Hi Adam, > Now that our test results are in line with yours, we can begin looking into setting up the production environment. Please, let me know if you need any help with it or waiting for an input from me. Regards, Andrew. On 10/10/23 17:09, Adam Hassick wrote: > Hi Andrew, > > Thank you for taking a look at our log. Netplan was attempting to run > DHCP on our test links, and additionally I discovered that the NIC > firmware was transmitting LLDP packets, causing tests to fail in the > same way. Now that these problems have been fixed, our pass rate on > the XL710 is approximately 91%. Now that our test results are in line > with yours, we can begin looking into setting up the production > environment. > > First, is it possible to run the test agent on ARM hosts? Our ARM > testbeds have the best topology for running this test suite, with > separate tester and DUT servers. > > We are testing this test suite on two x86 development servers using > the test suite's recommended server topology. In contrast, our > existing x86 production testbeds which run DTS have a single server > topology. This single server has both the tester NIC and the device > under test NIC installed, with NUMA node separation between TRex and > DPDK. We're going to test running the two test agent processes on the > single-server testbeds if we cannot run this on ARM. Is there any > reason you can think of that would prevent this setup from working? > > Once we figure out where this can live in production, then we will > begin setting up log storage, Jenkins integration, and Bublik. > > Thanks, > Adam > > On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko > wrote: > > Hi Adam, > > > Do these default to vfio-pci? > > Yes, vfio-pci is the default. > However, it does not work in the case of Mellanox which uses > bifurcated driver. It should mlx5_core for Mellanox NICs. > > > Here is the text log from a run on our Intel XL710 NICs, with > the expected result profile set to the X710. > > It is hard to analyze all tests using text logs, but I definitely > see one common problem. Tests receive unexpected packets and fail > because of it. > Tests are written very strict from this point of view and it > brought fruits in the past when HW had bugs. > Are DUT and tester connected back-to-back on tested interfaces or > via switch? > If via switch, is it possible to isolate it from everything else? > If back-to-back, it could be some embedded SW/FW which regenerates > these packet. > I definitely see unexpected DHCP packets. > > > We haven't set up the Jenkins integration yet, however if this > is required to import the logs then we will prioritize that. > > Unfortunately manual runs do not generate all artifacts required > to import logs. However, we have almost solved it right now. > Hopefully we'll finalize it in a day or two. I'll let you know > when these changes are available. > > Regards, > Andrew. > > On 10/4/23 16:48, Adam Hassick wrote: >> Hi Andrew, >> >> Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set >> anywhere in the default configurations for the Intel X710. Do >> these default to vfio-pci? >> We have IOMMU enabled on our development testbed, and should be >> able to bind vfio-pci. >> Here is the text log from a run on our Intel XL710 NICs, with the >> expected result profile set to the X710. We haven't set up the >> Jenkins integration yet, however if this is required to import >> the logs then we will prioritize that. >> log.txt.tar.gz >> >> >> Thanks, >> Adam >> >> On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko >> wrote: >> >> On 9/18/23 17:44, Adam Hassick wrote: >>> Hi Andrew and Konstantin, >>> >>> Thank you for adding the tester-dial feature, this opens up >>> the possibility for us to do CI integrated testing in the >>> future. >>> >>> Our Mellanox pass rate is similar to yours (about ~2400 >>> passing, ~4400 failing), however our Intel pass rates are >>> far worse. >>> I will try running tests on the XL710 with the trc-tags >>> argument set and see if it improves the pass rate. >>> Another thing I noticed in the results you uploaded is that >>> the results are tagged with vfio-pci and not i40e. >>> Though in the environment dump, the driver on the test >>> machine and the DUT are set to use the i40e driver. Is this >>> important at all? >> >> I think it is a misunderstanding here. There are two kinds of >> driver in configuration: net driver and so-called DPDK driver. >> Net driver is a Linux kernel network device driver used on >> Tester side. >> DPDK driver is a Linux kernel driver to bind device to to use >> it with DPDK. So, it is NOT a driver inside DPDK (drivers/net/*). >> In the case of bifurcated driver (like mlx5_core) it is the >> same in both cases. >> In non-bifurcated case DPDK driver is some UIO >> driver(vfio-pci, uio-pci-generic or igb_uio). >> Some expectations depend on used UIO. For example, >> uio-pci-generic do not support many interrupts (used by >> usecases/rx_intr test cases). >> That's why we care corresponding TRC tag. >> >> TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's >> Intel case. Or uio-pci-generic if IOMMU is turned off on >> corresponding machines and Linux distro does not support VFIO >> no IOMMU mode. >> >> Andrew. >> >>> There isn't anything preventing us from pushing our results >>> up to the existing Bublik instance running at ts-factory.io >>> that I can think of at the moment. >>> We will have to work out how to submit our results to your >>> Bublik instance in a controlled and secure manner in that case. >>> As far as I know we won't need access controls for the >>> results themselves. I'll discuss this with Patrick and will >>> let you know once we confirm that it's fine. >>> >>> Thanks, >>> Adam >>> >>> On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko >>> wrote: >>> >>> On 9/18/23 09:23, Konstantin Ushakov wrote: >>>> >>>> Hi Andrew, >>>> >>>> should we always auto-assign the tags or you don’t do >>>> it since it slows down (by some seconds) the TE startup? >>>> >>> >>> Tags are auto-assigned, but I guess it differs in Adam's >>> case since NIC is a bit different. Below test will help >>> to understand if it is the root cause of very different >>> expectations. If pass rate will be close to mine, I'll >>> simply update TRC database to share expectations for >>> mine NIC and NIC used by Adam. >>> >>>> Hi Adam, >>>> >>>> I think I second the question from Andrew - happy to >>>> help you with the triage so that we get to the same >>>> baseline. Do you have a good way for us to share the >>>> logs? I.e. say upload to ts-factory if we add strict >>>> permissions system so it’s not publishing or any other way. >>>> >>>> Thanks, >>>> Konstantin >>>> >>>> >>>> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote: >>>> >>>> Hi Adam, >>>> >>>> I've uploaded fresh testing results to >>>> ts-factory.io [1] to be on >>>> the same page. >>>> >>>> I think I know why your and mine results on Intel >>>> 710 series NICs differ so much. Testing results >>>> expectations database (dpdk-ethdev-ts/trc/*) is >>>> filled in in terms of TRC tags.  I.e. expectations >>>> depends on TRC tags discovered by helper scripts >>>> when testing is started. These tags identify >>>> various aspects of what is tested. Ideally >>>> expectations should be written in terms of root >>>> cause of the expected behaviour. If it is a driver >>>> expectations, driver tag should be used. If it is >>>> HW limitation, tags with PCI IDs should be used. >>>> However, it is not always easy to classify it >>>> correctly if you're not involved in driver >>>> development. So, in order case expectations for >>>> 710's Intel are filled in in terms of PCI IDs. I >>>> guess PCI ID differ in your case and that's why >>>> expectations filled in for my NIC do not apply to >>>> your runs. >>>> >>>> Just try to add the following option when you run >>>> on your 710's Intel in order to mimic mine and see >>>> if it helps to achieve better pass rate. >>>> --trc-tag=pci-8086-1572 >>>> >>>> BTW, fresh TE tag v1.21.0 has improved algorithm to >>>> choose tests for --tester-dial option. It should >>>> have better coverage now. >>>> >>>> Andrew. >>>> >>>> [1] >>>> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1 >>>> >>>> >>>> On 9/13/23 18:45, Andrew Rybchenko wrote: >>>>> Hi Adam, >>>>> >>>>> I've pushed new TE tag v1.20.0 which supported a >>>>> new command-line option --tester-dial=NUM where >>>>> NUM is from 0 to 100. it allows to choose >>>>> percentage of tests to run. If you want stable >>>>> set, you should pass --tester-random-seed=0 (or >>>>> other integer). It is the first sketch and we have >>>>> plans to improve it, but feedback would be welcome. >>>>> >>>>> > Is it needed on the tester? >>>>> >>>>> It is hard to say if it is strictly required for >>>>> simple tests. However, it is better to update >>>>> Tester as well, since performance tests run DPDK >>>>> on Tester as well. >>>>> >>>>> > Are there any other manual setup steps for these >>>>> devices that I might be missing? >>>>> >>>>> I don't remember anything else. >>>>> >>>>> I think it is better to get down to details and >>>>> take a look at logs. I'm ready to help with it and >>>>> explain what's happening there. May be it will >>>>> help to understand if it is a problem with >>>>> setup/configuration. >>>>> >>>>> Text logs are not very convenient. Ideally logs >>>>> should be imported to bublik, however, manual runs >>>>> do not provide all required artifacts right now >>>>> (Jenkins jobs generate all required artifacts). >>>>> Other option is 'tmp_raw_log' file (should be >>>>> packed to make it smaller) which could be >>>>> converted to various log formats. >>>>> Would it be OK for you if I import your logs to >>>>> bublik at ts-factory.io ? Or >>>>> is it a problem that it is publicly available? >>>>> Would it help if we add authentication and access >>>>> control there? >>>>> >>>>> Andrew. >>>>> >>>>> On 9/8/23 17:57, Adam Hassick wrote: >>>>>> Hi Andrew, >>>>>> >>>>>> I have a couple questions about needed setup of >>>>>> the NICs for the ethdev test suite. >>>>>> >>>>>> Our MCX5s and XL710s are failing the checkup >>>>>> tests. The pass rate appears to be much worse on >>>>>> the XL710s (40 of 73 tests failed, 3 passed >>>>>> unexpectedly). >>>>>> >>>>>> For the XL710s, I've updated the driver and NVM >>>>>> versions to match the minimum supported versions >>>>>> in the compatibility matrix found on the DPDK >>>>>> documentation. This did not change the failure >>>>>> rate much. >>>>>> For the MCX5s, I've installed the latest LTS >>>>>> version of the OFED bifurcated driver on the DUT. >>>>>> Is it needed on the tester? >>>>>> >>>>>> Are there any other manual setup steps for these >>>>>> devices that I might be missing? >>>>>> >>>>>> Thanks, >>>>>> Adam >>>>>> >>>>>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick >>>>>> wrote: >>>>>> >>>>>> Hi Andrew, >>>>>> >>>>>> Yes, I copied the X710 configs to set up >>>>>> XL710 configs. I changed the environment >>>>>> variable names from the X710 suffix to XL710 >>>>>> suffix in the script, and forgot to change >>>>>> them in the corresponding environment file. >>>>>> That fixed the issue. >>>>>> >>>>>> I got the checkup tests working on the XL710 >>>>>> now. Most of them are failing, which leads me >>>>>> to believe this is an issue with our testbed. >>>>>> Based on the DPDK documentation for i40e, the >>>>>> firmware and driver versions are much older >>>>>> than what DPDK 22.11 LTS and main prefer, so >>>>>> I'll try updating those. >>>>>> >>>>>> For now I'm working on getting the XL710 >>>>>> checkup tests passing, and will pick up >>>>>> getting the E810 configured properly next. >>>>>> I'll let you know if I run into any more >>>>>> issues in relation to the test engine. >>>>>> >>>>>> Thanks, >>>>>> Adam >>>>>> >>>>>> On Wed, Sep 6, 2023 at 7:36 AM Andrew >>>>>> Rybchenko wrote: >>>>>> >>>>>> Hi Adam, >>>>>> >>>>>> On 9/5/23 18:01, Adam Hassick wrote: >>>>>>> Hi Andrew, >>>>>>> >>>>>>> The compilation warning issue is now >>>>>>> resolved. Again, thank you guys for >>>>>>> fixing this for us. I can run the tests >>>>>>> on the Mellanox CX5s again, however I'm >>>>>>> running into a couple new issues with >>>>>>> running the prologues on the Intel cards. >>>>>>> >>>>>>> When running testing on the Intel >>>>>>> XL710s, I see this error appear in the log: >>>>>>> >>>>>>> ERROR  prologue  Environment LIB >>>>>>>  14:16:13.650 >>>>>>> Too few networks in available >>>>>>> configuration (0) in comparison with >>>>>>> required (1) >>>>>>> >>>>>>> >>>>>>> This seems like a trivial configuration >>>>>>> error, perhaps this is something I need >>>>>>> to set up in ts-rigs. I briefly searched >>>>>>> through the examples there and didn't >>>>>>> see any mention of how to set up a network. >>>>>>> I will attach this log just in case you >>>>>>> need more information. >>>>>> >>>>>> Unfortunately logs are insufficient to >>>>>> understand it. I've pushed new tag to TE >>>>>> v1.19.0 which add log message with TE_* >>>>>> environment variables. >>>>>> Most likely something is wrong with >>>>>> variables which are used as conditions >>>>>> when available networks are defined in >>>>>> ts-conf/cs/inc.net_cfg_pci_fns.yml: >>>>>> TE_PCI_INSTANCE_IUT_TST1 >>>>>> TE_PCI_INSTANCE_IUT_TST1a >>>>>> TE_PCI_INSTANCE_TST1a_IUT >>>>>> TE_PCI_INSTANCE_TST1_IUT >>>>>> My guess it that you change naming a bit, >>>>>> but script like >>>>>> ts-rigs-sample/scripts/iut.h1-x710 is not >>>>>> included or not updated. >>>>>> >>>>>>> There is a different error when running >>>>>>> on the Intel E810s. It appears to me >>>>>>> like it starts DPDK, does some >>>>>>> configuration inside DPDK and on the >>>>>>> device, and then fails to bring the >>>>>>> device back up. Since this error seems >>>>>>> very non-trivial, I will also attach >>>>>>> this log. >>>>>> >>>>>> This one is a bit simpler. Few lines >>>>>> after the first ERROR in log I see the >>>>>> following: >>>>>> WARN  RCF  DPDK 13:06:00.144 >>>>>> ice_program_hw_rx_queue(): currently >>>>>> package doesn't support RXDID (22) >>>>>> ice_rx_queue_start(): fail to program RX >>>>>> queue 0 >>>>>> ice_dev_start(): fail to start Rx queue 0 >>>>>> Device with port_id=0 already stopped >>>>>> >>>>>> It is stdout/stderr from test agent which >>>>>> runs DPDK. Same logs in plain format are >>>>>> available in ta.DPDK file. >>>>>> I'm not an expert here, but I vaguely >>>>>> remember that E810 requires correct >>>>>> firmware and DDP to be loaded. >>>>>> There is some information in >>>>>> dpdk/doc/guides/nics/ice.rst. >>>>>> >>>>>> You can try to add >>>>>> --dev-args=safe-mode-support=1 >>>>>> command-line option described there. >>>>>> >>>>>> Hope it helps, >>>>>> Andrew. >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Adam >>>>>>> >>>>>>> On Fri, Sep 1, 2023 at 3:59 AM Andrew >>>>>>> Rybchenko >>>>>>> wrote: >>>>>>> >>>>>>> Hi Adam, >>>>>>> >>>>>>> On 8/31/23 22:38, Adam Hassick wrote: >>>>>>>> Hi Andrew, >>>>>>>> >>>>>>>> I have one additional question as >>>>>>>> well: Does the test engine support >>>>>>>> running tests on two ARMv8 test agents? >>>>>>>> >>>>>>>> 1. We'll sort out warnings this >>>>>>>> week. Thanks for heads up. >>>>>>>> >>>>>>>> >>>>>>>> Great. Let me know when that's fixed. >>>>>>> >>>>>>> Done. We also fixed a number of >>>>>>> warnings in TE. >>>>>>> Also we fixed root test package name >>>>>>> to be consistent with the repository >>>>>>> name. >>>>>>> >>>>>>>> Support for old LTS branches >>>>>>>> was dropped some time ago, but >>>>>>>> in the future it is definitely >>>>>>>> possible to keep it for new LTS >>>>>>>> branches. I think 22.11 is >>>>>>>> supported, but I'm not sure >>>>>>>> about older LTS releases. >>>>>>>> >>>>>>>> >>>>>>>> Good to know. >>>>>>>> >>>>>>>> 2. You can add command-line >>>>>>>> option --sanity to run tests >>>>>>>> marked with TEST_HARNESS_SANITY >>>>>>>> requirement (see >>>>>>>> dpdk-ethdev-ts/scripts/run.sh >>>>>>>> and grep TEST_HARNESS_SANITY >>>>>>>> dpdk-ethdev-ts to see which >>>>>>>> tests are marked). Yes, there >>>>>>>> is a space for terminology >>>>>>>> improvement here. We'll do it. >>>>>>>> >>>>>>> >>>>>>> Done. Now it is called --checkup. >>>>>>> >>>>>>>> >>>>>>>> Also it takes a lot of time >>>>>>>> because of failures and tests >>>>>>>> which wait for some timeout. >>>>>>>> >>>>>>>> >>>>>>>> That makes sense to me. We'll use >>>>>>>> the time to complete tests on >>>>>>>> virtio or the Intel devices as a >>>>>>>> reference for how long the tests >>>>>>>> really take to complete. >>>>>>>> We will explore the possibility of >>>>>>>> periodically running the sanity >>>>>>>> tests for patches. >>>>>>> >>>>>>> I'll double-check and let you know >>>>>>> how long entire TS runs on Intel >>>>>>> X710, E810, Mellanox CX5 and virtio >>>>>>> net. Just to ensure that time >>>>>>> observed in your case looks the same. >>>>>>> >>>>>>>> >>>>>>>> The test harness can provide >>>>>>>> coverage reports based on gcov, >>>>>>>> but I'm not sure what you mean >>>>>>>> by a "dial" to control test >>>>>>>> coverage. Provided reports are >>>>>>>> rather for human to analyze. >>>>>>>> >>>>>>>> >>>>>>>> The general idea is to have some >>>>>>>> kind of parameter on the test >>>>>>>> suite, which could be an integer >>>>>>>> ranging from zero to ten, that >>>>>>>> controls how many tests are run >>>>>>>> based on how important the test is. >>>>>>>> >>>>>>>> Similar to how some command line >>>>>>>> interfaces provide a verbosity >>>>>>>> level parameter (some number of >>>>>>>> "-v" arguments) to control the >>>>>>>> importance of the information in >>>>>>>> the log. >>>>>>>> The verbosity level zero only >>>>>>>> prints very important log messages, >>>>>>>> while ten prints everything. >>>>>>>> >>>>>>>> In much the same manner as above, >>>>>>>> this "dial" parameter controls what >>>>>>>> tests are run and with what >>>>>>>> parameters based on how important >>>>>>>> those tests and test parameter >>>>>>>> combinations are. >>>>>>>> Coverage Level zero tells the suite >>>>>>>> to run a very basic set of >>>>>>>> important tests, with minimal >>>>>>>> parameterization. This mode would >>>>>>>> take only ~5-10 minutes to run. >>>>>>>> In contrast, Coverage Level ten >>>>>>>> includes all the edge cases, every >>>>>>>> combination of test parameters, >>>>>>>> everything the test suite can do, >>>>>>>> which takes the normal several >>>>>>>> hours to run. >>>>>>>> The values 1 - 9 are between those >>>>>>>> two extremes, allowing the user to >>>>>>>> get a gradient of test coverage in >>>>>>>> the results and to limit the >>>>>>>> running time. >>>>>>>> >>>>>>>> Then we could, for example, run the >>>>>>>> "run.sh" with a level of 2 or 3 for >>>>>>>> incoming patches that need quick >>>>>>>> results, and with a level of 10 for >>>>>>>> the less often run periodic tests >>>>>>>> performed on main or LTS branches. >>>>>>> >>>>>>> Understood now. Thanks a lot for the >>>>>>> idea. We'll discuss it and come back. >>>>>>> >>>>>>>> 3. Yes, really many tests on >>>>>>>> Mellanox CX5 NICs report >>>>>>>> unexpected testing results. >>>>>>>> Unfortunately it is time >>>>>>>> consuming to fill in >>>>>>>> expectations database since it >>>>>>>> is necessary to analyze testing >>>>>>>> results and classify if it is a >>>>>>>> bug or just acceptable >>>>>>>> behaviour aspect. >>>>>>>> >>>>>>>> Bublik allows to compare >>>>>>>> results of two runs. It is >>>>>>>> useful for human, but still not >>>>>>>> good for automation. >>>>>>>> >>>>>>>> I have local patch for mlx5 >>>>>>>> driver which reports Tx ring >>>>>>>> size maximum. It makes pass >>>>>>>> rate higher. It is a problem >>>>>>>> for test harness that mlx5 does >>>>>>>> not report limits right now. >>>>>>>> >>>>>>>> Pass rate on Intel X710 is >>>>>>>> about 92% on my test rig. Pass >>>>>>>> rate on virtio net is 99% right >>>>>>>> now and could be done 100% >>>>>>>> easily (just one thing to fix >>>>>>>> in expectations). >>>>>>>> >>>>>>>> I think logs storage setup is >>>>>>>> essential for logs analysis. Of >>>>>>>> course, you can request HTML >>>>>>>> logs when you run tests >>>>>>>> (--log-html=html) or generate >>>>>>>> after run using >>>>>>>> dpdk-ethdev-ts/scripts/html-log.sh >>>>>>>> and open index.html in a >>>>>>>> browser, but logs storage makes >>>>>>>> it more convenient. >>>>>>>> >>>>>>>> >>>>>>>> We are interested in setting up >>>>>>>> Bublik, potentially as an >>>>>>>> externally-facing component, once >>>>>>>> we have our process of running the >>>>>>>> test suite stabilized. >>>>>>>> Once we are able to run the test >>>>>>>> suite again, I'll see what the pass >>>>>>>> rate is on our other hardware. >>>>>>>> Good to know that it isn't an issue >>>>>>>> with our dev testbed causing the >>>>>>>> high fail rate. >>>>>>>> >>>>>>>> For Intel hardware, we have an >>>>>>>> XL710 and an Intel E810-C in our >>>>>>>> development testbed. Although they >>>>>>>> are slightly different devices, >>>>>>>> ideally the pass rate will be >>>>>>>> identical or similar. I have yet to >>>>>>>> set up a VM pair for virtio, but we >>>>>>>> will soon. >>>>>>>> >>>>>>>> Latest version of >>>>>>>> test-environment has examples >>>>>>>> of our CGI scripts which we use >>>>>>>> for log storage (see >>>>>>>> tools/log_server/README.md). >>>>>>>> >>>>>>>> Also all bits for Jenkins setup >>>>>>>> are available. See >>>>>>>> dpdk-ethdev-ts/jenkins/README.md >>>>>>>> and examples of jenkins files >>>>>>>> in ts-rigs-sample. >>>>>>>> >>>>>>>> >>>>>>>> Jenkins integration, setting up >>>>>>>> production rig configurations, and >>>>>>>> permanent log storage will be our >>>>>>>> next steps once I am able to run >>>>>>>> the tests again. >>>>>>>> Unless there is an easy way to have >>>>>>>> meson not pass "-Werror" into GCC. >>>>>>>> Then I would be able to run the >>>>>>>> test suite. >>>>>>> >>>>>>> Hopefully it is resolved now. >>>>>>> >>>>>>> I thought a bit more about your >>>>>>> usecase for Jenkins. I'm not 100% >>>>>>> sure that existing pipelines are >>>>>>> convenient for your usecase. >>>>>>> Fill free to ask questions when you >>>>>>> are on it. >>>>>>> >>>>>>> Thanks, >>>>>>> Andrew. >>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Adam >>>>>>>> >>>>>>>> >>>>>>>> On 8/29/23 17:02, Adam Hassick >>>>>>>> wrote: >>>>>>>>> Hi Andrew, >>>>>>>>> >>>>>>>>> That fix seems to have >>>>>>>>> resolved the issue, thanks for >>>>>>>>> the quick turnaround time on >>>>>>>>> that patch. >>>>>>>>> Now that we have the RCF >>>>>>>>> timeout issue resolved, there >>>>>>>>> are a few other questions and >>>>>>>>> issues that we have about the >>>>>>>>> tests themselves. >>>>>>>>> >>>>>>>>> 1. The test suite fails to >>>>>>>>> build with a couple warnings. >>>>>>>>> >>>>>>>>> Below is the stderr log from >>>>>>>>> compilation: >>>>>>>>> >>>>>>>>> FAILED: >>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o >>>>>>>>> cc >>>>>>>>> -Ilib/76b5a35@@ts_dpdk_pmd@sta >>>>>>>>> -Ilib -I../../lib >>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include >>>>>>>>> -fdiagnostics-color=always >>>>>>>>> -pipe >>>>>>>>> -D_FILE_OFFSET_BITS=64 >>>>>>>>> -Wall -Winvalid-pch >>>>>>>>> -Werror -g -D_GNU_SOURCE >>>>>>>>> -O0 -ggdb -Wall -W -fPIC >>>>>>>>> -MD -MQ >>>>>>>>> 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' >>>>>>>>> -MF >>>>>>>>> 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' >>>>>>>>> -o >>>>>>>>> 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' >>>>>>>>> -c ../../lib/dpdk_pmd_ts.c >>>>>>>>> ../../lib/dpdk_pmd_ts.c: >>>>>>>>> In function >>>>>>>>> ‘test_create_traffic_generator_params’: >>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: >>>>>>>>> error: format not a string >>>>>>>>> literal and no format >>>>>>>>> arguments >>>>>>>>> [-Werror=format-security] >>>>>>>>> 5577 |     rc = >>>>>>>>> te_kvpair_add(result, buf, >>>>>>>>> mode); >>>>>>>>> |     ^~ >>>>>>>>> cc1: all warnings being >>>>>>>>> treated as errors >>>>>>>>> ninja: build stopped: >>>>>>>>> subcommand failed. >>>>>>>>> ninja: Entering directory `.' >>>>>>>>> FAILED: >>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o >>>>>>>>> cc >>>>>>>>> -Ilib/76b5a35@@ts_dpdk_pmd@sta >>>>>>>>> -Ilib -I../../lib >>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include >>>>>>>>> -fdiagnostics-color=always >>>>>>>>> -pipe >>>>>>>>> -D_FILE_OFFSET_BITS=64 >>>>>>>>> -Wall -Winvalid-pch >>>>>>>>> -Werror -g -D_GNU_SOURCE >>>>>>>>> -O0 -ggdb -Wall -W -fPIC >>>>>>>>> -MD -MQ >>>>>>>>> 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' >>>>>>>>> -MF >>>>>>>>> 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' >>>>>>>>> -o >>>>>>>>> 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' >>>>>>>>> -c ../../lib/dpdk_pmd_ts.c >>>>>>>>> ../../lib/dpdk_pmd_ts.c: >>>>>>>>> In function >>>>>>>>> ‘test_create_traffic_generator_params’: >>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: >>>>>>>>> error: format not a string >>>>>>>>> literal and no format >>>>>>>>> arguments >>>>>>>>> [-Werror=format-security] >>>>>>>>> 5577 |     rc = >>>>>>>>> te_kvpair_add(result, buf, >>>>>>>>> mode); >>>>>>>>> |     ^~ >>>>>>>>> cc1: all warnings being >>>>>>>>> treated as errors >>>>>>>>> >>>>>>>>> >>>>>>>>> This error wasn't occurring >>>>>>>>> last week, which was the last >>>>>>>>> time I ran the tests. >>>>>>>>> The TE host and the DUT have >>>>>>>>> GCC v9.4.0 installed, and the >>>>>>>>> tester has GCC v11.4.0 >>>>>>>>> installed, if this information >>>>>>>>> is helpful. >>>>>>>>> >>>>>>>>> 2. On the Mellanox CX5s, there >>>>>>>>> are over 6,000 tests run, >>>>>>>>> which collectively take around >>>>>>>>> 9 hours. Is it possible, and >>>>>>>>> would it make sense, to lower >>>>>>>>> the test coverage and have the >>>>>>>>> test suite run faster? >>>>>>>>> >>>>>>>>> For some context, we run >>>>>>>>> immediate testing on incoming >>>>>>>>> patches for DPDK main and >>>>>>>>> development branches, as well >>>>>>>>> as periodic test runs on the >>>>>>>>> main, stable, and LTS branches. >>>>>>>>> For us to consider including >>>>>>>>> this test suite as part of our >>>>>>>>> immediate testing on patches, >>>>>>>>> we would have to reduce the >>>>>>>>> test coverage to the most >>>>>>>>> important tests. >>>>>>>>> This is primarily to reduce >>>>>>>>> the testing time to, for >>>>>>>>> example, less than 30 minutes. >>>>>>>>> Testing on patches can't take >>>>>>>>> too long because the lab can >>>>>>>>> receive numerous patches each >>>>>>>>> day, which each require >>>>>>>>> individual testing runs. >>>>>>>>> >>>>>>>>> At what frequency we run these >>>>>>>>> tests, and on what, still >>>>>>>>> needs to be discussed with the >>>>>>>>> DPDK community, but it would >>>>>>>>> be nice to know if the test >>>>>>>>> suite had a "dial" to control >>>>>>>>> the testing coverage. >>>>>>>>> >>>>>>>>> 3. We see a lot of test >>>>>>>>> failures on our Mellanox CX5 >>>>>>>>> NICs. Around 2,300 of ~6,600 >>>>>>>>> tests passed. Is there >>>>>>>>> anything we can do to diagnose >>>>>>>>> these test failures? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Adam >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Aug 29, 2023 at >>>>>>>>> 8:07 AM Andrew Rybchenko >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Adam, >>>>>>>>> >>>>>>>>> I've pushed the fix in >>>>>>>>> main branch and a new tag >>>>>>>>> v1.18.1. It should solve >>>>>>>>> the problem with IPv6 >>>>>>>>> address from DNS. >>>>>>>>> >>>>>>>>> Andrew. >>>>>>>>> >>>>>>>>> On 8/29/23 00:05, Andrew >>>>>>>>> Rybchenko wrote: >>>>>>>>>> Hi Adam, >>>>>>>>>> >>>>>>>>>> > Does the test engine >>>>>>>>>> prefer to use IPv6 over >>>>>>>>>> IPv4 for initiating the >>>>>>>>>> RCF connection to the >>>>>>>>>> test bed hosts? And if >>>>>>>>>> so, is there a way to >>>>>>>>>> force it to use IPv4? >>>>>>>>>> >>>>>>>>>> Brilliant idea. If DNS >>>>>>>>>> returns both IPv4 and >>>>>>>>>> IPv6 addresses in your >>>>>>>>>> case, I guess it is the >>>>>>>>>> root cause of the problem. >>>>>>>>>> Of course, it is TE >>>>>>>>>> problem since I see >>>>>>>>>> really weird code in >>>>>>>>>> lib/comm_net_engine/comm_net_engine.c >>>>>>>>>> line 135. >>>>>>>>>> >>>>>>>>>> I've pushed fix to the >>>>>>>>>> branch >>>>>>>>>> user/arybchik/fix_ipv4_only >>>>>>>>>> in >>>>>>>>>> ts-factory/test-environment >>>>>>>>>> repository. Please, try. >>>>>>>>>> >>>>>>>>>> It is late night fix with >>>>>>>>>> minimal testing and no >>>>>>>>>> review. I'll pass it >>>>>>>>>> through review process >>>>>>>>>> tomorrow and >>>>>>>>>> hopefully it will be >>>>>>>>>> released in one-two days. >>>>>>>>>> >>>>>>>>>> Andrew. >>>>>>>>>> >>>>>>>>>> On 8/28/23 18:02, Adam >>>>>>>>>> Hassick wrote: >>>>>>>>>>> Hi Andrew, >>>>>>>>>>> >>>>>>>>>>> We have yet to notice a >>>>>>>>>>> distinct pattern with >>>>>>>>>>> the failures. Sometimes, >>>>>>>>>>> the RCF will start and >>>>>>>>>>> connect without issue a >>>>>>>>>>> few times in a row >>>>>>>>>>> before failing to >>>>>>>>>>> connect again. Once the >>>>>>>>>>> issue begins to occur, >>>>>>>>>>> neither rebooting all of >>>>>>>>>>> the hosts (test engine >>>>>>>>>>> VM, tester, IUT) or >>>>>>>>>>> deleting all of the >>>>>>>>>>> build directories >>>>>>>>>>> (suites, agents, inst) >>>>>>>>>>> and rebooting the hosts >>>>>>>>>>> afterward resolves the >>>>>>>>>>> issue. When it begins >>>>>>>>>>> working again seems very >>>>>>>>>>> arbitrary to us. >>>>>>>>>>> >>>>>>>>>>> I do usually try to >>>>>>>>>>> terminate the test >>>>>>>>>>> engine with Ctrl+C, but >>>>>>>>>>> when it hangs while >>>>>>>>>>> trying to start RCF, >>>>>>>>>>> that does not work. >>>>>>>>>>> >>>>>>>>>>> Does the test engine >>>>>>>>>>> prefer to use IPv6 over >>>>>>>>>>> IPv4 for initiating the >>>>>>>>>>> RCF connection to the >>>>>>>>>>> test bed hosts? And if >>>>>>>>>>> so, is there a way to >>>>>>>>>>> force it to use IPv4? >>>>>>>>>>> >>>>>>>>>>>  - Adam >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 25, 2023 at >>>>>>>>>>> 1:35 PM Andrew Rybchenko >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> > I'll double-check >>>>>>>>>>> test engine on >>>>>>>>>>> Ubuntu 20.04 and >>>>>>>>>>> Ubuntu 22.04. >>>>>>>>>>> >>>>>>>>>>> Done. It works fine >>>>>>>>>>> for me without any >>>>>>>>>>> issues. >>>>>>>>>>> >>>>>>>>>>> Have you noticed any >>>>>>>>>>> pattern when it >>>>>>>>>>> works or does not work? >>>>>>>>>>> May be it is a >>>>>>>>>>> problem of not clean >>>>>>>>>>> state after termination? >>>>>>>>>>> Does it work fine >>>>>>>>>>> the first time after >>>>>>>>>>> DUTs reboot? >>>>>>>>>>> How do you terminate >>>>>>>>>>> testing? It should >>>>>>>>>>> be done using Ctrl+C >>>>>>>>>>> in terminal where >>>>>>>>>>> you execute run.sh >>>>>>>>>>> command. >>>>>>>>>>>  In this case it >>>>>>>>>>> should shutdown >>>>>>>>>>> gracefully and close >>>>>>>>>>> all test agents and >>>>>>>>>>> engine applications. >>>>>>>>>>> >>>>>>>>>>> (I'm trying to >>>>>>>>>>> understand why >>>>>>>>>>> you've seen many >>>>>>>>>>> test agent >>>>>>>>>>> processes. It should >>>>>>>>>>> not happen.) >>>>>>>>>>> >>>>>>>>>>> Andrew. >>>>>>>>>>> >>>>>>>>>>> On 8/25/23 17:41, >>>>>>>>>>> Andrew Rybchenko wrote: >>>>>>>>>>>> On 8/25/23 17:06, >>>>>>>>>>>> Adam Hassick wrote: >>>>>>>>>>>>> Hi Andrew, >>>>>>>>>>>>> >>>>>>>>>>>>> Two of our systems >>>>>>>>>>>>> (the Test Engine >>>>>>>>>>>>> runner and the DUT >>>>>>>>>>>>> host) are running >>>>>>>>>>>>> Ubuntu 20.04 LTS, >>>>>>>>>>>>> however this >>>>>>>>>>>>> morning I noticed >>>>>>>>>>>>> that the tester >>>>>>>>>>>>> system (the one >>>>>>>>>>>>> having issues) is >>>>>>>>>>>>> running Ubuntu >>>>>>>>>>>>> 22.04 LTS. >>>>>>>>>>>>> This could be the >>>>>>>>>>>>> source of the >>>>>>>>>>>>> problem. I >>>>>>>>>>>>> encountered a >>>>>>>>>>>>> dependency issue >>>>>>>>>>>>> trying to run the >>>>>>>>>>>>> Test Engine on >>>>>>>>>>>>> 22.04 LTS, so I >>>>>>>>>>>>> downgraded the >>>>>>>>>>>>> system. Since the >>>>>>>>>>>>> tester is also the >>>>>>>>>>>>> host having >>>>>>>>>>>>> connection issues, >>>>>>>>>>>>> I will try >>>>>>>>>>>>> downgrading that >>>>>>>>>>>>> system to 20.04, >>>>>>>>>>>>> and see if that >>>>>>>>>>>>> changes anything. >>>>>>>>>>>> >>>>>>>>>>>> Unlikely, but who >>>>>>>>>>>> knows. We run tests >>>>>>>>>>>> (DUTs) on Ubuntu >>>>>>>>>>>> 20.04, Ubuntu >>>>>>>>>>>> 22.04, Ubuntu >>>>>>>>>>>> 22.10, Ubuntu >>>>>>>>>>>> 23.04, Debian 11 >>>>>>>>>>>> and Fedora 38 every >>>>>>>>>>>> night. >>>>>>>>>>>> Right now Debian 11 >>>>>>>>>>>> is used for test >>>>>>>>>>>> engine in nightly >>>>>>>>>>>> regressions. >>>>>>>>>>>> >>>>>>>>>>>> I'll double-check >>>>>>>>>>>> test engine on >>>>>>>>>>>> Ubuntu 20.04 and >>>>>>>>>>>> Ubuntu 22.04. >>>>>>>>>>>> >>>>>>>>>>>>> I did try passing >>>>>>>>>>>>> in the "--vg-rcf" >>>>>>>>>>>>> argument to the >>>>>>>>>>>>> run.sh script of >>>>>>>>>>>>> the test suite >>>>>>>>>>>>> after installing >>>>>>>>>>>>> valgrind, but >>>>>>>>>>>>> there was no >>>>>>>>>>>>> additional output >>>>>>>>>>>>> that I saw. >>>>>>>>>>>> >>>>>>>>>>>> Sorry, I should >>>>>>>>>>>> valgrind output >>>>>>>>>>>> should be in >>>>>>>>>>>> valgrind.te_rcf >>>>>>>>>>>> (direction where >>>>>>>>>>>> you run test engine). >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I will try pulling >>>>>>>>>>>>> in the changes >>>>>>>>>>>>> you've pushed up, >>>>>>>>>>>>> and will see if >>>>>>>>>>>>> that fixes anything. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Adam >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 25, >>>>>>>>>>>>> 2023 at 9:57 AM >>>>>>>>>>>>> Andrew Rybchenko >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hello Adam, >>>>>>>>>>>>> >>>>>>>>>>>>> On 8/24/23 >>>>>>>>>>>>> 23:54, Andrew >>>>>>>>>>>>> Rybchenko wrote: >>>>>>>>>>>>>> I'd like to >>>>>>>>>>>>>> try to repeat >>>>>>>>>>>>>> the problem >>>>>>>>>>>>>> locally. >>>>>>>>>>>>>> Which Linux >>>>>>>>>>>>>> distro is >>>>>>>>>>>>>> running on >>>>>>>>>>>>>> test engine >>>>>>>>>>>>>> and agents? >>>>>>>>>>>>>> >>>>>>>>>>>>>> In fact I >>>>>>>>>>>>>> know one >>>>>>>>>>>>>> problem with >>>>>>>>>>>>>> Debian 12 and >>>>>>>>>>>>>> Fedora 38 and >>>>>>>>>>>>>> we have >>>>>>>>>>>>>> patch in >>>>>>>>>>>>>> review to fix >>>>>>>>>>>>>> it, however, >>>>>>>>>>>>>> the behaviour >>>>>>>>>>>>>> is different in >>>>>>>>>>>>>> this case, so >>>>>>>>>>>>>> it is unlike >>>>>>>>>>>>>> the same problem. >>>>>>>>>>>>> >>>>>>>>>>>>> I've just >>>>>>>>>>>>> published a >>>>>>>>>>>>> new tag which >>>>>>>>>>>>> fixes known >>>>>>>>>>>>> test engine >>>>>>>>>>>>> side problems >>>>>>>>>>>>> on Debian 12 >>>>>>>>>>>>> and Fedora 38. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> One more idea >>>>>>>>>>>>>> is to install >>>>>>>>>>>>>> valgrind on >>>>>>>>>>>>>> the test >>>>>>>>>>>>>> engine host and >>>>>>>>>>>>>> run with >>>>>>>>>>>>>> option >>>>>>>>>>>>>> --vg-rcf to >>>>>>>>>>>>>> check if >>>>>>>>>>>>>> something >>>>>>>>>>>>>> weird is >>>>>>>>>>>>>> happening. >>>>>>>>>>>>>> >>>>>>>>>>>>>> What I don't >>>>>>>>>>>>>> understand >>>>>>>>>>>>>> right now is >>>>>>>>>>>>>> why I see >>>>>>>>>>>>>> just one >>>>>>>>>>>>>> failed attempt >>>>>>>>>>>>>> to connect in >>>>>>>>>>>>>> your log.txt >>>>>>>>>>>>>> and then >>>>>>>>>>>>>> Logger >>>>>>>>>>>>>> shutdown >>>>>>>>>>>>>> after 9 minutes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Andrew. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 8/24/23 >>>>>>>>>>>>>> 23:29, Adam >>>>>>>>>>>>>> Hassick wrote: >>>>>>>>>>>>>>>  > Is there >>>>>>>>>>>>>>> any firewall >>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>> network or >>>>>>>>>>>>>>> on test >>>>>>>>>>>>>>> hosts which >>>>>>>>>>>>>>> could block >>>>>>>>>>>>>>> incoming TCP >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> to the port >>>>>>>>>>>>>>> 23571 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> from the >>>>>>>>>>>>>>> host where >>>>>>>>>>>>>>> you run test >>>>>>>>>>>>>>> engine? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Our test >>>>>>>>>>>>>>> engine host >>>>>>>>>>>>>>> and the >>>>>>>>>>>>>>> testbed are >>>>>>>>>>>>>>> on the same >>>>>>>>>>>>>>> subnet. The >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> does work >>>>>>>>>>>>>>> sometimes. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>  > If >>>>>>>>>>>>>>> behaviour >>>>>>>>>>>>>>> the same on >>>>>>>>>>>>>>> the next try >>>>>>>>>>>>>>> and you see >>>>>>>>>>>>>>> that test >>>>>>>>>>>>>>> agent is >>>>>>>>>>>>>>> kept >>>>>>>>>>>>>>> running, >>>>>>>>>>>>>>> could you >>>>>>>>>>>>>>> check using >>>>>>>>>>>>>>>  > >>>>>>>>>>>>>>>  > # netstat >>>>>>>>>>>>>>> -tnlp >>>>>>>>>>>>>>>  > >>>>>>>>>>>>>>>  > that Test >>>>>>>>>>>>>>> Agent is >>>>>>>>>>>>>>> listening on >>>>>>>>>>>>>>> the port and >>>>>>>>>>>>>>> try to >>>>>>>>>>>>>>> establish >>>>>>>>>>>>>>> TCP >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> from test >>>>>>>>>>>>>>> agent using >>>>>>>>>>>>>>>  > >>>>>>>>>>>>>>>  > $ telnet >>>>>>>>>>>>>>> iol-dts-tester.dpdklab.iol.unh.edu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 23571 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>  > >>>>>>>>>>>>>>>  > and check >>>>>>>>>>>>>>> if TCP >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> could be >>>>>>>>>>>>>>> established. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I was able >>>>>>>>>>>>>>> to replicate >>>>>>>>>>>>>>> the same >>>>>>>>>>>>>>> behavior >>>>>>>>>>>>>>> again, where >>>>>>>>>>>>>>> it hangs >>>>>>>>>>>>>>> while RCF is >>>>>>>>>>>>>>> trying to start. >>>>>>>>>>>>>>> Running this >>>>>>>>>>>>>>> command, I >>>>>>>>>>>>>>> see this in >>>>>>>>>>>>>>> the output: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> tcp        0 >>>>>>>>>>>>>>>    0 >>>>>>>>>>>>>>> 0.0.0.0:23571 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 0.0.0.0:* >>>>>>>>>>>>>>> LISTEN  18599/ta >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So it seems >>>>>>>>>>>>>>> like it is >>>>>>>>>>>>>>> listening on >>>>>>>>>>>>>>> the correct >>>>>>>>>>>>>>> port. >>>>>>>>>>>>>>> Additionally, >>>>>>>>>>>>>>> I was able >>>>>>>>>>>>>>> to connect >>>>>>>>>>>>>>> to the >>>>>>>>>>>>>>> Tester >>>>>>>>>>>>>>> machine from >>>>>>>>>>>>>>> our Test >>>>>>>>>>>>>>> Engine host >>>>>>>>>>>>>>> using >>>>>>>>>>>>>>> telnet. It >>>>>>>>>>>>>>> printed the >>>>>>>>>>>>>>> PID of the >>>>>>>>>>>>>>> process once >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> was opened. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I tried >>>>>>>>>>>>>>> running the >>>>>>>>>>>>>>> "ta" >>>>>>>>>>>>>>> application >>>>>>>>>>>>>>> manually on >>>>>>>>>>>>>>> the command >>>>>>>>>>>>>>> line, and it >>>>>>>>>>>>>>> didn't print >>>>>>>>>>>>>>> anything at all. >>>>>>>>>>>>>>> Maybe the >>>>>>>>>>>>>>> issue is >>>>>>>>>>>>>>> something on >>>>>>>>>>>>>>> the Test >>>>>>>>>>>>>>> Engine side. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Aug >>>>>>>>>>>>>>> 24, 2023 at >>>>>>>>>>>>>>> 2:35 PM >>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>> Rybchenko >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     Hi Adam, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>      > On >>>>>>>>>>>>>>> the tester >>>>>>>>>>>>>>> host (which >>>>>>>>>>>>>>> appears to >>>>>>>>>>>>>>> be the Peer >>>>>>>>>>>>>>> agent), there >>>>>>>>>>>>>>>     are four >>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>> that I see >>>>>>>>>>>>>>> running, >>>>>>>>>>>>>>> which look >>>>>>>>>>>>>>> like the test >>>>>>>>>>>>>>>     agent >>>>>>>>>>>>>>> processes. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     Before >>>>>>>>>>>>>>> the next try >>>>>>>>>>>>>>> I'd >>>>>>>>>>>>>>> recommend to >>>>>>>>>>>>>>> kill these >>>>>>>>>>>>>>> processes. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     Is there >>>>>>>>>>>>>>> any firewall >>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>> network or >>>>>>>>>>>>>>> on test >>>>>>>>>>>>>>> hosts which >>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>     block >>>>>>>>>>>>>>> incoming TCP >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> to the port >>>>>>>>>>>>>>> 23571 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> from the host >>>>>>>>>>>>>>>     where >>>>>>>>>>>>>>> you run test >>>>>>>>>>>>>>> engine? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     If >>>>>>>>>>>>>>> behaviour >>>>>>>>>>>>>>> the same on >>>>>>>>>>>>>>> the next try >>>>>>>>>>>>>>> and you see >>>>>>>>>>>>>>> that test >>>>>>>>>>>>>>> agent is >>>>>>>>>>>>>>>     kept >>>>>>>>>>>>>>> running, >>>>>>>>>>>>>>> could you >>>>>>>>>>>>>>> check using >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     # >>>>>>>>>>>>>>> netstat -tnlp >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     that >>>>>>>>>>>>>>> Test Agent >>>>>>>>>>>>>>> is listening >>>>>>>>>>>>>>> on the port >>>>>>>>>>>>>>> and try to >>>>>>>>>>>>>>> establish TCP >>>>>>>>>>>>>>>     >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> from test >>>>>>>>>>>>>>> agent using >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     $ telnet >>>>>>>>>>>>>>> iol-dts-tester.dpdklab.iol.unh.edu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 23571 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     and >>>>>>>>>>>>>>> check if TCP >>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>> could be >>>>>>>>>>>>>>> established. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     Another >>>>>>>>>>>>>>> idea is to >>>>>>>>>>>>>>> login Tester >>>>>>>>>>>>>>> under root >>>>>>>>>>>>>>> as testing >>>>>>>>>>>>>>> does, get >>>>>>>>>>>>>>>     start TA >>>>>>>>>>>>>>> command from >>>>>>>>>>>>>>> the log and >>>>>>>>>>>>>>> try it by >>>>>>>>>>>>>>> hands >>>>>>>>>>>>>>> without -n and >>>>>>>>>>>>>>>     remove >>>>>>>>>>>>>>> extra escaping. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     # sudo >>>>>>>>>>>>>>> PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1 >>>>>>>>>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 >>>>>>>>>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta >>>>>>>>>>>>>>> Peer 23571 >>>>>>>>>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell= >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     >>>>>>>>>>>>>>> Hopefully in >>>>>>>>>>>>>>> this case >>>>>>>>>>>>>>> test agent >>>>>>>>>>>>>>> directory >>>>>>>>>>>>>>> remains in >>>>>>>>>>>>>>> the /tmp and >>>>>>>>>>>>>>>     you >>>>>>>>>>>>>>> don't need >>>>>>>>>>>>>>> to copy it >>>>>>>>>>>>>>> as testing does. >>>>>>>>>>>>>>>     May be >>>>>>>>>>>>>>> output could >>>>>>>>>>>>>>> shed some >>>>>>>>>>>>>>> light on >>>>>>>>>>>>>>> what's going on. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     Andrew. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>     On >>>>>>>>>>>>>>> 8/24/23 >>>>>>>>>>>>>>> 17:30, Adam >>>>>>>>>>>>>>> Hassick wrote: >>>>>>>>>>>>>>>> Hi Andrew, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     This is >>>>>>>>>>>>>>>> the output >>>>>>>>>>>>>>>> that I see >>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>> terminal >>>>>>>>>>>>>>>> when this >>>>>>>>>>>>>>>> failure >>>>>>>>>>>>>>>>     occurs, >>>>>>>>>>>>>>>> after the >>>>>>>>>>>>>>>> test agent >>>>>>>>>>>>>>>> binaries >>>>>>>>>>>>>>>> build and >>>>>>>>>>>>>>>> the test engine >>>>>>>>>>>>>>>>     starts: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     >>>>>>>>>>>>>>>> Platform >>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>> build - pass >>>>>>>>>>>>>>>>     Simple >>>>>>>>>>>>>>>> RCF >>>>>>>>>>>>>>>> consistency >>>>>>>>>>>>>>>> check succeeded >>>>>>>>>>>>>>>> --->>> >>>>>>>>>>>>>>>> Starting >>>>>>>>>>>>>>>> Logger...done >>>>>>>>>>>>>>>> --->>> >>>>>>>>>>>>>>>> Starting >>>>>>>>>>>>>>>> RCF...rcf_net_engine_connect(): >>>>>>>>>>>>>>>> Connection >>>>>>>>>>>>>>>> timed >>>>>>>>>>>>>>>>     out >>>>>>>>>>>>>>>> iol-dts-tester.dpdklab.iol.unh.edu:23571 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     Then, >>>>>>>>>>>>>>>> it hangs >>>>>>>>>>>>>>>> here until >>>>>>>>>>>>>>>> I kill the >>>>>>>>>>>>>>>> "te_rcf" >>>>>>>>>>>>>>>> and "te_tee" >>>>>>>>>>>>>>>>     >>>>>>>>>>>>>>>> processes. >>>>>>>>>>>>>>>> I let it >>>>>>>>>>>>>>>> hang for >>>>>>>>>>>>>>>> around 9 >>>>>>>>>>>>>>>> minutes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     On the >>>>>>>>>>>>>>>> tester host >>>>>>>>>>>>>>>> (which >>>>>>>>>>>>>>>> appears to >>>>>>>>>>>>>>>> be the Peer >>>>>>>>>>>>>>>> agent), >>>>>>>>>>>>>>>> there are >>>>>>>>>>>>>>>>     four >>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>> that I see >>>>>>>>>>>>>>>> running, >>>>>>>>>>>>>>>> which look >>>>>>>>>>>>>>>> like the >>>>>>>>>>>>>>>> test agent >>>>>>>>>>>>>>>>     processes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     ta.Peer >>>>>>>>>>>>>>>> is an empty >>>>>>>>>>>>>>>> file. I've >>>>>>>>>>>>>>>> attached >>>>>>>>>>>>>>>> the log.txt >>>>>>>>>>>>>>>> from this run. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>      - Adam >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     On Thu, >>>>>>>>>>>>>>>> Aug 24, >>>>>>>>>>>>>>>> 2023 at >>>>>>>>>>>>>>>> 4:22 AM >>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>> Rybchenko >>>>>>>>>>>>>>>>     >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>         Hi >>>>>>>>>>>>>>>> Adam, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> Yes, >>>>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT >>>>>>>>>>>>>>>> is in >>>>>>>>>>>>>>>> seconds. >>>>>>>>>>>>>>>> I've >>>>>>>>>>>>>>>> double-checked >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> that it >>>>>>>>>>>>>>>> goes to >>>>>>>>>>>>>>>> 'copy_timeout' >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> ts-conf/rcf.conf. >>>>>>>>>>>>>>>> Description >>>>>>>>>>>>>>>> in in >>>>>>>>>>>>>>>> doc/sphinx/pages/group_te_engine_rcf.rst >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> says that >>>>>>>>>>>>>>>> copy_timeout >>>>>>>>>>>>>>>> is in >>>>>>>>>>>>>>>> seconds and >>>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> lib/rcfunix/rcfunix.c >>>>>>>>>>>>>>>> passes the >>>>>>>>>>>>>>>> value to >>>>>>>>>>>>>>>> select() >>>>>>>>>>>>>>>> tv_sec. >>>>>>>>>>>>>>>> Theoretically >>>>>>>>>>>>>>>> select() >>>>>>>>>>>>>>>> could be >>>>>>>>>>>>>>>> interrupted >>>>>>>>>>>>>>>> by signal, >>>>>>>>>>>>>>>> but I >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> think it is >>>>>>>>>>>>>>>> unlikely here. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>         I'm >>>>>>>>>>>>>>>> not sure >>>>>>>>>>>>>>>> that I >>>>>>>>>>>>>>>> understand >>>>>>>>>>>>>>>> what do you >>>>>>>>>>>>>>>> mean by RCF >>>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>>> timeout. >>>>>>>>>>>>>>>> Does it >>>>>>>>>>>>>>>> happen on >>>>>>>>>>>>>>>> TE startup >>>>>>>>>>>>>>>> when RCF >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> starts test >>>>>>>>>>>>>>>> agents. If >>>>>>>>>>>>>>>> so, >>>>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT >>>>>>>>>>>>>>>> could help. Or >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> does it >>>>>>>>>>>>>>>> happen when >>>>>>>>>>>>>>>> tests are >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> progress, >>>>>>>>>>>>>>>> e.g. in the >>>>>>>>>>>>>>>> middle >>>>>>>>>>>>>>>>         of >>>>>>>>>>>>>>>> a test. If >>>>>>>>>>>>>>>> so, >>>>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> unrelated >>>>>>>>>>>>>>>> and most >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> likely >>>>>>>>>>>>>>>> either host >>>>>>>>>>>>>>>> with test >>>>>>>>>>>>>>>> agent dies >>>>>>>>>>>>>>>> or test >>>>>>>>>>>>>>>> agent itself >>>>>>>>>>>>>>>> crashes. It >>>>>>>>>>>>>>>> would be >>>>>>>>>>>>>>>> easier for >>>>>>>>>>>>>>>> me if >>>>>>>>>>>>>>>> classify it >>>>>>>>>>>>>>>> if you share >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> text log >>>>>>>>>>>>>>>> (log.txt, >>>>>>>>>>>>>>>> full or >>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>> corresponding >>>>>>>>>>>>>>>> fragment with >>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>> context). >>>>>>>>>>>>>>>> Also >>>>>>>>>>>>>>>> content of >>>>>>>>>>>>>>>> ta.DPDK or >>>>>>>>>>>>>>>> ta.Peer file >>>>>>>>>>>>>>>> depending >>>>>>>>>>>>>>>> on which >>>>>>>>>>>>>>>> agent has >>>>>>>>>>>>>>>> problems >>>>>>>>>>>>>>>> could shed >>>>>>>>>>>>>>>> some light. >>>>>>>>>>>>>>>> Corresponding >>>>>>>>>>>>>>>> files >>>>>>>>>>>>>>>> contain >>>>>>>>>>>>>>>> stdout/stderr >>>>>>>>>>>>>>>> of test agents. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Andrew. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>         On >>>>>>>>>>>>>>>> 8/23/23 >>>>>>>>>>>>>>>> 17:45, Adam >>>>>>>>>>>>>>>> Hassick wrote: >>>>>>>>>>>>>>>>> Hi Andrew, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> I've set >>>>>>>>>>>>>>>>> up a test >>>>>>>>>>>>>>>>> rig >>>>>>>>>>>>>>>>> repository >>>>>>>>>>>>>>>>> here, and >>>>>>>>>>>>>>>>> have created >>>>>>>>>>>>>>>>> configurations >>>>>>>>>>>>>>>>> for our >>>>>>>>>>>>>>>>> development >>>>>>>>>>>>>>>>> testbed >>>>>>>>>>>>>>>>> based off >>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>> examples. >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> We've been >>>>>>>>>>>>>>>>> able to >>>>>>>>>>>>>>>>> get the >>>>>>>>>>>>>>>>> test suite >>>>>>>>>>>>>>>>> to run >>>>>>>>>>>>>>>>> manually on >>>>>>>>>>>>>>>>> Mellanox >>>>>>>>>>>>>>>>> CX5 >>>>>>>>>>>>>>>>> devices once. >>>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>> we are >>>>>>>>>>>>>>>>> running >>>>>>>>>>>>>>>>> into an >>>>>>>>>>>>>>>>> issue >>>>>>>>>>>>>>>>> where, >>>>>>>>>>>>>>>>> when RCF >>>>>>>>>>>>>>>>> starts, >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> the RCF >>>>>>>>>>>>>>>>> connection >>>>>>>>>>>>>>>>> times out >>>>>>>>>>>>>>>>> very >>>>>>>>>>>>>>>>> frequently. >>>>>>>>>>>>>>>>> We aren't sure >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> why this >>>>>>>>>>>>>>>>> is the case. >>>>>>>>>>>>>>>>>         It >>>>>>>>>>>>>>>>> works >>>>>>>>>>>>>>>>> sometimes, >>>>>>>>>>>>>>>>> but most >>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>> time when >>>>>>>>>>>>>>>>> we try to run >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> the test >>>>>>>>>>>>>>>>> engine, it >>>>>>>>>>>>>>>>> encounters >>>>>>>>>>>>>>>>> this issue. >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> I've tried >>>>>>>>>>>>>>>>> changing >>>>>>>>>>>>>>>>> the RCF >>>>>>>>>>>>>>>>> port by >>>>>>>>>>>>>>>>> setting >>>>>>>>>>>>>>>>> "TE_RCF_PORT=>>>>>>>>>>>>>>>> port >>>>>>>>>>>>>>>>> number>" >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> rebooting >>>>>>>>>>>>>>>>> the testbed >>>>>>>>>>>>>>>>> machines. >>>>>>>>>>>>>>>>> Neither >>>>>>>>>>>>>>>>> seems to >>>>>>>>>>>>>>>>> fix the issue. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>         It >>>>>>>>>>>>>>>>> also seems >>>>>>>>>>>>>>>>> like the >>>>>>>>>>>>>>>>> timeout >>>>>>>>>>>>>>>>> takes far >>>>>>>>>>>>>>>>> longer than 60 >>>>>>>>>>>>>>>>> seconds, >>>>>>>>>>>>>>>>> even when >>>>>>>>>>>>>>>>> running >>>>>>>>>>>>>>>>> "export >>>>>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT=60" >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> before I >>>>>>>>>>>>>>>>> try to run >>>>>>>>>>>>>>>>> the test >>>>>>>>>>>>>>>>> suite. >>>>>>>>>>>>>>>>>         I >>>>>>>>>>>>>>>>> assume the >>>>>>>>>>>>>>>>> unit for >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> variable >>>>>>>>>>>>>>>>> is seconds? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>         Adam >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>         On >>>>>>>>>>>>>>>>> Mon, Aug >>>>>>>>>>>>>>>>> 21, 2023 >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> 10:19 AM >>>>>>>>>>>>>>>>> Adam Hassick >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>             >>>>>>>>>>>>>>>>> Hi Andrew, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> I've >>>>>>>>>>>>>>>>> cloned the >>>>>>>>>>>>>>>>> example >>>>>>>>>>>>>>>>> repository >>>>>>>>>>>>>>>>> and will start >>>>>>>>>>>>>>>>> setting up >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>> for our >>>>>>>>>>>>>>>>> development >>>>>>>>>>>>>>>>> testbed >>>>>>>>>>>>>>>>> today. >>>>>>>>>>>>>>>>> I'll let >>>>>>>>>>>>>>>>> you know >>>>>>>>>>>>>>>>> if I run >>>>>>>>>>>>>>>>> into any >>>>>>>>>>>>>>>>> difficulties >>>>>>>>>>>>>>>>>             >>>>>>>>>>>>>>>>> or have >>>>>>>>>>>>>>>>> any questions. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>             >>>>>>>>>>>>>>>>>  - Adam >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>             >>>>>>>>>>>>>>>>> On Sun, >>>>>>>>>>>>>>>>> Aug 20, >>>>>>>>>>>>>>>>> 2023 at >>>>>>>>>>>>>>>>> 4:40 AM >>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>> Rybchenko >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> Hi Adam, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> I've published >>>>>>>>>>>>>>>>> https://github.com/ts-factory/ts-rigs-sample >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> Hopefully >>>>>>>>>>>>>>>>> it will >>>>>>>>>>>>>>>>> help to >>>>>>>>>>>>>>>>> define >>>>>>>>>>>>>>>>> your test >>>>>>>>>>>>>>>>> rigs and >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> successfully >>>>>>>>>>>>>>>>> run some >>>>>>>>>>>>>>>>> tests >>>>>>>>>>>>>>>>> manually. >>>>>>>>>>>>>>>>> Feel free to >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> ask any >>>>>>>>>>>>>>>>> questions >>>>>>>>>>>>>>>>> and I'll >>>>>>>>>>>>>>>>> answer >>>>>>>>>>>>>>>>> here and >>>>>>>>>>>>>>>>> try to >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> update >>>>>>>>>>>>>>>>> documentation. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> Meanwhile >>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>> prepare >>>>>>>>>>>>>>>>> missing >>>>>>>>>>>>>>>>> bits for >>>>>>>>>>>>>>>>> steps (2) and >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> (3). >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> Hopefully >>>>>>>>>>>>>>>>> everything >>>>>>>>>>>>>>>>> is in >>>>>>>>>>>>>>>>> place for >>>>>>>>>>>>>>>>> step (4), >>>>>>>>>>>>>>>>> but we >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> need to >>>>>>>>>>>>>>>>> make steps >>>>>>>>>>>>>>>>> (2) and >>>>>>>>>>>>>>>>> (3) first. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> Andrew. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>> On 8/18/23 >>>>>>>>>>>>>>>>> 21:40, >>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>> Rybchenko >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> Hi Adam, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> > I've >>>>>>>>>>>>>>>>>> conferred >>>>>>>>>>>>>>>>>> with the >>>>>>>>>>>>>>>>>> rest of >>>>>>>>>>>>>>>>>> the team, >>>>>>>>>>>>>>>>>> and we >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> think it >>>>>>>>>>>>>>>>>> would be >>>>>>>>>>>>>>>>>> best to >>>>>>>>>>>>>>>>>> move >>>>>>>>>>>>>>>>>> forward >>>>>>>>>>>>>>>>>> with mainly >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> option B. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> OK, I'll >>>>>>>>>>>>>>>>>> provide >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> sample on >>>>>>>>>>>>>>>>>> Monday >>>>>>>>>>>>>>>>>> for you. >>>>>>>>>>>>>>>>>> It is >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> almost >>>>>>>>>>>>>>>>>> ready >>>>>>>>>>>>>>>>>> right >>>>>>>>>>>>>>>>>> now, but >>>>>>>>>>>>>>>>>> I need to >>>>>>>>>>>>>>>>>> double-check >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> it before >>>>>>>>>>>>>>>>>> publishing. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> Andrew. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>> On >>>>>>>>>>>>>>>>>> 8/17/23 >>>>>>>>>>>>>>>>>> 20:03, >>>>>>>>>>>>>>>>>> Adam >>>>>>>>>>>>>>>>>> Hassick >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> Hi Andrew, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> I'm >>>>>>>>>>>>>>>>>>> adding >>>>>>>>>>>>>>>>>>> the CI >>>>>>>>>>>>>>>>>>> mailing >>>>>>>>>>>>>>>>>>> list to this >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> conversation. >>>>>>>>>>>>>>>>>>> Others >>>>>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>>>>> community >>>>>>>>>>>>>>>>>>> might find >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>> conversation >>>>>>>>>>>>>>>>>>> valuable. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> We do >>>>>>>>>>>>>>>>>>> want to >>>>>>>>>>>>>>>>>>> run >>>>>>>>>>>>>>>>>>> testing >>>>>>>>>>>>>>>>>>> on a >>>>>>>>>>>>>>>>>>> regular >>>>>>>>>>>>>>>>>>> basis. The >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> Jenkins >>>>>>>>>>>>>>>>>>> integration >>>>>>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>>>>>> very >>>>>>>>>>>>>>>>>>> useful >>>>>>>>>>>>>>>>>>> for us, as >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> most of >>>>>>>>>>>>>>>>>>> our CI >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> orchestrated >>>>>>>>>>>>>>>>>>> by Jenkins. >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> I've >>>>>>>>>>>>>>>>>>> conferred >>>>>>>>>>>>>>>>>>> with the >>>>>>>>>>>>>>>>>>> rest of >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> team, and we >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> think it >>>>>>>>>>>>>>>>>>> would be >>>>>>>>>>>>>>>>>>> best to >>>>>>>>>>>>>>>>>>> move >>>>>>>>>>>>>>>>>>> forward >>>>>>>>>>>>>>>>>>> with mainly >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> option B. >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> If you >>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>> like to >>>>>>>>>>>>>>>>>>> know >>>>>>>>>>>>>>>>>>> anything >>>>>>>>>>>>>>>>>>> about our >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> testbeds >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>> help you >>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> creating an >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> example >>>>>>>>>>>>>>>>>>> ts-rigs >>>>>>>>>>>>>>>>>>> repo, >>>>>>>>>>>>>>>>>>> I'd be >>>>>>>>>>>>>>>>>>> happy to >>>>>>>>>>>>>>>>>>> answer any >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> questions >>>>>>>>>>>>>>>>>>> you have. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> We have >>>>>>>>>>>>>>>>>>> multiple >>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>> rigs (we >>>>>>>>>>>>>>>>>>> call these >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> "DUT-tester >>>>>>>>>>>>>>>>>>> pairs") >>>>>>>>>>>>>>>>>>> that we >>>>>>>>>>>>>>>>>>> run our >>>>>>>>>>>>>>>>>>> existing >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> hardware >>>>>>>>>>>>>>>>>>> testing >>>>>>>>>>>>>>>>>>> on, with >>>>>>>>>>>>>>>>>>> differing >>>>>>>>>>>>>>>>>>> network >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> hardware >>>>>>>>>>>>>>>>>>> and CPU >>>>>>>>>>>>>>>>>>> architecture. >>>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>> figured >>>>>>>>>>>>>>>>>>> this might >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> be an >>>>>>>>>>>>>>>>>>> important >>>>>>>>>>>>>>>>>>> detail. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> Adam >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> On Thu, >>>>>>>>>>>>>>>>>>> Aug 17, >>>>>>>>>>>>>>>>>>> 2023 at >>>>>>>>>>>>>>>>>>> 11:44 AM >>>>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>>>> Rybchenko >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Greatings >>>>>>>>>>>>>>>>>>> Adam, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> I'm >>>>>>>>>>>>>>>>>>> happy to >>>>>>>>>>>>>>>>>>> hear >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> you're >>>>>>>>>>>>>>>>>>> trying >>>>>>>>>>>>>>>>>>> to bring >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> it up. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> As I >>>>>>>>>>>>>>>>>>> understand >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> final >>>>>>>>>>>>>>>>>>> goal is >>>>>>>>>>>>>>>>>>> to run it on >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> regular >>>>>>>>>>>>>>>>>>> basis. >>>>>>>>>>>>>>>>>>> So, we >>>>>>>>>>>>>>>>>>> need to >>>>>>>>>>>>>>>>>>> make it >>>>>>>>>>>>>>>>>>> properly >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> from the >>>>>>>>>>>>>>>>>>> very >>>>>>>>>>>>>>>>>>> beginning. >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Bring up >>>>>>>>>>>>>>>>>>> of all >>>>>>>>>>>>>>>>>>> features >>>>>>>>>>>>>>>>>>> consists >>>>>>>>>>>>>>>>>>> of 4 steps: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> 1. >>>>>>>>>>>>>>>>>>> Create >>>>>>>>>>>>>>>>>>> site-specific >>>>>>>>>>>>>>>>>>> repository >>>>>>>>>>>>>>>>>>> (we call it >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> ts-rigs) >>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>> contains >>>>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>> about test >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> rigs and >>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>> site-specific >>>>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> where to >>>>>>>>>>>>>>>>>>> send >>>>>>>>>>>>>>>>>>> mails, >>>>>>>>>>>>>>>>>>> where to >>>>>>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>> logs etc. >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> It is >>>>>>>>>>>>>>>>>>> required >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> manual >>>>>>>>>>>>>>>>>>> execution >>>>>>>>>>>>>>>>>>> as well, >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> since >>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>> rigs >>>>>>>>>>>>>>>>>>> description >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> essential. >>>>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> return >>>>>>>>>>>>>>>>>>> to the >>>>>>>>>>>>>>>>>>> topic below. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> 2. Setup >>>>>>>>>>>>>>>>>>> logs >>>>>>>>>>>>>>>>>>> storage >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> automated >>>>>>>>>>>>>>>>>>> runs. >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Basically >>>>>>>>>>>>>>>>>>> it is a >>>>>>>>>>>>>>>>>>> disk >>>>>>>>>>>>>>>>>>> space >>>>>>>>>>>>>>>>>>> plus >>>>>>>>>>>>>>>>>>> apache2 web >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> server >>>>>>>>>>>>>>>>>>> with few >>>>>>>>>>>>>>>>>>> CGI >>>>>>>>>>>>>>>>>>> scripts >>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>> help a >>>>>>>>>>>>>>>>>>> lot to >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> save >>>>>>>>>>>>>>>>>>> disk space. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> 3. Setup >>>>>>>>>>>>>>>>>>> Bublik >>>>>>>>>>>>>>>>>>> web >>>>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>> provides >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> web >>>>>>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>>>> to view >>>>>>>>>>>>>>>>>>> testing >>>>>>>>>>>>>>>>>>> results. >>>>>>>>>>>>>>>>>>> Same as >>>>>>>>>>>>>>>>>>> https://ts-factory.io/bublik >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> 4. Setup >>>>>>>>>>>>>>>>>>> Jenkins >>>>>>>>>>>>>>>>>>> to run >>>>>>>>>>>>>>>>>>> tests on >>>>>>>>>>>>>>>>>>> regularly, >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> save >>>>>>>>>>>>>>>>>>> logs in >>>>>>>>>>>>>>>>>>> log >>>>>>>>>>>>>>>>>>> storage >>>>>>>>>>>>>>>>>>> (2) and >>>>>>>>>>>>>>>>>>> import it to >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> bublik (3). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Last few >>>>>>>>>>>>>>>>>>> month we >>>>>>>>>>>>>>>>>>> spent on >>>>>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>>>> homework >>>>>>>>>>>>>>>>>>> to make >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>> simpler >>>>>>>>>>>>>>>>>>> to bring >>>>>>>>>>>>>>>>>>> up >>>>>>>>>>>>>>>>>>> automated >>>>>>>>>>>>>>>>>>> execution >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>>>>> Jenkins - >>>>>>>>>>>>>>>>>>> https://github.com/ts-factory/te-jenkins >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Corresponding >>>>>>>>>>>>>>>>>>> bits in >>>>>>>>>>>>>>>>>>> dpdk-ethdev-ts >>>>>>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> available >>>>>>>>>>>>>>>>>>> tomorrow. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Let's >>>>>>>>>>>>>>>>>>> return >>>>>>>>>>>>>>>>>>> to the >>>>>>>>>>>>>>>>>>> step (1). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Unfortunately >>>>>>>>>>>>>>>>>>> there is >>>>>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>> publicly >>>>>>>>>>>>>>>>>>> available >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> example >>>>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>>>> ts-rigs >>>>>>>>>>>>>>>>>>> repository >>>>>>>>>>>>>>>>>>> since >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> sensitive >>>>>>>>>>>>>>>>>>> site-specific >>>>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>> is located >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> there. >>>>>>>>>>>>>>>>>>> But I'm >>>>>>>>>>>>>>>>>>> ready to >>>>>>>>>>>>>>>>>>> help you >>>>>>>>>>>>>>>>>>> to create it >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> for UNH. >>>>>>>>>>>>>>>>>>> I see >>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>> options >>>>>>>>>>>>>>>>>>> here: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> (A) I'll >>>>>>>>>>>>>>>>>>> ask >>>>>>>>>>>>>>>>>>> questions >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> based on >>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> answers >>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>>> draft >>>>>>>>>>>>>>>>>>> with my >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> comments. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> (B) I'll >>>>>>>>>>>>>>>>>>> make a >>>>>>>>>>>>>>>>>>> template/example >>>>>>>>>>>>>>>>>>> ts-rigs >>>>>>>>>>>>>>>>>>> repo, >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> publish >>>>>>>>>>>>>>>>>>> it and >>>>>>>>>>>>>>>>>>> you'll >>>>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>>>> UNH >>>>>>>>>>>>>>>>>>> ts-rigs >>>>>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> on it. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Of >>>>>>>>>>>>>>>>>>> course, >>>>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>>>> help to >>>>>>>>>>>>>>>>>>> debug >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> finally >>>>>>>>>>>>>>>>>>> bring >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> it up in >>>>>>>>>>>>>>>>>>> any case. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> (A) is a >>>>>>>>>>>>>>>>>>> bit >>>>>>>>>>>>>>>>>>> simpler >>>>>>>>>>>>>>>>>>> for me >>>>>>>>>>>>>>>>>>> and you, >>>>>>>>>>>>>>>>>>> but (B) is >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> a bit >>>>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>> generic >>>>>>>>>>>>>>>>>>> and will >>>>>>>>>>>>>>>>>>> help other >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> potential >>>>>>>>>>>>>>>>>>> users to >>>>>>>>>>>>>>>>>>> bring it up. >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> We can >>>>>>>>>>>>>>>>>>> combine >>>>>>>>>>>>>>>>>>> (A)+(B). >>>>>>>>>>>>>>>>>>> I.e. >>>>>>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>> from (A). >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> What do >>>>>>>>>>>>>>>>>>> you think? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> Andrew. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>> On >>>>>>>>>>>>>>>>>>> 8/17/23 >>>>>>>>>>>>>>>>>>> 15:18, >>>>>>>>>>>>>>>>>>> Konstantin >>>>>>>>>>>>>>>>>>> Ushakov >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> Greetings >>>>>>>>>>>>>>>>>>>> Adam, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>> contacting >>>>>>>>>>>>>>>>>>>> us. I >>>>>>>>>>>>>>>>>>>> copy >>>>>>>>>>>>>>>>>>>> Andrew who >>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>> happy >>>>>>>>>>>>>>>>>>>> to help >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>> Konstantin >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 16 >>>>>>>>>>>>>>>>>>>>> Aug >>>>>>>>>>>>>>>>>>>>> 2023, >>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>> 21:50, >>>>>>>>>>>>>>>>>>>>> Adam >>>>>>>>>>>>>>>>>>>>> Hassick >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>>  >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> Greetings >>>>>>>>>>>>>>>>>>>>> Konstantin, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> I am >>>>>>>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>>>>>>> process >>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>> setting >>>>>>>>>>>>>>>>>>>>> up the >>>>>>>>>>>>>>>>>>>>> DPDK >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> Poll >>>>>>>>>>>>>>>>>>>>> Mode >>>>>>>>>>>>>>>>>>>>> Driver >>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>> suite >>>>>>>>>>>>>>>>>>>>> as an >>>>>>>>>>>>>>>>>>>>> addition >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>>>>>> testing >>>>>>>>>>>>>>>>>>>>> coverage >>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>> DPDK >>>>>>>>>>>>>>>>>>>>> at the >>>>>>>>>>>>>>>>>>>>> UNH lab. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> I have >>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>> questions >>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>> how to >>>>>>>>>>>>>>>>>>>>> set the >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>> suite >>>>>>>>>>>>>>>>>>>>> arguments. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> I have >>>>>>>>>>>>>>>>>>>>> been >>>>>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> configure >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> Test >>>>>>>>>>>>>>>>>>>>> Engine >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> connect >>>>>>>>>>>>>>>>>>>>> to the >>>>>>>>>>>>>>>>>>>>> hosts >>>>>>>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>>>>>>> testbed. >>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> RCF, >>>>>>>>>>>>>>>>>>>>> Configurator, >>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> Tester >>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>> begin to >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> run, >>>>>>>>>>>>>>>>>>>>> however >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> prelude >>>>>>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>>>>>> test suite >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> fails >>>>>>>>>>>>>>>>>>>>> to run. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>> documentation >>>>>>>>>>>>>>>>>>>>> mentions >>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>> there are >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> several >>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>> suite, >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> IUT >>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>> link >>>>>>>>>>>>>>>>>>>>> MAC, >>>>>>>>>>>>>>>>>>>>> etc. These >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> seem >>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>> they >>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>> need >>>>>>>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>>>> somewhere >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> to run >>>>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>>>>>> tests. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> I see >>>>>>>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>>>>>>> Test >>>>>>>>>>>>>>>>>>>>> Engine >>>>>>>>>>>>>>>>>>>>> documentation, >>>>>>>>>>>>>>>>>>>>> there >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>> instructions >>>>>>>>>>>>>>>>>>>>> on how >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> create new >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>> suites >>>>>>>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>>>>>>> Tester >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> configuration, >>>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>> there >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> nothing >>>>>>>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>>>> guide >>>>>>>>>>>>>>>>>>>>> or in >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> Tester >>>>>>>>>>>>>>>>>>>>> guide >>>>>>>>>>>>>>>>>>>>> for how to >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> arguments >>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> running >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>> suite >>>>>>>>>>>>>>>>>>>>> that I >>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>> find. I'm >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>> sure >>>>>>>>>>>>>>>>>>>>> if I >>>>>>>>>>>>>>>>>>>>> need >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> write >>>>>>>>>>>>>>>>>>>>> my own >>>>>>>>>>>>>>>>>>>>> Tester >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> config, >>>>>>>>>>>>>>>>>>>>> or if >>>>>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>> setting >>>>>>>>>>>>>>>>>>>>> these in >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>> other way. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> How >>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>>>> values >>>>>>>>>>>>>>>>>>>>> be set? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> I'm >>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>> sure >>>>>>>>>>>>>>>>>>>>> what >>>>>>>>>>>>>>>>>>>>> environment >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> variables/arguments >>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>> strictly >>>>>>>>>>>>>>>>>>>>> necessary >>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>> optional. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> Adam >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> *Adam >>>>>>>>>>>>>>>>>>>>> Hassick* >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> Senior >>>>>>>>>>>>>>>>>>>>> Developer >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> UNH >>>>>>>>>>>>>>>>>>>>> InterOperability >>>>>>>>>>>>>>>>>>>>> Lab >>>>>>>>>>>>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> iol.unh.edu >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>                     >>>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>>> (603) >>>>>>>>>>>>>>>>>>>>> 475-8248 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> *Adam >>>>>>>>>>>>>>>>>>> Hassick* >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> Senior >>>>>>>>>>>>>>>>>>> Developer >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> UNH >>>>>>>>>>>>>>>>>>> InterOperability >>>>>>>>>>>>>>>>>>> Lab >>>>>>>>>>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> iol.unh.edu >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>                 >>>>>>>>>>>>>>>>>>> +1 (603) >>>>>>>>>>>>>>>>>>> 475-8248 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>             >>>>>>>>>>>>>>>>> -- *Adam >>>>>>>>>>>>>>>>> Hassick* >>>>>>>>>>>>>>>>> Senior >>>>>>>>>>>>>>>>> Developer >>>>>>>>>>>>>>>>> UNH >>>>>>>>>>>>>>>>> InterOperability >>>>>>>>>>>>>>>>> Lab >>>>>>>>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> iol.unh.edu >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>             >>>>>>>>>>>>>>>>> +1 (603) >>>>>>>>>>>>>>>>> 475-8248 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>         -- >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> *Adam Hassick* >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> Senior >>>>>>>>>>>>>>>>> Developer >>>>>>>>>>>>>>>>>         >>>>>>>>>>>>>>>>> UNH >>>>>>>>>>>>>>>>> InterOperability >>>>>>>>>>>>>>>>> Lab >>>>>>>>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> iol.unh.edu >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>         +1 >>>>>>>>>>>>>>>>> (603) 475-8248 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     -- >>>>>>>>>>>>>>>> *Adam Hassick* >>>>>>>>>>>>>>>>     Senior >>>>>>>>>>>>>>>> Developer >>>>>>>>>>>>>>>>     UNH >>>>>>>>>>>>>>>> InterOperability >>>>>>>>>>>>>>>> Lab >>>>>>>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> iol.unh.edu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>     +1 >>>>>>>>>>>>>>>> (603) 475-8248 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> *Adam Hassick* >>>>>>>>>>>>>>> Senior Developer >>>>>>>>>>>>>>> UNH >>>>>>>>>>>>>>> InterOperability >>>>>>>>>>>>>>> Lab >>>>>>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> iol.unh.edu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 (603) >>>>>>>>>>>>>>> 475-8248 >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Adam Hassick* >>>>>>>>>>>>> Senior Developer >>>>>>>>>>>>> UNH >>>>>>>>>>>>> InterOperability Lab >>>>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>>>> iol.unh.edu >>>>>>>>>>>>> >>>>>>>>>>>>> +1 (603) 475-8248 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Adam Hassick* >>>>>>>>>>> Senior Developer >>>>>>>>>>> UNH InterOperability Lab >>>>>>>>>>> ahassick@iol.unh.edu >>>>>>>>>>> iol.unh.edu >>>>>>>>>>> >>>>>>>>>>> +1 (603) 475-8248 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Adam Hassick* >>>>>>>>> Senior Developer >>>>>>>>> UNH InterOperability Lab >>>>>>>>> ahassick@iol.unh.edu >>>>>>>>> iol.unh.edu >>>>>>>>> >>>>>>>>> +1 (603) 475-8248 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Adam Hassick* >>>>>>> Senior Developer >>>>>>> UNH InterOperability Lab >>>>>>> ahassick@iol.unh.edu >>>>>>> iol.unh.edu >>>>>>> +1 (603) 475-8248 >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Adam Hassick* >>>>>> Senior Developer >>>>>> UNH InterOperability Lab >>>>>> ahassick@iol.unh.edu >>>>>> iol.unh.edu >>>>>> +1 (603) 475-8248 >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Adam Hassick* >>>>>> Senior Developer >>>>>> UNH InterOperability Lab >>>>>> ahassick@iol.unh.edu >>>>>> iol.unh.edu >>>>>> +1 (603) 475-8248 >>>>> >>>> >>> >>> >>> >>> -- >>> *Adam Hassick* >>> Senior Developer >>> UNH InterOperability Lab >>> ahassick@iol.unh.edu >>> iol.unh.edu >>> +1 (603) 475-8248 >> >> >> >> -- >> *Adam Hassick* >> Senior Developer >> UNH InterOperability Lab >> ahassick@iol.unh.edu >> iol.unh.edu >> +1 (603) 475-8248 > > > > -- > *Adam Hassick* > Senior Developer > UNH InterOperability Lab > ahassick@iol.unh.edu > iol.unh.edu > +1 (603) 475-8248