Re: Setting up DPDK PMD Test Suite

[Adam Hassick <ahassick@iol.unh.edu>]

[-- Attachment #1.1: Type: text/plain, Size: 61031 bytes --]

Hi Andrew,

I've attached the public key we'd like to use.

Thanks,
Adam

On Thu, Oct 26, 2023 at 8:19 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> On 10/25/23 23:27, Adam Hassick wrote:
>
> Hi Andrew,
>
> Sorry about the two week radio silence, we're still trying to sort out
> logistics for deployment on our end.
>
> > I've created setup on ts-factory.io which allows to publish logs. Let
> me know if you'd like to try it and I'll provide credentials, script and
> short instruction.
>
> We're interested in publishing some test logs to the ts-factory Bublik
> instance in the meantime.
>
>
> Please, send me SSH public key which you'd like to use to upload logs.
> I'll provide helper script and instructions to do it.
>
> Andrew.
>
>
> Thanks,
> Adam
>
> On Mon, Oct 23, 2023 at 7:11 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> > Now that our test results are in line with yours, we can begin looking
>> into setting up the production environment.
>>
>> Please, let me know if you need any help with it or waiting for an input
>> from me.
>>
>> Regards,
>> Andrew.
>>
>> On 10/10/23 17:09, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> Thank you for taking a look at our log. Netplan was attempting to run
>> DHCP on our test links, and additionally I discovered that the NIC firmware
>> was transmitting LLDP packets, causing tests to fail in the same way. Now
>> that these problems have been fixed, our pass rate on the XL710 is
>> approximately 91%. Now that our test results are in line with yours, we can
>> begin looking into setting up the production environment.
>>
>> First, is it possible to run the test agent on ARM hosts? Our ARM
>> testbeds have the best topology for running this test suite, with separate
>> tester and DUT servers.
>>
>> We are testing this test suite on two x86 development servers using the
>> test suite's recommended server topology. In contrast, our existing x86
>> production testbeds which run DTS have a single server topology. This
>> single server has both the tester NIC and the device under test NIC
>> installed, with NUMA node separation between TRex and DPDK. We're going to
>> test running the two test agent processes on the single-server testbeds if
>> we cannot run this on ARM. Is there any reason you can think of that would
>> prevent this setup from working?
>>
>> Once we figure out where this can live in production, then we will begin
>> setting up log storage, Jenkins integration, and Bublik.
>>
>> Thanks,
>> Adam
>>
>> On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> > Do these default to vfio-pci?
>>>
>>> Yes, vfio-pci is the default.
>>> However, it does not work in the case of Mellanox which uses bifurcated
>>> driver. It should mlx5_core for Mellanox NICs.
>>>
>>> > Here is the text log from a run on our Intel XL710 NICs, with the
>>> expected result profile set to the X710.
>>>
>>> It is hard to analyze all tests using text logs, but I definitely see
>>> one common problem. Tests receive unexpected packets and fail because of it.
>>> Tests are written very strict from this point of view and it brought
>>> fruits in the past when HW had bugs.
>>> Are DUT and tester connected back-to-back on tested interfaces or via
>>> switch?
>>> If via switch, is it possible to isolate it from everything else?
>>> If back-to-back, it could be some embedded SW/FW which regenerates these
>>> packet.
>>> I definitely see unexpected DHCP packets.
>>>
>>> > We haven't set up the Jenkins integration yet, however if this is
>>> required to import the logs then we will prioritize that.
>>>
>>> Unfortunately manual runs do not generate all artifacts required to
>>> import logs. However, we have almost solved it right now. Hopefully we'll
>>> finalize it in a day or two. I'll let you know when these changes are
>>> available.
>>>
>>> Regards,
>>> Andrew.
>>>
>>> On 10/4/23 16:48, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set anywhere
>>> in the default configurations for the Intel X710. Do these default to
>>> vfio-pci?
>>> We have IOMMU enabled on our development testbed, and should be able to
>>> bind vfio-pci.
>>> Here is the text log from a run on our Intel XL710 NICs, with the
>>> expected result profile set to the X710. We haven't set up the Jenkins
>>> integration yet, however if this is required to import the logs then we
>>> will prioritize that.
>>>  log.txt.tar.gz
>>> <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> On 9/18/23 17:44, Adam Hassick wrote:
>>>>
>>>> Hi Andrew and Konstantin,
>>>>
>>>> Thank you for adding the tester-dial feature, this opens up the
>>>> possibility for us to do CI integrated testing in the future.
>>>>
>>>> Our Mellanox pass rate is similar to yours (about ~2400 passing, ~4400
>>>> failing), however our Intel pass rates are far worse.
>>>> I will try running tests on the XL710 with the trc-tags argument set
>>>> and see if it improves the pass rate.
>>>> Another thing I noticed in the results you uploaded is that the results
>>>> are tagged with vfio-pci and not i40e.
>>>> Though in the environment dump, the driver on the test machine and the
>>>> DUT are set to use the i40e driver. Is this important at all?
>>>>
>>>>
>>>> I think it is a misunderstanding here. There are two kinds of driver in
>>>> configuration: net driver and so-called DPDK driver.
>>>> Net driver is a Linux kernel network device driver used on Tester side.
>>>> DPDK driver is a Linux kernel driver to bind device to to use it with
>>>> DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
>>>> In the case of bifurcated driver (like mlx5_core) it is the same in
>>>> both cases.
>>>> In non-bifurcated case DPDK driver is some UIO driver(vfio-pci,
>>>> uio-pci-generic or igb_uio).
>>>> Some expectations depend on used UIO. For example, uio-pci-generic do
>>>> not support many interrupts (used by usecases/rx_intr test cases).
>>>> That's why we care corresponding TRC tag.
>>>>
>>>> TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's Intel case.
>>>> Or uio-pci-generic if IOMMU is turned off on corresponding machines and
>>>> Linux distro does not support VFIO no IOMMU mode.
>>>>
>>>> Andrew.
>>>>
>>>> There isn't anything preventing us from pushing our results up to the
>>>> existing Bublik instance running at ts-factory.io that I can think of
>>>> at the moment.
>>>> We will have to work out how to submit our results to your Bublik
>>>> instance in a controlled and secure manner in that case.
>>>> As far as I know we won't need access controls for the results
>>>> themselves. I'll discuss this with Patrick and will let you know once we
>>>> confirm that it's fine.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> should we always auto-assign the tags or you don’t do it since it
>>>>> slows down (by some seconds) the TE startup?
>>>>>
>>>>>
>>>>> Tags are auto-assigned, but I guess it differs in Adam's case since
>>>>> NIC is a bit different. Below test will help to understand if it is the
>>>>> root cause of very different expectations. If pass rate will be close to
>>>>> mine, I'll simply update TRC database to share expectations for mine NIC
>>>>> and NIC used by Adam.
>>>>>
>>>>> Hi Adam,
>>>>>
>>>>> I think I second the question from Andrew - happy to help you with the
>>>>> triage so that we get to the same baseline. Do you have a good way for us
>>>>> to share the logs? I.e. say upload to ts-factory if we add strict
>>>>> permissions system so it’s not publishing or any other way.
>>>>>
>>>>> Thanks,
>>>>> Konstantin
>>>>>
>>>>> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>>>
>>>>> Hi Adam,
>>>>>
>>>>> I've uploaded fresh testing results to ts-factory.io [1] to be on the
>>>>> same page.
>>>>>
>>>>> I think I know why your and mine results on Intel 710 series NICs
>>>>> differ so much. Testing results expectations database
>>>>> (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags.  I.e.
>>>>> expectations depends on TRC tags discovered by helper scripts when testing
>>>>> is started. These tags identify various aspects of what is tested. Ideally
>>>>> expectations should be written in terms of root cause of the expected
>>>>> behaviour. If it is a driver expectations, driver tag should be used. If it
>>>>> is HW limitation, tags with PCI IDs should be used. However, it is not
>>>>> always easy to classify it correctly if you're not involved in driver
>>>>> development. So, in order case expectations for 710's Intel are filled in
>>>>> in terms of PCI IDs. I guess PCI ID differ in your case and that's why
>>>>> expectations filled in for my NIC do not apply to your runs.
>>>>>
>>>>> Just try to add the following option when you run on your 710's Intel
>>>>> in order to mimic mine and see if it helps to achieve better pass rate.
>>>>> --trc-tag=pci-8086-1572
>>>>>
>>>>> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for
>>>>> --tester-dial option. It should have better coverage now.
>>>>>
>>>>> Andrew.
>>>>>
>>>>> [1]
>>>>> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>>>
>>>>> On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>>
>>>>> Hi Adam,
>>>>>
>>>>> I've pushed new TE tag v1.20.0 which supported a new command-line
>>>>> option --tester-dial=NUM where NUM is from 0 to 100. it allows to choose
>>>>> percentage of tests to run. If you want stable set, you should pass
>>>>> --tester-random-seed=0 (or other integer). It is the first sketch and we
>>>>> have plans to improve it, but feedback would be welcome.
>>>>>
>>>>> > Is it needed on the tester?
>>>>>
>>>>> It is hard to say if it is strictly required for simple tests.
>>>>> However, it is better to update Tester as well, since performance tests run
>>>>> DPDK on Tester as well.
>>>>>
>>>>> > Are there any other manual setup steps for these devices that I
>>>>> might be missing?
>>>>>
>>>>> I don't remember anything else.
>>>>>
>>>>> I think it is better to get down to details and take a look at logs.
>>>>> I'm ready to help with it and explain what's happening there. May be it
>>>>> will help to understand if it is a problem with setup/configuration.
>>>>>
>>>>> Text logs are not very convenient. Ideally logs should be imported to
>>>>> bublik, however, manual runs do not provide all required artifacts right
>>>>> now (Jenkins jobs generate all required artifacts).
>>>>> Other option is 'tmp_raw_log' file (should be packed to make it
>>>>> smaller) which could be converted to various log formats.
>>>>> Would it be OK for you if I import your logs to bublik at
>>>>> ts-factory.io? Or is it a problem that it is publicly available?
>>>>> Would it help if we add authentication and access control there?
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 9/8/23 17:57, Adam Hassick wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> I have a couple questions about needed setup of the NICs for the
>>>>> ethdev test suite.
>>>>>
>>>>> Our MCX5s and XL710s are failing the checkup tests. The pass rate
>>>>> appears to be much worse on the XL710s (40 of 73 tests failed, 3 passed
>>>>> unexpectedly).
>>>>>
>>>>> For the XL710s, I've updated the driver and NVM versions to match the
>>>>> minimum supported versions in the compatibility matrix found on the DPDK
>>>>> documentation. This did not change the failure rate much.
>>>>> For the MCX5s, I've installed the latest LTS version of the OFED
>>>>> bifurcated driver on the DUT. Is it needed on the tester?
>>>>>
>>>>> Are there any other manual setup steps for these devices that I might
>>>>> be missing?
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> Yes, I copied the X710 configs to set up XL710 configs. I changed the
>>>>>> environment variable names from the X710 suffix to XL710 suffix in the
>>>>>> script, and forgot to change them in the corresponding environment file.
>>>>>> That fixed the issue.
>>>>>>
>>>>>> I got the checkup tests working on the XL710 now. Most of them are
>>>>>> failing, which leads me to believe this is an issue with our testbed. Based
>>>>>> on the DPDK documentation for i40e, the firmware and driver versions are
>>>>>> much older than what DPDK 22.11 LTS and main prefer, so I'll try updating
>>>>>> those.
>>>>>>
>>>>>> For now I'm working on getting the XL710 checkup tests passing, and
>>>>>> will pick up getting the E810 configured properly next. I'll let you know
>>>>>> if I run into any more issues in relation to the test engine.
>>>>>>
>>>>>> Thanks,
>>>>>> Adam
>>>>>>
>>>>>> On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>> Hi Adam,
>>>>>>>
>>>>>>> On 9/5/23 18:01, Adam Hassick wrote:
>>>>>>>
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> The compilation warning issue is now resolved. Again, thank you guys
>>>>>>> for fixing this for us. I can run the tests on the Mellanox CX5s again,
>>>>>>> however I'm running into a couple new issues with running the prologues on
>>>>>>> the Intel cards.
>>>>>>>
>>>>>>> When running testing on the Intel XL710s, I see this error appear in
>>>>>>> the log:
>>>>>>>
>>>>>>> ERROR  prologue  Environment LIB  14:16:13.650
>>>>>>>> Too few networks in available configuration (0) in comparison with
>>>>>>>> required (1)
>>>>>>>>
>>>>>>>
>>>>>>> This seems like a trivial configuration error, perhaps this is
>>>>>>> something I need to set up in ts-rigs. I briefly searched through the
>>>>>>> examples there and didn't see any mention of how to set up a network.
>>>>>>> I will attach this log just in case you need more information.
>>>>>>>
>>>>>>>
>>>>>>> Unfortunately logs are insufficient to understand it. I've pushed
>>>>>>> new tag to TE v1.19.0 which add log message with TE_* environment variables.
>>>>>>> Most likely something is wrong with variables which are used as
>>>>>>> conditions when available networks are defined in
>>>>>>> ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>>> TE_PCI_INSTANCE_IUT_TST1
>>>>>>> TE_PCI_INSTANCE_IUT_TST1a
>>>>>>> TE_PCI_INSTANCE_TST1a_IUT
>>>>>>> TE_PCI_INSTANCE_TST1_IUT
>>>>>>> My guess it that you change naming a bit, but script like
>>>>>>> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>>>>>>>
>>>>>>> There is a different error when running on the Intel E810s. It
>>>>>>> appears to me like it starts DPDK, does some configuration inside DPDK and
>>>>>>> on the device, and then fails to bring the device back up. Since this error
>>>>>>> seems very non-trivial, I will also attach this log.
>>>>>>>
>>>>>>>
>>>>>>> This one is a bit simpler. Few lines after the first ERROR in log I
>>>>>>> see the following:
>>>>>>> WARN  RCF  DPDK  13:06:00.144
>>>>>>> ice_program_hw_rx_queue(): currently package doesn't support RXDID
>>>>>>> (22)
>>>>>>> ice_rx_queue_start(): fail to program RX queue 0
>>>>>>> ice_dev_start(): fail to start Rx queue 0
>>>>>>> Device with port_id=0 already stopped
>>>>>>>
>>>>>>> It is stdout/stderr from test agent which runs DPDK. Same logs in
>>>>>>> plain format are available in ta.DPDK file.
>>>>>>> I'm not an expert here, but I vaguely remember that E810 requires
>>>>>>> correct firmware and DDP to be loaded.
>>>>>>> There is some information in dpdk/doc/guides/nics/ice.rst.
>>>>>>>
>>>>>>> You can try to add --dev-args=safe-mode-support=1 command-line
>>>>>>> option described there.
>>>>>>>
>>>>>>> Hope it helps,
>>>>>>> Andrew.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Adam
>>>>>>>
>>>>>>> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>> Hi Adam,
>>>>>>>>
>>>>>>>> On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> I have one additional question as well: Does the test engine
>>>>>>>> support running tests on two ARMv8 test agents?
>>>>>>>>
>>>>>>>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Great. Let me know when that's fixed.
>>>>>>>>
>>>>>>>>
>>>>>>>> Done. We also fixed a number of warnings in TE.
>>>>>>>> Also we fixed root test package name to be consistent with the
>>>>>>>> repository name.
>>>>>>>>
>>>>>>>> Support for old LTS branches was dropped some time ago, but in the
>>>>>>>>> future it is definitely possible to keep it for new LTS branches. I think
>>>>>>>>> 22.11 is supported, but I'm not sure about older LTS releases.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Good to know.
>>>>>>>>
>>>>>>>>
>>>>>>>>> 2. You can add command-line option --sanity to run tests marked
>>>>>>>>> with TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and
>>>>>>>>> grep TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked).
>>>>>>>>> Yes, there is a space for terminology improvement here. We'll do it.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Done. Now it is called --checkup.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also it takes a lot of time because of failures and tests which
>>>>>>>>> wait for some timeout.
>>>>>>>>>
>>>>>>>>
>>>>>>>> That makes sense to me. We'll use the time to complete tests on
>>>>>>>> virtio or the Intel devices as a reference for how long the tests really
>>>>>>>> take to complete.
>>>>>>>> We will explore the possibility of periodically running the sanity
>>>>>>>> tests for patches.
>>>>>>>>
>>>>>>>>
>>>>>>>> I'll double-check and let you know how long entire TS runs on Intel
>>>>>>>> X710, E810, Mellanox CX5 and virtio net. Just to ensure that time observed
>>>>>>>> in your case looks the same.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> The test harness can provide coverage reports based on gcov, but
>>>>>>>>> I'm not sure what you mean by a "dial" to control test coverage. Provided
>>>>>>>>> reports are rather for human to analyze.
>>>>>>>>>
>>>>>>>>
>>>>>>>> The general idea is to have some kind of parameter on the test
>>>>>>>> suite, which could be an integer ranging from zero to ten, that controls
>>>>>>>> how many tests are run based on how important the test is.
>>>>>>>>
>>>>>>>> Similar to how some command line interfaces provide a verbosity
>>>>>>>> level parameter (some number of "-v" arguments) to control the importance
>>>>>>>> of the information in the log.
>>>>>>>> The verbosity level zero only prints very important log messages,
>>>>>>>> while ten prints everything.
>>>>>>>>
>>>>>>>> In much the same manner as above, this "dial" parameter controls
>>>>>>>> what tests are run and with what parameters based on how important those
>>>>>>>> tests and test parameter combinations are.
>>>>>>>> Coverage Level zero tells the suite to run a very basic set of
>>>>>>>> important tests, with minimal parameterization. This mode would take only
>>>>>>>> ~5-10 minutes to run.
>>>>>>>> In contrast, Coverage Level ten includes all the edge cases, every
>>>>>>>> combination of test parameters, everything the test suite can do, which
>>>>>>>> takes the normal several hours to run.
>>>>>>>> The values 1 - 9 are between those two extremes, allowing the user
>>>>>>>> to get a gradient of test coverage in the results and to limit the running
>>>>>>>> time.
>>>>>>>>
>>>>>>>> Then we could, for example, run the "run.sh" with a level of 2 or 3
>>>>>>>> for incoming patches that need quick results, and with a level of 10 for
>>>>>>>> the less often run periodic tests performed on main or LTS branches.
>>>>>>>>
>>>>>>>>
>>>>>>>> Understood now. Thanks a lot for the idea. We'll discuss it and
>>>>>>>> come back.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>>>>>>>>> testing results. Unfortunately it is time consuming to fill in expectations
>>>>>>>>> database since it is necessary to analyze testing results and classify if
>>>>>>>>> it is a bug or just acceptable behaviour aspect.
>>>>>>>>>
>>>>>>>>> Bublik allows to compare results of two runs. It is useful for
>>>>>>>>> human, but still not good for automation.
>>>>>>>>>
>>>>>>>>> I have local patch for mlx5 driver which reports Tx ring size
>>>>>>>>> maximum. It makes pass rate higher. It is a problem for test harness that
>>>>>>>>> mlx5 does not report limits right now.
>>>>>>>>>
>>>>>>>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>>>>>>>>> virtio net is 99% right now and could be done 100% easily (just one thing
>>>>>>>>> to fix in expectations).
>>>>>>>>>
>>>>>>>>> I think logs storage setup is essential for logs analysis. Of
>>>>>>>>> course, you can request HTML logs when you run tests (--log-html=html) or
>>>>>>>>> generate after run using dpdk-ethdev-ts/scripts/html-log.sh and open
>>>>>>>>> index.html in a browser, but logs storage makes it more convenient.
>>>>>>>>>
>>>>>>>>
>>>>>>>> We are interested in setting up Bublik, potentially as an
>>>>>>>> externally-facing component, once we have our process of running the test
>>>>>>>> suite stabilized.
>>>>>>>> Once we are able to run the test suite again, I'll see what the
>>>>>>>> pass rate is on our other hardware.
>>>>>>>> Good to know that it isn't an issue with our dev testbed causing
>>>>>>>> the high fail rate.
>>>>>>>>
>>>>>>>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>>>>>>>> development testbed. Although they are slightly different devices, ideally
>>>>>>>> the pass rate will be identical or similar. I have yet to set up a VM pair
>>>>>>>> for virtio, but we will soon.
>>>>>>>>
>>>>>>>> Latest version of test-environment has examples of our CGI scripts
>>>>>>>>> which we use for log storage (see tools/log_server/README.md).
>>>>>>>>>
>>>>>>>>> Also all bits for Jenkins setup are available. See
>>>>>>>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>>>>>>>> ts-rigs-sample.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Jenkins integration, setting up production rig configurations, and
>>>>>>>> permanent log storage will be our next steps once I am able to run the
>>>>>>>> tests again.
>>>>>>>> Unless there is an easy way to have meson not pass "-Werror" into
>>>>>>>> GCC. Then I would be able to run the test suite.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hopefully it is resolved now.
>>>>>>>>
>>>>>>>> I thought a bit more about your usecase for Jenkins. I'm not 100%
>>>>>>>> sure that existing pipelines are convenient for your usecase.
>>>>>>>> Fill free to ask questions when you are on it.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Adam
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>> Hi Andrew,
>>>>>>>>>
>>>>>>>>> That fix seems to have resolved the issue, thanks for the quick
>>>>>>>>> turnaround time on that patch.
>>>>>>>>> Now that we have the RCF timeout issue resolved, there are a few
>>>>>>>>> other questions and issues that we have about the tests themselves.
>>>>>>>>>
>>>>>>>>> 1. The test suite fails to build with a couple warnings.
>>>>>>>>>
>>>>>>>>> Below is the stderr log from compilation:
>>>>>>>>>
>>>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>>>>>>>>> literal and no format arguments [-Werror=format-security]
>>>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>>>> |     ^~
>>>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>>>> ninja: build stopped: subcommand failed.
>>>>>>>>>> ninja: Entering directory `.'
>>>>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>>>>>>>>> literal and no format arguments [-Werror=format-security]
>>>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>>>> |     ^~
>>>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This error wasn't occurring last week, which was the last time I
>>>>>>>>> ran the tests.
>>>>>>>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester
>>>>>>>>> has GCC v11.4.0 installed, if this information is helpful.
>>>>>>>>>
>>>>>>>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>>>>>>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>>>>>>>> to lower the test coverage and have the test suite run faster?
>>>>>>>>>
>>>>>>>>> For some context, we run immediate testing on incoming patches for
>>>>>>>>> DPDK main and development branches, as well as periodic test runs on the
>>>>>>>>> main, stable, and LTS branches.
>>>>>>>>> For us to consider including this test suite as part of our
>>>>>>>>> immediate testing on patches, we would have to reduce the test coverage to
>>>>>>>>> the most important tests.
>>>>>>>>> This is primarily to reduce the testing time to, for example, less
>>>>>>>>> than 30 minutes. Testing on patches can't take too long because the lab can
>>>>>>>>> receive numerous patches each day, which each require individual testing
>>>>>>>>> runs.
>>>>>>>>>
>>>>>>>>> At what frequency we run these tests, and on what, still needs to
>>>>>>>>> be discussed with the DPDK community, but it would be nice to know if the
>>>>>>>>> test suite had a "dial" to control the testing coverage.
>>>>>>>>>
>>>>>>>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>>>>>>>> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose these
>>>>>>>>> test failures?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Adam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Adam,
>>>>>>>>>>
>>>>>>>>>> I've pushed the fix in main branch and a new tag v1.18.1. It
>>>>>>>>>> should solve the problem with IPv6 address from DNS.
>>>>>>>>>>
>>>>>>>>>> Andrew.
>>>>>>>>>>
>>>>>>>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Adam,
>>>>>>>>>>
>>>>>>>>>> > Does the test engine prefer to use IPv6 over IPv4 for
>>>>>>>>>> initiating the RCF connection to the test bed hosts? And if so, is there a
>>>>>>>>>> way to force it to use IPv4?
>>>>>>>>>>
>>>>>>>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in
>>>>>>>>>> your case, I guess it is the root cause of the problem.
>>>>>>>>>> Of course, it is TE problem since I see really weird code in
>>>>>>>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>>>>>>
>>>>>>>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>>>>>>>> ts-factory/test-environment repository. Please, try.
>>>>>>>>>>
>>>>>>>>>> It is late night fix with minimal testing and no review. I'll
>>>>>>>>>> pass it through review process tomorrow and
>>>>>>>>>> hopefully it will be released in one-two days.
>>>>>>>>>>
>>>>>>>>>> Andrew.
>>>>>>>>>>
>>>>>>>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Andrew,
>>>>>>>>>>
>>>>>>>>>> We have yet to notice a distinct pattern with the failures.
>>>>>>>>>> Sometimes, the RCF will start and connect without issue a few times in a
>>>>>>>>>> row before failing to connect again. Once the issue begins to occur,
>>>>>>>>>> neither rebooting all of the hosts (test engine VM, tester, IUT) or
>>>>>>>>>> deleting all of the build directories (suites, agents, inst) and rebooting
>>>>>>>>>> the hosts afterward resolves the issue. When it begins working again seems
>>>>>>>>>> very arbitrary to us.
>>>>>>>>>>
>>>>>>>>>> I do usually try to terminate the test engine with Ctrl+C, but
>>>>>>>>>> when it hangs while trying to start RCF, that does not work.
>>>>>>>>>>
>>>>>>>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>>>>>> force it to use IPv4?
>>>>>>>>>>
>>>>>>>>>>  - Adam
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>>
>>>>>>>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>>>
>>>>>>>>>>> Done. It works fine for me without any issues.
>>>>>>>>>>>
>>>>>>>>>>> Have you noticed any pattern when it works or does not work?
>>>>>>>>>>> May be it is a problem of not clean state after termination?
>>>>>>>>>>> Does it work fine the first time after DUTs reboot?
>>>>>>>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>>>>>>>> terminal where you execute run.sh command.
>>>>>>>>>>>  In this case it should shutdown gracefully and close all test
>>>>>>>>>>> agents and engine applications.
>>>>>>>>>>>
>>>>>>>>>>> (I'm trying to understand why you've seen many test agent
>>>>>>>>>>> processes. It should not happen.)
>>>>>>>>>>>
>>>>>>>>>>> Andrew.
>>>>>>>>>>>
>>>>>>>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>>>>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>>>>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>>>>>>>> This could be the source of the problem. I encountered a
>>>>>>>>>>> dependency issue trying to run the Test Engine on 22.04 LTS, so I
>>>>>>>>>>> downgraded the system. Since the tester is also the host having connection
>>>>>>>>>>> issues, I will try downgrading that system to 20.04, and see if that
>>>>>>>>>>> changes anything.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>>>>>>>>>> Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every
>>>>>>>>>>> night.
>>>>>>>>>>> Right now Debian 11 is used for test engine in nightly
>>>>>>>>>>> regressions.
>>>>>>>>>>>
>>>>>>>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>>>
>>>>>>>>>>> I did try passing in the "--vg-rcf" argument to the run.sh
>>>>>>>>>>> script of the test suite after installing valgrind, but there was no
>>>>>>>>>>> additional output that I saw.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>>>>>>>> (direction where you run test engine).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I will try pulling in the changes you've pushed up, and will see
>>>>>>>>>>> if that fixes anything.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Adam
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello Adam,
>>>>>>>>>>>>
>>>>>>>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to try to repeat the problem locally. Which Linux
>>>>>>>>>>>> distro is running on test engine and agents?
>>>>>>>>>>>>
>>>>>>>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we
>>>>>>>>>>>> have
>>>>>>>>>>>> patch in review to fix it, however, the behaviour is different
>>>>>>>>>>>> in
>>>>>>>>>>>> this case, so it is unlike the same problem.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I've just published a new tag which fixes known test engine
>>>>>>>>>>>> side problems on Debian 12 and Fedora 38.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>>>>>>>> run with option --vg-rcf to check if something weird is
>>>>>>>>>>>> happening.
>>>>>>>>>>>>
>>>>>>>>>>>> What I don't understand right now is why I see just one failed
>>>>>>>>>>>> attempt
>>>>>>>>>>>> to connect in your log.txt and then Logger shutdown after 9
>>>>>>>>>>>> minutes.
>>>>>>>>>>>>
>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>  > Is there any firewall in the network or on test hosts which
>>>>>>>>>>>> could block incoming TCP connection to the port 23571
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the
>>>>>>>>>>>> host where you run test engine?
>>>>>>>>>>>>
>>>>>>>>>>>> Our test engine host and the testbed are on the same subnet.
>>>>>>>>>>>> The connection does work sometimes.
>>>>>>>>>>>>
>>>>>>>>>>>>  > If behaviour the same on the next try and you see that test
>>>>>>>>>>>> agent is kept running, could you check using
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > # netstat -tnlp
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > that Test Agent is listening on the port and try to
>>>>>>>>>>>> establish TCP connection from test agent using
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > and check if TCP connection could be established.
>>>>>>>>>>>>
>>>>>>>>>>>> I was able to replicate the same behavior again, where it hangs
>>>>>>>>>>>> while RCF is trying to start.
>>>>>>>>>>>> Running this command, I see this in the output:
>>>>>>>>>>>>
>>>>>>>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*
>>>>>>>>>>>> LISTEN      18599/ta
>>>>>>>>>>>>
>>>>>>>>>>>> So it seems like it is listening on the correct port.
>>>>>>>>>>>> Additionally, I was able to connect to the Tester machine from
>>>>>>>>>>>> our Test Engine host using telnet. It printed the PID of the process once
>>>>>>>>>>>> the connection was opened.
>>>>>>>>>>>>
>>>>>>>>>>>> I tried running the "ta" application manually on the command
>>>>>>>>>>>> line, and it didn't print anything at all.
>>>>>>>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>>>>>>>> andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>     Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>      > On the tester host (which appears to be the Peer agent),
>>>>>>>>>>>> there
>>>>>>>>>>>>     are four processes that I see running, which look like the
>>>>>>>>>>>> test
>>>>>>>>>>>>     agent processes.
>>>>>>>>>>>>
>>>>>>>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>>>>>>>
>>>>>>>>>>>>     Is there any firewall in the network or on test hosts which
>>>>>>>>>>>> could
>>>>>>>>>>>>     block incoming TCP connection to the port 23571
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>>>>>     where you run test engine?
>>>>>>>>>>>>
>>>>>>>>>>>>     If behaviour the same on the next try and you see that test
>>>>>>>>>>>> agent is
>>>>>>>>>>>>     kept running, could you check using
>>>>>>>>>>>>
>>>>>>>>>>>>     # netstat -tnlp
>>>>>>>>>>>>
>>>>>>>>>>>>     that Test Agent is listening on the port and try to
>>>>>>>>>>>> establish TCP
>>>>>>>>>>>>     connection from test agent using
>>>>>>>>>>>>
>>>>>>>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>     and check if TCP connection could be established.
>>>>>>>>>>>>
>>>>>>>>>>>>     Another idea is to login Tester under root as testing does,
>>>>>>>>>>>> get
>>>>>>>>>>>>     start TA command from the log and try it by hands without
>>>>>>>>>>>> -n and
>>>>>>>>>>>>     remove extra escaping.
>>>>>>>>>>>>
>>>>>>>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>
>>>>>>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>>>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>
>>>>>>>>>>>>     Hopefully in this case test agent directory remains in the
>>>>>>>>>>>> /tmp and
>>>>>>>>>>>>     you don't need to copy it as testing does.
>>>>>>>>>>>>     May be output could shed some light on what's going on.
>>>>>>>>>>>>
>>>>>>>>>>>>     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>     Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>     This is the output that I see in the terminal when this
>>>>>>>>>>>> failure
>>>>>>>>>>>>     occurs, after the test agent binaries build and the test
>>>>>>>>>>>> engine
>>>>>>>>>>>>     starts:
>>>>>>>>>>>>
>>>>>>>>>>>>     Platform default build - pass
>>>>>>>>>>>>     Simple RCF consistency check succeeded
>>>>>>>>>>>>     --->>> Starting Logger...done
>>>>>>>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>>>>>>>>> timed
>>>>>>>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>>>>>>>
>>>>>>>>>>>>     On the tester host (which appears to be the Peer agent),
>>>>>>>>>>>> there are
>>>>>>>>>>>>     four processes that I see running, which look like the test
>>>>>>>>>>>> agent
>>>>>>>>>>>>     processes.
>>>>>>>>>>>>
>>>>>>>>>>>>     ta.Peer is an empty file. I've attached the log.txt from
>>>>>>>>>>>> this run.
>>>>>>>>>>>>
>>>>>>>>>>>>      - Adam
>>>>>>>>>>>>
>>>>>>>>>>>>     On Thu, Aug 24, 2023 at 4:22��AM Andrew Rybchenko
>>>>>>>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>         Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>>>>>>> double-checked
>>>>>>>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>>>>>>         Description in in
>>>>>>>>>>>> doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>         says that copy_timeout is in seconds and implementation
>>>>>>>>>>>> in
>>>>>>>>>>>>         lib/rcfunix/rcfunix.c passes the value to select()
>>>>>>>>>>>> tv_sec.
>>>>>>>>>>>>         Theoretically select() could be interrupted by signal,
>>>>>>>>>>>> but I
>>>>>>>>>>>>         think it is unlikely here.
>>>>>>>>>>>>
>>>>>>>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>>>>>>>         connection timeout. Does it happen on TE startup when
>>>>>>>>>>>> RCF
>>>>>>>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>>> help. Or
>>>>>>>>>>>>         does it happen when tests are in progress, e.g. in the
>>>>>>>>>>>> middle
>>>>>>>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and
>>>>>>>>>>>> most
>>>>>>>>>>>>         likely either host with test agent dies or test agent
>>>>>>>>>>>> itself
>>>>>>>>>>>>         crashes. It would be easier for me if classify it if
>>>>>>>>>>>> you share
>>>>>>>>>>>>         text log (log.txt, full or just corresponding fragment
>>>>>>>>>>>> with
>>>>>>>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>>>>>>>         depending on which agent has problems could shed some
>>>>>>>>>>>> light.
>>>>>>>>>>>>         Corresponding files contain stdout/stderr of test
>>>>>>>>>>>> agents.
>>>>>>>>>>>>
>>>>>>>>>>>>         Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>         Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>         I've set up a test rig repository here, and have created
>>>>>>>>>>>>         configurations for our development testbed based off of
>>>>>>>>>>>> the
>>>>>>>>>>>>         examples.
>>>>>>>>>>>>         We've been able to get the test suite to run manually on
>>>>>>>>>>>>         Mellanox CX5 devices once.
>>>>>>>>>>>>         However, we are running into an issue where, when RCF
>>>>>>>>>>>> starts,
>>>>>>>>>>>>         the RCF connection times out very frequently. We aren't
>>>>>>>>>>>> sure
>>>>>>>>>>>>         why this is the case.
>>>>>>>>>>>>         It works sometimes, but most of the time when we try to
>>>>>>>>>>>> run
>>>>>>>>>>>>         the test engine, it encounters this issue.
>>>>>>>>>>>>         I've tried changing the RCF port by setting
>>>>>>>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the
>>>>>>>>>>>> testbed
>>>>>>>>>>>>         machines. Neither seems to fix the issue.
>>>>>>>>>>>>
>>>>>>>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>>>>>>>         seconds, even when running "export
>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>         before I try to run the test suite.
>>>>>>>>>>>>         I assume the unit for this variable is seconds?
>>>>>>>>>>>>
>>>>>>>>>>>>         Thanks,
>>>>>>>>>>>>         Adam
>>>>>>>>>>>>
>>>>>>>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>             Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>             Thanks, I've cloned the example repository and will
>>>>>>>>>>>> start
>>>>>>>>>>>>             setting up a configuration for our development
>>>>>>>>>>>> testbed
>>>>>>>>>>>>             today. I'll let you know if I run into any
>>>>>>>>>>>> difficulties
>>>>>>>>>>>>             or have any questions.
>>>>>>>>>>>>
>>>>>>>>>>>>              - Adam
>>>>>>>>>>>>
>>>>>>>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                 I've published
>>>>>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>                 Hopefully it will help to define your test rigs
>>>>>>>>>>>> and
>>>>>>>>>>>>                 successfully run some tests manually. Feel free
>>>>>>>>>>>> to
>>>>>>>>>>>>                 ask any questions and I'll answer here and try
>>>>>>>>>>>> to
>>>>>>>>>>>>                 update documentation.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Meanwhile I'll prepare missing bits for steps
>>>>>>>>>>>> (2) and
>>>>>>>>>>>>                 (3).
>>>>>>>>>>>>                 Hopefully everything is in place for step (4),
>>>>>>>>>>>> but we
>>>>>>>>>>>>                 need to make steps (2) and (3) first.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                 > I've conferred with the rest of the team, and
>>>>>>>>>>>> we
>>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>> mainly
>>>>>>>>>>>>                 option B.
>>>>>>>>>>>>
>>>>>>>>>>>>                 OK, I'll provide the sample on Monday for you.
>>>>>>>>>>>> It is
>>>>>>>>>>>>                 almost ready right now, but I need to
>>>>>>>>>>>> double-check
>>>>>>>>>>>>                 it before publishing.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Regards,
>>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>>>>>                 conversation. Others in the community might find
>>>>>>>>>>>>                 this conversation valuable.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We do want to run testing on a regular basis.
>>>>>>>>>>>> The
>>>>>>>>>>>>                 Jenkins integration will be very useful for us,
>>>>>>>>>>>> as
>>>>>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>> mainly
>>>>>>>>>>>>                 option B.
>>>>>>>>>>>>                 If you would like to know anything about our
>>>>>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>>>>>>>                 questions you have.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>>>>>                 hardware and CPU architecture. I figured this
>>>>>>>>>>>> might
>>>>>>>>>>>>                 be an important detail.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Thanks,
>>>>>>>>>>>>                 Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>>>>>> Rybchenko
>>>>>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                     Greatings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                     I'm happy to hear that you're trying to
>>>>>>>>>>>> bring
>>>>>>>>>>>>                     it up.
>>>>>>>>>>>>
>>>>>>>>>>>>                     As I understand the final goal is to run it
>>>>>>>>>>>> on
>>>>>>>>>>>>                     regular basis. So, we need to make it
>>>>>>>>>>>> properly
>>>>>>>>>>>>                     from the very beginning.
>>>>>>>>>>>>                     Bring up of all features consists of 4
>>>>>>>>>>>> steps:
>>>>>>>>>>>>
>>>>>>>>>>>>                     1. Create site-specific repository (we call
>>>>>>>>>>>> it
>>>>>>>>>>>>                     ts-rigs) which contains information about
>>>>>>>>>>>> test
>>>>>>>>>>>>                     rigs and other site-specific information
>>>>>>>>>>>> like
>>>>>>>>>>>>                     where to send mails, where to store logs
>>>>>>>>>>>> etc.
>>>>>>>>>>>>                     It is required for manual execution as well,
>>>>>>>>>>>>                     since test rigs description is essential.
>>>>>>>>>>>> I'll
>>>>>>>>>>>>                     return to the topic below.
>>>>>>>>>>>>
>>>>>>>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>>>>>>>                     Basically it is a disk space plus apache2
>>>>>>>>>>>> web
>>>>>>>>>>>>                     server with few CGI scripts which help a
>>>>>>>>>>>> lot to
>>>>>>>>>>>>                     save disk space.
>>>>>>>>>>>>
>>>>>>>>>>>>                     3. Setup Bublik web application which
>>>>>>>>>>>> provides
>>>>>>>>>>>>                     web interface to view testing results. Same
>>>>>>>>>>>> as
>>>>>>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>>>>>> <https://ts-factory.io/bublik>
>>>>>>>>>>>>
>>>>>>>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>>>>>>>                     save logs in log storage (2) and import it
>>>>>>>>>>>> to
>>>>>>>>>>>>                     bublik (3).
>>>>>>>>>>>>
>>>>>>>>>>>>                     Last few month we spent on our homework to
>>>>>>>>>>>> make
>>>>>>>>>>>>                     it simpler to bring up automated execution
>>>>>>>>>>>>                     using Jenkins -
>>>>>>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>>>>>                     available tomorrow.
>>>>>>>>>>>>
>>>>>>>>>>>>                     Let's return to the step (1).
>>>>>>>>>>>>
>>>>>>>>>>>>                     Unfortunately there is no publicly available
>>>>>>>>>>>>                     example of the ts-rigs repository since
>>>>>>>>>>>>                     sensitive site-specific information is
>>>>>>>>>>>> located
>>>>>>>>>>>>                     there. But I'm ready to help you to create
>>>>>>>>>>>> it
>>>>>>>>>>>>                     for UNH. I see two options here:
>>>>>>>>>>>>
>>>>>>>>>>>>                     (A) I'll ask questions and based on your
>>>>>>>>>>>>                     answers will create the first draft with my
>>>>>>>>>>>>                     comments.
>>>>>>>>>>>>
>>>>>>>>>>>>                     (B) I'll make a template/example ts-rigs
>>>>>>>>>>>> repo,
>>>>>>>>>>>>                     publish it and you'll create UNH ts-rigs
>>>>>>>>>>>> based
>>>>>>>>>>>>                     on it.
>>>>>>>>>>>>
>>>>>>>>>>>>                     Of course, I'll help to debug and finally
>>>>>>>>>>>> bring
>>>>>>>>>>>>                     it up in any case.
>>>>>>>>>>>>
>>>>>>>>>>>>                     (A) is a bit simpler for me and you, but
>>>>>>>>>>>> (B) is
>>>>>>>>>>>>                     a bit more generic and will help other
>>>>>>>>>>>>                     potential users to bring it up.
>>>>>>>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>>>>>>>                     What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>>                     Thanks,
>>>>>>>>>>>>                     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                     Greetings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>>>>>>                     would be happy to help
>>>>>>>>>>>>
>>>>>>>>>>>>                     Thanks,
>>>>>>>>>>>>                     Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>                     <ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>>                     Greetings Konstantin,
>>>>>>>>>>>>
>>>>>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>>>>>                     Poll Mode Driver test suite as an addition
>>>>>>>>>>>> to
>>>>>>>>>>>>                     our testing coverage for DPDK at the UNH
>>>>>>>>>>>> lab.
>>>>>>>>>>>>
>>>>>>>>>>>>                     I have some questions about how to set the
>>>>>>>>>>>>                     test suite arguments.
>>>>>>>>>>>>
>>>>>>>>>>>>                     I have been able to configure the Test
>>>>>>>>>>>> Engine
>>>>>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>>>>>                     fails to run.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>
>>>>>>>>>>>>                     The documentation mentions that there are
>>>>>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>>>>>                     seem like they would need to be set
>>>>>>>>>>>> somewhere
>>>>>>>>>>>>                     to run many of the tests.
>>>>>>>>>>>>
>>>>>>>>>>>>                     I see in the Test Engine documentation,
>>>>>>>>>>>> there
>>>>>>>>>>>>                     are instructions on how to create new
>>>>>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>>>>>                     some other way.
>>>>>>>>>>>>
>>>>>>>>>>>>                     How should these values be set?
>>>>>>>>>>>>
>>>>>>>>>>>>                     I'm also not sure what environment
>>>>>>>>>>>>                     variables/arguments are strictly necessary
>>>>>>>>>>>> or
>>>>>>>>>>>>                     which are optional.
>>>>>>>>>>>>
>>>>>>>>>>>>                     Regards,
>>>>>>>>>>>>                     Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>>>>>                     Senior Developer
>>>>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 --                 *Adam Hassick*
>>>>>>>>>>>>                 Senior Developer
>>>>>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>>> <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>             --             *Adam Hassick*
>>>>>>>>>>>>             Senior Developer
>>>>>>>>>>>>             UNH InterOperability Lab
>>>>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>             +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>         --         *Adam Hassick*
>>>>>>>>>>>>         Senior Developer
>>>>>>>>>>>>         UNH InterOperability Lab
>>>>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>         +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>     --     *Adam Hassick*
>>>>>>>>>>>>     Senior Developer
>>>>>>>>>>>>     UNH InterOperability Lab
>>>>>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>>> Senior Developer
>>>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>> Senior Developer
>>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Adam Hassick*
>>>>>>>>>> Senior Developer
>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Adam Hassick*
>>>>>>>>> Senior Developer
>>>>>>>>> UNH InterOperability Lab
>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Adam Hassick*
>>>>>>> Senior Developer
>>>>>>> UNH InterOperability Lab
>>>>>>> ahassick@iol.unh.edu
>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Adam Hassick*
>>>>>> Senior Developer
>>>>>> UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu
>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 239754 bytes --]

[-- Attachment #2: id_ed25519.pub --]
[-- Type: application/octet-stream, Size: 104 bytes --]

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKKuH2LWlFDdPqgnoNM9FFRIMZGV6/a1O5C0JV0JqemZ root@ts-factory-runner