DPDK CI discussions
 help / color / mirror / Atom feed
From: Konstantin Ushakov <Konstantin.Ushakov@oktetlabs.ru>
To: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Cc: Adam Hassick <ahassick@iol.unh.edu>,
	Patrick Robb <probb@iol.unh.edu>,
	ci@dpdk.org
Subject: Re: Setting up DPDK PMD Test Suite
Date: Mon, 18 Sep 2023 09:23:39 +0300	[thread overview]
Message-ID: <AAF8D161-89D0-4192-920D-EB58B61100D9@oktetlabs.ru> (raw)
In-Reply-To: <ec55a18a-0f0e-fedf-01eb-577e8286c660@oktetlabs.ru>

[-- Attachment #1: Type: text/plain, Size: 91484 bytes --]

Hi Andrew,

should we always auto-assign the tags or you don’t do it since it 
slows down (by some seconds) the TE startup?

Hi Adam,

I think I second the question from Andrew - happy to help you with the 
triage so that we get to the same baseline. Do you have a good way for 
us to share the logs? I.e. say upload to ts-factory if we add strict 
permissions system so it’s not publishing or any other way.

Thanks,
Konstantin


On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:

> Hi Adam,
>
> I've uploaded fresh testing results to ts-factory.io [1] to be on the 
> same page.
>
> I think I know why your and mine results on Intel 710 series NICs 
> differ so much. Testing results expectations database 
> (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags.  I.e. 
> expectations depends on TRC tags discovered by helper scripts when 
> testing is started. These tags identify various aspects of what is 
> tested. Ideally expectations should be written in terms of root cause 
> of the expected behaviour. If it is a driver expectations, driver tag 
> should be used. If it is HW limitation, tags with PCI IDs should be 
> used. However, it is not always easy to classify it correctly if 
> you're not involved in driver development. So, in order case 
> expectations for 710's Intel are filled in in terms of PCI IDs. I 
> guess PCI ID differ in your case and that's why expectations filled in 
> for my NIC do not apply to your runs.
>
> Just try to add the following option when you run on your 710's Intel 
> in order to mimic mine and see if it helps to achieve better pass 
> rate.
> --trc-tag=pci-8086-1572
>
> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for 
> --tester-dial option. It should have better coverage now.
>
> Andrew.
>
> [1] 
> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>
> On 9/13/23 18:45, Andrew Rybchenko wrote:
>> Hi Adam,
>>
>> I've pushed new TE tag v1.20.0 which supported a new command-line 
>> option --tester-dial=NUM where NUM is from 0 to 100. it allows to 
>> choose percentage of tests to run. If you want stable set, you should 
>> pass --tester-random-seed=0 (or other integer). It is the first 
>> sketch and we have plans to improve it, but feedback would be 
>> welcome.
>>
>>> Is it needed on the tester?
>>
>> It is hard to say if it is strictly required for simple tests. 
>> However, it is better to update Tester as well, since performance 
>> tests run DPDK on Tester as well.
>>
>>> Are there any other manual setup steps for these devices that I 
>>> might be missing?
>>
>> I don't remember anything else.
>>
>> I think it is better to get down to details and take a look at logs. 
>> I'm ready to help with it and explain what's happening there. May be 
>> it will help to understand if it is a problem with 
>> setup/configuration.
>>
>> Text logs are not very convenient. Ideally logs should be imported to 
>> bublik, however, manual runs do not provide all required artifacts 
>> right now (Jenkins jobs generate all required artifacts).
>> Other option is 'tmp_raw_log' file (should be packed to make it 
>> smaller) which could be converted to various log formats.
>> Would it be OK for you if I import your logs to bublik at 
>> ts-factory.io? Or is it a problem that it is publicly available?
>> Would it help if we add authentication and access control there?
>>
>> Andrew.
>>
>> On 9/8/23 17:57, Adam Hassick wrote:
>>> Hi Andrew,
>>>
>>> I have a couple questions about needed setup of the NICs for the 
>>> ethdev test suite.
>>>
>>> Our MCX5s and XL710s are failing the checkup tests. The pass rate 
>>> appears to be much worse on the XL710s (40 of 73 tests failed, 3 
>>> passed unexpectedly).
>>>
>>> For the XL710s, I've updated the driver and NVM versions to match 
>>> the minimum supported versions in the compatibility matrix found on 
>>> the DPDK documentation. This did not change the failure rate much.
>>> For the MCX5s, I've installed the latest LTS version of the OFED 
>>> bifurcated driver on the DUT. Is it needed on the tester?
>>>
>>> Are there any other manual setup steps for these devices that I 
>>> might be missing?
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick 
>>> <ahassick@iol.unh.edu> wrote:
>>>
>>>     Hi Andrew,
>>>
>>>     Yes, I copied the X710 configs to set up XL710 configs. I 
>>> changed
>>>     the environment variable names from the X710 suffix to XL710
>>>     suffix in the script, and forgot to change them in the
>>>     corresponding environment file.
>>>     That fixed the issue.
>>>
>>>     I got the checkup tests working on the XL710 now. Most of them
>>>     are failing, which leads me to believe this is an issue with our
>>>     testbed. Based on the DPDK documentation for i40e, the firmware
>>>     and driver versions are much older than what DPDK 22.11 LTS and
>>>     main prefer, so I'll try updating those.
>>>
>>>     For now I'm working on getting the XL710 checkup tests passing,
>>>     and will pick up getting the E810 configured properly next. I'll
>>>     let you know if I run into any more issues in relation to the
>>>     test engine.
>>>
>>>     Thanks,
>>>     Adam
>>>
>>>     On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>         Hi Adam,
>>>
>>>         On 9/5/23 18:01, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         The compilation warning issue is now resolved. Again, thank
>>>>         you guys for fixing this for us. I can run the tests on the
>>>>         Mellanox CX5s again, however I'm running into a couple new
>>>>         issues with running the prologues on the Intel cards.
>>>>
>>>>         When running testing on the Intel XL710s, I see this error
>>>>         appear in the log:
>>>>
>>>>             ERROR  prologue  Environment LIB  14:16:13.650
>>>>             Too few networks in available configuration (0) in
>>>>             comparison with required (1)
>>>>
>>>>
>>>>         This seems like a trivial configuration error, perhaps this
>>>>         is something I need to set up in ts-rigs. I briefly 
>>>> searched
>>>>         through the examples there and didn't see any mention of 
>>>> how
>>>>         to set up a network.
>>>>         I will attach this log just in case you need more 
>>>> information.
>>>
>>>         Unfortunately logs are insufficient to understand it. I've
>>>         pushed new tag to TE v1.19.0 which add log message with TE_*
>>>         environment variables.
>>>         Most likely something is wrong with variables which are used
>>>         as conditions when available networks are defined in
>>>         ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>         TE_PCI_INSTANCE_IUT_TST1
>>>         TE_PCI_INSTANCE_IUT_TST1a
>>>         TE_PCI_INSTANCE_TST1a_IUT
>>>         TE_PCI_INSTANCE_TST1_IUT
>>>         My guess it that you change naming a bit, but script like
>>>         ts-rigs-sample/scripts/iut.h1-x710 is not included or not
>>>         updated.
>>>
>>>>         There is a different error when running on the Intel E810s.
>>>>         It appears to me like it starts DPDK, does some
>>>>         configuration inside DPDK and on the device, and then fails
>>>>         to bring the device back up. Since this error seems very
>>>>         non-trivial, I will also attach this log.
>>>
>>>         This one is a bit simpler. Few lines after the first ERROR 
>>> in
>>>         log I see the following:
>>>         WARN  RCF  DPDK  13:06:00.144
>>>         ice_program_hw_rx_queue(): currently package doesn't support
>>>         RXDID (22)
>>>         ice_rx_queue_start(): fail to program RX queue 0
>>>         ice_dev_start(): fail to start Rx queue 0
>>>         Device with port_id=0 already stopped
>>>
>>>         It is stdout/stderr from test agent which runs DPDK. Same
>>>         logs in plain format are available in ta.DPDK file.
>>>         I'm not an expert here, but I vaguely remember that E810
>>>         requires correct firmware and DDP to be loaded.
>>>         There is some information in dpdk/doc/guides/nics/ice.rst.
>>>
>>>         You can try to add --dev-args=safe-mode-support=1
>>>         command-line option described there.
>>>
>>>         Hope it helps,
>>>         Andrew.
>>>
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko
>>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>             Hi Adam,
>>>>
>>>>             On 8/31/23 22:38, Adam Hassick wrote:
>>>>>             Hi Andrew,
>>>>>
>>>>>             I have one additional question as well: Does the test
>>>>>             engine support running tests on two ARMv8 test agents?
>>>>>
>>>>>                 1. We'll sort out warnings this week. Thanks for
>>>>>                 heads up.
>>>>>
>>>>>
>>>>>             Great. Let me know when that's fixed.
>>>>
>>>>             Done. We also fixed a number of warnings in TE.
>>>>             Also we fixed root test package name to be consistent
>>>>             with the repository name.
>>>>
>>>>>                 Support for old LTS branches was dropped some time
>>>>>                 ago, but in the future it is definitely possible 
>>>>> to
>>>>>                 keep it for new LTS branches. I think 22.11 is
>>>>>                 supported, but I'm not sure about older LTS 
>>>>> releases.
>>>>>
>>>>>
>>>>>             Good to know.
>>>>>
>>>>>                 2. You can add command-line option --sanity to run
>>>>>                 tests marked with TEST_HARNESS_SANITY requirement
>>>>>                 (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>>>                 TEST_HARNESS_SANITY dpdk-ethdev-ts to see which
>>>>>                 tests are marked). Yes, there is a space for
>>>>>                 terminology improvement here. We'll do it.
>>>>>
>>>>
>>>>             Done. Now it is called --checkup.
>>>>
>>>>>
>>>>>                 Also it takes a lot of time because of failures 
>>>>> and
>>>>>                 tests which wait for some timeout.
>>>>>
>>>>>
>>>>>             That makes sense to me. We'll use the time to complete
>>>>>             tests on virtio or the Intel devices as a reference 
>>>>> for
>>>>>             how long the tests really take to complete.
>>>>>             We will explore the possibility of periodically 
>>>>> running
>>>>>             the sanity tests for patches.
>>>>
>>>>             I'll double-check and let you know how long entire TS
>>>>             runs on Intel X710, E810, Mellanox CX5 and virtio net.
>>>>             Just to ensure that time observed in your case looks 
>>>> the
>>>>             same.
>>>>
>>>>>
>>>>>                 The test harness can provide coverage reports 
>>>>> based
>>>>>                 on gcov, but I'm not sure what you mean by a 
>>>>> "dial"
>>>>>                 to control test coverage. Provided reports are
>>>>>                 rather for human to analyze.
>>>>>
>>>>>
>>>>>             The general idea is to have some kind of parameter on
>>>>>             the test suite, which could be an integer ranging from
>>>>>             zero to ten, that controls how many tests are run 
>>>>> based
>>>>>             on how important the test is.
>>>>>
>>>>>             Similar to how some command line interfaces provide a
>>>>>             verbosity level parameter (some number of "-v"
>>>>>             arguments) to control the importance of the 
>>>>> information
>>>>>             in the log.
>>>>>             The verbosity level zero only prints very important 
>>>>> log
>>>>>             messages, while ten prints everything.
>>>>>
>>>>>             In much the same manner as above, this "dial" 
>>>>> parameter
>>>>>             controls what tests are run and with what parameters
>>>>>             based on how important those tests and test parameter
>>>>>             combinations are.
>>>>>             Coverage Level zero tells the suite to run a very 
>>>>> basic
>>>>>             set of important tests, with minimal parameterization.
>>>>>             This mode would take only ~5-10 minutes to run.
>>>>>             In contrast, Coverage Level ten includes all the edge
>>>>>             cases, every combination of test parameters, 
>>>>> everything
>>>>>             the test suite can do, which takes the normal several
>>>>>             hours to run.
>>>>>             The values 1 - 9 are between those two extremes,
>>>>>             allowing the user to get a gradient of test coverage 
>>>>> in
>>>>>             the results and to limit the running time.
>>>>>
>>>>>             Then we could, for example, run the "run.sh" with a
>>>>>             level of 2 or 3 for incoming patches that need quick
>>>>>             results, and with a level of 10 for the less often run
>>>>>             periodic tests performed on main or LTS branches.
>>>>
>>>>             Understood now. Thanks a lot for the idea. We'll 
>>>> discuss
>>>>             it and come back.
>>>>
>>>>>                 3. Yes, really many tests on Mellanox CX5 NICs
>>>>>                 report unexpected testing results. Unfortunately 
>>>>> it
>>>>>                 is time consuming to fill in expectations database
>>>>>                 since it is necessary to analyze testing results
>>>>>                 and classify if it is a bug or just acceptable
>>>>>                 behaviour aspect.
>>>>>
>>>>>                 Bublik allows to compare results of two runs. It 
>>>>> is
>>>>>                 useful for human, but still not good for 
>>>>> automation.
>>>>>
>>>>>                 I have local patch for mlx5 driver which reports 
>>>>> Tx
>>>>>                 ring size maximum. It makes pass rate higher. It 
>>>>> is
>>>>>                 a problem for test harness that mlx5 does not
>>>>>                 report limits right now.
>>>>>
>>>>>                 Pass rate on Intel X710 is about 92% on my test
>>>>>                 rig. Pass rate on virtio net is 99% right now and
>>>>>                 could be done 100% easily (just one thing to fix 
>>>>> in
>>>>>                 expectations).
>>>>>
>>>>>                 I think logs storage setup is essential for logs
>>>>>                 analysis. Of course, you can request HTML logs 
>>>>> when
>>>>>                 you run tests (--log-html=html) or generate after
>>>>>                 run using dpdk-ethdev-ts/scripts/html-log.sh and
>>>>>                 open index.html in a browser, but logs storage
>>>>>                 makes it more convenient.
>>>>>
>>>>>
>>>>>             We are interested in setting up Bublik, potentially as
>>>>>             an externally-facing component, once we have our
>>>>>             process of running the test suite stabilized.
>>>>>             Once we are able to run the test suite again, I'll see
>>>>>             what the pass rate is on our other hardware.
>>>>>             Good to know that it isn't an issue with our dev
>>>>>             testbed causing the high fail rate.
>>>>>
>>>>>             For Intel hardware, we have an XL710 and an Intel
>>>>>             E810-C in our development testbed. Although they are
>>>>>             slightly different devices, ideally the pass rate will
>>>>>             be identical or similar. I have yet to set up a VM 
>>>>> pair
>>>>>             for virtio, but we will soon.
>>>>>
>>>>>                 Latest version of test-environment has examples of
>>>>>                 our CGI scripts which we use for log storage (see
>>>>>                 tools/log_server/README.md).
>>>>>
>>>>>                 Also all bits for Jenkins setup are available. See
>>>>>                 dpdk-ethdev-ts/jenkins/README.md and examples of
>>>>>                 jenkins files in ts-rigs-sample.
>>>>>
>>>>>
>>>>>             Jenkins integration, setting up production rig
>>>>>             configurations, and permanent log storage will be our
>>>>>             next steps once I am able to run the tests again.
>>>>>             Unless there is an easy way to have meson not pass
>>>>>             "-Werror" into GCC. Then I would be able to run the
>>>>>             test suite.
>>>>
>>>>             Hopefully it is resolved now.
>>>>
>>>>             I thought a bit more about your usecase for Jenkins. 
>>>> I'm
>>>>             not 100% sure that existing pipelines are convenient 
>>>> for
>>>>             your usecase.
>>>>             Fill free to ask questions when you are on it.
>>>>
>>>>             Thanks,
>>>>             Andrew.
>>>>
>>>>>
>>>>>             Thanks,
>>>>>             Adam
>>>>>
>>>>>
>>>>>                 On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 That fix seems to have resolved the issue, thanks
>>>>>>                 for the quick turnaround time on that patch.
>>>>>>                 Now that we have the RCF timeout issue resolved,
>>>>>>                 there are a few other questions and issues that 
>>>>>> we
>>>>>>                 have about the tests themselves.
>>>>>>
>>>>>>                 1. The test suite fails to build with a couple
>>>>>>                 warnings.
>>>>>>
>>>>>>                 Below is the stderr log from compilation:
>>>>>>
>>>>>>                     FAILED:
>>>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>>                     -I../../lib
>>>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>                     -fdiagnostics-color=always -pipe
>>>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>>                     -fPIC -MD -MQ
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -MF
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>                     -o
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>>>                     ‘test_create_traffic_generator_params’:
>>>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>>>                     not a string literal and no format arguments
>>>>>>                     [-Werror=format-security]
>>>>>>                     5577 |     rc = te_kvpair_add(result, buf, 
>>>>>> mode);
>>>>>>                     |     ^~
>>>>>>                     cc1: all warnings being treated as errors
>>>>>>                     ninja: build stopped: subcommand failed.
>>>>>>                     ninja: Entering directory `.'
>>>>>>                     FAILED:
>>>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>>                     -I../../lib
>>>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>                     -fdiagnostics-color=always -pipe
>>>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>>                     -fPIC -MD -MQ
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -MF
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>                     -o
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>>>                     ‘test_create_traffic_generator_params’:
>>>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>>>                     not a string literal and no format arguments
>>>>>>                     [-Werror=format-security]
>>>>>>                     5577 |     rc = te_kvpair_add(result, buf, 
>>>>>> mode);
>>>>>>                     |     ^~
>>>>>>                     cc1: all warnings being treated as errors
>>>>>>
>>>>>>
>>>>>>                 This error wasn't occurring last week, which was
>>>>>>                 the last time I ran the tests.
>>>>>>                 The TE host and the DUT have GCC v9.4.0 
>>>>>> installed,
>>>>>>                 and the tester has GCC v11.4.0 installed, if this
>>>>>>                 information is helpful.
>>>>>>
>>>>>>                 2. On the Mellanox CX5s, there are over 6,000
>>>>>>                 tests run, which collectively take around 9 
>>>>>> hours.
>>>>>>                 Is it possible, and would it make sense, to lower
>>>>>>                 the test coverage and have the test suite run 
>>>>>> faster?
>>>>>>
>>>>>>                 For some context, we run immediate testing on
>>>>>>                 incoming patches for DPDK main and development
>>>>>>                 branches, as well as periodic test runs on the
>>>>>>                 main, stable, and LTS branches.
>>>>>>                 For us to consider including this test suite as
>>>>>>                 part of our immediate testing on patches, we 
>>>>>> would
>>>>>>                 have to reduce the test coverage to the most
>>>>>>                 important tests.
>>>>>>                 This is primarily to reduce the testing time to,
>>>>>>                 for example, less than 30 minutes. Testing on
>>>>>>                 patches can't take too long because the lab can
>>>>>>                 receive numerous patches each day, which each
>>>>>>                 require individual testing runs.
>>>>>>
>>>>>>                 At what frequency we run these tests, and on 
>>>>>> what,
>>>>>>                 still needs to be discussed with the DPDK
>>>>>>                 community, but it would be nice to know if the
>>>>>>                 test suite had a "dial" to control the testing
>>>>>>                 coverage.
>>>>>>
>>>>>>                 3. We see a lot of test failures on our Mellanox
>>>>>>                 CX5 NICs. Around 2,300 of ~6,600 tests passed. Is
>>>>>>                 there anything we can do to diagnose these test
>>>>>>                 failures?
>>>>>>
>>>>>>                 Thanks,
>>>>>>                 Adam
>>>>>>
>>>>>>
>>>>>>                 On Tue, Aug 29, 2023 at 8:07 AM Andrew 
>>>>>> Rybchenko
>>>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     I've pushed the fix in main branch and a new
>>>>>>                     tag v1.18.1. It should solve the problem with
>>>>>>                     IPv6 address from DNS.
>>>>>>
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>                     Hi Adam,
>>>>>>>
>>>>>>>                     > Does the test engine prefer to use IPv6
>>>>>>>                     over IPv4 for initiating the RCF connection
>>>>>>>                     to the test bed hosts? And if so, is there a
>>>>>>>                     way to force it to use IPv4?
>>>>>>>
>>>>>>>                     Brilliant idea. If DNS returns both IPv4 and
>>>>>>>                     IPv6 addresses in your case, I guess it is
>>>>>>>                     the root cause of the problem.
>>>>>>>                     Of course, it is TE problem since I see
>>>>>>>                     really weird code in
>>>>>>>                     lib/comm_net_engine/comm_net_engine.c line 
>>>>>>> 135.
>>>>>>>
>>>>>>>                     I've pushed fix to the branch
>>>>>>>                     user/arybchik/fix_ipv4_only in
>>>>>>>                     ts-factory/test-environment repository.
>>>>>>>                     Please, try.
>>>>>>>
>>>>>>>                     It is late night fix with minimal testing 
>>>>>>> and
>>>>>>>                     no review. I'll pass it through review
>>>>>>>                     process tomorrow and
>>>>>>>                     hopefully it will be released in one-two 
>>>>>>> days.
>>>>>>>
>>>>>>>                     Andrew.
>>>>>>>
>>>>>>>                     On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>                     Hi Andrew,
>>>>>>>>
>>>>>>>>                     We have yet to notice a distinct pattern
>>>>>>>>                     with the failures. Sometimes, the RCF will
>>>>>>>>                     start and connect without issue a few times
>>>>>>>>                     in a row before failing to connect again.
>>>>>>>>                     Once the issue begins to occur, neither
>>>>>>>>                     rebooting all of the hosts (test engine VM,
>>>>>>>>                     tester, IUT) or deleting all of the build
>>>>>>>>                     directories (suites, agents, inst) and
>>>>>>>>                     rebooting the hosts afterward resolves the
>>>>>>>>                     issue. When it begins working again seems
>>>>>>>>                     very arbitrary to us.
>>>>>>>>
>>>>>>>>                     I do usually try to terminate the test
>>>>>>>>                     engine with Ctrl+C, but when it hangs while
>>>>>>>>                     trying to start RCF, that does not work.
>>>>>>>>
>>>>>>>>                     Does the test engine prefer to use IPv6 
>>>>>>>> over
>>>>>>>>                     IPv4 for initiating the RCF connection to
>>>>>>>>                     the test bed hosts? And if so, is there a
>>>>>>>>                     way to force it to use IPv4?
>>>>>>>>
>>>>>>>>                      - Adam
>>>>>>>>
>>>>>>>>                     On Fri, Aug 25, 2023 at 1:35 PM Andrew
>>>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>                         > I'll double-check test engine on
>>>>>>>>                         Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>
>>>>>>>>                         Done. It works fine for me without any
>>>>>>>>                         issues.
>>>>>>>>
>>>>>>>>                         Have you noticed any pattern when it
>>>>>>>>                         works or does not work?
>>>>>>>>                         May be it is a problem of not clean
>>>>>>>>                         state after termination?
>>>>>>>>                         Does it work fine the first time after
>>>>>>>>                         DUTs reboot?
>>>>>>>>                         How do you terminate testing? It should
>>>>>>>>                         be done using Ctrl+C in terminal where
>>>>>>>>                         you execute run.sh command.
>>>>>>>>                          In this case it should shutdown
>>>>>>>>                         gracefully and close all test agents 
>>>>>>>> and
>>>>>>>>                         engine applications.
>>>>>>>>
>>>>>>>>                         (I'm trying to understand why you've
>>>>>>>>                         seen many test agent processes. It
>>>>>>>>                         should not happen.)
>>>>>>>>
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/25/23 17:41, Andrew Rybchenko 
>>>>>>>> wrote:
>>>>>>>>>                         On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>>                         Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                         Two of our systems (the Test Engine
>>>>>>>>>>                         runner and the DUT host) are running
>>>>>>>>>>                         Ubuntu 20.04 LTS, however this 
>>>>>>>>>> morning
>>>>>>>>>>                         I noticed that the tester system (the
>>>>>>>>>>                         one having issues) is running Ubuntu
>>>>>>>>>>                         22.04 LTS.
>>>>>>>>>>                         This could be the source of the
>>>>>>>>>>                         problem. I encountered a dependency
>>>>>>>>>>                         issue trying to run the Test Engine 
>>>>>>>>>> on
>>>>>>>>>>                         22.04 LTS, so I downgraded the 
>>>>>>>>>> system.
>>>>>>>>>>                         Since the tester is also the host
>>>>>>>>>>                         having connection issues, I will try
>>>>>>>>>>                         downgrading that system to 20.04, and
>>>>>>>>>>                         see if that changes anything.
>>>>>>>>>
>>>>>>>>>                         Unlikely, but who knows. We run tests
>>>>>>>>>                         (DUTs) on Ubuntu 20.04, Ubuntu 22.04,
>>>>>>>>>                         Ubuntu 22.10, Ubuntu 23.04, Debian 11
>>>>>>>>>                         and Fedora 38 every night.
>>>>>>>>>                         Right now Debian 11 is used for test
>>>>>>>>>                         engine in nightly regressions.
>>>>>>>>>
>>>>>>>>>                         I'll double-check test engine on 
>>>>>>>>> Ubuntu
>>>>>>>>>                         20.04 and Ubuntu 22.04.
>>>>>>>>>
>>>>>>>>>>                         I did try passing in the "--vg-rcf"
>>>>>>>>>>                         argument to the run.sh script of the
>>>>>>>>>>                         test suite after installing valgrind,
>>>>>>>>>>                         but there was no additional output
>>>>>>>>>>                         that I saw.
>>>>>>>>>
>>>>>>>>>                         Sorry, I should valgrind output should
>>>>>>>>>                         be in valgrind.te_rcf (direction where
>>>>>>>>>                         you run test engine).
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         I will try pulling in the changes
>>>>>>>>>>                         you've pushed up, and will see if 
>>>>>>>>>> that
>>>>>>>>>>                         fixes anything.
>>>>>>>>>>
>>>>>>>>>>                         Thanks,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         On Fri, Aug 25, 2023 at 9:57 AM 
>>>>>>>>>> Andrew
>>>>>>>>>>                         Rybchenko
>>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> 
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>                             Hello Adam,
>>>>>>>>>>
>>>>>>>>>>                             On 8/24/23 23:54, Andrew 
>>>>>>>>>> Rybchenko
>>>>>>>>>>                             wrote:
>>>>>>>>>>>                             I'd like to try to repeat the
>>>>>>>>>>>                             problem locally. Which Linux
>>>>>>>>>>>                             distro is running on test engine
>>>>>>>>>>>                             and agents?
>>>>>>>>>>>
>>>>>>>>>>>                             In fact I know one problem with
>>>>>>>>>>>                             Debian 12 and Fedora 38 and we 
>>>>>>>>>>> have
>>>>>>>>>>>                             patch in review to fix it,
>>>>>>>>>>>                             however, the behaviour is
>>>>>>>>>>>                             different in
>>>>>>>>>>>                             this case, so it is unlike the
>>>>>>>>>>>                             same problem.
>>>>>>>>>>
>>>>>>>>>>                             I've just published a new tag
>>>>>>>>>>                             which fixes known test engine 
>>>>>>>>>> side
>>>>>>>>>>                             problems on Debian 12 and Fedora 
>>>>>>>>>> 38.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             One more idea is to install
>>>>>>>>>>>                             valgrind on the test engine host 
>>>>>>>>>>> and
>>>>>>>>>>>                             run with option --vg-rcf to 
>>>>>>>>>>> check
>>>>>>>>>>>                             if something weird is happening.
>>>>>>>>>>>
>>>>>>>>>>>                             What I don't understand right 
>>>>>>>>>>> now
>>>>>>>>>>>                             is why I see just one failed 
>>>>>>>>>>> attempt
>>>>>>>>>>>                             to connect in your log.txt and
>>>>>>>>>>>                             then Logger shutdown after 9
>>>>>>>>>>>                             minutes.
>>>>>>>>>>>
>>>>>>>>>>>                             Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                             On 8/24/23 23:29, Adam Hassick
>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>                              > Is there any firewall in 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             network or on test hosts which
>>>>>>>>>>>>                             could block incoming TCP
>>>>>>>>>>>>                             connection to the port 23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             from the host where you run 
>>>>>>>>>>>> test
>>>>>>>>>>>>                             engine?
>>>>>>>>>>>>
>>>>>>>>>>>>                             Our test engine host and the
>>>>>>>>>>>>                             testbed are on the same subnet.
>>>>>>>>>>>>                             The connection does work 
>>>>>>>>>>>> sometimes.
>>>>>>>>>>>>
>>>>>>>>>>>>                              > If behaviour the same on 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>>>                             agent is kept running, could 
>>>>>>>>>>>> you
>>>>>>>>>>>>                             check using
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > # netstat -tnlp
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > that Test Agent is 
>>>>>>>>>>>> listening
>>>>>>>>>>>>                             on the port and try to 
>>>>>>>>>>>> establish
>>>>>>>>>>>>                             TCP connection from test agent
>>>>>>>>>>>>                             using
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > $ telnet
>>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > and check if TCP connection
>>>>>>>>>>>>                             could be established.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I was able to replicate the 
>>>>>>>>>>>> same
>>>>>>>>>>>>                             behavior again, where it hangs
>>>>>>>>>>>>                             while RCF is trying to start.
>>>>>>>>>>>>                             Running this command, I see 
>>>>>>>>>>>> this
>>>>>>>>>>>>                             in the output:
>>>>>>>>>>>>
>>>>>>>>>>>>                             tcp        0    0 
>>>>>>>>>>>> 0.0.0.0:23571
>>>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>>>                             <http://0.0.0.0:23571> 
>>>>>>>>>>>> 0.0.0.0:*
>>>>>>>>>>>>                             LISTEN  18599/ta
>>>>>>>>>>>>
>>>>>>>>>>>>                             So it seems like it is 
>>>>>>>>>>>> listening
>>>>>>>>>>>>                             on the correct port.
>>>>>>>>>>>>                             Additionally, I was able to
>>>>>>>>>>>>                             connect to the Tester machine
>>>>>>>>>>>>                             from our Test Engine host using
>>>>>>>>>>>>                             telnet. It printed the PID of
>>>>>>>>>>>>                             the process once the connection
>>>>>>>>>>>>                             was opened.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I tried running the "ta"
>>>>>>>>>>>>                             application manually on the
>>>>>>>>>>>>                             command line, and it didn't
>>>>>>>>>>>>                             print anything at all.
>>>>>>>>>>>>                             Maybe the issue is something on
>>>>>>>>>>>>                             the Test Engine side.
>>>>>>>>>>>>
>>>>>>>>>>>>                             On Thu, Aug 24, 2023 at 
>>>>>>>>>>>> 2:35 PM
>>>>>>>>>>>>                             Andrew Rybchenko
>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                  > On the tester host 
>>>>>>>>>>>> (which
>>>>>>>>>>>>                             appears to be the Peer agent),
>>>>>>>>>>>>                             there
>>>>>>>>>>>>                                 are four processes that 
>>>>>>>>>>>> I
>>>>>>>>>>>>                             see running, which look like 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             test
>>>>>>>>>>>>                                 agent processes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Before the next try I'd
>>>>>>>>>>>>                             recommend to kill these 
>>>>>>>>>>>> processes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Is there any firewall in 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             network or on test hosts which
>>>>>>>>>>>>                             could
>>>>>>>>>>>>                                 block incoming TCP
>>>>>>>>>>>>                             connection to the port 23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             from the host
>>>>>>>>>>>>                                 where you run test 
>>>>>>>>>>>> engine?
>>>>>>>>>>>>
>>>>>>>>>>>>                                 If behaviour the same on 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>>>                             agent is
>>>>>>>>>>>>                                 kept running, could you
>>>>>>>>>>>>                             check using
>>>>>>>>>>>>
>>>>>>>>>>>>                                 # netstat -tnlp
>>>>>>>>>>>>
>>>>>>>>>>>>                                 that Test Agent is 
>>>>>>>>>>>> listening
>>>>>>>>>>>>                             on the port and try to 
>>>>>>>>>>>> establish
>>>>>>>>>>>>                             TCP
>>>>>>>>>>>>                                 connection from test 
>>>>>>>>>>>> agent
>>>>>>>>>>>>                             using
>>>>>>>>>>>>
>>>>>>>>>>>>                                 $ telnet
>>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                 and check if TCP 
>>>>>>>>>>>> connection
>>>>>>>>>>>>                             could be established.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Another idea is to login
>>>>>>>>>>>>                             Tester under root as testing
>>>>>>>>>>>>                             does, get
>>>>>>>>>>>>                                 start TA command from 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             log and try it by hands without
>>>>>>>>>>>>                             -n and
>>>>>>>>>>>>                                 remove extra escaping.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 # sudo
>>>>>>>>>>>>                             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>
>>>>>>>>>>>>                             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>                             /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>                             Peer 23571
>>>>>>>>>>>>                             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Hopefully in this case 
>>>>>>>>>>>> test
>>>>>>>>>>>>                             agent directory remains in the
>>>>>>>>>>>>                             /tmp and
>>>>>>>>>>>>                                 you don't need to copy 
>>>>>>>>>>>> it as
>>>>>>>>>>>>                             testing does.
>>>>>>>>>>>>                                 May be output could shed
>>>>>>>>>>>>                             some light on what's going on.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 On 8/24/23 17:30, Adam
>>>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 This is the output that 
>>>>>>>>>>>>> I
>>>>>>>>>>>>>                             see in the terminal when this
>>>>>>>>>>>>>                             failure
>>>>>>>>>>>>>                                 occurs, after the test
>>>>>>>>>>>>>                             agent binaries build and the
>>>>>>>>>>>>>                             test engine
>>>>>>>>>>>>>                                 starts:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 Platform default build 
>>>>>>>>>>>>> - pass
>>>>>>>>>>>>>                                 Simple RCF consistency
>>>>>>>>>>>>>                             check succeeded
>>>>>>>>>>>>>                             --->>> Starting Logger...done
>>>>>>>>>>>>>                             --->>> Starting
>>>>>>>>>>>>>                             RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>                             Connection timed
>>>>>>>>>>>>>                                 out
>>>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 Then, it hangs here 
>>>>>>>>>>>>> until I
>>>>>>>>>>>>>                             kill the "te_rcf" and "te_tee"
>>>>>>>>>>>>>                                 processes. I let it 
>>>>>>>>>>>>> hang
>>>>>>>>>>>>>                             for around 9 minutes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 On the tester host 
>>>>>>>>>>>>> (which
>>>>>>>>>>>>>                             appears to be the Peer agent),
>>>>>>>>>>>>>                             there are
>>>>>>>>>>>>>                                 four processes that I 
>>>>>>>>>>>>> see
>>>>>>>>>>>>>                             running, which look like the
>>>>>>>>>>>>>                             test agent
>>>>>>>>>>>>>                                 processes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 ta.Peer is an empty 
>>>>>>>>>>>>> file.
>>>>>>>>>>>>>                             I've attached the log.txt from
>>>>>>>>>>>>>                             this run.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                  - Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 On Thu, Aug 24, 2023 at
>>>>>>>>>>>>>                             4:22 AM Andrew Rybchenko
>>>>>>>>>>>>>                                 
>>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Yes, 
>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>                             is in seconds. I've 
>>>>>>>>>>>>> double-checked
>>>>>>>>>>>>>                                     that it goes to
>>>>>>>>>>>>>                             'copy_timeout' in
>>>>>>>>>>>>>                             ts-conf/rcf.conf.
>>>>>>>>>>>>>                             Description in in
>>>>>>>>>>>>>                             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     says that 
>>>>>>>>>>>>> copy_timeout
>>>>>>>>>>>>>                             is in seconds and
>>>>>>>>>>>>>                             implementation in
>>>>>>>>>>>>>                             lib/rcfunix/rcfunix.c passes
>>>>>>>>>>>>>                             the value to select() tv_sec.
>>>>>>>>>>>>>                             Theoretically select() could 
>>>>>>>>>>>>> be
>>>>>>>>>>>>>                             interrupted by signal, but I
>>>>>>>>>>>>>                                     think it is 
>>>>>>>>>>>>> unlikely here.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     I'm not sure 
>>>>>>>>>>>>> that I
>>>>>>>>>>>>>                             understand what do you mean by 
>>>>>>>>>>>>> RCF
>>>>>>>>>>>>>                             connection timeout. Does it
>>>>>>>>>>>>>                             happen on TE startup when RCF
>>>>>>>>>>>>>                                     starts test 
>>>>>>>>>>>>> agents. If
>>>>>>>>>>>>>                             so, TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>>>>                             help. Or
>>>>>>>>>>>>>                                     does it happen 
>>>>>>>>>>>>> when
>>>>>>>>>>>>>                             tests are in progress, e.g. in
>>>>>>>>>>>>>                             the middle
>>>>>>>>>>>>>                                     of a test. If 
>>>>>>>>>>>>> so,
>>>>>>>>>>>>>                             TE_RCFUNIX_TIMEOUT is 
>>>>>>>>>>>>> unrelated
>>>>>>>>>>>>>                             and most
>>>>>>>>>>>>>                                     likely either 
>>>>>>>>>>>>> host with
>>>>>>>>>>>>>                             test agent dies or test agent
>>>>>>>>>>>>>                             itself
>>>>>>>>>>>>>                             crashes. It would be easier 
>>>>>>>>>>>>> for
>>>>>>>>>>>>>                             me if classify it if you share
>>>>>>>>>>>>>                                     text log 
>>>>>>>>>>>>> (log.txt, full
>>>>>>>>>>>>>                             or just corresponding fragment
>>>>>>>>>>>>>                             with
>>>>>>>>>>>>>                                     some context). 
>>>>>>>>>>>>> Also
>>>>>>>>>>>>>                             content of ta.DPDK or ta.Peer 
>>>>>>>>>>>>> file
>>>>>>>>>>>>>                             depending on which agent has
>>>>>>>>>>>>>                             problems could shed some 
>>>>>>>>>>>>> light.
>>>>>>>>>>>>>                             Corresponding files contain
>>>>>>>>>>>>>                             stdout/stderr of test agents.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                             Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     On 8/23/23 
>>>>>>>>>>>>> 17:45, Adam
>>>>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I've set up a 
>>>>>>>>>>>>>> test rig
>>>>>>>>>>>>>>                             repository here, and have 
>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>                             configurations for our
>>>>>>>>>>>>>>                             development testbed based off
>>>>>>>>>>>>>>                             of the
>>>>>>>>>>>>>>                             examples.
>>>>>>>>>>>>>>                                     We've been 
>>>>>>>>>>>>>> able to get
>>>>>>>>>>>>>>                             the test suite to run 
>>>>>>>>>>>>>> manually on
>>>>>>>>>>>>>>                             Mellanox CX5 devices once.
>>>>>>>>>>>>>>                             However, we are running into
>>>>>>>>>>>>>>                             an issue where, when RCF 
>>>>>>>>>>>>>> starts,
>>>>>>>>>>>>>>                                     the RCF 
>>>>>>>>>>>>>> connection
>>>>>>>>>>>>>>                             times out very frequently. We
>>>>>>>>>>>>>>                             aren't sure
>>>>>>>>>>>>>>                                     why this is 
>>>>>>>>>>>>>> the case.
>>>>>>>>>>>>>>                                     It works 
>>>>>>>>>>>>>> sometimes,
>>>>>>>>>>>>>>                             but most of the time when we
>>>>>>>>>>>>>>                             try to run
>>>>>>>>>>>>>>                                     the test 
>>>>>>>>>>>>>> engine, it
>>>>>>>>>>>>>>                             encounters this issue.
>>>>>>>>>>>>>>                                     I've tried 
>>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>                             the RCF port by setting
>>>>>>>>>>>>>>                             "TE_RCF_PORT=<some port
>>>>>>>>>>>>>>                             number>" and rebooting the
>>>>>>>>>>>>>>                             testbed
>>>>>>>>>>>>>>                             machines. Neither seems to 
>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>                             the issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     It also seems 
>>>>>>>>>>>>>> like the
>>>>>>>>>>>>>>                             timeout takes far longer than 
>>>>>>>>>>>>>> 60
>>>>>>>>>>>>>>                             seconds, even when running
>>>>>>>>>>>>>>                             "export 
>>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>                                     before I try 
>>>>>>>>>>>>>> to run
>>>>>>>>>>>>>>                             the test suite.
>>>>>>>>>>>>>>                                     I assume the 
>>>>>>>>>>>>>> unit for
>>>>>>>>>>>>>>                             this variable is seconds?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     On Mon, Aug 
>>>>>>>>>>>>>> 21, 2023
>>>>>>>>>>>>>>                             at 10:19 AM Adam Hassick
>>>>>>>>>>>>>>                                     
>>>>>>>>>>>>>> <ahassick@iol.unh.edu
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         Hi 
>>>>>>>>>>>>>> Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                             Thanks, I've cloned the
>>>>>>>>>>>>>>                             example repository and will 
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>                             setting up a configuration 
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>                             our development testbed
>>>>>>>>>>>>>>                             today. I'll let you know if I
>>>>>>>>>>>>>>                             run into any difficulties
>>>>>>>>>>>>>>                                         or 
>>>>>>>>>>>>>> have any
>>>>>>>>>>>>>>                             questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                          - 
>>>>>>>>>>>>>> Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         On 
>>>>>>>>>>>>>> Sun, Aug 20,
>>>>>>>>>>>>>>                             2023 at 4:40 AM Andrew 
>>>>>>>>>>>>>> Rybchenko
>>>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Hi Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> I've published
>>>>>>>>>>>>>>                            https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample> 
>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Hopefully it
>>>>>>>>>>>>>>                             will help to define your test
>>>>>>>>>>>>>>                             rigs and
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> successfully
>>>>>>>>>>>>>>                             run some tests manually. Feel
>>>>>>>>>>>>>>                             free to
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> ask any
>>>>>>>>>>>>>>                             questions and I'll answer 
>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>                             and try to
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>                             documentation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Meanwhile I'll
>>>>>>>>>>>>>>                             prepare missing bits for 
>>>>>>>>>>>>>> steps
>>>>>>>>>>>>>>                             (2) and
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> (3).
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Hopefully
>>>>>>>>>>>>>>                             everything is in place for
>>>>>>>>>>>>>>                             step (4), but we
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> need to make
>>>>>>>>>>>>>>                             steps (2) and (3) first.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> On 8/18/23
>>>>>>>>>>>>>>                             21:40, Andrew Rybchenko 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>                             Hi Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> > I've
>>>>>>>>>>>>>>>                             conferred with the rest of
>>>>>>>>>>>>>>>                             the team, and we
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> think it
>>>>>>>>>>>>>>>                             would be best to move 
>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>                             with mainly
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> option B.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> OK, I'll
>>>>>>>>>>>>>>>                             provide the sample on Monday
>>>>>>>>>>>>>>>                             for you. It is
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> almost ready
>>>>>>>>>>>>>>>                             right now, but I need to
>>>>>>>>>>>>>>>                             double-check
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> it before
>>>>>>>>>>>>>>>                             publishing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> On 8/17/23
>>>>>>>>>>>>>>>                             20:03, Adam Hassick wrote:
>>>>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> I'm adding
>>>>>>>>>>>>>>>>                             the CI mailing list to this
>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>                             conversation. Others in the
>>>>>>>>>>>>>>>>                             community might find
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>                             conversation valuable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> We do want
>>>>>>>>>>>>>>>>                             to run testing on a regular
>>>>>>>>>>>>>>>>                             basis. The
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Jenkins
>>>>>>>>>>>>>>>>                             integration will be very
>>>>>>>>>>>>>>>>                             useful for us, as
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> most of our
>>>>>>>>>>>>>>>>                             CI is orchestrated by 
>>>>>>>>>>>>>>>> Jenkins.
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>                             conferred with the rest of
>>>>>>>>>>>>>>>>                             the team, and we
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> think it
>>>>>>>>>>>>>>>>                             would be best to move
>>>>>>>>>>>>>>>>                             forward with mainly
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> option B.
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> If you would
>>>>>>>>>>>>>>>>                             like to know anything about 
>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> testbeds
>>>>>>>>>>>>>>>>                             that would help you with
>>>>>>>>>>>>>>>>                             creating an
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> example
>>>>>>>>>>>>>>>>                             ts-rigs repo, I'd be happy
>>>>>>>>>>>>>>>>                             to answer any
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> questions
>>>>>>>>>>>>>>>>                             you have.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> We have
>>>>>>>>>>>>>>>>                             multiple test rigs (we call
>>>>>>>>>>>>>>>>                             these
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> "DUT-tester
>>>>>>>>>>>>>>>>                             pairs") that we run our
>>>>>>>>>>>>>>>>                             existing
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> hardware
>>>>>>>>>>>>>>>>                             testing on, with differing
>>>>>>>>>>>>>>>>                             network
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> hardware and
>>>>>>>>>>>>>>>>                             CPU architecture. I figured
>>>>>>>>>>>>>>>>                             this might
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> be an
>>>>>>>>>>>>>>>>                             important detail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> On Thu, Aug
>>>>>>>>>>>>>>>>                             17, 2023 at 11:44 AM 
>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>                             Rybchenko
>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Greatings Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>                             happy to hear that you're
>>>>>>>>>>>>>>>>                             trying to bring
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> it up.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> As I
>>>>>>>>>>>>>>>>                             understand the final goal 
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>                             to run it on
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> regular
>>>>>>>>>>>>>>>>                             basis. So, we need to make
>>>>>>>>>>>>>>>>                             it properly
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>                             very beginning.
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Bring up
>>>>>>>>>>>>>>>>                             of all features consists of
>>>>>>>>>>>>>>>>                             4 steps:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>                             Create site-specific
>>>>>>>>>>>>>>>>                             repository (we call it
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> ts-rigs)
>>>>>>>>>>>>>>>>                             which contains information
>>>>>>>>>>>>>>>>                             about test
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> rigs and
>>>>>>>>>>>>>>>>                             other site-specific
>>>>>>>>>>>>>>>>                             information like
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> where to
>>>>>>>>>>>>>>>>                             send mails, where to store
>>>>>>>>>>>>>>>>                             logs etc.
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> It is
>>>>>>>>>>>>>>>>                             required for manual
>>>>>>>>>>>>>>>>                             execution as well,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>                             test rigs description is
>>>>>>>>>>>>>>>>                             essential. I'll
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> return
>>>>>>>>>>>>>>>>                             to the topic below.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 2. Setup
>>>>>>>>>>>>>>>>                             logs storage for automated
>>>>>>>>>>>>>>>>                             runs.
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Basically it is a disk 
>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>>                             plus apache2 web
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>                             with few CGI scripts which
>>>>>>>>>>>>>>>>                             help a lot to
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>                             disk space.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 3. Setup
>>>>>>>>>>>>>>>>                             Bublik web application 
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>                             provides
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> web
>>>>>>>>>>>>>>>>                             interface to view testing
>>>>>>>>>>>>>>>>                             results. Same as
>>>>>>>>>>>>>>>>                            https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 4. Setup
>>>>>>>>>>>>>>>>                             Jenkins to run tests on
>>>>>>>>>>>>>>>>                             regularly,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>                             logs in log storage (2) and
>>>>>>>>>>>>>>>>                             import it to
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> bublik (3).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Last few
>>>>>>>>>>>>>>>>                             month we spent on our
>>>>>>>>>>>>>>>>                             homework to make
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>                             simpler to bring up
>>>>>>>>>>>>>>>>                             automated execution
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>                             Jenkins -
>>>>>>>>>>>>>>>>                            https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Corresponding bits in
>>>>>>>>>>>>>>>>                             dpdk-ethdev-ts will be
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             available tomorrow.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Let's
>>>>>>>>>>>>>>>>                             return to the step (1).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Unfortunately there is no
>>>>>>>>>>>>>>>>                             publicly available
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> example
>>>>>>>>>>>>>>>>                             of the ts-rigs repository 
>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             sensitive site-specific
>>>>>>>>>>>>>>>>                             information is located
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>                             But I'm ready to help you 
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>                             create it
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> for UNH.
>>>>>>>>>>>>>>>>                             I see two options here:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> (A) I'll
>>>>>>>>>>>>>>>>                             ask questions and based on 
>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> answers
>>>>>>>>>>>>>>>>                             will create the first draft
>>>>>>>>>>>>>>>>                             with my
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> (B) I'll
>>>>>>>>>>>>>>>>                             make a template/example
>>>>>>>>>>>>>>>>                             ts-rigs repo,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> publish
>>>>>>>>>>>>>>>>                             it and you'll create UNH
>>>>>>>>>>>>>>>>                             ts-rigs based
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> on it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Of
>>>>>>>>>>>>>>>>                             course, I'll help to debug
>>>>>>>>>>>>>>>>                             and finally bring
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> it up in
>>>>>>>>>>>>>>>>                             any case.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> (A) is a
>>>>>>>>>>>>>>>>                             bit simpler for me and you,
>>>>>>>>>>>>>>>>                             but (B) is
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> a bit
>>>>>>>>>>>>>>>>                             more generic and will help
>>>>>>>>>>>>>>>>                             other
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             potential users to bring it 
>>>>>>>>>>>>>>>> up.
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> We can
>>>>>>>>>>>>>>>>                             combine (A)+(B). I.e. start
>>>>>>>>>>>>>>>>                             from (A).
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> What do
>>>>>>>>>>>>>>>>                             you think?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> On
>>>>>>>>>>>>>>>>                             8/17/23 15:18, Konstantin
>>>>>>>>>>>>>>>>                             Ushakov wrote:
>>>>>>>>>>>>>>>>>                             Greetings Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>                             for contacting us. I copy
>>>>>>>>>>>>>>>>>                             Andrew who
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>                             be happy to help
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> Konstantin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                             On 16 Aug 2023, at 21:50,
>>>>>>>>>>>>>>>>>>                             Adam Hassick
>>>>>>>>>>>>>>>>>>                             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>>>>>>                             in the process of setting
>>>>>>>>>>>>>>>>>>                             up the DPDK
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Poll
>>>>>>>>>>>>>>>>>>                             Mode Driver test suite as
>>>>>>>>>>>>>>>>>>                             an addition to
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>>>                             testing coverage for DPDK
>>>>>>>>>>>>>>>>>>                             at the UNH lab.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>>                             some questions about how
>>>>>>>>>>>>>>>>>>                             to set the
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>                             suite arguments.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>>                             been able to configure 
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>                             Test Engine
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>                             connect to the hosts in
>>>>>>>>>>>>>>>>>>                             the testbed. The
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> RCF,
>>>>>>>>>>>>>>>>>>                             Configurator, and Tester
>>>>>>>>>>>>>>>>>>                             all begin to
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> run,
>>>>>>>>>>>>>>>>>>                             however the prelude of 
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>                             test suite
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>>>                             to run.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                            https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>                             documentation mentions
>>>>>>>>>>>>>>>>>>                             that there are
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             several test parameters
>>>>>>>>>>>>>>>>>>                             for the test suite,
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>                             for the IUT test link 
>>>>>>>>>>>>>>>>>> MAC,
>>>>>>>>>>>>>>>>>>                             etc. These
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>                             like they would need to 
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>                             set somewhere
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> to run
>>>>>>>>>>>>>>>>>>                             many of the tests.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>                             in the Test Engine
>>>>>>>>>>>>>>>>>>                             documentation, there
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>                             instructions on how to
>>>>>>>>>>>>>>>>>>                             create new
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             parameters for test 
>>>>>>>>>>>>>>>>>> suites
>>>>>>>>>>>>>>>>>>                             in the Tester
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             configuration, but there
>>>>>>>>>>>>>>>>>>                             is nothing in the
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>                             guide or in the Tester
>>>>>>>>>>>>>>>>>>                             guide for how to
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>                             the arguments for the
>>>>>>>>>>>>>>>>>>                             parameters when
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             running the test suite
>>>>>>>>>>>>>>>>>>                             that I can find. I'm
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>                             sure if I need to write 
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>                             own Tester
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             config, or if I should be
>>>>>>>>>>>>>>>>>>                             setting these in
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>                             other way.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>                             should these values be 
>>>>>>>>>>>>>>>>>> set?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>                             also not sure what
>>>>>>>>>>>>>>>>>>                             environment
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             variables/arguments are
>>>>>>>>>>>>>>>>>>                             strictly necessary or
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>                             are optional.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Adam
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> *Adam
>>>>>>>>>>>>>>>>>>                             Hassick*
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Senior
>>>>>>>>>>>>>>>>>>                             Developer
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> UNH
>>>>>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>>>>                             (603) 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Senior
>>>>>>>>>>>>>>>>                             Developer
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> UNH
>>>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> +1 (603)
>>>>>>>>>>>>>>>>                             475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         -- 
>>>>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                         +1 
>>>>>>>>>>>>>> (603) 475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     -- 
>>>>>>>>>>>>>>         *Adam Hassick*
>>>>>>>>>>>>>>                                     Senior 
>>>>>>>>>>>>>> Developer
>>>>>>>>>>>>>>                                     UNH 
>>>>>>>>>>>>>> InterOperability Lab
>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                     +1 (603) 
>>>>>>>>>>>>>> 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 -- *Adam Hassick*
>>>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>>>                                 UNH InterOperability 
>>>>>>>>>>>>> Lab
>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                             --
>>>>>>>>>>>>                             *Adam Hassick*
>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             iol.unh.edu 
>>>>>>>>>>>> <http://iol.unh.edu>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         --
>>>>>>>>>>                         *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>>>>                         iol.unh.edu 
>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                     --
>>>>>>>>                     *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --
>>>>>>                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>                 ahassick@iol.unh.edu
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>         --
>>>>         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>     --
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>>
>>>
>>>
>>> -- 
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>

[-- Attachment #2: Type: text/html, Size: 77968 bytes --]

  reply	other threads:[~2023-09-18  7:50 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CAC-YWqiQfH4Rx-Et1jGHhGK9i47d0AArKy-B2P77iYbbM+Lpig@mail.gmail.com>
     [not found] ` <C3B08390-DA6D-4BDC-BBD7-98561F92FE33@oktetlabs.ru>
     [not found]   ` <35340484-1d7e-7e5f-cad4-c965ba541397@oktetlabs.ru>
2023-08-17 17:03     ` Adam Hassick
2023-08-18 18:40       ` Andrew Rybchenko
2023-08-20  8:40         ` Andrew Rybchenko
2023-08-21 14:19           ` Adam Hassick
2023-08-23 14:45             ` Adam Hassick
2023-08-24  8:22               ` Andrew Rybchenko
2023-08-24 14:30                 ` Adam Hassick
2023-08-24 18:34                   ` Andrew Rybchenko
2023-08-24 20:29                     ` Adam Hassick
2023-08-24 20:54                       ` Andrew Rybchenko
2023-08-25 13:57                         ` Andrew Rybchenko
2023-08-25 14:06                           ` Adam Hassick
2023-08-25 14:41                             ` Andrew Rybchenko
2023-08-25 17:35                               ` Andrew Rybchenko
2023-08-28 15:02                                 ` Adam Hassick
2023-08-28 21:05                                   ` Andrew Rybchenko
2023-08-29 12:07                                     ` Andrew Rybchenko
2023-08-29 14:02                                       ` Adam Hassick
2023-08-29 20:43                                         ` Andrew Rybchenko
2023-08-31 19:38                                           ` Adam Hassick
2023-09-01  7:59                                             ` Andrew Rybchenko
2023-09-05 15:01                                               ` Adam Hassick
2023-09-06 11:36                                                 ` Andrew Rybchenko
2023-09-06 15:00                                                   ` Adam Hassick
2023-09-08 14:57                                                     ` Adam Hassick
2023-09-13 15:45                                                       ` Andrew Rybchenko
2023-09-18  6:15                                                         ` Andrew Rybchenko
2023-09-18  6:23                                                           ` Konstantin Ushakov [this message]
2023-09-18  6:26                                                             ` Andrew Rybchenko
2023-09-18 14:44                                                               ` Adam Hassick
2023-09-18 15:04                                                                 ` Andrew Rybchenko
2023-10-04 13:48                                                                   ` Adam Hassick
2023-10-05 10:25                                                                     ` Andrew Rybchenko
2023-10-10 14:09                                                                       ` Adam Hassick
2023-10-11 11:46                                                                         ` Andrew Rybchenko
2023-10-23 11:11                                                                         ` Andrew Rybchenko
2023-10-25 20:27                                                                           ` Adam Hassick
2023-10-26 12:19                                                                             ` Andrew Rybchenko
2023-10-26 17:44                                                                               ` Adam Hassick
2023-10-27  8:01                                                                                 ` Andrew Rybchenko
2023-10-27 19:13                                                                                 ` Andrew Rybchenko
2023-11-06 23:16                                                                                   ` Adam Hassick
2023-11-07 16:57                                                                                     ` Andrew Rybchenko
2023-11-07 20:30                                                                                       ` Adam Hassick
2023-11-08  7:20                                                                                         ` Andrew Rybchenko
2023-11-16 20:03                                                                                           ` Adam Hassick
2023-11-16 20:38                                                                                             ` DPDK Coverity test run Mcnamara, John
2023-11-16 20:43                                                                                               ` Patrick Robb
2023-11-16 20:56                                                                                                 ` Mcnamara, John
2023-11-20 17:18                                                                                             ` Setting up DPDK PMD Test Suite Andrew Rybchenko
2023-12-01 14:39                                                                                               ` Andrew Rybchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AAF8D161-89D0-4192-920D-EB58B61100D9@oktetlabs.ru \
    --to=konstantin.ushakov@oktetlabs.ru \
    --cc=ahassick@iol.unh.edu \
    --cc=andrew.rybchenko@oktetlabs.ru \
    --cc=ci@dpdk.org \
    --cc=probb@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).