Re: Setting up DPDK PMD Test Suite

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 4789 bytes --]

Hi Andrew,

I'm adding the CI mailing list to this conversation. Others in the
community might find this conversation valuable.

We do want to run testing on a regular basis. The Jenkins integration will
be very useful for us, as most of our CI is orchestrated by Jenkins.
I've conferred with the rest of the team, and we think it would be best to
move forward with mainly option B.
If you would like to know anything about our testbeds that would help you
with creating an example ts-rigs repo, I'd be happy to answer any questions
you have.

We have multiple test rigs (we call these "DUT-tester pairs") that we run
our existing hardware testing on, with differing network hardware and CPU
architecture. I figured this might be an important detail.

Thanks,
Adam

On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Greatings Adam,
>
> I'm happy to hear that you're trying to bring it up.
>
> As I understand the final goal is to run it on regular basis. So, we need
> to make it properly from the very beginning.
> Bring up of all features consists of 4 steps:
>
> 1. Create site-specific repository (we call it ts-rigs) which contains
> information about test rigs and other site-specific information like where
> to send mails, where to store logs etc. It is required for manual execution
> as well, since test rigs description is essential. I'll return to the topic
> below.
>
> 2. Setup logs storage for automated runs. Basically it is a disk space
> plus apache2 web server with few CGI scripts which help a lot to save disk
> space.
>
> 3. Setup Bublik web application which provides web interface to view
> testing results. Same as https://ts-factory.io/bublik
>
> 4. Setup Jenkins to run tests on regularly, save logs in log storage (2)
> and import it to bublik (3).
>
> Last few month we spent on our homework to make it simpler to bring up
> automated execution using Jenkins -
> https://github.com/ts-factory/te-jenkins
> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>
> Let's return to the step (1).
>
> Unfortunately there is no publicly available example of the ts-rigs
> repository since sensitive site-specific information is located there. But
> I'm ready to help you to create it for UNH. I see two options here:
>
> (A) I'll ask questions and based on your answers will create the first
> draft with my comments.
>
> (B) I'll make a template/example ts-rigs repo, publish it and you'll
> create UNH ts-rigs based on it.
>
> Of course, I'll help to debug and finally bring it up in any case.
>
> (A) is a bit simpler for me and you, but (B) is a bit more generic and
> will help other potential users to bring it up.
> We can combine (A)+(B). I.e. start from (A). What do you think?
>
> Thanks,
> Andrew.
>
> On 8/17/23 15:18, Konstantin Ushakov wrote:
>
> Greetings Adam,
>
>
> Thanks for contacting us. I copy Andrew who would be happy to help
>
> Thanks,
> Konstantin
>
> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu> wrote:
>
> 
> Greetings Konstantin,
>
> I am in the process of setting up the DPDK Poll Mode Driver test suite as
> an addition to our testing coverage for DPDK at the UNH lab.
>
> I have some questions about how to set the test suite arguments.
>
> I have been able to configure the Test Engine to connect to the hosts in
> the testbed. The RCF, Configurator, and Tester all begin to run, however
> the prelude of the test suite fails to run.
>
> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>
> The documentation mentions that there are several test parameters for the
> test suite, like for the IUT test link MAC, etc. These seem like they would
> need to be set somewhere to run many of the tests.
>
> I see in the Test Engine documentation, there are instructions on how to
> create new parameters for test suites in the Tester configuration, but
> there is nothing in the user guide or in the Tester guide for how to set
> the arguments for the parameters when running the test suite that I can
> find. I'm not sure if I need to write my own Tester config, or if I should
> be setting these in some other way.
>
> How should these values be set?
>
> I'm also not sure what environment variables/arguments are strictly
> necessary or which are optional.
>
> Regards,
> Adam
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 9885 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 5439 bytes --]

Hi Adam,

 > I've conferred with the rest of the team, and we think it would be 
best to move forward with mainly option B.

OK, I'll provide the sample on Monday for you. It is almost ready right 
now, but I need to double-check it before publishing.

Regards,
Andrew.

On 8/17/23 20:03, Adam Hassick wrote:
> Hi Andrew,
>
> I'm adding the CI mailing list to this conversation. Others in the 
> community might find this conversation valuable.
>
> We do want to run testing on a regular basis. The Jenkins integration 
> will be very useful for us, as most of our CI is orchestrated by Jenkins.
> I've conferred with the rest of the team, and we think it would be 
> best to move forward with mainly option B.
> If you would like to know anything about our testbeds that would help 
> you with creating an example ts-rigs repo, I'd be happy to answer any 
> questions you have.
>
> We have multiple test rigs (we call these "DUT-tester pairs") that we 
> run our existing hardware testing on, with differing network hardware 
> and CPU architecture. I figured this might be an important detail.
>
> Thanks,
> Adam
>
> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Greatings Adam,
>
>     I'm happy to hear that you're trying to bring it up.
>
>     As I understand the final goal is to run it on regular basis. So,
>     we need to make it properly from the very beginning.
>     Bring up of all features consists of 4 steps:
>
>     1. Create site-specific repository (we call it ts-rigs) which
>     contains information about test rigs and other site-specific
>     information like where to send mails, where to store logs etc. It
>     is required for manual execution as well, since test rigs
>     description is essential. I'll return to the topic below.
>
>     2. Setup logs storage for automated runs. Basically it is a disk
>     space plus apache2 web server with few CGI scripts which help a
>     lot to save disk space.
>
>     3. Setup Bublik web application which provides web interface to
>     view testing results. Same as https://ts-factory.io/bublik
>
>     4. Setup Jenkins to run tests on regularly, save logs in log
>     storage (2) and import it to bublik (3).
>
>     Last few month we spent on our homework to make it simpler to
>     bring up automated execution using Jenkins -
>     https://github.com/ts-factory/te-jenkins
>     Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>
>     Let's return to the step (1).
>
>     Unfortunately there is no publicly available example of the
>     ts-rigs repository since sensitive site-specific information is
>     located there. But I'm ready to help you to create it for UNH. I
>     see two options here:
>
>     (A) I'll ask questions and based on your answers will create the
>     first draft with my comments.
>
>     (B) I'll make a template/example ts-rigs repo, publish it and
>     you'll create UNH ts-rigs based on it.
>
>     Of course, I'll help to debug and finally bring it up in any case.
>
>     (A) is a bit simpler for me and you, but (B) is a bit more generic
>     and will help other potential users to bring it up.
>     We can combine (A)+(B). I.e. start from (A). What do you think?
>
>     Thanks,
>     Andrew.
>
>     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>     Greetings Adam,
>>
>>
>>     Thanks for contacting us. I copy Andrew who would be happy to help
>>
>>     Thanks,
>>     Konstantin
>>
>>>     On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>
>>>     
>>>     Greetings Konstantin,
>>>
>>>     I am in the process of setting up the DPDK Poll Mode Driver test
>>>     suite as an addition to our testing coverage for DPDK at the UNH
>>>     lab.
>>>
>>>     I have some questions about how to set the test suite arguments.
>>>
>>>     I have been able to configure the Test Engine to connect to the
>>>     hosts in the testbed. The RCF, Configurator, and Tester all
>>>     begin to run, however the prelude of the test suite fails to run.
>>>
>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>
>>>     The documentation mentions that there are several test
>>>     parameters for the test suite, like for the IUT test link MAC,
>>>     etc. These seem like they would need to be set somewhere to run
>>>     many of the tests.
>>>
>>>     I see in the Test Engine documentation, there are instructions
>>>     on how to create new parameters for test suites in the Tester
>>>     configuration, but there is nothing in the user guide or in the
>>>     Tester guide for how to set the arguments for the parameters
>>>     when running the test suite that I can find. I'm not sure if I
>>>     need to write my own Tester config, or if I should be setting
>>>     these in some other way.
>>>
>>>     How should these values be set?
>>>
>>>     I'm also not sure what environment variables/arguments are
>>>     strictly necessary or which are optional.
>>>
>>>     Regards,
>>>     Adam
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 13621 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 6053 bytes --]

Hi Adam,

I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully 
it will help to define your test rigs and successfully run some tests 
manually. Feel free to ask any questions and I'll answer here and try to 
update documentation.

Meanwhile I'll prepare missing bits for steps (2) and (3).
Hopefully everything is in place for step (4), but we need to make steps 
(2) and (3) first.

Andrew.

On 8/18/23 21:40, Andrew Rybchenko wrote:
> Hi Adam,
>
> > I've conferred with the rest of the team, and we think it would be 
> best to move forward with mainly option B.
>
> OK, I'll provide the sample on Monday for you. It is almost ready 
> right now, but I need to double-check it before publishing.
>
> Regards,
> Andrew.
>
> On 8/17/23 20:03, Adam Hassick wrote:
>> Hi Andrew,
>>
>> I'm adding the CI mailing list to this conversation. Others in the 
>> community might find this conversation valuable.
>>
>> We do want to run testing on a regular basis. The Jenkins integration 
>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>> I've conferred with the rest of the team, and we think it would be 
>> best to move forward with mainly option B.
>> If you would like to know anything about our testbeds that would help 
>> you with creating an example ts-rigs repo, I'd be happy to answer any 
>> questions you have.
>>
>> We have multiple test rigs (we call these "DUT-tester pairs") that we 
>> run our existing hardware testing on, with differing network hardware 
>> and CPU architecture. I figured this might be an important detail.
>>
>> Thanks,
>> Adam
>>
>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>     Greatings Adam,
>>
>>     I'm happy to hear that you're trying to bring it up.
>>
>>     As I understand the final goal is to run it on regular basis. So,
>>     we need to make it properly from the very beginning.
>>     Bring up of all features consists of 4 steps:
>>
>>     1. Create site-specific repository (we call it ts-rigs) which
>>     contains information about test rigs and other site-specific
>>     information like where to send mails, where to store logs etc. It
>>     is required for manual execution as well, since test rigs
>>     description is essential. I'll return to the topic below.
>>
>>     2. Setup logs storage for automated runs. Basically it is a disk
>>     space plus apache2 web server with few CGI scripts which help a
>>     lot to save disk space.
>>
>>     3. Setup Bublik web application which provides web interface to
>>     view testing results. Same as https://ts-factory.io/bublik
>>
>>     4. Setup Jenkins to run tests on regularly, save logs in log
>>     storage (2) and import it to bublik (3).
>>
>>     Last few month we spent on our homework to make it simpler to
>>     bring up automated execution using Jenkins -
>>     https://github.com/ts-factory/te-jenkins
>>     Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>
>>     Let's return to the step (1).
>>
>>     Unfortunately there is no publicly available example of the
>>     ts-rigs repository since sensitive site-specific information is
>>     located there. But I'm ready to help you to create it for UNH. I
>>     see two options here:
>>
>>     (A) I'll ask questions and based on your answers will create the
>>     first draft with my comments.
>>
>>     (B) I'll make a template/example ts-rigs repo, publish it and
>>     you'll create UNH ts-rigs based on it.
>>
>>     Of course, I'll help to debug and finally bring it up in any case.
>>
>>     (A) is a bit simpler for me and you, but (B) is a bit more
>>     generic and will help other potential users to bring it up.
>>     We can combine (A)+(B). I.e. start from (A). What do you think?
>>
>>     Thanks,
>>     Andrew.
>>
>>     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>     Greetings Adam,
>>>
>>>
>>>     Thanks for contacting us. I copy Andrew who would be happy to help
>>>
>>>     Thanks,
>>>     Konstantin
>>>
>>>>     On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>>
>>>>     
>>>>     Greetings Konstantin,
>>>>
>>>>     I am in the process of setting up the DPDK Poll Mode Driver
>>>>     test suite as an addition to our testing coverage for DPDK at
>>>>     the UNH lab.
>>>>
>>>>     I have some questions about how to set the test suite arguments.
>>>>
>>>>     I have been able to configure the Test Engine to connect to the
>>>>     hosts in the testbed. The RCF, Configurator, and Tester all
>>>>     begin to run, however the prelude of the test suite fails to run.
>>>>
>>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>
>>>>     The documentation mentions that there are several test
>>>>     parameters for the test suite, like for the IUT test link MAC,
>>>>     etc. These seem like they would need to be set somewhere to run
>>>>     many of the tests.
>>>>
>>>>     I see in the Test Engine documentation, there are instructions
>>>>     on how to create new parameters for test suites in the Tester
>>>>     configuration, but there is nothing in the user guide or in the
>>>>     Tester guide for how to set the arguments for the parameters
>>>>     when running the test suite that I can find. I'm not sure if I
>>>>     need to write my own Tester config, or if I should be setting
>>>>     these in some other way.
>>>>
>>>>     How should these values be set?
>>>>
>>>>     I'm also not sure what environment variables/arguments are
>>>>     strictly necessary or which are optional.
>>>>
>>>>     Regards,
>>>>     Adam
>>>>
>>>>     -- 
>>>>     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu
>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 15205 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 6258 bytes --]

Hi Andrew,

Thanks, I've cloned the example repository and will start setting up a
configuration for our development testbed today. I'll let you know if I run
into any difficulties or have any questions.

 - Adam

On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully it
> will help to define your test rigs and successfully run some tests
> manually. Feel free to ask any questions and I'll answer here and try to
> update documentation.
>
> Meanwhile I'll prepare missing bits for steps (2) and (3).
> Hopefully everything is in place for step (4), but we need to make steps
> (2) and (3) first.
>
> Andrew.
>
> On 8/18/23 21:40, Andrew Rybchenko wrote:
>
> Hi Adam,
>
> > I've conferred with the rest of the team, and we think it would be best
> to move forward with mainly option B.
>
> OK, I'll provide the sample on Monday for you. It is almost ready right
> now, but I need to double-check it before publishing.
>
> Regards,
> Andrew.
>
> On 8/17/23 20:03, Adam Hassick wrote:
>
> Hi Andrew,
>
> I'm adding the CI mailing list to this conversation. Others in the
> community might find this conversation valuable.
>
> We do want to run testing on a regular basis. The Jenkins integration will
> be very useful for us, as most of our CI is orchestrated by Jenkins.
> I've conferred with the rest of the team, and we think it would be best to
> move forward with mainly option B.
> If you would like to know anything about our testbeds that would help you
> with creating an example ts-rigs repo, I'd be happy to answer any questions
> you have.
>
> We have multiple test rigs (we call these "DUT-tester pairs") that we run
> our existing hardware testing on, with differing network hardware and CPU
> architecture. I figured this might be an important detail.
>
> Thanks,
> Adam
>
> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Greatings Adam,
>>
>> I'm happy to hear that you're trying to bring it up.
>>
>> As I understand the final goal is to run it on regular basis. So, we need
>> to make it properly from the very beginning.
>> Bring up of all features consists of 4 steps:
>>
>> 1. Create site-specific repository (we call it ts-rigs) which contains
>> information about test rigs and other site-specific information like where
>> to send mails, where to store logs etc. It is required for manual execution
>> as well, since test rigs description is essential. I'll return to the topic
>> below.
>>
>> 2. Setup logs storage for automated runs. Basically it is a disk space
>> plus apache2 web server with few CGI scripts which help a lot to save disk
>> space.
>>
>> 3. Setup Bublik web application which provides web interface to view
>> testing results. Same as https://ts-factory.io/bublik
>>
>> 4. Setup Jenkins to run tests on regularly, save logs in log storage (2)
>> and import it to bublik (3).
>>
>> Last few month we spent on our homework to make it simpler to bring up
>> automated execution using Jenkins -
>> https://github.com/ts-factory/te-jenkins
>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>
>> Let's return to the step (1).
>>
>> Unfortunately there is no publicly available example of the ts-rigs
>> repository since sensitive site-specific information is located there. But
>> I'm ready to help you to create it for UNH. I see two options here:
>>
>> (A) I'll ask questions and based on your answers will create the first
>> draft with my comments.
>>
>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>> create UNH ts-rigs based on it.
>>
>> Of course, I'll help to debug and finally bring it up in any case.
>>
>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>> will help other potential users to bring it up.
>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>
>> Thanks,
>> Andrew.
>>
>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>
>> Greetings Adam,
>>
>>
>> Thanks for contacting us. I copy Andrew who would be happy to help
>>
>> Thanks,
>> Konstantin
>>
>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu> wrote:
>>
>> 
>> Greetings Konstantin,
>>
>> I am in the process of setting up the DPDK Poll Mode Driver test suite as
>> an addition to our testing coverage for DPDK at the UNH lab.
>>
>> I have some questions about how to set the test suite arguments.
>>
>> I have been able to configure the Test Engine to connect to the hosts in
>> the testbed. The RCF, Configurator, and Tester all begin to run, however
>> the prelude of the test suite fails to run.
>>
>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>
>> The documentation mentions that there are several test parameters for the
>> test suite, like for the IUT test link MAC, etc. These seem like they would
>> need to be set somewhere to run many of the tests.
>>
>> I see in the Test Engine documentation, there are instructions on how to
>> create new parameters for test suites in the Tester configuration, but
>> there is nothing in the user guide or in the Tester guide for how to set
>> the arguments for the parameters when running the test suite that I can
>> find. I'm not sure if I need to write my own Tester config, or if I should
>> be setting these in some other way.
>>
>> How should these values be set?
>>
>> I'm also not sure what environment variables/arguments are strictly
>> necessary or which are optional.
>>
>> Regards,
>> Adam
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 15492 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 7543 bytes --]

Hi Andrew,

I've set up a test rig repository here, and have created configurations for
our development testbed based off of the examples.
We've been able to get the test suite to run manually on Mellanox CX5
devices once.
However, we are running into an issue where, when RCF starts, the RCF
connection times out very frequently. We aren't sure why this is the case.
It works sometimes, but most of the time when we try to run the test
engine, it encounters this issue.
I've tried changing the RCF port by setting "TE_RCF_PORT=<some port
number>" and rebooting the testbed machines. Neither seems to fix the issue.

It also seems like the timeout takes far longer than 60 seconds, even when
running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the test suite.
I assume the unit for this variable is seconds?

Thanks,
Adam

On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu> wrote:

> Hi Andrew,
>
> Thanks, I've cloned the example repository and will start setting up a
> configuration for our development testbed today. I'll let you know if I run
> into any difficulties or have any questions.
>
>  - Adam
>
> On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully
>> it will help to define your test rigs and successfully run some tests
>> manually. Feel free to ask any questions and I'll answer here and try to
>> update documentation.
>>
>> Meanwhile I'll prepare missing bits for steps (2) and (3).
>> Hopefully everything is in place for step (4), but we need to make steps
>> (2) and (3) first.
>>
>> Andrew.
>>
>> On 8/18/23 21:40, Andrew Rybchenko wrote:
>>
>> Hi Adam,
>>
>> > I've conferred with the rest of the team, and we think it would be best
>> to move forward with mainly option B.
>>
>> OK, I'll provide the sample on Monday for you. It is almost ready right
>> now, but I need to double-check it before publishing.
>>
>> Regards,
>> Andrew.
>>
>> On 8/17/23 20:03, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> I'm adding the CI mailing list to this conversation. Others in the
>> community might find this conversation valuable.
>>
>> We do want to run testing on a regular basis. The Jenkins integration
>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>> I've conferred with the rest of the team, and we think it would be best
>> to move forward with mainly option B.
>> If you would like to know anything about our testbeds that would help you
>> with creating an example ts-rigs repo, I'd be happy to answer any questions
>> you have.
>>
>> We have multiple test rigs (we call these "DUT-tester pairs") that we run
>> our existing hardware testing on, with differing network hardware and CPU
>> architecture. I figured this might be an important detail.
>>
>> Thanks,
>> Adam
>>
>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Greatings Adam,
>>>
>>> I'm happy to hear that you're trying to bring it up.
>>>
>>> As I understand the final goal is to run it on regular basis. So, we
>>> need to make it properly from the very beginning.
>>> Bring up of all features consists of 4 steps:
>>>
>>> 1. Create site-specific repository (we call it ts-rigs) which contains
>>> information about test rigs and other site-specific information like where
>>> to send mails, where to store logs etc. It is required for manual execution
>>> as well, since test rigs description is essential. I'll return to the topic
>>> below.
>>>
>>> 2. Setup logs storage for automated runs. Basically it is a disk space
>>> plus apache2 web server with few CGI scripts which help a lot to save disk
>>> space.
>>>
>>> 3. Setup Bublik web application which provides web interface to view
>>> testing results. Same as https://ts-factory.io/bublik
>>>
>>> 4. Setup Jenkins to run tests on regularly, save logs in log storage (2)
>>> and import it to bublik (3).
>>>
>>> Last few month we spent on our homework to make it simpler to bring up
>>> automated execution using Jenkins -
>>> https://github.com/ts-factory/te-jenkins
>>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>>
>>> Let's return to the step (1).
>>>
>>> Unfortunately there is no publicly available example of the ts-rigs
>>> repository since sensitive site-specific information is located there. But
>>> I'm ready to help you to create it for UNH. I see two options here:
>>>
>>> (A) I'll ask questions and based on your answers will create the first
>>> draft with my comments.
>>>
>>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>>> create UNH ts-rigs based on it.
>>>
>>> Of course, I'll help to debug and finally bring it up in any case.
>>>
>>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>>> will help other potential users to bring it up.
>>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>>
>>> Thanks,
>>> Andrew.
>>>
>>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>
>>> Greetings Adam,
>>>
>>>
>>> Thanks for contacting us. I copy Andrew who would be happy to help
>>>
>>> Thanks,
>>> Konstantin
>>>
>>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu> wrote:
>>>
>>> 
>>> Greetings Konstantin,
>>>
>>> I am in the process of setting up the DPDK Poll Mode Driver test suite
>>> as an addition to our testing coverage for DPDK at the UNH lab.
>>>
>>> I have some questions about how to set the test suite arguments.
>>>
>>> I have been able to configure the Test Engine to connect to the hosts in
>>> the testbed. The RCF, Configurator, and Tester all begin to run, however
>>> the prelude of the test suite fails to run.
>>>
>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>
>>> The documentation mentions that there are several test parameters for
>>> the test suite, like for the IUT test link MAC, etc. These seem like they
>>> would need to be set somewhere to run many of the tests.
>>>
>>> I see in the Test Engine documentation, there are instructions on how to
>>> create new parameters for test suites in the Tester configuration, but
>>> there is nothing in the user guide or in the Tester guide for how to set
>>> the arguments for the parameters when running the test suite that I can
>>> find. I'm not sure if I need to write my own Tester config, or if I should
>>> be setting these in some other way.
>>>
>>> How should these values be set?
>>>
>>> I'm also not sure what environment variables/arguments are strictly
>>> necessary or which are optional.
>>>
>>> Regards,
>>> Adam
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>


-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 17766 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 10146 bytes --]

Hi Adam,

Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it goes 
to 'copy_timeout' in ts-conf/rcf.conf.
Description in in doc/sphinx/pages/group_te_engine_rcf.rst says that 
copy_timeout is in seconds and implementation in lib/rcfunix/rcfunix.c 
passes the value to select() tv_sec. Theoretically select() could be 
interrupted by signal, but I think it is unlikely here.

I'm not sure that I understand what do you mean by RCF connection 
timeout. Does it happen on TE startup when RCF starts test agents. If 
so, TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests are in 
progress, e.g. in the middle of a test. If so, TE_RCFUNIX_TIMEOUT is 
unrelated and most likely either host with test agent dies or test agent 
itself crashes. It would be easier for me if classify it if you share 
text log (log.txt, full or just corresponding fragment with some 
context). Also content of ta.DPDK or ta.Peer file depending on which 
agent has problems could shed some light. Corresponding files contain 
stdout/stderr of test agents.

Andrew.

On 8/23/23 17:45, Adam Hassick wrote:
> Hi Andrew,
>
> I've set up a test rig repository here, and have created 
> configurations for our development testbed based off of the examples.
> We've been able to get the test suite to run manually on Mellanox CX5 
> devices once.
> However, we are running into an issue where, when RCF starts, the RCF 
> connection times out very frequently. We aren't sure why this is the case.
> It works sometimes, but most of the time when we try to run the test 
> engine, it encounters this issue.
> I've tried changing the RCF port by setting "TE_RCF_PORT=<some port 
> number>" and rebooting the testbed machines. Neither seems to fix the 
> issue.
>
> It also seems like the timeout takes far longer than 60 seconds, even 
> when running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the 
> test suite.
> I assume the unit for this variable is seconds?
>
> Thanks,
> Adam
>
> On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu> 
> wrote:
>
>     Hi Andrew,
>
>     Thanks, I've cloned the example repository and will start setting
>     up a configuration for our development testbed today. I'll let you
>     know if I run into any difficulties or have any questions.
>
>      - Adam
>
>     On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>     <andrew.rybchenko@oktetlabs.ru> wrote:
>
>         Hi Adam,
>
>         I've published https://github.com/ts-factory/ts-rigs-sample.
>         Hopefully it will help to define your test rigs and
>         successfully run some tests manually. Feel free to ask any
>         questions and I'll answer here and try to update documentation.
>
>         Meanwhile I'll prepare missing bits for steps (2) and (3).
>         Hopefully everything is in place for step (4), but we need to
>         make steps (2) and (3) first.
>
>         Andrew.
>
>         On 8/18/23 21:40, Andrew Rybchenko wrote:
>>         Hi Adam,
>>
>>         > I've conferred with the rest of the team, and we think it
>>         would be best to move forward with mainly option B.
>>
>>         OK, I'll provide the sample on Monday for you. It is almost
>>         ready right now, but I need to double-check it before publishing.
>>
>>         Regards,
>>         Andrew.
>>
>>         On 8/17/23 20:03, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         I'm adding the CI mailing list to this conversation. Others
>>>         in the community might find this conversation valuable.
>>>
>>>         We do want to run testing on a regular basis. The Jenkins
>>>         integration will be very useful for us, as most of our CI is
>>>         orchestrated by Jenkins.
>>>         I've conferred with the rest of the team, and we think it
>>>         would be best to move forward with mainly option B.
>>>         If you would like to know anything about our testbeds that
>>>         would help you with creating an example ts-rigs repo, I'd be
>>>         happy to answer any questions you have.
>>>
>>>         We have multiple test rigs (we call these "DUT-tester
>>>         pairs") that we run our existing hardware testing on, with
>>>         differing network hardware and CPU architecture. I figured
>>>         this might be an important detail.
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             Greatings Adam,
>>>
>>>             I'm happy to hear that you're trying to bring it up.
>>>
>>>             As I understand the final goal is to run it on regular
>>>             basis. So, we need to make it properly from the very
>>>             beginning.
>>>             Bring up of all features consists of 4 steps:
>>>
>>>             1. Create site-specific repository (we call it ts-rigs)
>>>             which contains information about test rigs and other
>>>             site-specific information like where to send mails,
>>>             where to store logs etc. It is required for manual
>>>             execution as well, since test rigs description is
>>>             essential. I'll return to the topic below.
>>>
>>>             2. Setup logs storage for automated runs. Basically it
>>>             is a disk space plus apache2 web server with few CGI
>>>             scripts which help a lot to save disk space.
>>>
>>>             3. Setup Bublik web application which provides web
>>>             interface to view testing results. Same as
>>>             https://ts-factory.io/bublik
>>>
>>>             4. Setup Jenkins to run tests on regularly, save logs in
>>>             log storage (2) and import it to bublik (3).
>>>
>>>             Last few month we spent on our homework to make it
>>>             simpler to bring up automated execution using Jenkins -
>>>             https://github.com/ts-factory/te-jenkins
>>>             Corresponding bits in dpdk-ethdev-ts will be available
>>>             tomorrow.
>>>
>>>             Let's return to the step (1).
>>>
>>>             Unfortunately there is no publicly available example of
>>>             the ts-rigs repository since sensitive site-specific
>>>             information is located there. But I'm ready to help you
>>>             to create it for UNH. I see two options here:
>>>
>>>             (A) I'll ask questions and based on your answers will
>>>             create the first draft with my comments.
>>>
>>>             (B) I'll make a template/example ts-rigs repo, publish
>>>             it and you'll create UNH ts-rigs based on it.
>>>
>>>             Of course, I'll help to debug and finally bring it up in
>>>             any case.
>>>
>>>             (A) is a bit simpler for me and you, but (B) is a bit
>>>             more generic and will help other potential users to
>>>             bring it up.
>>>             We can combine (A)+(B). I.e. start from (A). What do you
>>>             think?
>>>
>>>             Thanks,
>>>             Andrew.
>>>
>>>             On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>             Greetings Adam,
>>>>
>>>>
>>>>             Thanks for contacting us. I copy Andrew who would be
>>>>             happy to help
>>>>
>>>>             Thanks,
>>>>             Konstantin
>>>>
>>>>>             On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>             <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>             wrote:
>>>>>
>>>>>             
>>>>>             Greetings Konstantin,
>>>>>
>>>>>             I am in the process of setting up the DPDK Poll Mode
>>>>>             Driver test suite as an addition to our testing
>>>>>             coverage for DPDK at the UNH lab.
>>>>>
>>>>>             I have some questions about how to set the test suite
>>>>>             arguments.
>>>>>
>>>>>             I have been able to configure the Test Engine to
>>>>>             connect to the hosts in the testbed. The RCF,
>>>>>             Configurator, and Tester all begin to run, however the
>>>>>             prelude of the test suite fails to run.
>>>>>
>>>>>             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>
>>>>>             The documentation mentions that there are several test
>>>>>             parameters for the test suite, like for the IUT test
>>>>>             link MAC, etc. These seem like they would need to be
>>>>>             set somewhere to run many of the tests.
>>>>>
>>>>>             I see in the Test Engine documentation, there are
>>>>>             instructions on how to create new parameters for test
>>>>>             suites in the Tester configuration, but there is
>>>>>             nothing in the user guide or in the Tester guide for
>>>>>             how to set the arguments for the parameters when
>>>>>             running the test suite that I can find. I'm not sure
>>>>>             if I need to write my own Tester config, or if I
>>>>>             should be setting these in some other way.
>>>>>
>>>>>             How should these values be set?
>>>>>
>>>>>             I'm also not sure what environment variables/arguments
>>>>>             are strictly necessary or which are optional.
>>>>>
>>>>>             Regards,
>>>>>             Adam
>>>>>
>>>>>             -- 
>>>>>             *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>             ahassick@iol.unh.edu
>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>
>
>
>     -- 
>     *Adam Hassick*
>     Senior Developer
>     UNH InterOperability Lab
>     ahassick@iol.unh.edu
>     iol.unh.edu <https://www.iol.unh.edu/>
>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 28191 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1.1: Type: text/plain, Size: 9906 bytes --]

Hi Andrew,

This is the output that I see in the terminal when this failure occurs,
after the test agent binaries build and the test engine starts:

Platform default build - pass
Simple RCF consistency check succeeded
--->>> Starting Logger...done
--->>> Starting RCF...rcf_net_engine_connect(): Connection timed out
iol-dts-tester.dpdklab.iol.unh.edu:23571

Then, it hangs here until I kill the "te_rcf" and "te_tee" processes. I let
it hang for around 9 minutes.

On the tester host (which appears to be the Peer agent), there are four
processes that I see running, which look like the test agent processes.

ta.Peer is an empty file. I've attached the log.txt from this run.

 - Adam

On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it goes to
> 'copy_timeout' in ts-conf/rcf.conf.
> Description in in doc/sphinx/pages/group_te_engine_rcf.rst says that
> copy_timeout is in seconds and implementation in lib/rcfunix/rcfunix.c
> passes the value to select() tv_sec. Theoretically select() could be
> interrupted by signal, but I think it is unlikely here.
>
> I'm not sure that I understand what do you mean by RCF connection timeout.
> Does it happen on TE startup when RCF starts test agents. If so,
> TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests are in
> progress, e.g. in the middle of a test. If so, TE_RCFUNIX_TIMEOUT is
> unrelated and most likely either host with test agent dies or test agent
> itself crashes. It would be easier for me if classify it if you share text
> log (log.txt, full or just corresponding fragment with some context). Also
> content of ta.DPDK or ta.Peer file depending on which agent has problems
> could shed some light. Corresponding files contain stdout/stderr of test
> agents.
>
> Andrew.
>
> On 8/23/23 17:45, Adam Hassick wrote:
>
> Hi Andrew,
>
> I've set up a test rig repository here, and have created configurations
> for our development testbed based off of the examples.
> We've been able to get the test suite to run manually on Mellanox CX5
> devices once.
> However, we are running into an issue where, when RCF starts, the RCF
> connection times out very frequently. We aren't sure why this is the case.
> It works sometimes, but most of the time when we try to run the test
> engine, it encounters this issue.
> I've tried changing the RCF port by setting "TE_RCF_PORT=<some port
> number>" and rebooting the testbed machines. Neither seems to fix the issue.
>
> It also seems like the timeout takes far longer than 60 seconds, even when
> running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the test suite.
> I assume the unit for this variable is seconds?
>
> Thanks,
> Adam
>
> On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu>
> wrote:
>
>> Hi Andrew,
>>
>> Thanks, I've cloned the example repository and will start setting up a
>> configuration for our development testbed today. I'll let you know if I run
>> into any difficulties or have any questions.
>>
>>  - Adam
>>
>> On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully
>>> it will help to define your test rigs and successfully run some tests
>>> manually. Feel free to ask any questions and I'll answer here and try to
>>> update documentation.
>>>
>>> Meanwhile I'll prepare missing bits for steps (2) and (3).
>>> Hopefully everything is in place for step (4), but we need to make steps
>>> (2) and (3) first.
>>>
>>> Andrew.
>>>
>>> On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>
>>> Hi Adam,
>>>
>>> > I've conferred with the rest of the team, and we think it would be
>>> best to move forward with mainly option B.
>>>
>>> OK, I'll provide the sample on Monday for you. It is almost ready right
>>> now, but I need to double-check it before publishing.
>>>
>>> Regards,
>>> Andrew.
>>>
>>> On 8/17/23 20:03, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> I'm adding the CI mailing list to this conversation. Others in the
>>> community might find this conversation valuable.
>>>
>>> We do want to run testing on a regular basis. The Jenkins integration
>>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>>> I've conferred with the rest of the team, and we think it would be best
>>> to move forward with mainly option B.
>>> If you would like to know anything about our testbeds that would help
>>> you with creating an example ts-rigs repo, I'd be happy to answer any
>>> questions you have.
>>>
>>> We have multiple test rigs (we call these "DUT-tester pairs") that we
>>> run our existing hardware testing on, with differing network hardware and
>>> CPU architecture. I figured this might be an important detail.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Greatings Adam,
>>>>
>>>> I'm happy to hear that you're trying to bring it up.
>>>>
>>>> As I understand the final goal is to run it on regular basis. So, we
>>>> need to make it properly from the very beginning.
>>>> Bring up of all features consists of 4 steps:
>>>>
>>>> 1. Create site-specific repository (we call it ts-rigs) which contains
>>>> information about test rigs and other site-specific information like where
>>>> to send mails, where to store logs etc. It is required for manual execution
>>>> as well, since test rigs description is essential. I'll return to the topic
>>>> below.
>>>>
>>>> 2. Setup logs storage for automated runs. Basically it is a disk space
>>>> plus apache2 web server with few CGI scripts which help a lot to save disk
>>>> space.
>>>>
>>>> 3. Setup Bublik web application which provides web interface to view
>>>> testing results. Same as https://ts-factory.io/bublik
>>>>
>>>> 4. Setup Jenkins to run tests on regularly, save logs in log storage
>>>> (2) and import it to bublik (3).
>>>>
>>>> Last few month we spent on our homework to make it simpler to bring up
>>>> automated execution using Jenkins -
>>>> https://github.com/ts-factory/te-jenkins
>>>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>>>
>>>> Let's return to the step (1).
>>>>
>>>> Unfortunately there is no publicly available example of the ts-rigs
>>>> repository since sensitive site-specific information is located there. But
>>>> I'm ready to help you to create it for UNH. I see two options here:
>>>>
>>>> (A) I'll ask questions and based on your answers will create the first
>>>> draft with my comments.
>>>>
>>>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>>>> create UNH ts-rigs based on it.
>>>>
>>>> Of course, I'll help to debug and finally bring it up in any case.
>>>>
>>>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>>>> will help other potential users to bring it up.
>>>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>>>
>>>> Thanks,
>>>> Andrew.
>>>>
>>>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>
>>>> Greetings Adam,
>>>>
>>>>
>>>> Thanks for contacting us. I copy Andrew who would be happy to help
>>>>
>>>> Thanks,
>>>> Konstantin
>>>>
>>>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu> wrote:
>>>>
>>>> 
>>>> Greetings Konstantin,
>>>>
>>>> I am in the process of setting up the DPDK Poll Mode Driver test suite
>>>> as an addition to our testing coverage for DPDK at the UNH lab.
>>>>
>>>> I have some questions about how to set the test suite arguments.
>>>>
>>>> I have been able to configure the Test Engine to connect to the hosts
>>>> in the testbed. The RCF, Configurator, and Tester all begin to run, however
>>>> the prelude of the test suite fails to run.
>>>>
>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>
>>>> The documentation mentions that there are several test parameters for
>>>> the test suite, like for the IUT test link MAC, etc. These seem like they
>>>> would need to be set somewhere to run many of the tests.
>>>>
>>>> I see in the Test Engine documentation, there are instructions on how
>>>> to create new parameters for test suites in the Tester configuration, but
>>>> there is nothing in the user guide or in the Tester guide for how to set
>>>> the arguments for the parameters when running the test suite that I can
>>>> find. I'm not sure if I need to write my own Tester config, or if I should
>>>> be setting these in some other way.
>>>>
>>>> How should these values be set?
>>>>
>>>> I'm also not sure what environment variables/arguments are strictly
>>>> necessary or which are optional.
>>>>
>>>> Regards,
>>>> Adam
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #1.2: Type: text/html, Size: 27971 bytes --]

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 3752 bytes --]

Log report
~~~~~~~~~~

RING  Dispatcher  Command-line options  13:55:53.387
--conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --opts=run/iol-dts-mcx5 --opts=opts.ts

RING  Dispatcher  Expanded command-line options  13:55:53.402
 --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --script=env/iol-dts --script=env/mlx-cx5 --script=scripts/iut.h1 --script=scripts/iut.h1-mcx5 --conf-cs=cs/dpdk-pmd-ts.yml --script=scripts/ta-def --script=scripts/defaults --tester-script=scripts/dpdk-trc-tags --tester-script=scripts/os-trc-tags --script=scripts/net-modules --script=scripts/iut-net-driver-loaded --script=scripts/disable_unused_agts

RING  Dispatcher  Start  14:01:03.417
Starting TEN applications

RING  Dispatcher  Start  14:01:03.435
Start Logger:  /opt/tsf/ts-rigs/logger.conf

RING  Logger  Cfg file  14:01:03.461
Opening config file: /opt/tsf/ts-rigs/logger.conf

RING  Logger  Log streaming  14:01:03.462
Current listeners configuration:
Listeners:
Filters:

RING  Dispatcher  Start  14:01:03.481
Start RCF:  /opt/tsf/ts-conf/rcf.conf

RING  RCF  RCF Unix  14:01:03.487
Starting TA 'Peer' type 'linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell='

RING  RCF  RCF Unix  14:01:03.487
CMD to copy: ssh -qxTn -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_76872_1692885663_1" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_76872_1692885663_1 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  14:01:05.063
Command to detect shell name: ssh -qxTn -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  14:01:05.321
Shell is: /bin/bash

RING  RCF  RCF Unix  14:01:05.321
Command to start TA: ssh -qxTn -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_76872_1692885663_1 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=" 2>&1 | te_tee RCF Peer 10 >ta.Peer 

WARN  RCF  RCF Unix  14:11:16.984
Connecting to TA Peer iol-dts-tester.dpdklab.iol.unh.edu:23571 failed (COMM-ETIMEDOUT) - connect again after delay

RING  RCF  RCF Unix  14:11:16.984
Sleeping 1 seconds

RING  Dispatcher  Start  14:20:25.622
Shutdown Logger

RING  Logger  Self  14:20:25.624
Logger shutdown ...

WARN  Logger  Self  14:20:25.624
Logger is shut down without polling of TAs

WARN  Logger  Log streaming  14:20:25.624
Not all messages in listener queue have been processed

RING  Logger  Self  14:20:25.624
Shutdown is completed

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 13978 bytes --]

Hi Adam,

 > On the tester host (which appears to be the Peer agent), there are 
four processes that I see running, which look like the test agent processes.

Before the next try I'd recommend to kill these processes.

Is there any firewall in the network or on test hosts which could block 
incoming TCP connection to the port 23571 
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where 
you run test engine?

If behaviour the same on the next try and you see that test agent is 
kept running, could you check using

# netstat -tnlp

that Test Agent is listening on the port and try to establish TCP 
connection from test agent using

$ telnet iol-dts-tester.dpdklab.iol.unh.edu 
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571 
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571>

and check if TCP connection could be established.

Another idea is to login Tester under root as testing does, get start TA 
command from the log and try it by hands without -n and remove extra 
escaping.

# sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1 
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=

Hopefully in this case test agent directory remains in the /tmp and you 
don't need to copy it as testing does.
May be output could shed some light on what's going on.

Andrew.

On 8/24/23 17:30, Adam Hassick wrote:
> Hi Andrew,
>
> This is the output that I see in the terminal when this failure 
> occurs, after the test agent binaries build and the test engine starts:
>
> Platform default build - pass
> Simple RCF consistency check succeeded
> --->>> Starting Logger...done
> --->>> Starting RCF...rcf_net_engine_connect(): Connection timed out 
> iol-dts-tester.dpdklab.iol.unh.edu:23571 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
> Then, it hangs here until I kill the "te_rcf" and "te_tee" processes. 
> I let it hang for around 9 minutes.
>
> On the tester host (which appears to be the Peer agent), there are 
> four processes that I see running, which look like the test agent 
> processes.
>
> ta.Peer is an empty file. I've attached the log.txt from this run.
>
>  - Adam
>
> On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it
>     goes to 'copy_timeout' in ts-conf/rcf.conf.
>     Description in in doc/sphinx/pages/group_te_engine_rcf.rst says
>     that copy_timeout is in seconds and implementation in
>     lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>     Theoretically select() could be interrupted by signal, but I think
>     it is unlikely here.
>
>     I'm not sure that I understand what do you mean by RCF connection
>     timeout. Does it happen on TE startup when RCF starts test agents.
>     If so, TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests
>     are in progress, e.g. in the middle of a test. If so,
>     TE_RCFUNIX_TIMEOUT is unrelated and most likely either host with
>     test agent dies or test agent itself crashes. It would be easier
>     for me if classify it if you share text log (log.txt, full or just
>     corresponding fragment with some context). Also content of ta.DPDK
>     or ta.Peer file depending on which agent has problems could shed
>     some light. Corresponding files contain stdout/stderr of test agents.
>
>     Andrew.
>
>     On 8/23/23 17:45, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     I've set up a test rig repository here, and have created
>>     configurations for our development testbed based off of the examples.
>>     We've been able to get the test suite to run manually on Mellanox
>>     CX5 devices once.
>>     However, we are running into an issue where, when RCF starts, the
>>     RCF connection times out very frequently. We aren't sure why this
>>     is the case.
>>     It works sometimes, but most of the time when we try to run the
>>     test engine, it encounters this issue.
>>     I've tried changing the RCF port by setting "TE_RCF_PORT=<some
>>     port number>" and rebooting the testbed machines. Neither seems
>>     to fix the issue.
>>
>>     It also seems like the timeout takes far longer than 60 seconds,
>>     even when running "export TE_RCFUNIX_TIMEOUT=60" before I try to
>>     run the test suite.
>>     I assume the unit for this variable is seconds?
>>
>>     Thanks,
>>     Adam
>>
>>     On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>     <ahassick@iol.unh.edu> wrote:
>>
>>         Hi Andrew,
>>
>>         Thanks, I've cloned the example repository and will start
>>         setting up a configuration for our development testbed today.
>>         I'll let you know if I run into any difficulties or have any
>>         questions.
>>
>>          - Adam
>>
>>         On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>             Hi Adam,
>>
>>             I've published
>>             https://github.com/ts-factory/ts-rigs-sample. Hopefully
>>             it will help to define your test rigs and successfully
>>             run some tests manually. Feel free to ask any questions
>>             and I'll answer here and try to update documentation.
>>
>>             Meanwhile I'll prepare missing bits for steps (2) and (3).
>>             Hopefully everything is in place for step (4), but we
>>             need to make steps (2) and (3) first.
>>
>>             Andrew.
>>
>>             On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>             Hi Adam,
>>>
>>>             > I've conferred with the rest of the team, and we think
>>>             it would be best to move forward with mainly option B.
>>>
>>>             OK, I'll provide the sample on Monday for you. It is
>>>             almost ready right now, but I need to double-check it
>>>             before publishing.
>>>
>>>             Regards,
>>>             Andrew.
>>>
>>>             On 8/17/23 20:03, Adam Hassick wrote:
>>>>             Hi Andrew,
>>>>
>>>>             I'm adding the CI mailing list to this conversation.
>>>>             Others in the community might find this conversation
>>>>             valuable.
>>>>
>>>>             We do want to run testing on a regular basis. The
>>>>             Jenkins integration will be very useful for us, as most
>>>>             of our CI is orchestrated by Jenkins.
>>>>             I've conferred with the rest of the team, and we think
>>>>             it would be best to move forward with mainly option B.
>>>>             If you would like to know anything about our testbeds
>>>>             that would help you with creating an example ts-rigs
>>>>             repo, I'd be happy to answer any questions you have.
>>>>
>>>>             We have multiple test rigs (we call these "DUT-tester
>>>>             pairs") that we run our existing hardware testing on,
>>>>             with differing network hardware and CPU architecture. I
>>>>             figured this might be an important detail.
>>>>
>>>>             Thanks,
>>>>             Adam
>>>>
>>>>             On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>                 Greatings Adam,
>>>>
>>>>                 I'm happy to hear that you're trying to bring it up.
>>>>
>>>>                 As I understand the final goal is to run it on
>>>>                 regular basis. So, we need to make it properly from
>>>>                 the very beginning.
>>>>                 Bring up of all features consists of 4 steps:
>>>>
>>>>                 1. Create site-specific repository (we call it
>>>>                 ts-rigs) which contains information about test rigs
>>>>                 and other site-specific information like where to
>>>>                 send mails, where to store logs etc. It is required
>>>>                 for manual execution as well, since test rigs
>>>>                 description is essential. I'll return to the topic
>>>>                 below.
>>>>
>>>>                 2. Setup logs storage for automated runs. Basically
>>>>                 it is a disk space plus apache2 web server with few
>>>>                 CGI scripts which help a lot to save disk space.
>>>>
>>>>                 3. Setup Bublik web application which provides web
>>>>                 interface to view testing results. Same as
>>>>                 https://ts-factory.io/bublik
>>>>
>>>>                 4. Setup Jenkins to run tests on regularly, save
>>>>                 logs in log storage (2) and import it to bublik (3).
>>>>
>>>>                 Last few month we spent on our homework to make it
>>>>                 simpler to bring up automated execution using
>>>>                 Jenkins - https://github.com/ts-factory/te-jenkins
>>>>                 Corresponding bits in dpdk-ethdev-ts will be
>>>>                 available tomorrow.
>>>>
>>>>                 Let's return to the step (1).
>>>>
>>>>                 Unfortunately there is no publicly available
>>>>                 example of the ts-rigs repository since sensitive
>>>>                 site-specific information is located there. But I'm
>>>>                 ready to help you to create it for UNH. I see two
>>>>                 options here:
>>>>
>>>>                 (A) I'll ask questions and based on your answers
>>>>                 will create the first draft with my comments.
>>>>
>>>>                 (B) I'll make a template/example ts-rigs repo,
>>>>                 publish it and you'll create UNH ts-rigs based on it.
>>>>
>>>>                 Of course, I'll help to debug and finally bring it
>>>>                 up in any case.
>>>>
>>>>                 (A) is a bit simpler for me and you, but (B) is a
>>>>                 bit more generic and will help other potential
>>>>                 users to bring it up.
>>>>                 We can combine (A)+(B). I.e. start from (A). What
>>>>                 do you think?
>>>>
>>>>                 Thanks,
>>>>                 Andrew.
>>>>
>>>>                 On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>                 Greetings Adam,
>>>>>
>>>>>
>>>>>                 Thanks for contacting us. I copy Andrew who would
>>>>>                 be happy to help
>>>>>
>>>>>                 Thanks,
>>>>>                 Konstantin
>>>>>
>>>>>>                 On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>                 <ahassick@iol.unh.edu>
>>>>>>                 <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>
>>>>>>                 
>>>>>>                 Greetings Konstantin,
>>>>>>
>>>>>>                 I am in the process of setting up the DPDK Poll
>>>>>>                 Mode Driver test suite as an addition to our
>>>>>>                 testing coverage for DPDK at the UNH lab.
>>>>>>
>>>>>>                 I have some questions about how to set the test
>>>>>>                 suite arguments.
>>>>>>
>>>>>>                 I have been able to configure the Test Engine to
>>>>>>                 connect to the hosts in the testbed. The RCF,
>>>>>>                 Configurator, and Tester all begin to run,
>>>>>>                 however the prelude of the test suite fails to run.
>>>>>>
>>>>>>                 https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>
>>>>>>                 The documentation mentions that there are several
>>>>>>                 test parameters for the test suite, like for the
>>>>>>                 IUT test link MAC, etc. These seem like they
>>>>>>                 would need to be set somewhere to run many of the
>>>>>>                 tests.
>>>>>>
>>>>>>                 I see in the Test Engine documentation, there are
>>>>>>                 instructions on how to create new parameters for
>>>>>>                 test suites in the Tester configuration, but
>>>>>>                 there is nothing in the user guide or in the
>>>>>>                 Tester guide for how to set the arguments for the
>>>>>>                 parameters when running the test suite that I can
>>>>>>                 find. I'm not sure if I need to write my own
>>>>>>                 Tester config, or if I should be setting these in
>>>>>>                 some other way.
>>>>>>
>>>>>>                 How should these values be set?
>>>>>>
>>>>>>                 I'm also not sure what environment
>>>>>>                 variables/arguments are strictly necessary or
>>>>>>                 which are optional.
>>>>>>
>>>>>>                 Regards,
>>>>>>                 Adam
>>>>>>
>>>>>>                 -- 
>>>>>>                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>                 ahassick@iol.unh.edu
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>
>>
>>
>>
>>         -- 
>>         *Adam Hassick*
>>         Senior Developer
>>         UNH InterOperability Lab
>>         ahassick@iol.unh.edu
>>         iol.unh.edu <https://www.iol.unh.edu/>
>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 39041 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 13548 bytes --]

> Is there any firewall in the network or on test hosts which could block
incoming TCP connection to the port 23571
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where you
run test engine?

Our test engine host and the testbed are on the same subnet. The connection
does work sometimes.

> If behaviour the same on the next try and you see that test agent is kept
running, could you check using
>
> # netstat -tnlp
>
> that Test Agent is listening on the port and try to establish TCP
connection from test agent using
>
> $ telnet iol-dts-tester.dpdklab.iol.unh.edu
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
> and check if TCP connection could be established.

I was able to replicate the same behavior again, where it hangs while RCF
is trying to start.
Running this command, I see this in the output:

tcp        0      0 0.0.0.0:23571           0.0.0.0:*               LISTEN
     18599/ta

So it seems like it is listening on the correct port.
Additionally, I was able to connect to the Tester machine from our Test
Engine host using telnet. It printed the PID of the process once the
connection was opened.

I tried running the "ta" application manually on the command line, and it
didn't print anything at all.
Maybe the issue is something on the Test Engine side.

On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> > On the tester host (which appears to be the Peer agent), there are four
> processes that I see running, which look like the test agent processes.
>
> Before the next try I'd recommend to kill these processes.
>
> Is there any firewall in the network or on test hosts which could block
> incoming TCP connection to the port 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where you
> run test engine?
>
> If behaviour the same on the next try and you see that test agent is kept
> running, could you check using
>
> # netstat -tnlp
>
> that Test Agent is listening on the port and try to establish TCP
> connection from test agent using
>
> $ telnet iol-dts-tester.dpdklab.iol.unh.edu
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
> and check if TCP connection could be established.
>
> Another idea is to login Tester under root as testing does, get start TA
> command from the log and try it by hands without -n and remove extra
> escaping.
>
> # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
> host=iol-dts-tester.dpdklab.iol.unh.edu:
> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>
> Hopefully in this case test agent directory remains in the /tmp and you
> don't need to copy it as testing does.
> May be output could shed some light on what's going on.
>
> Andrew.
>
> On 8/24/23 17:30, Adam Hassick wrote:
>
> Hi Andrew,
>
> This is the output that I see in the terminal when this failure occurs,
> after the test agent binaries build and the test engine starts:
>
> Platform default build - pass
> Simple RCF consistency check succeeded
> --->>> Starting Logger...done
> --->>> Starting RCF...rcf_net_engine_connect(): Connection timed out
> iol-dts-tester.dpdklab.iol.unh.edu:23571
>
> Then, it hangs here until I kill the "te_rcf" and "te_tee" processes. I
> let it hang for around 9 minutes.
>
> On the tester host (which appears to be the Peer agent), there are four
> processes that I see running, which look like the test agent processes.
>
> ta.Peer is an empty file. I've attached the log.txt from this run.
>
>  - Adam
>
> On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it goes
>> to 'copy_timeout' in ts-conf/rcf.conf.
>> Description in in doc/sphinx/pages/group_te_engine_rcf.rst says that
>> copy_timeout is in seconds and implementation in lib/rcfunix/rcfunix.c
>> passes the value to select() tv_sec. Theoretically select() could be
>> interrupted by signal, but I think it is unlikely here.
>>
>> I'm not sure that I understand what do you mean by RCF connection
>> timeout. Does it happen on TE startup when RCF starts test agents. If so,
>> TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests are in
>> progress, e.g. in the middle of a test. If so, TE_RCFUNIX_TIMEOUT is
>> unrelated and most likely either host with test agent dies or test agent
>> itself crashes. It would be easier for me if classify it if you share text
>> log (log.txt, full or just corresponding fragment with some context). Also
>> content of ta.DPDK or ta.Peer file depending on which agent has problems
>> could shed some light. Corresponding files contain stdout/stderr of test
>> agents.
>>
>> Andrew.
>>
>> On 8/23/23 17:45, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> I've set up a test rig repository here, and have created configurations
>> for our development testbed based off of the examples.
>> We've been able to get the test suite to run manually on Mellanox CX5
>> devices once.
>> However, we are running into an issue where, when RCF starts, the RCF
>> connection times out very frequently. We aren't sure why this is the case.
>> It works sometimes, but most of the time when we try to run the test
>> engine, it encounters this issue.
>> I've tried changing the RCF port by setting "TE_RCF_PORT=<some port
>> number>" and rebooting the testbed machines. Neither seems to fix the issue.
>>
>> It also seems like the timeout takes far longer than 60 seconds, even
>> when running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the test
>> suite.
>> I assume the unit for this variable is seconds?
>>
>> Thanks,
>> Adam
>>
>> On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> Thanks, I've cloned the example repository and will start setting up a
>>> configuration for our development testbed today. I'll let you know if I run
>>> into any difficulties or have any questions.
>>>
>>>  - Adam
>>>
>>> On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hi Adam,
>>>>
>>>> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully
>>>> it will help to define your test rigs and successfully run some tests
>>>> manually. Feel free to ask any questions and I'll answer here and try to
>>>> update documentation.
>>>>
>>>> Meanwhile I'll prepare missing bits for steps (2) and (3).
>>>> Hopefully everything is in place for step (4), but we need to make
>>>> steps (2) and (3) first.
>>>>
>>>> Andrew.
>>>>
>>>> On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>
>>>> Hi Adam,
>>>>
>>>> > I've conferred with the rest of the team, and we think it would be
>>>> best to move forward with mainly option B.
>>>>
>>>> OK, I'll provide the sample on Monday for you. It is almost ready right
>>>> now, but I need to double-check it before publishing.
>>>>
>>>> Regards,
>>>> Andrew.
>>>>
>>>> On 8/17/23 20:03, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> I'm adding the CI mailing list to this conversation. Others in the
>>>> community might find this conversation valuable.
>>>>
>>>> We do want to run testing on a regular basis. The Jenkins integration
>>>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>>>> I've conferred with the rest of the team, and we think it would be best
>>>> to move forward with mainly option B.
>>>> If you would like to know anything about our testbeds that would help
>>>> you with creating an example ts-rigs repo, I'd be happy to answer any
>>>> questions you have.
>>>>
>>>> We have multiple test rigs (we call these "DUT-tester pairs") that we
>>>> run our existing hardware testing on, with differing network hardware and
>>>> CPU architecture. I figured this might be an important detail.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> Greatings Adam,
>>>>>
>>>>> I'm happy to hear that you're trying to bring it up.
>>>>>
>>>>> As I understand the final goal is to run it on regular basis. So, we
>>>>> need to make it properly from the very beginning.
>>>>> Bring up of all features consists of 4 steps:
>>>>>
>>>>> 1. Create site-specific repository (we call it ts-rigs) which contains
>>>>> information about test rigs and other site-specific information like where
>>>>> to send mails, where to store logs etc. It is required for manual execution
>>>>> as well, since test rigs description is essential. I'll return to the topic
>>>>> below.
>>>>>
>>>>> 2. Setup logs storage for automated runs. Basically it is a disk space
>>>>> plus apache2 web server with few CGI scripts which help a lot to save disk
>>>>> space.
>>>>>
>>>>> 3. Setup Bublik web application which provides web interface to view
>>>>> testing results. Same as https://ts-factory.io/bublik
>>>>>
>>>>> 4. Setup Jenkins to run tests on regularly, save logs in log storage
>>>>> (2) and import it to bublik (3).
>>>>>
>>>>> Last few month we spent on our homework to make it simpler to bring up
>>>>> automated execution using Jenkins -
>>>>> https://github.com/ts-factory/te-jenkins
>>>>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>>>>
>>>>> Let's return to the step (1).
>>>>>
>>>>> Unfortunately there is no publicly available example of the ts-rigs
>>>>> repository since sensitive site-specific information is located there. But
>>>>> I'm ready to help you to create it for UNH. I see two options here:
>>>>>
>>>>> (A) I'll ask questions and based on your answers will create the first
>>>>> draft with my comments.
>>>>>
>>>>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>>>>> create UNH ts-rigs based on it.
>>>>>
>>>>> Of course, I'll help to debug and finally bring it up in any case.
>>>>>
>>>>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>>>>> will help other potential users to bring it up.
>>>>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>>>>
>>>>> Thanks,
>>>>> Andrew.
>>>>>
>>>>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>
>>>>> Greetings Adam,
>>>>>
>>>>>
>>>>> Thanks for contacting us. I copy Andrew who would be happy to help
>>>>>
>>>>> Thanks,
>>>>> Konstantin
>>>>>
>>>>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>
>>>>> 
>>>>> Greetings Konstantin,
>>>>>
>>>>> I am in the process of setting up the DPDK Poll Mode Driver test suite
>>>>> as an addition to our testing coverage for DPDK at the UNH lab.
>>>>>
>>>>> I have some questions about how to set the test suite arguments.
>>>>>
>>>>> I have been able to configure the Test Engine to connect to the hosts
>>>>> in the testbed. The RCF, Configurator, and Tester all begin to run, however
>>>>> the prelude of the test suite fails to run.
>>>>>
>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>
>>>>> The documentation mentions that there are several test parameters for
>>>>> the test suite, like for the IUT test link MAC, etc. These seem like they
>>>>> would need to be set somewhere to run many of the tests.
>>>>>
>>>>> I see in the Test Engine documentation, there are instructions on how
>>>>> to create new parameters for test suites in the Tester configuration, but
>>>>> there is nothing in the user guide or in the Tester guide for how to set
>>>>> the arguments for the parameters when running the test suite that I can
>>>>> find. I'm not sure if I need to write my own Tester config, or if I should
>>>>> be setting these in some other way.
>>>>>
>>>>> How should these values be set?
>>>>>
>>>>> I'm also not sure what environment variables/arguments are strictly
>>>>> necessary or which are optional.
>>>>>
>>>>> Regards,
>>>>> Adam
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 39065 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

I'd like to try to repeat the problem locally. Which Linux distro is 
running on test engine and agents?

In fact I know one problem with Debian 12 and Fedora 38 and we have
patch in review to fix it, however, the behaviour is different in
this case, so it is unlike the same problem.

One more idea is to install valgrind on the test engine host and
run with option --vg-rcf to check if something weird is happening.

What I don't understand right now is why I see just one failed attempt
to connect in your log.txt and then Logger shutdown after 9 minutes.

Andrew.

On 8/24/23 23:29, Adam Hassick wrote:
>  > Is there any firewall in the network or on test hosts which could 
> block incoming TCP connection to the port 23571 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where 
> you run test engine?
> 
> Our test engine host and the testbed are on the same subnet. The 
> connection does work sometimes.
> 
>  > If behaviour the same on the next try and you see that test agent is 
> kept running, could you check using
>  >
>  > # netstat -tnlp
>  >
>  > that Test Agent is listening on the port and try to establish TCP 
> connection from test agent using
>  >
>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>  >
>  > and check if TCP connection could be established.
> 
> I was able to replicate the same behavior again, where it hangs while 
> RCF is trying to start.
> Running this command, I see this in the output:
> 
> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>           
> 0.0.0.0:*               LISTEN      18599/ta
> 
> So it seems like it is listening on the correct port.
> Additionally, I was able to connect to the Tester machine from our Test 
> Engine host using telnet. It printed the PID of the process once the 
> connection was opened.
> 
> I tried running the "ta" application manually on the command line, and 
> it didn't print anything at all.
> Maybe the issue is something on the Test Engine side.
> 
> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>> 
> wrote:
> 
>     Hi Adam,
> 
>      > On the tester host (which appears to be the Peer agent), there
>     are four processes that I see running, which look like the test
>     agent processes.
> 
>     Before the next try I'd recommend to kill these processes.
> 
>     Is there any firewall in the network or on test hosts which could
>     block incoming TCP connection to the port 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>     where you run test engine?
> 
>     If behaviour the same on the next try and you see that test agent is
>     kept running, could you check using
> 
>     # netstat -tnlp
> 
>     that Test Agent is listening on the port and try to establish TCP
>     connection from test agent using
> 
>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> 
>     and check if TCP connection could be established.
> 
>     Another idea is to login Tester under root as testing does, get
>     start TA command from the log and try it by hands without -n and
>     remove extra escaping.
> 
>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
> 
>     Hopefully in this case test agent directory remains in the /tmp and
>     you don't need to copy it as testing does.
>     May be output could shed some light on what's going on.
> 
>     Andrew.
> 
>     On 8/24/23 17:30, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     This is the output that I see in the terminal when this failure
>>     occurs, after the test agent binaries build and the test engine
>>     starts:
>>
>>     Platform default build - pass
>>     Simple RCF consistency check succeeded
>>     --->>> Starting Logger...done
>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>     processes. I let it hang for around 9 minutes.
>>
>>     On the tester host (which appears to be the Peer agent), there are
>>     four processes that I see running, which look like the test agent
>>     processes.
>>
>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>
>>      - Adam
>>
>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru
>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>         Hi Adam,
>>
>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>         says that copy_timeout is in seconds and implementation in
>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>         Theoretically select() could be interrupted by signal, but I
>>         think it is unlikely here.
>>
>>         I'm not sure that I understand what do you mean by RCF
>>         connection timeout. Does it happen on TE startup when RCF
>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>         does it happen when tests are in progress, e.g. in the middle
>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>         likely either host with test agent dies or test agent itself
>>         crashes. It would be easier for me if classify it if you share
>>         text log (log.txt, full or just corresponding fragment with
>>         some context). Also content of ta.DPDK or ta.Peer file
>>         depending on which agent has problems could shed some light.
>>         Corresponding files contain stdout/stderr of test agents.
>>
>>         Andrew.
>>
>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         I've set up a test rig repository here, and have created
>>>         configurations for our development testbed based off of the
>>>         examples.
>>>         We've been able to get the test suite to run manually on
>>>         Mellanox CX5 devices once.
>>>         However, we are running into an issue where, when RCF starts,
>>>         the RCF connection times out very frequently. We aren't sure
>>>         why this is the case.
>>>         It works sometimes, but most of the time when we try to run
>>>         the test engine, it encounters this issue.
>>>         I've tried changing the RCF port by setting
>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>         machines. Neither seems to fix the issue.
>>>
>>>         It also seems like the timeout takes far longer than 60
>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>         before I try to run the test suite.
>>>         I assume the unit for this variable is seconds?
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>> wrote:
>>>
>>>             Hi Andrew,
>>>
>>>             Thanks, I've cloned the example repository and will start
>>>             setting up a configuration for our development testbed
>>>             today. I'll let you know if I run into any difficulties
>>>             or have any questions.
>>>
>>>              - Adam
>>>
>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>             <andrew.rybchenko@oktetlabs.ru
>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>                 Hi Adam,
>>>
>>>                 I've published
>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>                 <https://github.com/ts-factory/ts-rigs-sample>.
>>>                 Hopefully it will help to define your test rigs and
>>>                 successfully run some tests manually. Feel free to
>>>                 ask any questions and I'll answer here and try to
>>>                 update documentation.
>>>
>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>                 (3).
>>>                 Hopefully everything is in place for step (4), but we
>>>                 need to make steps (2) and (3) first.
>>>
>>>                 Andrew.
>>>
>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>                 Hi Adam,
>>>>
>>>>                 > I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>
>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>                 almost ready right now, but I need to double-check
>>>>                 it before publishing.
>>>>
>>>>                 Regards,
>>>>                 Andrew.
>>>>
>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 I'm adding the CI mailing list to this
>>>>>                 conversation. Others in the community might find
>>>>>                 this conversation valuable.
>>>>>
>>>>>                 We do want to run testing on a regular basis. The
>>>>>                 Jenkins integration will be very useful for us, as
>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>                 I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>                 If you would like to know anything about our
>>>>>                 testbeds that would help you with creating an
>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>                 questions you have.
>>>>>
>>>>>                 We have multiple test rigs (we call these
>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>                 hardware testing on, with differing network
>>>>>                 hardware and CPU architecture. I figured this might
>>>>>                 be an important detail.
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                     Greatings Adam,
>>>>>
>>>>>                     I'm happy to hear that you're trying to bring
>>>>>                     it up.
>>>>>
>>>>>                     As I understand the final goal is to run it on
>>>>>                     regular basis. So, we need to make it properly
>>>>>                     from the very beginning.
>>>>>                     Bring up of all features consists of 4 steps:
>>>>>
>>>>>                     1. Create site-specific repository (we call it
>>>>>                     ts-rigs) which contains information about test
>>>>>                     rigs and other site-specific information like
>>>>>                     where to send mails, where to store logs etc.
>>>>>                     It is required for manual execution as well,
>>>>>                     since test rigs description is essential. I'll
>>>>>                     return to the topic below.
>>>>>
>>>>>                     2. Setup logs storage for automated runs.
>>>>>                     Basically it is a disk space plus apache2 web
>>>>>                     server with few CGI scripts which help a lot to
>>>>>                     save disk space.
>>>>>
>>>>>                     3. Setup Bublik web application which provides
>>>>>                     web interface to view testing results. Same as
>>>>>                     https://ts-factory.io/bublik
>>>>>                     <https://ts-factory.io/bublik>
>>>>>
>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>                     save logs in log storage (2) and import it to
>>>>>                     bublik (3).
>>>>>
>>>>>                     Last few month we spent on our homework to make
>>>>>                     it simpler to bring up automated execution
>>>>>                     using Jenkins -
>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>                     available tomorrow.
>>>>>
>>>>>                     Let's return to the step (1).
>>>>>
>>>>>                     Unfortunately there is no publicly available
>>>>>                     example of the ts-rigs repository since
>>>>>                     sensitive site-specific information is located
>>>>>                     there. But I'm ready to help you to create it
>>>>>                     for UNH. I see two options here:
>>>>>
>>>>>                     (A) I'll ask questions and based on your
>>>>>                     answers will create the first draft with my
>>>>>                     comments.
>>>>>
>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>                     on it.
>>>>>
>>>>>                     Of course, I'll help to debug and finally bring
>>>>>                     it up in any case.
>>>>>
>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>                     a bit more generic and will help other
>>>>>                     potential users to bring it up.
>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>                     What do you think?
>>>>>
>>>>>                     Thanks,
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>                     Greetings Adam,
>>>>>>
>>>>>>
>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>                     would be happy to help
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Konstantin
>>>>>>
>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>                     <ahassick@iol.unh.edu>
>>>>>>>                     <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>
>>>>>>>                     
>>>>>>>                     Greetings Konstantin,
>>>>>>>
>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>
>>>>>>>                     I have some questions about how to set the
>>>>>>>                     test suite arguments.
>>>>>>>
>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>                     run, however the prelude of the test suite
>>>>>>>                     fails to run.
>>>>>>>
>>>>>>>                     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>
>>>>>>>                     The documentation mentions that there are
>>>>>>>                     several test parameters for the test suite,
>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>                     to run many of the tests.
>>>>>>>
>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>                     are instructions on how to create new
>>>>>>>                     parameters for test suites in the Tester
>>>>>>>                     configuration, but there is nothing in the
>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>                     set the arguments for the parameters when
>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>                     config, or if I should be setting these in
>>>>>>>                     some other way.
>>>>>>>
>>>>>>>                     How should these values be set?
>>>>>>>
>>>>>>>                     I'm also not sure what environment
>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>                     which are optional.
>>>>>>>
>>>>>>>                     Regards,
>>>>>>>                     Adam
>>>>>>>
>>>>>>>                     -- 
>>>>>>>                     *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>                 -- 
>>>>>                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>             -- 
>>>             *Adam Hassick*
>>>             Senior Developer
>>>             UNH InterOperability Lab
>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
> 
> 
> 
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 22293 bytes --]

Hello Adam,

On 8/24/23 23:54, Andrew Rybchenko wrote:
> I'd like to try to repeat the problem locally. Which Linux distro is 
> running on test engine and agents?
>
> In fact I know one problem with Debian 12 and Fedora 38 and we have
> patch in review to fix it, however, the behaviour is different in
> this case, so it is unlike the same problem.

I've just published a new tag which fixes known test engine side 
problems on Debian 12 and Fedora 38.

>
> One more idea is to install valgrind on the test engine host and
> run with option --vg-rcf to check if something weird is happening.
>
> What I don't understand right now is why I see just one failed attempt
> to connect in your log.txt and then Logger shutdown after 9 minutes.
>
> Andrew.
>
> On 8/24/23 23:29, Adam Hassick wrote:
>>  > Is there any firewall in the network or on test hosts which could 
>> block incoming TCP connection to the port 23571 
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where 
>> you run test engine?
>>
>> Our test engine host and the testbed are on the same subnet. The 
>> connection does work sometimes.
>>
>>  > If behaviour the same on the next try and you see that test agent 
>> is kept running, could you check using
>>  >
>>  > # netstat -tnlp
>>  >
>>  > that Test Agent is listening on the port and try to establish TCP 
>> connection from test agent using
>>  >
>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu 
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571 
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>  >
>>  > and check if TCP connection could be established.
>>
>> I was able to replicate the same behavior again, where it hangs while 
>> RCF is trying to start.
>> Running this command, I see this in the output:
>>
>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>           
>> 0.0.0.0:* LISTEN      18599/ta
>>
>> So it seems like it is listening on the correct port.
>> Additionally, I was able to connect to the Tester machine from our 
>> Test Engine host using telnet. It printed the PID of the process once 
>> the connection was opened.
>>
>> I tried running the "ta" application manually on the command line, 
>> and it didn't print anything at all.
>> Maybe the issue is something on the Test Engine side.
>>
>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru 
>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>     Hi Adam,
>>
>>      > On the tester host (which appears to be the Peer agent), there
>>     are four processes that I see running, which look like the test
>>     agent processes.
>>
>>     Before the next try I'd recommend to kill these processes.
>>
>>     Is there any firewall in the network or on test hosts which could
>>     block incoming TCP connection to the port 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>     where you run test engine?
>>
>>     If behaviour the same on the next try and you see that test agent is
>>     kept running, could you check using
>>
>>     # netstat -tnlp
>>
>>     that Test Agent is listening on the port and try to establish TCP
>>     connection from test agent using
>>
>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     and check if TCP connection could be established.
>>
>>     Another idea is to login Tester under root as testing does, get
>>     start TA command from the log and try it by hands without -n and
>>     remove extra escaping.
>>
>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 
>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 
>> host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>
>>     Hopefully in this case test agent directory remains in the /tmp and
>>     you don't need to copy it as testing does.
>>     May be output could shed some light on what's going on.
>>
>>     Andrew.
>>
>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     This is the output that I see in the terminal when this failure
>>>     occurs, after the test agent binaries build and the test engine
>>>     starts:
>>>
>>>     Platform default build - pass
>>>     Simple RCF consistency check succeeded
>>>     --->>> Starting Logger...done
>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>     processes. I let it hang for around 9 minutes.
>>>
>>>     On the tester host (which appears to be the Peer agent), there are
>>>     four processes that I see running, which look like the test agent
>>>     processes.
>>>
>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>
>>>      - Adam
>>>
>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru
>>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>         Hi Adam,
>>>
>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>         says that copy_timeout is in seconds and implementation in
>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>         Theoretically select() could be interrupted by signal, but I
>>>         think it is unlikely here.
>>>
>>>         I'm not sure that I understand what do you mean by RCF
>>>         connection timeout. Does it happen on TE startup when RCF
>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>         does it happen when tests are in progress, e.g. in the middle
>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>         likely either host with test agent dies or test agent itself
>>>         crashes. It would be easier for me if classify it if you share
>>>         text log (log.txt, full or just corresponding fragment with
>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>         depending on which agent has problems could shed some light.
>>>         Corresponding files contain stdout/stderr of test agents.
>>>
>>>         Andrew.
>>>
>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         I've set up a test rig repository here, and have created
>>>>         configurations for our development testbed based off of the
>>>>         examples.
>>>>         We've been able to get the test suite to run manually on
>>>>         Mellanox CX5 devices once.
>>>>         However, we are running into an issue where, when RCF starts,
>>>>         the RCF connection times out very frequently. We aren't sure
>>>>         why this is the case.
>>>>         It works sometimes, but most of the time when we try to run
>>>>         the test engine, it encounters this issue.
>>>>         I've tried changing the RCF port by setting
>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>         machines. Neither seems to fix the issue.
>>>>
>>>>         It also seems like the timeout takes far longer than 60
>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>         before I try to run the test suite.
>>>>         I assume the unit for this variable is seconds?
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>> wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             Thanks, I've cloned the example repository and will start
>>>>             setting up a configuration for our development testbed
>>>>             today. I'll let you know if I run into any difficulties
>>>>             or have any questions.
>>>>
>>>>              - Adam
>>>>
>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru
>>>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 I've published
>>>> https://github.com/ts-factory/ts-rigs-sample
>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>                 Hopefully it will help to define your test rigs and
>>>>                 successfully run some tests manually. Feel free to
>>>>                 ask any questions and I'll answer here and try to
>>>>                 update documentation.
>>>>
>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>                 (3).
>>>>                 Hopefully everything is in place for step (4), but we
>>>>                 need to make steps (2) and (3) first.
>>>>
>>>>                 Andrew.
>>>>
>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>                 Hi Adam,
>>>>>
>>>>>                 > I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>
>>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>>                 almost ready right now, but I need to double-check
>>>>>                 it before publishing.
>>>>>
>>>>>                 Regards,
>>>>>                 Andrew.
>>>>>
>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 I'm adding the CI mailing list to this
>>>>>>                 conversation. Others in the community might find
>>>>>>                 this conversation valuable.
>>>>>>
>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>                 Jenkins integration will be very useful for us, as
>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>                 think it would be best to move forward with mainly
>>>>>>                 option B.
>>>>>>                 If you would like to know anything about our
>>>>>>                 testbeds that would help you with creating an
>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>                 questions you have.
>>>>>>
>>>>>>                 We have multiple test rigs (we call these
>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>                 hardware testing on, with differing network
>>>>>>                 hardware and CPU architecture. I figured this might
>>>>>>                 be an important detail.
>>>>>>
>>>>>>                 Thanks,
>>>>>>                 Adam
>>>>>>
>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                     Greatings Adam,
>>>>>>
>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>                     it up.
>>>>>>
>>>>>>                     As I understand the final goal is to run it on
>>>>>>                     regular basis. So, we need to make it properly
>>>>>>                     from the very beginning.
>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>
>>>>>>                     1. Create site-specific repository (we call it
>>>>>>                     ts-rigs) which contains information about test
>>>>>>                     rigs and other site-specific information like
>>>>>>                     where to send mails, where to store logs etc.
>>>>>>                     It is required for manual execution as well,
>>>>>>                     since test rigs description is essential. I'll
>>>>>>                     return to the topic below.
>>>>>>
>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>                     server with few CGI scripts which help a lot to
>>>>>>                     save disk space.
>>>>>>
>>>>>>                     3. Setup Bublik web application which provides
>>>>>>                     web interface to view testing results. Same as
>>>>>> https://ts-factory.io/bublik
>>>>>> <https://ts-factory.io/bublik>
>>>>>>
>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>                     save logs in log storage (2) and import it to
>>>>>>                     bublik (3).
>>>>>>
>>>>>>                     Last few month we spent on our homework to make
>>>>>>                     it simpler to bring up automated execution
>>>>>>                     using Jenkins -
>>>>>> https://github.com/ts-factory/te-jenkins
>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>                     available tomorrow.
>>>>>>
>>>>>>                     Let's return to the step (1).
>>>>>>
>>>>>>                     Unfortunately there is no publicly available
>>>>>>                     example of the ts-rigs repository since
>>>>>>                     sensitive site-specific information is located
>>>>>>                     there. But I'm ready to help you to create it
>>>>>>                     for UNH. I see two options here:
>>>>>>
>>>>>>                     (A) I'll ask questions and based on your
>>>>>>                     answers will create the first draft with my
>>>>>>                     comments.
>>>>>>
>>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>>                     on it.
>>>>>>
>>>>>>                     Of course, I'll help to debug and finally bring
>>>>>>                     it up in any case.
>>>>>>
>>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>>                     a bit more generic and will help other
>>>>>>                     potential users to bring it up.
>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>                     What do you think?
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>                     Greetings Adam,
>>>>>>>
>>>>>>>
>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>                     would be happy to help
>>>>>>>
>>>>>>>                     Thanks,
>>>>>>>                     Konstantin
>>>>>>>
>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>> <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>
>>>>>>>>                     
>>>>>>>>                     Greetings Konstantin,
>>>>>>>>
>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>>
>>>>>>>>                     I have some questions about how to set the
>>>>>>>>                     test suite arguments.
>>>>>>>>
>>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>                     fails to run.
>>>>>>>>
>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters 
>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters> 
>>>>>>>>
>>>>>>>>
>>>>>>>>                     The documentation mentions that there are
>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>>                     to run many of the tests.
>>>>>>>>
>>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>>                     are instructions on how to create new
>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>                     some other way.
>>>>>>>>
>>>>>>>>                     How should these values be set?
>>>>>>>>
>>>>>>>>                     I'm also not sure what environment
>>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>>                     which are optional.
>>>>>>>>
>>>>>>>>                     Regards,
>>>>>>>>                     Adam
>>>>>>>>
>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>> ahassick@iol.unh.edu
>>>>>>>> <mailto:ahassick@iol.unh.edu>
>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>             --             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>     --     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 39090 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 20482 bytes --]

Hi Andrew,

Two of our systems (the Test Engine runner and the DUT host) are running
Ubuntu 20.04 LTS, however this morning I noticed that the tester system
(the one having issues) is running Ubuntu 22.04 LTS.
This could be the source of the problem. I encountered a dependency issue
trying to run the Test Engine on 22.04 LTS, so I downgraded the system.
Since the tester is also the host having connection issues, I will try
downgrading that system to 20.04, and see if that changes anything.

I did try passing in the "--vg-rcf" argument to the run.sh script of the
test suite after installing valgrind, but there was no additional output
that I saw.

I will try pulling in the changes you've pushed up, and will see if that
fixes anything.

Thanks,
Adam

On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hello Adam,
>
> On 8/24/23 23:54, Andrew Rybchenko wrote:
>
> I'd like to try to repeat the problem locally. Which Linux distro is
> running on test engine and agents?
>
> In fact I know one problem with Debian 12 and Fedora 38 and we have
> patch in review to fix it, however, the behaviour is different in
> this case, so it is unlike the same problem.
>
>
> I've just published a new tag which fixes known test engine side problems
> on Debian 12 and Fedora 38.
>
>
> One more idea is to install valgrind on the test engine host and
> run with option --vg-rcf to check if something weird is happening.
>
> What I don't understand right now is why I see just one failed attempt
> to connect in your log.txt and then Logger shutdown after 9 minutes.
>
> Andrew.
>
> On 8/24/23 23:29, Adam Hassick wrote:
>
>  > Is there any firewall in the network or on test hosts which could block
> incoming TCP connection to the port 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where you
> run test engine?
>
> Our test engine host and the testbed are on the same subnet. The
> connection does work sometimes.
>
>  > If behaviour the same on the next try and you see that test agent is
> kept running, could you check using
>  >
>  > # netstat -tnlp
>  >
>  > that Test Agent is listening on the port and try to establish TCP
> connection from test agent using
>  >
>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>  >
>  > and check if TCP connection could be established.
>
> I was able to replicate the same behavior again, where it hangs while RCF
> is trying to start.
> Running this command, I see this in the output:
>
> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>  18599/ta
>
> So it seems like it is listening on the correct port.
> Additionally, I was able to connect to the Tester machine from our Test
> Engine host using telnet. It printed the PID of the process once the
> connection was opened.
>
> I tried running the "ta" application manually on the command line, and it
> didn't print anything at all.
> Maybe the issue is something on the Test Engine side.
>
> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>     Hi Adam,
>
>      > On the tester host (which appears to be the Peer agent), there
>     are four processes that I see running, which look like the test
>     agent processes.
>
>     Before the next try I'd recommend to kill these processes.
>
>     Is there any firewall in the network or on test hosts which could
>     block incoming TCP connection to the port 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>     where you run test engine?
>
>     If behaviour the same on the next try and you see that test agent is
>     kept running, could you check using
>
>     # netstat -tnlp
>
>     that Test Agent is listening on the port and try to establish TCP
>     connection from test agent using
>
>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
>     and check if TCP connection could be established.
>
>     Another idea is to login Tester under root as testing does, get
>     start TA command from the log and try it by hands without -n and
>     remove extra escaping.
>
>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>
> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
> host=iol-dts-tester.dpdklab.iol.unh.edu:
> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>
>     Hopefully in this case test agent directory remains in the /tmp and
>     you don't need to copy it as testing does.
>     May be output could shed some light on what's going on.
>
>     Andrew.
>
>     On 8/24/23 17:30, Adam Hassick wrote:
>
>     Hi Andrew,
>
>     This is the output that I see in the terminal when this failure
>     occurs, after the test agent binaries build and the test engine
>     starts:
>
>     Platform default build - pass
>     Simple RCF consistency check succeeded
>     --->>> Starting Logger...done
>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>     processes. I let it hang for around 9 minutes.
>
>     On the tester host (which appears to be the Peer agent), there are
>     four processes that I see running, which look like the test agent
>     processes.
>
>     ta.Peer is an empty file. I've attached the log.txt from this run.
>
>      - Adam
>
>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>     <andrew.rybchenko@oktetlabs.ru
>     <mailto:andrew.rybchenko@oktetlabs.ru> <andrew.rybchenko@oktetlabs.ru>>
> wrote:
>
>         Hi Adam,
>
>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>         says that copy_timeout is in seconds and implementation in
>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>         Theoretically select() could be interrupted by signal, but I
>         think it is unlikely here.
>
>         I'm not sure that I understand what do you mean by RCF
>         connection timeout. Does it happen on TE startup when RCF
>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>         does it happen when tests are in progress, e.g. in the middle
>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>         likely either host with test agent dies or test agent itself
>         crashes. It would be easier for me if classify it if you share
>         text log (log.txt, full or just corresponding fragment with
>         some context). Also content of ta.DPDK or ta.Peer file
>         depending on which agent has problems could shed some light.
>         Corresponding files contain stdout/stderr of test agents.
>
>         Andrew.
>
>         On 8/23/23 17:45, Adam Hassick wrote:
>
>         Hi Andrew,
>
>         I've set up a test rig repository here, and have created
>         configurations for our development testbed based off of the
>         examples.
>         We've been able to get the test suite to run manually on
>         Mellanox CX5 devices once.
>         However, we are running into an issue where, when RCF starts,
>         the RCF connection times out very frequently. We aren't sure
>         why this is the case.
>         It works sometimes, but most of the time when we try to run
>         the test engine, it encounters this issue.
>         I've tried changing the RCF port by setting
>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>         machines. Neither seems to fix the issue.
>
>         It also seems like the timeout takes far longer than 60
>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>         before I try to run the test suite.
>         I assume the unit for this variable is seconds?
>
>         Thanks,
>         Adam
>
>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>> wrote:
>
>             Hi Andrew,
>
>             Thanks, I've cloned the example repository and will start
>             setting up a configuration for our development testbed
>             today. I'll let you know if I run into any difficulties
>             or have any questions.
>
>              - Adam
>
>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>             <andrew.rybchenko@oktetlabs.ru
>             <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>                 Hi Adam,
>
>                 I've published
>                 https://github.com/ts-factory/ts-rigs-sample
>                 <https://github.com/ts-factory/ts-rigs-sample>
> <https://github.com/ts-factory/ts-rigs-sample>.
>                 Hopefully it will help to define your test rigs and
>                 successfully run some tests manually. Feel free to
>                 ask any questions and I'll answer here and try to
>                 update documentation.
>
>                 Meanwhile I'll prepare missing bits for steps (2) and
>                 (3).
>                 Hopefully everything is in place for step (4), but we
>                 need to make steps (2) and (3) first.
>
>                 Andrew.
>
>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>
>                 Hi Adam,
>
>                 > I've conferred with the rest of the team, and we
>                 think it would be best to move forward with mainly
>                 option B.
>
>                 OK, I'll provide the sample on Monday for you. It is
>                 almost ready right now, but I need to double-check
>                 it before publishing.
>
>                 Regards,
>                 Andrew.
>
>                 On 8/17/23 20:03, Adam Hassick wrote:
>
>                 Hi Andrew,
>
>                 I'm adding the CI mailing list to this
>                 conversation. Others in the community might find
>                 this conversation valuable.
>
>                 We do want to run testing on a regular basis. The
>                 Jenkins integration will be very useful for us, as
>                 most of our CI is orchestrated by Jenkins.
>                 I've conferred with the rest of the team, and we
>                 think it would be best to move forward with mainly
>                 option B.
>                 If you would like to know anything about our
>                 testbeds that would help you with creating an
>                 example ts-rigs repo, I'd be happy to answer any
>                 questions you have.
>
>                 We have multiple test rigs (we call these
>                 "DUT-tester pairs") that we run our existing
>                 hardware testing on, with differing network
>                 hardware and CPU architecture. I figured this might
>                 be an important detail.
>
>                 Thanks,
>                 Adam
>
>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>                 <andrew.rybchenko@oktetlabs.ru
>                 <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>                     Greatings Adam,
>
>                     I'm happy to hear that you're trying to bring
>                     it up.
>
>                     As I understand the final goal is to run it on
>                     regular basis. So, we need to make it properly
>                     from the very beginning.
>                     Bring up of all features consists of 4 steps:
>
>                     1. Create site-specific repository (we call it
>                     ts-rigs) which contains information about test
>                     rigs and other site-specific information like
>                     where to send mails, where to store logs etc.
>                     It is required for manual execution as well,
>                     since test rigs description is essential. I'll
>                     return to the topic below.
>
>                     2. Setup logs storage for automated runs.
>                     Basically it is a disk space plus apache2 web
>                     server with few CGI scripts which help a lot to
>                     save disk space.
>
>                     3. Setup Bublik web application which provides
>                     web interface to view testing results. Same as
>                     https://ts-factory.io/bublik
>                     <https://ts-factory.io/bublik>
> <https://ts-factory.io/bublik>
>
>                     4. Setup Jenkins to run tests on regularly,
>                     save logs in log storage (2) and import it to
>                     bublik (3).
>
>                     Last few month we spent on our homework to make
>                     it simpler to bring up automated execution
>                     using Jenkins -
>                     https://github.com/ts-factory/te-jenkins
>                     <https://github.com/ts-factory/te-jenkins>
> <https://github.com/ts-factory/te-jenkins>
>                     Corresponding bits in dpdk-ethdev-ts will be
>                     available tomorrow.
>
>                     Let's return to the step (1).
>
>                     Unfortunately there is no publicly available
>                     example of the ts-rigs repository since
>                     sensitive site-specific information is located
>                     there. But I'm ready to help you to create it
>                     for UNH. I see two options here:
>
>                     (A) I'll ask questions and based on your
>                     answers will create the first draft with my
>                     comments.
>
>                     (B) I'll make a template/example ts-rigs repo,
>                     publish it and you'll create UNH ts-rigs based
>                     on it.
>
>                     Of course, I'll help to debug and finally bring
>                     it up in any case.
>
>                     (A) is a bit simpler for me and you, but (B) is
>                     a bit more generic and will help other
>                     potential users to bring it up.
>                     We can combine (A)+(B). I.e. start from (A).
>                     What do you think?
>
>                     Thanks,
>                     Andrew.
>
>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>
>                     Greetings Adam,
>
>
>                     Thanks for contacting us. I copy Andrew who
>                     would be happy to help
>
>                     Thanks,
>                     Konstantin
>
>                     On 16 Aug 2023, at 21:50, Adam Hassick
>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
> wrote:
>
>                     
>                     Greetings Konstantin,
>
>                     I am in the process of setting up the DPDK
>                     Poll Mode Driver test suite as an addition to
>                     our testing coverage for DPDK at the UNH lab.
>
>                     I have some questions about how to set the
>                     test suite arguments.
>
>                     I have been able to configure the Test Engine
>                     to connect to the hosts in the testbed. The
>                     RCF, Configurator, and Tester all begin to
>                     run, however the prelude of the test suite
>                     fails to run.
>
>
> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>
>                     The documentation mentions that there are
>                     several test parameters for the test suite,
>                     like for the IUT test link MAC, etc. These
>                     seem like they would need to be set somewhere
>                     to run many of the tests.
>
>                     I see in the Test Engine documentation, there
>                     are instructions on how to create new
>                     parameters for test suites in the Tester
>                     configuration, but there is nothing in the
>                     user guide or in the Tester guide for how to
>                     set the arguments for the parameters when
>                     running the test suite that I can find. I'm
>                     not sure if I need to write my own Tester
>                     config, or if I should be setting these in
>                     some other way.
>
>                     How should these values be set?
>
>                     I'm also not sure what environment
>                     variables/arguments are strictly necessary or
>                     which are optional.
>
>                     Regards,
>                     Adam
>
>                     --                     *Adam Hassick*
>                     Senior Developer
>                     UNH InterOperability Lab
>                     ahassick@iol.unh.edu
>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>                     iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>                     +1 (603) 475-8248
>
>
>
>
>                 --                 *Adam Hassick*
>                 Senior Developer
>                 UNH InterOperability Lab
>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>                 iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>                 +1 (603) 475-8248
>
>
>
>
>
>             --             *Adam Hassick*
>             Senior Developer
>             UNH InterOperability Lab
>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>             iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>             +1 (603) 475-8248
>
>
>
>         --         *Adam Hassick*
>         Senior Developer
>         UNH InterOperability Lab
>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>         iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>         +1 (603) 475-8248
>
>
>
>
>     --     *Adam Hassick*
>     Senior Developer
>     UNH InterOperability Lab
>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>     +1 (603) 475-8248
>
>
>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 42032 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 28066 bytes --]

On 8/25/23 17:06, Adam Hassick wrote:
> Hi Andrew,
>
> Two of our systems (the Test Engine runner and the DUT host) are 
> running Ubuntu 20.04 LTS, however this morning I noticed that the 
> tester system (the one having issues) is running Ubuntu 22.04 LTS.
> This could be the source of the problem. I encountered a dependency 
> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the 
> system. Since the tester is also the host having connection issues, I 
> will try downgrading that system to 20.04, and see if that changes 
> anything.

Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu 
22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
Right now Debian 11 is used for test engine in nightly regressions.

I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.

> I did try passing in the "--vg-rcf" argument to the run.sh script of 
> the test suite after installing valgrind, but there was no additional 
> output that I saw.

Sorry, I should valgrind output should be in valgrind.te_rcf (direction 
where you run test engine).

>
> I will try pulling in the changes you've pushed up, and will see if 
> that fixes anything.
>
> Thanks,
> Adam
>
> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hello Adam,
>
>     On 8/24/23 23:54, Andrew Rybchenko wrote:
>>     I'd like to try to repeat the problem locally. Which Linux distro
>>     is running on test engine and agents?
>>
>>     In fact I know one problem with Debian 12 and Fedora 38 and we have
>>     patch in review to fix it, however, the behaviour is different in
>>     this case, so it is unlike the same problem.
>
>     I've just published a new tag which fixes known test engine side
>     problems on Debian 12 and Fedora 38.
>
>>
>>     One more idea is to install valgrind on the test engine host and
>>     run with option --vg-rcf to check if something weird is happening.
>>
>>     What I don't understand right now is why I see just one failed
>>     attempt
>>     to connect in your log.txt and then Logger shutdown after 9 minutes.
>>
>>     Andrew.
>>
>>     On 8/24/23 23:29, Adam Hassick wrote:
>>>      > Is there any firewall in the network or on test hosts which
>>>     could block incoming TCP connection to the port 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>     where you run test engine?
>>>
>>>     Our test engine host and the testbed are on the same subnet. The
>>>     connection does work sometimes.
>>>
>>>      > If behaviour the same on the next try and you see that test
>>>     agent is kept running, could you check using
>>>      >
>>>      > # netstat -tnlp
>>>      >
>>>      > that Test Agent is listening on the port and try to establish
>>>     TCP connection from test agent using
>>>      >
>>>      > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>      >
>>>      > and check if TCP connection could be established.
>>>
>>>     I was able to replicate the same behavior again, where it hangs
>>>     while RCF is trying to start.
>>>     Running this command, I see this in the output:
>>>
>>>     tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>     <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*        
>>>           LISTEN      18599/ta
>>>
>>>     So it seems like it is listening on the correct port.
>>>     Additionally, I was able to connect to the Tester machine from
>>>     our Test Engine host using telnet. It printed the PID of the
>>>     process once the connection was opened.
>>>
>>>     I tried running the "ta" application manually on the command
>>>     line, and it didn't print anything at all.
>>>     Maybe the issue is something on the Test Engine side.
>>>
>>>     On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru
>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>         Hi Adam,
>>>
>>>          > On the tester host (which appears to be the Peer agent),
>>>     there
>>>         are four processes that I see running, which look like the test
>>>         agent processes.
>>>
>>>         Before the next try I'd recommend to kill these processes.
>>>
>>>         Is there any firewall in the network or on test hosts which
>>>     could
>>>         block incoming TCP connection to the port 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>         where you run test engine?
>>>
>>>         If behaviour the same on the next try and you see that test
>>>     agent is
>>>         kept running, could you check using
>>>
>>>         # netstat -tnlp
>>>
>>>         that Test Agent is listening on the port and try to
>>>     establish TCP
>>>         connection from test agent using
>>>
>>>         $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>         and check if TCP connection could be established.
>>>
>>>         Another idea is to login Tester under root as testing does, get
>>>         start TA command from the log and try it by hands without -n
>>>     and
>>>         remove extra escaping.
>>>
>>>         # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>     /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>     host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>
>>>         Hopefully in this case test agent directory remains in the
>>>     /tmp and
>>>         you don't need to copy it as testing does.
>>>         May be output could shed some light on what's going on.
>>>
>>>         Andrew.
>>>
>>>         On 8/24/23 17:30, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         This is the output that I see in the terminal when this
>>>>     failure
>>>>         occurs, after the test agent binaries build and the test
>>>>     engine
>>>>         starts:
>>>>
>>>>         Platform default build - pass
>>>>         Simple RCF consistency check succeeded
>>>>         --->>> Starting Logger...done
>>>>         --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>     timed
>>>>         out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>         Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>         processes. I let it hang for around 9 minutes.
>>>>
>>>>         On the tester host (which appears to be the Peer agent),
>>>>     there are
>>>>         four processes that I see running, which look like the test
>>>>     agent
>>>>         processes.
>>>>
>>>>         ta.Peer is an empty file. I've attached the log.txt from
>>>>     this run.
>>>>
>>>>          - Adam
>>>>
>>>>         On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>             Hi Adam,
>>>>
>>>>             Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>             that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>             Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>             says that copy_timeout is in seconds and implementation in
>>>>             lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>             Theoretically select() could be interrupted by signal,
>>>>     but I
>>>>             think it is unlikely here.
>>>>
>>>>             I'm not sure that I understand what do you mean by RCF
>>>>             connection timeout. Does it happen on TE startup when RCF
>>>>             starts test agents. If so, TE_RCFUNIX_TIMEOUT could
>>>>     help. Or
>>>>             does it happen when tests are in progress, e.g. in the
>>>>     middle
>>>>             of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>             likely either host with test agent dies or test agent
>>>>     itself
>>>>             crashes. It would be easier for me if classify it if
>>>>     you share
>>>>             text log (log.txt, full or just corresponding fragment
>>>>     with
>>>>             some context). Also content of ta.DPDK or ta.Peer file
>>>>             depending on which agent has problems could shed some
>>>>     light.
>>>>             Corresponding files contain stdout/stderr of test agents.
>>>>
>>>>             Andrew.
>>>>
>>>>             On 8/23/23 17:45, Adam Hassick wrote:
>>>>>             Hi Andrew,
>>>>>
>>>>>             I've set up a test rig repository here, and have created
>>>>>             configurations for our development testbed based off
>>>>>     of the
>>>>>             examples.
>>>>>             We've been able to get the test suite to run manually on
>>>>>             Mellanox CX5 devices once.
>>>>>             However, we are running into an issue where, when RCF
>>>>>     starts,
>>>>>             the RCF connection times out very frequently. We
>>>>>     aren't sure
>>>>>             why this is the case.
>>>>>             It works sometimes, but most of the time when we try
>>>>>     to run
>>>>>             the test engine, it encounters this issue.
>>>>>             I've tried changing the RCF port by setting
>>>>>             "TE_RCF_PORT=<some port number>" and rebooting the
>>>>>     testbed
>>>>>             machines. Neither seems to fix the issue.
>>>>>
>>>>>             It also seems like the timeout takes far longer than 60
>>>>>             seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>             before I try to run the test suite.
>>>>>             I assume the unit for this variable is seconds?
>>>>>
>>>>>             Thanks,
>>>>>             Adam
>>>>>
>>>>>             On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>             <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 Thanks, I've cloned the example repository and
>>>>>     will start
>>>>>                 setting up a configuration for our development
>>>>>     testbed
>>>>>                 today. I'll let you know if I run into any
>>>>>     difficulties
>>>>>                 or have any questions.
>>>>>
>>>>>                  - Adam
>>>>>
>>>>>                 On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                     Hi Adam,
>>>>>
>>>>>                     I've published
>>>>>     https://github.com/ts-factory/ts-rigs-sample
>>>>>     <https://github.com/ts-factory/ts-rigs-sample>
>>>>>     <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>                     Hopefully it will help to define your test
>>>>>     rigs and
>>>>>                     successfully run some tests manually. Feel
>>>>>     free to
>>>>>                     ask any questions and I'll answer here and try to
>>>>>                     update documentation.
>>>>>
>>>>>                     Meanwhile I'll prepare missing bits for steps
>>>>>     (2) and
>>>>>                     (3).
>>>>>                     Hopefully everything is in place for step (4),
>>>>>     but we
>>>>>                     need to make steps (2) and (3) first.
>>>>>
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     > I've conferred with the rest of the team,
>>>>>>     and we
>>>>>>                     think it would be best to move forward with
>>>>>>     mainly
>>>>>>                     option B.
>>>>>>
>>>>>>                     OK, I'll provide the sample on Monday for
>>>>>>     you. It is
>>>>>>                     almost ready right now, but I need to
>>>>>>     double-check
>>>>>>                     it before publishing.
>>>>>>
>>>>>>                     Regards,
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>                     Hi Andrew,
>>>>>>>
>>>>>>>                     I'm adding the CI mailing list to this
>>>>>>>                     conversation. Others in the community might
>>>>>>>     find
>>>>>>>                     this conversation valuable.
>>>>>>>
>>>>>>>                     We do want to run testing on a regular
>>>>>>>     basis. The
>>>>>>>                     Jenkins integration will be very useful for
>>>>>>>     us, as
>>>>>>>                     most of our CI is orchestrated by Jenkins.
>>>>>>>                     I've conferred with the rest of the team,
>>>>>>>     and we
>>>>>>>                     think it would be best to move forward with
>>>>>>>     mainly
>>>>>>>                     option B.
>>>>>>>                     If you would like to know anything about our
>>>>>>>                     testbeds that would help you with creating an
>>>>>>>                     example ts-rigs repo, I'd be happy to answer
>>>>>>>     any
>>>>>>>                     questions you have.
>>>>>>>
>>>>>>>                     We have multiple test rigs (we call these
>>>>>>>                     "DUT-tester pairs") that we run our existing
>>>>>>>                     hardware testing on, with differing network
>>>>>>>                     hardware and CPU architecture. I figured
>>>>>>>     this might
>>>>>>>                     be an important detail.
>>>>>>>
>>>>>>>                     Thanks,
>>>>>>>                     Adam
>>>>>>>
>>>>>>>                     On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>     Rybchenko
>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                         Greatings Adam,
>>>>>>>
>>>>>>>                         I'm happy to hear that you're trying to
>>>>>>>     bring
>>>>>>>                         it up.
>>>>>>>
>>>>>>>                         As I understand the final goal is to run
>>>>>>>     it on
>>>>>>>                         regular basis. So, we need to make it
>>>>>>>     properly
>>>>>>>                         from the very beginning.
>>>>>>>                         Bring up of all features consists of 4
>>>>>>>     steps:
>>>>>>>
>>>>>>>                         1. Create site-specific repository (we
>>>>>>>     call it
>>>>>>>                         ts-rigs) which contains information
>>>>>>>     about test
>>>>>>>                         rigs and other site-specific information
>>>>>>>     like
>>>>>>>                         where to send mails, where to store logs
>>>>>>>     etc.
>>>>>>>                         It is required for manual execution as
>>>>>>>     well,
>>>>>>>                         since test rigs description is
>>>>>>>     essential. I'll
>>>>>>>                         return to the topic below.
>>>>>>>
>>>>>>>                         2. Setup logs storage for automated runs.
>>>>>>>                         Basically it is a disk space plus
>>>>>>>     apache2 web
>>>>>>>                         server with few CGI scripts which help a
>>>>>>>     lot to
>>>>>>>                         save disk space.
>>>>>>>
>>>>>>>                         3. Setup Bublik web application which
>>>>>>>     provides
>>>>>>>                         web interface to view testing results.
>>>>>>>     Same as
>>>>>>>     https://ts-factory.io/bublik
>>>>>>>     <https://ts-factory.io/bublik> <https://ts-factory.io/bublik>
>>>>>>>
>>>>>>>                         4. Setup Jenkins to run tests on regularly,
>>>>>>>                         save logs in log storage (2) and import
>>>>>>>     it to
>>>>>>>                         bublik (3).
>>>>>>>
>>>>>>>                         Last few month we spent on our homework
>>>>>>>     to make
>>>>>>>                         it simpler to bring up automated execution
>>>>>>>                         using Jenkins -
>>>>>>>     https://github.com/ts-factory/te-jenkins
>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>                         Corresponding bits in dpdk-ethdev-ts
>>>>>>>     will be
>>>>>>>                         available tomorrow.
>>>>>>>
>>>>>>>                         Let's return to the step (1).
>>>>>>>
>>>>>>>                         Unfortunately there is no publicly
>>>>>>>     available
>>>>>>>                         example of the ts-rigs repository since
>>>>>>>                         sensitive site-specific information is
>>>>>>>     located
>>>>>>>                         there. But I'm ready to help you to
>>>>>>>     create it
>>>>>>>                         for UNH. I see two options here:
>>>>>>>
>>>>>>>                         (A) I'll ask questions and based on your
>>>>>>>                         answers will create the first draft with my
>>>>>>>                         comments.
>>>>>>>
>>>>>>>                         (B) I'll make a template/example ts-rigs
>>>>>>>     repo,
>>>>>>>                         publish it and you'll create UNH ts-rigs
>>>>>>>     based
>>>>>>>                         on it.
>>>>>>>
>>>>>>>                         Of course, I'll help to debug and
>>>>>>>     finally bring
>>>>>>>                         it up in any case.
>>>>>>>
>>>>>>>                         (A) is a bit simpler for me and you, but
>>>>>>>     (B) is
>>>>>>>                         a bit more generic and will help other
>>>>>>>                         potential users to bring it up.
>>>>>>>                         We can combine (A)+(B). I.e. start from
>>>>>>>     (A).
>>>>>>>                         What do you think?
>>>>>>>
>>>>>>>                         Thanks,
>>>>>>>                         Andrew.
>>>>>>>
>>>>>>>                         On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>     Greetings Adam,
>>>>>>>>
>>>>>>>>
>>>>>>>>                         Thanks for contacting us. I copy Andrew
>>>>>>>>     who
>>>>>>>>                         would be happy to help
>>>>>>>>
>>>>>>>>                         Thanks,
>>>>>>>>                         Konstantin
>>>>>>>>
>>>>>>>>>                         On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>     <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>
>>>>>>>>>                         
>>>>>>>>>                         Greetings Konstantin,
>>>>>>>>>
>>>>>>>>>                         I am in the process of setting up the
>>>>>>>>>     DPDK
>>>>>>>>>                         Poll Mode Driver test suite as an
>>>>>>>>>     addition to
>>>>>>>>>                         our testing coverage for DPDK at the
>>>>>>>>>     UNH lab.
>>>>>>>>>
>>>>>>>>>                         I have some questions about how to set
>>>>>>>>>     the
>>>>>>>>>                         test suite arguments.
>>>>>>>>>
>>>>>>>>>                         I have been able to configure the Test
>>>>>>>>>     Engine
>>>>>>>>>                         to connect to the hosts in the
>>>>>>>>>     testbed. The
>>>>>>>>>                         RCF, Configurator, and Tester all
>>>>>>>>>     begin to
>>>>>>>>>                         run, however the prelude of the test
>>>>>>>>>     suite
>>>>>>>>>                         fails to run.
>>>>>>>>>
>>>>>>>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         The documentation mentions that there are
>>>>>>>>>                         several test parameters for the test
>>>>>>>>>     suite,
>>>>>>>>>                         like for the IUT test link MAC, etc.
>>>>>>>>>     These
>>>>>>>>>                         seem like they would need to be set
>>>>>>>>>     somewhere
>>>>>>>>>                         to run many of the tests.
>>>>>>>>>
>>>>>>>>>                         I see in the Test Engine
>>>>>>>>>     documentation, there
>>>>>>>>>                         are instructions on how to create new
>>>>>>>>>                         parameters for test suites in the Tester
>>>>>>>>>                         configuration, but there is nothing in
>>>>>>>>>     the
>>>>>>>>>                         user guide or in the Tester guide for
>>>>>>>>>     how to
>>>>>>>>>                         set the arguments for the parameters when
>>>>>>>>>                         running the test suite that I can
>>>>>>>>>     find. I'm
>>>>>>>>>                         not sure if I need to write my own Tester
>>>>>>>>>                         config, or if I should be setting
>>>>>>>>>     these in
>>>>>>>>>                         some other way.
>>>>>>>>>
>>>>>>>>>                         How should these values be set?
>>>>>>>>>
>>>>>>>>>                         I'm also not sure what environment
>>>>>>>>>                         variables/arguments are strictly
>>>>>>>>>     necessary or
>>>>>>>>>                         which are optional.
>>>>>>>>>
>>>>>>>>>                         Regards,
>>>>>>>>>                         Adam
>>>>>>>>>
>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>                         Senior Developer
>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>     ahassick@iol.unh.edu
>>>>>>>>>     <mailto:ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>     <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                     --                 *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>>     <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 --             *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>     <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>             --         *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>     <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --     *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>     <mailto:ahassick@iol.unh.edu>
>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>     <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>     <mailto:ahassick@iol.unh.edu>
>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>     <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 48828 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 29469 bytes --]

 > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.

Done. It works fine for me without any issues.

Have you noticed any pattern when it works or does not work?
May be it is a problem of not clean state after termination?
Does it work fine the first time after DUTs reboot?
How do you terminate testing? It should be done using Ctrl+C in terminal 
where you execute run.sh command.
  In this case it should shutdown gracefully and close all test agents 
and engine applications.

(I'm trying to understand why you've seen many test agent processes. It 
should not happen.)

Andrew.

On 8/25/23 17:41, Andrew Rybchenko wrote:
> On 8/25/23 17:06, Adam Hassick wrote:
>> Hi Andrew,
>>
>> Two of our systems (the Test Engine runner and the DUT host) are 
>> running Ubuntu 20.04 LTS, however this morning I noticed that the 
>> tester system (the one having issues) is running Ubuntu 22.04 LTS.
>> This could be the source of the problem. I encountered a dependency 
>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the 
>> system. Since the tester is also the host having connection issues, I 
>> will try downgrading that system to 20.04, and see if that changes 
>> anything.
>
> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu 
> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
> Right now Debian 11 is used for test engine in nightly regressions.
>
> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
>> I did try passing in the "--vg-rcf" argument to the run.sh script of 
>> the test suite after installing valgrind, but there was no additional 
>> output that I saw.
>
> Sorry, I should valgrind output should be in valgrind.te_rcf 
> (direction where you run test engine).
>
>>
>> I will try pulling in the changes you've pushed up, and will see if 
>> that fixes anything.
>>
>> Thanks,
>> Adam
>>
>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>     Hello Adam,
>>
>>     On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>     I'd like to try to repeat the problem locally. Which Linux
>>>     distro is running on test engine and agents?
>>>
>>>     In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>     patch in review to fix it, however, the behaviour is different in
>>>     this case, so it is unlike the same problem.
>>
>>     I've just published a new tag which fixes known test engine side
>>     problems on Debian 12 and Fedora 38.
>>
>>>
>>>     One more idea is to install valgrind on the test engine host and
>>>     run with option --vg-rcf to check if something weird is happening.
>>>
>>>     What I don't understand right now is why I see just one failed
>>>     attempt
>>>     to connect in your log.txt and then Logger shutdown after 9
>>>     minutes.
>>>
>>>     Andrew.
>>>
>>>     On 8/24/23 23:29, Adam Hassick wrote:
>>>>      > Is there any firewall in the network or on test hosts which
>>>>     could block incoming TCP connection to the port 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>     where you run test engine?
>>>>
>>>>     Our test engine host and the testbed are on the same subnet.
>>>>     The connection does work sometimes.
>>>>
>>>>      > If behaviour the same on the next try and you see that test
>>>>     agent is kept running, could you check using
>>>>      >
>>>>      > # netstat -tnlp
>>>>      >
>>>>      > that Test Agent is listening on the port and try to
>>>>     establish TCP connection from test agent using
>>>>      >
>>>>      > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>      >
>>>>      > and check if TCP connection could be established.
>>>>
>>>>     I was able to replicate the same behavior again, where it hangs
>>>>     while RCF is trying to start.
>>>>     Running this command, I see this in the output:
>>>>
>>>>     tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>     <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*        
>>>>           LISTEN      18599/ta
>>>>
>>>>     So it seems like it is listening on the correct port.
>>>>     Additionally, I was able to connect to the Tester machine from
>>>>     our Test Engine host using telnet. It printed the PID of the
>>>>     process once the connection was opened.
>>>>
>>>>     I tried running the "ta" application manually on the command
>>>>     line, and it didn't print anything at all.
>>>>     Maybe the issue is something on the Test Engine side.
>>>>
>>>>     On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>         Hi Adam,
>>>>
>>>>          > On the tester host (which appears to be the Peer agent),
>>>>     there
>>>>         are four processes that I see running, which look like the
>>>>     test
>>>>         agent processes.
>>>>
>>>>         Before the next try I'd recommend to kill these processes.
>>>>
>>>>         Is there any firewall in the network or on test hosts which
>>>>     could
>>>>         block incoming TCP connection to the port 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>         where you run test engine?
>>>>
>>>>         If behaviour the same on the next try and you see that test
>>>>     agent is
>>>>         kept running, could you check using
>>>>
>>>>         # netstat -tnlp
>>>>
>>>>         that Test Agent is listening on the port and try to
>>>>     establish TCP
>>>>         connection from test agent using
>>>>
>>>>         $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>         and check if TCP connection could be established.
>>>>
>>>>         Another idea is to login Tester under root as testing does,
>>>>     get
>>>>         start TA command from the log and try it by hands without
>>>>     -n and
>>>>         remove extra escaping.
>>>>
>>>>         # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>     /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>     host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>
>>>>         Hopefully in this case test agent directory remains in the
>>>>     /tmp and
>>>>         you don't need to copy it as testing does.
>>>>         May be output could shed some light on what's going on.
>>>>
>>>>         Andrew.
>>>>
>>>>         On 8/24/23 17:30, Adam Hassick wrote:
>>>>>         Hi Andrew,
>>>>>
>>>>>         This is the output that I see in the terminal when this
>>>>>     failure
>>>>>         occurs, after the test agent binaries build and the test
>>>>>     engine
>>>>>         starts:
>>>>>
>>>>>         Platform default build - pass
>>>>>         Simple RCF consistency check succeeded
>>>>>         --->>> Starting Logger...done
>>>>>         --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>>     timed
>>>>>         out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>         Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>         processes. I let it hang for around 9 minutes.
>>>>>
>>>>>         On the tester host (which appears to be the Peer agent),
>>>>>     there are
>>>>>         four processes that I see running, which look like the
>>>>>     test agent
>>>>>         processes.
>>>>>
>>>>>         ta.Peer is an empty file. I've attached the log.txt from
>>>>>     this run.
>>>>>
>>>>>          - Adam
>>>>>
>>>>>         On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>             Hi Adam,
>>>>>
>>>>>             Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>     double-checked
>>>>>             that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>             Description in in
>>>>>     doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>             says that copy_timeout is in seconds and
>>>>>     implementation in
>>>>>             lib/rcfunix/rcfunix.c passes the value to select()
>>>>>     tv_sec.
>>>>>             Theoretically select() could be interrupted by signal,
>>>>>     but I
>>>>>             think it is unlikely here.
>>>>>
>>>>>             I'm not sure that I understand what do you mean by RCF
>>>>>             connection timeout. Does it happen on TE startup when RCF
>>>>>             starts test agents. If so, TE_RCFUNIX_TIMEOUT could
>>>>>     help. Or
>>>>>             does it happen when tests are in progress, e.g. in the
>>>>>     middle
>>>>>             of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and
>>>>>     most
>>>>>             likely either host with test agent dies or test agent
>>>>>     itself
>>>>>             crashes. It would be easier for me if classify it if
>>>>>     you share
>>>>>             text log (log.txt, full or just corresponding fragment
>>>>>     with
>>>>>             some context). Also content of ta.DPDK or ta.Peer file
>>>>>             depending on which agent has problems could shed some
>>>>>     light.
>>>>>             Corresponding files contain stdout/stderr of test agents.
>>>>>
>>>>>             Andrew.
>>>>>
>>>>>             On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>             Hi Andrew,
>>>>>>
>>>>>>             I've set up a test rig repository here, and have created
>>>>>>             configurations for our development testbed based off
>>>>>>     of the
>>>>>>             examples.
>>>>>>             We've been able to get the test suite to run manually on
>>>>>>             Mellanox CX5 devices once.
>>>>>>             However, we are running into an issue where, when RCF
>>>>>>     starts,
>>>>>>             the RCF connection times out very frequently. We
>>>>>>     aren't sure
>>>>>>             why this is the case.
>>>>>>             It works sometimes, but most of the time when we try
>>>>>>     to run
>>>>>>             the test engine, it encounters this issue.
>>>>>>             I've tried changing the RCF port by setting
>>>>>>             "TE_RCF_PORT=<some port number>" and rebooting the
>>>>>>     testbed
>>>>>>             machines. Neither seems to fix the issue.
>>>>>>
>>>>>>             It also seems like the timeout takes far longer than 60
>>>>>>             seconds, even when running "export
>>>>>>     TE_RCFUNIX_TIMEOUT=60"
>>>>>>             before I try to run the test suite.
>>>>>>             I assume the unit for this variable is seconds?
>>>>>>
>>>>>>             Thanks,
>>>>>>             Adam
>>>>>>
>>>>>>             On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>             <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>     <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 Thanks, I've cloned the example repository and
>>>>>>     will start
>>>>>>                 setting up a configuration for our development
>>>>>>     testbed
>>>>>>                 today. I'll let you know if I run into any
>>>>>>     difficulties
>>>>>>                 or have any questions.
>>>>>>
>>>>>>                  - Adam
>>>>>>
>>>>>>                 On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     I've published
>>>>>>     https://github.com/ts-factory/ts-rigs-sample
>>>>>>     <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>     <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>                     Hopefully it will help to define your test
>>>>>>     rigs and
>>>>>>                     successfully run some tests manually. Feel
>>>>>>     free to
>>>>>>                     ask any questions and I'll answer here and
>>>>>>     try to
>>>>>>                     update documentation.
>>>>>>
>>>>>>                     Meanwhile I'll prepare missing bits for steps
>>>>>>     (2) and
>>>>>>                     (3).
>>>>>>                     Hopefully everything is in place for step
>>>>>>     (4), but we
>>>>>>                     need to make steps (2) and (3) first.
>>>>>>
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>                     Hi Adam,
>>>>>>>
>>>>>>>                     > I've conferred with the rest of the team,
>>>>>>>     and we
>>>>>>>                     think it would be best to move forward with
>>>>>>>     mainly
>>>>>>>                     option B.
>>>>>>>
>>>>>>>                     OK, I'll provide the sample on Monday for
>>>>>>>     you. It is
>>>>>>>                     almost ready right now, but I need to
>>>>>>>     double-check
>>>>>>>                     it before publishing.
>>>>>>>
>>>>>>>                     Regards,
>>>>>>>                     Andrew.
>>>>>>>
>>>>>>>                     On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>                     Hi Andrew,
>>>>>>>>
>>>>>>>>                     I'm adding the CI mailing list to this
>>>>>>>>                     conversation. Others in the community might
>>>>>>>>     find
>>>>>>>>                     this conversation valuable.
>>>>>>>>
>>>>>>>>                     We do want to run testing on a regular
>>>>>>>>     basis. The
>>>>>>>>                     Jenkins integration will be very useful for
>>>>>>>>     us, as
>>>>>>>>                     most of our CI is orchestrated by Jenkins.
>>>>>>>>                     I've conferred with the rest of the team,
>>>>>>>>     and we
>>>>>>>>                     think it would be best to move forward with
>>>>>>>>     mainly
>>>>>>>>                     option B.
>>>>>>>>                     If you would like to know anything about our
>>>>>>>>                     testbeds that would help you with creating an
>>>>>>>>                     example ts-rigs repo, I'd be happy to
>>>>>>>>     answer any
>>>>>>>>                     questions you have.
>>>>>>>>
>>>>>>>>                     We have multiple test rigs (we call these
>>>>>>>>                     "DUT-tester pairs") that we run our existing
>>>>>>>>                     hardware testing on, with differing network
>>>>>>>>                     hardware and CPU architecture. I figured
>>>>>>>>     this might
>>>>>>>>                     be an important detail.
>>>>>>>>
>>>>>>>>                     Thanks,
>>>>>>>>                     Adam
>>>>>>>>
>>>>>>>>                     On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>>     Rybchenko
>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                         Greatings Adam,
>>>>>>>>
>>>>>>>>                         I'm happy to hear that you're trying to
>>>>>>>>     bring
>>>>>>>>                         it up.
>>>>>>>>
>>>>>>>>                         As I understand the final goal is to
>>>>>>>>     run it on
>>>>>>>>                         regular basis. So, we need to make it
>>>>>>>>     properly
>>>>>>>>                         from the very beginning.
>>>>>>>>                         Bring up of all features consists of 4
>>>>>>>>     steps:
>>>>>>>>
>>>>>>>>                         1. Create site-specific repository (we
>>>>>>>>     call it
>>>>>>>>                         ts-rigs) which contains information
>>>>>>>>     about test
>>>>>>>>                         rigs and other site-specific
>>>>>>>>     information like
>>>>>>>>                         where to send mails, where to store
>>>>>>>>     logs etc.
>>>>>>>>                         It is required for manual execution as
>>>>>>>>     well,
>>>>>>>>                         since test rigs description is
>>>>>>>>     essential. I'll
>>>>>>>>                         return to the topic below.
>>>>>>>>
>>>>>>>>                         2. Setup logs storage for automated runs.
>>>>>>>>                         Basically it is a disk space plus
>>>>>>>>     apache2 web
>>>>>>>>                         server with few CGI scripts which help
>>>>>>>>     a lot to
>>>>>>>>                         save disk space.
>>>>>>>>
>>>>>>>>                         3. Setup Bublik web application which
>>>>>>>>     provides
>>>>>>>>                         web interface to view testing results.
>>>>>>>>     Same as
>>>>>>>>     https://ts-factory.io/bublik
>>>>>>>>     <https://ts-factory.io/bublik> <https://ts-factory.io/bublik>
>>>>>>>>
>>>>>>>>                         4. Setup Jenkins to run tests on
>>>>>>>>     regularly,
>>>>>>>>                         save logs in log storage (2) and import
>>>>>>>>     it to
>>>>>>>>                         bublik (3).
>>>>>>>>
>>>>>>>>                         Last few month we spent on our homework
>>>>>>>>     to make
>>>>>>>>                         it simpler to bring up automated execution
>>>>>>>>                         using Jenkins -
>>>>>>>>     https://github.com/ts-factory/te-jenkins
>>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>                         Corresponding bits in dpdk-ethdev-ts
>>>>>>>>     will be
>>>>>>>>                         available tomorrow.
>>>>>>>>
>>>>>>>>                         Let's return to the step (1).
>>>>>>>>
>>>>>>>>                         Unfortunately there is no publicly
>>>>>>>>     available
>>>>>>>>                         example of the ts-rigs repository since
>>>>>>>>                         sensitive site-specific information is
>>>>>>>>     located
>>>>>>>>                         there. But I'm ready to help you to
>>>>>>>>     create it
>>>>>>>>                         for UNH. I see two options here:
>>>>>>>>
>>>>>>>>                         (A) I'll ask questions and based on your
>>>>>>>>                         answers will create the first draft
>>>>>>>>     with my
>>>>>>>>                         comments.
>>>>>>>>
>>>>>>>>                         (B) I'll make a template/example
>>>>>>>>     ts-rigs repo,
>>>>>>>>                         publish it and you'll create UNH
>>>>>>>>     ts-rigs based
>>>>>>>>                         on it.
>>>>>>>>
>>>>>>>>                         Of course, I'll help to debug and
>>>>>>>>     finally bring
>>>>>>>>                         it up in any case.
>>>>>>>>
>>>>>>>>                         (A) is a bit simpler for me and you,
>>>>>>>>     but (B) is
>>>>>>>>                         a bit more generic and will help other
>>>>>>>>                         potential users to bring it up.
>>>>>>>>                         We can combine (A)+(B). I.e. start from
>>>>>>>>     (A).
>>>>>>>>                         What do you think?
>>>>>>>>
>>>>>>>>                         Thanks,
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/17/23 15:18, Konstantin Ushakov
>>>>>>>>     wrote:
>>>>>>>>>     Greetings Adam,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         Thanks for contacting us. I copy
>>>>>>>>>     Andrew who
>>>>>>>>>                         would be happy to help
>>>>>>>>>
>>>>>>>>>                         Thanks,
>>>>>>>>>                         Konstantin
>>>>>>>>>
>>>>>>>>>>     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>     <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>                         
>>>>>>>>>>                         Greetings Konstantin,
>>>>>>>>>>
>>>>>>>>>>                         I am in the process of setting up the
>>>>>>>>>>     DPDK
>>>>>>>>>>                         Poll Mode Driver test suite as an
>>>>>>>>>>     addition to
>>>>>>>>>>                         our testing coverage for DPDK at the
>>>>>>>>>>     UNH lab.
>>>>>>>>>>
>>>>>>>>>>                         I have some questions about how to
>>>>>>>>>>     set the
>>>>>>>>>>                         test suite arguments.
>>>>>>>>>>
>>>>>>>>>>                         I have been able to configure the
>>>>>>>>>>     Test Engine
>>>>>>>>>>                         to connect to the hosts in the
>>>>>>>>>>     testbed. The
>>>>>>>>>>                         RCF, Configurator, and Tester all
>>>>>>>>>>     begin to
>>>>>>>>>>                         run, however the prelude of the test
>>>>>>>>>>     suite
>>>>>>>>>>                         fails to run.
>>>>>>>>>>
>>>>>>>>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         The documentation mentions that there
>>>>>>>>>>     are
>>>>>>>>>>                         several test parameters for the test
>>>>>>>>>>     suite,
>>>>>>>>>>                         like for the IUT test link MAC, etc.
>>>>>>>>>>     These
>>>>>>>>>>                         seem like they would need to be set
>>>>>>>>>>     somewhere
>>>>>>>>>>                         to run many of the tests.
>>>>>>>>>>
>>>>>>>>>>                         I see in the Test Engine
>>>>>>>>>>     documentation, there
>>>>>>>>>>                         are instructions on how to create new
>>>>>>>>>>                         parameters for test suites in the Tester
>>>>>>>>>>                         configuration, but there is nothing
>>>>>>>>>>     in the
>>>>>>>>>>                         user guide or in the Tester guide for
>>>>>>>>>>     how to
>>>>>>>>>>                         set the arguments for the parameters
>>>>>>>>>>     when
>>>>>>>>>>                         running the test suite that I can
>>>>>>>>>>     find. I'm
>>>>>>>>>>                         not sure if I need to write my own
>>>>>>>>>>     Tester
>>>>>>>>>>                         config, or if I should be setting
>>>>>>>>>>     these in
>>>>>>>>>>                         some other way.
>>>>>>>>>>
>>>>>>>>>>                         How should these values be set?
>>>>>>>>>>
>>>>>>>>>>                         I'm also not sure what environment
>>>>>>>>>>                         variables/arguments are strictly
>>>>>>>>>>     necessary or
>>>>>>>>>>                         which are optional.
>>>>>>>>>>
>>>>>>>>>>                         Regards,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>     ahassick@iol.unh.edu
>>>>>>>>>>     <mailto:ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>     <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                     --                 *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>>>     <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --             *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>     <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>             --         *Adam Hassick*
>>>>>>             Senior Developer
>>>>>>             UNH InterOperability Lab
>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>     <https://www.iol.unh.edu/>
>>>>>>             +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>         --     *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>     <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>     -- 
>>>>     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>     <mailto:ahassick@iol.unh.edu>
>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>     <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>>
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 51855 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 23185 bytes --]

Hi Andrew,

We have yet to notice a distinct pattern with the failures. Sometimes, the
RCF will start and connect without issue a few times in a row before
failing to connect again. Once the issue begins to occur, neither rebooting
all of the hosts (test engine VM, tester, IUT) or deleting all of the build
directories (suites, agents, inst) and rebooting the hosts afterward
resolves the issue. When it begins working again seems very arbitrary to us.

I do usually try to terminate the test engine with Ctrl+C, but when it
hangs while trying to start RCF, that does not work.

Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
connection to the test bed hosts? And if so, is there a way to force it to
use IPv4?

 - Adam

On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
> Done. It works fine for me without any issues.
>
> Have you noticed any pattern when it works or does not work?
> May be it is a problem of not clean state after termination?
> Does it work fine the first time after DUTs reboot?
> How do you terminate testing? It should be done using Ctrl+C in terminal
> where you execute run.sh command.
>  In this case it should shutdown gracefully and close all test agents and
> engine applications.
>
> (I'm trying to understand why you've seen many test agent processes. It
> should not happen.)
>
> Andrew.
>
> On 8/25/23 17:41, Andrew Rybchenko wrote:
>
> On 8/25/23 17:06, Adam Hassick wrote:
>
> Hi Andrew,
>
> Two of our systems (the Test Engine runner and the DUT host) are running
> Ubuntu 20.04 LTS, however this morning I noticed that the tester system
> (the one having issues) is running Ubuntu 22.04 LTS.
> This could be the source of the problem. I encountered a dependency issue
> trying to run the Test Engine on 22.04 LTS, so I downgraded the system.
> Since the tester is also the host having connection issues, I will try
> downgrading that system to 20.04, and see if that changes anything.
>
>
> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
> Right now Debian 11 is used for test engine in nightly regressions.
>
> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
> I did try passing in the "--vg-rcf" argument to the run.sh script of the
> test suite after installing valgrind, but there was no additional output
> that I saw.
>
>
> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
> where you run test engine).
>
>
> I will try pulling in the changes you've pushed up, and will see if that
> fixes anything.
>
> Thanks,
> Adam
>
> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hello Adam,
>>
>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>
>> I'd like to try to repeat the problem locally. Which Linux distro is
>> running on test engine and agents?
>>
>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>> patch in review to fix it, however, the behaviour is different in
>> this case, so it is unlike the same problem.
>>
>>
>> I've just published a new tag which fixes known test engine side problems
>> on Debian 12 and Fedora 38.
>>
>>
>> One more idea is to install valgrind on the test engine host and
>> run with option --vg-rcf to check if something weird is happening.
>>
>> What I don't understand right now is why I see just one failed attempt
>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>
>> Andrew.
>>
>> On 8/24/23 23:29, Adam Hassick wrote:
>>
>>  > Is there any firewall in the network or on test hosts which could
>> block incoming TCP connection to the port 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>> you run test engine?
>>
>> Our test engine host and the testbed are on the same subnet. The
>> connection does work sometimes.
>>
>>  > If behaviour the same on the next try and you see that test agent is
>> kept running, could you check using
>>  >
>>  > # netstat -tnlp
>>  >
>>  > that Test Agent is listening on the port and try to establish TCP
>> connection from test agent using
>>  >
>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>  >
>>  > and check if TCP connection could be established.
>>
>> I was able to replicate the same behavior again, where it hangs while RCF
>> is trying to start.
>> Running this command, I see this in the output:
>>
>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>  18599/ta
>>
>> So it seems like it is listening on the correct port.
>> Additionally, I was able to connect to the Tester machine from our Test
>> Engine host using telnet. It printed the PID of the process once the
>> connection was opened.
>>
>> I tried running the "ta" application manually on the command line, and it
>> didn't print anything at all.
>> Maybe the issue is something on the Test Engine side.
>>
>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>     Hi Adam,
>>
>>      > On the tester host (which appears to be the Peer agent), there
>>     are four processes that I see running, which look like the test
>>     agent processes.
>>
>>     Before the next try I'd recommend to kill these processes.
>>
>>     Is there any firewall in the network or on test hosts which could
>>     block incoming TCP connection to the port 23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>     where you run test engine?
>>
>>     If behaviour the same on the next try and you see that test agent is
>>     kept running, could you check using
>>
>>     # netstat -tnlp
>>
>>     that Test Agent is listening on the port and try to establish TCP
>>     connection from test agent using
>>
>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     and check if TCP connection could be established.
>>
>>     Another idea is to login Tester under root as testing does, get
>>     start TA command from the log and try it by hands without -n and
>>     remove extra escaping.
>>
>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>
>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>
>>     Hopefully in this case test agent directory remains in the /tmp and
>>     you don't need to copy it as testing does.
>>     May be output could shed some light on what's going on.
>>
>>     Andrew.
>>
>>     On 8/24/23 17:30, Adam Hassick wrote:
>>
>>     Hi Andrew,
>>
>>     This is the output that I see in the terminal when this failure
>>     occurs, after the test agent binaries build and the test engine
>>     starts:
>>
>>     Platform default build - pass
>>     Simple RCF consistency check succeeded
>>     --->>> Starting Logger...done
>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>     processes. I let it hang for around 9 minutes.
>>
>>     On the tester host (which appears to be the Peer agent), there are
>>     four processes that I see running, which look like the test agent
>>     processes.
>>
>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>
>>      - Adam
>>
>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru
>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>         Hi Adam,
>>
>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>         says that copy_timeout is in seconds and implementation in
>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>         Theoretically select() could be interrupted by signal, but I
>>         think it is unlikely here.
>>
>>         I'm not sure that I understand what do you mean by RCF
>>         connection timeout. Does it happen on TE startup when RCF
>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>         does it happen when tests are in progress, e.g. in the middle
>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>         likely either host with test agent dies or test agent itself
>>         crashes. It would be easier for me if classify it if you share
>>         text log (log.txt, full or just corresponding fragment with
>>         some context). Also content of ta.DPDK or ta.Peer file
>>         depending on which agent has problems could shed some light.
>>         Corresponding files contain stdout/stderr of test agents.
>>
>>         Andrew.
>>
>>         On 8/23/23 17:45, Adam Hassick wrote:
>>
>>         Hi Andrew,
>>
>>         I've set up a test rig repository here, and have created
>>         configurations for our development testbed based off of the
>>         examples.
>>         We've been able to get the test suite to run manually on
>>         Mellanox CX5 devices once.
>>         However, we are running into an issue where, when RCF starts,
>>         the RCF connection times out very frequently. We aren't sure
>>         why this is the case.
>>         It works sometimes, but most of the time when we try to run
>>         the test engine, it encounters this issue.
>>         I've tried changing the RCF port by setting
>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>         machines. Neither seems to fix the issue.
>>
>>         It also seems like the timeout takes far longer than 60
>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>         before I try to run the test suite.
>>         I assume the unit for this variable is seconds?
>>
>>         Thanks,
>>         Adam
>>
>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>> wrote:
>>
>>             Hi Andrew,
>>
>>             Thanks, I've cloned the example repository and will start
>>             setting up a configuration for our development testbed
>>             today. I'll let you know if I run into any difficulties
>>             or have any questions.
>>
>>              - Adam
>>
>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>             <andrew.rybchenko@oktetlabs.ru
>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>                 Hi Adam,
>>
>>                 I've published
>>                 https://github.com/ts-factory/ts-rigs-sample
>>                 <https://github.com/ts-factory/ts-rigs-sample>
>> <https://github.com/ts-factory/ts-rigs-sample>.
>>                 Hopefully it will help to define your test rigs and
>>                 successfully run some tests manually. Feel free to
>>                 ask any questions and I'll answer here and try to
>>                 update documentation.
>>
>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>                 (3).
>>                 Hopefully everything is in place for step (4), but we
>>                 need to make steps (2) and (3) first.
>>
>>                 Andrew.
>>
>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>
>>                 Hi Adam,
>>
>>                 > I've conferred with the rest of the team, and we
>>                 think it would be best to move forward with mainly
>>                 option B.
>>
>>                 OK, I'll provide the sample on Monday for you. It is
>>                 almost ready right now, but I need to double-check
>>                 it before publishing.
>>
>>                 Regards,
>>                 Andrew.
>>
>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>
>>                 Hi Andrew,
>>
>>                 I'm adding the CI mailing list to this
>>                 conversation. Others in the community might find
>>                 this conversation valuable.
>>
>>                 We do want to run testing on a regular basis. The
>>                 Jenkins integration will be very useful for us, as
>>                 most of our CI is orchestrated by Jenkins.
>>                 I've conferred with the rest of the team, and we
>>                 think it would be best to move forward with mainly
>>                 option B.
>>                 If you would like to know anything about our
>>                 testbeds that would help you with creating an
>>                 example ts-rigs repo, I'd be happy to answer any
>>                 questions you have.
>>
>>                 We have multiple test rigs (we call these
>>                 "DUT-tester pairs") that we run our existing
>>                 hardware testing on, with differing network
>>                 hardware and CPU architecture. I figured this might
>>                 be an important detail.
>>
>>                 Thanks,
>>                 Adam
>>
>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>                 <andrew.rybchenko@oktetlabs.ru
>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>                     Greatings Adam,
>>
>>                     I'm happy to hear that you're trying to bring
>>                     it up.
>>
>>                     As I understand the final goal is to run it on
>>                     regular basis. So, we need to make it properly
>>                     from the very beginning.
>>                     Bring up of all features consists of 4 steps:
>>
>>                     1. Create site-specific repository (we call it
>>                     ts-rigs) which contains information about test
>>                     rigs and other site-specific information like
>>                     where to send mails, where to store logs etc.
>>                     It is required for manual execution as well,
>>                     since test rigs description is essential. I'll
>>                     return to the topic below.
>>
>>                     2. Setup logs storage for automated runs.
>>                     Basically it is a disk space plus apache2 web
>>                     server with few CGI scripts which help a lot to
>>                     save disk space.
>>
>>                     3. Setup Bublik web application which provides
>>                     web interface to view testing results. Same as
>>                     https://ts-factory.io/bublik
>>                     <https://ts-factory.io/bublik>
>> <https://ts-factory.io/bublik>
>>
>>                     4. Setup Jenkins to run tests on regularly,
>>                     save logs in log storage (2) and import it to
>>                     bublik (3).
>>
>>                     Last few month we spent on our homework to make
>>                     it simpler to bring up automated execution
>>                     using Jenkins -
>>                     https://github.com/ts-factory/te-jenkins
>>                     <https://github.com/ts-factory/te-jenkins>
>> <https://github.com/ts-factory/te-jenkins>
>>                     Corresponding bits in dpdk-ethdev-ts will be
>>                     available tomorrow.
>>
>>                     Let's return to the step (1).
>>
>>                     Unfortunately there is no publicly available
>>                     example of the ts-rigs repository since
>>                     sensitive site-specific information is located
>>                     there. But I'm ready to help you to create it
>>                     for UNH. I see two options here:
>>
>>                     (A) I'll ask questions and based on your
>>                     answers will create the first draft with my
>>                     comments.
>>
>>                     (B) I'll make a template/example ts-rigs repo,
>>                     publish it and you'll create UNH ts-rigs based
>>                     on it.
>>
>>                     Of course, I'll help to debug and finally bring
>>                     it up in any case.
>>
>>                     (A) is a bit simpler for me and you, but (B) is
>>                     a bit more generic and will help other
>>                     potential users to bring it up.
>>                     We can combine (A)+(B). I.e. start from (A).
>>                     What do you think?
>>
>>                     Thanks,
>>                     Andrew.
>>
>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>
>>                     Greetings Adam,
>>
>>
>>                     Thanks for contacting us. I copy Andrew who
>>                     would be happy to help
>>
>>                     Thanks,
>>                     Konstantin
>>
>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>> wrote:
>>
>>                     
>>                     Greetings Konstantin,
>>
>>                     I am in the process of setting up the DPDK
>>                     Poll Mode Driver test suite as an addition to
>>                     our testing coverage for DPDK at the UNH lab.
>>
>>                     I have some questions about how to set the
>>                     test suite arguments.
>>
>>                     I have been able to configure the Test Engine
>>                     to connect to the hosts in the testbed. The
>>                     RCF, Configurator, and Tester all begin to
>>                     run, however the prelude of the test suite
>>                     fails to run.
>>
>>
>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>
>>                     The documentation mentions that there are
>>                     several test parameters for the test suite,
>>                     like for the IUT test link MAC, etc. These
>>                     seem like they would need to be set somewhere
>>                     to run many of the tests.
>>
>>                     I see in the Test Engine documentation, there
>>                     are instructions on how to create new
>>                     parameters for test suites in the Tester
>>                     configuration, but there is nothing in the
>>                     user guide or in the Tester guide for how to
>>                     set the arguments for the parameters when
>>                     running the test suite that I can find. I'm
>>                     not sure if I need to write my own Tester
>>                     config, or if I should be setting these in
>>                     some other way.
>>
>>                     How should these values be set?
>>
>>                     I'm also not sure what environment
>>                     variables/arguments are strictly necessary or
>>                     which are optional.
>>
>>                     Regards,
>>                     Adam
>>
>>                     --                     *Adam Hassick*
>>                     Senior Developer
>>                     UNH InterOperability Lab
>>                     ahassick@iol.unh.edu
>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>                     iol.unh.edu <https://www.iol.unh.edu/>
>> <https://www.iol.unh.edu/>
>>                     +1 (603) 475-8248
>>
>>
>>
>>
>>                 --                 *Adam Hassick*
>>                 Senior Developer
>>                 UNH InterOperability Lab
>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>                 iol.unh.edu <https://www.iol.unh.edu/>
>> <https://www.iol.unh.edu/>
>>                 +1 (603) 475-8248
>>
>>
>>
>>
>>
>>             --             *Adam Hassick*
>>             Senior Developer
>>             UNH InterOperability Lab
>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>             iol.unh.edu <https://www.iol.unh.edu/>
>> <https://www.iol.unh.edu/>
>>             +1 (603) 475-8248
>>
>>
>>
>>         --         *Adam Hassick*
>>         Senior Developer
>>         UNH InterOperability Lab
>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>         iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>         +1 (603) 475-8248
>>
>>
>>
>>
>>     --     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>>
>>
>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 49907 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 34391 bytes --]

Hi Adam,

 > Does the test engine prefer to use IPv6 over IPv4 for initiating the 
RCF connection to the test bed hosts? And if so, is there a way to force 
it to use IPv4?

Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your 
case, I guess it is the root cause of the problem.
Of course, it is TE problem since I see really weird code in 
lib/comm_net_engine/comm_net_engine.c line 135.

I've pushed fix to the branch user/arybchik/fix_ipv4_only in 
ts-factory/test-environment repository. Please, try.

It is late night fix with minimal testing and no review. I'll pass it 
through review process tomorrow and
hopefully it will be released in one-two days.

Andrew.

On 8/28/23 18:02, Adam Hassick wrote:
> Hi Andrew,
>
> We have yet to notice a distinct pattern with the failures. Sometimes, 
> the RCF will start and connect without issue a few times in a row 
> before failing to connect again. Once the issue begins to occur, 
> neither rebooting all of the hosts (test engine VM, tester, IUT) or 
> deleting all of the build directories (suites, agents, inst) and 
> rebooting the hosts afterward resolves the issue. When it begins 
> working again seems very arbitrary to us.
>
> I do usually try to terminate the test engine with Ctrl+C, but when it 
> hangs while trying to start RCF, that does not work.
>
> Does the test engine prefer to use IPv6 over IPv4 for initiating the 
> RCF connection to the test bed hosts? And if so, is there a way to 
> force it to use IPv4?
>
>  - Adam
>
> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
>     Done. It works fine for me without any issues.
>
>     Have you noticed any pattern when it works or does not work?
>     May be it is a problem of not clean state after termination?
>     Does it work fine the first time after DUTs reboot?
>     How do you terminate testing? It should be done using Ctrl+C in
>     terminal where you execute run.sh command.
>      In this case it should shutdown gracefully and close all test
>     agents and engine applications.
>
>     (I'm trying to understand why you've seen many test agent
>     processes. It should not happen.)
>
>     Andrew.
>
>     On 8/25/23 17:41, Andrew Rybchenko wrote:
>>     On 8/25/23 17:06, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     Two of our systems (the Test Engine runner and the DUT host) are
>>>     running Ubuntu 20.04 LTS, however this morning I noticed that
>>>     the tester system (the one having issues) is running Ubuntu
>>>     22.04 LTS.
>>>     This could be the source of the problem. I encountered a
>>>     dependency issue trying to run the Test Engine on 22.04 LTS, so
>>>     I downgraded the system. Since the tester is also the host
>>>     having connection issues, I will try downgrading that system to
>>>     20.04, and see if that changes anything.
>>
>>     Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>     Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38
>>     every night.
>>     Right now Debian 11 is used for test engine in nightly regressions.
>>
>>     I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>>>     I did try passing in the "--vg-rcf" argument to the run.sh
>>>     script of the test suite after installing valgrind, but there
>>>     was no additional output that I saw.
>>
>>     Sorry, I should valgrind output should be in valgrind.te_rcf
>>     (direction where you run test engine).
>>
>>>
>>>     I will try pulling in the changes you've pushed up, and will see
>>>     if that fixes anything.
>>>
>>>     Thanks,
>>>     Adam
>>>
>>>     On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>         Hello Adam,
>>>
>>>         On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>         I'd like to try to repeat the problem locally. Which Linux
>>>>         distro is running on test engine and agents?
>>>>
>>>>         In fact I know one problem with Debian 12 and Fedora 38 and
>>>>         we have
>>>>         patch in review to fix it, however, the behaviour is
>>>>         different in
>>>>         this case, so it is unlike the same problem.
>>>
>>>         I've just published a new tag which fixes known test engine
>>>         side problems on Debian 12 and Fedora 38.
>>>
>>>>
>>>>         One more idea is to install valgrind on the test engine
>>>>         host and
>>>>         run with option --vg-rcf to check if something weird is
>>>>         happening.
>>>>
>>>>         What I don't understand right now is why I see just one
>>>>         failed attempt
>>>>         to connect in your log.txt and then Logger shutdown after 9
>>>>         minutes.
>>>>
>>>>         Andrew.
>>>>
>>>>         On 8/24/23 23:29, Adam Hassick wrote:
>>>>>          > Is there any firewall in the network or on test hosts
>>>>>         which could block incoming TCP connection to the port
>>>>>         23571 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the
>>>>>         host where you run test engine?
>>>>>
>>>>>         Our test engine host and the testbed are on the same
>>>>>         subnet. The connection does work sometimes.
>>>>>
>>>>>          > If behaviour the same on the next try and you see that
>>>>>         test agent is kept running, could you check using
>>>>>          >
>>>>>          > # netstat -tnlp
>>>>>          >
>>>>>          > that Test Agent is listening on the port and try to
>>>>>         establish TCP connection from test agent using
>>>>>          >
>>>>>          > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>          >
>>>>>          > and check if TCP connection could be established.
>>>>>
>>>>>         I was able to replicate the same behavior again, where it
>>>>>         hangs while RCF is trying to start.
>>>>>         Running this command, I see this in the output:
>>>>>
>>>>>         tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>         <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*  
>>>>>                     LISTEN      18599/ta
>>>>>
>>>>>         So it seems like it is listening on the correct port.
>>>>>         Additionally, I was able to connect to the Tester machine
>>>>>         from our Test Engine host using telnet. It printed the PID
>>>>>         of the process once the connection was opened.
>>>>>
>>>>>         I tried running the "ta" application manually on the
>>>>>         command line, and it didn't print anything at all.
>>>>>         Maybe the issue is something on the Test Engine side.
>>>>>
>>>>>         On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>             Hi Adam,
>>>>>
>>>>>              > On the tester host (which appears to be the Peer
>>>>>         agent), there
>>>>>             are four processes that I see running, which look like
>>>>>         the test
>>>>>             agent processes.
>>>>>
>>>>>             Before the next try I'd recommend to kill these
>>>>>         processes.
>>>>>
>>>>>             Is there any firewall in the network or on test hosts
>>>>>         which could
>>>>>             block incoming TCP connection to the port 23571
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the
>>>>>         host
>>>>>             where you run test engine?
>>>>>
>>>>>             If behaviour the same on the next try and you see that
>>>>>         test agent is
>>>>>             kept running, could you check using
>>>>>
>>>>>             # netstat -tnlp
>>>>>
>>>>>             that Test Agent is listening on the port and try to
>>>>>         establish TCP
>>>>>             connection from test agent using
>>>>>
>>>>>             $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>             and check if TCP connection could be established.
>>>>>
>>>>>             Another idea is to login Tester under root as testing
>>>>>         does, get
>>>>>             start TA command from the log and try it by hands
>>>>>         without -n and
>>>>>             remove extra escaping.
>>>>>
>>>>>             # sudo
>>>>>         PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>         LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>         /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>         host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>
>>>>>             Hopefully in this case test agent directory remains in
>>>>>         the /tmp and
>>>>>             you don't need to copy it as testing does.
>>>>>             May be output could shed some light on what's going on.
>>>>>
>>>>>             Andrew.
>>>>>
>>>>>             On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>             Hi Andrew,
>>>>>>
>>>>>>             This is the output that I see in the terminal when
>>>>>>         this failure
>>>>>>             occurs, after the test agent binaries build and the
>>>>>>         test engine
>>>>>>             starts:
>>>>>>
>>>>>>             Platform default build - pass
>>>>>>             Simple RCF consistency check succeeded
>>>>>>             --->>> Starting Logger...done
>>>>>>             --->>> Starting RCF...rcf_net_engine_connect():
>>>>>>         Connection timed
>>>>>>             out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>
>>>>>>             Then, it hangs here until I kill the "te_rcf" and
>>>>>>         "te_tee"
>>>>>>             processes. I let it hang for around 9 minutes.
>>>>>>
>>>>>>             On the tester host (which appears to be the Peer
>>>>>>         agent), there are
>>>>>>             four processes that I see running, which look like
>>>>>>         the test agent
>>>>>>             processes.
>>>>>>
>>>>>>             ta.Peer is an empty file. I've attached the log.txt
>>>>>>         from this run.
>>>>>>
>>>>>>              - Adam
>>>>>>
>>>>>>             On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                 Hi Adam,
>>>>>>
>>>>>>                 Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>         double-checked
>>>>>>                 that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>                 Description in in
>>>>>>         doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>                 says that copy_timeout is in seconds and
>>>>>>         implementation in
>>>>>>                 lib/rcfunix/rcfunix.c passes the value to
>>>>>>         select() tv_sec.
>>>>>>                 Theoretically select() could be interrupted by
>>>>>>         signal, but I
>>>>>>                 think it is unlikely here.
>>>>>>
>>>>>>                 I'm not sure that I understand what do you mean
>>>>>>         by RCF
>>>>>>                 connection timeout. Does it happen on TE startup
>>>>>>         when RCF
>>>>>>                 starts test agents. If so, TE_RCFUNIX_TIMEOUT
>>>>>>         could help. Or
>>>>>>                 does it happen when tests are in progress, e.g.
>>>>>>         in the middle
>>>>>>                 of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated
>>>>>>         and most
>>>>>>                 likely either host with test agent dies or test
>>>>>>         agent itself
>>>>>>                 crashes. It would be easier for me if classify it
>>>>>>         if you share
>>>>>>                 text log (log.txt, full or just corresponding
>>>>>>         fragment with
>>>>>>                 some context). Also content of ta.DPDK or ta.Peer
>>>>>>         file
>>>>>>                 depending on which agent has problems could shed
>>>>>>         some light.
>>>>>>                 Corresponding files contain stdout/stderr of test
>>>>>>         agents.
>>>>>>
>>>>>>                 Andrew.
>>>>>>
>>>>>>                 On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>                 Hi Andrew,
>>>>>>>
>>>>>>>                 I've set up a test rig repository here, and have
>>>>>>>         created
>>>>>>>                 configurations for our development testbed based
>>>>>>>         off of the
>>>>>>>                 examples.
>>>>>>>                 We've been able to get the test suite to run
>>>>>>>         manually on
>>>>>>>                 Mellanox CX5 devices once.
>>>>>>>                 However, we are running into an issue where,
>>>>>>>         when RCF starts,
>>>>>>>                 the RCF connection times out very frequently. We
>>>>>>>         aren't sure
>>>>>>>                 why this is the case.
>>>>>>>                 It works sometimes, but most of the time when we
>>>>>>>         try to run
>>>>>>>                 the test engine, it encounters this issue.
>>>>>>>                 I've tried changing the RCF port by setting
>>>>>>>                 "TE_RCF_PORT=<some port number>" and rebooting
>>>>>>>         the testbed
>>>>>>>                 machines. Neither seems to fix the issue.
>>>>>>>
>>>>>>>                 It also seems like the timeout takes far longer
>>>>>>>         than 60
>>>>>>>                 seconds, even when running "export
>>>>>>>         TE_RCFUNIX_TIMEOUT=60"
>>>>>>>                 before I try to run the test suite.
>>>>>>>                 I assume the unit for this variable is seconds?
>>>>>>>
>>>>>>>                 Thanks,
>>>>>>>                 Adam
>>>>>>>
>>>>>>>                 On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>                 <ahassick@iol.unh.edu
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>
>>>>>>>                     Hi Andrew,
>>>>>>>
>>>>>>>                     Thanks, I've cloned the example repository
>>>>>>>         and will start
>>>>>>>                     setting up a configuration for our
>>>>>>>         development testbed
>>>>>>>                     today. I'll let you know if I run into any
>>>>>>>         difficulties
>>>>>>>                     or have any questions.
>>>>>>>
>>>>>>>                      - Adam
>>>>>>>
>>>>>>>                     On Sun, Aug 20, 2023 at 4:40 AM Andrew
>>>>>>>         Rybchenko
>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                         Hi Adam,
>>>>>>>
>>>>>>>                         I've published
>>>>>>>         https://github.com/ts-factory/ts-rigs-sample
>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>                         Hopefully it will help to define your
>>>>>>>         test rigs and
>>>>>>>                         successfully run some tests manually.
>>>>>>>         Feel free to
>>>>>>>                         ask any questions and I'll answer here
>>>>>>>         and try to
>>>>>>>                         update documentation.
>>>>>>>
>>>>>>>                         Meanwhile I'll prepare missing bits for
>>>>>>>         steps (2) and
>>>>>>>                         (3).
>>>>>>>                         Hopefully everything is in place for
>>>>>>>         step (4), but we
>>>>>>>                         need to make steps (2) and (3) first.
>>>>>>>
>>>>>>>                         Andrew.
>>>>>>>
>>>>>>>                         On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>                         Hi Adam,
>>>>>>>>
>>>>>>>>                         > I've conferred with the rest of the
>>>>>>>>         team, and we
>>>>>>>>                         think it would be best to move forward
>>>>>>>>         with mainly
>>>>>>>>                         option B.
>>>>>>>>
>>>>>>>>                         OK, I'll provide the sample on Monday
>>>>>>>>         for you. It is
>>>>>>>>                         almost ready right now, but I need to
>>>>>>>>         double-check
>>>>>>>>                         it before publishing.
>>>>>>>>
>>>>>>>>                         Regards,
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>         Hi Andrew,
>>>>>>>>>
>>>>>>>>>                         I'm adding the CI mailing list to this
>>>>>>>>>                         conversation. Others in the community
>>>>>>>>>         might find
>>>>>>>>>                         this conversation valuable.
>>>>>>>>>
>>>>>>>>>                         We do want to run testing on a regular
>>>>>>>>>         basis. The
>>>>>>>>>                         Jenkins integration will be very
>>>>>>>>>         useful for us, as
>>>>>>>>>                         most of our CI is orchestrated by
>>>>>>>>>         Jenkins.
>>>>>>>>>                         I've conferred with the rest of the
>>>>>>>>>         team, and we
>>>>>>>>>                         think it would be best to move forward
>>>>>>>>>         with mainly
>>>>>>>>>                         option B.
>>>>>>>>>                         If you would like to know anything
>>>>>>>>>         about our
>>>>>>>>>                         testbeds that would help you with
>>>>>>>>>         creating an
>>>>>>>>>                         example ts-rigs repo, I'd be happy to
>>>>>>>>>         answer any
>>>>>>>>>                         questions you have.
>>>>>>>>>
>>>>>>>>>                         We have multiple test rigs (we call these
>>>>>>>>>                         "DUT-tester pairs") that we run our
>>>>>>>>>         existing
>>>>>>>>>                         hardware testing on, with differing
>>>>>>>>>         network
>>>>>>>>>                         hardware and CPU architecture. I
>>>>>>>>>         figured this might
>>>>>>>>>                         be an important detail.
>>>>>>>>>
>>>>>>>>>                         Thanks,
>>>>>>>>>                         Adam
>>>>>>>>>
>>>>>>>>>                         On Thu, Aug 17, 2023 at 11:44 AM
>>>>>>>>>         Andrew Rybchenko
>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                             Greatings Adam,
>>>>>>>>>
>>>>>>>>>                             I'm happy to hear that you're
>>>>>>>>>         trying to bring
>>>>>>>>>                             it up.
>>>>>>>>>
>>>>>>>>>                             As I understand the final goal is
>>>>>>>>>         to run it on
>>>>>>>>>                             regular basis. So, we need to make
>>>>>>>>>         it properly
>>>>>>>>>                             from the very beginning.
>>>>>>>>>                             Bring up of all features consists
>>>>>>>>>         of 4 steps:
>>>>>>>>>
>>>>>>>>>                             1. Create site-specific repository
>>>>>>>>>         (we call it
>>>>>>>>>                             ts-rigs) which contains
>>>>>>>>>         information about test
>>>>>>>>>                             rigs and other site-specific
>>>>>>>>>         information like
>>>>>>>>>                             where to send mails, where to
>>>>>>>>>         store logs etc.
>>>>>>>>>                             It is required for manual
>>>>>>>>>         execution as well,
>>>>>>>>>                             since test rigs description is
>>>>>>>>>         essential. I'll
>>>>>>>>>                             return to the topic below.
>>>>>>>>>
>>>>>>>>>                             2. Setup logs storage for
>>>>>>>>>         automated runs.
>>>>>>>>>                             Basically it is a disk space plus
>>>>>>>>>         apache2 web
>>>>>>>>>                             server with few CGI scripts which
>>>>>>>>>         help a lot to
>>>>>>>>>                             save disk space.
>>>>>>>>>
>>>>>>>>>                             3. Setup Bublik web application
>>>>>>>>>         which provides
>>>>>>>>>                             web interface to view testing
>>>>>>>>>         results. Same as
>>>>>>>>>         https://ts-factory.io/bublik
>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>
>>>>>>>>>                             4. Setup Jenkins to run tests on
>>>>>>>>>         regularly,
>>>>>>>>>                             save logs in log storage (2) and
>>>>>>>>>         import it to
>>>>>>>>>                             bublik (3).
>>>>>>>>>
>>>>>>>>>                             Last few month we spent on our
>>>>>>>>>         homework to make
>>>>>>>>>                             it simpler to bring up automated
>>>>>>>>>         execution
>>>>>>>>>                             using Jenkins -
>>>>>>>>>         https://github.com/ts-factory/te-jenkins
>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>                             Corresponding bits in
>>>>>>>>>         dpdk-ethdev-ts will be
>>>>>>>>>                             available tomorrow.
>>>>>>>>>
>>>>>>>>>                             Let's return to the step (1).
>>>>>>>>>
>>>>>>>>>                             Unfortunately there is no publicly
>>>>>>>>>         available
>>>>>>>>>                             example of the ts-rigs repository
>>>>>>>>>         since
>>>>>>>>>                             sensitive site-specific
>>>>>>>>>         information is located
>>>>>>>>>                             there. But I'm ready to help you
>>>>>>>>>         to create it
>>>>>>>>>                             for UNH. I see two options here:
>>>>>>>>>
>>>>>>>>>                             (A) I'll ask questions and based
>>>>>>>>>         on your
>>>>>>>>>                             answers will create the first
>>>>>>>>>         draft with my
>>>>>>>>>                             comments.
>>>>>>>>>
>>>>>>>>>                             (B) I'll make a template/example
>>>>>>>>>         ts-rigs repo,
>>>>>>>>>                             publish it and you'll create UNH
>>>>>>>>>         ts-rigs based
>>>>>>>>>                             on it.
>>>>>>>>>
>>>>>>>>>                             Of course, I'll help to debug and
>>>>>>>>>         finally bring
>>>>>>>>>                             it up in any case.
>>>>>>>>>
>>>>>>>>>                             (A) is a bit simpler for me and
>>>>>>>>>         you, but (B) is
>>>>>>>>>                             a bit more generic and will help
>>>>>>>>>         other
>>>>>>>>>                             potential users to bring it up.
>>>>>>>>>                             We can combine (A)+(B). I.e. start
>>>>>>>>>         from (A).
>>>>>>>>>                             What do you think?
>>>>>>>>>
>>>>>>>>>                             Thanks,
>>>>>>>>>                             Andrew.
>>>>>>>>>
>>>>>>>>>                             On 8/17/23 15:18, Konstantin
>>>>>>>>>         Ushakov wrote:
>>>>>>>>>>         Greetings Adam,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             Thanks for contacting us. I copy
>>>>>>>>>>         Andrew who
>>>>>>>>>>                             would be happy to help
>>>>>>>>>>
>>>>>>>>>>                             Thanks,
>>>>>>>>>>                             Konstantin
>>>>>>>>>>
>>>>>>>>>>>         On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>         <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>                             
>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>
>>>>>>>>>>>                             I am in the process of setting
>>>>>>>>>>>         up the DPDK
>>>>>>>>>>>                             Poll Mode Driver test suite as
>>>>>>>>>>>         an addition to
>>>>>>>>>>>                             our testing coverage for DPDK at
>>>>>>>>>>>         the UNH lab.
>>>>>>>>>>>
>>>>>>>>>>>                             I have some questions about how
>>>>>>>>>>>         to set the
>>>>>>>>>>>                             test suite arguments.
>>>>>>>>>>>
>>>>>>>>>>>                             I have been able to configure
>>>>>>>>>>>         the Test Engine
>>>>>>>>>>>                             to connect to the hosts in the
>>>>>>>>>>>         testbed. The
>>>>>>>>>>>                             RCF, Configurator, and Tester
>>>>>>>>>>>         all begin to
>>>>>>>>>>>                             run, however the prelude of the
>>>>>>>>>>>         test suite
>>>>>>>>>>>                             fails to run.
>>>>>>>>>>>
>>>>>>>>>>>         https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             The documentation mentions that
>>>>>>>>>>>         there are
>>>>>>>>>>>                             several test parameters for the
>>>>>>>>>>>         test suite,
>>>>>>>>>>>                             like for the IUT test link MAC,
>>>>>>>>>>>         etc. These
>>>>>>>>>>>                             seem like they would need to be
>>>>>>>>>>>         set somewhere
>>>>>>>>>>>                             to run many of the tests.
>>>>>>>>>>>
>>>>>>>>>>>                             I see in the Test Engine
>>>>>>>>>>>         documentation, there
>>>>>>>>>>>                             are instructions on how to
>>>>>>>>>>>         create new
>>>>>>>>>>>                             parameters for test suites in
>>>>>>>>>>>         the Tester
>>>>>>>>>>>                             configuration, but there is
>>>>>>>>>>>         nothing in the
>>>>>>>>>>>                             user guide or in the Tester
>>>>>>>>>>>         guide for how to
>>>>>>>>>>>                             set the arguments for the
>>>>>>>>>>>         parameters when
>>>>>>>>>>>                             running the test suite that I
>>>>>>>>>>>         can find. I'm
>>>>>>>>>>>                             not sure if I need to write my
>>>>>>>>>>>         own Tester
>>>>>>>>>>>                             config, or if I should be
>>>>>>>>>>>         setting these in
>>>>>>>>>>>                             some other way.
>>>>>>>>>>>
>>>>>>>>>>>                             How should these values be set?
>>>>>>>>>>>
>>>>>>>>>>>                             I'm also not sure what environment
>>>>>>>>>>>         variables/arguments are strictly necessary or
>>>>>>>>>>>                             which are optional.
>>>>>>>>>>>
>>>>>>>>>>>                             Regards,
>>>>>>>>>>>                             Adam
>>>>>>>>>>>
>>>>>>>>>>>                             --                     *Adam
>>>>>>>>>>>         Hassick*
>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>         ahassick@iol.unh.edu
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>                         Senior Developer
>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                     --             *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                 --         *Adam Hassick*
>>>>>>>                 Senior Developer
>>>>>>>                 UNH InterOperability Lab
>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>                 +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>             --     *Adam Hassick*
>>>>>>             Senior Developer
>>>>>>             UNH InterOperability Lab
>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>             +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>         -- 
>>>>>         *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 65155 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 35242 bytes --]

Hi Adam,

I've pushed the fix in main branch and a new tag v1.18.1. It should 
solve the problem with IPv6 address from DNS.

Andrew.

On 8/29/23 00:05, Andrew Rybchenko wrote:
> Hi Adam,
>
> > Does the test engine prefer to use IPv6 over IPv4 for initiating the 
> RCF connection to the test bed hosts? And if so, is there a way to 
> force it to use IPv4?
>
> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your 
> case, I guess it is the root cause of the problem.
> Of course, it is TE problem since I see really weird code in 
> lib/comm_net_engine/comm_net_engine.c line 135.
>
> I've pushed fix to the branch user/arybchik/fix_ipv4_only in 
> ts-factory/test-environment repository. Please, try.
>
> It is late night fix with minimal testing and no review. I'll pass it 
> through review process tomorrow and
> hopefully it will be released in one-two days.
>
> Andrew.
>
> On 8/28/23 18:02, Adam Hassick wrote:
>> Hi Andrew,
>>
>> We have yet to notice a distinct pattern with the failures. 
>> Sometimes, the RCF will start and connect without issue a few times 
>> in a row before failing to connect again. Once the issue begins to 
>> occur, neither rebooting all of the hosts (test engine VM, tester, 
>> IUT) or deleting all of the build directories (suites, agents, inst) 
>> and rebooting the hosts afterward resolves the issue. When it begins 
>> working again seems very arbitrary to us.
>>
>> I do usually try to terminate the test engine with Ctrl+C, but when 
>> it hangs while trying to start RCF, that does not work.
>>
>> Does the test engine prefer to use IPv6 over IPv4 for initiating the 
>> RCF connection to the test bed hosts? And if so, is there a way to 
>> force it to use IPv4?
>>
>>  - Adam
>>
>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>     > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>>     Done. It works fine for me without any issues.
>>
>>     Have you noticed any pattern when it works or does not work?
>>     May be it is a problem of not clean state after termination?
>>     Does it work fine the first time after DUTs reboot?
>>     How do you terminate testing? It should be done using Ctrl+C in
>>     terminal where you execute run.sh command.
>>      In this case it should shutdown gracefully and close all test
>>     agents and engine applications.
>>
>>     (I'm trying to understand why you've seen many test agent
>>     processes. It should not happen.)
>>
>>     Andrew.
>>
>>     On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>     On 8/25/23 17:06, Adam Hassick wrote:
>>>>     Hi Andrew,
>>>>
>>>>     Two of our systems (the Test Engine runner and the DUT host)
>>>>     are running Ubuntu 20.04 LTS, however this morning I noticed
>>>>     that the tester system (the one having issues) is running
>>>>     Ubuntu 22.04 LTS.
>>>>     This could be the source of the problem. I encountered a
>>>>     dependency issue trying to run the Test Engine on 22.04 LTS, so
>>>>     I downgraded the system. Since the tester is also the host
>>>>     having connection issues, I will try downgrading that system to
>>>>     20.04, and see if that changes anything.
>>>
>>>     Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>>     Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora
>>>     38 every night.
>>>     Right now Debian 11 is used for test engine in nightly regressions.
>>>
>>>     I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>>>     I did try passing in the "--vg-rcf" argument to the run.sh
>>>>     script of the test suite after installing valgrind, but there
>>>>     was no additional output that I saw.
>>>
>>>     Sorry, I should valgrind output should be in valgrind.te_rcf
>>>     (direction where you run test engine).
>>>
>>>>
>>>>     I will try pulling in the changes you've pushed up, and will
>>>>     see if that fixes anything.
>>>>
>>>>     Thanks,
>>>>     Adam
>>>>
>>>>     On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>         Hello Adam,
>>>>
>>>>         On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>         I'd like to try to repeat the problem locally. Which Linux
>>>>>         distro is running on test engine and agents?
>>>>>
>>>>>         In fact I know one problem with Debian 12 and Fedora 38
>>>>>         and we have
>>>>>         patch in review to fix it, however, the behaviour is
>>>>>         different in
>>>>>         this case, so it is unlike the same problem.
>>>>
>>>>         I've just published a new tag which fixes known test engine
>>>>         side problems on Debian 12 and Fedora 38.
>>>>
>>>>>
>>>>>         One more idea is to install valgrind on the test engine
>>>>>         host and
>>>>>         run with option --vg-rcf to check if something weird is
>>>>>         happening.
>>>>>
>>>>>         What I don't understand right now is why I see just one
>>>>>         failed attempt
>>>>>         to connect in your log.txt and then Logger shutdown after
>>>>>         9 minutes.
>>>>>
>>>>>         Andrew.
>>>>>
>>>>>         On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>          > Is there any firewall in the network or on test hosts
>>>>>>         which could block incoming TCP connection to the port
>>>>>>         23571 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from
>>>>>>         the host where you run test engine?
>>>>>>
>>>>>>         Our test engine host and the testbed are on the same
>>>>>>         subnet. The connection does work sometimes.
>>>>>>
>>>>>>          > If behaviour the same on the next try and you see that
>>>>>>         test agent is kept running, could you check using
>>>>>>          >
>>>>>>          > # netstat -tnlp
>>>>>>          >
>>>>>>          > that Test Agent is listening on the port and try to
>>>>>>         establish TCP connection from test agent using
>>>>>>          >
>>>>>>          > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>          >
>>>>>>          > and check if TCP connection could be established.
>>>>>>
>>>>>>         I was able to replicate the same behavior again, where it
>>>>>>         hangs while RCF is trying to start.
>>>>>>         Running this command, I see this in the output:
>>>>>>
>>>>>>         tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>         <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*  
>>>>>>                     LISTEN      18599/ta
>>>>>>
>>>>>>         So it seems like it is listening on the correct port.
>>>>>>         Additionally, I was able to connect to the Tester machine
>>>>>>         from our Test Engine host using telnet. It printed the
>>>>>>         PID of the process once the connection was opened.
>>>>>>
>>>>>>         I tried running the "ta" application manually on the
>>>>>>         command line, and it didn't print anything at all.
>>>>>>         Maybe the issue is something on the Test Engine side.
>>>>>>
>>>>>>         On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>             Hi Adam,
>>>>>>
>>>>>>              > On the tester host (which appears to be the Peer
>>>>>>         agent), there
>>>>>>             are four processes that I see running, which look
>>>>>>         like the test
>>>>>>             agent processes.
>>>>>>
>>>>>>             Before the next try I'd recommend to kill these
>>>>>>         processes.
>>>>>>
>>>>>>             Is there any firewall in the network or on test hosts
>>>>>>         which could
>>>>>>             block incoming TCP connection to the port 23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from
>>>>>>         the host
>>>>>>             where you run test engine?
>>>>>>
>>>>>>             If behaviour the same on the next try and you see
>>>>>>         that test agent is
>>>>>>             kept running, could you check using
>>>>>>
>>>>>>             # netstat -tnlp
>>>>>>
>>>>>>             that Test Agent is listening on the port and try to
>>>>>>         establish TCP
>>>>>>             connection from test agent using
>>>>>>
>>>>>>             $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>
>>>>>>             and check if TCP connection could be established.
>>>>>>
>>>>>>             Another idea is to login Tester under root as testing
>>>>>>         does, get
>>>>>>             start TA command from the log and try it by hands
>>>>>>         without -n and
>>>>>>             remove extra escaping.
>>>>>>
>>>>>>             # sudo
>>>>>>         PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>         LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>         /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>         host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>
>>>>>>             Hopefully in this case test agent directory remains
>>>>>>         in the /tmp and
>>>>>>             you don't need to copy it as testing does.
>>>>>>             May be output could shed some light on what's going on.
>>>>>>
>>>>>>             Andrew.
>>>>>>
>>>>>>             On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>             Hi Andrew,
>>>>>>>
>>>>>>>             This is the output that I see in the terminal when
>>>>>>>         this failure
>>>>>>>             occurs, after the test agent binaries build and the
>>>>>>>         test engine
>>>>>>>             starts:
>>>>>>>
>>>>>>>             Platform default build - pass
>>>>>>>             Simple RCF consistency check succeeded
>>>>>>>             --->>> Starting Logger...done
>>>>>>>             --->>> Starting RCF...rcf_net_engine_connect():
>>>>>>>         Connection timed
>>>>>>>             out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>
>>>>>>>             Then, it hangs here until I kill the "te_rcf" and
>>>>>>>         "te_tee"
>>>>>>>             processes. I let it hang for around 9 minutes.
>>>>>>>
>>>>>>>             On the tester host (which appears to be the Peer
>>>>>>>         agent), there are
>>>>>>>             four processes that I see running, which look like
>>>>>>>         the test agent
>>>>>>>             processes.
>>>>>>>
>>>>>>>             ta.Peer is an empty file. I've attached the log.txt
>>>>>>>         from this run.
>>>>>>>
>>>>>>>              - Adam
>>>>>>>
>>>>>>>             On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                 Hi Adam,
>>>>>>>
>>>>>>>                 Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>>         double-checked
>>>>>>>                 that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>                 Description in in
>>>>>>>         doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>                 says that copy_timeout is in seconds and
>>>>>>>         implementation in
>>>>>>>                 lib/rcfunix/rcfunix.c passes the value to
>>>>>>>         select() tv_sec.
>>>>>>>                 Theoretically select() could be interrupted by
>>>>>>>         signal, but I
>>>>>>>                 think it is unlikely here.
>>>>>>>
>>>>>>>                 I'm not sure that I understand what do you mean
>>>>>>>         by RCF
>>>>>>>                 connection timeout. Does it happen on TE startup
>>>>>>>         when RCF
>>>>>>>                 starts test agents. If so, TE_RCFUNIX_TIMEOUT
>>>>>>>         could help. Or
>>>>>>>                 does it happen when tests are in progress, e.g.
>>>>>>>         in the middle
>>>>>>>                 of a test. If so, TE_RCFUNIX_TIMEOUT is
>>>>>>>         unrelated and most
>>>>>>>                 likely either host with test agent dies or test
>>>>>>>         agent itself
>>>>>>>                 crashes. It would be easier for me if classify
>>>>>>>         it if you share
>>>>>>>                 text log (log.txt, full or just corresponding
>>>>>>>         fragment with
>>>>>>>                 some context). Also content of ta.DPDK or
>>>>>>>         ta.Peer file
>>>>>>>                 depending on which agent has problems could shed
>>>>>>>         some light.
>>>>>>>                 Corresponding files contain stdout/stderr of
>>>>>>>         test agents.
>>>>>>>
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>                 Hi Andrew,
>>>>>>>>
>>>>>>>>                 I've set up a test rig repository here, and
>>>>>>>>         have created
>>>>>>>>                 configurations for our development testbed
>>>>>>>>         based off of the
>>>>>>>>                 examples.
>>>>>>>>                 We've been able to get the test suite to run
>>>>>>>>         manually on
>>>>>>>>                 Mellanox CX5 devices once.
>>>>>>>>                 However, we are running into an issue where,
>>>>>>>>         when RCF starts,
>>>>>>>>                 the RCF connection times out very frequently.
>>>>>>>>         We aren't sure
>>>>>>>>                 why this is the case.
>>>>>>>>                 It works sometimes, but most of the time when
>>>>>>>>         we try to run
>>>>>>>>                 the test engine, it encounters this issue.
>>>>>>>>                 I've tried changing the RCF port by setting
>>>>>>>>                 "TE_RCF_PORT=<some port number>" and rebooting
>>>>>>>>         the testbed
>>>>>>>>                 machines. Neither seems to fix the issue.
>>>>>>>>
>>>>>>>>                 It also seems like the timeout takes far longer
>>>>>>>>         than 60
>>>>>>>>                 seconds, even when running "export
>>>>>>>>         TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>                 before I try to run the test suite.
>>>>>>>>                 I assume the unit for this variable is seconds?
>>>>>>>>
>>>>>>>>                 Thanks,
>>>>>>>>                 Adam
>>>>>>>>
>>>>>>>>                 On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>                 <ahassick@iol.unh.edu
>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>         <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>>
>>>>>>>>                     Hi Andrew,
>>>>>>>>
>>>>>>>>                     Thanks, I've cloned the example repository
>>>>>>>>         and will start
>>>>>>>>                     setting up a configuration for our
>>>>>>>>         development testbed
>>>>>>>>                     today. I'll let you know if I run into any
>>>>>>>>         difficulties
>>>>>>>>                     or have any questions.
>>>>>>>>
>>>>>>>>                      - Adam
>>>>>>>>
>>>>>>>>                     On Sun, Aug 20, 2023 at 4:40 AM Andrew
>>>>>>>>         Rybchenko
>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                         Hi Adam,
>>>>>>>>
>>>>>>>>                         I've published
>>>>>>>>         https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>                         Hopefully it will help to define your
>>>>>>>>         test rigs and
>>>>>>>>                         successfully run some tests manually.
>>>>>>>>         Feel free to
>>>>>>>>                         ask any questions and I'll answer here
>>>>>>>>         and try to
>>>>>>>>                         update documentation.
>>>>>>>>
>>>>>>>>                         Meanwhile I'll prepare missing bits for
>>>>>>>>         steps (2) and
>>>>>>>>                         (3).
>>>>>>>>                         Hopefully everything is in place for
>>>>>>>>         step (4), but we
>>>>>>>>                         need to make steps (2) and (3) first.
>>>>>>>>
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>>         Hi Adam,
>>>>>>>>>
>>>>>>>>>                         > I've conferred with the rest of the
>>>>>>>>>         team, and we
>>>>>>>>>                         think it would be best to move forward
>>>>>>>>>         with mainly
>>>>>>>>>                         option B.
>>>>>>>>>
>>>>>>>>>                         OK, I'll provide the sample on Monday
>>>>>>>>>         for you. It is
>>>>>>>>>                         almost ready right now, but I need to
>>>>>>>>>         double-check
>>>>>>>>>                         it before publishing.
>>>>>>>>>
>>>>>>>>>                         Regards,
>>>>>>>>>                         Andrew.
>>>>>>>>>
>>>>>>>>>                         On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>>         Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                         I'm adding the CI mailing list to this
>>>>>>>>>>                         conversation. Others in the community
>>>>>>>>>>         might find
>>>>>>>>>>                         this conversation valuable.
>>>>>>>>>>
>>>>>>>>>>                         We do want to run testing on a
>>>>>>>>>>         regular basis. The
>>>>>>>>>>                         Jenkins integration will be very
>>>>>>>>>>         useful for us, as
>>>>>>>>>>                         most of our CI is orchestrated by
>>>>>>>>>>         Jenkins.
>>>>>>>>>>                         I've conferred with the rest of the
>>>>>>>>>>         team, and we
>>>>>>>>>>                         think it would be best to move
>>>>>>>>>>         forward with mainly
>>>>>>>>>>                         option B.
>>>>>>>>>>                         If you would like to know anything
>>>>>>>>>>         about our
>>>>>>>>>>                         testbeds that would help you with
>>>>>>>>>>         creating an
>>>>>>>>>>                         example ts-rigs repo, I'd be happy to
>>>>>>>>>>         answer any
>>>>>>>>>>                         questions you have.
>>>>>>>>>>
>>>>>>>>>>                         We have multiple test rigs (we call
>>>>>>>>>>         these
>>>>>>>>>>                         "DUT-tester pairs") that we run our
>>>>>>>>>>         existing
>>>>>>>>>>                         hardware testing on, with differing
>>>>>>>>>>         network
>>>>>>>>>>                         hardware and CPU architecture. I
>>>>>>>>>>         figured this might
>>>>>>>>>>                         be an important detail.
>>>>>>>>>>
>>>>>>>>>>                         Thanks,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         On Thu, Aug 17, 2023 at 11:44 AM
>>>>>>>>>>         Andrew Rybchenko
>>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>                             Greatings Adam,
>>>>>>>>>>
>>>>>>>>>>                             I'm happy to hear that you're
>>>>>>>>>>         trying to bring
>>>>>>>>>>                             it up.
>>>>>>>>>>
>>>>>>>>>>                             As I understand the final goal is
>>>>>>>>>>         to run it on
>>>>>>>>>>                             regular basis. So, we need to
>>>>>>>>>>         make it properly
>>>>>>>>>>                             from the very beginning.
>>>>>>>>>>                             Bring up of all features consists
>>>>>>>>>>         of 4 steps:
>>>>>>>>>>
>>>>>>>>>>                             1. Create site-specific
>>>>>>>>>>         repository (we call it
>>>>>>>>>>                             ts-rigs) which contains
>>>>>>>>>>         information about test
>>>>>>>>>>                             rigs and other site-specific
>>>>>>>>>>         information like
>>>>>>>>>>                             where to send mails, where to
>>>>>>>>>>         store logs etc.
>>>>>>>>>>                             It is required for manual
>>>>>>>>>>         execution as well,
>>>>>>>>>>                             since test rigs description is
>>>>>>>>>>         essential. I'll
>>>>>>>>>>                             return to the topic below.
>>>>>>>>>>
>>>>>>>>>>                             2. Setup logs storage for
>>>>>>>>>>         automated runs.
>>>>>>>>>>                             Basically it is a disk space plus
>>>>>>>>>>         apache2 web
>>>>>>>>>>                             server with few CGI scripts which
>>>>>>>>>>         help a lot to
>>>>>>>>>>                             save disk space.
>>>>>>>>>>
>>>>>>>>>>                             3. Setup Bublik web application
>>>>>>>>>>         which provides
>>>>>>>>>>                             web interface to view testing
>>>>>>>>>>         results. Same as
>>>>>>>>>>         https://ts-factory.io/bublik
>>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>>
>>>>>>>>>>                             4. Setup Jenkins to run tests on
>>>>>>>>>>         regularly,
>>>>>>>>>>                             save logs in log storage (2) and
>>>>>>>>>>         import it to
>>>>>>>>>>                             bublik (3).
>>>>>>>>>>
>>>>>>>>>>                             Last few month we spent on our
>>>>>>>>>>         homework to make
>>>>>>>>>>                             it simpler to bring up automated
>>>>>>>>>>         execution
>>>>>>>>>>                             using Jenkins -
>>>>>>>>>>         https://github.com/ts-factory/te-jenkins
>>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>                             Corresponding bits in
>>>>>>>>>>         dpdk-ethdev-ts will be
>>>>>>>>>>                             available tomorrow.
>>>>>>>>>>
>>>>>>>>>>                             Let's return to the step (1).
>>>>>>>>>>
>>>>>>>>>>                             Unfortunately there is no
>>>>>>>>>>         publicly available
>>>>>>>>>>                             example of the ts-rigs repository
>>>>>>>>>>         since
>>>>>>>>>>                             sensitive site-specific
>>>>>>>>>>         information is located
>>>>>>>>>>                             there. But I'm ready to help you
>>>>>>>>>>         to create it
>>>>>>>>>>                             for UNH. I see two options here:
>>>>>>>>>>
>>>>>>>>>>                             (A) I'll ask questions and based
>>>>>>>>>>         on your
>>>>>>>>>>                             answers will create the first
>>>>>>>>>>         draft with my
>>>>>>>>>>                             comments.
>>>>>>>>>>
>>>>>>>>>>                             (B) I'll make a template/example
>>>>>>>>>>         ts-rigs repo,
>>>>>>>>>>                             publish it and you'll create UNH
>>>>>>>>>>         ts-rigs based
>>>>>>>>>>                             on it.
>>>>>>>>>>
>>>>>>>>>>                             Of course, I'll help to debug and
>>>>>>>>>>         finally bring
>>>>>>>>>>                             it up in any case.
>>>>>>>>>>
>>>>>>>>>>                             (A) is a bit simpler for me and
>>>>>>>>>>         you, but (B) is
>>>>>>>>>>                             a bit more generic and will help
>>>>>>>>>>         other
>>>>>>>>>>                             potential users to bring it up.
>>>>>>>>>>                             We can combine (A)+(B). I.e.
>>>>>>>>>>         start from (A).
>>>>>>>>>>                             What do you think?
>>>>>>>>>>
>>>>>>>>>>                             Thanks,
>>>>>>>>>>                             Andrew.
>>>>>>>>>>
>>>>>>>>>>                             On 8/17/23 15:18, Konstantin
>>>>>>>>>>         Ushakov wrote:
>>>>>>>>>>>         Greetings Adam,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             Thanks for contacting us. I copy
>>>>>>>>>>>         Andrew who
>>>>>>>>>>>                             would be happy to help
>>>>>>>>>>>
>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>                             Konstantin
>>>>>>>>>>>
>>>>>>>>>>>>         On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>         <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                             
>>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>>
>>>>>>>>>>>>                             I am in the process of setting
>>>>>>>>>>>>         up the DPDK
>>>>>>>>>>>>                             Poll Mode Driver test suite as
>>>>>>>>>>>>         an addition to
>>>>>>>>>>>>                             our testing coverage for DPDK
>>>>>>>>>>>>         at the UNH lab.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I have some questions about how
>>>>>>>>>>>>         to set the
>>>>>>>>>>>>                             test suite arguments.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I have been able to configure
>>>>>>>>>>>>         the Test Engine
>>>>>>>>>>>>                             to connect to the hosts in the
>>>>>>>>>>>>         testbed. The
>>>>>>>>>>>>                             RCF, Configurator, and Tester
>>>>>>>>>>>>         all begin to
>>>>>>>>>>>>                             run, however the prelude of the
>>>>>>>>>>>>         test suite
>>>>>>>>>>>>                             fails to run.
>>>>>>>>>>>>
>>>>>>>>>>>>         https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                             The documentation mentions that
>>>>>>>>>>>>         there are
>>>>>>>>>>>>                             several test parameters for the
>>>>>>>>>>>>         test suite,
>>>>>>>>>>>>                             like for the IUT test link MAC,
>>>>>>>>>>>>         etc. These
>>>>>>>>>>>>                             seem like they would need to be
>>>>>>>>>>>>         set somewhere
>>>>>>>>>>>>                             to run many of the tests.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I see in the Test Engine
>>>>>>>>>>>>         documentation, there
>>>>>>>>>>>>                             are instructions on how to
>>>>>>>>>>>>         create new
>>>>>>>>>>>>                             parameters for test suites in
>>>>>>>>>>>>         the Tester
>>>>>>>>>>>>         configuration, but there is nothing in the
>>>>>>>>>>>>                             user guide or in the Tester
>>>>>>>>>>>>         guide for how to
>>>>>>>>>>>>                             set the arguments for the
>>>>>>>>>>>>         parameters when
>>>>>>>>>>>>                             running the test suite that I
>>>>>>>>>>>>         can find. I'm
>>>>>>>>>>>>                             not sure if I need to write my
>>>>>>>>>>>>         own Tester
>>>>>>>>>>>>                             config, or if I should be
>>>>>>>>>>>>         setting these in
>>>>>>>>>>>>                             some other way.
>>>>>>>>>>>>
>>>>>>>>>>>>                             How should these values be set?
>>>>>>>>>>>>
>>>>>>>>>>>>                             I'm also not sure what environment
>>>>>>>>>>>>         variables/arguments are strictly necessary or
>>>>>>>>>>>>                             which are optional.
>>>>>>>>>>>>
>>>>>>>>>>>>                             Regards,
>>>>>>>>>>>>                             Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                             --                     *Adam
>>>>>>>>>>>>         Hassick*
>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>         ahassick@iol.unh.edu
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                     --             *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 --         *Adam Hassick*
>>>>>>>>                 Senior Developer
>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>                 +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             --     *Adam Hassick*
>>>>>>>             Senior Developer
>>>>>>>             UNH InterOperability Lab
>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>             +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>         -- 
>>>>>>         *Adam Hassick*
>>>>>>         Senior Developer
>>>>>>         UNH InterOperability Lab
>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>         +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>     -- 
>>>>     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu
>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>>
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 68596 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 28452 bytes --]

Hi Andrew,

That fix seems to have resolved the issue, thanks for the quick turnaround
time on that patch.
Now that we have the RCF timeout issue resolved, there are a few other
questions and issues that we have about the tests themselves.

1. The test suite fails to build with a couple warnings.

Below is the stderr log from compilation:

FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ 'lib/76b5a35@
> @ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
> -o 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
> ../../lib/dpdk_pmd_ts.c
> ../../lib/dpdk_pmd_ts.c: In function
> ‘test_create_traffic_generator_params’:
> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
> format arguments [-Werror=format-security]
> 5577 |     rc = te_kvpair_add(result, buf, mode);
> |     ^~
> cc1: all warnings being treated as errors
> ninja: build stopped: subcommand failed.
> ninja: Entering directory `.'
> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ 'lib/76b5a35@
> @ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
> -o 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
> ../../lib/dpdk_pmd_ts.c
> ../../lib/dpdk_pmd_ts.c: In function
> ‘test_create_traffic_generator_params’:
> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
> format arguments [-Werror=format-security]
> 5577 |     rc = te_kvpair_add(result, buf, mode);
> |     ^~
> cc1: all warnings being treated as errors
>

This error wasn't occurring last week, which was the last time I ran the
tests.
The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC
v11.4.0 installed, if this information is helpful.

2. On the Mellanox CX5s, there are over 6,000 tests run, which collectively
take around 9 hours. Is it possible, and would it make sense, to lower the
test coverage and have the test suite run faster?

For some context, we run immediate testing on incoming patches for DPDK
main and development branches, as well as periodic test runs on the main,
stable, and LTS branches.
For us to consider including this test suite as part of our immediate
testing on patches, we would have to reduce the test coverage to the most
important tests.
This is primarily to reduce the testing time to, for example, less than 30
minutes. Testing on patches can't take too long because the lab can receive
numerous patches each day, which each require individual testing runs.

At what frequency we run these tests, and on what, still needs to be
discussed with the DPDK community, but it would be nice to know if the test
suite had a "dial" to control the testing coverage.

3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300 of
~6,600 tests passed. Is there anything we can do to diagnose these test
failures?

Thanks,
Adam


On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> I've pushed the fix in main branch and a new tag v1.18.1. It should solve
> the problem with IPv6 address from DNS.
>
> Andrew.
>
> On 8/29/23 00:05, Andrew Rybchenko wrote:
>
> Hi Adam,
>
> > Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
> connection to the test bed hosts? And if so, is there a way to force it to
> use IPv4?
>
> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your case,
> I guess it is the root cause of the problem.
> Of course, it is TE problem since I see really weird code in
> lib/comm_net_engine/comm_net_engine.c line 135.
>
> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
> ts-factory/test-environment repository. Please, try.
>
> It is late night fix with minimal testing and no review. I'll pass it
> through review process tomorrow and
> hopefully it will be released in one-two days.
>
> Andrew.
>
> On 8/28/23 18:02, Adam Hassick wrote:
>
> Hi Andrew,
>
> We have yet to notice a distinct pattern with the failures. Sometimes, the
> RCF will start and connect without issue a few times in a row before
> failing to connect again. Once the issue begins to occur, neither rebooting
> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
> directories (suites, agents, inst) and rebooting the hosts afterward
> resolves the issue. When it begins working again seems very arbitrary to us.
>
> I do usually try to terminate the test engine with Ctrl+C, but when it
> hangs while trying to start RCF, that does not work.
>
> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
> connection to the test bed hosts? And if so, is there a way to force it to
> use IPv4?
>
>  - Adam
>
> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>> Done. It works fine for me without any issues.
>>
>> Have you noticed any pattern when it works or does not work?
>> May be it is a problem of not clean state after termination?
>> Does it work fine the first time after DUTs reboot?
>> How do you terminate testing? It should be done using Ctrl+C in terminal
>> where you execute run.sh command.
>>  In this case it should shutdown gracefully and close all test agents and
>> engine applications.
>>
>> (I'm trying to understand why you've seen many test agent processes. It
>> should not happen.)
>>
>> Andrew.
>>
>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>
>> On 8/25/23 17:06, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> Two of our systems (the Test Engine runner and the DUT host) are running
>> Ubuntu 20.04 LTS, however this morning I noticed that the tester system
>> (the one having issues) is running Ubuntu 22.04 LTS.
>> This could be the source of the problem. I encountered a dependency issue
>> trying to run the Test Engine on 22.04 LTS, so I downgraded the system.
>> Since the tester is also the host having connection issues, I will try
>> downgrading that system to 20.04, and see if that changes anything.
>>
>>
>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>> Right now Debian 11 is used for test engine in nightly regressions.
>>
>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>> I did try passing in the "--vg-rcf" argument to the run.sh script of the
>> test suite after installing valgrind, but there was no additional output
>> that I saw.
>>
>>
>> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
>> where you run test engine).
>>
>>
>> I will try pulling in the changes you've pushed up, and will see if that
>> fixes anything.
>>
>> Thanks,
>> Adam
>>
>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hello Adam,
>>>
>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>
>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>> running on test engine and agents?
>>>
>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>> patch in review to fix it, however, the behaviour is different in
>>> this case, so it is unlike the same problem.
>>>
>>>
>>> I've just published a new tag which fixes known test engine side
>>> problems on Debian 12 and Fedora 38.
>>>
>>>
>>> One more idea is to install valgrind on the test engine host and
>>> run with option --vg-rcf to check if something weird is happening.
>>>
>>> What I don't understand right now is why I see just one failed attempt
>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>
>>> Andrew.
>>>
>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>
>>>  > Is there any firewall in the network or on test hosts which could
>>> block incoming TCP connection to the port 23571
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>>> you run test engine?
>>>
>>> Our test engine host and the testbed are on the same subnet. The
>>> connection does work sometimes.
>>>
>>>  > If behaviour the same on the next try and you see that test agent is
>>> kept running, could you check using
>>>  >
>>>  > # netstat -tnlp
>>>  >
>>>  > that Test Agent is listening on the port and try to establish TCP
>>> connection from test agent using
>>>  >
>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>  >
>>>  > and check if TCP connection could be established.
>>>
>>> I was able to replicate the same behavior again, where it hangs while
>>> RCF is trying to start.
>>> Running this command, I see this in the output:
>>>
>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>  18599/ta
>>>
>>> So it seems like it is listening on the correct port.
>>> Additionally, I was able to connect to the Tester machine from our Test
>>> Engine host using telnet. It printed the PID of the process once the
>>> connection was opened.
>>>
>>> I tried running the "ta" application manually on the command line, and
>>> it didn't print anything at all.
>>> Maybe the issue is something on the Test Engine side.
>>>
>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>     Hi Adam,
>>>
>>>      > On the tester host (which appears to be the Peer agent), there
>>>     are four processes that I see running, which look like the test
>>>     agent processes.
>>>
>>>     Before the next try I'd recommend to kill these processes.
>>>
>>>     Is there any firewall in the network or on test hosts which could
>>>     block incoming TCP connection to the port 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>     where you run test engine?
>>>
>>>     If behaviour the same on the next try and you see that test agent is
>>>     kept running, could you check using
>>>
>>>     # netstat -tnlp
>>>
>>>     that Test Agent is listening on the port and try to establish TCP
>>>     connection from test agent using
>>>
>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>     and check if TCP connection could be established.
>>>
>>>     Another idea is to login Tester under root as testing does, get
>>>     start TA command from the log and try it by hands without -n and
>>>     remove extra escaping.
>>>
>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>
>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>
>>>     Hopefully in this case test agent directory remains in the /tmp and
>>>     you don't need to copy it as testing does.
>>>     May be output could shed some light on what's going on.
>>>
>>>     Andrew.
>>>
>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>
>>>     Hi Andrew,
>>>
>>>     This is the output that I see in the terminal when this failure
>>>     occurs, after the test agent binaries build and the test engine
>>>     starts:
>>>
>>>     Platform default build - pass
>>>     Simple RCF consistency check succeeded
>>>     --->>> Starting Logger...done
>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>     processes. I let it hang for around 9 minutes.
>>>
>>>     On the tester host (which appears to be the Peer agent), there are
>>>     four processes that I see running, which look like the test agent
>>>     processes.
>>>
>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>
>>>      - Adam
>>>
>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru
>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>         Hi Adam,
>>>
>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>         says that copy_timeout is in seconds and implementation in
>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>         Theoretically select() could be interrupted by signal, but I
>>>         think it is unlikely here.
>>>
>>>         I'm not sure that I understand what do you mean by RCF
>>>         connection timeout. Does it happen on TE startup when RCF
>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>         does it happen when tests are in progress, e.g. in the middle
>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>         likely either host with test agent dies or test agent itself
>>>         crashes. It would be easier for me if classify it if you share
>>>         text log (log.txt, full or just corresponding fragment with
>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>         depending on which agent has problems could shed some light.
>>>         Corresponding files contain stdout/stderr of test agents.
>>>
>>>         Andrew.
>>>
>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>
>>>         Hi Andrew,
>>>
>>>         I've set up a test rig repository here, and have created
>>>         configurations for our development testbed based off of the
>>>         examples.
>>>         We've been able to get the test suite to run manually on
>>>         Mellanox CX5 devices once.
>>>         However, we are running into an issue where, when RCF starts,
>>>         the RCF connection times out very frequently. We aren't sure
>>>         why this is the case.
>>>         It works sometimes, but most of the time when we try to run
>>>         the test engine, it encounters this issue.
>>>         I've tried changing the RCF port by setting
>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>         machines. Neither seems to fix the issue.
>>>
>>>         It also seems like the timeout takes far longer than 60
>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>         before I try to run the test suite.
>>>         I assume the unit for this variable is seconds?
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>> wrote:
>>>
>>>             Hi Andrew,
>>>
>>>             Thanks, I've cloned the example repository and will start
>>>             setting up a configuration for our development testbed
>>>             today. I'll let you know if I run into any difficulties
>>>             or have any questions.
>>>
>>>              - Adam
>>>
>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>             <andrew.rybchenko@oktetlabs.ru
>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>                 Hi Adam,
>>>
>>>                 I've published
>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>                 Hopefully it will help to define your test rigs and
>>>                 successfully run some tests manually. Feel free to
>>>                 ask any questions and I'll answer here and try to
>>>                 update documentation.
>>>
>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>                 (3).
>>>                 Hopefully everything is in place for step (4), but we
>>>                 need to make steps (2) and (3) first.
>>>
>>>                 Andrew.
>>>
>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>
>>>                 Hi Adam,
>>>
>>>                 > I've conferred with the rest of the team, and we
>>>                 think it would be best to move forward with mainly
>>>                 option B.
>>>
>>>                 OK, I'll provide the sample on Monday for you. It is
>>>                 almost ready right now, but I need to double-check
>>>                 it before publishing.
>>>
>>>                 Regards,
>>>                 Andrew.
>>>
>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>
>>>                 Hi Andrew,
>>>
>>>                 I'm adding the CI mailing list to this
>>>                 conversation. Others in the community might find
>>>                 this conversation valuable.
>>>
>>>                 We do want to run testing on a regular basis. The
>>>                 Jenkins integration will be very useful for us, as
>>>                 most of our CI is orchestrated by Jenkins.
>>>                 I've conferred with the rest of the team, and we
>>>                 think it would be best to move forward with mainly
>>>                 option B.
>>>                 If you would like to know anything about our
>>>                 testbeds that would help you with creating an
>>>                 example ts-rigs repo, I'd be happy to answer any
>>>                 questions you have.
>>>
>>>                 We have multiple test rigs (we call these
>>>                 "DUT-tester pairs") that we run our existing
>>>                 hardware testing on, with differing network
>>>                 hardware and CPU architecture. I figured this might
>>>                 be an important detail.
>>>
>>>                 Thanks,
>>>                 Adam
>>>
>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>                 <andrew.rybchenko@oktetlabs.ru
>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>                     Greatings Adam,
>>>
>>>                     I'm happy to hear that you're trying to bring
>>>                     it up.
>>>
>>>                     As I understand the final goal is to run it on
>>>                     regular basis. So, we need to make it properly
>>>                     from the very beginning.
>>>                     Bring up of all features consists of 4 steps:
>>>
>>>                     1. Create site-specific repository (we call it
>>>                     ts-rigs) which contains information about test
>>>                     rigs and other site-specific information like
>>>                     where to send mails, where to store logs etc.
>>>                     It is required for manual execution as well,
>>>                     since test rigs description is essential. I'll
>>>                     return to the topic below.
>>>
>>>                     2. Setup logs storage for automated runs.
>>>                     Basically it is a disk space plus apache2 web
>>>                     server with few CGI scripts which help a lot to
>>>                     save disk space.
>>>
>>>                     3. Setup Bublik web application which provides
>>>                     web interface to view testing results. Same as
>>>                     https://ts-factory.io/bublik
>>>                     <https://ts-factory.io/bublik>
>>> <https://ts-factory.io/bublik>
>>>
>>>                     4. Setup Jenkins to run tests on regularly,
>>>                     save logs in log storage (2) and import it to
>>>                     bublik (3).
>>>
>>>                     Last few month we spent on our homework to make
>>>                     it simpler to bring up automated execution
>>>                     using Jenkins -
>>>                     https://github.com/ts-factory/te-jenkins
>>>                     <https://github.com/ts-factory/te-jenkins>
>>> <https://github.com/ts-factory/te-jenkins>
>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>                     available tomorrow.
>>>
>>>                     Let's return to the step (1).
>>>
>>>                     Unfortunately there is no publicly available
>>>                     example of the ts-rigs repository since
>>>                     sensitive site-specific information is located
>>>                     there. But I'm ready to help you to create it
>>>                     for UNH. I see two options here:
>>>
>>>                     (A) I'll ask questions and based on your
>>>                     answers will create the first draft with my
>>>                     comments.
>>>
>>>                     (B) I'll make a template/example ts-rigs repo,
>>>                     publish it and you'll create UNH ts-rigs based
>>>                     on it.
>>>
>>>                     Of course, I'll help to debug and finally bring
>>>                     it up in any case.
>>>
>>>                     (A) is a bit simpler for me and you, but (B) is
>>>                     a bit more generic and will help other
>>>                     potential users to bring it up.
>>>                     We can combine (A)+(B). I.e. start from (A).
>>>                     What do you think?
>>>
>>>                     Thanks,
>>>                     Andrew.
>>>
>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>
>>>                     Greetings Adam,
>>>
>>>
>>>                     Thanks for contacting us. I copy Andrew who
>>>                     would be happy to help
>>>
>>>                     Thanks,
>>>                     Konstantin
>>>
>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>> wrote:
>>>
>>>                     
>>>                     Greetings Konstantin,
>>>
>>>                     I am in the process of setting up the DPDK
>>>                     Poll Mode Driver test suite as an addition to
>>>                     our testing coverage for DPDK at the UNH lab.
>>>
>>>                     I have some questions about how to set the
>>>                     test suite arguments.
>>>
>>>                     I have been able to configure the Test Engine
>>>                     to connect to the hosts in the testbed. The
>>>                     RCF, Configurator, and Tester all begin to
>>>                     run, however the prelude of the test suite
>>>                     fails to run.
>>>
>>>
>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>
>>>                     The documentation mentions that there are
>>>                     several test parameters for the test suite,
>>>                     like for the IUT test link MAC, etc. These
>>>                     seem like they would need to be set somewhere
>>>                     to run many of the tests.
>>>
>>>                     I see in the Test Engine documentation, there
>>>                     are instructions on how to create new
>>>                     parameters for test suites in the Tester
>>>                     configuration, but there is nothing in the
>>>                     user guide or in the Tester guide for how to
>>>                     set the arguments for the parameters when
>>>                     running the test suite that I can find. I'm
>>>                     not sure if I need to write my own Tester
>>>                     config, or if I should be setting these in
>>>                     some other way.
>>>
>>>                     How should these values be set?
>>>
>>>                     I'm also not sure what environment
>>>                     variables/arguments are strictly necessary or
>>>                     which are optional.
>>>
>>>                     Regards,
>>>                     Adam
>>>
>>>                     --                     *Adam Hassick*
>>>                     Senior Developer
>>>                     UNH InterOperability Lab
>>>                     ahassick@iol.unh.edu
>>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>                     +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>>                 --                 *Adam Hassick*
>>>                 Senior Developer
>>>                 UNH InterOperability Lab
>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>                 +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>>
>>>             --             *Adam Hassick*
>>>             Senior Developer
>>>             UNH InterOperability Lab
>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         --         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>>     --     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 67404 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 43784 bytes --]

1. We'll sort out warnings this week. Thanks for heads up.

2. You can add command-line option --sanity to run tests marked with 
TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and 
grep TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). 
Yes, there is a space for terminology improvement here. We'll do it.

Also it takes a lot of time because of failures and tests which wait for 
some timeout.

Support for old LTS branches was dropped some time ago, but in the 
future it is definitely possible to keep it for new LTS branches. I 
think 22.11 is supported, but I'm not sure about older LTS releases.

The test harness can provide coverage reports based on gcov, but I'm not 
sure what you mean by a "dial" to control test coverage. Provided 
reports are rather for human to analyze.

3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing 
results. Unfortunately it is time consuming to fill in expectations 
database since it is necessary to analyze testing results and classify 
if it is a bug or just acceptable behaviour aspect.

Bublik allows to compare results of two runs. It is useful for human, 
but still not good for automation.

I have local patch for mlx5 driver which reports Tx ring size maximum. 
It makes pass rate higher. It is a problem for test harness that mlx5 
does not report limits right now.

Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio 
net is 99% right now and could be done 100% easily (just one thing to 
fix in expectations).

I think logs storage setup is essential for logs analysis. Of course, 
you can request HTML logs when you run tests (--log-html=html) or 
generate after run using dpdk-ethdev-ts/scripts/html-log.sh and open 
index.html in a browser, but logs storage makes it more convenient.

Latest version of test-environment has examples of our CGI scripts which 
we use for log storage (see tools/log_server/README.md).

Also all bits for Jenkins setup are available. See 
dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in 
ts-rigs-sample.

Of course, bublik makes logs analysis even more convenient, but it uses 
logs storage anyway.

Andrew.

On 8/29/23 17:02, Adam Hassick wrote:
> Hi Andrew,
>
> That fix seems to have resolved the issue, thanks for the quick 
> turnaround time on that patch.
> Now that we have the RCF timeout issue resolved, there are a few other 
> questions and issues that we have about the tests themselves.
>
> 1. The test suite fails to build with a couple warnings.
>
> Below is the stderr log from compilation:
>
>     FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>     -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>     -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC
>     -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>     ../../lib/dpdk_pmd_ts.c
>     ../../lib/dpdk_pmd_ts.c: In function
>     ‘test_create_traffic_generator_params’:
>     ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>     and no format arguments [-Werror=format-security]
>     5577 |     rc = te_kvpair_add(result, buf, mode);
>     |     ^~
>     cc1: all warnings being treated as errors
>     ninja: build stopped: subcommand failed.
>     ninja: Entering directory `.'
>     FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>     -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>     -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC
>     -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>     ../../lib/dpdk_pmd_ts.c
>     ../../lib/dpdk_pmd_ts.c: In function
>     ‘test_create_traffic_generator_params’:
>     ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>     and no format arguments [-Werror=format-security]
>     5577 |     rc = te_kvpair_add(result, buf, mode);
>     |     ^~
>     cc1: all warnings being treated as errors
>
>
> This error wasn't occurring last week, which was the last time I ran 
> the tests.
> The TE host and the DUT have GCC v9.4.0 installed, and the tester has 
> GCC v11.4.0 installed, if this information is helpful.
>
> 2. On the Mellanox CX5s, there are over 6,000 tests run, which 
> collectively take around 9 hours. Is it possible, and would it make 
> sense, to lower the test coverage and have the test suite run faster?
>
> For some context, we run immediate testing on incoming patches for 
> DPDK main and development branches, as well as periodic test runs on 
> the main, stable, and LTS branches.
> For us to consider including this test suite as part of our immediate 
> testing on patches, we would have to reduce the test coverage to the 
> most important tests.
> This is primarily to reduce the testing time to, for example, less 
> than 30 minutes. Testing on patches can't take too long because the 
> lab can receive numerous patches each day, which each require 
> individual testing runs.
>
> At what frequency we run these tests, and on what, still needs to be 
> discussed with the DPDK community, but it would be nice to know if the 
> test suite had a "dial" to control the testing coverage.
>
> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 
> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose 
> these test failures?
>
> Thanks,
> Adam
>
>
> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     I've pushed the fix in main branch and a new tag v1.18.1. It
>     should solve the problem with IPv6 address from DNS.
>
>     Andrew.
>
>     On 8/29/23 00:05, Andrew Rybchenko wrote:
>>     Hi Adam,
>>
>>     > Does the test engine prefer to use IPv6 over IPv4 for
>>     initiating the RCF connection to the test bed hosts? And if so,
>>     is there a way to force it to use IPv4?
>>
>>     Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in
>>     your case, I guess it is the root cause of the problem.
>>     Of course, it is TE problem since I see really weird code in
>>     lib/comm_net_engine/comm_net_engine.c line 135.
>>
>>     I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>     ts-factory/test-environment repository. Please, try.
>>
>>     It is late night fix with minimal testing and no review. I'll
>>     pass it through review process tomorrow and
>>     hopefully it will be released in one-two days.
>>
>>     Andrew.
>>
>>     On 8/28/23 18:02, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     We have yet to notice a distinct pattern with the failures.
>>>     Sometimes, the RCF will start and connect without issue a few
>>>     times in a row before failing to connect again. Once the issue
>>>     begins to occur, neither rebooting all of the hosts (test engine
>>>     VM, tester, IUT) or deleting all of the build directories
>>>     (suites, agents, inst) and rebooting the hosts afterward
>>>     resolves the issue. When it begins working again seems very
>>>     arbitrary to us.
>>>
>>>     I do usually try to terminate the test engine with Ctrl+C, but
>>>     when it hangs while trying to start RCF, that does not work.
>>>
>>>     Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>     the RCF connection to the test bed hosts? And if so, is there a
>>>     way to force it to use IPv4?
>>>
>>>      - Adam
>>>
>>>     On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>         > I'll double-check test engine on Ubuntu 20.04 and Ubuntu
>>>         22.04.
>>>
>>>         Done. It works fine for me without any issues.
>>>
>>>         Have you noticed any pattern when it works or does not work?
>>>         May be it is a problem of not clean state after termination?
>>>         Does it work fine the first time after DUTs reboot?
>>>         How do you terminate testing? It should be done using Ctrl+C
>>>         in terminal where you execute run.sh command.
>>>          In this case it should shutdown gracefully and close all
>>>         test agents and engine applications.
>>>
>>>         (I'm trying to understand why you've seen many test agent
>>>         processes. It should not happen.)
>>>
>>>         Andrew.
>>>
>>>         On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>         On 8/25/23 17:06, Adam Hassick wrote:
>>>>>         Hi Andrew,
>>>>>
>>>>>         Two of our systems (the Test Engine runner and the DUT
>>>>>         host) are running Ubuntu 20.04 LTS, however this morning I
>>>>>         noticed that the tester system (the one having issues) is
>>>>>         running Ubuntu 22.04 LTS.
>>>>>         This could be the source of the problem. I encountered a
>>>>>         dependency issue trying to run the Test Engine on 22.04
>>>>>         LTS, so I downgraded the system. Since the tester is also
>>>>>         the host having connection issues, I will try downgrading
>>>>>         that system to 20.04, and see if that changes anything.
>>>>
>>>>         Unlikely, but who knows. We run tests (DUTs) on Ubuntu
>>>>         20.04, Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11
>>>>         and Fedora 38 every night.
>>>>         Right now Debian 11 is used for test engine in nightly
>>>>         regressions.
>>>>
>>>>         I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>
>>>>>         I did try passing in the "--vg-rcf" argument to the run.sh
>>>>>         script of the test suite after installing valgrind, but
>>>>>         there was no additional output that I saw.
>>>>
>>>>         Sorry, I should valgrind output should be in
>>>>         valgrind.te_rcf (direction where you run test engine).
>>>>
>>>>>
>>>>>         I will try pulling in the changes you've pushed up, and
>>>>>         will see if that fixes anything.
>>>>>
>>>>>         Thanks,
>>>>>         Adam
>>>>>
>>>>>         On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>             Hello Adam,
>>>>>
>>>>>             On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>             I'd like to try to repeat the problem locally. Which
>>>>>>             Linux distro is running on test engine and agents?
>>>>>>
>>>>>>             In fact I know one problem with Debian 12 and Fedora
>>>>>>             38 and we have
>>>>>>             patch in review to fix it, however, the behaviour is
>>>>>>             different in
>>>>>>             this case, so it is unlike the same problem.
>>>>>
>>>>>             I've just published a new tag which fixes known test
>>>>>             engine side problems on Debian 12 and Fedora 38.
>>>>>
>>>>>>
>>>>>>             One more idea is to install valgrind on the test
>>>>>>             engine host and
>>>>>>             run with option --vg-rcf to check if something weird
>>>>>>             is happening.
>>>>>>
>>>>>>             What I don't understand right now is why I see just
>>>>>>             one failed attempt
>>>>>>             to connect in your log.txt and then Logger shutdown
>>>>>>             after 9 minutes.
>>>>>>
>>>>>>             Andrew.
>>>>>>
>>>>>>             On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>              > Is there any firewall in the network or on test
>>>>>>>             hosts which could block incoming TCP connection to
>>>>>>>             the port 23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             from the host where you run test engine?
>>>>>>>
>>>>>>>             Our test engine host and the testbed are on the same
>>>>>>>             subnet. The connection does work sometimes.
>>>>>>>
>>>>>>>              > If behaviour the same on the next try and you see
>>>>>>>             that test agent is kept running, could you check using
>>>>>>>              >
>>>>>>>              > # netstat -tnlp
>>>>>>>              >
>>>>>>>              > that Test Agent is listening on the port and try
>>>>>>>             to establish TCP connection from test agent using
>>>>>>>              >
>>>>>>>              > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>              >
>>>>>>>              > and check if TCP connection could be established.
>>>>>>>
>>>>>>>             I was able to replicate the same behavior again,
>>>>>>>             where it hangs while RCF is trying to start.
>>>>>>>             Running this command, I see this in the output:
>>>>>>>
>>>>>>>             tcp        0      0 0.0.0.0:23571
>>>>>>>             <http://0.0.0.0:23571> <http://0.0.0.0:23571>
>>>>>>>             <http://0.0.0.0:23571> 0.0.0.0:*              
>>>>>>>             LISTEN  18599/ta
>>>>>>>
>>>>>>>             So it seems like it is listening on the correct port.
>>>>>>>             Additionally, I was able to connect to the Tester
>>>>>>>             machine from our Test Engine host using telnet. It
>>>>>>>             printed the PID of the process once the connection
>>>>>>>             was opened.
>>>>>>>
>>>>>>>             I tried running the "ta" application manually on the
>>>>>>>             command line, and it didn't print anything at all.
>>>>>>>             Maybe the issue is something on the Test Engine side.
>>>>>>>
>>>>>>>             On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                 Hi Adam,
>>>>>>>
>>>>>>>                  > On the tester host (which appears to be the
>>>>>>>             Peer agent), there
>>>>>>>                 are four processes that I see running, which
>>>>>>>             look like the test
>>>>>>>                 agent processes.
>>>>>>>
>>>>>>>                 Before the next try I'd recommend to kill these
>>>>>>>             processes.
>>>>>>>
>>>>>>>                 Is there any firewall in the network or on test
>>>>>>>             hosts which could
>>>>>>>                 block incoming TCP connection to the port 23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             from the host
>>>>>>>                 where you run test engine?
>>>>>>>
>>>>>>>                 If behaviour the same on the next try and you
>>>>>>>             see that test agent is
>>>>>>>                 kept running, could you check using
>>>>>>>
>>>>>>>                 # netstat -tnlp
>>>>>>>
>>>>>>>                 that Test Agent is listening on the port and try
>>>>>>>             to establish TCP
>>>>>>>                 connection from test agent using
>>>>>>>
>>>>>>>                 $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>
>>>>>>>                 and check if TCP connection could be established.
>>>>>>>
>>>>>>>                 Another idea is to login Tester under root as
>>>>>>>             testing does, get
>>>>>>>                 start TA command from the log and try it by
>>>>>>>             hands without -n and
>>>>>>>                 remove extra escaping.
>>>>>>>
>>>>>>>                 # sudo
>>>>>>>             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>             /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>
>>>>>>>                 Hopefully in this case test agent directory
>>>>>>>             remains in the /tmp and
>>>>>>>                 you don't need to copy it as testing does.
>>>>>>>                 May be output could shed some light on what's
>>>>>>>             going on.
>>>>>>>
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>                 Hi Andrew,
>>>>>>>>
>>>>>>>>                 This is the output that I see in the terminal
>>>>>>>>             when this failure
>>>>>>>>                 occurs, after the test agent binaries build and
>>>>>>>>             the test engine
>>>>>>>>                 starts:
>>>>>>>>
>>>>>>>>                 Platform default build - pass
>>>>>>>>                 Simple RCF consistency check succeeded
>>>>>>>>                 --->>> Starting Logger...done
>>>>>>>>                 --->>> Starting RCF...rcf_net_engine_connect():
>>>>>>>>             Connection timed
>>>>>>>>                 out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>
>>>>>>>>                 Then, it hangs here until I kill the "te_rcf"
>>>>>>>>             and "te_tee"
>>>>>>>>                 processes. I let it hang for around 9 minutes.
>>>>>>>>
>>>>>>>>                 On the tester host (which appears to be the
>>>>>>>>             Peer agent), there are
>>>>>>>>                 four processes that I see running, which look
>>>>>>>>             like the test agent
>>>>>>>>                 processes.
>>>>>>>>
>>>>>>>>                 ta.Peer is an empty file. I've attached the
>>>>>>>>             log.txt from this run.
>>>>>>>>
>>>>>>>>                  - Adam
>>>>>>>>
>>>>>>>>                 On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                     Hi Adam,
>>>>>>>>
>>>>>>>>                     Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>>>             double-checked
>>>>>>>>                     that it goes to 'copy_timeout' in
>>>>>>>>             ts-conf/rcf.conf.
>>>>>>>>                     Description in in
>>>>>>>>             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>                     says that copy_timeout is in seconds and
>>>>>>>>             implementation in
>>>>>>>>                     lib/rcfunix/rcfunix.c passes the value to
>>>>>>>>             select() tv_sec.
>>>>>>>>                     Theoretically select() could be interrupted
>>>>>>>>             by signal, but I
>>>>>>>>                     think it is unlikely here.
>>>>>>>>
>>>>>>>>                     I'm not sure that I understand what do you
>>>>>>>>             mean by RCF
>>>>>>>>                     connection timeout. Does it happen on TE
>>>>>>>>             startup when RCF
>>>>>>>>                     starts test agents. If so,
>>>>>>>>             TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>>>                     does it happen when tests are in progress,
>>>>>>>>             e.g. in the middle
>>>>>>>>                     of a test. If so, TE_RCFUNIX_TIMEOUT is
>>>>>>>>             unrelated and most
>>>>>>>>                     likely either host with test agent dies or
>>>>>>>>             test agent itself
>>>>>>>>                     crashes. It would be easier for me if
>>>>>>>>             classify it if you share
>>>>>>>>                     text log (log.txt, full or just
>>>>>>>>             corresponding fragment with
>>>>>>>>                     some context). Also content of ta.DPDK or
>>>>>>>>             ta.Peer file
>>>>>>>>                     depending on which agent has problems could
>>>>>>>>             shed some light.
>>>>>>>>                     Corresponding files contain stdout/stderr
>>>>>>>>             of test agents.
>>>>>>>>
>>>>>>>>                     Andrew.
>>>>>>>>
>>>>>>>>                     On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>                     Hi Andrew,
>>>>>>>>>
>>>>>>>>>                     I've set up a test rig repository here,
>>>>>>>>>             and have created
>>>>>>>>>                     configurations for our development testbed
>>>>>>>>>             based off of the
>>>>>>>>>                     examples.
>>>>>>>>>                     We've been able to get the test suite to
>>>>>>>>>             run manually on
>>>>>>>>>                     Mellanox CX5 devices once.
>>>>>>>>>                     However, we are running into an issue
>>>>>>>>>             where, when RCF starts,
>>>>>>>>>                     the RCF connection times out very
>>>>>>>>>             frequently. We aren't sure
>>>>>>>>>                     why this is the case.
>>>>>>>>>                     It works sometimes, but most of the time
>>>>>>>>>             when we try to run
>>>>>>>>>                     the test engine, it encounters this issue.
>>>>>>>>>                     I've tried changing the RCF port by setting
>>>>>>>>>                     "TE_RCF_PORT=<some port number>" and
>>>>>>>>>             rebooting the testbed
>>>>>>>>>                     machines. Neither seems to fix the issue.
>>>>>>>>>
>>>>>>>>>                     It also seems like the timeout takes far
>>>>>>>>>             longer than 60
>>>>>>>>>                     seconds, even when running "export
>>>>>>>>>             TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>                     before I try to run the test suite.
>>>>>>>>>                     I assume the unit for this variable is
>>>>>>>>>             seconds?
>>>>>>>>>
>>>>>>>>>                     Thanks,
>>>>>>>>>                     Adam
>>>>>>>>>
>>>>>>>>>                     On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>>                     <ahassick@iol.unh.edu
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>>>
>>>>>>>>>                         Hi Andrew,
>>>>>>>>>
>>>>>>>>>                         Thanks, I've cloned the example
>>>>>>>>>             repository and will start
>>>>>>>>>                         setting up a configuration for our
>>>>>>>>>             development testbed
>>>>>>>>>                         today. I'll let you know if I run into
>>>>>>>>>             any difficulties
>>>>>>>>>                         or have any questions.
>>>>>>>>>
>>>>>>>>>                          - Adam
>>>>>>>>>
>>>>>>>>>                         On Sun, Aug 20, 2023 at 4:40 AM Andrew
>>>>>>>>>             Rybchenko
>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                             Hi Adam,
>>>>>>>>>
>>>>>>>>>                             I've published
>>>>>>>>>             https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>             <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>                             Hopefully it will help to define
>>>>>>>>>             your test rigs and
>>>>>>>>>                             successfully run some tests
>>>>>>>>>             manually. Feel free to
>>>>>>>>>                             ask any questions and I'll answer
>>>>>>>>>             here and try to
>>>>>>>>>                             update documentation.
>>>>>>>>>
>>>>>>>>>                             Meanwhile I'll prepare missing
>>>>>>>>>             bits for steps (2) and
>>>>>>>>>                             (3).
>>>>>>>>>                             Hopefully everything is in place
>>>>>>>>>             for step (4), but we
>>>>>>>>>                             need to make steps (2) and (3) first.
>>>>>>>>>
>>>>>>>>>                             Andrew.
>>>>>>>>>
>>>>>>>>>                             On 8/18/23 21:40, Andrew Rybchenko
>>>>>>>>>             wrote:
>>>>>>>>>>             Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                             > I've conferred with the rest of
>>>>>>>>>>             the team, and we
>>>>>>>>>>                             think it would be best to move
>>>>>>>>>>             forward with mainly
>>>>>>>>>>                             option B.
>>>>>>>>>>
>>>>>>>>>>                             OK, I'll provide the sample on
>>>>>>>>>>             Monday for you. It is
>>>>>>>>>>                             almost ready right now, but I
>>>>>>>>>>             need to double-check
>>>>>>>>>>                             it before publishing.
>>>>>>>>>>
>>>>>>>>>>                             Regards,
>>>>>>>>>>                             Andrew.
>>>>>>>>>>
>>>>>>>>>>                             On 8/17/23 20:03, Adam Hassick
>>>>>>>>>>             wrote:
>>>>>>>>>>>             Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                             I'm adding the CI mailing list
>>>>>>>>>>>             to this
>>>>>>>>>>>             conversation. Others in the community might find
>>>>>>>>>>>                             this conversation valuable.
>>>>>>>>>>>
>>>>>>>>>>>                             We do want to run testing on a
>>>>>>>>>>>             regular basis. The
>>>>>>>>>>>                             Jenkins integration will be very
>>>>>>>>>>>             useful for us, as
>>>>>>>>>>>                             most of our CI is orchestrated
>>>>>>>>>>>             by Jenkins.
>>>>>>>>>>>                             I've conferred with the rest of
>>>>>>>>>>>             the team, and we
>>>>>>>>>>>                             think it would be best to move
>>>>>>>>>>>             forward with mainly
>>>>>>>>>>>                             option B.
>>>>>>>>>>>                             If you would like to know
>>>>>>>>>>>             anything about our
>>>>>>>>>>>                             testbeds that would help you
>>>>>>>>>>>             with creating an
>>>>>>>>>>>                             example ts-rigs repo, I'd be
>>>>>>>>>>>             happy to answer any
>>>>>>>>>>>                             questions you have.
>>>>>>>>>>>
>>>>>>>>>>>                             We have multiple test rigs (we
>>>>>>>>>>>             call these
>>>>>>>>>>>                             "DUT-tester pairs") that we run
>>>>>>>>>>>             our existing
>>>>>>>>>>>                             hardware testing on, with
>>>>>>>>>>>             differing network
>>>>>>>>>>>                             hardware and CPU architecture. I
>>>>>>>>>>>             figured this might
>>>>>>>>>>>                             be an important detail.
>>>>>>>>>>>
>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>                             Adam
>>>>>>>>>>>
>>>>>>>>>>>                             On Thu, Aug 17, 2023 at 11:44 AM
>>>>>>>>>>>             Andrew Rybchenko
>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>             Greatings Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                 I'm happy to hear that
>>>>>>>>>>>             you're trying to bring
>>>>>>>>>>>                                 it up.
>>>>>>>>>>>
>>>>>>>>>>>                                 As I understand the final
>>>>>>>>>>>             goal is to run it on
>>>>>>>>>>>                                 regular basis. So, we need
>>>>>>>>>>>             to make it properly
>>>>>>>>>>>                                 from the very beginning.
>>>>>>>>>>>                                 Bring up of all features
>>>>>>>>>>>             consists of 4 steps:
>>>>>>>>>>>
>>>>>>>>>>>                                 1. Create site-specific
>>>>>>>>>>>             repository (we call it
>>>>>>>>>>>                                 ts-rigs) which contains
>>>>>>>>>>>             information about test
>>>>>>>>>>>                                 rigs and other site-specific
>>>>>>>>>>>             information like
>>>>>>>>>>>                                 where to send mails, where
>>>>>>>>>>>             to store logs etc.
>>>>>>>>>>>                                 It is required for manual
>>>>>>>>>>>             execution as well,
>>>>>>>>>>>                                 since test rigs description
>>>>>>>>>>>             is essential. I'll
>>>>>>>>>>>                                 return to the topic below.
>>>>>>>>>>>
>>>>>>>>>>>                                 2. Setup logs storage for
>>>>>>>>>>>             automated runs.
>>>>>>>>>>>             Basically it is a disk space plus apache2 web
>>>>>>>>>>>                                 server with few CGI scripts
>>>>>>>>>>>             which help a lot to
>>>>>>>>>>>                                 save disk space.
>>>>>>>>>>>
>>>>>>>>>>>                                 3. Setup Bublik web
>>>>>>>>>>>             application which provides
>>>>>>>>>>>                                 web interface to view
>>>>>>>>>>>             testing results. Same as
>>>>>>>>>>>             https://ts-factory.io/bublik
>>>>>>>>>>>             <https://ts-factory.io/bublik>
>>>>>>>>>>>             <https://ts-factory.io/bublik>
>>>>>>>>>>>
>>>>>>>>>>>                                 4. Setup Jenkins to run
>>>>>>>>>>>             tests on regularly,
>>>>>>>>>>>                                 save logs in log storage (2)
>>>>>>>>>>>             and import it to
>>>>>>>>>>>                                 bublik (3).
>>>>>>>>>>>
>>>>>>>>>>>                                 Last few month we spent on
>>>>>>>>>>>             our homework to make
>>>>>>>>>>>                                 it simpler to bring up
>>>>>>>>>>>             automated execution
>>>>>>>>>>>                                 using Jenkins -
>>>>>>>>>>>             https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>             Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>>>>             available tomorrow.
>>>>>>>>>>>
>>>>>>>>>>>                                 Let's return to the step (1).
>>>>>>>>>>>
>>>>>>>>>>>             Unfortunately there is no publicly available
>>>>>>>>>>>                                 example of the ts-rigs
>>>>>>>>>>>             repository since
>>>>>>>>>>>             sensitive site-specific information is located
>>>>>>>>>>>                                 there. But I'm ready to help
>>>>>>>>>>>             you to create it
>>>>>>>>>>>                                 for UNH. I see two options
>>>>>>>>>>>             here:
>>>>>>>>>>>
>>>>>>>>>>>                                 (A) I'll ask questions and
>>>>>>>>>>>             based on your
>>>>>>>>>>>                                 answers will create the
>>>>>>>>>>>             first draft with my
>>>>>>>>>>>             comments.
>>>>>>>>>>>
>>>>>>>>>>>                                 (B) I'll make a
>>>>>>>>>>>             template/example ts-rigs repo,
>>>>>>>>>>>                                 publish it and you'll create
>>>>>>>>>>>             UNH ts-rigs based
>>>>>>>>>>>                                 on it.
>>>>>>>>>>>
>>>>>>>>>>>                                 Of course, I'll help to
>>>>>>>>>>>             debug and finally bring
>>>>>>>>>>>                                 it up in any case.
>>>>>>>>>>>
>>>>>>>>>>>                                 (A) is a bit simpler for me
>>>>>>>>>>>             and you, but (B) is
>>>>>>>>>>>                                 a bit more generic and will
>>>>>>>>>>>             help other
>>>>>>>>>>>             potential users to bring it up.
>>>>>>>>>>>                                 We can combine (A)+(B). I.e.
>>>>>>>>>>>             start from (A).
>>>>>>>>>>>                                 What do you think?
>>>>>>>>>>>
>>>>>>>>>>>                                 Thanks,
>>>>>>>>>>>                                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                 On 8/17/23 15:18, Konstantin
>>>>>>>>>>>             Ushakov wrote:
>>>>>>>>>>>>             Greetings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Thanks for contacting us. I
>>>>>>>>>>>>             copy Andrew who
>>>>>>>>>>>>                                 would be happy to help
>>>>>>>>>>>>
>>>>>>>>>>>>             Thanks,
>>>>>>>>>>>>             Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>>>             On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>>             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 
>>>>>>>>>>>>>             Greetings Konstantin,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I am in the process of
>>>>>>>>>>>>>             setting up the DPDK
>>>>>>>>>>>>>                                 Poll Mode Driver test
>>>>>>>>>>>>>             suite as an addition to
>>>>>>>>>>>>>                                 our testing coverage for
>>>>>>>>>>>>>             DPDK at the UNH lab.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I have some questions
>>>>>>>>>>>>>             about how to set the
>>>>>>>>>>>>>                                 test suite arguments.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I have been able to
>>>>>>>>>>>>>             configure the Test Engine
>>>>>>>>>>>>>                                 to connect to the hosts in
>>>>>>>>>>>>>             the testbed. The
>>>>>>>>>>>>>                                 RCF, Configurator, and
>>>>>>>>>>>>>             Tester all begin to
>>>>>>>>>>>>>                                 run, however the prelude
>>>>>>>>>>>>>             of the test suite
>>>>>>>>>>>>>             fails to run.
>>>>>>>>>>>>>
>>>>>>>>>>>>>             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 The documentation mentions
>>>>>>>>>>>>>             that there are
>>>>>>>>>>>>>             several test parameters for the test suite,
>>>>>>>>>>>>>                                 like for the IUT test link
>>>>>>>>>>>>>             MAC, etc. These
>>>>>>>>>>>>>                                 seem like they would need
>>>>>>>>>>>>>             to be set somewhere
>>>>>>>>>>>>>                                 to run many of the tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I see in the Test Engine
>>>>>>>>>>>>>             documentation, there
>>>>>>>>>>>>>                                 are instructions on how to
>>>>>>>>>>>>>             create new
>>>>>>>>>>>>>             parameters for test suites in the Tester
>>>>>>>>>>>>>             configuration, but there is nothing in the
>>>>>>>>>>>>>                                 user guide or in the
>>>>>>>>>>>>>             Tester guide for how to
>>>>>>>>>>>>>                                 set the arguments for the
>>>>>>>>>>>>>             parameters when
>>>>>>>>>>>>>             running the test suite that I can find. I'm
>>>>>>>>>>>>>                                 not sure if I need to
>>>>>>>>>>>>>             write my own Tester
>>>>>>>>>>>>>             config, or if I should be setting these in
>>>>>>>>>>>>>                                 some other way.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 How should these values be
>>>>>>>>>>>>>             set?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I'm also not sure what
>>>>>>>>>>>>>             environment
>>>>>>>>>>>>>             variables/arguments are strictly necessary or
>>>>>>>>>>>>>             which are optional.
>>>>>>>>>>>>>
>>>>>>>>>>>>>             Regards,
>>>>>>>>>>>>>                                 Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 -- *Adam Hassick*
>>>>>>>>>>>>>             Senior Developer
>>>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>>>             ahassick@iol.unh.edu
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             --                 *Adam Hassick*
>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>             ahassick@iol.unh.edu
>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         --             *Adam Hassick*
>>>>>>>>>                         Senior Developer
>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                     --         *Adam Hassick*
>>>>>>>>>                     Senior Developer
>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 --     *Adam Hassick*
>>>>>>>>                 Senior Developer
>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>                 +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             -- 
>>>>>>>             *Adam Hassick*
>>>>>>>             Senior Developer
>>>>>>>             UNH InterOperability Lab
>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>             +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>         -- 
>>>>>         *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>         ahassick@iol.unh.edu
>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 94983 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 34163 bytes --]

Hi Andrew,

I have one additional question as well: Does the test engine support
running tests on two ARMv8 test agents?

1. We'll sort out warnings this week. Thanks for heads up.
>

Great. Let me know when that's fixed.

Support for old LTS branches was dropped some time ago, but in the future
> it is definitely possible to keep it for new LTS branches. I think 22.11 is
> supported, but I'm not sure about older LTS releases.
>

Good to know.


> 2. You can add command-line option --sanity to run tests marked with
> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
> there is a space for terminology improvement here. We'll do it.
>
> Also it takes a lot of time because of failures and tests which wait for
> some timeout.
>

That makes sense to me. We'll use the time to complete tests on virtio or
the Intel devices as a reference for how long the tests really take to
complete.
We will explore the possibility of periodically running the sanity tests
for patches.


> The test harness can provide coverage reports based on gcov, but I'm not
> sure what you mean by a "dial" to control test coverage. Provided reports
> are rather for human to analyze.
>

The general idea is to have some kind of parameter on the test suite, which
could be an integer ranging from zero to ten, that controls how many tests
are run based on how important the test is.

Similar to how some command line interfaces provide a verbosity level
parameter (some number of "-v" arguments) to control the importance of the
information in the log.
The verbosity level zero only prints very important log messages, while ten
prints everything.

In much the same manner as above, this "dial" parameter controls what tests
are run and with what parameters based on how important those tests and
test parameter combinations are.
Coverage Level zero tells the suite to run a very basic set of important
tests, with minimal parameterization. This mode would take only ~5-10
minutes to run.
In contrast, Coverage Level ten includes all the edge cases, every
combination of test parameters, everything the test suite can do, which
takes the normal several hours to run.
The values 1 - 9 are between those two extremes, allowing the user to get a
gradient of test coverage in the results and to limit the running time.

Then we could, for example, run the "run.sh" with a level of 2 or 3 for
incoming patches that need quick results, and with a level of 10 for the
less often run periodic tests performed on main or LTS branches.


> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing
> results. Unfortunately it is time consuming to fill in expectations
> database since it is necessary to analyze testing results and classify if
> it is a bug or just acceptable behaviour aspect.
>
> Bublik allows to compare results of two runs. It is useful for human, but
> still not good for automation.
>
> I have local patch for mlx5 driver which reports Tx ring size maximum. It
> makes pass rate higher. It is a problem for test harness that mlx5 does not
> report limits right now.
>
> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio
> net is 99% right now and could be done 100% easily (just one thing to fix
> in expectations).
>
> I think logs storage setup is essential for logs analysis. Of course, you
> can request HTML logs when you run tests (--log-html=html) or generate
> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
> browser, but logs storage makes it more convenient.
>

We are interested in setting up Bublik, potentially as an externally-facing
component, once we have our process of running the test suite stabilized.
Once we are able to run the test suite again, I'll see what the pass rate
is on our other hardware.
Good to know that it isn't an issue with our dev testbed causing the high
fail rate.

For Intel hardware, we have an XL710 and an Intel E810-C in our development
testbed. Although they are slightly different devices, ideally the pass
rate will be identical or similar. I have yet to set up a VM pair for
virtio, but we will soon.

Latest version of test-environment has examples of our CGI scripts which we
> use for log storage (see tools/log_server/README.md).
>
> Also all bits for Jenkins setup are available. See
> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
> ts-rigs-sample.
>

Jenkins integration, setting up production rig configurations, and
permanent log storage will be our next steps once I am able to run the
tests again.
Unless there is an easy way to have meson not pass "-Werror" into GCC. Then
I would be able to run the test suite.

Thanks,
Adam



>
> On 8/29/23 17:02, Adam Hassick wrote:
>
> Hi Andrew,
>
> That fix seems to have resolved the issue, thanks for the quick turnaround
> time on that patch.
> Now that we have the RCF timeout issue resolved, there are a few other
> questions and issues that we have about the tests themselves.
>
> 1. The test suite fails to build with a couple warnings.
>
> Below is the stderr log from compilation:
>
> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>> ../../lib/dpdk_pmd_ts.c: In function
>> ‘test_create_traffic_generator_params’:
>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
>> format arguments [-Werror=format-security]
>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>> |     ^~
>> cc1: all warnings being treated as errors
>> ninja: build stopped: subcommand failed.
>> ninja: Entering directory `.'
>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>> ../../lib/dpdk_pmd_ts.c: In function
>> ‘test_create_traffic_generator_params’:
>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
>> format arguments [-Werror=format-security]
>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>> |     ^~
>> cc1: all warnings being treated as errors
>>
>
> This error wasn't occurring last week, which was the last time I ran the
> tests.
> The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC
> v11.4.0 installed, if this information is helpful.
>
> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
> collectively take around 9 hours. Is it possible, and would it make sense,
> to lower the test coverage and have the test suite run faster?
>
> For some context, we run immediate testing on incoming patches for DPDK
> main and development branches, as well as periodic test runs on the main,
> stable, and LTS branches.
> For us to consider including this test suite as part of our immediate
> testing on patches, we would have to reduce the test coverage to the most
> important tests.
> This is primarily to reduce the testing time to, for example, less than 30
> minutes. Testing on patches can't take too long because the lab can receive
> numerous patches each day, which each require individual testing runs.
>
> At what frequency we run these tests, and on what, still needs to be
> discussed with the DPDK community, but it would be nice to know if the test
> suite had a "dial" to control the testing coverage.
>
> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300 of
> ~6,600 tests passed. Is there anything we can do to diagnose these test
> failures?
>
> Thanks,
> Adam
>
>
> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> I've pushed the fix in main branch and a new tag v1.18.1. It should solve
>> the problem with IPv6 address from DNS.
>>
>> Andrew.
>>
>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>
>> Hi Adam,
>>
>> > Does the test engine prefer to use IPv6 over IPv4 for initiating the
>> RCF connection to the test bed hosts? And if so, is there a way to force it
>> to use IPv4?
>>
>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your case,
>> I guess it is the root cause of the problem.
>> Of course, it is TE problem since I see really weird code in
>> lib/comm_net_engine/comm_net_engine.c line 135.
>>
>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>> ts-factory/test-environment repository. Please, try.
>>
>> It is late night fix with minimal testing and no review. I'll pass it
>> through review process tomorrow and
>> hopefully it will be released in one-two days.
>>
>> Andrew.
>>
>> On 8/28/23 18:02, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> We have yet to notice a distinct pattern with the failures. Sometimes,
>> the RCF will start and connect without issue a few times in a row before
>> failing to connect again. Once the issue begins to occur, neither rebooting
>> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
>> directories (suites, agents, inst) and rebooting the hosts afterward
>> resolves the issue. When it begins working again seems very arbitrary to us.
>>
>> I do usually try to terminate the test engine with Ctrl+C, but when it
>> hangs while trying to start RCF, that does not work.
>>
>> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
>> connection to the test bed hosts? And if so, is there a way to force it to
>> use IPv4?
>>
>>  - Adam
>>
>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>> Done. It works fine for me without any issues.
>>>
>>> Have you noticed any pattern when it works or does not work?
>>> May be it is a problem of not clean state after termination?
>>> Does it work fine the first time after DUTs reboot?
>>> How do you terminate testing? It should be done using Ctrl+C in terminal
>>> where you execute run.sh command.
>>>  In this case it should shutdown gracefully and close all test agents
>>> and engine applications.
>>>
>>> (I'm trying to understand why you've seen many test agent processes. It
>>> should not happen.)
>>>
>>> Andrew.
>>>
>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>
>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> Two of our systems (the Test Engine runner and the DUT host) are running
>>> Ubuntu 20.04 LTS, however this morning I noticed that the tester system
>>> (the one having issues) is running Ubuntu 22.04 LTS.
>>> This could be the source of the problem. I encountered a dependency
>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>> system. Since the tester is also the host having connection issues, I will
>>> try downgrading that system to 20.04, and see if that changes anything.
>>>
>>>
>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>
>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>> I did try passing in the "--vg-rcf" argument to the run.sh script of the
>>> test suite after installing valgrind, but there was no additional output
>>> that I saw.
>>>
>>>
>>> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
>>> where you run test engine).
>>>
>>>
>>> I will try pulling in the changes you've pushed up, and will see if that
>>> fixes anything.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hello Adam,
>>>>
>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>
>>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>>> running on test engine and agents?
>>>>
>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>> patch in review to fix it, however, the behaviour is different in
>>>> this case, so it is unlike the same problem.
>>>>
>>>>
>>>> I've just published a new tag which fixes known test engine side
>>>> problems on Debian 12 and Fedora 38.
>>>>
>>>>
>>>> One more idea is to install valgrind on the test engine host and
>>>> run with option --vg-rcf to check if something weird is happening.
>>>>
>>>> What I don't understand right now is why I see just one failed attempt
>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>
>>>> Andrew.
>>>>
>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>
>>>>  > Is there any firewall in the network or on test hosts which could
>>>> block incoming TCP connection to the port 23571
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>>>> you run test engine?
>>>>
>>>> Our test engine host and the testbed are on the same subnet. The
>>>> connection does work sometimes.
>>>>
>>>>  > If behaviour the same on the next try and you see that test agent is
>>>> kept running, could you check using
>>>>  >
>>>>  > # netstat -tnlp
>>>>  >
>>>>  > that Test Agent is listening on the port and try to establish TCP
>>>> connection from test agent using
>>>>  >
>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>  >
>>>>  > and check if TCP connection could be established.
>>>>
>>>> I was able to replicate the same behavior again, where it hangs while
>>>> RCF is trying to start.
>>>> Running this command, I see this in the output:
>>>>
>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>  18599/ta
>>>>
>>>> So it seems like it is listening on the correct port.
>>>> Additionally, I was able to connect to the Tester machine from our Test
>>>> Engine host using telnet. It printed the PID of the process once the
>>>> connection was opened.
>>>>
>>>> I tried running the "ta" application manually on the command line, and
>>>> it didn't print anything at all.
>>>> Maybe the issue is something on the Test Engine side.
>>>>
>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>     Hi Adam,
>>>>
>>>>      > On the tester host (which appears to be the Peer agent), there
>>>>     are four processes that I see running, which look like the test
>>>>     agent processes.
>>>>
>>>>     Before the next try I'd recommend to kill these processes.
>>>>
>>>>     Is there any firewall in the network or on test hosts which could
>>>>     block incoming TCP connection to the port 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>     where you run test engine?
>>>>
>>>>     If behaviour the same on the next try and you see that test agent
>>>> is
>>>>     kept running, could you check using
>>>>
>>>>     # netstat -tnlp
>>>>
>>>>     that Test Agent is listening on the port and try to establish TCP
>>>>     connection from test agent using
>>>>
>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>     and check if TCP connection could be established.
>>>>
>>>>     Another idea is to login Tester under root as testing does, get
>>>>     start TA command from the log and try it by hands without -n and
>>>>     remove extra escaping.
>>>>
>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>
>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>
>>>>     Hopefully in this case test agent directory remains in the /tmp and
>>>>     you don't need to copy it as testing does.
>>>>     May be output could shed some light on what's going on.
>>>>
>>>>     Andrew.
>>>>
>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>
>>>>     Hi Andrew,
>>>>
>>>>     This is the output that I see in the terminal when this failure
>>>>     occurs, after the test agent binaries build and the test engine
>>>>     starts:
>>>>
>>>>     Platform default build - pass
>>>>     Simple RCF consistency check succeeded
>>>>     --->>> Starting Logger...done
>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>     processes. I let it hang for around 9 minutes.
>>>>
>>>>     On the tester host (which appears to be the Peer agent), there are
>>>>     four processes that I see running, which look like the test agent
>>>>     processes.
>>>>
>>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>>
>>>>      - Adam
>>>>
>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>         Hi Adam,
>>>>
>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>         says that copy_timeout is in seconds and implementation in
>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>         Theoretically select() could be interrupted by signal, but I
>>>>         think it is unlikely here.
>>>>
>>>>         I'm not sure that I understand what do you mean by RCF
>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>         does it happen when tests are in progress, e.g. in the middle
>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>         likely either host with test agent dies or test agent itself
>>>>         crashes. It would be easier for me if classify it if you share
>>>>         text log (log.txt, full or just corresponding fragment with
>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>         depending on which agent has problems could shed some light.
>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>
>>>>         Andrew.
>>>>
>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>
>>>>         Hi Andrew,
>>>>
>>>>         I've set up a test rig repository here, and have created
>>>>         configurations for our development testbed based off of the
>>>>         examples.
>>>>         We've been able to get the test suite to run manually on
>>>>         Mellanox CX5 devices once.
>>>>         However, we are running into an issue where, when RCF starts,
>>>>         the RCF connection times out very frequently. We aren't sure
>>>>         why this is the case.
>>>>         It works sometimes, but most of the time when we try to run
>>>>         the test engine, it encounters this issue.
>>>>         I've tried changing the RCF port by setting
>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>         machines. Neither seems to fix the issue.
>>>>
>>>>         It also seems like the timeout takes far longer than 60
>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>         before I try to run the test suite.
>>>>         I assume the unit for this variable is seconds?
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>> wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             Thanks, I've cloned the example repository and will start
>>>>             setting up a configuration for our development testbed
>>>>             today. I'll let you know if I run into any difficulties
>>>>             or have any questions.
>>>>
>>>>              - Adam
>>>>
>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 I've published
>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>                 Hopefully it will help to define your test rigs and
>>>>                 successfully run some tests manually. Feel free to
>>>>                 ask any questions and I'll answer here and try to
>>>>                 update documentation.
>>>>
>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>                 (3).
>>>>                 Hopefully everything is in place for step (4), but we
>>>>                 need to make steps (2) and (3) first.
>>>>
>>>>                 Andrew.
>>>>
>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 > I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>
>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>                 almost ready right now, but I need to double-check
>>>>                 it before publishing.
>>>>
>>>>                 Regards,
>>>>                 Andrew.
>>>>
>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>
>>>>                 Hi Andrew,
>>>>
>>>>                 I'm adding the CI mailing list to this
>>>>                 conversation. Others in the community might find
>>>>                 this conversation valuable.
>>>>
>>>>                 We do want to run testing on a regular basis. The
>>>>                 Jenkins integration will be very useful for us, as
>>>>                 most of our CI is orchestrated by Jenkins.
>>>>                 I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>                 If you would like to know anything about our
>>>>                 testbeds that would help you with creating an
>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>                 questions you have.
>>>>
>>>>                 We have multiple test rigs (we call these
>>>>                 "DUT-tester pairs") that we run our existing
>>>>                 hardware testing on, with differing network
>>>>                 hardware and CPU architecture. I figured this might
>>>>                 be an important detail.
>>>>
>>>>                 Thanks,
>>>>                 Adam
>>>>
>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                     Greatings Adam,
>>>>
>>>>                     I'm happy to hear that you're trying to bring
>>>>                     it up.
>>>>
>>>>                     As I understand the final goal is to run it on
>>>>                     regular basis. So, we need to make it properly
>>>>                     from the very beginning.
>>>>                     Bring up of all features consists of 4 steps:
>>>>
>>>>                     1. Create site-specific repository (we call it
>>>>                     ts-rigs) which contains information about test
>>>>                     rigs and other site-specific information like
>>>>                     where to send mails, where to store logs etc.
>>>>                     It is required for manual execution as well,
>>>>                     since test rigs description is essential. I'll
>>>>                     return to the topic below.
>>>>
>>>>                     2. Setup logs storage for automated runs.
>>>>                     Basically it is a disk space plus apache2 web
>>>>                     server with few CGI scripts which help a lot to
>>>>                     save disk space.
>>>>
>>>>                     3. Setup Bublik web application which provides
>>>>                     web interface to view testing results. Same as
>>>>                     https://ts-factory.io/bublik
>>>>                     <https://ts-factory.io/bublik>
>>>> <https://ts-factory.io/bublik>
>>>>
>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>                     save logs in log storage (2) and import it to
>>>>                     bublik (3).
>>>>
>>>>                     Last few month we spent on our homework to make
>>>>                     it simpler to bring up automated execution
>>>>                     using Jenkins -
>>>>                     https://github.com/ts-factory/te-jenkins
>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>> <https://github.com/ts-factory/te-jenkins>
>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>                     available tomorrow.
>>>>
>>>>                     Let's return to the step (1).
>>>>
>>>>                     Unfortunately there is no publicly available
>>>>                     example of the ts-rigs repository since
>>>>                     sensitive site-specific information is located
>>>>                     there. But I'm ready to help you to create it
>>>>                     for UNH. I see two options here:
>>>>
>>>>                     (A) I'll ask questions and based on your
>>>>                     answers will create the first draft with my
>>>>                     comments.
>>>>
>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>                     publish it and you'll create UNH ts-rigs based
>>>>                     on it.
>>>>
>>>>                     Of course, I'll help to debug and finally bring
>>>>                     it up in any case.
>>>>
>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>                     a bit more generic and will help other
>>>>                     potential users to bring it up.
>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>                     What do you think?
>>>>
>>>>                     Thanks,
>>>>                     Andrew.
>>>>
>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>
>>>>                     Greetings Adam,
>>>>
>>>>
>>>>                     Thanks for contacting us. I copy Andrew who
>>>>                     would be happy to help
>>>>
>>>>                     Thanks,
>>>>                     Konstantin
>>>>
>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>                     <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu> wrote:
>>>>
>>>>                     
>>>>                     Greetings Konstantin,
>>>>
>>>>                     I am in the process of setting up the DPDK
>>>>                     Poll Mode Driver test suite as an addition to
>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>
>>>>                     I have some questions about how to set the
>>>>                     test suite arguments.
>>>>
>>>>                     I have been able to configure the Test Engine
>>>>                     to connect to the hosts in the testbed. The
>>>>                     RCF, Configurator, and Tester all begin to
>>>>                     run, however the prelude of the test suite
>>>>                     fails to run.
>>>>
>>>>
>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>
>>>>                     The documentation mentions that there are
>>>>                     several test parameters for the test suite,
>>>>                     like for the IUT test link MAC, etc. These
>>>>                     seem like they would need to be set somewhere
>>>>                     to run many of the tests.
>>>>
>>>>                     I see in the Test Engine documentation, there
>>>>                     are instructions on how to create new
>>>>                     parameters for test suites in the Tester
>>>>                     configuration, but there is nothing in the
>>>>                     user guide or in the Tester guide for how to
>>>>                     set the arguments for the parameters when
>>>>                     running the test suite that I can find. I'm
>>>>                     not sure if I need to write my own Tester
>>>>                     config, or if I should be setting these in
>>>>                     some other way.
>>>>
>>>>                     How should these values be set?
>>>>
>>>>                     I'm also not sure what environment
>>>>                     variables/arguments are strictly necessary or
>>>>                     which are optional.
>>>>
>>>>                     Regards,
>>>>                     Adam
>>>>
>>>>                     --                     *Adam Hassick*
>>>>                     Senior Developer
>>>>                     UNH InterOperability Lab
>>>>                     ahassick@iol.unh.edu
>>>>                     <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>                     +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>                 --                 *Adam Hassick*
>>>>                 Senior Developer
>>>>                 UNH InterOperability Lab
>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>                 +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>             --             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>     --     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

[-- Attachment #2: Type: text/html, Size: 90038 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 51510 bytes --]

Hi Adam,

On 8/31/23 22:38, Adam Hassick wrote:
> Hi Andrew,
>
> I have one additional question as well: Does the test engine support 
> running tests on two ARMv8 test agents?
>
>     1. We'll sort out warnings this week. Thanks for heads up.
>
>
> Great. Let me know when that's fixed.

Done. We also fixed a number of warnings in TE.
Also we fixed root test package name to be consistent with the 
repository name.

>     Support for old LTS branches was dropped some time ago, but in the
>     future it is definitely possible to keep it for new LTS branches.
>     I think 22.11 is supported, but I'm not sure about older LTS releases.
>
>
> Good to know.
>
>     2. You can add command-line option --sanity to run tests marked
>     with TEST_HARNESS_SANITY requirement (see
>     dpdk-ethdev-ts/scripts/run.sh and grep TEST_HARNESS_SANITY
>     dpdk-ethdev-ts to see which tests are marked). Yes, there is a
>     space for terminology improvement here. We'll do it.
>

Done. Now it is called --checkup.

>
>     Also it takes a lot of time because of failures and tests which
>     wait for some timeout.
>
>
> That makes sense to me. We'll use the time to complete tests on virtio 
> or the Intel devices as a reference for how long the tests really take 
> to complete.
> We will explore the possibility of periodically running the sanity 
> tests for patches.

I'll double-check and let you know how long entire TS runs on Intel 
X710, E810, Mellanox CX5 and virtio net. Just to ensure that time 
observed in your case looks the same.

>
>     The test harness can provide coverage reports based on gcov, but
>     I'm not sure what you mean by a "dial" to control test coverage.
>     Provided reports are rather for human to analyze.
>
>
> The general idea is to have some kind of parameter on the test suite, 
> which could be an integer ranging from zero to ten, that controls how 
> many tests are run based on how important the test is.
>
> Similar to how some command line interfaces provide a verbosity level 
> parameter (some number of "-v" arguments) to control the importance of 
> the information in the log.
> The verbosity level zero only prints very important log messages, 
> while ten prints everything.
>
> In much the same manner as above, this "dial" parameter controls what 
> tests are run and with what parameters based on how important those 
> tests and test parameter combinations are.
> Coverage Level zero tells the suite to run a very basic set of 
> important tests, with minimal parameterization. This mode would take 
> only ~5-10 minutes to run.
> In contrast, Coverage Level ten includes all the edge cases, every 
> combination of test parameters, everything the test suite can do, 
> which takes the normal several hours to run.
> The values 1 - 9 are between those two extremes, allowing the user to 
> get a gradient of test coverage in the results and to limit the 
> running time.
>
> Then we could, for example, run the "run.sh" with a level of 2 or 3 
> for incoming patches that need quick results, and with a level of 10 
> for the less often run periodic tests performed on main or LTS branches.

Understood now. Thanks a lot for the idea. We'll discuss it and come back.

>     3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>     testing results. Unfortunately it is time consuming to fill in
>     expectations database since it is necessary to analyze testing
>     results and classify if it is a bug or just acceptable behaviour
>     aspect.
>
>     Bublik allows to compare results of two runs. It is useful for
>     human, but still not good for automation.
>
>     I have local patch for mlx5 driver which reports Tx ring size
>     maximum. It makes pass rate higher. It is a problem for test
>     harness that mlx5 does not report limits right now.
>
>     Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>     virtio net is 99% right now and could be done 100% easily (just
>     one thing to fix in expectations).
>
>     I think logs storage setup is essential for logs analysis. Of
>     course, you can request HTML logs when you run tests
>     (--log-html=html) or generate after run using
>     dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>     browser, but logs storage makes it more convenient.
>
>
> We are interested in setting up Bublik, potentially as an 
> externally-facing component, once we have our process of running the 
> test suite stabilized.
> Once we are able to run the test suite again, I'll see what the pass 
> rate is on our other hardware.
> Good to know that it isn't an issue with our dev testbed causing the 
> high fail rate.
>
> For Intel hardware, we have an XL710 and an Intel E810-C in our 
> development testbed. Although they are slightly different devices, 
> ideally the pass rate will be identical or similar. I have yet to set 
> up a VM pair for virtio, but we will soon.
>
>     Latest version of test-environment has examples of our CGI scripts
>     which we use for log storage (see tools/log_server/README.md).
>
>     Also all bits for Jenkins setup are available. See
>     dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>     ts-rigs-sample.
>
>
> Jenkins integration, setting up production rig configurations, and 
> permanent log storage will be our next steps once I am able to run the 
> tests again.
> Unless there is an easy way to have meson not pass "-Werror" into GCC. 
> Then I would be able to run the test suite.

Hopefully it is resolved now.

I thought a bit more about your usecase for Jenkins. I'm not 100% sure 
that existing pipelines are convenient for your usecase.
Fill free to ask questions when you are on it.

Thanks,
Andrew.

>
> Thanks,
> Adam
>
>
>     On 8/29/23 17:02, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     That fix seems to have resolved the issue, thanks for the quick
>>     turnaround time on that patch.
>>     Now that we have the RCF timeout issue resolved, there are a few
>>     other questions and issues that we have about the tests themselves.
>>
>>     1. The test suite fails to build with a couple warnings.
>>
>>     Below is the stderr log from compilation:
>>
>>         FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>         cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>         -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>>         -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>         -fPIC -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>         -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>>         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>         ../../lib/dpdk_pmd_ts.c
>>         ../../lib/dpdk_pmd_ts.c: In function
>>         ‘test_create_traffic_generator_params’:
>>         ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>         literal and no format arguments [-Werror=format-security]
>>         5577 |     rc = te_kvpair_add(result, buf, mode);
>>         |     ^~
>>         cc1: all warnings being treated as errors
>>         ninja: build stopped: subcommand failed.
>>         ninja: Entering directory `.'
>>         FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>         cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>         -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>>         -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>         -fPIC -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>         -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>>         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>         ../../lib/dpdk_pmd_ts.c
>>         ../../lib/dpdk_pmd_ts.c: In function
>>         ‘test_create_traffic_generator_params’:
>>         ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>         literal and no format arguments [-Werror=format-security]
>>         5577 |     rc = te_kvpair_add(result, buf, mode);
>>         |     ^~
>>         cc1: all warnings being treated as errors
>>
>>
>>     This error wasn't occurring last week, which was the last time I
>>     ran the tests.
>>     The TE host and the DUT have GCC v9.4.0 installed, and the tester
>>     has GCC v11.4.0 installed, if this information is helpful.
>>
>>     2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>     collectively take around 9 hours. Is it possible, and would it
>>     make sense, to lower the test coverage and have the test suite
>>     run faster?
>>
>>     For some context, we run immediate testing on incoming patches
>>     for DPDK main and development branches, as well as periodic test
>>     runs on the main, stable, and LTS branches.
>>     For us to consider including this test suite as part of our
>>     immediate testing on patches, we would have to reduce the test
>>     coverage to the most important tests.
>>     This is primarily to reduce the testing time to, for example,
>>     less than 30 minutes. Testing on patches can't take too long
>>     because the lab can receive numerous patches each day, which each
>>     require individual testing runs.
>>
>>     At what frequency we run these tests, and on what, still needs to
>>     be discussed with the DPDK community, but it would be nice to
>>     know if the test suite had a "dial" to control the testing coverage.
>>
>>     3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>     2,300 of ~6,600 tests passed. Is there anything we can do to
>>     diagnose these test failures?
>>
>>     Thanks,
>>     Adam
>>
>>
>>     On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         Hi Adam,
>>
>>         I've pushed the fix in main branch and a new tag v1.18.1. It
>>         should solve the problem with IPv6 address from DNS.
>>
>>         Andrew.
>>
>>         On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>         Hi Adam,
>>>
>>>         > Does the test engine prefer to use IPv6 over IPv4 for
>>>         initiating the RCF connection to the test bed hosts? And if
>>>         so, is there a way to force it to use IPv4?
>>>
>>>         Brilliant idea. If DNS returns both IPv4 and IPv6 addresses
>>>         in your case, I guess it is the root cause of the problem.
>>>         Of course, it is TE problem since I see really weird code in
>>>         lib/comm_net_engine/comm_net_engine.c line 135.
>>>
>>>         I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>         ts-factory/test-environment repository. Please, try.
>>>
>>>         It is late night fix with minimal testing and no review.
>>>         I'll pass it through review process tomorrow and
>>>         hopefully it will be released in one-two days.
>>>
>>>         Andrew.
>>>
>>>         On 8/28/23 18:02, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         We have yet to notice a distinct pattern with the failures.
>>>>         Sometimes, the RCF will start and connect without issue a
>>>>         few times in a row before failing to connect again. Once
>>>>         the issue begins to occur, neither rebooting all of the
>>>>         hosts (test engine VM, tester, IUT) or deleting all of the
>>>>         build directories (suites, agents, inst) and rebooting the
>>>>         hosts afterward resolves the issue. When it begins working
>>>>         again seems very arbitrary to us.
>>>>
>>>>         I do usually try to terminate the test engine with Ctrl+C,
>>>>         but when it hangs while trying to start RCF, that does not
>>>>         work.
>>>>
>>>>         Does the test engine prefer to use IPv6 over IPv4 for
>>>>         initiating the RCF connection to the test bed hosts? And if
>>>>         so, is there a way to force it to use IPv4?
>>>>
>>>>          - Adam
>>>>
>>>>         On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko
>>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>             > I'll double-check test engine on Ubuntu 20.04 and
>>>>             Ubuntu 22.04.
>>>>
>>>>             Done. It works fine for me without any issues.
>>>>
>>>>             Have you noticed any pattern when it works or does not
>>>>             work?
>>>>             May be it is a problem of not clean state after
>>>>             termination?
>>>>             Does it work fine the first time after DUTs reboot?
>>>>             How do you terminate testing? It should be done using
>>>>             Ctrl+C in terminal where you execute run.sh command.
>>>>              In this case it should shutdown gracefully and close
>>>>             all test agents and engine applications.
>>>>
>>>>             (I'm trying to understand why you've seen many test
>>>>             agent processes. It should not happen.)
>>>>
>>>>             Andrew.
>>>>
>>>>             On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>             On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>             Hi Andrew,
>>>>>>
>>>>>>             Two of our systems (the Test Engine runner and the
>>>>>>             DUT host) are running Ubuntu 20.04 LTS, however this
>>>>>>             morning I noticed that the tester system (the one
>>>>>>             having issues) is running Ubuntu 22.04 LTS.
>>>>>>             This could be the source of the problem. I
>>>>>>             encountered a dependency issue trying to run the Test
>>>>>>             Engine on 22.04 LTS, so I downgraded the system.
>>>>>>             Since the tester is also the host having connection
>>>>>>             issues, I will try downgrading that system to 20.04,
>>>>>>             and see if that changes anything.
>>>>>
>>>>>             Unlikely, but who knows. We run tests (DUTs) on Ubuntu
>>>>>             20.04, Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04,
>>>>>             Debian 11 and Fedora 38 every night.
>>>>>             Right now Debian 11 is used for test engine in nightly
>>>>>             regressions.
>>>>>
>>>>>             I'll double-check test engine on Ubuntu 20.04 and
>>>>>             Ubuntu 22.04.
>>>>>
>>>>>>             I did try passing in the "--vg-rcf" argument to the
>>>>>>             run.sh script of the test suite after installing
>>>>>>             valgrind, but there was no additional output that I saw.
>>>>>
>>>>>             Sorry, I should valgrind output should be in
>>>>>             valgrind.te_rcf (direction where you run test engine).
>>>>>
>>>>>>
>>>>>>             I will try pulling in the changes you've pushed up,
>>>>>>             and will see if that fixes anything.
>>>>>>
>>>>>>             Thanks,
>>>>>>             Adam
>>>>>>
>>>>>>             On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                 Hello Adam,
>>>>>>
>>>>>>                 On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>                 I'd like to try to repeat the problem locally.
>>>>>>>                 Which Linux distro is running on test engine and
>>>>>>>                 agents?
>>>>>>>
>>>>>>>                 In fact I know one problem with Debian 12 and
>>>>>>>                 Fedora 38 and we have
>>>>>>>                 patch in review to fix it, however, the
>>>>>>>                 behaviour is different in
>>>>>>>                 this case, so it is unlike the same problem.
>>>>>>
>>>>>>                 I've just published a new tag which fixes known
>>>>>>                 test engine side problems on Debian 12 and Fedora 38.
>>>>>>
>>>>>>>
>>>>>>>                 One more idea is to install valgrind on the test
>>>>>>>                 engine host and
>>>>>>>                 run with option --vg-rcf to check if something
>>>>>>>                 weird is happening.
>>>>>>>
>>>>>>>                 What I don't understand right now is why I see
>>>>>>>                 just one failed attempt
>>>>>>>                 to connect in your log.txt and then Logger
>>>>>>>                 shutdown after 9 minutes.
>>>>>>>
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>                  > Is there any firewall in the network or on
>>>>>>>>                 test hosts which could block incoming TCP
>>>>>>>>                 connection to the port 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 from the host where you run test engine?
>>>>>>>>
>>>>>>>>                 Our test engine host and the testbed are on the
>>>>>>>>                 same subnet. The connection does work sometimes.
>>>>>>>>
>>>>>>>>                  > If behaviour the same on the next try and
>>>>>>>>                 you see that test agent is kept running, could
>>>>>>>>                 you check using
>>>>>>>>                  >
>>>>>>>>                  > # netstat -tnlp
>>>>>>>>                  >
>>>>>>>>                  > that Test Agent is listening on the port and
>>>>>>>>                 try to establish TCP connection from test agent
>>>>>>>>                 using
>>>>>>>>                  >
>>>>>>>>                  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                  >
>>>>>>>>                  > and check if TCP connection could be
>>>>>>>>                 established.
>>>>>>>>
>>>>>>>>                 I was able to replicate the same behavior
>>>>>>>>                 again, where it hangs while RCF is trying to
>>>>>>>>                 start.
>>>>>>>>                 Running this command, I see this in the output:
>>>>>>>>
>>>>>>>>                 tcp        0      0 0.0.0.0:23571
>>>>>>>>                 <http://0.0.0.0:23571> <http://0.0.0.0:23571>
>>>>>>>>                 <http://0.0.0.0:23571> 0.0.0.0:* LISTEN    
>>>>>>>>                  18599/ta
>>>>>>>>
>>>>>>>>                 So it seems like it is listening on the correct
>>>>>>>>                 port.
>>>>>>>>                 Additionally, I was able to connect to the
>>>>>>>>                 Tester machine from our Test Engine host using
>>>>>>>>                 telnet. It printed the PID of the process once
>>>>>>>>                 the connection was opened.
>>>>>>>>
>>>>>>>>                 I tried running the "ta" application manually
>>>>>>>>                 on the command line, and it didn't print
>>>>>>>>                 anything at all.
>>>>>>>>                 Maybe the issue is something on the Test Engine
>>>>>>>>                 side.
>>>>>>>>
>>>>>>>>                 On Thu, Aug 24, 2023 at 2:35 PM Andrew
>>>>>>>>                 Rybchenko <andrew.rybchenko@oktetlabs.ru
>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                     Hi Adam,
>>>>>>>>
>>>>>>>>                      > On the tester host (which appears to be
>>>>>>>>                 the Peer agent), there
>>>>>>>>                     are four processes that I see running,
>>>>>>>>                 which look like the test
>>>>>>>>                     agent processes.
>>>>>>>>
>>>>>>>>                     Before the next try I'd recommend to kill
>>>>>>>>                 these processes.
>>>>>>>>
>>>>>>>>                     Is there any firewall in the network or on
>>>>>>>>                 test hosts which could
>>>>>>>>                     block incoming TCP connection to the port
>>>>>>>>                 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 from the host
>>>>>>>>                     where you run test engine?
>>>>>>>>
>>>>>>>>                     If behaviour the same on the next try and
>>>>>>>>                 you see that test agent is
>>>>>>>>                     kept running, could you check using
>>>>>>>>
>>>>>>>>                     # netstat -tnlp
>>>>>>>>
>>>>>>>>                     that Test Agent is listening on the port
>>>>>>>>                 and try to establish TCP
>>>>>>>>                     connection from test agent using
>>>>>>>>
>>>>>>>>                     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>
>>>>>>>>                     and check if TCP connection could be
>>>>>>>>                 established.
>>>>>>>>
>>>>>>>>                     Another idea is to login Tester under root
>>>>>>>>                 as testing does, get
>>>>>>>>                     start TA command from the log and try it by
>>>>>>>>                 hands without -n and
>>>>>>>>                     remove extra escaping.
>>>>>>>>
>>>>>>>>                     # sudo
>>>>>>>>                 PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>
>>>>>>>>                 LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>                 /tmp/linux_x86_root_76872_1692885663_1/ta Peer
>>>>>>>>                 23571
>>>>>>>>                 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>
>>>>>>>>                     Hopefully in this case test agent directory
>>>>>>>>                 remains in the /tmp and
>>>>>>>>                     you don't need to copy it as testing does.
>>>>>>>>                     May be output could shed some light on
>>>>>>>>                 what's going on.
>>>>>>>>
>>>>>>>>                     Andrew.
>>>>>>>>
>>>>>>>>                     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>                 Hi Andrew,
>>>>>>>>>
>>>>>>>>>                     This is the output that I see in the
>>>>>>>>>                 terminal when this failure
>>>>>>>>>                     occurs, after the test agent binaries
>>>>>>>>>                 build and the test engine
>>>>>>>>>                     starts:
>>>>>>>>>
>>>>>>>>>                     Platform default build - pass
>>>>>>>>>                     Simple RCF consistency check succeeded
>>>>>>>>>                     --->>> Starting Logger...done
>>>>>>>>>                     --->>> Starting
>>>>>>>>>                 RCF...rcf_net_engine_connect(): Connection timed
>>>>>>>>>                     out
>>>>>>>>>                 iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>
>>>>>>>>>                     Then, it hangs here until I kill the
>>>>>>>>>                 "te_rcf" and "te_tee"
>>>>>>>>>                     processes. I let it hang for around 9
>>>>>>>>>                 minutes.
>>>>>>>>>
>>>>>>>>>                     On the tester host (which appears to be
>>>>>>>>>                 the Peer agent), there are
>>>>>>>>>                     four processes that I see running, which
>>>>>>>>>                 look like the test agent
>>>>>>>>>                     processes.
>>>>>>>>>
>>>>>>>>>                     ta.Peer is an empty file. I've attached
>>>>>>>>>                 the log.txt from this run.
>>>>>>>>>
>>>>>>>>>                      - Adam
>>>>>>>>>
>>>>>>>>>                     On Thu, Aug 24, 2023 at 4:22 AM Andrew
>>>>>>>>>                 Rybchenko
>>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                         Hi Adam,
>>>>>>>>>
>>>>>>>>>                         Yes, TE_RCFUNIX_TIMEOUT is in seconds.
>>>>>>>>>                 I've double-checked
>>>>>>>>>                         that it goes to 'copy_timeout' in
>>>>>>>>>                 ts-conf/rcf.conf.
>>>>>>>>>                         Description in in
>>>>>>>>>                 doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>                         says that copy_timeout is in seconds
>>>>>>>>>                 and implementation in
>>>>>>>>>                 lib/rcfunix/rcfunix.c passes the value to
>>>>>>>>>                 select() tv_sec.
>>>>>>>>>                         Theoretically select() could be
>>>>>>>>>                 interrupted by signal, but I
>>>>>>>>>                         think it is unlikely here.
>>>>>>>>>
>>>>>>>>>                         I'm not sure that I understand what do
>>>>>>>>>                 you mean by RCF
>>>>>>>>>                         connection timeout. Does it happen on
>>>>>>>>>                 TE startup when RCF
>>>>>>>>>                         starts test agents. If so,
>>>>>>>>>                 TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>>>>                         does it happen when tests are in
>>>>>>>>>                 progress, e.g. in the middle
>>>>>>>>>                         of a test. If so, TE_RCFUNIX_TIMEOUT
>>>>>>>>>                 is unrelated and most
>>>>>>>>>                         likely either host with test agent
>>>>>>>>>                 dies or test agent itself
>>>>>>>>>                         crashes. It would be easier for me if
>>>>>>>>>                 classify it if you share
>>>>>>>>>                         text log (log.txt, full or just
>>>>>>>>>                 corresponding fragment with
>>>>>>>>>                         some context). Also content of ta.DPDK
>>>>>>>>>                 or ta.Peer file
>>>>>>>>>                         depending on which agent has problems
>>>>>>>>>                 could shed some light.
>>>>>>>>>                         Corresponding files contain
>>>>>>>>>                 stdout/stderr of test agents.
>>>>>>>>>
>>>>>>>>>                         Andrew.
>>>>>>>>>
>>>>>>>>>                         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                         I've set up a test rig repository
>>>>>>>>>>                 here, and have created
>>>>>>>>>>                 configurations for our development testbed
>>>>>>>>>>                 based off of the
>>>>>>>>>>                         examples.
>>>>>>>>>>                         We've been able to get the test suite
>>>>>>>>>>                 to run manually on
>>>>>>>>>>                         Mellanox CX5 devices once.
>>>>>>>>>>                         However, we are running into an issue
>>>>>>>>>>                 where, when RCF starts,
>>>>>>>>>>                         the RCF connection times out very
>>>>>>>>>>                 frequently. We aren't sure
>>>>>>>>>>                         why this is the case.
>>>>>>>>>>                         It works sometimes, but most of the
>>>>>>>>>>                 time when we try to run
>>>>>>>>>>                         the test engine, it encounters this
>>>>>>>>>>                 issue.
>>>>>>>>>>                         I've tried changing the RCF port by
>>>>>>>>>>                 setting
>>>>>>>>>>                 "TE_RCF_PORT=<some port number>" and
>>>>>>>>>>                 rebooting the testbed
>>>>>>>>>>                         machines. Neither seems to fix the
>>>>>>>>>>                 issue.
>>>>>>>>>>
>>>>>>>>>>                         It also seems like the timeout takes
>>>>>>>>>>                 far longer than 60
>>>>>>>>>>                         seconds, even when running "export
>>>>>>>>>>                 TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>                         before I try to run the test suite.
>>>>>>>>>>                         I assume the unit for this variable
>>>>>>>>>>                 is seconds?
>>>>>>>>>>
>>>>>>>>>>                         Thanks,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         On Mon, Aug 21, 2023 at 10:19 AM Adam
>>>>>>>>>>                 Hassick
>>>>>>>>>>                         <ahassick@iol.unh.edu
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>>>>
>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                             Thanks, I've cloned the example
>>>>>>>>>>                 repository and will start
>>>>>>>>>>                             setting up a configuration for
>>>>>>>>>>                 our development testbed
>>>>>>>>>>                             today. I'll let you know if I run
>>>>>>>>>>                 into any difficulties
>>>>>>>>>>                             or have any questions.
>>>>>>>>>>
>>>>>>>>>>                              - Adam
>>>>>>>>>>
>>>>>>>>>>                             On Sun, Aug 20, 2023 at 4:40 AM
>>>>>>>>>>                 Andrew Rybchenko
>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>                                 Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                                 I've published
>>>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>                 Hopefully it will help to define your test
>>>>>>>>>>                 rigs and
>>>>>>>>>>                 successfully run some tests manually. Feel
>>>>>>>>>>                 free to
>>>>>>>>>>                                 ask any questions and I'll
>>>>>>>>>>                 answer here and try to
>>>>>>>>>>                 update documentation.
>>>>>>>>>>
>>>>>>>>>>                 Meanwhile I'll prepare missing bits for steps
>>>>>>>>>>                 (2) and
>>>>>>>>>>                                 (3).
>>>>>>>>>>                 Hopefully everything is in place for step
>>>>>>>>>>                 (4), but we
>>>>>>>>>>                                 need to make steps (2) and
>>>>>>>>>>                 (3) first.
>>>>>>>>>>
>>>>>>>>>>                 Andrew.
>>>>>>>>>>
>>>>>>>>>>                                 On 8/18/23 21:40, Andrew
>>>>>>>>>>                 Rybchenko wrote:
>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                 > I've conferred with the rest of the team,
>>>>>>>>>>>                 and we
>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>                 mainly
>>>>>>>>>>>                 option B.
>>>>>>>>>>>
>>>>>>>>>>>                 OK, I'll provide the sample on Monday for
>>>>>>>>>>>                 you. It is
>>>>>>>>>>>                 almost ready right now, but I need to
>>>>>>>>>>>                 double-check
>>>>>>>>>>>                                 it before publishing.
>>>>>>>>>>>
>>>>>>>>>>>                 Regards,
>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                 On 8/17/23 20:03, Adam
>>>>>>>>>>>                 Hassick wrote:
>>>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>>>>>                 conversation. Others in the community might
>>>>>>>>>>>>                 find
>>>>>>>>>>>>                 this conversation valuable.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We do want to run testing on a regular
>>>>>>>>>>>>                 basis. The
>>>>>>>>>>>>                 Jenkins integration will be very useful for
>>>>>>>>>>>>                 us, as
>>>>>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>>>>>                 I've conferred with the rest of the team,
>>>>>>>>>>>>                 and we
>>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>>                 mainly
>>>>>>>>>>>>                 option B.
>>>>>>>>>>>>                 If you would like to know anything about our
>>>>>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>>>>>                 example ts-rigs repo, I'd be happy to
>>>>>>>>>>>>                 answer any
>>>>>>>>>>>>                 questions you have.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>>>>>                 hardware and CPU architecture. I figured
>>>>>>>>>>>>                 this might
>>>>>>>>>>>>                 be an important detail.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Thanks,
>>>>>>>>>>>>                 Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>>>>>>                 Rybchenko
>>>>>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Greatings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                     I'm happy to hear that
>>>>>>>>>>>>                 you're trying to bring
>>>>>>>>>>>>                                     it up.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     As I understand the
>>>>>>>>>>>>                 final goal is to run it on
>>>>>>>>>>>>                                     regular basis. So, we
>>>>>>>>>>>>                 need to make it properly
>>>>>>>>>>>>                                     from the very beginning.
>>>>>>>>>>>>                                     Bring up of all
>>>>>>>>>>>>                 features consists of 4 steps:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     1. Create site-specific
>>>>>>>>>>>>                 repository (we call it
>>>>>>>>>>>>                                     ts-rigs) which contains
>>>>>>>>>>>>                 information about test
>>>>>>>>>>>>                                     rigs and other
>>>>>>>>>>>>                 site-specific information like
>>>>>>>>>>>>                                     where to send mails,
>>>>>>>>>>>>                 where to store logs etc.
>>>>>>>>>>>>                                     It is required for
>>>>>>>>>>>>                 manual execution as well,
>>>>>>>>>>>>                                     since test rigs
>>>>>>>>>>>>                 description is essential. I'll
>>>>>>>>>>>>                                     return to the topic below.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     2. Setup logs storage
>>>>>>>>>>>>                 for automated runs.
>>>>>>>>>>>>                                     Basically it is a disk
>>>>>>>>>>>>                 space plus apache2 web
>>>>>>>>>>>>                                     server with few CGI
>>>>>>>>>>>>                 scripts which help a lot to
>>>>>>>>>>>>                                     save disk space.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     3. Setup Bublik web
>>>>>>>>>>>>                 application which provides
>>>>>>>>>>>>                                     web interface to view
>>>>>>>>>>>>                 testing results. Same as
>>>>>>>>>>>>                 https://ts-factory.io/bublik
>>>>>>>>>>>>                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>
>>>>>>>>>>>>                                     4. Setup Jenkins to run
>>>>>>>>>>>>                 tests on regularly,
>>>>>>>>>>>>                                     save logs in log
>>>>>>>>>>>>                 storage (2) and import it to
>>>>>>>>>>>>                                     bublik (3).
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Last few month we spent
>>>>>>>>>>>>                 on our homework to make
>>>>>>>>>>>>                                     it simpler to bring up
>>>>>>>>>>>>                 automated execution
>>>>>>>>>>>>                                     using Jenkins -
>>>>>>>>>>>>                 https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>                                     Corresponding bits in
>>>>>>>>>>>>                 dpdk-ethdev-ts will be
>>>>>>>>>>>>                                     available tomorrow.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Let's return to the
>>>>>>>>>>>>                 step (1).
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Unfortunately there is
>>>>>>>>>>>>                 no publicly available
>>>>>>>>>>>>                                     example of the ts-rigs
>>>>>>>>>>>>                 repository since
>>>>>>>>>>>>                                     sensitive site-specific
>>>>>>>>>>>>                 information is located
>>>>>>>>>>>>                                     there. But I'm ready to
>>>>>>>>>>>>                 help you to create it
>>>>>>>>>>>>                                     for UNH. I see two
>>>>>>>>>>>>                 options here:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     (A) I'll ask questions
>>>>>>>>>>>>                 and based on your
>>>>>>>>>>>>                                     answers will create the
>>>>>>>>>>>>                 first draft with my
>>>>>>>>>>>>                                     comments.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     (B) I'll make a
>>>>>>>>>>>>                 template/example ts-rigs repo,
>>>>>>>>>>>>                                     publish it and you'll
>>>>>>>>>>>>                 create UNH ts-rigs based
>>>>>>>>>>>>                                     on it.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Of course, I'll help to
>>>>>>>>>>>>                 debug and finally bring
>>>>>>>>>>>>                                     it up in any case.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     (A) is a bit simpler
>>>>>>>>>>>>                 for me and you, but (B) is
>>>>>>>>>>>>                                     a bit more generic and
>>>>>>>>>>>>                 will help other
>>>>>>>>>>>>                                     potential users to
>>>>>>>>>>>>                 bring it up.
>>>>>>>>>>>>                                     We can combine (A)+(B).
>>>>>>>>>>>>                 I.e. start from (A).
>>>>>>>>>>>>                                     What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On 8/17/23 15:18,
>>>>>>>>>>>>                 Konstantin Ushakov wrote:
>>>>>>>>>>>>>                 Greetings Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Thanks for contacting
>>>>>>>>>>>>>                 us. I copy Andrew who
>>>>>>>>>>>>>                                     would be happy to help
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>>                                     Konstantin
>>>>>>>>>>>>>
>>>>>>>>>>>>>>                 On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>>>                 <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     
>>>>>>>>>>>>>>                                     Greetings Konstantin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I am in the process
>>>>>>>>>>>>>>                 of setting up the DPDK
>>>>>>>>>>>>>>                                     Poll Mode Driver test
>>>>>>>>>>>>>>                 suite as an addition to
>>>>>>>>>>>>>>                                     our testing coverage
>>>>>>>>>>>>>>                 for DPDK at the UNH lab.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I have some questions
>>>>>>>>>>>>>>                 about how to set the
>>>>>>>>>>>>>>                                     test suite arguments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I have been able to
>>>>>>>>>>>>>>                 configure the Test Engine
>>>>>>>>>>>>>>                                     to connect to the
>>>>>>>>>>>>>>                 hosts in the testbed. The
>>>>>>>>>>>>>>                                     RCF, Configurator,
>>>>>>>>>>>>>>                 and Tester all begin to
>>>>>>>>>>>>>>                                     run, however the
>>>>>>>>>>>>>>                 prelude of the test suite
>>>>>>>>>>>>>>                                     fails to run.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                 https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     The documentation
>>>>>>>>>>>>>>                 mentions that there are
>>>>>>>>>>>>>>                                     several test
>>>>>>>>>>>>>>                 parameters for the test suite,
>>>>>>>>>>>>>>                                     like for the IUT test
>>>>>>>>>>>>>>                 link MAC, etc. These
>>>>>>>>>>>>>>                                     seem like they would
>>>>>>>>>>>>>>                 need to be set somewhere
>>>>>>>>>>>>>>                                     to run many of the
>>>>>>>>>>>>>>                 tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I see in the Test
>>>>>>>>>>>>>>                 Engine documentation, there
>>>>>>>>>>>>>>                                     are instructions on
>>>>>>>>>>>>>>                 how to create new
>>>>>>>>>>>>>>                                     parameters for test
>>>>>>>>>>>>>>                 suites in the Tester
>>>>>>>>>>>>>>                                     configuration, but
>>>>>>>>>>>>>>                 there is nothing in the
>>>>>>>>>>>>>>                                     user guide or in the
>>>>>>>>>>>>>>                 Tester guide for how to
>>>>>>>>>>>>>>                                     set the arguments for
>>>>>>>>>>>>>>                 the parameters when
>>>>>>>>>>>>>>                                     running the test
>>>>>>>>>>>>>>                 suite that I can find. I'm
>>>>>>>>>>>>>>                                     not sure if I need to
>>>>>>>>>>>>>>                 write my own Tester
>>>>>>>>>>>>>>                                     config, or if I
>>>>>>>>>>>>>>                 should be setting these in
>>>>>>>>>>>>>>                                     some other way.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     How should these
>>>>>>>>>>>>>>                 values be set?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I'm also not sure
>>>>>>>>>>>>>>                 what environment
>>>>>>>>>>>>>>                                     variables/arguments
>>>>>>>>>>>>>>                 are strictly necessary or
>>>>>>>>>>>>>>                                     which are optional.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     Regards,
>>>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     --
>>>>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 -- *Adam Hassick*
>>>>>>>>>>>>                 Senior Developer
>>>>>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             --             *Adam Hassick*
>>>>>>>>>>                             Senior Developer
>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                     --     *Adam Hassick*
>>>>>>>>>                     Senior Developer
>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 -- 
>>>>>>>>                 *Adam Hassick*
>>>>>>>>                 Senior Developer
>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>                 +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             -- 
>>>>>>             *Adam Hassick*
>>>>>>             Senior Developer
>>>>>>             UNH InterOperability Lab
>>>>>>             ahassick@iol.unh.edu
>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>             +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>         -- 
>>>>         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 115388 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1.1: Type: text/plain, Size: 37124 bytes --]

Hi Andrew,

The compilation warning issue is now resolved. Again, thank you guys for
fixing this for us. I can run the tests on the Mellanox CX5s again, however
I'm running into a couple new issues with running the prologues on the
Intel cards.

When running testing on the Intel XL710s, I see this error appear in the
log:

ERROR  prologue  Environment LIB  14:16:13.650
> Too few networks in available configuration (0) in comparison with
> required (1)
>

This seems like a trivial configuration error, perhaps this is something I
need to set up in ts-rigs. I briefly searched through the examples there
and didn't see any mention of how to set up a network.
I will attach this log just in case you need more information.

There is a different error when running on the Intel E810s. It appears to
me like it starts DPDK, does some configuration inside DPDK and on the
device, and then fails to bring the device back up. Since this error seems
very non-trivial, I will also attach this log.

Thanks,
Adam

On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> On 8/31/23 22:38, Adam Hassick wrote:
>
> Hi Andrew,
>
> I have one additional question as well: Does the test engine support
> running tests on two ARMv8 test agents?
>
> 1. We'll sort out warnings this week. Thanks for heads up.
>>
>
> Great. Let me know when that's fixed.
>
>
> Done. We also fixed a number of warnings in TE.
> Also we fixed root test package name to be consistent with the repository
> name.
>
> Support for old LTS branches was dropped some time ago, but in the future
>> it is definitely possible to keep it for new LTS branches. I think 22.11 is
>> supported, but I'm not sure about older LTS releases.
>>
>
> Good to know.
>
>
>> 2. You can add command-line option --sanity to run tests marked with
>> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
>> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
>> there is a space for terminology improvement here. We'll do it.
>>
>
> Done. Now it is called --checkup.
>
>
>> Also it takes a lot of time because of failures and tests which wait for
>> some timeout.
>>
>
> That makes sense to me. We'll use the time to complete tests on virtio or
> the Intel devices as a reference for how long the tests really take to
> complete.
> We will explore the possibility of periodically running the sanity tests
> for patches.
>
>
> I'll double-check and let you know how long entire TS runs on Intel X710,
> E810, Mellanox CX5 and virtio net. Just to ensure that time observed in
> your case looks the same.
>
>
>
>> The test harness can provide coverage reports based on gcov, but I'm not
>> sure what you mean by a "dial" to control test coverage. Provided reports
>> are rather for human to analyze.
>>
>
> The general idea is to have some kind of parameter on the test suite,
> which could be an integer ranging from zero to ten, that controls how many
> tests are run based on how important the test is.
>
> Similar to how some command line interfaces provide a verbosity level
> parameter (some number of "-v" arguments) to control the importance of the
> information in the log.
> The verbosity level zero only prints very important log messages, while
> ten prints everything.
>
> In much the same manner as above, this "dial" parameter controls what
> tests are run and with what parameters based on how important those tests
> and test parameter combinations are.
> Coverage Level zero tells the suite to run a very basic set of important
> tests, with minimal parameterization. This mode would take only ~5-10
> minutes to run.
> In contrast, Coverage Level ten includes all the edge cases, every
> combination of test parameters, everything the test suite can do, which
> takes the normal several hours to run.
> The values 1 - 9 are between those two extremes, allowing the user to get
> a gradient of test coverage in the results and to limit the running time.
>
> Then we could, for example, run the "run.sh" with a level of 2 or 3 for
> incoming patches that need quick results, and with a level of 10 for the
> less often run periodic tests performed on main or LTS branches.
>
>
> Understood now. Thanks a lot for the idea. We'll discuss it and come back.
>
>
>
>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing
>> results. Unfortunately it is time consuming to fill in expectations
>> database since it is necessary to analyze testing results and classify if
>> it is a bug or just acceptable behaviour aspect.
>>
>> Bublik allows to compare results of two runs. It is useful for human, but
>> still not good for automation.
>>
>> I have local patch for mlx5 driver which reports Tx ring size maximum. It
>> makes pass rate higher. It is a problem for test harness that mlx5 does not
>> report limits right now.
>>
>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio
>> net is 99% right now and could be done 100% easily (just one thing to fix
>> in expectations).
>>
>> I think logs storage setup is essential for logs analysis. Of course, you
>> can request HTML logs when you run tests (--log-html=html) or generate
>> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>> browser, but logs storage makes it more convenient.
>>
>
> We are interested in setting up Bublik, potentially as an
> externally-facing component, once we have our process of running the test
> suite stabilized.
> Once we are able to run the test suite again, I'll see what the pass rate
> is on our other hardware.
> Good to know that it isn't an issue with our dev testbed causing the high
> fail rate.
>
> For Intel hardware, we have an XL710 and an Intel E810-C in our
> development testbed. Although they are slightly different devices, ideally
> the pass rate will be identical or similar. I have yet to set up a VM pair
> for virtio, but we will soon.
>
> Latest version of test-environment has examples of our CGI scripts which
>> we use for log storage (see tools/log_server/README.md).
>>
>> Also all bits for Jenkins setup are available. See
>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>> ts-rigs-sample.
>>
>
> Jenkins integration, setting up production rig configurations, and
> permanent log storage will be our next steps once I am able to run the
> tests again.
> Unless there is an easy way to have meson not pass "-Werror" into GCC.
> Then I would be able to run the test suite.
>
>
> Hopefully it is resolved now.
>
> I thought a bit more about your usecase for Jenkins. I'm not 100% sure
> that existing pipelines are convenient for your usecase.
> Fill free to ask questions when you are on it.
>
> Thanks,
> Andrew.
>
>
> Thanks,
> Adam
>
>
>
>>
>> On 8/29/23 17:02, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> That fix seems to have resolved the issue, thanks for the quick
>> turnaround time on that patch.
>> Now that we have the RCF timeout issue resolved, there are a few other
>> questions and issues that we have about the tests themselves.
>>
>> 1. The test suite fails to build with a couple warnings.
>>
>> Below is the stderr log from compilation:
>>
>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>>> ../../lib/dpdk_pmd_ts.c: In function
>>> ‘test_create_traffic_generator_params’:
>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>> no format arguments [-Werror=format-security]
>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>> |     ^~
>>> cc1: all warnings being treated as errors
>>> ninja: build stopped: subcommand failed.
>>> ninja: Entering directory `.'
>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>>> ../../lib/dpdk_pmd_ts.c: In function
>>> ‘test_create_traffic_generator_params’:
>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>> no format arguments [-Werror=format-security]
>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>> |     ^~
>>> cc1: all warnings being treated as errors
>>>
>>
>> This error wasn't occurring last week, which was the last time I ran the
>> tests.
>> The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC
>> v11.4.0 installed, if this information is helpful.
>>
>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>> collectively take around 9 hours. Is it possible, and would it make sense,
>> to lower the test coverage and have the test suite run faster?
>>
>> For some context, we run immediate testing on incoming patches for DPDK
>> main and development branches, as well as periodic test runs on the main,
>> stable, and LTS branches.
>> For us to consider including this test suite as part of our immediate
>> testing on patches, we would have to reduce the test coverage to the most
>> important tests.
>> This is primarily to reduce the testing time to, for example, less than
>> 30 minutes. Testing on patches can't take too long because the lab can
>> receive numerous patches each day, which each require individual testing
>> runs.
>>
>> At what frequency we run these tests, and on what, still needs to be
>> discussed with the DPDK community, but it would be nice to know if the test
>> suite had a "dial" to control the testing coverage.
>>
>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300
>> of ~6,600 tests passed. Is there anything we can do to diagnose these test
>> failures?
>>
>> Thanks,
>> Adam
>>
>>
>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> I've pushed the fix in main branch and a new tag v1.18.1. It should
>>> solve the problem with IPv6 address from DNS.
>>>
>>> Andrew.
>>>
>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>
>>> Hi Adam,
>>>
>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>> to use IPv4?
>>>
>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your
>>> case, I guess it is the root cause of the problem.
>>> Of course, it is TE problem since I see really weird code in
>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>
>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>> ts-factory/test-environment repository. Please, try.
>>>
>>> It is late night fix with minimal testing and no review. I'll pass it
>>> through review process tomorrow and
>>> hopefully it will be released in one-two days.
>>>
>>> Andrew.
>>>
>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> We have yet to notice a distinct pattern with the failures. Sometimes,
>>> the RCF will start and connect without issue a few times in a row before
>>> failing to connect again. Once the issue begins to occur, neither rebooting
>>> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
>>> directories (suites, agents, inst) and rebooting the hosts afterward
>>> resolves the issue. When it begins working again seems very arbitrary to us.
>>>
>>> I do usually try to terminate the test engine with Ctrl+C, but when it
>>> hangs while trying to start RCF, that does not work.
>>>
>>> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
>>> connection to the test bed hosts? And if so, is there a way to force it to
>>> use IPv4?
>>>
>>>  - Adam
>>>
>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>
>>>> Done. It works fine for me without any issues.
>>>>
>>>> Have you noticed any pattern when it works or does not work?
>>>> May be it is a problem of not clean state after termination?
>>>> Does it work fine the first time after DUTs reboot?
>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>> terminal where you execute run.sh command.
>>>>  In this case it should shutdown gracefully and close all test agents
>>>> and engine applications.
>>>>
>>>> (I'm trying to understand why you've seen many test agent processes. It
>>>> should not happen.)
>>>>
>>>> Andrew.
>>>>
>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>
>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>> This could be the source of the problem. I encountered a dependency
>>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>>> system. Since the tester is also the host having connection issues, I will
>>>> try downgrading that system to 20.04, and see if that changes anything.
>>>>
>>>>
>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>>
>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>
>>>> I did try passing in the "--vg-rcf" argument to the run.sh script of
>>>> the test suite after installing valgrind, but there was no additional
>>>> output that I saw.
>>>>
>>>>
>>>> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
>>>> where you run test engine).
>>>>
>>>>
>>>> I will try pulling in the changes you've pushed up, and will see if
>>>> that fixes anything.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> Hello Adam,
>>>>>
>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>
>>>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>>>> running on test engine and agents?
>>>>>
>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>>> patch in review to fix it, however, the behaviour is different in
>>>>> this case, so it is unlike the same problem.
>>>>>
>>>>>
>>>>> I've just published a new tag which fixes known test engine side
>>>>> problems on Debian 12 and Fedora 38.
>>>>>
>>>>>
>>>>> One more idea is to install valgrind on the test engine host and
>>>>> run with option --vg-rcf to check if something weird is happening.
>>>>>
>>>>> What I don't understand right now is why I see just one failed attempt
>>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>
>>>>>  > Is there any firewall in the network or on test hosts which could
>>>>> block incoming TCP connection to the port 23571
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>>>>> you run test engine?
>>>>>
>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>> connection does work sometimes.
>>>>>
>>>>>  > If behaviour the same on the next try and you see that test agent
>>>>> is kept running, could you check using
>>>>>  >
>>>>>  > # netstat -tnlp
>>>>>  >
>>>>>  > that Test Agent is listening on the port and try to establish TCP
>>>>> connection from test agent using
>>>>>  >
>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>  >
>>>>>  > and check if TCP connection could be established.
>>>>>
>>>>> I was able to replicate the same behavior again, where it hangs while
>>>>> RCF is trying to start.
>>>>> Running this command, I see this in the output:
>>>>>
>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>  18599/ta
>>>>>
>>>>> So it seems like it is listening on the correct port.
>>>>> Additionally, I was able to connect to the Tester machine from our
>>>>> Test Engine host using telnet. It printed the PID of the process once the
>>>>> connection was opened.
>>>>>
>>>>> I tried running the "ta" application manually on the command line, and
>>>>> it didn't print anything at all.
>>>>> Maybe the issue is something on the Test Engine side.
>>>>>
>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>     Hi Adam,
>>>>>
>>>>>      > On the tester host (which appears to be the Peer agent), there
>>>>>     are four processes that I see running, which look like the test
>>>>>     agent processes.
>>>>>
>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>
>>>>>     Is there any firewall in the network or on test hosts which could
>>>>>     block incoming TCP connection to the port 23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>     where you run test engine?
>>>>>
>>>>>     If behaviour the same on the next try and you see that test agent
>>>>> is
>>>>>     kept running, could you check using
>>>>>
>>>>>     # netstat -tnlp
>>>>>
>>>>>     that Test Agent is listening on the port and try to establish TCP
>>>>>     connection from test agent using
>>>>>
>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>     and check if TCP connection could be established.
>>>>>
>>>>>     Another idea is to login Tester under root as testing does, get
>>>>>     start TA command from the log and try it by hands without -n and
>>>>>     remove extra escaping.
>>>>>
>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>
>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>
>>>>>     Hopefully in this case test agent directory remains in the /tmp
>>>>> and
>>>>>     you don't need to copy it as testing does.
>>>>>     May be output could shed some light on what's going on.
>>>>>
>>>>>     Andrew.
>>>>>
>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>
>>>>>     Hi Andrew,
>>>>>
>>>>>     This is the output that I see in the terminal when this failure
>>>>>     occurs, after the test agent binaries build and the test engine
>>>>>     starts:
>>>>>
>>>>>     Platform default build - pass
>>>>>     Simple RCF consistency check succeeded
>>>>>     --->>> Starting Logger...done
>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>     processes. I let it hang for around 9 minutes.
>>>>>
>>>>>     On the tester host (which appears to be the Peer agent), there are
>>>>>     four processes that I see running, which look like the test agent
>>>>>     processes.
>>>>>
>>>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>>>
>>>>>      - Adam
>>>>>
>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>         Hi Adam,
>>>>>
>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>         says that copy_timeout is in seconds and implementation in
>>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>>         Theoretically select() could be interrupted by signal, but I
>>>>>         think it is unlikely here.
>>>>>
>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>>         does it happen when tests are in progress, e.g. in the middle
>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>         likely either host with test agent dies or test agent itself
>>>>>         crashes. It would be easier for me if classify it if you share
>>>>>         text log (log.txt, full or just corresponding fragment with
>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>         depending on which agent has problems could shed some light.
>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>
>>>>>         Andrew.
>>>>>
>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>
>>>>>         Hi Andrew,
>>>>>
>>>>>         I've set up a test rig repository here, and have created
>>>>>         configurations for our development testbed based off of the
>>>>>         examples.
>>>>>         We've been able to get the test suite to run manually on
>>>>>         Mellanox CX5 devices once.
>>>>>         However, we are running into an issue where, when RCF starts,
>>>>>         the RCF connection times out very frequently. We aren't sure
>>>>>         why this is the case.
>>>>>         It works sometimes, but most of the time when we try to run
>>>>>         the test engine, it encounters this issue.
>>>>>         I've tried changing the RCF port by setting
>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>>         machines. Neither seems to fix the issue.
>>>>>
>>>>>         It also seems like the timeout takes far longer than 60
>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>         before I try to run the test suite.
>>>>>         I assume the unit for this variable is seconds?
>>>>>
>>>>>         Thanks,
>>>>>         Adam
>>>>>
>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>
>>>>>             Hi Andrew,
>>>>>
>>>>>             Thanks, I've cloned the example repository and will start
>>>>>             setting up a configuration for our development testbed
>>>>>             today. I'll let you know if I run into any difficulties
>>>>>             or have any questions.
>>>>>
>>>>>              - Adam
>>>>>
>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                 Hi Adam,
>>>>>
>>>>>                 I've published
>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>                 Hopefully it will help to define your test rigs and
>>>>>                 successfully run some tests manually. Feel free to
>>>>>                 ask any questions and I'll answer here and try to
>>>>>                 update documentation.
>>>>>
>>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>>                 (3).
>>>>>                 Hopefully everything is in place for step (4), but we
>>>>>                 need to make steps (2) and (3) first.
>>>>>
>>>>>                 Andrew.
>>>>>
>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>
>>>>>                 Hi Adam,
>>>>>
>>>>>                 > I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>
>>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>>                 almost ready right now, but I need to double-check
>>>>>                 it before publishing.
>>>>>
>>>>>                 Regards,
>>>>>                 Andrew.
>>>>>
>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 I'm adding the CI mailing list to this
>>>>>                 conversation. Others in the community might find
>>>>>                 this conversation valuable.
>>>>>
>>>>>                 We do want to run testing on a regular basis. The
>>>>>                 Jenkins integration will be very useful for us, as
>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>                 I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>                 If you would like to know anything about our
>>>>>                 testbeds that would help you with creating an
>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>                 questions you have.
>>>>>
>>>>>                 We have multiple test rigs (we call these
>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>                 hardware testing on, with differing network
>>>>>                 hardware and CPU architecture. I figured this might
>>>>>                 be an important detail.
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                     Greatings Adam,
>>>>>
>>>>>                     I'm happy to hear that you're trying to bring
>>>>>                     it up.
>>>>>
>>>>>                     As I understand the final goal is to run it on
>>>>>                     regular basis. So, we need to make it properly
>>>>>                     from the very beginning.
>>>>>                     Bring up of all features consists of 4 steps:
>>>>>
>>>>>                     1. Create site-specific repository (we call it
>>>>>                     ts-rigs) which contains information about test
>>>>>                     rigs and other site-specific information like
>>>>>                     where to send mails, where to store logs etc.
>>>>>                     It is required for manual execution as well,
>>>>>                     since test rigs description is essential. I'll
>>>>>                     return to the topic below.
>>>>>
>>>>>                     2. Setup logs storage for automated runs.
>>>>>                     Basically it is a disk space plus apache2 web
>>>>>                     server with few CGI scripts which help a lot to
>>>>>                     save disk space.
>>>>>
>>>>>                     3. Setup Bublik web application which provides
>>>>>                     web interface to view testing results. Same as
>>>>>                     https://ts-factory.io/bublik
>>>>>                     <https://ts-factory.io/bublik>
>>>>> <https://ts-factory.io/bublik>
>>>>>
>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>                     save logs in log storage (2) and import it to
>>>>>                     bublik (3).
>>>>>
>>>>>                     Last few month we spent on our homework to make
>>>>>                     it simpler to bring up automated execution
>>>>>                     using Jenkins -
>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>                     available tomorrow.
>>>>>
>>>>>                     Let's return to the step (1).
>>>>>
>>>>>                     Unfortunately there is no publicly available
>>>>>                     example of the ts-rigs repository since
>>>>>                     sensitive site-specific information is located
>>>>>                     there. But I'm ready to help you to create it
>>>>>                     for UNH. I see two options here:
>>>>>
>>>>>                     (A) I'll ask questions and based on your
>>>>>                     answers will create the first draft with my
>>>>>                     comments.
>>>>>
>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>                     on it.
>>>>>
>>>>>                     Of course, I'll help to debug and finally bring
>>>>>                     it up in any case.
>>>>>
>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>                     a bit more generic and will help other
>>>>>                     potential users to bring it up.
>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>                     What do you think?
>>>>>
>>>>>                     Thanks,
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>
>>>>>                     Greetings Adam,
>>>>>
>>>>>
>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>                     would be happy to help
>>>>>
>>>>>                     Thanks,
>>>>>                     Konstantin
>>>>>
>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>
>>>>>                     
>>>>>                     Greetings Konstantin,
>>>>>
>>>>>                     I am in the process of setting up the DPDK
>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>
>>>>>                     I have some questions about how to set the
>>>>>                     test suite arguments.
>>>>>
>>>>>                     I have been able to configure the Test Engine
>>>>>                     to connect to the hosts in the testbed. The
>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>                     run, however the prelude of the test suite
>>>>>                     fails to run.
>>>>>
>>>>>
>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>
>>>>>                     The documentation mentions that there are
>>>>>                     several test parameters for the test suite,
>>>>>                     like for the IUT test link MAC, etc. These
>>>>>                     seem like they would need to be set somewhere
>>>>>                     to run many of the tests.
>>>>>
>>>>>                     I see in the Test Engine documentation, there
>>>>>                     are instructions on how to create new
>>>>>                     parameters for test suites in the Tester
>>>>>                     configuration, but there is nothing in the
>>>>>                     user guide or in the Tester guide for how to
>>>>>                     set the arguments for the parameters when
>>>>>                     running the test suite that I can find. I'm
>>>>>                     not sure if I need to write my own Tester
>>>>>                     config, or if I should be setting these in
>>>>>                     some other way.
>>>>>
>>>>>                     How should these values be set?
>>>>>
>>>>>                     I'm also not sure what environment
>>>>>                     variables/arguments are strictly necessary or
>>>>>                     which are optional.
>>>>>
>>>>>                     Regards,
>>>>>                     Adam
>>>>>
>>>>>                     --                     *Adam Hassick*
>>>>>                     Senior Developer
>>>>>                     UNH InterOperability Lab
>>>>>                     ahassick@iol.unh.edu
>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>                     +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 --                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>             --             *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>         --         *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>     --     *Adam Hassick*
>>>>>     Senior Developer
>>>>>     UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>     +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #1.2: Type: text/html, Size: 107232 bytes --]

[-- Attachment #2: intel-e810-main-fail.txt --]
[-- Type: text/plain, Size: 369975 bytes --]

Log report
~~~~~~~~~~

RING  Dispatcher  Command-line options  13:01:03.175
--conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --tester-req=TEST_HARNESS_CHECKUP --tester-req=!NO_TEST_HARNESS_CHECKUP --opts=run/iol-dts-e810 --opts=opts.ts

RING  Dispatcher  Expanded command-line options  13:01:03.188
 --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --tester-req=TEST_HARNESS_CHECKUP --tester-req=!NO_TEST_HARNESS_CHECKUP --script=env/iol-dts --script=env/intel-e810 --script=scripts/iut.h1 --script=scripts/iut.h1-e810 --conf-cs=cs/dpdk-pmd-ts.yml --script=scripts/ta-def --script=scripts/defaults --tester-script=scripts/dpdk-trc-tags --tester-script=scripts/os-trc-tags --script=scripts/net-modules --script=scripts/iut-net-driver-loaded --script=scripts/disable_unused_agts

RING  Dispatcher  Start  13:04:57.364
Starting TEN applications

RING  Dispatcher  Start  13:04:57.381
Start Logger:  /opt/tsf/ts-rigs/logger.conf

RING  Logger  Cfg file  13:04:57.393
Opening config file: /opt/tsf/ts-rigs/logger.conf

RING  Logger  Log streaming  13:04:57.393
Current listeners configuration:
Listeners:
Filters:

RING  Dispatcher  Start  13:04:57.410
Start RCF:  /opt/tsf/ts-conf/rcf.conf

RING  RCF  RCF Unix  13:04:57.415
Starting TA 'Peer' type 'linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell='

RING  RCF  RCF Unix  13:04:57.415
CMD to copy: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_353319_1693919097_1" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_353319_1693919097_1 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  13:04:59.168
Command to detect shell name: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  13:04:59.421
Shell is: /bin/bash

RING  RCF  RCF Unix  13:04:59.421
Command to start TA: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_353319_1693919097_1 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_353319_1693919097_1 /tmp/linux_x86_root_353319_1693919097_1/ta Peer 23571 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=" 2>&1 | te_tee RCF Peer 10 >ta.Peer 

RING  @Peer  Main  13:04:59.679
Starting

RING  @Peer  TA unix VM  13:04:59.680
KVM is supported

RING  RCF  RCF Unix  13:05:00.423
Starting TA 'DPDK' type 'linux_x86_64_linux_gnu__glibc2_31__kernel5_4_0_156__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-dut.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:ssh_proxy=:copy_timeout=15:kill_timeout=15:sudo=:shell=:ld_preload='

RING  RCF  RCF Unix  13:05:00.423
CMD to copy: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_353319_1693919100_2" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_31__kernel5_4_0_156__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_353319_1693919100_2 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  13:05:08.494
Command to detect shell name: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  13:05:12.304
Shell is: /bin/bash

RING  RCF  RCF Unix  13:05:12.304
Command to start TA: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_353319_1693919100_2 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_353319_1693919100_2 /tmp/linux_x86_root_353319_1693919100_2/ta DPDK 23571 host=iol-dts-dut.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:ssh_proxy=:copy_timeout=15:kill_timeout=15:sudo=:shell=:ld_preload=" 2>&1 | te_tee RCF DPDK 10 >ta.DPDK 

RING  @DPDK  Main  13:05:16.111
Starting

WARN  @DPDK  TA unix VM  13:05:16.112
KVM is not supported

RING  Dispatcher  Start  13:05:16.328
Start Configurator: --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf:/opt/tsf/dpdk-ethdev-ts/ts/inst/default/share/cm/ /opt/tsf/ts-conf/cs/dpdk-pmd-ts.yml

RING  Configurator  RCF API  13:05:22.938
Set /agent:DPDK/rpcprovider: to dpdkrpc: OK

RING  Configurator  RCF API  13:05:22.938
Set /agent:Peer/rpcprovider: to ta_rpcs: OK

RING  Configurator  RCF API  13:05:22.939
Set /agent:DPDK/sys:/core_pattern: to /var/tmp/core.te.%h-%p-%t: OK

RING  Configurator  RCF API  13:05:22.940
Set /agent:Peer/sys:/core_pattern: to /var/tmp/core.te.%h-%p-%t: OK

RING  Dispatcher  Start  13:05:38.273
Start Tester: --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --req="TEST_HARNESS_CHECKUP" --req="!NO_TEST_HARNESS_CHECKUP" --trc-tag=dpdk-23.11.0-rc0 --trc-tag=dpdk:23110000 --trc-tag=vfio-pci --trc-tag=pci-8086-1592 --trc-tag=pci-8086 --trc-tag=pci-sub-8086-0002 --trc-tag=pci-sub-8086 --trc-tag=num_vfs:128 --trc-tag=kernel-linux --trc-tag=linux-mm:504 /opt/tsf/dpdk-ethdev-ts/conf/tester.conf

MI  Tester  Process Info  13:05:38.991
{
  "type": "tester_pid",
  "version": 1,
  "pid": 353414
}

RING  Tester  Target Requirements  13:05:38.991
TEST_HARNESS_CHECKUP & !NO_TEST_HARNESS_CHECKUP

MI  Tester  TRC tags  13:05:38.991
TRC tags:
 dpdk-23.11.0-rc0
 dpdk:23110000
 vfio-pci
 pci-8086-1592
 pci-8086
 pci-sub-8086-0002
 pci-sub-8086
 num_vfs:128
 kernel-linux
 linux-mm:504

RING  Tester  Self  13:05:38.991
Random seed is 1693919138

RING  Tester  Build  13:05:38.991
Build Test Suite 'dpdk-ethdev-ts' from '${TE_TS_SRC}'

RING  Tester  Self  13:05:41.745
Starting...

RING  Tester  Globals  13:05:41.745
Tester global variables list:
  env.peer2peer='net':IUT{'iut_host'{{'iut_rpcs':IUT,if:'iut_port'},addr:'iut_addr':inet:unicast,addr:'mcast_addr':ether:multicast,addr:'bcast_addr':ether:broadcast,addr:'iut_alien_mac':ether:alien, addr:'alien_addr':inet:alien},'tst_host'{{'tst_rpcs':tester},addr:'tst_addr':inet:unicast,if:'tst_if',addr:'tst_lladdr':ether:fake, addr:'tst_alien_mac':ether:alien}}
  env.peer2peer_ip6='net':IUT{'iut_host'{{'iut_rpcs':IUT,if:'iut_port'},addr:'iut_addr':inet6:unicast,addr:'mcast_addr':ether:multicast,addr:'bcast_addr':ether:broadcast,addr:'iut_alien_mac':ether:alien},'tst_host'{{'tst_rpcs':tester},addr:'tst_addr':inet6:unicast,if:'tst_if',addr:'tst_lladdr':ether:fake, addr:'tst_alien_mac':ether:alien}}
  tmpl.any.eth={
              pdus {
                eth:{
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2tst.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2mcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.mcast_addr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2bcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2mcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.mcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2bcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2tst.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.icmp4={
                pdus {
                    icmp4:{
                        code plain: 1,
                        type plain: 1
                    },
                    ip4:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" },
                        protocol plain: 1
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.icmp4={
                pdus {
                    icmp4:{
                        code plain: 1,
                        type plain: 1
                    },
                    ip4:{
                        src-addr env:{ name "env.addr.iut_addr" },
                        dst-addr env:{ name "env.addr.tst_addr" },
                        protocol plain: 1
                    },
                    eth:{
                        dst-addr env:{ name "env.addr.tst_lladdr" },
                        src-addr env:{ name "param.iut_mac" }
                    }
                }
            }
  tmpl.tst2iut.udp6={
                pdus {
                    udp:{
                        src-port env:{ name "env.addr.tst_addr.port" },
                        dst-port env:{ name "env.addr.iut_addr.port" }
                    },
                    ip6:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" }
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" },
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.tcp6={
                pdus {
                    tcp:{
                        src-port env:{ name "env.addr.tst_addr.port" },
                        dst-port env:{ name "env.addr.iut_addr.port" },
                        seqn script:"expr:rand()",
                        ackn plain:0
                    },
                    ip6:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" }
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vlan.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }

  tmpl.tst2iut.vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "par
  tmpl.tst2iut.vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
  tmpl.tst2iut.vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                e
  tmpl.tst2iut.vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                
  tmpl.tst2iut.vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }

  tmpl.tst2iut.vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "par
  tmpl.tst2iut.vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
  tmpl.tst2iut.vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                e
  tmpl.tst2iut.vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                
  tmpl.tst2iut.vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.vlan_vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.vlan_vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.vlan_vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
         
  tmpl.tst2iut.vlan_vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.vlan_vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.vlan_vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.vlan_vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
         
  tmpl.tst2iut.vlan_vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.vlan_vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.qinq_vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.qinq_vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
               
  tmpl.tst2iut.qinq_vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
       
  tmpl.tst2iut.qinq_vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.qinq_vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.qinq_vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.qinq_vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
               
  tmpl.tst2iut.qinq_vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
       
  tmpl.tst2iut.qinq_vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.qinq_vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" 
  tmpl.tst2iut.geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "p
  tmpl.tst2iut.geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr"
  tmpl.tst2iut.geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
               
  tmpl.tst2iut.geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.ad
  tmpl.tst2iut.geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" 
  tmpl.tst2iut.geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "p
  tmpl.tst2iut.geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr"
  tmpl.tst2iut.geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
               
  tmpl.tst2iut.geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.ad
  tmpl.tst2iut.vlan_geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.vlan_geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.vlan_geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
       
  tmpl.tst2iut.vlan_geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.vlan_geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.vlan_geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.vlan_geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.vlan_geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.vlan_geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.vlan_geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
       
  tmpl.tst2iut.vlan_geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.vlan_geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.vlan_geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.vlan_geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.qinq_geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.qinq_geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.qinq_geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
             
  tmpl.tst2iut.qinq_geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
     
  tmpl.tst2iut.qinq_geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.qinq_geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.qinq_geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.qinq_geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.qinq_geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.qinq_geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.qinq_geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
             
  tmpl.tst2iut.qinq_geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
     
  tmpl.tst2iut.qinq_geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.qinq_geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.qinq_geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.qinq_geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },

  tmpl.tst2iut.nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" 
  tmpl.tst2iut.nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
  tmpl.tst2iut.nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
        
  tmpl.tst2iut.nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },

  tmpl.tst2iut.nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" 
  tmpl.tst2iut.nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
  tmpl.tst2iut.nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
        
  tmpl.tst2iut.vlan_nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vl
  tmpl.tst2iut.vlan_nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    v
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.vlan_nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.vlan_nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vl
  tmpl.tst2iut.vlan_nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    v
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.vlan_nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.qinq_nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
               
  tmpl.tst2iut.qinq_nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq_nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
     
  tmpl.tst2iut.qinq_nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
              
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.qinq_nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.qinq_nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
               
  tmpl.tst2iut.qinq_nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq_nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
     
  tmpl.tst2iut.qinq_nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
              
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.qinq_nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.iut2tst.vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env
  tmpl.iut2tst.vxlan4.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{

  tmpl.iut2tst.vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env
  tmpl.iut2tst.vxlan6.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{

  tmpl.iut2tst.geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "e
  tmpl.iut2tst.geneve4.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:
  tmpl.iut2tst.geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "e
  tmpl.iut2tst.geneve6.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:
  tmpl.iut2tst.vxlan4.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr
  tmpl.iut2tst.vxlan4.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_ad
  tmpl.iut2tst.vxlan6.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr
  tmpl.iut2tst.vxlan6.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_ad
  tmpl.iut2tst.geneve4.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.ad
  tmpl.iut2tst.geneve4.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_
  tmpl.iut2tst.geneve6.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.ad
  tmpl.iut2tst.geneve6.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_
  tmpl.iut2tst.nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "pa
  flow_rule_pattern.ethertype={
              eth:{
                length-type plain:8111,
              }
            }
  flow_rule_pattern.ethertype.arp={
              eth:{
              },
              arp:{
              }
            }
  flow_rule_pattern.ethertype.pppoed={
              eth:{
                length-type plain:34915
              },
              pppoe:{
              }
            }
  flow_rule_pattern.ethertype.pppoes={
              eth:{
                length-type plain:34916
              },
              pppoe:{
              }
            }
  flow_rule_pattern.ethertype.ip4={
              eth:{
              },
              ip4:{
              }
            }
  flow_rule_pattern.ethertype.ip6={
              eth:{
              },
              ip6:{
              }
            }
  flow_rule_pattern.ethertype.outer_vid={
              eth:{
                length-type plain:8111,
                tagged tagged:{
                  vlan-id plain:13,
                }
              }
            }
  flow_rule_pattern.dst_mac.outer_vid={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H,
                tagged tagged:{
                  vlan-id plain:13,
                }
              }
            }
  flow_rule_pattern.ethertype.outer_vid.inner_vid={
              eth:{
                length-type plain:8111,
                tagged double-tagged:{
                  outer {
                    vid plain:13
                  },
                  inner {
                    vid plain:15
                  }
                }
              }
            }
  flow_rule_pattern.dst_mac={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              }
            }
  flow_rule_pattern.unknown_ucast_dst_mac={
              eth:{
                dst-addr range:{
                  first '00 01 02 03 04 05'H,
                  mask '01 00 00 00 00 00'H
                }
              }
            }
  flow_rule_pattern.unknown_mcast_dst_mac={
              eth:{
                dst-addr range:{
                  first '01 02 03 04 05 06'H,
                  mask '01 00 00 00 00 00'H
                }
              }
            }
  flow_rule_pattern.dst_mac.ethertype={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H,
                length-type plain:8111
              }
            }
  flow_rule_pattern.dst_mac.ethertype.ip4={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              }
            }
  flow_rule_pattern.outer_vid.ip_proto={
              eth:{
                tagged tagged:{
                  vlan-id plain:13,
                }
              },
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.ip_proto={
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.ip_proto.icmp4={
              eth:{
              },
              ip4:{
              },
              icmp4:{
              }
            }
  flow_rule_pattern.ip_proto.udp={
              eth:{
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.ip_proto.tcp={
              eth:{
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.dst_mac.ip_proto={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.dst_mac.ip_proto.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.3tuple.udp={
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.3tuple.udp6={
              ip6:{
                dst-addr plain:'20010DB885A3000000008A2E03707334'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.5tuple.udp={
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.5tuple.udp6={
              ip6:{
                src-addr plain:'20010DB885A3000000008A2E03707334'H,
                dst-addr plain:'20010DB885A3000000008A2E03707335'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.3tuple.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.dst_mac.5tuple.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.3tuple.tcp={
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              tcp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.5tuple.tcp={
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              tcp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.3tuple.tcp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              tcp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.dst_mac.5tuple.tcp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              tcp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.vxlan={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.vxlan6={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip6:{
              },
              udp:{
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.vsid.ifrm_dst_mac.vxlan={
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ip4.udp_dst.vsid.ifrm_dst_mac.vxlan={
              eth:{
              },
              ip4:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ip6.udp_dst.vsid.ifrm_dst_mac.vxlan={
              eth:{
              },
              ip6:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.vsid.vxlan={
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.ip4.udp_dst.vsid.vxlan={
              eth:{
              },
              ip4:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.ip6.udp_dst.vsid.vxlan={
              eth:{
              },
              ip6:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.geneve={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              },
              geneve:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.geneve6={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip6:{
              },
              udp:{
              },
              geneve:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.nvgre={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              gre:{
                opt-key nvgre:{
                  vsid plain:13
                }
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ethertype.4tuple.vxlan={
              eth:{
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
              }
            }
  flow_rule_actions.flag={
              {
                type flag
              }
            }
  flow_rule_actions.mark={
              {
                type mark
              }
            }
  flow_rule.rss.3tuple.tcp.default={
              attr {
                ingress 1
              },
              pattern {
                ip4:{
                  dst-addr plain:'0a 89 03 01'H,
                  src-addr script:"expr:rand()"
                },
                tcp:{
                  dst-port plain:9201,
                  src-port script:"expr:rand()"
                }
              },
              actions {
                {
                  type rss,
                  conf rss:{
                    queue {
                      1, 5, 2, 4
                    }
                  }
                }
              }
            }
  flow_rule.rss.3tuple.tcp.custom={
              attr {
                ingress 1
              },
              pattern {
                ip4:{
                  dst-addr plain:'0a 89 03 01'H,
                  src-addr script:"expr:rand()"
                },
                tcp:{
                  dst-port plain:9201,
                  src-port script:"expr:rand()"
                }
              },
              actions {
                {
                  type rss,
                  conf rss:{
                    rss-conf {
                      rss-key '6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a'H,
                      rss-hf {
                        ip 1,
                        tcp 1,
                      }
                    },
                    queue {
                      1, 5, 
  env.perf.iut_only='net':IUT{
                'iut_host'{
                    {'iut_jobs_ctrl':tester, if:'iut_port'}
                }
            }
  env.perf.peer2peer='net':IUT{
                'iut_host'{
                    {'iut_jobs_ctrl':tester, if:'iut_port'}
                },
                'tst_host'{
                    {'tst_jobs_ctrl':tester, if:'tst_port'}
                }
            }

MI  Tester  Execution Plan  13:05:41.776
{
  "type": "test_plan",
  "version": 1,
  "plan": {
    "type": "pkg",
    "name": "dpdk-ethdev-ts",
    "objective": "DPDM EthDev Test Suite",
    "authors": [
      {
        "name": null,
        "email": "Andrew.Rybchenko@oktetlabs.ru"
      }
    ],
    "prologue": {
      "type": "test",
      "name": "prologue"
    },
    "children": [
      {
        "type": "pkg",
        "name": "usecases",
        "objective": "Main use cases of the PMD",
        "authors": [
          {
            "name": null,
            "email": "Andrew.Rybchenko@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "test",
            "name": "tx_burst_simple",
            "iterations": 6
          },
          {
            "type": "test",
            "name": "rx_burst_simple",
            "iterations": 6
          },
          {
            "type": "test",
            "name": "rss",
            "iterations": 5
          },
          {
            "type": "test",
            "name": "rx_scatter",
            "iterations": 2
          },
          {
            "type": "test",
            "name": "tx_stats",
            "iterations": 3
          },
          {
            "type": "test",
            "name": "rx_offload_checksum",
            "iterations": 20
          }
        ]
      },
      {
        "type": "pkg",
        "name": "xmit",
        "objective": "Transmit Functionality",
        "authors": [
          {
            "name": null,
            "email": "Ivan.Malov@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "session",
            "name": "one_packet_all",
            "children": [
              {
                "type": "test",
                "name": "one_packet_tunnel",
                "iterations": 12
              },
              {
                "type": "session",
                "name": "one_packet_ip4",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4",
                    "children": [
                      {
                        "type": "test",
                        "name": "tso_packet_ip4_seg"
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip4"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip4",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4",
                    "children": [
                      {
                        "type": "test",
                        "name": "tso_packet_ip4_seg"
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip4"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip6",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip6"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip6",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip6"
                  }
                ]
              }
            ]
          },
          {
            "type": "session",
            "name": "one_packet_with_dpdk_rx_all",
            "prologue": {
              "type": "test",
              "name": "one_packet_with_dpdk_rx_prologue"
            },
            "children": [
              {
                "type": "session",
                "name": "session",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_with_dpdk_rx_cksum_offloads",
                    "children": [
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_plain",
                        "children": [
                          {
                            "type": "test",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_plain_ip4",
                            "iterations": 8
                          }
                        ]
                      },
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_encap",
                        "children": [
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip6"
                          }
                        ]
                      },
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_encap",
                        "children": [
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip6"
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_with_dpdk_rx_corner_cases",
                    "children": [
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_outgoing_frames"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_outgoing_frames"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_header_segments"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_header_segments"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_payload_segments"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_long_payload"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_long_payload"
                      }
                    ]
                  }
                ]
              }
            ],
            "epilogue": {
              "type": "test",
              "name": "one_packet_with_dpdk_rx_epilogue"
            }
          }
        ]
      },
      {
        "type": "pkg",
        "name": "filters",
        "objective": "Filters",
        "authors": [
          {
            "name": null,
            "email": "Roman.Zhukov@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "test",
            "name": "flow_rule_in2q",
            "iterations": 6
          },
          {
            "type": "session",
            "name": "flow_rule_drop_and_count"
          },
          {
            "type": "session",
            "name": "flow_rule_mark_and_flag"
          }
        ]
      },
      {
        "type": "pkg",
        "name": "representors",
        "objective": "Representors",
        "authors": [
          {
            "name": null,
            "email": "Igor.Romanov@oktetlabs.ru"
          }
        ],
        "prologue": {
          "type": "test",
          "name": "rep_prologue"
        }
      },
      {
        "type": "pkg",
        "name": "perf",
        "objective": "Performance",
        "authors": [
          {
            "name": null,
            "email": "Igor.Romanov@oktetlabs.ru"
          }
        ],
        "prologue": {
          "type": "test",
          "name": "perf_prologue"
        },
        "children": [
          {
            "type": "test",
            "name": "l2fwd_simple"
          }
        ]
      }
    ]
  }
}

RING  Tester  Run  13:05:41.776
Running configuration:
File:        /opt/tsf/dpdk-ethdev-ts/conf/tester.conf
Maintainers: mailto:Andrew.Rybchenko@oktetlabs.ru
Description: DPDK EthDev Testing

MI  Tester  Control  13:05:41.777
PACKAGE "dpdk-ethdev-ts" started
Node ID 1, Parent ID 0, Plan ID 0
Authors:
 *   <Andrew.Rybchenko@oktetlabs.ru>
Objective: DPDM EthDev Test Suite

MI  Tester  Control  13:05:41.777
TEST "prologue" started
Node ID 2, Parent ID 1, Plan ID 1
Parameters:
 *  env = VAR.env.peer2peer

RING  prologue  Step  13:05:41.786
Test start

INFO  prologue  TAPI Jumps  13:05:41.786
Set jump point ../../prologue.c:721

RING  prologue  Self  13:05:41.786
Pseudo-random seed is 1217406619

INFO  prologue  TAPI Jumps  13:05:41.787
Remove jump point ../../prologue.c:721 at ../../prologue.c:721

INFO  prologue  TAPI Jumps  13:05:41.787
Set jump point ../../prologue.c:721

RING  prologue  Configurator API  13:05:41.790
Set /agent:DPDK/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919100_2:/usr/local/sbin

RING  prologue  Configurator API  13:05:41.793
Set /agent:DPDK/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919100_2:/usr/local/sbin:/usr/sbin

RING  prologue  Configurator API  13:05:41.796
Set /agent:DPDK/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919100_2:/usr/local/sbin:/usr/sbin:/sbin

RING  prologue  Configurator API  13:05:41.799
Set /agent:Peer/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919097_1:/usr/local/sbin

RING  prologue  Configurator API  13:05:41.802
Set /agent:Peer/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919097_1:/usr/local/sbin:/usr/sbin

RING  prologue  Configurator API  13:05:41.805
Set /agent:Peer/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919097_1:/usr/local/sbin:/usr/sbin:/sbin

RING  prologue  Configurator API  13:05:41.841
Added /agent:DPDK/rsrc:pci_fn:8086:1592:0 = /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:0

RING  prologue  Configurator API  13:05:41.892
Added /agent:Peer/rsrc:pci_fn:8086:1592:0 = /agent:Peer/hardware:/pci:/vendor:8086/device:1592/instance:0

RING  prologue  Configurator API  13:05:41.933
Added /agent:DPDK/rsrc:pci_fn:8086:1592:1 = /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:1

RING  prologue  Configurator API  13:05:42.055
Added /agent:Peer/rsrc:pci_fn:8086:1592:1 = /agent:Peer/hardware:/pci:/vendor:8086/device:1592/instance:1

RING  prologue  Configurator API  13:05:42.097
Added /agent:DPDK/rsrc:module:vfio-pci = 

RING  prologue  Configurator API  13:05:42.097
Set /agent:DPDK/rsrc:module:vfio-pci/fallback_shared: = 1

RING  prologue  Configurator API  13:05:42.098
Set /agent:DPDK/rsrc:module:vfio-pci/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  13:05:42.100
Set /agent:DPDK/rsrc:module:vfio-pci/shared: = 0

RING  prologue  Configurator API  13:05:42.137
Set /agent:DPDK/rsrc:module:vfio-pci = /agent:DPDK/module:vfio-pci

RING  @DPDK  Unix Conf System Module  13:05:42.156
Module 'vfio-pci' already loaded

RING  prologue  Configurator API  13:05:42.162
Added /agent:DPDK/module:vfio-pci = (none)

RING  prologue  Configurator API  13:05:42.168
Set /agent:DPDK/module:vfio-pci/loaded: = 1

RING  prologue  Configurator API  13:05:42.171
Set /agent:DPDK/rsrc:module:vfio-pci/shared: = 1

RING  prologue  Configurator API  13:05:42.216
Added /agent:DPDK/rsrc:module:ice = 

RING  prologue  Configurator API  13:05:42.216
Set /agent:DPDK/rsrc:module:ice/fallback_shared: = 1

RING  prologue  Configurator API  13:05:42.217
Set /agent:DPDK/rsrc:module:ice/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  13:05:42.219
Set /agent:DPDK/rsrc:module:ice/shared: = 0

RING  prologue  Configurator API  13:05:42.258
Set /agent:DPDK/rsrc:module:ice = /agent:DPDK/module:ice

RING  prologue  Configurator API  13:05:42.282
Added /agent:DPDK/module:ice = (none)

RING  prologue  Configurator API  13:05:42.288
Set /agent:DPDK/module:ice/unload_holders: = 1

RING  @DPDK  Unix Conf System Module  13:05:42.720
Do 'rmmod ice': OK

RING  prologue  Configurator API  13:05:42.731
Set /agent:DPDK/module:ice/loaded: = 0

RING  prologue  Configurator API  13:05:42.732
Set /agent:DPDK/rsrc:module:ice/fallback_shared: = 1

RING  prologue  Configurator API  13:05:42.733
Set /agent:DPDK/rsrc:module:ice/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  13:05:42.735
Set /agent:DPDK/rsrc:module:ice/shared: = 0

RING  prologue  Configurator API  13:05:42.745
Set /agent:DPDK/module:ice/filename: = /tmp/linux_x86_root_353319_1693919100_2/lib/modules/5.4.0-156-generic/ice.ko

RING  prologue  Configurator API  13:05:42.747
Set /agent:DPDK/module:ice/filename:/load_dependencies: = 1

RING  prologue  Configurator API  13:05:42.750
Set /agent:DPDK/module:ice/filename:/fallback: = 1

RING  @DPDK  Unix Conf System Module  13:05:43.139
Do 'modprobe ice': OK

RING  prologue  Configurator API  13:05:43.155
Set /agent:DPDK/module:ice/loaded: = 1

RING  prologue  Configurator API  13:05:43.158
Set /agent:DPDK/rsrc:module:ice/shared: = 1

RING  prologue  Configurator API  13:05:43.236
Added /agent:Peer/rsrc:module:vfio-pci = 

RING  prologue  Configurator API  13:05:43.236
Set /agent:Peer/rsrc:module:vfio-pci/fallback_shared: = 1

RING  prologue  Configurator API  13:05:43.237
Set /agent:Peer/rsrc:module:vfio-pci/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  13:05:43.240
Set /agent:Peer/rsrc:module:vfio-pci/shared: = 0

RING  prologue  Configurator API  13:05:43.320
Set /agent:Peer/rsrc:module:vfio-pci = /agent:Peer/module:vfio-pci

RING  @Peer  Unix Conf System Module  13:05:43.348
Module 'vfio-pci' already loaded

RING  prologue  Configurator API  13:05:43.349
Added /agent:Peer/module:vfio-pci = (none)

RING  prologue  Configurator API  13:05:43.357
Set /agent:Peer/module:vfio-pci/loaded: = 1

RING  prologue  Configurator API  13:05:43.360
Set /agent:Peer/rsrc:module:vfio-pci/shared: = 1

RING  prologue  Configurator API  13:05:43.455
Added /agent:Peer/rsrc:module:ice = 

RING  prologue  Configurator API  13:05:43.456
Set /agent:Peer/rsrc:module:ice/fallback_shared: = 1

RING  prologue  Configurator API  13:05:43.456
Set /agent:Peer/rsrc:module:ice/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  13:05:43.459
Set /agent:Peer/rsrc:module:ice/shared: = 0

RING  prologue  Configurator API  13:05:43.543
Set /agent:Peer/rsrc:module:ice = /agent:Peer/module:ice

RING  prologue  Configurator API  13:05:43.567
Added /agent:Peer/module:ice = (none)

RING  prologue  Configurator API  13:05:43.573
Set /agent:Peer/module:ice/unload_holders: = 1

RING  @Peer  Unix Conf System Module  13:05:43.653
Do 'rmmod irdma': OK

RING  @Peer  Unix Conf System Module  13:05:44.173
Do 'rmmod ice': OK

RING  prologue  Configurator API  13:05:44.180
Set /agent:Peer/module:ice/loaded: = 0

RING  prologue  Configurator API  13:05:44.181
Set /agent:Peer/rsrc:module:ice/fallback_shared: = 1

RING  prologue  Configurator API  13:05:44.182
Set /agent:Peer/rsrc:module:ice/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  13:05:44.184
Set /agent:Peer/rsrc:module:ice/shared: = 0

RING  prologue  Configurator API  13:05:44.194
Set /agent:Peer/module:ice/filename: = /tmp/linux_x86_root_353319_1693919097_1/lib/modules/5.15.0-79-generic/ice.ko

RING  prologue  Configurator API  13:05:44.196
Set /agent:Peer/module:ice/filename:/load_dependencies: = 1

RING  prologue  Configurator API  13:05:44.199
Set /agent:Peer/module:ice/filename:/fallback: = 1

RING  @Peer  Unix Conf System Module  13:05:45.084
Do 'modprobe ice': OK

RING  prologue  Configurator API  13:05:45.091
Set /agent:Peer/module:ice/loaded: = 1

RING  prologue  Configurator API  13:05:45.093
Set /agent:Peer/rsrc:module:ice/shared: = 1

RING  prologue  Configurator API  13:05:45.140
Added /agent:DPDK/rsrc:module:vfio = 

RING  prologue  Configurator API  13:05:45.140
Set /agent:DPDK/rsrc:module:vfio/fallback_shared: = 1

RING  prologue  Configurator API  13:05:45.141
Set /agent:DPDK/rsrc:module:vfio/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  13:05:45.143
Set /agent:DPDK/rsrc:module:vfio/shared: = 0

RING  prologue  Configurator API  13:05:45.187
Set /agent:DPDK/rsrc:module:vfio = /agent:DPDK/module:vfio

RING  @DPDK  Unix Conf System Module  13:05:45.204
Module 'vfio' already loaded

RING  prologue  Configurator API  13:05:45.209
Added /agent:DPDK/module:vfio = (none)

RING  prologue  Configurator API  13:05:45.215
Set /agent:DPDK/module:vfio/loaded: = 1

RING  prologue  Configurator API  13:05:45.218
Set /agent:DPDK/rsrc:module:vfio/shared: = 1

RING  prologue  Configurator API  13:05:45.442
Set /agent:DPDK/hardware:/pci:/device:0000:61:00.0/driver: = vfio-pci

RING  prologue  Configurator API  13:05:45.683
Set /agent:DPDK/hardware:/pci:/device:0000:61:00.1/driver: = vfio-pci

RING  prologue  Configurator API  13:05:46.502
Added /agent:Peer/rsrc:ens2f0 = /agent:Peer/interface:ens2f0

RING  prologue  Configurator API  13:05:46.982
Added /agent:Peer/rsrc:ens2f1 = /agent:Peer/interface:ens2f1

RING  prologue  Configurator API  13:05:46.988
Set /agent:Peer/interface:ens2f0/feature:rx-gro = 0

RING  prologue  Configurator API  13:05:46.992
Set /agent:Peer/interface:ens2f0/feature:rx-lro = 0

RING  prologue  Configurator API  13:05:46.995
Set /agent:Peer/interface:ens2f0/feature:tx-checksum-ip-generic = 0

RING  prologue  Configurator API  13:05:46.998
Set /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv4 = 0

RING  prologue  Configurator API  13:05:47.000
Set /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv6 = 0

RING  prologue  Configurator API  13:05:47.003
Set /agent:Peer/interface:ens2f1/feature:rx-gro = 0

RING  prologue  Configurator API  13:05:47.006
Set /agent:Peer/interface:ens2f1/feature:rx-lro = 0

RING  prologue  Configurator API  13:05:47.009
Set /agent:Peer/interface:ens2f1/feature:tx-checksum-ip-generic = 0

RING  prologue  Configurator API  13:05:47.012
Set /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv4 = 0

RING  prologue  Configurator API  13:05:47.015
Set /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv6 = 0

RING  prologue  Configurator API  13:05:47.059
Set /agent:Peer/interface:ens2f0/status: = 1

RING  prologue  Configurator API  13:05:47.094
Set /agent:Peer/interface:ens2f1/status: = 1

RING  prologue  Configuration TAPI  13:05:47.114
Address 10.38.10.1 is added to pool entry '/net_pool:ip4/entry:10.38.10.0'

RING  prologue  Configuration TAPI  13:05:47.115
Address 10.38.10.2 is added to pool entry '/net_pool:ip4/entry:10.38.10.0'

RING  prologue  Configurator API  13:05:47.128
Added /agent:Peer/interface:ens2f0/net_addr:10.38.10.2 = 24

RING  prologue  Configurator API  13:05:47.129
Set /agent:Peer/interface:ens2f0/net_addr:10.38.10.2/broadcast: = 10.38.10.255

RING  prologue  Configuration TAPI  13:05:47.129
Address 10.38.11.1 is added to pool entry '/net_pool:ip4/entry:10.38.11.0'

RING  prologue  Configuration TAPI  13:05:47.130
Address 10.38.11.2 is added to pool entry '/net_pool:ip4/entry:10.38.11.0'

RING  prologue  Configurator API  13:05:47.145
Added /agent:Peer/interface:ens2f1/net_addr:10.38.11.2 = 24

RING  prologue  Configurator API  13:05:47.146
Set /agent:Peer/interface:ens2f1/net_addr:10.38.11.2/broadcast: = 10.38.11.255

RING  prologue  Configuration TAPI  13:05:47.150
Address fec0::1 is added to pool entry '/net_pool:ip6/entry:fec0:0:0::'

RING  prologue  Configuration TAPI  13:05:47.150
Address fec0::2 is added to pool entry '/net_pool:ip6/entry:fec0:0:0::'

RING  prologue  Configurator API  13:05:47.168
Added /agent:Peer/interface:ens2f0/net_addr:fec0::2 = 48

RING  prologue  Configuration TAPI  13:05:47.169
Address fec0:0:10::1 is added to pool entry '/net_pool:ip6/entry:fec0:0:10::'

RING  prologue  Configuration TAPI  13:05:47.169
Address fec0:0:10::2 is added to pool entry '/net_pool:ip6/entry:fec0:0:10::'

RING  prologue  Configurator API  13:05:47.190
Added /agent:Peer/interface:ens2f1/net_addr:fec0:0:10::2 = 48

RING  prologue  Configurator API  13:05:47.195
Set /agent:Peer/sys:/net:/ipv6:/conf:ens2f0/disable_ipv6: = 1

RING  prologue  Configurator API  13:05:47.207
Deleted /agent:Peer/interface:ens2f0/net_addr:10.38.10.2

RING  prologue  Configurator API  13:05:47.209
Set /agent:Peer/sys:/net:/ipv6:/conf:ens2f1/disable_ipv6: = 1

RING  prologue  Configurator API  13:05:47.216
Deleted /agent:Peer/interface:ens2f1/net_addr:10.38.11.2

RING  Configurator  Self  13:05:47.216
Sleep 5000 milliseconds to propagate configuration changes

RING  @Peer  Self  13:05:52.233
tst_rpcs.141600.3363830784: RPC server 'tst_rpcs' (64-bit) (re-)started (PID 141600, TID 3363830784)

RING  prologue  Configurator API  13:05:52.245
Added /agent:Peer/rpcserver:tst_rpcs = 

RING  prologue  Configurator API  13:05:52.248
Set /agent:Peer/rpcserver:tst_rpcs/sid: = 4

RING  @DPDK  Self  13:05:52.248
iut_rpcs.1375235.877186432: RPC server 'iut_rpcs' (64-bit) (re-)started (PID 1375235, TID 877186432)

RING  @DPDK  RPC  13:05:52.262
iut_rpcs.1375235.877186432: Dynamic library is set to ''

RING  prologue  Configurator API  13:05:52.265
Added /agent:DPDK/rpcserver:iut_rpcs = 

RING  prologue  Configurator API  13:05:52.268
Set /agent:DPDK/rpcserver:iut_rpcs/sid: = 4

RING  prologue  RCF RPC  13:05:53.157
RPC (DPDK,iut_rpcs) setlibname() -> 0 (OK)

RING  prologue  Configurator API  13:05:53.208
Added /agent:DPDK/rsrc:cpu:0:0:0:0 = /agent:DPDK/hardware:/node:0/cpu:0/core:0/thread:0

RING  prologue  Configurator API  13:05:53.592
Added /agent:Peer/rsrc:cpu:0:0:0:0 = /agent:Peer/hardware:/node:0/cpu:0/core:0/thread:0

RING  prologue  Configurator API  13:05:53.649
Set /agent:Peer/interface:ens2f0/mtu: = 9216

RING  prologue  Configurator API  13:05:53.650
Set /agent:Peer/interface:ens2f0/promisc: = 1

RING  Configurator  Self  13:05:53.650
Sleep 5000 milliseconds to propagate configuration changes

RING  Configurator  Self  13:05:59.377
tree of Instances /::
/: = 
  /conf_delay:if_mtu = /agent/interface/mtu
    /conf_delay:if_mtu/ta: = 5000
  /conf_delay:if_status = /agent/interface/status
    /conf_delay:if_status/ta: = 5000
  /conf_delay:net_addr = /agent/interface/net_addr
    /conf_delay:net_addr/ta: = 500
  /conf_delay:promisc = /agent/interface/promisc
    /conf_delay:promisc/ta: = 1000
  /conf_delay:route = /agent/route
    /conf_delay:route/ta: = 500
  /conf_delay:rule = /agent/rule
    /conf_delay:rule/ta: = 500
  /conf_delay:vlans = /agent/interface/vlans
    /conf_delay:vlans/ta: = 3000
  /conf_delay:xdp = /agent/interface/xdp
    /conf_delay:xdp/ta: = 500
  /local: = 
    /local:/ip4_alien: = 33.33.33.33
    /local:/ip6_alien: = fec0::2121:2121
    /local:/socklib: = 
    /local:/rpcserver_force_restart: = 0
    /local:/iut_errno_change_no_check: = 1
    /local:/no_reuse_pco: = 0
    /local:/use_static_arp: = 0
    /local:/mem_channels: = 0
    /local:/test: = 
      /local:/test:/behaviour:cleanup_fd_close_enforce_libc = 0
      /local:/test:/behaviour:cleanup_fd_leak_check = 1
      /local:/test:/behaviour:fail_verdict = 0
      /local:/test:/behaviour:iface_toggle_delay_ms = 0
      /local:/test:/behaviour:log_all_rpc = 0
      /local:/test:/behaviour:log_stack = 0
      /local:/test:/behaviour:log_test_fail_state = 0
      /local:/test:/behaviour:prologue_sleep = 0
      /local:/test:/behaviour:rpc_fail_verdict = 0
      /local:/test:/behaviour:use_chk_funcs = 0
      /local:/test:/behaviour:wait_on_cleanup = 0
      /local:/test:/behaviour:wait_on_fail = 0
    /local:/dpdk: = 
      /local:/dpdk:/app_prefix: = 
      /local:/dpdk:/extra_eal_args: =  
      /local:/dpdk:/link_speeds: = 
      /local:/dpdk:/mem_amount: = 0
      /local:/dpdk:/offloads: = 
        /local:/dpdk:/offloads:/dev: = 
          /local:/dpdk:/offloads:/dev:/rx: = 
          /local:/dpdk:/offloads:/dev:/tx: = 
      /local:/dpdk:/peer_max_mtu: = 9216
      /local:/dpdk:/reuse_rpcs: = 
      /local:/dpdk:/extra_tx_descs_per_packet: = 0
      /local:/dpdk:/tso_cutoff_barrier: = 0
      /local:/dpdk:/tso_ip_id_inc_algo: = 
  /local:DPDK = 
    /local:DPDK/dpdk_driver: = vfio-pci
    /local:DPDK/ip4_alien: = 
    /local:DPDK/ip6_alien: = 
    /local:DPDK/net_driver: = ice
    /local:DPDK/socklib: = 
    /local:DPDK/rpcserver_force_restart: = 0
    /local:DPDK/iut_errno_change_no_check: = 0
    /local:DPDK/no_reuse_pco: = 0
    /local:DPDK/use_static_arp: = 0
    /local:DPDK/mem_channels: = 2
    /local:DPDK/test: = 
    /local:DPDK/dpdk: = 
      /local:DPDK/dpdk:/app_prefix: = 
      /local:DPDK/dpdk:/dev_args:pci_fn:8086:1592: = 
      /local:DPDK/dpdk:/extra_eal_args: =  
      /local:DPDK/dpdk:/link_speeds: = 
      /local:DPDK/dpdk:/mem_amount: = 0
      /local:DPDK/dpdk:/offloads: = 
        /local:DPDK/dpdk:/offloads:/dev: = 
          /local:DPDK/dpdk:/offloads:/dev:/rx: = 
          /local:DPDK/dpdk:/offloads:/dev:/tx: = 
      /local:DPDK/dpdk:/peer_max_mtu: = 0
      /local:DPDK/dpdk:/required_service_cores:pci_fn:8086:1592: = 0
      /local:DPDK/dpdk:/reuse_rpcs: = 
      /local:DPDK/dpdk:/extra_tx_descs_per_packet: = 0
      /local:DPDK/dpdk:/tso_cutoff_barrier: = 0
      /local:DPDK/dpdk:/tso_ip_id_inc_algo: = 
  /local:Hypervisor = 
    /local:Hypervisor/ip4_alien: = 
    /local:Hypervisor/ip6_alien: = 
    /local:Hypervisor/socklib: = 
    /local:Hypervisor/rpcserver_force_restart: = 0
    /local:Hypervisor/iut_errno_change_no_check: = 0
    /local:Hypervisor/no_reuse_pco: = 0
    /local:Hypervisor/use_static_arp: = 0
    /local:Hypervisor/mem_channels: = 0
    /local:Hypervisor/test: = 
    /local:Hypervisor/dpdk: = 
      /local:Hypervisor/dpdk:/app_prefix: = 
      /local:Hypervisor/dpdk:/extra_eal_args: = 
      /local:Hypervisor/dpdk:/link_speeds: = 
      /local:Hypervisor/dpdk:/mem_amount: = 0
      /local:Hypervisor/dpdk:/offloads: = 
        /local:Hypervisor/dpdk:/offloads:/dev: = 
          /local:Hypervisor/dpdk:/offloads:/dev:/rx: = 
          /local:Hypervisor/dpdk:/offloads:/dev:/tx: = 
      /local:Hypervisor/dpdk:/peer_max_mtu: = 0
      /local:Hypervisor/dpdk:/reuse_rpcs: = 
      /local:Hypervisor/dpdk:/extra_tx_descs_per_packet: = 0
      /local:Hypervisor/dpdk:/tso_cutoff_barrier: = 0
      /local:Hypervisor/dpdk:/tso_ip_id_inc_algo: = 
  /local:Peer = 
    /local:Peer/dpdk_driver: = vfio-pci
    /local:Peer/ip4_alien: = 
    /local:Peer/ip6_alien: = 
    /local:Peer/net_driver: = ice
    /local:Peer/socklib: = 
    /local:Peer/rpcserver_force_restart: = 0
    /local:Peer/iut_errno_change_no_check: = 0
    /local:Peer/no_reuse_pco: = 0
    /local:Peer/use_static_arp: = 0
    /local:Peer/mem_channels: = 2
    /local:Peer/test: = 
    /local:Peer/dpdk: = 
      /local:Peer/dpdk:/app_prefix: = 
      /local:Peer/dpdk:/dev_args:pci_fn::: = 
      /local:Peer/dpdk:/extra_eal_args: =  
      /local:Peer/dpdk:/link_speeds: = 
      /local:Peer/dpdk:/mem_amount: = 0
      /local:Peer/dpdk:/offloads: = 
        /local:Peer/dpdk:/offloads:/dev: = 
          /local:Peer/dpdk:/offloads:/dev:/rx: = 
          /local:Peer/dpdk:/offloads:/dev:/tx: = 
      /local:Peer/dpdk:/peer_max_mtu: = 0
      /local:Peer/dpdk:/reuse_rpcs: = 
      /local:Peer/dpdk:/extra_tx_descs_per_packet: = 0
      /local:Peer/dpdk:/tso_cutoff_barrier: = 0
      /local:Peer/dpdk:/tso_ip_id_inc_algo: = 
  /net:net1 = 
    /net:net1/ip4_subnet:0x311000002b7 = 10.38.10.0
    /net:net1/ip6_subnet:0x3890000032f = fec0::
    /net:net1/node:A = /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:0
      /net:net1/node:A/ip4_address:0x70000000697 = 10.38.10.1
      /net:net1/node:A/ip6_address:0x71e000006b5 = fec0::1
      /net:net1/node:A/type: = 1
    /net:net1/node:B = /agent:Peer/interface:ens2f0
      /net:net1/node:B/ip4_address:0x70200000699 = 10.38.10.2
      /net:net1/node:B/ip6_address:0x720000006b7 = fec0::2
      /net:net1/node:B/type: = 0
  /net:net1a = 
    /net:net1a/ip4_subnet:0x315000002bb = 10.38.11.0
    /net:net1a/ip6_subnet:0x3b100000357 = fec0:0:10::
    /net:net1a/node:A = /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:1
      /net:net1a/node:A/ip4_address:0x70f000006a6 = 10.38.11.1
      /net:net1a/node:A/ip6_address:0x72d000006c4 = fec0:0:10::1
      /net:net1a/node:A/type: = 1
    /net:net1a/node:B = /agent:Peer/interface:ens2f1
      /net:net1a/node:B/ip4_address:0x711000006a8 = 10.38.11.2
      /net:net1a/node:B/ip6_address:0x72f000006c6 = fec0:0:10::2
      /net:net1a/node:B/type: = 0
  /net_pool:ip4 = 
    /net_pool:ip4/entry:10.38.10.0 = 1
      /net_pool:ip4/entry:10.38.10.0/prefix: = 24
      /net_pool:ip4/entry:10.38.10.0/n_entries: = 2
      /net_pool:ip4/entry:10.38.10.0/pool: = 
        /net_pool:ip4/entry:10.38.10.0/pool:/entry:10.38.10.1 = 1
        /net_pool:ip4/entry:10.38.10.0/pool:/entry:10.38.10.2 = 1
    /net_pool:ip4/entry:10.38.11.0 = 1
      /net_pool:ip4/entry:10.38.11.0/prefix: = 24
      /net_pool:ip4/entry:10.38.11.0/n_entries: = 2
      /net_pool:ip4/entry:10.38.11.0/pool: = 
        /net_pool:ip4/entry:10.38.11.0/pool:/entry:10.38.11.1 = 1
        /net_pool:ip4/entry:10.38.11.0/pool:/entry:10.38.11.2 = 1
    /net_pool:ip4/entry:10.38.110.0 = 0
      /net_pool:ip4/entry:10.38.110.0/prefix: = 24
      /net_pool:ip4/entry:10.38.110.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.110.0/pool: = 
    /net_pool:ip4/entry:10.38.111.0 = 0
      /net_pool:ip4/entry:10.38.111.0/prefix: = 24
      /net_pool:ip4/entry:10.38.111.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.111.0/pool: = 
    /net_pool:ip4/entry:10.38.112.0 = 0
      /net_pool:ip4/entry:10.38.112.0/prefix: = 24
      /net_pool:ip4/entry:10.38.112.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.112.0/pool: = 
    /net_pool:ip4/entry:10.38.113.0 = 0
      /net_pool:ip4/entry:10.38.113.0/prefix: = 24
      /net_pool:ip4/entry:10.38.113.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.113.0/pool: = 
    /net_pool:ip4/entry:10.38.114.0 = 0
      /net_pool:ip4/entry:10.38.114.0/prefix: = 24
      /net_pool:ip4/entry:10.38.114.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.114.0/pool: = 
    /net_pool:ip4/entry:10.38.115.0 = 0
      /net_pool:ip4/entry:10.38.115.0/prefix: = 24
      /net_pool:ip4/entry:10.38.115.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.115.0/pool: = 
    /net_pool:ip4/entry:10.38.116.0 = 0
      /net_pool:ip4/entry:10.38.116.0/prefix: = 24
      /net_pool:ip4/entry:10.38.116.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.116.0/pool: = 
    /net_pool:ip4/entry:10.38.117.0 = 0
      /net_pool:ip4/entry:10.38.117.0/prefix: = 24
      /net_pool:ip4/entry:10.38.117.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.117.0/pool: = 
    /net_pool:ip4/entry:10.38.118.0 = 0
      /net_pool:ip4/entry:10.38.118.0/prefix: = 24
      /net_pool:ip4/entry:10.38.118.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.118.0/pool: = 
    /net_pool:ip4/entry:10.38.119.0 = 0
      /net_pool:ip4/entry:10.38.119.0/prefix: = 24
      /net_pool:ip4/entry:10.38.119.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.119.0/pool: = 
    /net_pool:ip4/entry:10.38.12.0 = 0
      /net_pool:ip4/entry:10.38.12.0/prefix: = 24
      /net_pool:ip4/entry:10.38.12.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.12.0/pool: = 
    /net_pool:ip4/entry:10.38.120.0 = 0
      /net_pool:ip4/entry:10.38.120.0/prefix: = 24
      /net_pool:ip4/entry:10.38.120.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.120.0/pool: = 
    /net_pool:ip4/entry:10.38.121.0 = 0
      /net_pool:ip4/entry:10.38.121.0/prefix: = 24
      /net_pool:ip4/entry:10.38.121.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.121.0/pool: = 
    /net_pool:ip4/entry:10.38.122.0 = 0
      /net_pool:ip4/entry:10.38.122.0/prefix: = 24
      /net_pool:ip4/entry:10.38.122.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.122.0/pool: = 
    /net_pool:ip4/entry:10.38.123.0 = 0
      /net_pool:ip4/entry:10.38.123.0/prefix: = 24
      /net_pool:ip4/entry:10.38.123.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.123.0/pool: = 
    /net_pool:ip4/entry:10.38.124.0 = 0
      /net_pool:ip4/entry:10.38.124.0/prefix: = 24
      /net_pool:ip4/entry:10.38.124.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.124.0/pool: = 
    /net_pool:ip4/entry:10.38.125.0 = 0
      /net_pool:ip4/entry:10.38.125.0/prefix: = 24
      /net_pool:ip4/entry:10.38.125.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.125.0/pool: = 
    /net_pool:ip4/entry:10.38.126.0 = 0
      /net_pool:ip4/entry:10.38.126.0/prefix: = 24
      /net_pool:ip4/entry:10.38.126.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.126.0/pool: = 
    /net_pool:ip4/entry:10.38.127.0 = 0
      /net_pool:ip4/entry:10.38.127.0/prefix: = 24
      /net_pool:ip4/entry:10.38.127.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.127.0/pool: = 
    /net_pool:ip4/entry:10.38.128.0 = 0
      /net_pool:ip4/entry:10.38.128.0/prefix: = 24
      /net_pool:ip4/entry:10.38.128.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.128.0/pool: = 
    /net_pool:ip4/entry:10.38.129.0 = 0
      /net_pool:ip4/entry:10.38.129.0/prefix: = 24
      /net_pool:ip4/entry:10.38.129.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.129.0/pool: = 
    /net_pool:ip4/entry:10.38.13.0 = 0
      /net_pool:ip4/entry:10.38.13.0/prefix: = 24
      /net_pool:ip4/entry:10.38.13.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.13.0/pool: = 
    /net_pool:ip4/entry:10.38.14.0 = 0
      /net_pool:ip4/entry:10.38.14.0/prefix: = 24
      /net_pool:ip4/entry:10.38.14.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.14.0/pool: = 
    /net_pool:ip4/entry:10.38.15.0 = 0
      /net_pool:ip4/entry:10.38.15.0/prefix: = 24
      /net_pool:ip4/entry:10.38.15.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.15.0/pool: = 
    /net_pool:ip4/entry:10.38.16.0 = 0
      /net_pool:ip4/entry:10.38.16.0/prefix: = 24
      /net_pool:ip4/entry:10.38.16.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.16.0/pool: = 
    /net_pool:ip4/entry:10.38.17.0 = 0
      /net_pool:ip4/entry:10.38.17.0/prefix: = 24
      /net_pool:ip4/entry:10.38.17.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.17.0/pool: = 
    /net_pool:ip4/entry:10.38.18.0 = 0
      /net_pool:ip4/entry:10.38.18.0/prefix: = 24
      /net_pool:ip4/entry:10.38.18.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.18.0/pool: = 
    /net_pool:ip4/entry:10.38.19.0 = 0
      /net_pool:ip4/entry:10.38.19.0/prefix: = 24
      /net_pool:ip4/entry:10.38.19.0/n_entries: = 0
      /net_pool:ip4/entry:10.38.19.0/pool: = 
  /net_pool:ip6 = 
    /net_pool:ip6/entry:fec0:0:0:: = 1
      /net_pool:ip6/entry:fec0:0:0::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:0::/n_entries: = 2
      /net_pool:ip6/entry:fec0:0:0::/pool: = 
        /net_pool:ip6/entry:fec0:0:0::/pool:/entry:fec0::1 = 1
        /net_pool:ip6/entry:fec0:0:0::/pool:/entry:fec0::2 = 1
    /net_pool:ip6/entry:fec0:0:10:: = 1
      /net_pool:ip6/entry:fec0:0:10::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:10::/n_entries: = 2
      /net_pool:ip6/entry:fec0:0:10::/pool: = 
        /net_pool:ip6/entry:fec0:0:10::/pool:/entry:fec0:0:10::1 = 1
        /net_pool:ip6/entry:fec0:0:10::/pool:/entry:fec0:0:10::2 = 1
    /net_pool:ip6/entry:fec0:0:11:: = 0
      /net_pool:ip6/entry:fec0:0:11::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:11::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:11::/pool: = 
    /net_pool:ip6/entry:fec0:0:12:: = 0
      /net_pool:ip6/entry:fec0:0:12::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:12::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:12::/pool: = 
    /net_pool:ip6/entry:fec0:0:13:: = 0
      /net_pool:ip6/entry:fec0:0:13::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:13::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:13::/pool: = 
    /net_pool:ip6/entry:fec0:0:14:: = 0
      /net_pool:ip6/entry:fec0:0:14::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:14::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:14::/pool: = 
    /net_pool:ip6/entry:fec0:0:15:: = 0
      /net_pool:ip6/entry:fec0:0:15::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:15::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:15::/pool: = 
    /net_pool:ip6/entry:fec0:0:16:: = 0
      /net_pool:ip6/entry:fec0:0:16::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:16::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:16::/pool: = 
    /net_pool:ip6/entry:fec0:0:17:: = 0
      /net_pool:ip6/entry:fec0:0:17::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:17::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:17::/pool: = 
    /net_pool:ip6/entry:fec0:0:18:: = 0
      /net_pool:ip6/entry:fec0:0:18::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:18::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:18::/pool: = 
    /net_pool:ip6/entry:fec0:0:19:: = 0
      /net_pool:ip6/entry:fec0:0:19::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:19::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:19::/pool: = 
    /net_pool:ip6/entry:fec0:0:1:: = 0
      /net_pool:ip6/entry:fec0:0:1::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:1::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:1::/pool: = 
    /net_pool:ip6/entry:fec0:0:20:: = 0
      /net_pool:ip6/entry:fec0:0:20::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:20::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:20::/pool: = 
    /net_pool:ip6/entry:fec0:0:21:: = 0
      /net_pool:ip6/entry:fec0:0:21::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:21::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:21::/pool: = 
    /net_pool:ip6/entry:fec0:0:22:: = 0
      /net_pool:ip6/entry:fec0:0:22::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:22::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:22::/pool: = 
    /net_pool:ip6/entry:fec0:0:23:: = 0
      /net_pool:ip6/entry:fec0:0:23::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:23::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:23::/pool: = 
    /net_pool:ip6/entry:fec0:0:24:: = 0
      /net_pool:ip6/entry:fec0:0:24::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:24::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:24::/pool: = 
    /net_pool:ip6/entry:fec0:0:25:: = 0
      /net_pool:ip6/entry:fec0:0:25::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:25::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:25::/pool: = 
    /net_pool:ip6/entry:fec0:0:26:: = 0
      /net_pool:ip6/entry:fec0:0:26::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:26::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:26::/pool: = 
    /net_pool:ip6/entry:fec0:0:27:: = 0
      /net_pool:ip6/entry:fec0:0:27::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:27::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:27::/pool: = 
    /net_pool:ip6/entry:fec0:0:28:: = 0
      /net_pool:ip6/entry:fec0:0:28::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:28::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:28::/pool: = 
    /net_pool:ip6/entry:fec0:0:29:: = 0
      /net_pool:ip6/entry:fec0:0:29::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:29::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:29::/pool: = 
    /net_pool:ip6/entry:fec0:0:2:: = 0
      /net_pool:ip6/entry:fec0:0:2::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:2::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:2::/pool: = 
    /net_pool:ip6/entry:fec0:0:3:: = 0
      /net_pool:ip6/entry:fec0:0:3::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:3::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:3::/pool: = 
    /net_pool:ip6/entry:fec0:0:4:: = 0
      /net_pool:ip6/entry:fec0:0:4::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:4::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:4::/pool: = 
    /net_pool:ip6/entry:fec0:0:5:: = 0
      /net_pool:ip6/entry:fec0:0:5::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:5::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:5::/pool: = 
    /net_pool:ip6/entry:fec0:0:6:: = 0
      /net_pool:ip6/entry:fec0:0:6::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:6::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:6::/pool: = 
    /net_pool:ip6/entry:fec0:0:7:: = 0
      /net_pool:ip6/entry:fec0:0:7::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:7::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:7::/pool: = 
    /net_pool:ip6/entry:fec0:0:8:: = 0
      /net_pool:ip6/entry:fec0:0:8::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:8::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:8::/pool: = 
    /net_pool:ip6/entry:fec0:0:9:: = 0
      /net_pool:ip6/entry:fec0:0:9::/prefix: = 48
      /net_pool:ip6/entry:fec0:0:9::/n_entries: = 0
      /net_pool:ip6/entry:fec0:0:9::/pool: = 
  /rcf: = 
  /agent:DPDK = 
    /agent:DPDK/l4_port: = 
      /agent:DPDK/l4_port:/alloc: = 
        /agent:DPDK/l4_port:/alloc:/next: = 23600
          /agent:DPDK/l4_port:/alloc:/next:/type: = 0
          /agent:DPDK/l4_port:/alloc:/next:/family: = 0
    /agent:DPDK/hardware: = 
      /agent:DPDK/hardware:/pci: = 
        /agent:DPDK/hardware:/pci:/device:0000:61:00.0 = 
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/bus: = 61
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/class: = 00020000
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/device_id: = 1592
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/domain: = 0000
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/driver: = vfio-pci
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/fn: = 0
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/node: = /agent:DPDK/hardware:/node:0
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/serialno: = 
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/slot: = 00
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/sriov: = 128
            /agent:DPDK/hardware:/pci:/device:0000:61:00.0/sriov:/num_vfs: = 0
            /agent:DPDK/hardware:/pci:/device:0000:61:00.0/sriov:/pf: = 
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/subsystem_device: = 0002
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/subsystem_vendor: = 8086
          /agent:DPDK/hardware:/pci:/device:0000:61:00.0/vendor_id: = 8086
        /agent:DPDK/hardware:/pci:/device:0000:61:00.1 = 
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/bus: = 61
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/class: = 00020000
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/device_id: = 1592
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/domain: = 0000
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/driver: = vfio-pci
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/fn: = 1
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/node: = /agent:DPDK/hardware:/node:0
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/serialno: = 
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/slot: = 00
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/sriov: = 128
            /agent:DPDK/hardware:/pci:/device:0000:61:00.1/sriov:/num_vfs: = 0
            /agent:DPDK/hardware:/pci:/device:0000:61:00.1/sriov:/pf: = 
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/subsystem_device: = 0002
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/subsystem_vendor: = 8086
          /agent:DPDK/hardware:/pci:/device:0000:61:00.1/vendor_id: = 8086
        /agent:DPDK/hardware:/pci:/vendor:8086 = 
          /agent:DPDK/hardware:/pci:/vendor:8086/device:1592 = 
            /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:0 = /agent:DPDK/hardware:/pci:/device:0000:61:00.0
            /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:1 = /agent:DPDK/hardware:/pci:/device:0000:61:00.1
      /agent:DPDK/hardware:/iommu: = off
      /agent:DPDK/hardware:/node:0 = 
        /agent:DPDK/hardware:/node:0/memory: = 67089633280
          /agent:DPDK/hardware:/node:0/memory:/free: = 62281928704
        /agent:DPDK/hardware:/node:0/cpu:0 = 
          /agent:DPDK/hardware:/node:0/cpu:0/core:0 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:0/thread:0 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:0/thread:0/isolated: = 0
          /agent:DPDK/hardware:/node:0/cpu:0/core:1 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:1/thread:4 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:1/thread:4/isolated: = 0
          /agent:DPDK/hardware:/node:0/cpu:0/core:2 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:2/thread:1 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:2/thread:1/isolated: = 0
          /agent:DPDK/hardware:/node:0/cpu:0/core:3 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:3/thread:5 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:3/thread:5/isolated: = 0
          /agent:DPDK/hardware:/node:0/cpu:0/core:4 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:4/thread:2 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:4/thread:2/isolated: = 0
          /agent:DPDK/hardware:/node:0/cpu:0/core:5 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:5/thread:6 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:5/thread:6/isolated: = 0
          /agent:DPDK/hardware:/node:0/cpu:0/core:6 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:6/thread:3 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:6/thread:3/isolated: = 0
          /agent:DPDK/hardware:/node:0/cpu:0/core:7 = 
            /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:0 = Unified
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:0/assoc: = 16
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:0/size: = 1048576
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:0/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:0/level: = 2
            /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:1 = Data
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:1/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:1/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:1/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:1/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:2 = Instruction
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:2/assoc: = 8
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:2/size: = 32768
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:2/linesize: = 64
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/cache:2/level: = 1
            /agent:DPDK/hardware:/node:0/cpu:0/core:7/thread:7 = 0
              /agent:DPDK/hardware:/node:0/cpu:0/core:7/thread:7/isolated: = 0
    /agent:DPDK/module:ice = 
      /agent:DPDK/module:ice/driver:pci:ice = 
      /agent:DPDK/module:ice/filename: = /tmp/linux_x86_root_353319_1693919100_2/lib/modules/5.4.0-156-generic/ice.ko
        /agent:DPDK/module:ice/filename:/fallback: = 1
        /agent:DPDK/module:ice/filename:/load_dependencies: = 1
      /agent:DPDK/module:ice/loaded: = 1
      /agent:DPDK/module:ice/loaded_oper: = 1
      /agent:DPDK/module:ice/parameter:debug = -1
      /agent:DPDK/module:ice/unload_holders: = 1
      /agent:DPDK/module:ice/version: = 0.8.1-k
    /agent:DPDK/module:vfio = 
      /agent:DPDK/module:vfio/filename: = 
        /agent:DPDK/module:vfio/filename:/fallback: = 0
        /agent:DPDK/module:vfio/filename:/load_dependencies: = 0
      /agent:DPDK/module:vfio/loaded: = 1
      /agent:DPDK/module:vfio/loaded_oper: = 1
      /agent:DPDK/module:vfio/parameter:enable_unsafe_noiommu_mode = Y
      /agent:DPDK/module:vfio/unload_holders: = 0
      /agent:DPDK/module:vfio/version: = 0.3
    /agent:DPDK/module:vfio-pci = 
      /agent:DPDK/module:vfio-pci/driver:pci:vfio-pci = 
        /agent:DPDK/module:vfio-pci/driver:pci:vfio-pci/device:0000:61:00.0 = /agent:DPDK/hardware:/pci:/device:0000:61:00.0
        /agent:DPDK/module:vfio-pci/driver:pci:vfio-pci/device:0000:61:00.1 = /agent:DPDK/hardware:/pci:/device:0000:61:00.1
      /agent:DPDK/module:vfio-pci/filename: = 
        /agent:DPDK/module:vfio-pci/filename:/fallback: = 0
        /agent:DPDK/module:vfio-pci/filename:/load_dependencies: = 0
      /agent:DPDK/module:vfio-pci/loaded: = 1
      /agent:DPDK/module:vfio-pci/loaded_oper: = 1
      /agent:DPDK/module:vfio-pci/parameter:disable_idle_d3 = N
      /agent:DPDK/module:vfio-pci/parameter:disable_vga = N
      /agent:DPDK/module:vfio-pci/parameter:nointxmask = N
      /agent:DPDK/module:vfio-pci/unload_holders: = 0
      /agent:DPDK/module:vfio-pci/version: = 0.2
    /agent:DPDK/rpcserver:iut_rpcs = 
      /agent:DPDK/rpcserver:iut_rpcs/config: = 
      /agent:DPDK/rpcserver:iut_rpcs/dead: = 0
      /agent:DPDK/rpcserver:iut_rpcs/finished: = 0
      /agent:DPDK/rpcserver:iut_rpcs/sid: = 4
    /agent:DPDK/rsrc:cpu:0:0:0:0 = /agent:DPDK/hardware:/node:0/cpu:0/core:0/thread:0
      /agent:DPDK/rsrc:cpu:0:0:0:0/acquire_attempts_timeout: = 0
      /agent:DPDK/rsrc:cpu:0:0:0:0/fallback_shared: = 0
      /agent:DPDK/rsrc:cpu:0:0:0:0/shared: = 0
    /agent:DPDK/rsrc:module:ice = /agent:DPDK/module:ice
      /agent:DPDK/rsrc:module:ice/acquire_attempts_timeout: = 3000
      /agent:DPDK/rsrc:module:ice/fallback_shared: = 1
      /agent:DPDK/rsrc:module:ice/shared: = 1
    /agent:DPDK/rsrc:module:vfio = /agent:DPDK/module:vfio
      /agent:DPDK/rsrc:module:vfio/acquire_attempts_timeout: = 3000
      /agent:DPDK/rsrc:module:vfio/fallback_shared: = 1
      /agent:DPDK/rsrc:module:vfio/shared: = 1
    /agent:DPDK/rsrc:module:vfio-pci = /agent:DPDK/module:vfio-pci
      /agent:DPDK/rsrc:module:vfio-pci/acquire_attempts_timeout: = 3000
      /agent:DPDK/rsrc:module:vfio-pci/fallback_shared: = 1
      /agent:DPDK/rsrc:module:vfio-pci/shared: = 1
    /agent:DPDK/rsrc:pci_fn:8086:1592:0 = /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:0
      /agent:DPDK/rsrc:pci_fn:8086:1592:0/acquire_attempts_timeout: = 0
      /agent:DPDK/rsrc:pci_fn:8086:1592:0/fallback_shared: = 0
      /agent:DPDK/rsrc:pci_fn:8086:1592:0/shared: = 0
    /agent:DPDK/rsrc:pci_fn:8086:1592:1 = /agent:DPDK/hardware:/pci:/vendor:8086/device:1592/instance:1
      /agent:DPDK/rsrc:pci_fn:8086:1592:1/acquire_attempts_timeout: = 0
      /agent:DPDK/rsrc:pci_fn:8086:1592:1/fallback_shared: = 0
      /agent:DPDK/rsrc:pci_fn:8086:1592:1/shared: = 0
    /agent:DPDK/sys: = 
      /agent:DPDK/sys:/debug: = 
        /agent:DPDK/sys:/debug:/exception-trace: = 1
      /agent:DPDK/sys:/net: = 
        /agent:DPDK/sys:/net:/ipv6: = 
          /agent:DPDK/sys:/net:/ipv6:/fib_multipath_hash_policy: = 0
          /agent:DPDK/sys:/net:/ipv6:/route: = 
            /agent:DPDK/sys:/net:/ipv6:/route:/mtu_expires: = 600
          /agent:DPDK/sys:/net:/ipv6:/auto_flowlabels: = 1
        /agent:DPDK/sys:/net:/ipv4: = 
          /agent:DPDK/sys:/net:/ipv4:/route: = 
            /agent:DPDK/sys:/net:/ipv4:/route:/mtu_expires: = 600
          /agent:DPDK/sys:/net:/ipv4:/tcp_congestion_control: = cubic
          /agent:DPDK/sys:/net:/ipv4:/ip_default_ttl: = 64
          /agent:DPDK/sys:/net:/ipv4:/fib_multipath_hash_policy: = 0
          /agent:DPDK/sys:/net:/ipv4:/tcp_early_retrans: = 3
          /agent:DPDK/sys:/net:/ipv4:/igmp_max_memberships: = 20
          /agent:DPDK/sys:/net:/ipv4:/tcp_dsack: = 1
          /agent:DPDK/sys:/net:/ipv4:/tcp_sack: = 1
          /agent:DPDK/sys:/net:/ipv4:/tcp_fin_timeout: = 60
          /agent:DPDK/sys:/net:/ipv4:/tcp_rmem:0 = 4096
          /agent:DPDK/sys:/net:/ipv4:/tcp_rmem:1 = 131072
          /agent:DPDK/sys:/net:/ipv4:/tcp_rmem:2 = 6291456
          /agent:DPDK/sys:/net:/ipv4:/tcp_synack_retries: = 5
          /agent:DPDK/sys:/net:/ipv4:/tcp_syn_retries: = 6
          /agent:DPDK/sys:/net:/ipv4:/tcp_orphan_retries: = 0
          /agent:DPDK/sys:/net:/ipv4:/tcp_retries2: = 15
          /agent:DPDK/sys:/net:/ipv4:/tcp_keepalive_intvl: = 75
          /agent:DPDK/sys:/net:/ipv4:/tcp_keepalive_probes: = 9
          /agent:DPDK/sys:/net:/ipv4:/tcp_keepalive_time: = 7200
          /agent:DPDK/sys:/net:/ipv4:/tcp_max_syn_backlog: = 4096
          /agent:DPDK/sys:/net:/ipv4:/tcp_syncookies: = 1
          /agent:DPDK/sys:/net:/ipv4:/tcp_timestamps: = 1
          /agent:DPDK/sys:/net:/ipv4:/ipfrag_high_thresh: = 4194304
          /agent:DPDK/sys:/net:/ipv4:/ipfrag_max_dist: = 64
          /agent:DPDK/sys:/net:/ipv4:/ip_forward: = 0
          /agent:DPDK/sys:/net:/ipv4:/tcp_wmem:0 = 4096
          /agent:DPDK/sys:/net:/ipv4:/tcp_wmem:1 = 16384
          /agent:DPDK/sys:/net:/ipv4:/tcp_wmem:2 = 4194304
        /agent:DPDK/sys:/net:/netfilter: = 
          /agent:DPDK/sys:/net:/netfilter:/nf_conntrack_max: = 262144
        /agent:DPDK/sys:/net:/core: = 
          /agent:DPDK/sys:/net:/core:/optmem_max: = 20480
          /agent:DPDK/sys:/net:/core:/rmem_max: = 212992
          /agent:DPDK/sys:/net:/core:/rmem_default: = 212992
          /agent:DPDK/sys:/net:/core:/wmem_max: = 212992
          /agent:DPDK/sys:/net:/core:/wmem_default: = 212992
          /agent:DPDK/sys:/net:/core:/somaxconn: = 4096
          /agent:DPDK/sys:/net:/core:/busy_poll: = 0
          /agent:DPDK/sys:/net:/core:/busy_read: = 0
      /agent:DPDK/sys:/fs: = 
        /agent:DPDK/sys:/fs:/file-max: = 9223372036854775807
        /agent:DPDK/sys:/fs:/file-nr:0 = 2112
        /agent:DPDK/sys:/fs:/file-nr:1 = 0
        /agent:DPDK/sys:/fs:/file-nr:2 = 9223372036854775807
      /agent:DPDK/sys:/vm: = 
        /agent:DPDK/sys:/vm:/overcommit_memory: = 0
        /agent:DPDK/sys:/vm:/nr_hugepages: = 256
      /agent:DPDK/sys:/core_pattern: = /var/tmp/core.te.%h-%p-%t
      /agent:DPDK/sys:/console_loglevel: = 4
    /agent:DPDK/sniffer_settings: = 
      /agent:DPDK/sniffer_settings:/tmp_logs: = 
        /agent:DPDK/sniffer_settings:/tmp_logs:/overfill_meth: = 0
        /agent:DPDK/sniffer_settings:/tmp_logs:/rotation: = 4
        /agent:DPDK/sniffer_settings:/tmp_logs:/total_size: = 256
        /agent:DPDK/sniffer_settings:/tmp_logs:/file_size: = 16
        /agent:DPDK/sniffer_settings:/tmp_logs:/path: = /tmp/0_23571
      /agent:DPDK/sniffer_settings:/snaplen: = 0
      /agent:DPDK/sniffer_settings:/filter_exp_file: = 
      /agent:DPDK/sniffer_settings:/filter_exp_str: = 
      /agent:DPDK/sniffer_settings:/enable: = 0
    /agent:DPDK/rlimits: = 
      /agent:DPDK/rlimits:/memlock: = 
        /agent:DPDK/rlimits:/memlock:/max: = 67108864
        /agent:DPDK/rlimits:/memlock:/cur: = 67108864
      /agent:DPDK/rlimits:/nofile: = 
        /agent:DPDK/rlimits:/nofile:/max: = 1048576
        /agent:DPDK/rlimits:/nofile:/cur: = 1024
    /agent:DPDK/uname: = Linux
      /agent:DPDK/uname:/machine: = x86_64
      /agent:DPDK/uname:/release: = 5.4.0-156-generic
      /agent:DPDK/uname:/version: = #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023
    /agent:DPDK/env:LANG = en_US.UTF-8
    /agent:DPDK/env:LC_ALL = POSIX
    /agent:DPDK/env:LD_LIBRARY_PATH = /tmp/linux_x86_root_353319_1693919100_2
    /agent:DPDK/env:LOGNAME = root
    /agent:DPDK/env:MAIL = /var/mail/root
    /agent:DPDK/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919100_2:/usr/local/sbin:/usr/sbin:/sbin
    /agent:DPDK/env:SHELL = /bin/bash
    /agent:DPDK/env:SUDO_GID = 0
    /agent:DPDK/env:SUDO_UID = 0
    /agent:DPDK/env:SUDO_USER = root
    /agent:DPDK/env:TERM = unknown
    /agent:DPDK/env:USER = root
    /agent:DPDK/rp_filter_all: = 2
    /agent:DPDK/dns: = 127.0.0.53
    /agent:DPDK/rpc_default_timeout: = 0
    /agent:DPDK/rpcprovider: = dpdkrpc
    /agent:DPDK/ip6_rt_default_if: = eno1
    /agent:DPDK/ip4_rt_default_if: = eno1
    /agent:DPDK/rule:family=10,type=1,table=255 = 
    /agent:DPDK/rule:family=2,type=1,table=255 = 
    /agent:DPDK/rule:priority=32766,family=10,type=1,table=254 = 
    /agent:DPDK/rule:priority=32766,family=2,type=1,table=254 = 
    /agent:DPDK/rule:priority=32767,family=2,type=1,table=253 = 
    /agent:DPDK/stats: = 
      /agent:DPDK/stats:/icmp_out_addr_mask_reps: = 0
      /agent:DPDK/stats:/icmp_out_addr_masks: = 0
      /agent:DPDK/stats:/icmp_out_timestamp_reps: = 0
      /agent:DPDK/stats:/icmp_out_timestamps: = 2
      /agent:DPDK/stats:/icmp_out_echo_reps: = 0
      /agent:DPDK/stats:/icmp_out_echos: = 0
      /agent:DPDK/stats:/icmp_out_redirects: = 0
      /agent:DPDK/stats:/icmp_out_src_quenchs: = 0
      /agent:DPDK/stats:/icmp_out_parm_probs: = 0
      /agent:DPDK/stats:/icmp_out_time_excds: = 0
      /agent:DPDK/stats:/icmp_out_dest_unreachs: = 0
      /agent:DPDK/stats:/icmp_out_errs: = 2
      /agent:DPDK/stats:/icmp_out_msgs: = 0
      /agent:DPDK/stats:/icmp_in_addr_mask_reps: = 0
      /agent:DPDK/stats:/icmp_in_addr_masks: = 0
      /agent:DPDK/stats:/icmp_in_timestamp_reps: = 0
      /agent:DPDK/stats:/icmp_in_timestamps: = 0
      /agent:DPDK/stats:/icmp_in_echo_reps: = 2
      /agent:DPDK/stats:/icmp_in_echos: = 0
      /agent:DPDK/stats:/icmp_in_redirects: = 0
      /agent:DPDK/stats:/icmp_in_src_quenchs: = 0
      /agent:DPDK/stats:/icmp_in_parm_probs: = 0
      /agent:DPDK/stats:/icmp_in_time_excds: = 0
      /agent:DPDK/stats:/icmp_in_dest_unreachs: = 0
      /agent:DPDK/stats:/icmp_in_errs: = 0
      /agent:DPDK/stats:/icmp_in_msgs: = 2
      /agent:DPDK/stats:/ipv4_frag_creates: = 0
      /agent:DPDK/stats:/ipv4_frag_fails: = 0
      /agent:DPDK/stats:/ipv4_frag_oks: = 0
      /agent:DPDK/stats:/ipv4_reasm_fails: = 0
      /agent:DPDK/stats:/ipv4_reasm_oks: = 0
      /agent:DPDK/stats:/ipv4_reasm_reqds: = 0
      /agent:DPDK/stats:/ipv4_reasm_timeout: = 0
      /agent:DPDK/stats:/ipv4_out_no_routes: = 0
      /agent:DPDK/stats:/ipv4_out_discards: = 0
      /agent:DPDK/stats:/ipv4_out_requests: = 732367
      /agent:DPDK/stats:/ipv4_in_delivers: = 1413702
      /agent:DPDK/stats:/ipv4_in_discards: = 0
      /agent:DPDK/stats:/ipv4_in_unknown_protos: = 0
      /agent:DPDK/stats:/ipv4_forw_dgrams: = 0
      /agent:DPDK/stats:/ipv4_in_addr_errs: = 0
      /agent:DPDK/stats:/ipv4_in_hdr_errs: = 0
      /agent:DPDK/stats:/ipv4_in_recvs: = 1413705
    /agent:DPDK/loadavg: = 
      /agent:DPDK/loadavg:/latest_pid: = 1375236
      /agent:DPDK/loadavg:/total: = 282
      /agent:DPDK/loadavg:/runnable: = 1
      /agent:DPDK/loadavg:/min15: = 0.8
      /agent:DPDK/loadavg:/min5: = 1.59
      /agent:DPDK/loadavg:/min1: = 0.7
    /agent:DPDK/ip4_fw: = 0
    /agent:DPDK/lib_bin_dir: = /tmp/linux_x86_root_353319_1693919100_2/lib/x86_64-linux
    /agent:DPDK/lib_mod_dir: = /tmp/linux_x86_root_353319_1693919100_2/lib/modules/5.4.0-156-generic
    /agent:DPDK/tmp_dir: = /tmp/linux_x86_root_353319_1693919100_2/tmp/
    /agent:DPDK/dir: = /tmp/linux_x86_root_353319_1693919100_2
    /agent:DPDK/platform: = linux_x86_64_linux_gnu__glibc2_31__kernel5_4_0_156__cpu_avx512bw__cpu_bmi2
  /agent:Peer = 
    /agent:Peer/interface:ens2f0 = 
      /agent:Peer/interface:ens2f0/arp: = 1
      /agent:Peer/interface:ens2f0/bcast_link_addr: = ff:ff:ff:ff:ff:ff
      /agent:Peer/interface:ens2f0/channels: = 
        /agent:Peer/interface:ens2f0/channels:/combined: = 
          /agent:Peer/interface:ens2f0/channels:/combined:/current: = 8
          /agent:Peer/interface:ens2f0/channels:/combined:/maximum: = 8
        /agent:Peer/interface:ens2f0/channels:/other: = 
          /agent:Peer/interface:ens2f0/channels:/other:/current: = 1
          /agent:Peer/interface:ens2f0/channels:/other:/maximum: = 1
        /agent:Peer/interface:ens2f0/channels:/rx: = 
          /agent:Peer/interface:ens2f0/channels:/rx:/current: = 0
          /agent:Peer/interface:ens2f0/channels:/rx:/maximum: = 8
        /agent:Peer/interface:ens2f0/channels:/tx: = 
          /agent:Peer/interface:ens2f0/channels:/tx:/current: = 0
          /agent:Peer/interface:ens2f0/channels:/tx:/maximum: = 8
      /agent:Peer/interface:ens2f0/coalesce: = 
        /agent:Peer/interface:ens2f0/coalesce:/param:pkt_rate_high = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:pkt_rate_low = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rate_sample_interval = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_coalesce_usecs = 50
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_coalesce_usecs_high = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_coalesce_usecs_irq = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_coalesce_usecs_low = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_max_coalesced_frames = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_max_coalesced_frames_high = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_max_coalesced_frames_irq = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:rx_max_coalesced_frames_low = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:stats_block_coalesce_usecs = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_coalesce_usecs = 50
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_coalesce_usecs_high = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_coalesce_usecs_irq = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_coalesce_usecs_low = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_max_coalesced_frames = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_max_coalesced_frames_high = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_max_coalesced_frames_irq = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:tx_max_coalesced_frames_low = 0
        /agent:Peer/interface:ens2f0/coalesce:/param:use_adaptive_rx_coalesce = 1
        /agent:Peer/interface:ens2f0/coalesce:/param:use_adaptive_tx_coalesce = 1
      /agent:Peer/interface:ens2f0/deviceinfo: = 
        /agent:Peer/interface:ens2f0/deviceinfo:/drivername: = ice
        /agent:Peer/interface:ens2f0/deviceinfo:/driverversion: = 5.15.0-79-generic
        /agent:Peer/interface:ens2f0/deviceinfo:/firmwareversion: = 2.50 0x800077a6 1.2960.0
      /agent:Peer/interface:ens2f0/feature: = 0
        /agent:Peer/interface:ens2f0/feature:/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:esp-hw-offload = 0
        /agent:Peer/interface:ens2f0/feature:esp-hw-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:esp-tx-csum-hw-offload = 0
        /agent:Peer/interface:ens2f0/feature:esp-tx-csum-hw-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:fcoe-mtu = 0
        /agent:Peer/interface:ens2f0/feature:fcoe-mtu/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:highdma = 1
        /agent:Peer/interface:ens2f0/feature:highdma/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:hsr-dup-offload = 0
        /agent:Peer/interface:ens2f0/feature:hsr-dup-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:hsr-fwd-offload = 0
        /agent:Peer/interface:ens2f0/feature:hsr-fwd-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:hsr-tag-ins-offload = 0
        /agent:Peer/interface:ens2f0/feature:hsr-tag-ins-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:hsr-tag-rm-offload = 0
        /agent:Peer/interface:ens2f0/feature:hsr-tag-rm-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:hw-tc-offload = 0
        /agent:Peer/interface:ens2f0/feature:hw-tc-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:l2-fwd-offload = 0
        /agent:Peer/interface:ens2f0/feature:l2-fwd-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:loopback = 0
        /agent:Peer/interface:ens2f0/feature:loopback/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:macsec-hw-offload = 0
        /agent:Peer/interface:ens2f0/feature:macsec-hw-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:netns-local = 0
        /agent:Peer/interface:ens2f0/feature:netns-local/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:rx-all = 0
        /agent:Peer/interface:ens2f0/feature:rx-all/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:rx-checksum = 1
        /agent:Peer/interface:ens2f0/feature:rx-checksum/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-fcs = 0
        /agent:Peer/interface:ens2f0/feature:rx-fcs/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:rx-gro = 0
        /agent:Peer/interface:ens2f0/feature:rx-gro/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-gro-hw = 0
        /agent:Peer/interface:ens2f0/feature:rx-gro-hw/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:rx-gro-list = 0
        /agent:Peer/interface:ens2f0/feature:rx-gro-list/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-hashing = 1
        /agent:Peer/interface:ens2f0/feature:rx-hashing/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-lro = 0
        /agent:Peer/interface:ens2f0/feature:rx-lro/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:rx-ntuple-filter = 1
        /agent:Peer/interface:ens2f0/feature:rx-ntuple-filter/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-udp-gro-forwarding = 0
        /agent:Peer/interface:ens2f0/feature:rx-udp-gro-forwarding/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-udp_tunnel-port-offload = 1
        /agent:Peer/interface:ens2f0/feature:rx-udp_tunnel-port-offload/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-vlan-filter = 1
        /agent:Peer/interface:ens2f0/feature:rx-vlan-filter/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-vlan-hw-parse = 1
        /agent:Peer/interface:ens2f0/feature:rx-vlan-hw-parse/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:rx-vlan-stag-filter = 0
        /agent:Peer/interface:ens2f0/feature:rx-vlan-stag-filter/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:rx-vlan-stag-hw-parse = 0
        /agent:Peer/interface:ens2f0/feature:rx-vlan-stag-hw-parse/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tls-hw-record = 0
        /agent:Peer/interface:ens2f0/feature:tls-hw-record/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tls-hw-rx-offload = 0
        /agent:Peer/interface:ens2f0/feature:tls-hw-rx-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tls-hw-tx-offload = 0
        /agent:Peer/interface:ens2f0/feature:tls-hw-tx-offload/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-checksum-fcoe-crc = 0
        /agent:Peer/interface:ens2f0/feature:tx-checksum-fcoe-crc/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-checksum-ip-generic = 0
        /agent:Peer/interface:ens2f0/feature:tx-checksum-ip-generic/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv4 = 0
        /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv4/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv6 = 0
        /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv6/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-checksum-sctp = 1
        /agent:Peer/interface:ens2f0/feature:tx-checksum-sctp/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-esp-segmentation = 0
        /agent:Peer/interface:ens2f0/feature:tx-esp-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-fcoe-segmentation = 0
        /agent:Peer/interface:ens2f0/feature:tx-fcoe-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-generic-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-generic-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-gre-csum-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-gre-csum-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-gre-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-gre-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-gso-list = 0
        /agent:Peer/interface:ens2f0/feature:tx-gso-list/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-gso-partial = 1
        /agent:Peer/interface:ens2f0/feature:tx-gso-partial/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-gso-robust = 0
        /agent:Peer/interface:ens2f0/feature:tx-gso-robust/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-ipxip4-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-ipxip4-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-ipxip6-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-ipxip6-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-lockless = 0
        /agent:Peer/interface:ens2f0/feature:tx-lockless/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-nocache-copy = 0
        /agent:Peer/interface:ens2f0/feature:tx-nocache-copy/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-scatter-gather = 1
        /agent:Peer/interface:ens2f0/feature:tx-scatter-gather/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-scatter-gather-fraglist = 0
        /agent:Peer/interface:ens2f0/feature:tx-scatter-gather-fraglist/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-sctp-segmentation = 0
        /agent:Peer/interface:ens2f0/feature:tx-sctp-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-tcp-ecn-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-tcp-ecn-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-tcp-mangleid-segmentation = 0
        /agent:Peer/interface:ens2f0/feature:tx-tcp-mangleid-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-tcp-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-tcp-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-tcp6-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-tcp6-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-tunnel-remcsum-segmentation = 0
        /agent:Peer/interface:ens2f0/feature:tx-tunnel-remcsum-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:tx-udp-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-udp-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-udp_tnl-csum-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-udp_tnl-csum-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-udp_tnl-segmentation = 1
        /agent:Peer/interface:ens2f0/feature:tx-udp_tnl-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-vlan-hw-insert = 1
        /agent:Peer/interface:ens2f0/feature:tx-vlan-hw-insert/readonly: = 0
      /agent:Peer/interface:ens2f0/feature:tx-vlan-stag-hw-insert = 0
        /agent:Peer/interface:ens2f0/feature:tx-vlan-stag-hw-insert/readonly: = 1
      /agent:Peer/interface:ens2f0/feature:vlan-challenged = 0
        /agent:Peer/interface:ens2f0/feature:vlan-challenged/readonly: = 1
      /agent:Peer/interface:ens2f0/flow_control: = 
        /agent:Peer/interface:ens2f0/flow_control:/autoneg: = 1
        /agent:Peer/interface:ens2f0/flow_control:/rx: = 0
        /agent:Peer/interface:ens2f0/flow_control:/tx: = 0
      /agent:Peer/interface:ens2f0/index: = 22
      /agent:Peer/interface:ens2f0/kind: = 
      /agent:Peer/interface:ens2f0/link_addr: = 64:9d:99:ff:ef:a2
      /agent:Peer/interface:ens2f0/msglvl: = 7
      /agent:Peer/interface:ens2f0/mtu: = 9216
      /agent:Peer/interface:ens2f0/oper_status: = 0
      /agent:Peer/interface:ens2f0/parent: = 
      /agent:Peer/interface:ens2f0/phy: = 
        /agent:Peer/interface:ens2f0/phy:/autoneg: = 0
        /agent:Peer/interface:ens2f0/phy:/duplex_admin: = full
        /agent:Peer/interface:ens2f0/phy:/duplex_oper: = unknown
        /agent:Peer/interface:ens2f0/phy:/mode:100000baseCR4_Full = 1
        /agent:Peer/interface:ens2f0/phy:/mode:100000baseLR4_ER4_Full = 0
        /agent:Peer/interface:ens2f0/phy:/mode:100000baseSR4_Full = 0
        /agent:Peer/interface:ens2f0/phy:/mode:25000baseCR_Full = 1
        /agent:Peer/interface:ens2f0/phy:/mode:25000baseSR_Full = 0
        /agent:Peer/interface:ens2f0/phy:/mode:50000baseCR2_Full = 1
        /agent:Peer/interface:ens2f0/phy:/mode:50000baseSR2_Full = 0
        /agent:Peer/interface:ens2f0/phy:/mode:Autoneg = 1
        /agent:Peer/interface:ens2f0/phy:/mode:FEC_BASER = 1
        /agent:Peer/interface:ens2f0/phy:/mode:FEC_NONE = 1
        /agent:Peer/interface:ens2f0/phy:/mode:FEC_RS = 1
        /agent:Peer/interface:ens2f0/phy:/mode:FIBRE = 1
        /agent:Peer/interface:ens2f0/phy:/mode:Pause = 0
        /agent:Peer/interface:ens2f0/phy:/set_supported: = 1
        /agent:Peer/interface:ens2f0/phy:/speed_admin: = 100000
        /agent:Peer/interface:ens2f0/phy:/speed_oper: = -1
        /agent:Peer/interface:ens2f0/phy:/state: = 0
      /agent:Peer/interface:ens2f0/private: = 
        /agent:Peer/interface:ens2f0/private:/flag:fw-lldp-agent = 0
        /agent:Peer/interface:ens2f0/private:/flag:legacy-rx = 0
        /agent:Peer/interface:ens2f0/private:/flag:link-down-on-close = 0
        /agent:Peer/interface:ens2f0/private:/flag:mdd-auto-reset-vf = 0
        /agent:Peer/interface:ens2f0/private:/flag:vf-true-promisc-support = 0
      /agent:Peer/interface:ens2f0/promisc: = 1
      /agent:Peer/interface:ens2f0/reset: = 0
      /agent:Peer/interface:ens2f0/ring: = 
        /agent:Peer/interface:ens2f0/ring:/rx: = 
          /agent:Peer/interface:ens2f0/ring:/rx:/current: = 2048
          /agent:Peer/interface:ens2f0/ring:/rx:/max: = 8160
        /agent:Peer/interface:ens2f0/ring:/tx: = 
          /agent:Peer/interface:ens2f0/ring:/tx:/current: = 256
          /agent:Peer/interface:ens2f0/ring:/tx:/max: = 8160
      /agent:Peer/interface:ens2f0/stats: = 
        /agent:Peer/interface:ens2f0/stats:/in_discards: = 0
        /agent:Peer/interface:ens2f0/stats:/in_errors: = 0
        /agent:Peer/interface:ens2f0/stats:/in_nucast_pkts: = 0
        /agent:Peer/interface:ens2f0/stats:/in_octets: = 0
        /agent:Peer/interface:ens2f0/stats:/in_ucast_pkts: = 0
        /agent:Peer/interface:ens2f0/stats:/in_unknown_protos: = 0
        /agent:Peer/interface:ens2f0/stats:/out_discards: = 0
        /agent:Peer/interface:ens2f0/stats:/out_errors: = 0
        /agent:Peer/interface:ens2f0/stats:/out_nucast_pkts: = 0
        /agent:Peer/interface:ens2f0/stats:/out_octets: = 0
        /agent:Peer/interface:ens2f0/stats:/out_ucast_pkts: = 0
      /agent:Peer/interface:ens2f0/status: = 1
    /agent:Peer/interface:ens2f1 = 
      /agent:Peer/interface:ens2f1/arp: = 1
      /agent:Peer/interface:ens2f1/bcast_link_addr: = ff:ff:ff:ff:ff:ff
      /agent:Peer/interface:ens2f1/channels: = 
        /agent:Peer/interface:ens2f1/channels:/combined: = 
          /agent:Peer/interface:ens2f1/channels:/combined:/current: = 8
          /agent:Peer/interface:ens2f1/channels:/combined:/maximum: = 8
        /agent:Peer/interface:ens2f1/channels:/other: = 
          /agent:Peer/interface:ens2f1/channels:/other:/current: = 1
          /agent:Peer/interface:ens2f1/channels:/other:/maximum: = 1
        /agent:Peer/interface:ens2f1/channels:/rx: = 
          /agent:Peer/interface:ens2f1/channels:/rx:/current: = 0
          /agent:Peer/interface:ens2f1/channels:/rx:/maximum: = 8
        /agent:Peer/interface:ens2f1/channels:/tx: = 
          /agent:Peer/interface:ens2f1/channels:/tx:/current: = 0
          /agent:Peer/interface:ens2f1/channels:/tx:/maximum: = 8
      /agent:Peer/interface:ens2f1/coalesce: = 
        /agent:Peer/interface:ens2f1/coalesce:/param:pkt_rate_high = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:pkt_rate_low = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rate_sample_interval = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_coalesce_usecs = 50
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_coalesce_usecs_high = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_coalesce_usecs_irq = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_coalesce_usecs_low = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_max_coalesced_frames = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_max_coalesced_frames_high = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_max_coalesced_frames_irq = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:rx_max_coalesced_frames_low = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:stats_block_coalesce_usecs = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_coalesce_usecs = 50
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_coalesce_usecs_high = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_coalesce_usecs_irq = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_coalesce_usecs_low = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_max_coalesced_frames = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_max_coalesced_frames_high = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_max_coalesced_frames_irq = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:tx_max_coalesced_frames_low = 0
        /agent:Peer/interface:ens2f1/coalesce:/param:use_adaptive_rx_coalesce = 1
        /agent:Peer/interface:ens2f1/coalesce:/param:use_adaptive_tx_coalesce = 1
      /agent:Peer/interface:ens2f1/deviceinfo: = 
        /agent:Peer/interface:ens2f1/deviceinfo:/drivername: = ice
        /agent:Peer/interface:ens2f1/deviceinfo:/driverversion: = 5.15.0-79-generic
        /agent:Peer/interface:ens2f1/deviceinfo:/firmwareversion: = 2.50 0x800077a6 1.2960.0
      /agent:Peer/interface:ens2f1/feature: = 0
        /agent:Peer/interface:ens2f1/feature:/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:esp-hw-offload = 0
        /agent:Peer/interface:ens2f1/feature:esp-hw-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:esp-tx-csum-hw-offload = 0
        /agent:Peer/interface:ens2f1/feature:esp-tx-csum-hw-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:fcoe-mtu = 0
        /agent:Peer/interface:ens2f1/feature:fcoe-mtu/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:highdma = 1
        /agent:Peer/interface:ens2f1/feature:highdma/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:hsr-dup-offload = 0
        /agent:Peer/interface:ens2f1/feature:hsr-dup-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:hsr-fwd-offload = 0
        /agent:Peer/interface:ens2f1/feature:hsr-fwd-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:hsr-tag-ins-offload = 0
        /agent:Peer/interface:ens2f1/feature:hsr-tag-ins-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:hsr-tag-rm-offload = 0
        /agent:Peer/interface:ens2f1/feature:hsr-tag-rm-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:hw-tc-offload = 0
        /agent:Peer/interface:ens2f1/feature:hw-tc-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:l2-fwd-offload = 0
        /agent:Peer/interface:ens2f1/feature:l2-fwd-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:loopback = 0
        /agent:Peer/interface:ens2f1/feature:loopback/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:macsec-hw-offload = 0
        /agent:Peer/interface:ens2f1/feature:macsec-hw-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:netns-local = 0
        /agent:Peer/interface:ens2f1/feature:netns-local/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:rx-all = 0
        /agent:Peer/interface:ens2f1/feature:rx-all/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:rx-checksum = 1
        /agent:Peer/interface:ens2f1/feature:rx-checksum/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-fcs = 0
        /agent:Peer/interface:ens2f1/feature:rx-fcs/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:rx-gro = 0

RING  Configurator  Self  13:05:59.377
[continuation]
        /agent:Peer/interface:ens2f1/feature:rx-gro/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-gro-hw = 0
        /agent:Peer/interface:ens2f1/feature:rx-gro-hw/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:rx-gro-list = 0
        /agent:Peer/interface:ens2f1/feature:rx-gro-list/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-hashing = 1
        /agent:Peer/interface:ens2f1/feature:rx-hashing/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-lro = 0
        /agent:Peer/interface:ens2f1/feature:rx-lro/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:rx-ntuple-filter = 1
        /agent:Peer/interface:ens2f1/feature:rx-ntuple-filter/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-udp-gro-forwarding = 0
        /agent:Peer/interface:ens2f1/feature:rx-udp-gro-forwarding/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-udp_tunnel-port-offload = 1
        /agent:Peer/interface:ens2f1/feature:rx-udp_tunnel-port-offload/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-vlan-filter = 1
        /agent:Peer/interface:ens2f1/feature:rx-vlan-filter/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-vlan-hw-parse = 1
        /agent:Peer/interface:ens2f1/feature:rx-vlan-hw-parse/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:rx-vlan-stag-filter = 0
        /agent:Peer/interface:ens2f1/feature:rx-vlan-stag-filter/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:rx-vlan-stag-hw-parse = 0
        /agent:Peer/interface:ens2f1/feature:rx-vlan-stag-hw-parse/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tls-hw-record = 0
        /agent:Peer/interface:ens2f1/feature:tls-hw-record/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tls-hw-rx-offload = 0
        /agent:Peer/interface:ens2f1/feature:tls-hw-rx-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tls-hw-tx-offload = 0
        /agent:Peer/interface:ens2f1/feature:tls-hw-tx-offload/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-checksum-fcoe-crc = 0
        /agent:Peer/interface:ens2f1/feature:tx-checksum-fcoe-crc/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-checksum-ip-generic = 0
        /agent:Peer/interface:ens2f1/feature:tx-checksum-ip-generic/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv4 = 0
        /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv4/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv6 = 0
        /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv6/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-checksum-sctp = 1
        /agent:Peer/interface:ens2f1/feature:tx-checksum-sctp/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-esp-segmentation = 0
        /agent:Peer/interface:ens2f1/feature:tx-esp-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-fcoe-segmentation = 0
        /agent:Peer/interface:ens2f1/feature:tx-fcoe-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-generic-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-generic-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-gre-csum-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-gre-csum-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-gre-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-gre-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-gso-list = 0
        /agent:Peer/interface:ens2f1/feature:tx-gso-list/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-gso-partial = 1
        /agent:Peer/interface:ens2f1/feature:tx-gso-partial/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-gso-robust = 0
        /agent:Peer/interface:ens2f1/feature:tx-gso-robust/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-ipxip4-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-ipxip4-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-ipxip6-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-ipxip6-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-lockless = 0
        /agent:Peer/interface:ens2f1/feature:tx-lockless/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-nocache-copy = 0
        /agent:Peer/interface:ens2f1/feature:tx-nocache-copy/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-scatter-gather = 1
        /agent:Peer/interface:ens2f1/feature:tx-scatter-gather/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-scatter-gather-fraglist = 0
        /agent:Peer/interface:ens2f1/feature:tx-scatter-gather-fraglist/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-sctp-segmentation = 0
        /agent:Peer/interface:ens2f1/feature:tx-sctp-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-tcp-ecn-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-tcp-ecn-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-tcp-mangleid-segmentation = 0
        /agent:Peer/interface:ens2f1/feature:tx-tcp-mangleid-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-tcp-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-tcp-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-tcp6-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-tcp6-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-tunnel-remcsum-segmentation = 0
        /agent:Peer/interface:ens2f1/feature:tx-tunnel-remcsum-segmentation/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:tx-udp-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-udp-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-udp_tnl-csum-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-udp_tnl-csum-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-udp_tnl-segmentation = 1
        /agent:Peer/interface:ens2f1/feature:tx-udp_tnl-segmentation/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-vlan-hw-insert = 1
        /agent:Peer/interface:ens2f1/feature:tx-vlan-hw-insert/readonly: = 0
      /agent:Peer/interface:ens2f1/feature:tx-vlan-stag-hw-insert = 0
        /agent:Peer/interface:ens2f1/feature:tx-vlan-stag-hw-insert/readonly: = 1
      /agent:Peer/interface:ens2f1/feature:vlan-challenged = 0
        /agent:Peer/interface:ens2f1/feature:vlan-challenged/readonly: = 1
      /agent:Peer/interface:ens2f1/flow_control: = 
        /agent:Peer/interface:ens2f1/flow_control:/autoneg: = 1
        /agent:Peer/interface:ens2f1/flow_control:/rx: = 0
        /agent:Peer/interface:ens2f1/flow_control:/tx: = 0
      /agent:Peer/interface:ens2f1/index: = 23
      /agent:Peer/interface:ens2f1/kind: = 
      /agent:Peer/interface:ens2f1/link_addr: = 64:9d:99:ff:ef:a3
      /agent:Peer/interface:ens2f1/msglvl: = 7
      /agent:Peer/interface:ens2f1/mtu: = 1500
      /agent:Peer/interface:ens2f1/oper_status: = 0
      /agent:Peer/interface:ens2f1/parent: = 
      /agent:Peer/interface:ens2f1/phy: = 
        /agent:Peer/interface:ens2f1/phy:/autoneg: = 0
        /agent:Peer/interface:ens2f1/phy:/duplex_admin: = full
        /agent:Peer/interface:ens2f1/phy:/duplex_oper: = unknown
        /agent:Peer/interface:ens2f1/phy:/mode:100000baseCR4_Full = 1
        /agent:Peer/interface:ens2f1/phy:/mode:100000baseLR4_ER4_Full = 0
        /agent:Peer/interface:ens2f1/phy:/mode:100000baseSR4_Full = 0
        /agent:Peer/interface:ens2f1/phy:/mode:25000baseCR_Full = 1
        /agent:Peer/interface:ens2f1/phy:/mode:25000baseSR_Full = 0
        /agent:Peer/interface:ens2f1/phy:/mode:50000baseCR2_Full = 1
        /agent:Peer/interface:ens2f1/phy:/mode:50000baseSR2_Full = 0
        /agent:Peer/interface:ens2f1/phy:/mode:Autoneg = 1
        /agent:Peer/interface:ens2f1/phy:/mode:FEC_BASER = 1
        /agent:Peer/interface:ens2f1/phy:/mode:FEC_NONE = 1
        /agent:Peer/interface:ens2f1/phy:/mode:FEC_RS = 1
        /agent:Peer/interface:ens2f1/phy:/mode:Pause = 0
        /agent:Peer/interface:ens2f1/phy:/set_supported: = 1
        /agent:Peer/interface:ens2f1/phy:/speed_admin: = 100000
        /agent:Peer/interface:ens2f1/phy:/speed_oper: = -1
        /agent:Peer/interface:ens2f1/phy:/state: = 0
      /agent:Peer/interface:ens2f1/private: = 
        /agent:Peer/interface:ens2f1/private:/flag:fw-lldp-agent = 0
        /agent:Peer/interface:ens2f1/private:/flag:legacy-rx = 0
        /agent:Peer/interface:ens2f1/private:/flag:link-down-on-close = 0
        /agent:Peer/interface:ens2f1/private:/flag:mdd-auto-reset-vf = 0
        /agent:Peer/interface:ens2f1/private:/flag:vf-true-promisc-support = 0
      /agent:Peer/interface:ens2f1/promisc: = 0
      /agent:Peer/interface:ens2f1/reset: = 0
      /agent:Peer/interface:ens2f1/ring: = 
        /agent:Peer/interface:ens2f1/ring:/rx: = 
          /agent:Peer/interface:ens2f1/ring:/rx:/current: = 2048
          /agent:Peer/interface:ens2f1/ring:/rx:/max: = 8160
        /agent:Peer/interface:ens2f1/ring:/tx: = 
          /agent:Peer/interface:ens2f1/ring:/tx:/current: = 256
          /agent:Peer/interface:ens2f1/ring:/tx:/max: = 8160
      /agent:Peer/interface:ens2f1/stats: = 
        /agent:Peer/interface:ens2f1/stats:/in_discards: = 0
        /agent:Peer/interface:ens2f1/stats:/in_errors: = 0
        /agent:Peer/interface:ens2f1/stats:/in_nucast_pkts: = 0
        /agent:Peer/interface:ens2f1/stats:/in_octets: = 0
        /agent:Peer/interface:ens2f1/stats:/in_ucast_pkts: = 0
        /agent:Peer/interface:ens2f1/stats:/in_unknown_protos: = 0
        /agent:Peer/interface:ens2f1/stats:/out_discards: = 0
        /agent:Peer/interface:ens2f1/stats:/out_errors: = 0
        /agent:Peer/interface:ens2f1/stats:/out_nucast_pkts: = 0
        /agent:Peer/interface:ens2f1/stats:/out_octets: = 0
        /agent:Peer/interface:ens2f1/stats:/out_ucast_pkts: = 0
      /agent:Peer/interface:ens2f1/status: = 1
    /agent:Peer/l4_port: = 
      /agent:Peer/l4_port:/alloc: = 
        /agent:Peer/l4_port:/alloc:/next: = 23900
          /agent:Peer/l4_port:/alloc:/next:/type: = 0
          /agent:Peer/l4_port:/alloc:/next:/family: = 0
    /agent:Peer/hardware: = 
      /agent:Peer/hardware:/pci: = 
        /agent:Peer/hardware:/pci:/device:0000:61:00.0 = 
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/bus: = 61
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/class: = 00020000
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/device_id: = 1592
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/domain: = 0000
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/driver: = ice
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/fn: = 0
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/net: = ens2f0
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/node: = /agent:Peer/hardware:/node:0
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/serialno: = e2-00-70-ff-ff-d9-9f-d0
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/slot: = 00
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/sriov: = 128
            /agent:Peer/hardware:/pci:/device:0000:61:00.0/sriov:/num_vfs: = 0
            /agent:Peer/hardware:/pci:/device:0000:61:00.0/sriov:/pf: = 
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/subsystem_device: = 0002
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/subsystem_vendor: = 8086
          /agent:Peer/hardware:/pci:/device:0000:61:00.0/vendor_id: = 8086
        /agent:Peer/hardware:/pci:/device:0000:61:00.1 = 
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/bus: = 61
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/class: = 00020000
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/device_id: = 1592
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/domain: = 0000
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/driver: = ice
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/fn: = 1
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/net: = ens2f1
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/node: = /agent:Peer/hardware:/node:0
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/serialno: = e2-00-70-ff-ff-d9-9f-d0
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/slot: = 00
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/sriov: = 128
            /agent:Peer/hardware:/pci:/device:0000:61:00.1/sriov:/num_vfs: = 0
            /agent:Peer/hardware:/pci:/device:0000:61:00.1/sriov:/pf: = 
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/subsystem_device: = 0002
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/subsystem_vendor: = 8086
          /agent:Peer/hardware:/pci:/device:0000:61:00.1/vendor_id: = 8086
        /agent:Peer/hardware:/pci:/vendor:8086 = 
          /agent:Peer/hardware:/pci:/vendor:8086/device:1592 = 
            /agent:Peer/hardware:/pci:/vendor:8086/device:1592/instance:0 = /agent:Peer/hardware:/pci:/device:0000:61:00.0
            /agent:Peer/hardware:/pci:/vendor:8086/device:1592/instance:1 = /agent:Peer/hardware:/pci:/device:0000:61:00.1
      /agent:Peer/hardware:/iommu: = on
      /agent:Peer/hardware:/node:0 = 
        /agent:Peer/hardware:/node:0/memory: = 67081580544
          /agent:Peer/hardware:/node:0/memory:/free: = 24847597568
        /agent:Peer/hardware:/node:0/cpu:0 = 
          /agent:Peer/hardware:/node:0/cpu:0/core:0 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:0/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:0/thread:0 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:0/thread:0/isolated: = 0
          /agent:Peer/hardware:/node:0/cpu:0/core:1 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:1/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:1/thread:4 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:1/thread:4/isolated: = 0
          /agent:Peer/hardware:/node:0/cpu:0/core:2 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:2/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:2/thread:1 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:2/thread:1/isolated: = 0
          /agent:Peer/hardware:/node:0/cpu:0/core:3 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:3/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:3/thread:5 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:3/thread:5/isolated: = 0
          /agent:Peer/hardware:/node:0/cpu:0/core:4 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:4/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:4/thread:2 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:4/thread:2/isolated: = 0
          /agent:Peer/hardware:/node:0/cpu:0/core:5 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:5/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:5/thread:6 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:5/thread:6/isolated: = 0
          /agent:Peer/hardware:/node:0/cpu:0/core:6 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:6/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:6/thread:3 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:6/thread:3/isolated: = 0
          /agent:Peer/hardware:/node:0/cpu:0/core:7 = 
            /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:0 = Unified
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:0/assoc: = 16
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:0/size: = 1048576
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:0/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:0/level: = 2
            /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:1 = Data
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:1/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:1/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:1/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:1/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:2 = Instruction
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:2/assoc: = 8
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:2/size: = 32768
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:2/linesize: = 64
              /agent:Peer/hardware:/node:0/cpu:0/core:7/cache:2/level: = 1
            /agent:Peer/hardware:/node:0/cpu:0/core:7/thread:7 = 0
              /agent:Peer/hardware:/node:0/cpu:0/core:7/thread:7/isolated: = 0
    /agent:Peer/module:ice = 
      /agent:Peer/module:ice/driver:pci:ice = 
        /agent:Peer/module:ice/driver:pci:ice/device:0000:61:00.0 = /agent:Peer/hardware:/pci:/device:0000:61:00.0
        /agent:Peer/module:ice/driver:pci:ice/device:0000:61:00.1 = /agent:Peer/hardware:/pci:/device:0000:61:00.1
      /agent:Peer/module:ice/filename: = /tmp/linux_x86_root_353319_1693919097_1/lib/modules/5.15.0-79-generic/ice.ko
        /agent:Peer/module:ice/filename:/fallback: = 1
        /agent:Peer/module:ice/filename:/load_dependencies: = 1
      /agent:Peer/module:ice/loaded: = 1
      /agent:Peer/module:ice/loaded_oper: = 1
      /agent:Peer/module:ice/parameter:debug = -1
      /agent:Peer/module:ice/unload_holders: = 1
    /agent:Peer/module:vfio-pci = 
      /agent:Peer/module:vfio-pci/driver:pci:vfio-pci = 
      /agent:Peer/module:vfio-pci/filename: = 
        /agent:Peer/module:vfio-pci/filename:/fallback: = 0
        /agent:Peer/module:vfio-pci/filename:/load_dependencies: = 0
      /agent:Peer/module:vfio-pci/loaded: = 1
      /agent:Peer/module:vfio-pci/loaded_oper: = 1
      /agent:Peer/module:vfio-pci/parameter:disable_denylist = N
      /agent:Peer/module:vfio-pci/parameter:disable_idle_d3 = N
      /agent:Peer/module:vfio-pci/parameter:disable_vga = N
      /agent:Peer/module:vfio-pci/parameter:enable_sriov = N
      /agent:Peer/module:vfio-pci/parameter:nointxmask = N
      /agent:Peer/module:vfio-pci/unload_holders: = 0
    /agent:Peer/rpcserver:tst_rpcs = 
      /agent:Peer/rpcserver:tst_rpcs/config: = 
      /agent:Peer/rpcserver:tst_rpcs/dead: = 0
      /agent:Peer/rpcserver:tst_rpcs/finished: = 0
      /agent:Peer/rpcserver:tst_rpcs/sid: = 4
    /agent:Peer/rsrc:cpu:0:0:0:0 = /agent:Peer/hardware:/node:0/cpu:0/core:0/thread:0
      /agent:Peer/rsrc:cpu:0:0:0:0/acquire_attempts_timeout: = 0
      /agent:Peer/rsrc:cpu:0:0:0:0/fallback_shared: = 0
      /agent:Peer/rsrc:cpu:0:0:0:0/shared: = 0
    /agent:Peer/rsrc:ens2f0 = /agent:Peer/interface:ens2f0
      /agent:Peer/rsrc:ens2f0/acquire_attempts_timeout: = 0
      /agent:Peer/rsrc:ens2f0/fallback_shared: = 0
      /agent:Peer/rsrc:ens2f0/shared: = 0
    /agent:Peer/rsrc:ens2f1 = /agent:Peer/interface:ens2f1
      /agent:Peer/rsrc:ens2f1/acquire_attempts_timeout: = 0
      /agent:Peer/rsrc:ens2f1/fallback_shared: = 0
      /agent:Peer/rsrc:ens2f1/shared: = 0
    /agent:Peer/rsrc:module:ice = /agent:Peer/module:ice
      /agent:Peer/rsrc:module:ice/acquire_attempts_timeout: = 3000
      /agent:Peer/rsrc:module:ice/fallback_shared: = 1
      /agent:Peer/rsrc:module:ice/shared: = 1
    /agent:Peer/rsrc:module:vfio-pci = /agent:Peer/module:vfio-pci
      /agent:Peer/rsrc:module:vfio-pci/acquire_attempts_timeout: = 3000
      /agent:Peer/rsrc:module:vfio-pci/fallback_shared: = 1
      /agent:Peer/rsrc:module:vfio-pci/shared: = 1
    /agent:Peer/rsrc:pci_fn:8086:1592:0 = /agent:Peer/hardware:/pci:/vendor:8086/device:1592/instance:0
      /agent:Peer/rsrc:pci_fn:8086:1592:0/acquire_attempts_timeout: = 0
      /agent:Peer/rsrc:pci_fn:8086:1592:0/fallback_shared: = 0
      /agent:Peer/rsrc:pci_fn:8086:1592:0/shared: = 0
    /agent:Peer/rsrc:pci_fn:8086:1592:1 = /agent:Peer/hardware:/pci:/vendor:8086/device:1592/instance:1
      /agent:Peer/rsrc:pci_fn:8086:1592:1/acquire_attempts_timeout: = 0
      /agent:Peer/rsrc:pci_fn:8086:1592:1/fallback_shared: = 0
      /agent:Peer/rsrc:pci_fn:8086:1592:1/shared: = 0
    /agent:Peer/sys: = 
      /agent:Peer/sys:/debug: = 
        /agent:Peer/sys:/debug:/exception-trace: = 1
      /agent:Peer/sys:/net: = 
        /agent:Peer/sys:/net:/ipv6: = 
          /agent:Peer/sys:/net:/ipv6:/conf:ens2f0 = 
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f0/disable_ipv6: = 1
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f0/forwarding: = 0
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f0/hop_limit: = 64
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f0/keep_addr_on_down: = 0
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f0/proxy_ndp: = 0
          /agent:Peer/sys:/net:/ipv6:/conf:ens2f1 = 
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f1/disable_ipv6: = 1
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f1/forwarding: = 0
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f1/hop_limit: = 64
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f1/keep_addr_on_down: = 0
            /agent:Peer/sys:/net:/ipv6:/conf:ens2f1/proxy_ndp: = 0
          /agent:Peer/sys:/net:/ipv6:/fib_multipath_hash_policy: = 0
          /agent:Peer/sys:/net:/ipv6:/neigh:ens2f0 = 
          /agent:Peer/sys:/net:/ipv6:/neigh:ens2f1 = 
          /agent:Peer/sys:/net:/ipv6:/route: = 
            /agent:Peer/sys:/net:/ipv6:/route:/mtu_expires: = 600
          /agent:Peer/sys:/net:/ipv6:/auto_flowlabels: = 1
        /agent:Peer/sys:/net:/ipv4: = 
          /agent:Peer/sys:/net:/ipv4:/conf:ens2f0 = 
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f0/arp_ignore: = 0
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f0/forwarding: = 1
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f0/proxy_arp: = 0
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f0/rp_filter: = 2
          /agent:Peer/sys:/net:/ipv4:/conf:ens2f1 = 
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f1/arp_ignore: = 0
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f1/forwarding: = 1
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f1/proxy_arp: = 0
            /agent:Peer/sys:/net:/ipv4:/conf:ens2f1/rp_filter: = 2
          /agent:Peer/sys:/net:/ipv4:/neigh:ens2f0 = 
          /agent:Peer/sys:/net:/ipv4:/neigh:ens2f1 = 
          /agent:Peer/sys:/net:/ipv4:/route: = 
            /agent:Peer/sys:/net:/ipv4:/route:/mtu_expires: = 600
          /agent:Peer/sys:/net:/ipv4:/tcp_congestion_control: = cubic
          /agent:Peer/sys:/net:/ipv4:/ip_default_ttl: = 64
          /agent:Peer/sys:/net:/ipv4:/fib_multipath_hash_policy: = 0
          /agent:Peer/sys:/net:/ipv4:/tcp_early_retrans: = 3
          /agent:Peer/sys:/net:/ipv4:/igmp_max_memberships: = 20
          /agent:Peer/sys:/net:/ipv4:/tcp_dsack: = 1
          /agent:Peer/sys:/net:/ipv4:/tcp_sack: = 1
          /agent:Peer/sys:/net:/ipv4:/tcp_fin_timeout: = 60
          /agent:Peer/sys:/net:/ipv4:/tcp_rmem:0 = 4096
          /agent:Peer/sys:/net:/ipv4:/tcp_rmem:1 = 131072
          /agent:Peer/sys:/net:/ipv4:/tcp_rmem:2 = 6291456
          /agent:Peer/sys:/net:/ipv4:/tcp_synack_retries: = 5
          /agent:Peer/sys:/net:/ipv4:/tcp_syn_retries: = 6
          /agent:Peer/sys:/net:/ipv4:/tcp_orphan_retries: = 0
          /agent:Peer/sys:/net:/ipv4:/tcp_retries2: = 15
          /agent:Peer/sys:/net:/ipv4:/tcp_keepalive_intvl: = 75
          /agent:Peer/sys:/net:/ipv4:/tcp_keepalive_probes: = 9
          /agent:Peer/sys:/net:/ipv4:/tcp_keepalive_time: = 7200
          /agent:Peer/sys:/net:/ipv4:/tcp_max_syn_backlog: = 4096
          /agent:Peer/sys:/net:/ipv4:/tcp_syncookies: = 1
          /agent:Peer/sys:/net:/ipv4:/tcp_timestamps: = 1
          /agent:Peer/sys:/net:/ipv4:/ipfrag_high_thresh: = 4194304
          /agent:Peer/sys:/net:/ipv4:/ipfrag_max_dist: = 64
          /agent:Peer/sys:/net:/ipv4:/ip_forward: = 1
          /agent:Peer/sys:/net:/ipv4:/tcp_wmem:0 = 4096
          /agent:Peer/sys:/net:/ipv4:/tcp_wmem:1 = 16384
          /agent:Peer/sys:/net:/ipv4:/tcp_wmem:2 = 4194304
        /agent:Peer/sys:/net:/netfilter: = 
          /agent:Peer/sys:/net:/netfilter:/nf_conntrack_max: = 262144
        /agent:Peer/sys:/net:/core: = 
          /agent:Peer/sys:/net:/core:/optmem_max: = 20480
          /agent:Peer/sys:/net:/core:/rmem_max: = 212992
          /agent:Peer/sys:/net:/core:/rmem_default: = 212992
          /agent:Peer/sys:/net:/core:/wmem_max: = 212992
          /agent:Peer/sys:/net:/core:/wmem_default: = 212992
          /agent:Peer/sys:/net:/core:/somaxconn: = 4096
          /agent:Peer/sys:/net:/core:/busy_poll: = 0
          /agent:Peer/sys:/net:/core:/busy_read: = 0
      /agent:Peer/sys:/fs: = 
        /agent:Peer/sys:/fs:/file-max: = 9223372036854775807
        /agent:Peer/sys:/fs:/file-nr:0 = 2816
        /agent:Peer/sys:/fs:/file-nr:1 = 0
        /agent:Peer/sys:/fs:/file-nr:2 = 9223372036854775807
      /agent:Peer/sys:/vm: = 
        /agent:Peer/sys:/vm:/overcommit_memory: = 0
        /agent:Peer/sys:/vm:/nr_hugepages: = 32
      /agent:Peer/sys:/core_pattern: = /var/tmp/core.te.%h-%p-%t
      /agent:Peer/sys:/console_loglevel: = 4
    /agent:Peer/sniffer_settings: = 
      /agent:Peer/sniffer_settings:/tmp_logs: = 
        /agent:Peer/sniffer_settings:/tmp_logs:/overfill_meth: = 0
        /agent:Peer/sniffer_settings:/tmp_logs:/rotation: = 4
        /agent:Peer/sniffer_settings:/tmp_logs:/total_size: = 256
        /agent:Peer/sniffer_settings:/tmp_logs:/file_size: = 16
        /agent:Peer/sniffer_settings:/tmp_logs:/path: = /tmp/0_23571
      /agent:Peer/sniffer_settings:/snaplen: = 0
      /agent:Peer/sniffer_settings:/filter_exp_file: = 
      /agent:Peer/sniffer_settings:/filter_exp_str: = 
      /agent:Peer/sniffer_settings:/enable: = 0
    /agent:Peer/rlimits: = 
      /agent:Peer/rlimits:/memlock: = 
        /agent:Peer/rlimits:/memlock:/max: = 8385196032
        /agent:Peer/rlimits:/memlock:/cur: = 8385196032
      /agent:Peer/rlimits:/nofile: = 
        /agent:Peer/rlimits:/nofile:/max: = 1048576
        /agent:Peer/rlimits:/nofile:/cur: = 1024
    /agent:Peer/uname: = Linux
      /agent:Peer/uname:/machine: = x86_64
      /agent:Peer/uname:/release: = 5.15.0-79-generic
      /agent:Peer/uname:/version: = #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023
    /agent:Peer/env:LANG = en_US.UTF-8
    /agent:Peer/env:LC_ALL = POSIX
    /agent:Peer/env:LD_LIBRARY_PATH = /tmp/linux_x86_root_353319_1693919097_1
    /agent:Peer/env:LOGNAME = root
    /agent:Peer/env:MAIL = /var/mail/root
    /agent:Peer/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919097_1:/usr/local/sbin:/usr/sbin:/sbin
    /agent:Peer/env:SHELL = /bin/bash
    /agent:Peer/env:SUDO_GID = 0
    /agent:Peer/env:SUDO_UID = 0
    /agent:Peer/env:SUDO_USER = root
    /agent:Peer/env:TERM = unknown
    /agent:Peer/env:USER = root
    /agent:Peer/env:XDG_CACHE_HOME = /var/cache/user/root
    /agent:Peer/rp_filter_all: = 2
    /agent:Peer/dns: = 127.0.0.53
    /agent:Peer/rpc_default_timeout: = 0
    /agent:Peer/rpcprovider: = ta_rpcs
    /agent:Peer/ip6_rt_default_if: = eno1
    /agent:Peer/ip4_rt_default_if: = eno1
    /agent:Peer/rule:family=10,type=1,table=255 = 
    /agent:Peer/rule:family=2,type=1,table=255 = 
    /agent:Peer/rule:priority=32766,family=10,type=1,table=254 = 
    /agent:Peer/rule:priority=32766,family=2,type=1,table=254 = 
    /agent:Peer/rule:priority=32767,family=2,type=1,table=253 = 
    /agent:Peer/stats: = 
      /agent:Peer/stats:/icmp_out_addr_mask_reps: = 0
      /agent:Peer/stats:/icmp_out_addr_masks: = 0
      /agent:Peer/stats:/icmp_out_timestamp_reps: = 0
      /agent:Peer/stats:/icmp_out_timestamps: = 4
      /agent:Peer/stats:/icmp_out_echo_reps: = 0
      /agent:Peer/stats:/icmp_out_echos: = 0
      /agent:Peer/stats:/icmp_out_redirects: = 0
      /agent:Peer/stats:/icmp_out_src_quenchs: = 0
      /agent:Peer/stats:/icmp_out_parm_probs: = 0
      /agent:Peer/stats:/icmp_out_time_excds: = 2
      /agent:Peer/stats:/icmp_out_dest_unreachs: = 0
      /agent:Peer/stats:/icmp_out_errs: = 6
      /agent:Peer/stats:/icmp_out_msgs: = 0
      /agent:Peer/stats:/icmp_in_addr_mask_reps: = 0
      /agent:Peer/stats:/icmp_in_addr_masks: = 0
      /agent:Peer/stats:/icmp_in_timestamp_reps: = 0
      /agent:Peer/stats:/icmp_in_timestamps: = 0
      /agent:Peer/stats:/icmp_in_echo_reps: = 4
      /agent:Peer/stats:/icmp_in_echos: = 0
      /agent:Peer/stats:/icmp_in_redirects: = 0
      /agent:Peer/stats:/icmp_in_src_quenchs: = 0
      /agent:Peer/stats:/icmp_in_parm_probs: = 0
      /agent:Peer/stats:/icmp_in_time_excds: = 2
      /agent:Peer/stats:/icmp_in_dest_unreachs: = 0
      /agent:Peer/stats:/icmp_in_errs: = 0
      /agent:Peer/stats:/icmp_in_msgs: = 6
      /agent:Peer/stats:/ipv4_frag_creates: = 0
      /agent:Peer/stats:/ipv4_frag_fails: = 0
      /agent:Peer/stats:/ipv4_frag_oks: = 0
      /agent:Peer/stats:/ipv4_reasm_fails: = 0
      /agent:Peer/stats:/ipv4_reasm_oks: = 0
      /agent:Peer/stats:/ipv4_reasm_reqds: = 0
      /agent:Peer/stats:/ipv4_reasm_timeout: = 0
      /agent:Peer/stats:/ipv4_out_no_routes: = 0
      /agent:Peer/stats:/ipv4_out_discards: = 0
      /agent:Peer/stats:/ipv4_out_requests: = 275599
      /agent:Peer/stats:/ipv4_in_delivers: = 341522
      /agent:Peer/stats:/ipv4_in_discards: = 0
      /agent:Peer/stats:/ipv4_in_unknown_protos: = 0
      /agent:Peer/stats:/ipv4_forw_dgrams: = 0
      /agent:Peer/stats:/ipv4_in_addr_errs: = 2
      /agent:Peer/stats:/ipv4_in_hdr_errs: = 0
      /agent:Peer/stats:/ipv4_in_recvs: = 341527
    /agent:Peer/loadavg: = 
      /agent:Peer/loadavg:/latest_pid: = 141681
      /agent:Peer/loadavg:/total: = 361
      /agent:Peer/loadavg:/runnable: = 1
      /agent:Peer/loadavg:/min15: = 0.81
      /agent:Peer/loadavg:/min5: = 1.47
      /agent:Peer/loadavg:/min1: = 0.46
    /agent:Peer/ip4_fw: = 1
    /agent:Peer/lib_bin_dir: = /tmp/linux_x86_root_353319_1693919097_1/lib/x86_64-linux
    /agent:Peer/lib_mod_dir: = /tmp/linux_x86_root_353319_1693919097_1/lib/modules/5.15.0-79-generic
    /agent:Peer/tmp_dir: = /tmp/linux_x86_root_353319_1693919097_1/tmp/
    /agent:Peer/dir: = /tmp/linux_x86_root_353319_1693919097_1
    /agent:Peer/platform: = linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2
  /volatile: = 
    /volatile:/alien_link_addr: = 00:10:29:38:47:58
    /volatile:/fake_link_addr: = 00:10:29:38:47:57
    /volatile:/sockaddr_port: = 0

WARN  RCF  DPDK  13:05:59.422
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/DPDK/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)

WARN  RCF  DPDK  13:05:59.704
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.0 (socket 0)

WARN  RCF  DPDK  13:06:00.096
ice_load_pkg_type(): Active package is: 1.3.4.0, ICE OS Default Package (single VLAN mode)

RING  prologue  TAPI RPC  13:06:00.113
RPC (DPDK,iut_rpcs[2]): rte_eal_init(8, 0x55cd6aeffa90("iut_rpcs", "--allow=0000:61:00.0", "-n", "2", "--file-prefix", "DPDK", "-c", "0x01")) -> 7 (RPC-EAGAIN)

RING  prologue  Configurator API  13:06:00.116
Set /agent:DPDK/rpcserver:iut_rpcs/config: =  iut_rpcs --allow=0000:61:00.0 -n 2 --file-prefix DPDK -c 0x01

RING  prologue  TAPI RPC  13:06:00.117
RPC (DPDK,iut_rpcs[3]): rte_eth_dev_info_get(0) -> dev_info={ driver_name=net_ice, if_index=0, min_mtu=68, max_mtu=9702, min_rx_bufsize=1024, max_rx_pktlen=9728, max_rx_queues=256, max_tx_queues=256, max_mac_addrs=64, max_hash_mac_addrs=0, max_vfs=0, max_vmdq_pools=0, rx_queue_offload_capa=BUFFER_SPLIT, rx_offload_capa=VLAN_STRIP|IPV4_CKSUM|UDP_CKSUM|TCP_CKSUM|QINQ_STRIP|OUTER_IPV4_CKSUM|VLAN_FILTER|VLAN_EXTEND|SCATTER|TIMESTAMP|KEEP_CRC|RSS_HASH|BUFFER_SPLIT, tx_queue_offload_capa=MBUF_FAST_FREE, tx_offload_capa=VLAN_INSERT|IPV4_CKSUM|UDP_CKSUM|TCP_CKSUM|SCTP_CKSUM|TCP_TSO|OUTER_IPV4_CKSUM|QINQ_INSERT|MULTI_SEGS|MBUF_FAST_FREE|OUTER_UDP_CKSUM, reta_size=512, hash_key_size=52, flow_type_rss_offloads=IPV4|FRAG_IPV4|NONFRAG_IPV4_TCP|NONFRAG_IPV4_UDP|NONFRAG_IPV4_SCTP|NONFRAG_IPV4_OTHER|IPV6|FRAG_IPV6|NONFRAG_IPV6_TCP|NONFRAG_IPV6_UDP|NONFRAG_IPV6_SCTP|NONFRAG_IPV6_OTHER|L2_PAYLOAD, default_rxconf={ rx_thresh={ pthresh=8, hthresh=8, wthresh=0 }, rx_free_thresh=32, rx_drop_en=0, rx_deferred_start=0, offloads= }, default_txconf={ tx_thresh={ pthresh=32, hthresh=0, wthresh=0 }, tx_rs_thresh=32, tx_free_thresh=32, txq_flags=, tx_deferred_start=0, offloads= }, vmdq_queue_base=0, vmdq_queue_num=0, vmdq_pool_base=0, rx_desc_lim={ nb_max=4096, nb_min=64, nb_align=32, nb_seg_max=0, nb_mtu_seg_max=0 }, tx_desc_lim={ nb_max=4096, nb_min=64, nb_align=32, nb_seg_max=0, nb_mtu_seg_max=0 }, speed_capa=10M|100M|1G|2_5G|5G|10G|20G|25G|50G|100G, nb_rx_queues=0, nb_tx_queues=0, default_rxportconf={ burst_size=32, ring_size=1024, nb_queues=1 }, default_txportconf={ burst_size=32, ring_size=1024, nb_queues=1 }, dev_capa= } (RPC-EAGAIN)

RING  prologue  TAPI RPC  13:06:00.119
RPC (DPDK,iut_rpcs[4]): rte_eth_dev_get_name_by_port(port_id=0) -> name='0000:61:00.0'; 0 (RPC-EAGAIN)

WARN  RCF  DPDK  13:06:00.123
TELEMETRY: No legacy callbacks, legacy socket not created

RING  prologue  TAPI RPC  13:06:00.124
RPC (DPDK,iut_rpcs[5]): rte_eth_dev_configure(0, 1, 1, { link_speeds=0, rxmode={ mq_mode=NONE, mtu=0, offloads=, flags=HW_STRIP_CRC }, txmode={ mq_mode=NONE, offloads=, pvid=0, flags= }, lbpk_mode=0, rx_conf_adv={ rss_conf={rss_key=, rss_key_len=0, rss_hf=} }, dcb_cap_en=0, intr_conf={ lsc=0, rxq=0 } }) -> 0 (RPC-EAGAIN)

RING  prologue  TAPI RPC  13:06:00.124
RPC (DPDK,iut_rpcs[6]): rte_eth_dev_socket_id(0) -> 0 (RPC-EAGAIN)

RING  prologue  TAPI RPC  13:06:00.127
RPC (DPDK,iut_rpcs[7]): rte_pktmbuf_pool_create(name=pkts, n=544, cache_size=256, priv_size=0, data_room_size=2304, socket_id=0) -> rte_mempool(0x1) (RPC-ENOENT)

RING  prologue  TAPI RPC  13:06:00.127
RPC (DPDK,iut_rpcs[8]): rte_eth_rx_queue_setup(0, 0, 64, 0, (null), rte_mempool(0x1)) -> 0 (RPC-ENOENT)

RING  prologue  TAPI RPC  13:06:00.128
RPC (DPDK,iut_rpcs[9]): rte_eth_tx_queue_setup(0, 0, 64, 0, (null)) -> 0 (RPC-ENOENT)

RING  @DPDK  Self  13:06:00.128
iut_rpcs.1375235.877186432: rte_eth_dev_stop(0)

RING  @DPDK  Self  13:06:00.129
iut_rpcs.1375235.877186432: rte_eth_dev_close(0)

ERROR  prologue  TAPI RPC  13:06:00.130
RPC (DPDK,iut_rpcs[10]): rte_eth_dev_start(0) -> -EIO (RPC-ENOENT)

INFO  prologue  TAPI Jumps  13:06:00.130
Jump from ../src/lib/rpcc_dpdk/ethdev.c:826 to ../../prologue.c:721 rc=EFAIL

RING  prologue  Step  13:06:00.130
Test cleanup

RING  @Peer  Self  13:06:00.131
tst_rpcs.141600.3363830784: RPC server 'tst_rpcs' finishing status: OK

RING  prologue  Configurator API  13:06:00.133
Deleted /agent:Peer/rpcserver:tst_rpcs

WARN  RCF  DPDK  13:06:00.144
ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)
ice_rx_queue_start(): fail to program RX queue 0
ice_dev_start(): fail to start Rx queue 0
Device with port_id=0 already stopped

RING  @DPDK  Self  13:06:00.159
iut_rpcs.1375235.877186432: RPC server 'iut_rpcs' finishing status: OK

RING  prologue  Configurator API  13:06:00.317
Deleted /agent:DPDK/rpcserver:iut_rpcs

RING  Tester  Run  13:06:00.318

Obtained result is:
FAILED

Expected results are: default
PASSED

MI  Tester  Control  13:06:00.318
TEST "prologue" finished
Node ID 2, Parent ID 1, Plan ID 1

Obtained result:
Status: FAILED

ERROR: Unexpected test result

Expected results:

Status: PASSED

RING  @DPDK  Self  13:06:01.435
iut_rpcs.1375243.3577146752: RPC server 'iut_rpcs' (64-bit) (re-)started (PID 1375243, TID 3577146752)

RING  Configurator  RCF API  13:06:01.441
Add /agent:DPDK/rpcserver:iut_rpcs: OK

RING  @Peer  Self  13:06:01.458
tst_rpcs.141722.1908444160: RPC server 'tst_rpcs' (64-bit) (re-)started (PID 141722, TID 1908444160)

RING  Configurator  RCF API  13:06:01.459
Add /agent:Peer/rpcserver:tst_rpcs: OK

RING  Configurator  RCF API  13:06:01.474
Set /agent:Peer/interface:ens2f0/promisc: to 0: OK

RING  Configurator  RCF API  13:06:01.515
Set /agent:Peer/interface:ens2f0/mtu: to 1500: OK

RING  Configurator  RCF API  13:06:01.533
Delete /agent:Peer/rsrc:cpu:0:0:0:0: OK

RING  Configurator  RCF API  13:06:01.952
Delete /agent:DPDK/rsrc:cpu:0:0:0:0: OK

RING  @DPDK  Self  13:06:01.993
iut_rpcs.1375243.3577146752: RPC server 'iut_rpcs' finishing status: OK

RING  @Peer  Self  13:06:01.999
tst_rpcs.141722.1908444160: RPC server 'tst_rpcs' finishing status: OK

RING  Configurator  RCF API  13:06:01.999
Delete /agent:DPDK/rpcserver:iut_rpcs: OK

RING  Configurator  RCF API  13:06:02.001
Delete /agent:Peer/rpcserver:tst_rpcs: OK

RING  Configurator  RCF API  13:06:02.003
Add /agent:Peer/interface:ens2f1/net_addr:10.38.11.2 with value 24: OK

RING  Configurator  RCF API  13:06:02.035
Set /agent:Peer/sys:/net:/ipv6:/conf:ens2f1/disable_ipv6: to 0: OK

RING  Configurator  RCF API  13:06:02.037
Add /agent:Peer/interface:ens2f0/net_addr:10.38.10.2 with value 24: OK

RING  Configurator  RCF API  13:06:02.055
Set /agent:Peer/sys:/net:/ipv6:/conf:ens2f0/disable_ipv6: to 0: OK

RING  Configurator  RCF API  13:06:02.056
Set /agent:Peer/interface:ens2f1/net_addr:10.38.11.2/broadcast: to 10.38.11.255: OK

RING  Configurator  RCF API  13:06:02.057
Delete /agent:Peer/interface:ens2f1/net_addr:10.38.11.2: OK

RING  Configurator  RCF API  13:06:02.065
Set /agent:Peer/interface:ens2f0/net_addr:10.38.10.2/broadcast: to 10.38.10.255: OK

RING  Configurator  RCF API  13:06:02.066
Delete /agent:Peer/interface:ens2f0/net_addr:10.38.10.2: OK

RING  Configurator  RCF API  13:06:02.084
Set /agent:Peer/interface:ens2f1/status: to 0: OK

RING  Configurator  RCF API  13:06:02.121
Set /agent:Peer/interface:ens2f0/status: to 0: OK

RING  Configurator  RCF API  13:06:02.137
Set /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv6 to 1: OK

RING  Configurator  RCF API  13:06:02.139
Set /agent:Peer/interface:ens2f1/feature:tx-checksum-ipv4 to 1: OK

RING  Configurator  RCF API  13:06:02.142
Set /agent:Peer/interface:ens2f1/feature:tx-checksum-ip-generic to 0: OK

RING  Configurator  RCF API  13:06:02.144
Set /agent:Peer/interface:ens2f1/feature:rx-lro to 0: OK

RING  Configurator  RCF API  13:06:02.147
Set /agent:Peer/interface:ens2f1/feature:rx-gro to 1: OK

RING  Configurator  RCF API  13:06:02.149
Set /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv6 to 1: OK

RING  Configurator  RCF API  13:06:02.152
Set /agent:Peer/interface:ens2f0/feature:tx-checksum-ipv4 to 1: OK

RING  Configurator  RCF API  13:06:02.154
Set /agent:Peer/interface:ens2f0/feature:tx-checksum-ip-generic to 0: OK

RING  Configurator  RCF API  13:06:02.157
Set /agent:Peer/interface:ens2f0/feature:rx-lro to 0: OK

RING  Configurator  RCF API  13:06:02.160
Set /agent:Peer/interface:ens2f0/feature:rx-gro to 1: OK

RING  Configurator  RCF API  13:06:02.163
Delete /agent:Peer/rsrc:ens2f1: OK

RING  Configurator  RCF API  13:06:02.403
Delete /agent:Peer/rsrc:ens2f0: OK

RING  Configurator  RCF API  13:06:02.844
Set /agent:DPDK/hardware:/pci:/device:0000:61:00.1/driver: to ice: OK

RING  Configurator  RCF API  13:06:02.905
Set /agent:DPDK/hardware:/pci:/device:0000:61:00.0/driver: to ice: OK

RING  @DPDK  Unix Conf System Module  13:06:02.906
Module 'vfio' already loaded

RING  Configurator  RCF API  13:06:02.912
Set /agent:DPDK/module:vfio/loaded: to 1: OK

RING  Configurator  RCF API  13:06:02.919
Delete /agent:DPDK/module:vfio: OK

RING  Configurator  RCF API  13:06:02.922
Set /agent:DPDK/rsrc:module:vfio to : OK

RING  Configurator  RCF API  13:06:02.969
Set /agent:DPDK/rsrc:module:vfio/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  13:06:02.970
Set /agent:DPDK/rsrc:module:vfio/fallback_shared: to 0: OK

RING  Configurator  RCF API  13:06:02.970
Delete /agent:DPDK/rsrc:module:vfio: OK

RING  Configurator  RCF API  13:06:03.010
Set /agent:Peer/module:ice/loaded: to 0: OK

RING  @Peer  Unix Conf System Module  13:06:03.017
Module 'ice' already loaded

RING  Configurator  RCF API  13:06:03.018
Set /agent:Peer/rsrc:module:ice/acquire_attempts_timeout: to 3000: OK

RING  Configurator  RCF API  13:06:03.018
Set /agent:Peer/rsrc:module:ice/fallback_shared: to 1: OK

RING  Configurator  RCF API  13:06:03.018
Set /agent:Peer/module:ice/loaded: to 1: OK

RING  Configurator  RCF API  13:06:03.026
Set /agent:Peer/module:ice/unload_holders: to 0: OK

RING  Configurator  RCF API  13:06:03.026
Delete /agent:Peer/module:ice: OK

RING  Configurator  RCF API  13:06:03.029
Set /agent:Peer/rsrc:module:ice to : OK

RING  Configurator  RCF API  13:06:03.111
Set /agent:Peer/rsrc:module:ice/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  13:06:03.111
Set /agent:Peer/rsrc:module:ice/fallback_shared: to 0: OK

RING  Configurator  RCF API  13:06:03.112
Delete /agent:Peer/rsrc:module:ice: OK

RING  @Peer  Unix Conf System Module  13:06:03.187
Module 'vfio-pci' already loaded

RING  Configurator  RCF API  13:06:03.188
Set /agent:Peer/module:vfio-pci/loaded: to 1: OK

RING  Configurator  RCF API  13:06:03.196
Delete /agent:Peer/module:vfio-pci: OK

RING  Configurator  RCF API  13:06:03.199
Set /agent:Peer/rsrc:module:vfio-pci to : OK

RING  Configurator  RCF API  13:06:03.273
Set /agent:Peer/rsrc:module:vfio-pci/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  13:06:03.274
Set /agent:Peer/rsrc:module:vfio-pci/fallback_shared: to 0: OK

RING  Configurator  RCF API  13:06:03.274
Delete /agent:Peer/rsrc:module:vfio-pci: OK

RING  Configurator  RCF API  13:06:03.351
Set /agent:DPDK/module:ice/loaded: to 0: OK

RING  @DPDK  Unix Conf System Module  13:06:03.353
Module 'ice' already loaded

RING  Configurator  RCF API  13:06:03.358
Set /agent:DPDK/rsrc:module:ice/acquire_attempts_timeout: to 3000: OK

RING  Configurator  RCF API  13:06:03.358
Set /agent:DPDK/rsrc:module:ice/fallback_shared: to 1: OK

RING  Configurator  RCF API  13:06:03.358
Set /agent:DPDK/module:ice/loaded: to 1: OK

RING  Configurator  RCF API  13:06:03.365
Set /agent:DPDK/module:ice/unload_holders: to 0: OK

RING  Configurator  RCF API  13:06:03.365
Delete /agent:DPDK/module:ice: OK

RING  Configurator  RCF API  13:06:03.368
Set /agent:DPDK/rsrc:module:ice to : OK

RING  Configurator  RCF API  13:06:03.412
Set /agent:DPDK/rsrc:module:ice/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  13:06:03.412
Set /agent:DPDK/rsrc:module:ice/fallback_shared: to 0: OK

RING  Configurator  RCF API  13:06:03.413
Delete /agent:DPDK/rsrc:module:ice: OK

RING  @DPDK  Unix Conf System Module  13:06:03.444
Module 'vfio-pci' already loaded

RING  Configurator  RCF API  13:06:03.449
Set /agent:DPDK/module:vfio-pci/loaded: to 1: OK

RING  Configurator  RCF API  13:06:03.456
Delete /agent:DPDK/module:vfio-pci: OK

RING  Configurator  RCF API  13:06:03.459
Set /agent:DPDK/rsrc:module:vfio-pci to : OK

RING  Configurator  RCF API  13:06:03.497
Set /agent:DPDK/rsrc:module:vfio-pci/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  13:06:03.497
Set /agent:DPDK/rsrc:module:vfio-pci/fallback_shared: to 0: OK

RING  Configurator  RCF API  13:06:03.498
Delete /agent:DPDK/rsrc:module:vfio-pci: OK

RING  Configurator  RCF API  13:06:03.530
Delete /agent:Peer/rsrc:pci_fn:8086:1592:1: OK

RING  Configurator  RCF API  13:06:03.575
Delete /agent:DPDK/rsrc:pci_fn:8086:1592:1: OK

RING  Configurator  RCF API  13:06:03.600
Delete /agent:Peer/rsrc:pci_fn:8086:1592:0: OK

RING  Configurator  RCF API  13:06:03.616
Delete /agent:DPDK/rsrc:pci_fn:8086:1592:0: OK

RING  Configurator  RCF API  13:06:03.632
Set /agent:Peer/env:PATH to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919097_1: OK

RING  Configurator  RCF API  13:06:03.635
Set /agent:DPDK/env:PATH to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_353319_1693919100_2: OK

RING  Configurator  Self  13:06:03.637
Sleep 5000 milliseconds to propagate configuration changes

MI  Tester  Control  13:06:08.905
PACKAGE "dpdk-ethdev-ts" finished
Node ID 1, Parent ID 0, Plan ID 0

Obtained result:
Status: FAILED
Verdicts:
 *  Session prologue failed

ERROR: Session prologue failed

Expected results:

Status: PASSED

RING  Tester  Self  13:06:08.917
Done

RING  Dispatcher  Start  13:06:09.057
Shutdown Configurator

RING  Configurator  RCF API  13:06:09.061
Set /agent:Peer/sys:/core_pattern: to |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E: OK

RING  Configurator  RCF API  13:06:09.061
Set /agent:DPDK/sys:/core_pattern: to |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E: OK

RING  Configurator  RCF API  13:06:09.062
Set /agent:Peer/rpcprovider: to : OK

RING  Configurator  RCF API  13:06:09.062
Set /agent:DPDK/rpcprovider: to : OK

RING  Configurator  Self  13:06:09.129
Exit

RING  Dispatcher  Start  13:06:09.144
Flush log

RING  Dispatcher  Start  13:06:09.163
Shutdown RCF

RING  RCF  Self  13:06:09.165
Shutting down

RING  RCF  RCF Unix  13:06:09.367
Finish method is called for TA Peer

RING  RCF  RCF Unix  13:06:09.367
CMD to remove: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "rm -rf /tmp/linux_x86_root_353319_1693919097_1"

WARN  RCF  DPDK  13:06:09.370
Exiting: 0

RING  RCF  RCF Unix  13:06:10.164
Finish method is called for TA DPDK

RING  RCF  RCF Unix  13:06:10.164
CMD to remove: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "rm -rf /tmp/linux_x86_root_353319_1693919100_2"

RING  RCF  Self  13:06:13.968
Test Agents are stopped

RING  RCF  Self  13:06:13.968
Exit

ERROR  Logger  Self  13:06:13.969
rcf_ta_get_log(ta_name='Peer') returned fatal error RCF_API-EIPC, stop gathering logs from this TA

ERROR  Logger  Self  13:06:13.969
rcf_ta_get_log(ta_name='DPDK') returned fatal error RCF_API-EIPC, stop gathering logs from this TA

RING  Logger  Self  13:06:13.969
IPC Server 'TE_LOGGER_352374-ta-DPDK' closed

RING  Logger  Self  13:06:13.969
IPC Server 'TE_LOGGER_352374-ta-Peer' closed

RING  Dispatcher  Start  13:06:13.985
Shutdown Logger

RING  Logger  Self  13:06:13.987
Logger shutdown ...

WARN  Logger  Log streaming  13:06:13.987
Not all messages in listener queue have been processed

RING  Logger  Self  13:06:13.987
Shutdown is completed

[-- Attachment #3: intel-xl710-main-fail.txt --]
[-- Type: text/plain, Size: 246478 bytes --]

Log report
~~~~~~~~~~

RING  Dispatcher  Command-line options  14:13:20.501
--conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --tester-req=TEST_HARNESS_CHECKUP --tester-req=!NO_TEST_HARNESS_CHECKUP --opts=run/iol-dts-xl710 --opts=opts.ts

RING  Dispatcher  Expanded command-line options  14:13:20.516
 --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --tester-req=TEST_HARNESS_CHECKUP --tester-req=!NO_TEST_HARNESS_CHECKUP --script=env/iol-dts --script=env/intel-xl710 --script=scripts/iut.h1 --script=scripts/iut.h1-xl710 --conf-cs=cs/dpdk-pmd-ts.yml --script=scripts/ta-def --script=scripts/defaults --tester-script=scripts/dpdk-trc-tags --tester-script=scripts/os-trc-tags --script=scripts/net-modules --script=scripts/iut-net-driver-loaded --script=scripts/disable_unused_agts

RING  Dispatcher  Start  14:15:36.988
Starting TEN applications

RING  Dispatcher  Start  14:15:37.004
Start Logger:  /opt/tsf/ts-rigs/logger.conf

RING  Logger  Cfg file  14:15:37.015
Opening config file: /opt/tsf/ts-rigs/logger.conf

RING  Logger  Log streaming  14:15:37.015
Current listeners configuration:
Listeners:
Filters:

RING  Dispatcher  Start  14:15:37.032
Start RCF:  /opt/tsf/ts-conf/rcf.conf

RING  RCF  RCF Unix  14:15:37.037
Starting TA 'Peer' type 'linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell='

RING  RCF  RCF Unix  14:15:37.037
CMD to copy: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_356502_1693923337_1" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_356502_1693923337_1 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  14:15:38.785
Command to detect shell name: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  14:15:39.035
Shell is: /bin/bash

RING  RCF  RCF Unix  14:15:39.035
Command to start TA: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_356502_1693923337_1 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_356502_1693923337_1 /tmp/linux_x86_root_356502_1693923337_1/ta Peer 23571 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=" 2>&1 | te_tee RCF Peer 10 >ta.Peer 

RING  @Peer  Main  14:15:39.290
Starting

RING  @Peer  TA unix VM  14:15:39.290
KVM is supported

RING  RCF  RCF Unix  14:15:40.038
Starting TA 'DPDK' type 'linux_x86_64_linux_gnu__glibc2_31__kernel5_4_0_156__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-dut.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:ssh_proxy=:copy_timeout=15:kill_timeout=15:sudo=:shell=:ld_preload='

RING  RCF  RCF Unix  14:15:40.038
CMD to copy: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_356502_1693923340_2" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_31__kernel5_4_0_156__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_356502_1693923340_2 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  14:15:48.071
Command to detect shell name: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  14:15:51.896
Shell is: /bin/bash

RING  RCF  RCF Unix  14:15:51.896
Command to start TA: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_356502_1693923340_2 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_356502_1693923340_2 /tmp/linux_x86_root_356502_1693923340_2/ta DPDK 23571 host=iol-dts-dut.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:ssh_proxy=:copy_timeout=15:kill_timeout=15:sudo=:shell=:ld_preload=" 2>&1 | te_tee RCF DPDK 10 >ta.DPDK 

RING  @DPDK  Main  14:15:55.693
Starting

WARN  @DPDK  TA unix VM  14:15:55.694
KVM is not supported

RING  Dispatcher  Start  14:15:55.919
Start Configurator: --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf:/opt/tsf/dpdk-ethdev-ts/ts/inst/default/share/cm/ /opt/tsf/ts-conf/cs/dpdk-pmd-ts.yml

RING  Configurator  RCF API  14:16:02.307
Set /agent:DPDK/rpcprovider: to dpdkrpc: OK

RING  Configurator  RCF API  14:16:02.308
Set /agent:Peer/rpcprovider: to ta_rpcs: OK

RING  Configurator  RCF API  14:16:02.309
Set /agent:DPDK/sys:/core_pattern: to /var/tmp/core.te.%h-%p-%t: OK

RING  Configurator  RCF API  14:16:02.309
Set /agent:Peer/sys:/core_pattern: to /var/tmp/core.te.%h-%p-%t: OK

RING  Dispatcher  Start  14:16:09.953
Start Tester: --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --req="TEST_HARNESS_CHECKUP" --req="!NO_TEST_HARNESS_CHECKUP" --trc-tag=dpdk-23.11.0-rc0 --trc-tag=dpdk:23110000 --trc-tag=vfio-pci --trc-tag=kernel-linux --trc-tag=linux-mm:504 /opt/tsf/dpdk-ethdev-ts/conf/tester.conf

MI  Tester  Process Info  14:16:10.359
{
  "type": "tester_pid",
  "version": 1,
  "pid": 356574
}

RING  Tester  Target Requirements  14:16:10.359
TEST_HARNESS_CHECKUP & !NO_TEST_HARNESS_CHECKUP

MI  Tester  TRC tags  14:16:10.359
TRC tags:
 dpdk-23.11.0-rc0
 dpdk:23110000
 vfio-pci
 kernel-linux
 linux-mm:504

RING  Tester  Self  14:16:10.360
Random seed is 1693923369

RING  Tester  Build  14:16:10.360
Build Test Suite 'dpdk-ethdev-ts' from '${TE_TS_SRC}'

RING  Tester  Self  14:16:13.018
Starting...

RING  Tester  Globals  14:16:13.018
Tester global variables list:
  env.peer2peer='net':IUT{'iut_host'{{'iut_rpcs':IUT,if:'iut_port'},addr:'iut_addr':inet:unicast,addr:'mcast_addr':ether:multicast,addr:'bcast_addr':ether:broadcast,addr:'iut_alien_mac':ether:alien, addr:'alien_addr':inet:alien},'tst_host'{{'tst_rpcs':tester},addr:'tst_addr':inet:unicast,if:'tst_if',addr:'tst_lladdr':ether:fake, addr:'tst_alien_mac':ether:alien}}
  env.peer2peer_ip6='net':IUT{'iut_host'{{'iut_rpcs':IUT,if:'iut_port'},addr:'iut_addr':inet6:unicast,addr:'mcast_addr':ether:multicast,addr:'bcast_addr':ether:broadcast,addr:'iut_alien_mac':ether:alien},'tst_host'{{'tst_rpcs':tester},addr:'tst_addr':inet6:unicast,if:'tst_if',addr:'tst_lladdr':ether:fake, addr:'tst_alien_mac':ether:alien}}
  tmpl.any.eth={
              pdus {
                eth:{
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2tst.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2mcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.mcast_addr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2bcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2mcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.mcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2bcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2tst.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.icmp4={
                pdus {
                    icmp4:{
                        code plain: 1,
                        type plain: 1
                    },
                    ip4:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" },
                        protocol plain: 1
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.icmp4={
                pdus {
                    icmp4:{
                        code plain: 1,
                        type plain: 1
                    },
                    ip4:{
                        src-addr env:{ name "env.addr.iut_addr" },
                        dst-addr env:{ name "env.addr.tst_addr" },
                        protocol plain: 1
                    },
                    eth:{
                        dst-addr env:{ name "env.addr.tst_lladdr" },
                        src-addr env:{ name "param.iut_mac" }
                    }
                }
            }
  tmpl.tst2iut.udp6={
                pdus {
                    udp:{
                        src-port env:{ name "env.addr.tst_addr.port" },
                        dst-port env:{ name "env.addr.iut_addr.port" }
                    },
                    ip6:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" }
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" },
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.tcp6={
                pdus {
                    tcp:{
                        src-port env:{ name "env.addr.tst_addr.port" },
                        dst-port env:{ name "env.addr.iut_addr.port" },
                        seqn script:"expr:rand()",
                        ackn plain:0
                    },
                    ip6:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" }
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vlan.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }

  tmpl.tst2iut.vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "par
  tmpl.tst2iut.vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
  tmpl.tst2iut.vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                e
  tmpl.tst2iut.vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                
  tmpl.tst2iut.vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }

  tmpl.tst2iut.vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "par
  tmpl.tst2iut.vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
  tmpl.tst2iut.vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                e
  tmpl.tst2iut.vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                
  tmpl.tst2iut.vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.vlan_vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.vlan_vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.vlan_vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
         
  tmpl.tst2iut.vlan_vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.vlan_vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.vlan_vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.vlan_vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
         
  tmpl.tst2iut.vlan_vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.vlan_vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.qinq_vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.qinq_vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
               
  tmpl.tst2iut.qinq_vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
       
  tmpl.tst2iut.qinq_vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.qinq_vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.qinq_vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.qinq_vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
               
  tmpl.tst2iut.qinq_vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
       
  tmpl.tst2iut.qinq_vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.qinq_vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" 
  tmpl.tst2iut.geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "p
  tmpl.tst2iut.geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr"
  tmpl.tst2iut.geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
               
  tmpl.tst2iut.geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.ad
  tmpl.tst2iut.geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" 
  tmpl.tst2iut.geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "p
  tmpl.tst2iut.geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr"
  tmpl.tst2iut.geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
               
  tmpl.tst2iut.geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.ad
  tmpl.tst2iut.vlan_geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.vlan_geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.vlan_geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
       
  tmpl.tst2iut.vlan_geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.vlan_geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.vlan_geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.vlan_geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.vlan_geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.vlan_geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.vlan_geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
       
  tmpl.tst2iut.vlan_geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.vlan_geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.vlan_geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.vlan_geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.qinq_geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.qinq_geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.qinq_geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
             
  tmpl.tst2iut.qinq_geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
     
  tmpl.tst2iut.qinq_geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.qinq_geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.qinq_geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.qinq_geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.qinq_geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.qinq_geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.qinq_geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
             
  tmpl.tst2iut.qinq_geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
     
  tmpl.tst2iut.qinq_geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.qinq_geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.qinq_geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.qinq_geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },

  tmpl.tst2iut.nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" 
  tmpl.tst2iut.nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
  tmpl.tst2iut.nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
        
  tmpl.tst2iut.nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },

  tmpl.tst2iut.nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" 
  tmpl.tst2iut.nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
  tmpl.tst2iut.nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
        
  tmpl.tst2iut.vlan_nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vl
  tmpl.tst2iut.vlan_nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    v
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.vlan_nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.vlan_nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vl
  tmpl.tst2iut.vlan_nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    v
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.vlan_nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.qinq_nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
               
  tmpl.tst2iut.qinq_nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq_nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
     
  tmpl.tst2iut.qinq_nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
              
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.qinq_nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.qinq_nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
               
  tmpl.tst2iut.qinq_nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq_nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
     
  tmpl.tst2iut.qinq_nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
              
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.qinq_nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.iut2tst.vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env
  tmpl.iut2tst.vxlan4.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{

  tmpl.iut2tst.vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env
  tmpl.iut2tst.vxlan6.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{

  tmpl.iut2tst.geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "e
  tmpl.iut2tst.geneve4.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:
  tmpl.iut2tst.geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "e
  tmpl.iut2tst.geneve6.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:
  tmpl.iut2tst.vxlan4.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr
  tmpl.iut2tst.vxlan4.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_ad
  tmpl.iut2tst.vxlan6.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr
  tmpl.iut2tst.vxlan6.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_ad
  tmpl.iut2tst.geneve4.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.ad
  tmpl.iut2tst.geneve4.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_
  tmpl.iut2tst.geneve6.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.ad
  tmpl.iut2tst.geneve6.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_
  tmpl.iut2tst.nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "pa
  flow_rule_pattern.ethertype={
              eth:{
                length-type plain:8111,
              }
            }
  flow_rule_pattern.ethertype.arp={
              eth:{
              },
              arp:{
              }
            }
  flow_rule_pattern.ethertype.pppoed={
              eth:{
                length-type plain:34915
              },
              pppoe:{
              }
            }
  flow_rule_pattern.ethertype.pppoes={
              eth:{
                length-type plain:34916
              },
              pppoe:{
              }
            }
  flow_rule_pattern.ethertype.ip4={
              eth:{
              },
              ip4:{
              }
            }
  flow_rule_pattern.ethertype.ip6={
              eth:{
              },
              ip6:{
              }
            }
  flow_rule_pattern.ethertype.outer_vid={
              eth:{
                length-type plain:8111,
                tagged tagged:{
                  vlan-id plain:13,
                }
              }
            }
  flow_rule_pattern.dst_mac.outer_vid={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H,
                tagged tagged:{
                  vlan-id plain:13,
                }
              }
            }
  flow_rule_pattern.ethertype.outer_vid.inner_vid={
              eth:{
                length-type plain:8111,
                tagged double-tagged:{
                  outer {
                    vid plain:13
                  },
                  inner {
                    vid plain:15
                  }
                }
              }
            }
  flow_rule_pattern.dst_mac={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              }
            }
  flow_rule_pattern.unknown_ucast_dst_mac={
              eth:{
                dst-addr range:{
                  first '00 01 02 03 04 05'H,
                  mask '01 00 00 00 00 00'H
                }
              }
            }
  flow_rule_pattern.unknown_mcast_dst_mac={
              eth:{
                dst-addr range:{
                  first '01 02 03 04 05 06'H,
                  mask '01 00 00 00 00 00'H
                }
              }
            }
  flow_rule_pattern.dst_mac.ethertype={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H,
                length-type plain:8111
              }
            }
  flow_rule_pattern.dst_mac.ethertype.ip4={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              }
            }
  flow_rule_pattern.outer_vid.ip_proto={
              eth:{
                tagged tagged:{
                  vlan-id plain:13,
                }
              },
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.ip_proto={
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.ip_proto.icmp4={
              eth:{
              },
              ip4:{
              },
              icmp4:{
              }
            }
  flow_rule_pattern.ip_proto.udp={
              eth:{
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.ip_proto.tcp={
              eth:{
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.dst_mac.ip_proto={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.dst_mac.ip_proto.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.3tuple.udp={
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.3tuple.udp6={
              ip6:{
                dst-addr plain:'20010DB885A3000000008A2E03707334'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.5tuple.udp={
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.5tuple.udp6={
              ip6:{
                src-addr plain:'20010DB885A3000000008A2E03707334'H,
                dst-addr plain:'20010DB885A3000000008A2E03707335'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.3tuple.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.dst_mac.5tuple.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.3tuple.tcp={
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              tcp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.5tuple.tcp={
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              tcp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.3tuple.tcp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              tcp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.dst_mac.5tuple.tcp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              tcp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.vxlan={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.vxlan6={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip6:{
              },
              udp:{
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.vsid.ifrm_dst_mac.vxlan={
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ip4.udp_dst.vsid.ifrm_dst_mac.vxlan={
              eth:{
              },
              ip4:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ip6.udp_dst.vsid.ifrm_dst_mac.vxlan={
              eth:{
              },
              ip6:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.vsid.vxlan={
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.ip4.udp_dst.vsid.vxlan={
              eth:{
              },
              ip4:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.ip6.udp_dst.vsid.vxlan={
              eth:{
              },
              ip6:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.geneve={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              },
              geneve:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.geneve6={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip6:{
              },
              udp:{
              },
              geneve:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.nvgre={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              gre:{
                opt-key nvgre:{
                  vsid plain:13
                }
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ethertype.4tuple.vxlan={
              eth:{
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
              }
            }
  flow_rule_actions.flag={
              {
                type flag
              }
            }
  flow_rule_actions.mark={
              {
                type mark
              }
            }
  flow_rule.rss.3tuple.tcp.default={
              attr {
                ingress 1
              },
              pattern {
                ip4:{
                  dst-addr plain:'0a 89 03 01'H,
                  src-addr script:"expr:rand()"
                },
                tcp:{
                  dst-port plain:9201,
                  src-port script:"expr:rand()"
                }
              },
              actions {
                {
                  type rss,
                  conf rss:{
                    queue {
                      1, 5, 2, 4
                    }
                  }
                }
              }
            }
  flow_rule.rss.3tuple.tcp.custom={
              attr {
                ingress 1
              },
              pattern {
                ip4:{
                  dst-addr plain:'0a 89 03 01'H,
                  src-addr script:"expr:rand()"
                },
                tcp:{
                  dst-port plain:9201,
                  src-port script:"expr:rand()"
                }
              },
              actions {
                {
                  type rss,
                  conf rss:{
                    rss-conf {
                      rss-key '6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a'H,
                      rss-hf {
                        ip 1,
                        tcp 1,
                      }
                    },
                    queue {
                      1, 5, 
  env.perf.iut_only='net':IUT{
                'iut_host'{
                    {'iut_jobs_ctrl':tester, if:'iut_port'}
                }
            }
  env.perf.peer2peer='net':IUT{
                'iut_host'{
                    {'iut_jobs_ctrl':tester, if:'iut_port'}
                },
                'tst_host'{
                    {'tst_jobs_ctrl':tester, if:'tst_port'}
                }
            }

MI  Tester  Execution Plan  14:16:13.049
{
  "type": "test_plan",
  "version": 1,
  "plan": {
    "type": "pkg",
    "name": "dpdk-ethdev-ts",
    "objective": "DPDM EthDev Test Suite",
    "authors": [
      {
        "name": null,
        "email": "Andrew.Rybchenko@oktetlabs.ru"
      }
    ],
    "prologue": {
      "type": "test",
      "name": "prologue"
    },
    "children": [
      {
        "type": "pkg",
        "name": "usecases",
        "objective": "Main use cases of the PMD",
        "authors": [
          {
            "name": null,
            "email": "Andrew.Rybchenko@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "test",
            "name": "tx_burst_simple",
            "iterations": 6
          },
          {
            "type": "test",
            "name": "rx_burst_simple",
            "iterations": 6
          },
          {
            "type": "test",
            "name": "rss",
            "iterations": 5
          },
          {
            "type": "test",
            "name": "rx_scatter",
            "iterations": 2
          },
          {
            "type": "test",
            "name": "tx_stats",
            "iterations": 3
          },
          {
            "type": "test",
            "name": "rx_offload_checksum",
            "iterations": 20
          }
        ]
      },
      {
        "type": "pkg",
        "name": "xmit",
        "objective": "Transmit Functionality",
        "authors": [
          {
            "name": null,
            "email": "Ivan.Malov@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "session",
            "name": "one_packet_all",
            "children": [
              {
                "type": "test",
                "name": "one_packet_tunnel",
                "iterations": 12
              },
              {
                "type": "session",
                "name": "one_packet_ip4",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4",
                    "children": [
                      {
                        "type": "test",
                        "name": "tso_packet_ip4_seg"
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip4"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip4",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4",
                    "children": [
                      {
                        "type": "test",
                        "name": "tso_packet_ip4_seg"
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip4"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip6",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip6"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip6",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip6"
                  }
                ]
              }
            ]
          },
          {
            "type": "session",
            "name": "one_packet_with_dpdk_rx_all",
            "prologue": {
              "type": "test",
              "name": "one_packet_with_dpdk_rx_prologue"
            },
            "children": [
              {
                "type": "session",
                "name": "session",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_with_dpdk_rx_cksum_offloads",
                    "children": [
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_plain",
                        "children": [
                          {
                            "type": "test",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_plain_ip4",
                            "iterations": 8
                          }
                        ]
                      },
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_encap",
                        "children": [
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip6"
                          }
                        ]
                      },
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_encap",
                        "children": [
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip6"
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_with_dpdk_rx_corner_cases",
                    "children": [
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_outgoing_frames"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_outgoing_frames"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_header_segments"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_header_segments"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_many_payload_segments"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_long_payload"
                      },
                      {
                        "type": "session",
                        "name": "tso_packet_with_dpdk_rx_too_long_payload"
                      }
                    ]
                  }
                ]
              }
            ],
            "epilogue": {
              "type": "test",
              "name": "one_packet_with_dpdk_rx_epilogue"
            }
          }
        ]
      },
      {
        "type": "pkg",
        "name": "filters",
        "objective": "Filters",
        "authors": [
          {
            "name": null,
            "email": "Roman.Zhukov@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "test",
            "name": "flow_rule_in2q",
            "iterations": 6
          },
          {
            "type": "session",
            "name": "flow_rule_drop_and_count"
          },
          {
            "type": "session",
            "name": "flow_rule_mark_and_flag"
          }
        ]
      },
      {
        "type": "pkg",
        "name": "representors",
        "objective": "Representors",
        "authors": [
          {
            "name": null,
            "email": "Igor.Romanov@oktetlabs.ru"
          }
        ],
        "prologue": {
          "type": "test",
          "name": "rep_prologue"
        }
      },
      {
        "type": "pkg",
        "name": "perf",
        "objective": "Performance",
        "authors": [
          {
            "name": null,
            "email": "Igor.Romanov@oktetlabs.ru"
          }
        ],
        "prologue": {
          "type": "test",
          "name": "perf_prologue"
        },
        "children": [
          {
            "type": "test",
            "name": "l2fwd_simple"
          }
        ]
      }
    ]
  }
}

RING  Tester  Run  14:16:13.049
Running configuration:
File:        /opt/tsf/dpdk-ethdev-ts/conf/tester.conf
Maintainers: mailto:Andrew.Rybchenko@oktetlabs.ru
Description: DPDK EthDev Testing

MI  Tester  Control  14:16:13.050
PACKAGE "dpdk-ethdev-ts" started
Node ID 1, Parent ID 0, Plan ID 0
Authors:
 *   <Andrew.Rybchenko@oktetlabs.ru>
Objective: DPDM EthDev Test Suite

MI  Tester  Control  14:16:13.050
TEST "prologue" started
Node ID 2, Parent ID 1, Plan ID 1
Parameters:
 *  env = VAR.env.peer2peer

RING  prologue  Step  14:16:13.057
Test start

INFO  prologue  TAPI Jumps  14:16:13.058
Set jump point ../../prologue.c:721

RING  prologue  Self  14:16:13.058
Pseudo-random seed is 1870362436

INFO  prologue  TAPI Jumps  14:16:13.058
Remove jump point ../../prologue.c:721 at ../../prologue.c:721

INFO  prologue  TAPI Jumps  14:16:13.058
Set jump point ../../prologue.c:721

RING  prologue  Configurator API  14:16:13.061
Set /agent:DPDK/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923340_2:/usr/local/sbin

RING  prologue  Configurator API  14:16:13.065
Set /agent:DPDK/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923340_2:/usr/local/sbin:/usr/sbin

RING  prologue  Configurator API  14:16:13.067
Set /agent:DPDK/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923340_2:/usr/local/sbin:/usr/sbin:/sbin

RING  prologue  Configurator API  14:16:13.070
Set /agent:Peer/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923337_1:/usr/local/sbin

RING  prologue  Configurator API  14:16:13.073
Set /agent:Peer/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923337_1:/usr/local/sbin:/usr/sbin

RING  prologue  Configurator API  14:16:13.076
Set /agent:Peer/env:PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923337_1:/usr/local/sbin:/usr/sbin:/sbin

RING  prologue  Configurator API  14:16:13.101
Added /agent:DPDK/rsrc:module:vfio-pci = 

RING  prologue  Configurator API  14:16:13.102
Set /agent:DPDK/rsrc:module:vfio-pci/fallback_shared: = 1

RING  prologue  Configurator API  14:16:13.103
Set /agent:DPDK/rsrc:module:vfio-pci/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  14:16:13.105
Set /agent:DPDK/rsrc:module:vfio-pci/shared: = 0

RING  prologue  Configurator API  14:16:13.126
Set /agent:DPDK/rsrc:module:vfio-pci = /agent:DPDK/module:vfio-pci

RING  @DPDK  Unix Conf System Module  14:16:13.143
Module 'vfio-pci' already loaded

RING  prologue  Configurator API  14:16:13.150
Added /agent:DPDK/module:vfio-pci = (none)

RING  prologue  Configurator API  14:16:13.156
Set /agent:DPDK/module:vfio-pci/loaded: = 1

RING  prologue  Configurator API  14:16:13.159
Set /agent:DPDK/rsrc:module:vfio-pci/shared: = 1

RING  prologue  Configurator API  14:16:13.185
Added /agent:Peer/rsrc:module:vfio-pci = 

RING  prologue  Configurator API  14:16:13.186
Set /agent:Peer/rsrc:module:vfio-pci/fallback_shared: = 1

RING  prologue  Configurator API  14:16:13.186
Set /agent:Peer/rsrc:module:vfio-pci/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  14:16:13.189
Set /agent:Peer/rsrc:module:vfio-pci/shared: = 0

RING  prologue  Configurator API  14:16:13.212
Set /agent:Peer/rsrc:module:vfio-pci = /agent:Peer/module:vfio-pci

RING  @Peer  Unix Conf System Module  14:16:13.236
Module 'vfio-pci' already loaded

RING  prologue  Configurator API  14:16:13.237
Added /agent:Peer/module:vfio-pci = (none)

RING  prologue  Configurator API  14:16:13.245
Set /agent:Peer/module:vfio-pci/loaded: = 1

RING  prologue  Configurator API  14:16:13.248
Set /agent:Peer/rsrc:module:vfio-pci/shared: = 1

RING  prologue  Configurator API  14:16:13.276
Added /agent:DPDK/rsrc:module:vfio = 

RING  prologue  Configurator API  14:16:13.277
Set /agent:DPDK/rsrc:module:vfio/fallback_shared: = 1

RING  prologue  Configurator API  14:16:13.278
Set /agent:DPDK/rsrc:module:vfio/acquire_attempts_timeout: = 3000

RING  prologue  Configurator API  14:16:13.280
Set /agent:DPDK/rsrc:module:vfio/shared: = 0

RING  prologue  Configurator API  14:16:13.306
Set /agent:DPDK/rsrc:module:vfio = /agent:DPDK/module:vfio

RING  @DPDK  Unix Conf System Module  14:16:13.327
Module 'vfio' already loaded

RING  prologue  Configurator API  14:16:13.334
Added /agent:DPDK/module:vfio = (none)

RING  prologue  Configurator API  14:16:13.341
Set /agent:DPDK/module:vfio/loaded: = 1

RING  prologue  Configurator API  14:16:13.344
Set /agent:DPDK/rsrc:module:vfio/shared: = 1

ERROR  prologue  Environment LIB  14:16:13.650
Too few networks in available configuration (0) in comparison with required (1)

ERROR  prologue  Self  14:16:13.650
Test Failed in ../../prologue.c, line 807, main()

ERROR  prologue  Self  14:16:13.650
tapi_env_get() failed: 'net':IUT{'iut_host'{{'iut_rpcs':IUT,if:'iut_port'},addr:'iut_addr':inet:unicast,addr:'mcast_addr':ether:multicast,addr:'bcast_addr':ether:broadcast,addr:'iut_alien_mac':ether:alien, addr:'alien_addr':inet:alien},'tst_host'{{'tst_rpcs':tester},addr:'tst_addr':inet:unicast,if:'tst_if',addr:'tst_lladdr':ether:fake, addr:'tst_alien_mac':ether:alien}} : EENV

INFO  prologue  TAPI Jumps  14:16:13.650
Jump from ../../prologue.c:807 to ../../prologue.c:721 rc=EFAIL

RING  prologue  Step  14:16:13.650
Test cleanup

RING  Tester  Run  14:16:13.651

Obtained result is:
FAILED

Expected results are: default
PASSED

MI  Tester  Control  14:16:13.651
TEST "prologue" finished
Node ID 2, Parent ID 1, Plan ID 1

Obtained result:
Status: FAILED

ERROR: Unexpected test result

Expected results:

Status: PASSED

RING  @DPDK  Unix Conf System Module  14:16:14.206
Module 'vfio' already loaded

RING  Configurator  RCF API  14:16:14.213
Set /agent:DPDK/module:vfio/loaded: to 1: OK

RING  Configurator  RCF API  14:16:14.219
Delete /agent:DPDK/module:vfio: OK

RING  Configurator  RCF API  14:16:14.221
Set /agent:DPDK/rsrc:module:vfio to : OK

RING  Configurator  RCF API  14:16:14.245
Set /agent:DPDK/rsrc:module:vfio/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  14:16:14.245
Set /agent:DPDK/rsrc:module:vfio/fallback_shared: to 0: OK

RING  Configurator  RCF API  14:16:14.246
Delete /agent:DPDK/rsrc:module:vfio: OK

RING  @Peer  Unix Conf System Module  14:16:14.263
Module 'vfio-pci' already loaded

RING  Configurator  RCF API  14:16:14.265
Set /agent:Peer/module:vfio-pci/loaded: to 1: OK

RING  Configurator  RCF API  14:16:14.272
Delete /agent:Peer/module:vfio-pci: OK

RING  Configurator  RCF API  14:16:14.273
Set /agent:Peer/rsrc:module:vfio-pci to : OK

RING  Configurator  RCF API  14:16:14.295
Set /agent:Peer/rsrc:module:vfio-pci/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  14:16:14.295
Set /agent:Peer/rsrc:module:vfio-pci/fallback_shared: to 0: OK

RING  Configurator  RCF API  14:16:14.296
Delete /agent:Peer/rsrc:module:vfio-pci: OK

RING  @DPDK  Unix Conf System Module  14:16:14.304
Module 'vfio-pci' already loaded

RING  Configurator  RCF API  14:16:14.311
Set /agent:DPDK/module:vfio-pci/loaded: to 1: OK

RING  Configurator  RCF API  14:16:14.317
Delete /agent:DPDK/module:vfio-pci: OK

RING  Configurator  RCF API  14:16:14.319
Set /agent:DPDK/rsrc:module:vfio-pci to : OK

RING  Configurator  RCF API  14:16:14.340
Set /agent:DPDK/rsrc:module:vfio-pci/acquire_attempts_timeout: to 0: OK

RING  Configurator  RCF API  14:16:14.340
Set /agent:DPDK/rsrc:module:vfio-pci/fallback_shared: to 0: OK

RING  Configurator  RCF API  14:16:14.341
Delete /agent:DPDK/rsrc:module:vfio-pci: OK

RING  Configurator  RCF API  14:16:14.357
Set /agent:Peer/env:PATH to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923337_1: OK

RING  Configurator  RCF API  14:16:14.359
Set /agent:DPDK/env:PATH to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/tmp/linux_x86_root_356502_1693923340_2: OK

MI  Tester  Control  14:16:14.605
PACKAGE "dpdk-ethdev-ts" finished
Node ID 1, Parent ID 0, Plan ID 0

Obtained result:
Status: FAILED
Verdicts:
 *  Session prologue failed

ERROR: Session prologue failed

Expected results:

Status: PASSED

RING  Tester  Self  14:16:14.652
Done

RING  Dispatcher  Start  14:16:14.791
Shutdown Configurator

RING  Configurator  RCF API  14:16:14.794
Set /agent:Peer/sys:/core_pattern: to |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E: OK

RING  Configurator  RCF API  14:16:14.795
Set /agent:DPDK/sys:/core_pattern: to |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E: OK

RING  Configurator  RCF API  14:16:14.796
Set /agent:Peer/rpcprovider: to : OK

RING  Configurator  RCF API  14:16:14.796
Set /agent:DPDK/rpcprovider: to : OK

RING  Configurator  Self  14:16:14.864
Exit

RING  Dispatcher  Start  14:16:14.878
Flush log

RING  Dispatcher  Start  14:16:14.897
Shutdown RCF

RING  RCF  Self  14:16:14.898
Shutting down

RING  RCF  RCF Unix  14:16:15.000
Finish method is called for TA Peer

RING  RCF  RCF Unix  14:16:15.000
CMD to remove: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "rm -rf /tmp/linux_x86_root_356502_1693923337_1"

WARN  RCF  DPDK  14:16:15.004
Exiting: 0

RING  RCF  RCF Unix  14:16:15.267
Finish method is called for TA DPDK

RING  RCF  RCF Unix  14:16:15.267
CMD to remove: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "rm -rf /tmp/linux_x86_root_356502_1693923340_2"

RING  RCF  Self  14:16:19.064
Test Agents are stopped

RING  RCF  Self  14:16:19.065
Exit

ERROR  Logger  Self  14:16:19.065
rcf_ta_get_log(ta_name='DPDK') returned fatal error RCF_API-EIPC, stop gathering logs from this TA

RING  Logger  Self  14:16:19.065
IPC Server 'TE_LOGGER_355605-ta-DPDK' closed

ERROR  Logger  Self  14:16:19.065
rcf_ta_get_log(ta_name='Peer') returned fatal error RCF_API-EIPC, stop gathering logs from this TA

RING  Logger  Self  14:16:19.065
IPC Server 'TE_LOGGER_355605-ta-Peer' closed

RING  Dispatcher  Start  14:16:19.081
Shutdown Logger

RING  Logger  Self  14:16:19.082
Logger shutdown ...

WARN  Logger  Log streaming  14:16:19.082
Not all messages in listener queue have been processed

RING  Logger  Self  14:16:19.082
Shutdown is completed

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 60635 bytes --]

Hi Adam,

On 9/5/23 18:01, Adam Hassick wrote:
> Hi Andrew,
>
> The compilation warning issue is now resolved. Again, thank you guys 
> for fixing this for us. I can run the tests on the Mellanox CX5s 
> again, however I'm running into a couple new issues with running the 
> prologues on the Intel cards.
>
> When running testing on the Intel XL710s, I see this error appear in 
> the log:
>
>     ERROR  prologue  Environment LIB  14:16:13.650
>     Too few networks in available configuration (0) in comparison with
>     required (1)
>
>
> This seems like a trivial configuration error, perhaps this is 
> something I need to set up in ts-rigs. I briefly searched through the 
> examples there and didn't see any mention of how to set up a network.
> I will attach this log just in case you need more information.

Unfortunately logs are insufficient to understand it. I've pushed new 
tag to TE v1.19.0 which add log message with TE_* environment variables.
Most likely something is wrong with variables which are used as 
conditions when available networks are defined in 
ts-conf/cs/inc.net_cfg_pci_fns.yml:
TE_PCI_INSTANCE_IUT_TST1
TE_PCI_INSTANCE_IUT_TST1a
TE_PCI_INSTANCE_TST1a_IUT
TE_PCI_INSTANCE_TST1_IUT
My guess it that you change naming a bit, but script like 
ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.

> There is a different error when running on the Intel E810s. It appears 
> to me like it starts DPDK, does some configuration inside DPDK and on 
> the device, and then fails to bring the device back up. Since this 
> error seems very non-trivial, I will also attach this log.

This one is a bit simpler. Few lines after the first ERROR in log I see 
the following:
WARN  RCF  DPDK  13:06:00.144
ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)
ice_rx_queue_start(): fail to program RX queue 0
ice_dev_start(): fail to start Rx queue 0
Device with port_id=0 already stopped

It is stdout/stderr from test agent which runs DPDK. Same logs in plain 
format are available in ta.DPDK file.
I'm not an expert here, but I vaguely remember that E810 requires 
correct firmware and DDP to be loaded.
There is some information in dpdk/doc/guides/nics/ice.rst.

You can try to add --dev-args=safe-mode-support=1 command-line option 
described there.

Hope it helps,
Andrew.

>
> Thanks,
> Adam
>
> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     On 8/31/23 22:38, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     I have one additional question as well: Does the test engine
>>     support running tests on two ARMv8 test agents?
>>
>>         1. We'll sort out warnings this week. Thanks for heads up.
>>
>>
>>     Great. Let me know when that's fixed.
>
>     Done. We also fixed a number of warnings in TE.
>     Also we fixed root test package name to be consistent with the
>     repository name.
>
>>         Support for old LTS branches was dropped some time ago, but
>>         in the future it is definitely possible to keep it for new
>>         LTS branches. I think 22.11 is supported, but I'm not sure
>>         about older LTS releases.
>>
>>
>>     Good to know.
>>
>>         2. You can add command-line option --sanity to run tests
>>         marked with TEST_HARNESS_SANITY requirement (see
>>         dpdk-ethdev-ts/scripts/run.sh and grep TEST_HARNESS_SANITY
>>         dpdk-ethdev-ts to see which tests are marked). Yes, there is
>>         a space for terminology improvement here. We'll do it.
>>
>
>     Done. Now it is called --checkup.
>
>>
>>         Also it takes a lot of time because of failures and tests
>>         which wait for some timeout.
>>
>>
>>     That makes sense to me. We'll use the time to complete tests on
>>     virtio or the Intel devices as a reference for how long the tests
>>     really take to complete.
>>     We will explore the possibility of periodically running the
>>     sanity tests for patches.
>
>     I'll double-check and let you know how long entire TS runs on
>     Intel X710, E810, Mellanox CX5 and virtio net. Just to ensure that
>     time observed in your case looks the same.
>
>>
>>         The test harness can provide coverage reports based on gcov,
>>         but I'm not sure what you mean by a "dial" to control test
>>         coverage. Provided reports are rather for human to analyze.
>>
>>
>>     The general idea is to have some kind of parameter on the test
>>     suite, which could be an integer ranging from zero to ten, that
>>     controls how many tests are run based on how important the test is.
>>
>>     Similar to how some command line interfaces provide a verbosity
>>     level parameter (some number of "-v" arguments) to control the
>>     importance of the information in the log.
>>     The verbosity level zero only prints very important log messages,
>>     while ten prints everything.
>>
>>     In much the same manner as above, this "dial" parameter controls
>>     what tests are run and with what parameters based on how
>>     important those tests and test parameter combinations are.
>>     Coverage Level zero tells the suite to run a very basic set of
>>     important tests, with minimal parameterization. This mode would
>>     take only ~5-10 minutes to run.
>>     In contrast, Coverage Level ten includes all the edge cases,
>>     every combination of test parameters, everything the test suite
>>     can do, which takes the normal several hours to run.
>>     The values 1 - 9 are between those two extremes, allowing the
>>     user to get a gradient of test coverage in the results and to
>>     limit the running time.
>>
>>     Then we could, for example, run the "run.sh" with a level of 2 or
>>     3 for incoming patches that need quick results, and with a level
>>     of 10 for the less often run periodic tests performed on main or
>>     LTS branches.
>
>     Understood now. Thanks a lot for the idea. We'll discuss it and
>     come back.
>
>>         3. Yes, really many tests on Mellanox CX5 NICs report
>>         unexpected testing results. Unfortunately it is time
>>         consuming to fill in expectations database since it is
>>         necessary to analyze testing results and classify if it is a
>>         bug or just acceptable behaviour aspect.
>>
>>         Bublik allows to compare results of two runs. It is useful
>>         for human, but still not good for automation.
>>
>>         I have local patch for mlx5 driver which reports Tx ring size
>>         maximum. It makes pass rate higher. It is a problem for test
>>         harness that mlx5 does not report limits right now.
>>
>>         Pass rate on Intel X710 is about 92% on my test rig. Pass
>>         rate on virtio net is 99% right now and could be done 100%
>>         easily (just one thing to fix in expectations).
>>
>>         I think logs storage setup is essential for logs analysis. Of
>>         course, you can request HTML logs when you run tests
>>         (--log-html=html) or generate after run using
>>         dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>>         browser, but logs storage makes it more convenient.
>>
>>
>>     We are interested in setting up Bublik, potentially as an
>>     externally-facing component, once we have our process of running
>>     the test suite stabilized.
>>     Once we are able to run the test suite again, I'll see what the
>>     pass rate is on our other hardware.
>>     Good to know that it isn't an issue with our dev testbed causing
>>     the high fail rate.
>>
>>     For Intel hardware, we have an XL710 and an Intel E810-C in our
>>     development testbed. Although they are slightly different
>>     devices, ideally the pass rate will be identical or similar. I
>>     have yet to set up a VM pair for virtio, but we will soon.
>>
>>         Latest version of test-environment has examples of our CGI
>>         scripts which we use for log storage (see
>>         tools/log_server/README.md).
>>
>>         Also all bits for Jenkins setup are available. See
>>         dpdk-ethdev-ts/jenkins/README.md and examples of jenkins
>>         files in ts-rigs-sample.
>>
>>
>>     Jenkins integration, setting up production rig configurations,
>>     and permanent log storage will be our next steps once I am able
>>     to run the tests again.
>>     Unless there is an easy way to have meson not pass "-Werror" into
>>     GCC. Then I would be able to run the test suite.
>
>     Hopefully it is resolved now.
>
>     I thought a bit more about your usecase for Jenkins. I'm not 100%
>     sure that existing pipelines are convenient for your usecase.
>     Fill free to ask questions when you are on it.
>
>     Thanks,
>     Andrew.
>
>>
>>     Thanks,
>>     Adam
>>
>>
>>         On 8/29/23 17:02, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         That fix seems to have resolved the issue, thanks for the
>>>         quick turnaround time on that patch.
>>>         Now that we have the RCF timeout issue resolved, there are a
>>>         few other questions and issues that we have about the tests
>>>         themselves.
>>>
>>>         1. The test suite fails to build with a couple warnings.
>>>
>>>         Below is the stderr log from compilation:
>>>
>>>             FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>             cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>             -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>             -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64
>>>             -Wall -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb
>>>             -Wall -W -fPIC -MD -MQ
>>>             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF
>>>             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>>>             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>             ../../lib/dpdk_pmd_ts.c
>>>             ../../lib/dpdk_pmd_ts.c: In function
>>>             ‘test_create_traffic_generator_params’:
>>>             ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a
>>>             string literal and no format arguments
>>>             [-Werror=format-security]
>>>             5577 |     rc = te_kvpair_add(result, buf, mode);
>>>             |     ^~
>>>             cc1: all warnings being treated as errors
>>>             ninja: build stopped: subcommand failed.
>>>             ninja: Entering directory `.'
>>>             FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>             cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>             -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>             -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64
>>>             -Wall -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb
>>>             -Wall -W -fPIC -MD -MQ
>>>             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF
>>>             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>>>             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>             ../../lib/dpdk_pmd_ts.c
>>>             ../../lib/dpdk_pmd_ts.c: In function
>>>             ‘test_create_traffic_generator_params’:
>>>             ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a
>>>             string literal and no format arguments
>>>             [-Werror=format-security]
>>>             5577 |     rc = te_kvpair_add(result, buf, mode);
>>>             |     ^~
>>>             cc1: all warnings being treated as errors
>>>
>>>
>>>         This error wasn't occurring last week, which was the last
>>>         time I ran the tests.
>>>         The TE host and the DUT have GCC v9.4.0 installed, and the
>>>         tester has GCC v11.4.0 installed, if this information is
>>>         helpful.
>>>
>>>         2. On the Mellanox CX5s, there are over 6,000 tests run,
>>>         which collectively take around 9 hours. Is it possible, and
>>>         would it make sense, to lower the test coverage and have the
>>>         test suite run faster?
>>>
>>>         For some context, we run immediate testing on incoming
>>>         patches for DPDK main and development branches, as well as
>>>         periodic test runs on the main, stable, and LTS branches.
>>>         For us to consider including this test suite as part of our
>>>         immediate testing on patches, we would have to reduce the
>>>         test coverage to the most important tests.
>>>         This is primarily to reduce the testing time to, for
>>>         example, less than 30 minutes. Testing on patches can't take
>>>         too long because the lab can receive numerous patches each
>>>         day, which each require individual testing runs.
>>>
>>>         At what frequency we run these tests, and on what, still
>>>         needs to be discussed with the DPDK community, but it would
>>>         be nice to know if the test suite had a "dial" to control
>>>         the testing coverage.
>>>
>>>         3. We see a lot of test failures on our Mellanox CX5 NICs.
>>>         Around 2,300 of ~6,600 tests passed. Is there anything we
>>>         can do to diagnose these test failures?
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>
>>>         On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             Hi Adam,
>>>
>>>             I've pushed the fix in main branch and a new tag
>>>             v1.18.1. It should solve the problem with IPv6 address
>>>             from DNS.
>>>
>>>             Andrew.
>>>
>>>             On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>             Hi Adam,
>>>>
>>>>             > Does the test engine prefer to use IPv6 over IPv4 for
>>>>             initiating the RCF connection to the test bed hosts?
>>>>             And if so, is there a way to force it to use IPv4?
>>>>
>>>>             Brilliant idea. If DNS returns both IPv4 and IPv6
>>>>             addresses in your case, I guess it is the root cause of
>>>>             the problem.
>>>>             Of course, it is TE problem since I see really weird
>>>>             code in lib/comm_net_engine/comm_net_engine.c line 135.
>>>>
>>>>             I've pushed fix to the branch
>>>>             user/arybchik/fix_ipv4_only in
>>>>             ts-factory/test-environment repository. Please, try.
>>>>
>>>>             It is late night fix with minimal testing and no
>>>>             review. I'll pass it through review process tomorrow and
>>>>             hopefully it will be released in one-two days.
>>>>
>>>>             Andrew.
>>>>
>>>>             On 8/28/23 18:02, Adam Hassick wrote:
>>>>>             Hi Andrew,
>>>>>
>>>>>             We have yet to notice a distinct pattern with the
>>>>>             failures. Sometimes, the RCF will start and connect
>>>>>             without issue a few times in a row before failing to
>>>>>             connect again. Once the issue begins to occur, neither
>>>>>             rebooting all of the hosts (test engine VM, tester,
>>>>>             IUT) or deleting all of the build directories (suites,
>>>>>             agents, inst) and rebooting the hosts afterward
>>>>>             resolves the issue. When it begins working again seems
>>>>>             very arbitrary to us.
>>>>>
>>>>>             I do usually try to terminate the test engine with
>>>>>             Ctrl+C, but when it hangs while trying to start RCF,
>>>>>             that does not work.
>>>>>
>>>>>             Does the test engine prefer to use IPv6 over IPv4 for
>>>>>             initiating the RCF connection to the test bed hosts?
>>>>>             And if so, is there a way to force it to use IPv4?
>>>>>
>>>>>              - Adam
>>>>>
>>>>>             On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko
>>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>                 > I'll double-check test engine on Ubuntu 20.04
>>>>>                 and Ubuntu 22.04.
>>>>>
>>>>>                 Done. It works fine for me without any issues.
>>>>>
>>>>>                 Have you noticed any pattern when it works or does
>>>>>                 not work?
>>>>>                 May be it is a problem of not clean state after
>>>>>                 termination?
>>>>>                 Does it work fine the first time after DUTs reboot?
>>>>>                 How do you terminate testing? It should be done
>>>>>                 using Ctrl+C in terminal where you execute run.sh
>>>>>                 command.
>>>>>                  In this case it should shutdown gracefully and
>>>>>                 close all test agents and engine applications.
>>>>>
>>>>>                 (I'm trying to understand why you've seen many
>>>>>                 test agent processes. It should not happen.)
>>>>>
>>>>>                 Andrew.
>>>>>
>>>>>                 On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>                 On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>                 Hi Andrew,
>>>>>>>
>>>>>>>                 Two of our systems (the Test Engine runner and
>>>>>>>                 the DUT host) are running Ubuntu 20.04 LTS,
>>>>>>>                 however this morning I noticed that the tester
>>>>>>>                 system (the one having issues) is running Ubuntu
>>>>>>>                 22.04 LTS.
>>>>>>>                 This could be the source of the problem. I
>>>>>>>                 encountered a dependency issue trying to run the
>>>>>>>                 Test Engine on 22.04 LTS, so I downgraded the
>>>>>>>                 system. Since the tester is also the host having
>>>>>>>                 connection issues, I will try downgrading that
>>>>>>>                 system to 20.04, and see if that changes anything.
>>>>>>
>>>>>>                 Unlikely, but who knows. We run tests (DUTs) on
>>>>>>                 Ubuntu 20.04, Ubuntu 22.04, Ubuntu 22.10, Ubuntu
>>>>>>                 23.04, Debian 11 and Fedora 38 every night.
>>>>>>                 Right now Debian 11 is used for test engine in
>>>>>>                 nightly regressions.
>>>>>>
>>>>>>                 I'll double-check test engine on Ubuntu 20.04 and
>>>>>>                 Ubuntu 22.04.
>>>>>>
>>>>>>>                 I did try passing in the "--vg-rcf" argument to
>>>>>>>                 the run.sh script of the test suite after
>>>>>>>                 installing valgrind, but there was no additional
>>>>>>>                 output that I saw.
>>>>>>
>>>>>>                 Sorry, I should valgrind output should be in
>>>>>>                 valgrind.te_rcf (direction where you run test
>>>>>>                 engine).
>>>>>>
>>>>>>>
>>>>>>>                 I will try pulling in the changes you've pushed
>>>>>>>                 up, and will see if that fixes anything.
>>>>>>>
>>>>>>>                 Thanks,
>>>>>>>                 Adam
>>>>>>>
>>>>>>>                 On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>                     Hello Adam,
>>>>>>>
>>>>>>>                     On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>>                     I'd like to try to repeat the problem
>>>>>>>>                     locally. Which Linux distro is running on
>>>>>>>>                     test engine and agents?
>>>>>>>>
>>>>>>>>                     In fact I know one problem with Debian 12
>>>>>>>>                     and Fedora 38 and we have
>>>>>>>>                     patch in review to fix it, however, the
>>>>>>>>                     behaviour is different in
>>>>>>>>                     this case, so it is unlike the same problem.
>>>>>>>
>>>>>>>                     I've just published a new tag which fixes
>>>>>>>                     known test engine side problems on Debian 12
>>>>>>>                     and Fedora 38.
>>>>>>>
>>>>>>>>
>>>>>>>>                     One more idea is to install valgrind on the
>>>>>>>>                     test engine host and
>>>>>>>>                     run with option --vg-rcf to check if
>>>>>>>>                     something weird is happening.
>>>>>>>>
>>>>>>>>                     What I don't understand right now is why I
>>>>>>>>                     see just one failed attempt
>>>>>>>>                     to connect in your log.txt and then Logger
>>>>>>>>                     shutdown after 9 minutes.
>>>>>>>>
>>>>>>>>                     Andrew.
>>>>>>>>
>>>>>>>>                     On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>>                      > Is there any firewall in the network or
>>>>>>>>>                     on test hosts which could block incoming
>>>>>>>>>                     TCP connection to the port 23571
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     from the host where you run test engine?
>>>>>>>>>
>>>>>>>>>                     Our test engine host and the testbed are
>>>>>>>>>                     on the same subnet. The connection does
>>>>>>>>>                     work sometimes.
>>>>>>>>>
>>>>>>>>>                      > If behaviour the same on the next try
>>>>>>>>>                     and you see that test agent is kept
>>>>>>>>>                     running, could you check using
>>>>>>>>>                      >
>>>>>>>>>                      > # netstat -tnlp
>>>>>>>>>                      >
>>>>>>>>>                      > that Test Agent is listening on the
>>>>>>>>>                     port and try to establish TCP connection
>>>>>>>>>                     from test agent using
>>>>>>>>>                      >
>>>>>>>>>                      > $ telnet
>>>>>>>>>                     iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     23571
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>
>>>>>>>>>                      >
>>>>>>>>>                      > and check if TCP connection could be
>>>>>>>>>                     established.
>>>>>>>>>
>>>>>>>>>                     I was able to replicate the same behavior
>>>>>>>>>                     again, where it hangs while RCF is trying
>>>>>>>>>                     to start.
>>>>>>>>>                     Running this command, I see this in the
>>>>>>>>>                     output:
>>>>>>>>>
>>>>>>>>>                     tcp        0  0 0.0.0.0:23571
>>>>>>>>>                     <http://0.0.0.0:23571>
>>>>>>>>>                     <http://0.0.0.0:23571>
>>>>>>>>>                     <http://0.0.0.0:23571> 0.0.0.0:*        
>>>>>>>>>                     LISTEN    18599/ta
>>>>>>>>>
>>>>>>>>>                     So it seems like it is listening on the
>>>>>>>>>                     correct port.
>>>>>>>>>                     Additionally, I was able to connect to the
>>>>>>>>>                     Tester machine from our Test Engine host
>>>>>>>>>                     using telnet. It printed the PID of the
>>>>>>>>>                     process once the connection was opened.
>>>>>>>>>
>>>>>>>>>                     I tried running the "ta" application
>>>>>>>>>                     manually on the command line, and it
>>>>>>>>>                     didn't print anything at all.
>>>>>>>>>                     Maybe the issue is something on the Test
>>>>>>>>>                     Engine side.
>>>>>>>>>
>>>>>>>>>                     On Thu, Aug 24, 2023 at 2:35 PM Andrew
>>>>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>                     wrote:
>>>>>>>>>
>>>>>>>>>                         Hi Adam,
>>>>>>>>>
>>>>>>>>>                          > On the tester host (which appears
>>>>>>>>>                     to be the Peer agent), there
>>>>>>>>>                         are four processes that I see running,
>>>>>>>>>                     which look like the test
>>>>>>>>>                         agent processes.
>>>>>>>>>
>>>>>>>>>                         Before the next try I'd recommend to
>>>>>>>>>                     kill these processes.
>>>>>>>>>
>>>>>>>>>                         Is there any firewall in the network
>>>>>>>>>                     or on test hosts which could
>>>>>>>>>                         block incoming TCP connection to the
>>>>>>>>>                     port 23571
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     from the host
>>>>>>>>>                         where you run test engine?
>>>>>>>>>
>>>>>>>>>                         If behaviour the same on the next try
>>>>>>>>>                     and you see that test agent is
>>>>>>>>>                         kept running, could you check using
>>>>>>>>>
>>>>>>>>>                         # netstat -tnlp
>>>>>>>>>
>>>>>>>>>                         that Test Agent is listening on the
>>>>>>>>>                     port and try to establish TCP
>>>>>>>>>                         connection from test agent using
>>>>>>>>>
>>>>>>>>>                         $ telnet
>>>>>>>>>                     iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     23571
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         and check if TCP connection could be
>>>>>>>>>                     established.
>>>>>>>>>
>>>>>>>>>                         Another idea is to login Tester under
>>>>>>>>>                     root as testing does, get
>>>>>>>>>                         start TA command from the log and try
>>>>>>>>>                     it by hands without -n and
>>>>>>>>>                         remove extra escaping.
>>>>>>>>>
>>>>>>>>>                         # sudo
>>>>>>>>>                     PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>
>>>>>>>>>                     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>                     /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>                     Peer 23571
>>>>>>>>>                     host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>
>>>>>>>>>                         Hopefully in this case test agent
>>>>>>>>>                     directory remains in the /tmp and
>>>>>>>>>                         you don't need to copy it as testing
>>>>>>>>>                     does.
>>>>>>>>>                         May be output could shed some light on
>>>>>>>>>                     what's going on.
>>>>>>>>>
>>>>>>>>>                         Andrew.
>>>>>>>>>
>>>>>>>>>                         On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>>                     Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                         This is the output that I see in the
>>>>>>>>>>                     terminal when this failure
>>>>>>>>>>                         occurs, after the test agent binaries
>>>>>>>>>>                     build and the test engine
>>>>>>>>>>                         starts:
>>>>>>>>>>
>>>>>>>>>>                         Platform default build - pass
>>>>>>>>>>                         Simple RCF consistency check succeeded
>>>>>>>>>>                     --->>> Starting Logger...done
>>>>>>>>>>                     --->>> Starting
>>>>>>>>>>                     RCF...rcf_net_engine_connect():
>>>>>>>>>>                     Connection timed
>>>>>>>>>>                         out
>>>>>>>>>>                     iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>
>>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         Then, it hangs here until I kill the
>>>>>>>>>>                     "te_rcf" and "te_tee"
>>>>>>>>>>                         processes. I let it hang for around 9
>>>>>>>>>>                     minutes.
>>>>>>>>>>
>>>>>>>>>>                         On the tester host (which appears to
>>>>>>>>>>                     be the Peer agent), there are
>>>>>>>>>>                         four processes that I see running,
>>>>>>>>>>                     which look like the test agent
>>>>>>>>>>                         processes.
>>>>>>>>>>
>>>>>>>>>>                         ta.Peer is an empty file. I've
>>>>>>>>>>                     attached the log.txt from this run.
>>>>>>>>>>
>>>>>>>>>>                          - Adam
>>>>>>>>>>
>>>>>>>>>>                         On Thu, Aug 24, 2023 at 4:22 AM
>>>>>>>>>>                     Andrew Rybchenko
>>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>                     wrote:
>>>>>>>>>>
>>>>>>>>>>                             Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                             Yes, TE_RCFUNIX_TIMEOUT is in
>>>>>>>>>>                     seconds. I've double-checked
>>>>>>>>>>                             that it goes to 'copy_timeout' in
>>>>>>>>>>                     ts-conf/rcf.conf.
>>>>>>>>>>                     Description in in
>>>>>>>>>>                     doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>                             says that copy_timeout is in
>>>>>>>>>>                     seconds and implementation in
>>>>>>>>>>                     lib/rcfunix/rcfunix.c passes the value to
>>>>>>>>>>                     select() tv_sec.
>>>>>>>>>>                     Theoretically select() could be
>>>>>>>>>>                     interrupted by signal, but I
>>>>>>>>>>                             think it is unlikely here.
>>>>>>>>>>
>>>>>>>>>>                             I'm not sure that I understand
>>>>>>>>>>                     what do you mean by RCF
>>>>>>>>>>                     connection timeout. Does it happen on TE
>>>>>>>>>>                     startup when RCF
>>>>>>>>>>                             starts test agents. If so,
>>>>>>>>>>                     TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>>>>>                             does it happen when tests are in
>>>>>>>>>>                     progress, e.g. in the middle
>>>>>>>>>>                             of a test. If so,
>>>>>>>>>>                     TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>>>>>>                             likely either host with test
>>>>>>>>>>                     agent dies or test agent itself
>>>>>>>>>>                     crashes. It would be easier for me if
>>>>>>>>>>                     classify it if you share
>>>>>>>>>>                             text log (log.txt, full or just
>>>>>>>>>>                     corresponding fragment with
>>>>>>>>>>                             some context). Also content of
>>>>>>>>>>                     ta.DPDK or ta.Peer file
>>>>>>>>>>                     depending on which agent has problems
>>>>>>>>>>                     could shed some light.
>>>>>>>>>>                     Corresponding files contain stdout/stderr
>>>>>>>>>>                     of test agents.
>>>>>>>>>>
>>>>>>>>>>                     Andrew.
>>>>>>>>>>
>>>>>>>>>>                             On 8/23/23 17:45, Adam Hassick
>>>>>>>>>>                     wrote:
>>>>>>>>>>>                     Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                             I've set up a test rig
>>>>>>>>>>>                     repository here, and have created
>>>>>>>>>>>                     configurations for our development
>>>>>>>>>>>                     testbed based off of the
>>>>>>>>>>>                     examples.
>>>>>>>>>>>                             We've been able to get the test
>>>>>>>>>>>                     suite to run manually on
>>>>>>>>>>>                     Mellanox CX5 devices once.
>>>>>>>>>>>                     However, we are running into an issue
>>>>>>>>>>>                     where, when RCF starts,
>>>>>>>>>>>                             the RCF connection times out
>>>>>>>>>>>                     very frequently. We aren't sure
>>>>>>>>>>>                             why this is the case.
>>>>>>>>>>>                             It works sometimes, but most of
>>>>>>>>>>>                     the time when we try to run
>>>>>>>>>>>                             the test engine, it encounters
>>>>>>>>>>>                     this issue.
>>>>>>>>>>>                             I've tried changing the RCF port
>>>>>>>>>>>                     by setting
>>>>>>>>>>>                     "TE_RCF_PORT=<some port number>" and
>>>>>>>>>>>                     rebooting the testbed
>>>>>>>>>>>                     machines. Neither seems to fix the issue.
>>>>>>>>>>>
>>>>>>>>>>>                             It also seems like the timeout
>>>>>>>>>>>                     takes far longer than 60
>>>>>>>>>>>                     seconds, even when running "export
>>>>>>>>>>>                     TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>                             before I try to run the test suite.
>>>>>>>>>>>                             I assume the unit for this
>>>>>>>>>>>                     variable is seconds?
>>>>>>>>>>>
>>>>>>>>>>>                     Thanks,
>>>>>>>>>>>                             Adam
>>>>>>>>>>>
>>>>>>>>>>>                             On Mon, Aug 21, 2023 at 10:19 AM
>>>>>>>>>>>                     Adam Hassick
>>>>>>>>>>>                             <ahassick@iol.unh.edu
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                     Thanks, I've cloned the example
>>>>>>>>>>>                     repository and will start
>>>>>>>>>>>                     setting up a configuration for our
>>>>>>>>>>>                     development testbed
>>>>>>>>>>>                     today. I'll let you know if I run into
>>>>>>>>>>>                     any difficulties
>>>>>>>>>>>                                 or have any questions.
>>>>>>>>>>>
>>>>>>>>>>>                                  - Adam
>>>>>>>>>>>
>>>>>>>>>>>                                 On Sun, Aug 20, 2023 at
>>>>>>>>>>>                     4:40 AM Andrew Rybchenko
>>>>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>                     wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                     I've published
>>>>>>>>>>>                     https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>
>>>>>>>>>>>                     <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>                     <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>
>>>>>>>>>>>                                     Hopefully it will help
>>>>>>>>>>>                     to define your test rigs and
>>>>>>>>>>>                                     successfully run some
>>>>>>>>>>>                     tests manually. Feel free to
>>>>>>>>>>>                                     ask any questions and
>>>>>>>>>>>                     I'll answer here and try to
>>>>>>>>>>>                                     update documentation.
>>>>>>>>>>>
>>>>>>>>>>>                                     Meanwhile I'll prepare
>>>>>>>>>>>                     missing bits for steps (2) and
>>>>>>>>>>>                                     (3).
>>>>>>>>>>>                                     Hopefully everything is
>>>>>>>>>>>                     in place for step (4), but we
>>>>>>>>>>>                                     need to make steps (2)
>>>>>>>>>>>                     and (3) first.
>>>>>>>>>>>
>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                     On 8/18/23 21:40, Andrew
>>>>>>>>>>>                     Rybchenko wrote:
>>>>>>>>>>>>                     Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                     > I've conferred with
>>>>>>>>>>>>                     the rest of the team, and we
>>>>>>>>>>>>                                     think it would be best
>>>>>>>>>>>>                     to move forward with mainly
>>>>>>>>>>>>                                     option B.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     OK, I'll provide the
>>>>>>>>>>>>                     sample on Monday for you. It is
>>>>>>>>>>>>                                     almost ready right now,
>>>>>>>>>>>>                     but I need to double-check
>>>>>>>>>>>>                                     it before publishing.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Regards,
>>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On 8/17/23 20:03, Adam
>>>>>>>>>>>>                     Hassick wrote:
>>>>>>>>>>>>>                     Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     I'm adding the CI
>>>>>>>>>>>>>                     mailing list to this
>>>>>>>>>>>>>                                     conversation. Others
>>>>>>>>>>>>>                     in the community might find
>>>>>>>>>>>>>                                     this conversation
>>>>>>>>>>>>>                     valuable.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     We do want to run
>>>>>>>>>>>>>                     testing on a regular basis. The
>>>>>>>>>>>>>                                     Jenkins integration
>>>>>>>>>>>>>                     will be very useful for us, as
>>>>>>>>>>>>>                                     most of our CI is
>>>>>>>>>>>>>                     orchestrated by Jenkins.
>>>>>>>>>>>>>                                     I've conferred with
>>>>>>>>>>>>>                     the rest of the team, and we
>>>>>>>>>>>>>                                     think it would be best
>>>>>>>>>>>>>                     to move forward with mainly
>>>>>>>>>>>>>                                     option B.
>>>>>>>>>>>>>                                     If you would like to
>>>>>>>>>>>>>                     know anything about our
>>>>>>>>>>>>>                                     testbeds that would
>>>>>>>>>>>>>                     help you with creating an
>>>>>>>>>>>>>                                     example ts-rigs repo,
>>>>>>>>>>>>>                     I'd be happy to answer any
>>>>>>>>>>>>>                                     questions you have.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     We have multiple test
>>>>>>>>>>>>>                     rigs (we call these
>>>>>>>>>>>>>                                     "DUT-tester pairs")
>>>>>>>>>>>>>                     that we run our existing
>>>>>>>>>>>>>                                     hardware testing on,
>>>>>>>>>>>>>                     with differing network
>>>>>>>>>>>>>                                     hardware and CPU
>>>>>>>>>>>>>                     architecture. I figured this might
>>>>>>>>>>>>>                                     be an important detail.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     On Thu, Aug 17, 2023
>>>>>>>>>>>>>                     at 11:44 AM Andrew Rybchenko
>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>                     wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Greatings Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         I'm happy to hear
>>>>>>>>>>>>>                     that you're trying to bring
>>>>>>>>>>>>>                                         it up.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         As I understand
>>>>>>>>>>>>>                     the final goal is to run it on
>>>>>>>>>>>>>                                         regular basis. So,
>>>>>>>>>>>>>                     we need to make it properly
>>>>>>>>>>>>>                                         from the very
>>>>>>>>>>>>>                     beginning.
>>>>>>>>>>>>>                                         Bring up of all
>>>>>>>>>>>>>                     features consists of 4 steps:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         1. Create
>>>>>>>>>>>>>                     site-specific repository (we call it
>>>>>>>>>>>>>                                         ts-rigs) which
>>>>>>>>>>>>>                     contains information about test
>>>>>>>>>>>>>                                         rigs and other
>>>>>>>>>>>>>                     site-specific information like
>>>>>>>>>>>>>                                         where to send
>>>>>>>>>>>>>                     mails, where to store logs etc.
>>>>>>>>>>>>>                                         It is required for
>>>>>>>>>>>>>                     manual execution as well,
>>>>>>>>>>>>>                                         since test rigs
>>>>>>>>>>>>>                     description is essential. I'll
>>>>>>>>>>>>>                                         return to the
>>>>>>>>>>>>>                     topic below.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         2. Setup logs
>>>>>>>>>>>>>                     storage for automated runs.
>>>>>>>>>>>>>                                         Basically it is a
>>>>>>>>>>>>>                     disk space plus apache2 web
>>>>>>>>>>>>>                                         server with few
>>>>>>>>>>>>>                     CGI scripts which help a lot to
>>>>>>>>>>>>>                                         save disk space.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         3. Setup Bublik
>>>>>>>>>>>>>                     web application which provides
>>>>>>>>>>>>>                                         web interface to
>>>>>>>>>>>>>                     view testing results. Same as
>>>>>>>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         4. Setup Jenkins
>>>>>>>>>>>>>                     to run tests on regularly,
>>>>>>>>>>>>>                                         save logs in log
>>>>>>>>>>>>>                     storage (2) and import it to
>>>>>>>>>>>>>                                         bublik (3).
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Last few month we
>>>>>>>>>>>>>                     spent on our homework to make
>>>>>>>>>>>>>                                         it simpler to
>>>>>>>>>>>>>                     bring up automated execution
>>>>>>>>>>>>>                                         using Jenkins -
>>>>>>>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Corresponding bits
>>>>>>>>>>>>>                     in dpdk-ethdev-ts will be
>>>>>>>>>>>>>                                         available tomorrow.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Let's return to
>>>>>>>>>>>>>                     the step (1).
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Unfortunately
>>>>>>>>>>>>>                     there is no publicly available
>>>>>>>>>>>>>                                         example of the
>>>>>>>>>>>>>                     ts-rigs repository since
>>>>>>>>>>>>>                                         sensitive
>>>>>>>>>>>>>                     site-specific information is located
>>>>>>>>>>>>>                                         there. But I'm
>>>>>>>>>>>>>                     ready to help you to create it
>>>>>>>>>>>>>                                         for UNH. I see two
>>>>>>>>>>>>>                     options here:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         (A) I'll ask
>>>>>>>>>>>>>                     questions and based on your
>>>>>>>>>>>>>                                         answers will
>>>>>>>>>>>>>                     create the first draft with my
>>>>>>>>>>>>>                                         comments.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         (B) I'll make a
>>>>>>>>>>>>>                     template/example ts-rigs repo,
>>>>>>>>>>>>>                                         publish it and
>>>>>>>>>>>>>                     you'll create UNH ts-rigs based
>>>>>>>>>>>>>                                         on it.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Of course, I'll
>>>>>>>>>>>>>                     help to debug and finally bring
>>>>>>>>>>>>>                                         it up in any case.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         (A) is a bit
>>>>>>>>>>>>>                     simpler for me and you, but (B) is
>>>>>>>>>>>>>                                         a bit more generic
>>>>>>>>>>>>>                     and will help other
>>>>>>>>>>>>>                                         potential users to
>>>>>>>>>>>>>                     bring it up.
>>>>>>>>>>>>>                                         We can combine
>>>>>>>>>>>>>                     (A)+(B). I.e. start from (A).
>>>>>>>>>>>>>                                         What do you think?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         On 8/17/23 15:18,
>>>>>>>>>>>>>                     Konstantin Ushakov wrote:
>>>>>>>>>>>>>>                     Greetings Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         Thanks for
>>>>>>>>>>>>>>                     contacting us. I copy Andrew who
>>>>>>>>>>>>>>                                         would be happy to
>>>>>>>>>>>>>>                     help
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>>                                         Konstantin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>>>>                     <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         
>>>>>>>>>>>>>>>                                         Greetings
>>>>>>>>>>>>>>>                     Konstantin,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         I am in the
>>>>>>>>>>>>>>>                     process of setting up the DPDK
>>>>>>>>>>>>>>>                                         Poll Mode Driver
>>>>>>>>>>>>>>>                     test suite as an addition to
>>>>>>>>>>>>>>>                                         our testing
>>>>>>>>>>>>>>>                     coverage for DPDK at the UNH lab.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         I have some
>>>>>>>>>>>>>>>                     questions about how to set the
>>>>>>>>>>>>>>>                                         test suite
>>>>>>>>>>>>>>>                     arguments.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         I have been able
>>>>>>>>>>>>>>>                     to configure the Test Engine
>>>>>>>>>>>>>>>                                         to connect to
>>>>>>>>>>>>>>>                     the hosts in the testbed. The
>>>>>>>>>>>>>>>                                         RCF,
>>>>>>>>>>>>>>>                     Configurator, and Tester all begin to
>>>>>>>>>>>>>>>                                         run, however the
>>>>>>>>>>>>>>>                     prelude of the test suite
>>>>>>>>>>>>>>>                                         fails to run.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>                     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>                     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         The
>>>>>>>>>>>>>>>                     documentation mentions that there are
>>>>>>>>>>>>>>>                                         several test
>>>>>>>>>>>>>>>                     parameters for the test suite,
>>>>>>>>>>>>>>>                                         like for the IUT
>>>>>>>>>>>>>>>                     test link MAC, etc. These
>>>>>>>>>>>>>>>                                         seem like they
>>>>>>>>>>>>>>>                     would need to be set somewhere
>>>>>>>>>>>>>>>                                         to run many of
>>>>>>>>>>>>>>>                     the tests.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         I see in the
>>>>>>>>>>>>>>>                     Test Engine documentation, there
>>>>>>>>>>>>>>>                                         are instructions
>>>>>>>>>>>>>>>                     on how to create new
>>>>>>>>>>>>>>>                                         parameters for
>>>>>>>>>>>>>>>                     test suites in the Tester
>>>>>>>>>>>>>>>                                         configuration,
>>>>>>>>>>>>>>>                     but there is nothing in the
>>>>>>>>>>>>>>>                                         user guide or in
>>>>>>>>>>>>>>>                     the Tester guide for how to
>>>>>>>>>>>>>>>                                         set the
>>>>>>>>>>>>>>>                     arguments for the parameters when
>>>>>>>>>>>>>>>                                         running the test
>>>>>>>>>>>>>>>                     suite that I can find. I'm
>>>>>>>>>>>>>>>                                         not sure if I
>>>>>>>>>>>>>>>                     need to write my own Tester
>>>>>>>>>>>>>>>                                         config, or if I
>>>>>>>>>>>>>>>                     should be setting these in
>>>>>>>>>>>>>>>                                         some other way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         How should these
>>>>>>>>>>>>>>>                     values be set?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         I'm also not
>>>>>>>>>>>>>>>                     sure what environment
>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>                     variables/arguments are strictly
>>>>>>>>>>>>>>>                     necessary or
>>>>>>>>>>>>>>>                                         which are optional.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         Regards,
>>>>>>>>>>>>>>>                                         Adam
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         --
>>>>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>>>                     InterOperability Lab
>>>>>>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     --                
>>>>>>>>>>>>>                     *Adam Hassick*
>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                 -- *Adam Hassick*
>>>>>>>>>>>                     Senior Developer
>>>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             --         *Adam Hassick*
>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                     -- 
>>>>>>>>>                     *Adam Hassick*
>>>>>>>>>                     Senior Developer
>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>                     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>                     <https://www.iol.unh.edu/>
>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                 -- 
>>>>>>>                 *Adam Hassick*
>>>>>>>                 Senior Developer
>>>>>>>                 UNH InterOperability Lab
>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                 +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>             -- 
>>>>>             *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>             ahassick@iol.unh.edu
>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 129917 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 40524 bytes --]

Hi Andrew,

Yes, I copied the X710 configs to set up XL710 configs. I changed the
environment variable names from the X710 suffix to XL710 suffix in the
script, and forgot to change them in the corresponding environment file.
That fixed the issue.

I got the checkup tests working on the XL710 now. Most of them are failing,
which leads me to believe this is an issue with our testbed. Based on the
DPDK documentation for i40e, the firmware and driver versions are much
older than what DPDK 22.11 LTS and main prefer, so I'll try updating those.

For now I'm working on getting the XL710 checkup tests passing, and will
pick up getting the E810 configured properly next. I'll let you know if I
run into any more issues in relation to the test engine.

Thanks,
Adam

On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> On 9/5/23 18:01, Adam Hassick wrote:
>
> Hi Andrew,
>
> The compilation warning issue is now resolved. Again, thank you guys for
> fixing this for us. I can run the tests on the Mellanox CX5s again, however
> I'm running into a couple new issues with running the prologues on the
> Intel cards.
>
> When running testing on the Intel XL710s, I see this error appear in the
> log:
>
> ERROR  prologue  Environment LIB  14:16:13.650
>> Too few networks in available configuration (0) in comparison with
>> required (1)
>>
>
> This seems like a trivial configuration error, perhaps this is something I
> need to set up in ts-rigs. I briefly searched through the examples there
> and didn't see any mention of how to set up a network.
> I will attach this log just in case you need more information.
>
>
> Unfortunately logs are insufficient to understand it. I've pushed new tag
> to TE v1.19.0 which add log message with TE_* environment variables.
> Most likely something is wrong with variables which are used as conditions
> when available networks are defined in ts-conf/cs/inc.net_cfg_pci_fns.yml:
> TE_PCI_INSTANCE_IUT_TST1
> TE_PCI_INSTANCE_IUT_TST1a
> TE_PCI_INSTANCE_TST1a_IUT
> TE_PCI_INSTANCE_TST1_IUT
> My guess it that you change naming a bit, but script like
> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>
> There is a different error when running on the Intel E810s. It appears to
> me like it starts DPDK, does some configuration inside DPDK and on the
> device, and then fails to bring the device back up. Since this error seems
> very non-trivial, I will also attach this log.
>
>
> This one is a bit simpler. Few lines after the first ERROR in log I see
> the following:
> WARN  RCF  DPDK  13:06:00.144
> ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)
> ice_rx_queue_start(): fail to program RX queue 0
> ice_dev_start(): fail to start Rx queue 0
> Device with port_id=0 already stopped
>
> It is stdout/stderr from test agent which runs DPDK. Same logs in plain
> format are available in ta.DPDK file.
> I'm not an expert here, but I vaguely remember that E810 requires correct
> firmware and DDP to be loaded.
> There is some information in dpdk/doc/guides/nics/ice.rst.
>
> You can try to add --dev-args=safe-mode-support=1 command-line option
> described there.
>
> Hope it helps,
> Andrew.
>
>
> Thanks,
> Adam
>
> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> On 8/31/23 22:38, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> I have one additional question as well: Does the test engine support
>> running tests on two ARMv8 test agents?
>>
>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>
>>
>> Great. Let me know when that's fixed.
>>
>>
>> Done. We also fixed a number of warnings in TE.
>> Also we fixed root test package name to be consistent with the repository
>> name.
>>
>> Support for old LTS branches was dropped some time ago, but in the future
>>> it is definitely possible to keep it for new LTS branches. I think 22.11 is
>>> supported, but I'm not sure about older LTS releases.
>>>
>>
>> Good to know.
>>
>>
>>> 2. You can add command-line option --sanity to run tests marked with
>>> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
>>> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
>>> there is a space for terminology improvement here. We'll do it.
>>>
>>
>> Done. Now it is called --checkup.
>>
>>
>>> Also it takes a lot of time because of failures and tests which wait for
>>> some timeout.
>>>
>>
>> That makes sense to me. We'll use the time to complete tests on virtio or
>> the Intel devices as a reference for how long the tests really take to
>> complete.
>> We will explore the possibility of periodically running the sanity tests
>> for patches.
>>
>>
>> I'll double-check and let you know how long entire TS runs on Intel X710,
>> E810, Mellanox CX5 and virtio net. Just to ensure that time observed in
>> your case looks the same.
>>
>>
>>
>>> The test harness can provide coverage reports based on gcov, but I'm not
>>> sure what you mean by a "dial" to control test coverage. Provided reports
>>> are rather for human to analyze.
>>>
>>
>> The general idea is to have some kind of parameter on the test suite,
>> which could be an integer ranging from zero to ten, that controls how many
>> tests are run based on how important the test is.
>>
>> Similar to how some command line interfaces provide a verbosity level
>> parameter (some number of "-v" arguments) to control the importance of the
>> information in the log.
>> The verbosity level zero only prints very important log messages, while
>> ten prints everything.
>>
>> In much the same manner as above, this "dial" parameter controls what
>> tests are run and with what parameters based on how important those tests
>> and test parameter combinations are.
>> Coverage Level zero tells the suite to run a very basic set of important
>> tests, with minimal parameterization. This mode would take only ~5-10
>> minutes to run.
>> In contrast, Coverage Level ten includes all the edge cases, every
>> combination of test parameters, everything the test suite can do, which
>> takes the normal several hours to run.
>> The values 1 - 9 are between those two extremes, allowing the user to get
>> a gradient of test coverage in the results and to limit the running time.
>>
>> Then we could, for example, run the "run.sh" with a level of 2 or 3 for
>> incoming patches that need quick results, and with a level of 10 for the
>> less often run periodic tests performed on main or LTS branches.
>>
>>
>> Understood now. Thanks a lot for the idea. We'll discuss it and come back.
>>
>>
>>
>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing
>>> results. Unfortunately it is time consuming to fill in expectations
>>> database since it is necessary to analyze testing results and classify if
>>> it is a bug or just acceptable behaviour aspect.
>>>
>>> Bublik allows to compare results of two runs. It is useful for human,
>>> but still not good for automation.
>>>
>>> I have local patch for mlx5 driver which reports Tx ring size maximum.
>>> It makes pass rate higher. It is a problem for test harness that mlx5 does
>>> not report limits right now.
>>>
>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio
>>> net is 99% right now and could be done 100% easily (just one thing to fix
>>> in expectations).
>>>
>>> I think logs storage setup is essential for logs analysis. Of course,
>>> you can request HTML logs when you run tests (--log-html=html) or generate
>>> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>>> browser, but logs storage makes it more convenient.
>>>
>>
>> We are interested in setting up Bublik, potentially as an
>> externally-facing component, once we have our process of running the test
>> suite stabilized.
>> Once we are able to run the test suite again, I'll see what the pass rate
>> is on our other hardware.
>> Good to know that it isn't an issue with our dev testbed causing the high
>> fail rate.
>>
>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>> development testbed. Although they are slightly different devices, ideally
>> the pass rate will be identical or similar. I have yet to set up a VM pair
>> for virtio, but we will soon.
>>
>> Latest version of test-environment has examples of our CGI scripts which
>>> we use for log storage (see tools/log_server/README.md).
>>>
>>> Also all bits for Jenkins setup are available. See
>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>> ts-rigs-sample.
>>>
>>
>> Jenkins integration, setting up production rig configurations, and
>> permanent log storage will be our next steps once I am able to run the
>> tests again.
>> Unless there is an easy way to have meson not pass "-Werror" into GCC.
>> Then I would be able to run the test suite.
>>
>>
>> Hopefully it is resolved now.
>>
>> I thought a bit more about your usecase for Jenkins. I'm not 100% sure
>> that existing pipelines are convenient for your usecase.
>> Fill free to ask questions when you are on it.
>>
>> Thanks,
>> Andrew.
>>
>>
>> Thanks,
>> Adam
>>
>>
>>
>>>
>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> That fix seems to have resolved the issue, thanks for the quick
>>> turnaround time on that patch.
>>> Now that we have the RCF timeout issue resolved, there are a few other
>>> questions and issues that we have about the tests themselves.
>>>
>>> 1. The test suite fails to build with a couple warnings.
>>>
>>> Below is the stderr log from compilation:
>>>
>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>> ../../lib/dpdk_pmd_ts.c
>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>> ‘test_create_traffic_generator_params’:
>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>>> no format arguments [-Werror=format-security]
>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>> |     ^~
>>>> cc1: all warnings being treated as errors
>>>> ninja: build stopped: subcommand failed.
>>>> ninja: Entering directory `.'
>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>> ../../lib/dpdk_pmd_ts.c
>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>> ‘test_create_traffic_generator_params’:
>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>>> no format arguments [-Werror=format-security]
>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>> |     ^~
>>>> cc1: all warnings being treated as errors
>>>>
>>>
>>> This error wasn't occurring last week, which was the last time I ran the
>>> tests.
>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester has
>>> GCC v11.4.0 installed, if this information is helpful.
>>>
>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>> to lower the test coverage and have the test suite run faster?
>>>
>>> For some context, we run immediate testing on incoming patches for DPDK
>>> main and development branches, as well as periodic test runs on the main,
>>> stable, and LTS branches.
>>> For us to consider including this test suite as part of our immediate
>>> testing on patches, we would have to reduce the test coverage to the most
>>> important tests.
>>> This is primarily to reduce the testing time to, for example, less than
>>> 30 minutes. Testing on patches can't take too long because the lab can
>>> receive numerous patches each day, which each require individual testing
>>> runs.
>>>
>>> At what frequency we run these tests, and on what, still needs to be
>>> discussed with the DPDK community, but it would be nice to know if the test
>>> suite had a "dial" to control the testing coverage.
>>>
>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300
>>> of ~6,600 tests passed. Is there anything we can do to diagnose these test
>>> failures?
>>>
>>> Thanks,
>>> Adam
>>>
>>>
>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hi Adam,
>>>>
>>>> I've pushed the fix in main branch and a new tag v1.18.1. It should
>>>> solve the problem with IPv6 address from DNS.
>>>>
>>>> Andrew.
>>>>
>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>
>>>> Hi Adam,
>>>>
>>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>>> to use IPv4?
>>>>
>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your
>>>> case, I guess it is the root cause of the problem.
>>>> Of course, it is TE problem since I see really weird code in
>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>
>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>> ts-factory/test-environment repository. Please, try.
>>>>
>>>> It is late night fix with minimal testing and no review. I'll pass it
>>>> through review process tomorrow and
>>>> hopefully it will be released in one-two days.
>>>>
>>>> Andrew.
>>>>
>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> We have yet to notice a distinct pattern with the failures. Sometimes,
>>>> the RCF will start and connect without issue a few times in a row before
>>>> failing to connect again. Once the issue begins to occur, neither rebooting
>>>> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
>>>> directories (suites, agents, inst) and rebooting the hosts afterward
>>>> resolves the issue. When it begins working again seems very arbitrary to us.
>>>>
>>>> I do usually try to terminate the test engine with Ctrl+C, but when it
>>>> hangs while trying to start RCF, that does not work.
>>>>
>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>>> to use IPv4?
>>>>
>>>>  - Adam
>>>>
>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>
>>>>> Done. It works fine for me without any issues.
>>>>>
>>>>> Have you noticed any pattern when it works or does not work?
>>>>> May be it is a problem of not clean state after termination?
>>>>> Does it work fine the first time after DUTs reboot?
>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>> terminal where you execute run.sh command.
>>>>>  In this case it should shutdown gracefully and close all test agents
>>>>> and engine applications.
>>>>>
>>>>> (I'm trying to understand why you've seen many test agent processes.
>>>>> It should not happen.)
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>
>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>> This could be the source of the problem. I encountered a dependency
>>>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>>>> system. Since the tester is also the host having connection issues, I will
>>>>> try downgrading that system to 20.04, and see if that changes anything.
>>>>>
>>>>>
>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>>>
>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>
>>>>> I did try passing in the "--vg-rcf" argument to the run.sh script of
>>>>> the test suite after installing valgrind, but there was no additional
>>>>> output that I saw.
>>>>>
>>>>>
>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>> (direction where you run test engine).
>>>>>
>>>>>
>>>>> I will try pulling in the changes you've pushed up, and will see if
>>>>> that fixes anything.
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>> Hello Adam,
>>>>>>
>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>
>>>>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>>>>> running on test engine and agents?
>>>>>>
>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>>>> patch in review to fix it, however, the behaviour is different in
>>>>>> this case, so it is unlike the same problem.
>>>>>>
>>>>>>
>>>>>> I've just published a new tag which fixes known test engine side
>>>>>> problems on Debian 12 and Fedora 38.
>>>>>>
>>>>>>
>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>> run with option --vg-rcf to check if something weird is happening.
>>>>>>
>>>>>> What I don't understand right now is why I see just one failed
>>>>>> attempt
>>>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>>>
>>>>>> Andrew.
>>>>>>
>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>
>>>>>>  > Is there any firewall in the network or on test hosts which could
>>>>>> block incoming TCP connection to the port 23571
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>> where you run test engine?
>>>>>>
>>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>>> connection does work sometimes.
>>>>>>
>>>>>>  > If behaviour the same on the next try and you see that test agent
>>>>>> is kept running, could you check using
>>>>>>  >
>>>>>>  > # netstat -tnlp
>>>>>>  >
>>>>>>  > that Test Agent is listening on the port and try to establish TCP
>>>>>> connection from test agent using
>>>>>>  >
>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>  >
>>>>>>  > and check if TCP connection could be established.
>>>>>>
>>>>>> I was able to replicate the same behavior again, where it hangs while
>>>>>> RCF is trying to start.
>>>>>> Running this command, I see this in the output:
>>>>>>
>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>>  18599/ta
>>>>>>
>>>>>> So it seems like it is listening on the correct port.
>>>>>> Additionally, I was able to connect to the Tester machine from our
>>>>>> Test Engine host using telnet. It printed the PID of the process once the
>>>>>> connection was opened.
>>>>>>
>>>>>> I tried running the "ta" application manually on the command line,
>>>>>> and it didn't print anything at all.
>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>
>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>     Hi Adam,
>>>>>>
>>>>>>      > On the tester host (which appears to be the Peer agent), there
>>>>>>     are four processes that I see running, which look like the test
>>>>>>     agent processes.
>>>>>>
>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>
>>>>>>     Is there any firewall in the network or on test hosts which could
>>>>>>     block incoming TCP connection to the port 23571
>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>     where you run test engine?
>>>>>>
>>>>>>     If behaviour the same on the next try and you see that test agent
>>>>>> is
>>>>>>     kept running, could you check using
>>>>>>
>>>>>>     # netstat -tnlp
>>>>>>
>>>>>>     that Test Agent is listening on the port and try to establish TCP
>>>>>>     connection from test agent using
>>>>>>
>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>
>>>>>>     and check if TCP connection could be established.
>>>>>>
>>>>>>     Another idea is to login Tester under root as testing does, get
>>>>>>     start TA command from the log and try it by hands without -n and
>>>>>>     remove extra escaping.
>>>>>>
>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>
>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>
>>>>>>     Hopefully in this case test agent directory remains in the /tmp
>>>>>> and
>>>>>>     you don't need to copy it as testing does.
>>>>>>     May be output could shed some light on what's going on.
>>>>>>
>>>>>>     Andrew.
>>>>>>
>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>
>>>>>>     Hi Andrew,
>>>>>>
>>>>>>     This is the output that I see in the terminal when this failure
>>>>>>     occurs, after the test agent binaries build and the test engine
>>>>>>     starts:
>>>>>>
>>>>>>     Platform default build - pass
>>>>>>     Simple RCF consistency check succeeded
>>>>>>     --->>> Starting Logger...done
>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>
>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>
>>>>>>     On the tester host (which appears to be the Peer agent), there
>>>>>> are
>>>>>>     four processes that I see running, which look like the test agent
>>>>>>     processes.
>>>>>>
>>>>>>     ta.Peer is an empty file. I've attached the log.txt from this
>>>>>> run.
>>>>>>
>>>>>>      - Adam
>>>>>>
>>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>         Hi Adam,
>>>>>>
>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>         says that copy_timeout is in seconds and implementation in
>>>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>>>         Theoretically select() could be interrupted by signal, but I
>>>>>>         think it is unlikely here.
>>>>>>
>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>         does it happen when tests are in progress, e.g. in the middle
>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>>         likely either host with test agent dies or test agent itself
>>>>>>         crashes. It would be easier for me if classify it if you
>>>>>> share
>>>>>>         text log (log.txt, full or just corresponding fragment with
>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>         depending on which agent has problems could shed some light.
>>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>>
>>>>>>         Andrew.
>>>>>>
>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>
>>>>>>         Hi Andrew,
>>>>>>
>>>>>>         I've set up a test rig repository here, and have created
>>>>>>         configurations for our development testbed based off of the
>>>>>>         examples.
>>>>>>         We've been able to get the test suite to run manually on
>>>>>>         Mellanox CX5 devices once.
>>>>>>         However, we are running into an issue where, when RCF starts,
>>>>>>         the RCF connection times out very frequently. We aren't sure
>>>>>>         why this is the case.
>>>>>>         It works sometimes, but most of the time when we try to run
>>>>>>         the test engine, it encounters this issue.
>>>>>>         I've tried changing the RCF port by setting
>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>>>         machines. Neither seems to fix the issue.
>>>>>>
>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>         before I try to run the test suite.
>>>>>>         I assume the unit for this variable is seconds?
>>>>>>
>>>>>>         Thanks,
>>>>>>         Adam
>>>>>>
>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>
>>>>>>             Hi Andrew,
>>>>>>
>>>>>>             Thanks, I've cloned the example repository and will start
>>>>>>             setting up a configuration for our development testbed
>>>>>>             today. I'll let you know if I run into any difficulties
>>>>>>             or have any questions.
>>>>>>
>>>>>>              - Adam
>>>>>>
>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                 Hi Adam,
>>>>>>
>>>>>>                 I've published
>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>                 Hopefully it will help to define your test rigs and
>>>>>>                 successfully run some tests manually. Feel free to
>>>>>>                 ask any questions and I'll answer here and try to
>>>>>>                 update documentation.
>>>>>>
>>>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>>>                 (3).
>>>>>>                 Hopefully everything is in place for step (4), but we
>>>>>>                 need to make steps (2) and (3) first.
>>>>>>
>>>>>>                 Andrew.
>>>>>>
>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>
>>>>>>                 Hi Adam,
>>>>>>
>>>>>>                 > I've conferred with the rest of the team, and we
>>>>>>                 think it would be best to move forward with mainly
>>>>>>                 option B.
>>>>>>
>>>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>>>                 almost ready right now, but I need to double-check
>>>>>>                 it before publishing.
>>>>>>
>>>>>>                 Regards,
>>>>>>                 Andrew.
>>>>>>
>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 I'm adding the CI mailing list to this
>>>>>>                 conversation. Others in the community might find
>>>>>>                 this conversation valuable.
>>>>>>
>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>                 Jenkins integration will be very useful for us, as
>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>                 think it would be best to move forward with mainly
>>>>>>                 option B.
>>>>>>                 If you would like to know anything about our
>>>>>>                 testbeds that would help you with creating an
>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>                 questions you have.
>>>>>>
>>>>>>                 We have multiple test rigs (we call these
>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>                 hardware testing on, with differing network
>>>>>>                 hardware and CPU architecture. I figured this might
>>>>>>                 be an important detail.
>>>>>>
>>>>>>                 Thanks,
>>>>>>                 Adam
>>>>>>
>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                     Greatings Adam,
>>>>>>
>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>                     it up.
>>>>>>
>>>>>>                     As I understand the final goal is to run it on
>>>>>>                     regular basis. So, we need to make it properly
>>>>>>                     from the very beginning.
>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>
>>>>>>                     1. Create site-specific repository (we call it
>>>>>>                     ts-rigs) which contains information about test
>>>>>>                     rigs and other site-specific information like
>>>>>>                     where to send mails, where to store logs etc.
>>>>>>                     It is required for manual execution as well,
>>>>>>                     since test rigs description is essential. I'll
>>>>>>                     return to the topic below.
>>>>>>
>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>                     server with few CGI scripts which help a lot to
>>>>>>                     save disk space.
>>>>>>
>>>>>>                     3. Setup Bublik web application which provides
>>>>>>                     web interface to view testing results. Same as
>>>>>>                     https://ts-factory.io/bublik
>>>>>>                     <https://ts-factory.io/bublik>
>>>>>> <https://ts-factory.io/bublik>
>>>>>>
>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>                     save logs in log storage (2) and import it to
>>>>>>                     bublik (3).
>>>>>>
>>>>>>                     Last few month we spent on our homework to make
>>>>>>                     it simpler to bring up automated execution
>>>>>>                     using Jenkins -
>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>                     available tomorrow.
>>>>>>
>>>>>>                     Let's return to the step (1).
>>>>>>
>>>>>>                     Unfortunately there is no publicly available
>>>>>>                     example of the ts-rigs repository since
>>>>>>                     sensitive site-specific information is located
>>>>>>                     there. But I'm ready to help you to create it
>>>>>>                     for UNH. I see two options here:
>>>>>>
>>>>>>                     (A) I'll ask questions and based on your
>>>>>>                     answers will create the first draft with my
>>>>>>                     comments.
>>>>>>
>>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>>                     on it.
>>>>>>
>>>>>>                     Of course, I'll help to debug and finally bring
>>>>>>                     it up in any case.
>>>>>>
>>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>>                     a bit more generic and will help other
>>>>>>                     potential users to bring it up.
>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>                     What do you think?
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>
>>>>>>                     Greetings Adam,
>>>>>>
>>>>>>
>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>                     would be happy to help
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Konstantin
>>>>>>
>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>
>>>>>>                     
>>>>>>                     Greetings Konstantin,
>>>>>>
>>>>>>                     I am in the process of setting up the DPDK
>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>
>>>>>>                     I have some questions about how to set the
>>>>>>                     test suite arguments.
>>>>>>
>>>>>>                     I have been able to configure the Test Engine
>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>                     run, however the prelude of the test suite
>>>>>>                     fails to run.
>>>>>>
>>>>>>
>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>
>>>>>>                     The documentation mentions that there are
>>>>>>                     several test parameters for the test suite,
>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>                     seem like they would need to be set somewhere
>>>>>>                     to run many of the tests.
>>>>>>
>>>>>>                     I see in the Test Engine documentation, there
>>>>>>                     are instructions on how to create new
>>>>>>                     parameters for test suites in the Tester
>>>>>>                     configuration, but there is nothing in the
>>>>>>                     user guide or in the Tester guide for how to
>>>>>>                     set the arguments for the parameters when
>>>>>>                     running the test suite that I can find. I'm
>>>>>>                     not sure if I need to write my own Tester
>>>>>>                     config, or if I should be setting these in
>>>>>>                     some other way.
>>>>>>
>>>>>>                     How should these values be set?
>>>>>>
>>>>>>                     I'm also not sure what environment
>>>>>>                     variables/arguments are strictly necessary or
>>>>>>                     which are optional.
>>>>>>
>>>>>>                     Regards,
>>>>>>                     Adam
>>>>>>
>>>>>>                     --                     *Adam Hassick*
>>>>>>                     Senior Developer
>>>>>>                     UNH InterOperability Lab
>>>>>>                     ahassick@iol.unh.edu
>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu>
>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> <https://www.iol.unh.edu/>
>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu>
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             --             *Adam Hassick*
>>>>>>             Senior Developer
>>>>>>             UNH InterOperability Lab
>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu>
>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> <https://www.iol.unh.edu/>
>>>>>>             +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>         --         *Adam Hassick*
>>>>>>         Senior Developer
>>>>>>         UNH InterOperability Lab
>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu>
>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> <https://www.iol.unh.edu/>
>>>>>>         +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>     --     *Adam Hassick*
>>>>>>     Senior Developer
>>>>>>     UNH InterOperability Lab
>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu>
>>>>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>     +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Adam Hassick*
>>>>>> Senior Developer
>>>>>> UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>> <ahassick@iol.unh.edu>
>>>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>> +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 124026 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 42569 bytes --]

Hi Andrew,

I have a couple questions about needed setup of the NICs for the ethdev
test suite.

Our MCX5s and XL710s are failing the checkup tests. The pass rate appears
to be much worse on the XL710s (40 of 73 tests failed, 3 passed
unexpectedly).

For the XL710s, I've updated the driver and NVM versions to match the
minimum supported versions in the compatibility matrix found on the DPDK
documentation. This did not change the failure rate much.
For the MCX5s, I've installed the latest LTS version of the OFED bifurcated
driver on the DUT. Is it needed on the tester?

Are there any other manual setup steps for these devices that I might be
missing?

Thanks,
Adam

On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu> wrote:

> Hi Andrew,
>
> Yes, I copied the X710 configs to set up XL710 configs. I changed the
> environment variable names from the X710 suffix to XL710 suffix in the
> script, and forgot to change them in the corresponding environment file.
> That fixed the issue.
>
> I got the checkup tests working on the XL710 now. Most of them are
> failing, which leads me to believe this is an issue with our testbed. Based
> on the DPDK documentation for i40e, the firmware and driver versions are
> much older than what DPDK 22.11 LTS and main prefer, so I'll try updating
> those.
>
> For now I'm working on getting the XL710 checkup tests passing, and will
> pick up getting the E810 configured properly next. I'll let you know if I
> run into any more issues in relation to the test engine.
>
> Thanks,
> Adam
>
> On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> On 9/5/23 18:01, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> The compilation warning issue is now resolved. Again, thank you guys for
>> fixing this for us. I can run the tests on the Mellanox CX5s again, however
>> I'm running into a couple new issues with running the prologues on the
>> Intel cards.
>>
>> When running testing on the Intel XL710s, I see this error appear in the
>> log:
>>
>> ERROR  prologue  Environment LIB  14:16:13.650
>>> Too few networks in available configuration (0) in comparison with
>>> required (1)
>>>
>>
>> This seems like a trivial configuration error, perhaps this is something
>> I need to set up in ts-rigs. I briefly searched through the examples there
>> and didn't see any mention of how to set up a network.
>> I will attach this log just in case you need more information.
>>
>>
>> Unfortunately logs are insufficient to understand it. I've pushed new tag
>> to TE v1.19.0 which add log message with TE_* environment variables.
>> Most likely something is wrong with variables which are used as
>> conditions when available networks are defined in
>> ts-conf/cs/inc.net_cfg_pci_fns.yml:
>> TE_PCI_INSTANCE_IUT_TST1
>> TE_PCI_INSTANCE_IUT_TST1a
>> TE_PCI_INSTANCE_TST1a_IUT
>> TE_PCI_INSTANCE_TST1_IUT
>> My guess it that you change naming a bit, but script like
>> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>>
>> There is a different error when running on the Intel E810s. It appears to
>> me like it starts DPDK, does some configuration inside DPDK and on the
>> device, and then fails to bring the device back up. Since this error seems
>> very non-trivial, I will also attach this log.
>>
>>
>> This one is a bit simpler. Few lines after the first ERROR in log I see
>> the following:
>> WARN  RCF  DPDK  13:06:00.144
>> ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)
>> ice_rx_queue_start(): fail to program RX queue 0
>> ice_dev_start(): fail to start Rx queue 0
>> Device with port_id=0 already stopped
>>
>> It is stdout/stderr from test agent which runs DPDK. Same logs in plain
>> format are available in ta.DPDK file.
>> I'm not an expert here, but I vaguely remember that E810 requires correct
>> firmware and DDP to be loaded.
>> There is some information in dpdk/doc/guides/nics/ice.rst.
>>
>> You can try to add --dev-args=safe-mode-support=1 command-line option
>> described there.
>>
>> Hope it helps,
>> Andrew.
>>
>>
>> Thanks,
>> Adam
>>
>> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> On 8/31/23 22:38, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> I have one additional question as well: Does the test engine support
>>> running tests on two ARMv8 test agents?
>>>
>>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>>
>>>
>>> Great. Let me know when that's fixed.
>>>
>>>
>>> Done. We also fixed a number of warnings in TE.
>>> Also we fixed root test package name to be consistent with the
>>> repository name.
>>>
>>> Support for old LTS branches was dropped some time ago, but in the
>>>> future it is definitely possible to keep it for new LTS branches. I think
>>>> 22.11 is supported, but I'm not sure about older LTS releases.
>>>>
>>>
>>> Good to know.
>>>
>>>
>>>> 2. You can add command-line option --sanity to run tests marked with
>>>> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
>>>> there is a space for terminology improvement here. We'll do it.
>>>>
>>>
>>> Done. Now it is called --checkup.
>>>
>>>
>>>> Also it takes a lot of time because of failures and tests which wait
>>>> for some timeout.
>>>>
>>>
>>> That makes sense to me. We'll use the time to complete tests on virtio
>>> or the Intel devices as a reference for how long the tests really take to
>>> complete.
>>> We will explore the possibility of periodically running the sanity tests
>>> for patches.
>>>
>>>
>>> I'll double-check and let you know how long entire TS runs on Intel
>>> X710, E810, Mellanox CX5 and virtio net. Just to ensure that time observed
>>> in your case looks the same.
>>>
>>>
>>>
>>>> The test harness can provide coverage reports based on gcov, but I'm
>>>> not sure what you mean by a "dial" to control test coverage. Provided
>>>> reports are rather for human to analyze.
>>>>
>>>
>>> The general idea is to have some kind of parameter on the test suite,
>>> which could be an integer ranging from zero to ten, that controls how many
>>> tests are run based on how important the test is.
>>>
>>> Similar to how some command line interfaces provide a verbosity level
>>> parameter (some number of "-v" arguments) to control the importance of the
>>> information in the log.
>>> The verbosity level zero only prints very important log messages, while
>>> ten prints everything.
>>>
>>> In much the same manner as above, this "dial" parameter controls what
>>> tests are run and with what parameters based on how important those tests
>>> and test parameter combinations are.
>>> Coverage Level zero tells the suite to run a very basic set of important
>>> tests, with minimal parameterization. This mode would take only ~5-10
>>> minutes to run.
>>> In contrast, Coverage Level ten includes all the edge cases, every
>>> combination of test parameters, everything the test suite can do, which
>>> takes the normal several hours to run.
>>> The values 1 - 9 are between those two extremes, allowing the user to
>>> get a gradient of test coverage in the results and to limit the running
>>> time.
>>>
>>> Then we could, for example, run the "run.sh" with a level of 2 or 3 for
>>> incoming patches that need quick results, and with a level of 10 for the
>>> less often run periodic tests performed on main or LTS branches.
>>>
>>>
>>> Understood now. Thanks a lot for the idea. We'll discuss it and come
>>> back.
>>>
>>>
>>>
>>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>>>> testing results. Unfortunately it is time consuming to fill in expectations
>>>> database since it is necessary to analyze testing results and classify if
>>>> it is a bug or just acceptable behaviour aspect.
>>>>
>>>> Bublik allows to compare results of two runs. It is useful for human,
>>>> but still not good for automation.
>>>>
>>>> I have local patch for mlx5 driver which reports Tx ring size maximum.
>>>> It makes pass rate higher. It is a problem for test harness that mlx5 does
>>>> not report limits right now.
>>>>
>>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>>>> virtio net is 99% right now and could be done 100% easily (just one thing
>>>> to fix in expectations).
>>>>
>>>> I think logs storage setup is essential for logs analysis. Of course,
>>>> you can request HTML logs when you run tests (--log-html=html) or generate
>>>> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>>>> browser, but logs storage makes it more convenient.
>>>>
>>>
>>> We are interested in setting up Bublik, potentially as an
>>> externally-facing component, once we have our process of running the test
>>> suite stabilized.
>>> Once we are able to run the test suite again, I'll see what the pass
>>> rate is on our other hardware.
>>> Good to know that it isn't an issue with our dev testbed causing the
>>> high fail rate.
>>>
>>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>>> development testbed. Although they are slightly different devices, ideally
>>> the pass rate will be identical or similar. I have yet to set up a VM pair
>>> for virtio, but we will soon.
>>>
>>> Latest version of test-environment has examples of our CGI scripts which
>>>> we use for log storage (see tools/log_server/README.md).
>>>>
>>>> Also all bits for Jenkins setup are available. See
>>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>>> ts-rigs-sample.
>>>>
>>>
>>> Jenkins integration, setting up production rig configurations, and
>>> permanent log storage will be our next steps once I am able to run the
>>> tests again.
>>> Unless there is an easy way to have meson not pass "-Werror" into GCC.
>>> Then I would be able to run the test suite.
>>>
>>>
>>> Hopefully it is resolved now.
>>>
>>> I thought a bit more about your usecase for Jenkins. I'm not 100% sure
>>> that existing pipelines are convenient for your usecase.
>>> Fill free to ask questions when you are on it.
>>>
>>> Thanks,
>>> Andrew.
>>>
>>>
>>> Thanks,
>>> Adam
>>>
>>>
>>>
>>>>
>>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> That fix seems to have resolved the issue, thanks for the quick
>>>> turnaround time on that patch.
>>>> Now that we have the RCF timeout issue resolved, there are a few other
>>>> questions and issues that we have about the tests themselves.
>>>>
>>>> 1. The test suite fails to build with a couple warnings.
>>>>
>>>> Below is the stderr log from compilation:
>>>>
>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>> ../../lib/dpdk_pmd_ts.c
>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>> ‘test_create_traffic_generator_params’:
>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>>>> no format arguments [-Werror=format-security]
>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>> |     ^~
>>>>> cc1: all warnings being treated as errors
>>>>> ninja: build stopped: subcommand failed.
>>>>> ninja: Entering directory `.'
>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>> ../../lib/dpdk_pmd_ts.c
>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>> ‘test_create_traffic_generator_params’:
>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>>>> no format arguments [-Werror=format-security]
>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>> |     ^~
>>>>> cc1: all warnings being treated as errors
>>>>>
>>>>
>>>> This error wasn't occurring last week, which was the last time I ran
>>>> the tests.
>>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester has
>>>> GCC v11.4.0 installed, if this information is helpful.
>>>>
>>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>>> to lower the test coverage and have the test suite run faster?
>>>>
>>>> For some context, we run immediate testing on incoming patches for DPDK
>>>> main and development branches, as well as periodic test runs on the main,
>>>> stable, and LTS branches.
>>>> For us to consider including this test suite as part of our immediate
>>>> testing on patches, we would have to reduce the test coverage to the most
>>>> important tests.
>>>> This is primarily to reduce the testing time to, for example, less than
>>>> 30 minutes. Testing on patches can't take too long because the lab can
>>>> receive numerous patches each day, which each require individual testing
>>>> runs.
>>>>
>>>> At what frequency we run these tests, and on what, still needs to be
>>>> discussed with the DPDK community, but it would be nice to know if the test
>>>> suite had a "dial" to control the testing coverage.
>>>>
>>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300
>>>> of ~6,600 tests passed. Is there anything we can do to diagnose these test
>>>> failures?
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>>
>>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> Hi Adam,
>>>>>
>>>>> I've pushed the fix in main branch and a new tag v1.18.1. It should
>>>>> solve the problem with IPv6 address from DNS.
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>
>>>>> Hi Adam,
>>>>>
>>>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>>>> to use IPv4?
>>>>>
>>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your
>>>>> case, I guess it is the root cause of the problem.
>>>>> Of course, it is TE problem since I see really weird code in
>>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>
>>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>>> ts-factory/test-environment repository. Please, try.
>>>>>
>>>>> It is late night fix with minimal testing and no review. I'll pass it
>>>>> through review process tomorrow and
>>>>> hopefully it will be released in one-two days.
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> We have yet to notice a distinct pattern with the failures. Sometimes,
>>>>> the RCF will start and connect without issue a few times in a row before
>>>>> failing to connect again. Once the issue begins to occur, neither rebooting
>>>>> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
>>>>> directories (suites, agents, inst) and rebooting the hosts afterward
>>>>> resolves the issue. When it begins working again seems very arbitrary to us.
>>>>>
>>>>> I do usually try to terminate the test engine with Ctrl+C, but when it
>>>>> hangs while trying to start RCF, that does not work.
>>>>>
>>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>>>> to use IPv4?
>>>>>
>>>>>  - Adam
>>>>>
>>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>
>>>>>> Done. It works fine for me without any issues.
>>>>>>
>>>>>> Have you noticed any pattern when it works or does not work?
>>>>>> May be it is a problem of not clean state after termination?
>>>>>> Does it work fine the first time after DUTs reboot?
>>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>>> terminal where you execute run.sh command.
>>>>>>  In this case it should shutdown gracefully and close all test agents
>>>>>> and engine applications.
>>>>>>
>>>>>> (I'm trying to understand why you've seen many test agent processes.
>>>>>> It should not happen.)
>>>>>>
>>>>>> Andrew.
>>>>>>
>>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>
>>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>>> This could be the source of the problem. I encountered a dependency
>>>>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>>>>> system. Since the tester is also the host having connection issues, I will
>>>>>> try downgrading that system to 20.04, and see if that changes anything.
>>>>>>
>>>>>>
>>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>>>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>>>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>>>>
>>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>
>>>>>> I did try passing in the "--vg-rcf" argument to the run.sh script of
>>>>>> the test suite after installing valgrind, but there was no additional
>>>>>> output that I saw.
>>>>>>
>>>>>>
>>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>>> (direction where you run test engine).
>>>>>>
>>>>>>
>>>>>> I will try pulling in the changes you've pushed up, and will see if
>>>>>> that fixes anything.
>>>>>>
>>>>>> Thanks,
>>>>>> Adam
>>>>>>
>>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>> Hello Adam,
>>>>>>>
>>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>
>>>>>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>>>>>> running on test engine and agents?
>>>>>>>
>>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>>>>> patch in review to fix it, however, the behaviour is different in
>>>>>>> this case, so it is unlike the same problem.
>>>>>>>
>>>>>>>
>>>>>>> I've just published a new tag which fixes known test engine side
>>>>>>> problems on Debian 12 and Fedora 38.
>>>>>>>
>>>>>>>
>>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>>> run with option --vg-rcf to check if something weird is happening.
>>>>>>>
>>>>>>> What I don't understand right now is why I see just one failed
>>>>>>> attempt
>>>>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>>>>
>>>>>>> Andrew.
>>>>>>>
>>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>
>>>>>>>  > Is there any firewall in the network or on test hosts which could
>>>>>>> block incoming TCP connection to the port 23571
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>> where you run test engine?
>>>>>>>
>>>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>>>> connection does work sometimes.
>>>>>>>
>>>>>>>  > If behaviour the same on the next try and you see that test agent
>>>>>>> is kept running, could you check using
>>>>>>>  >
>>>>>>>  > # netstat -tnlp
>>>>>>>  >
>>>>>>>  > that Test Agent is listening on the port and try to establish TCP
>>>>>>> connection from test agent using
>>>>>>>  >
>>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>  >
>>>>>>>  > and check if TCP connection could be established.
>>>>>>>
>>>>>>> I was able to replicate the same behavior again, where it hangs
>>>>>>> while RCF is trying to start.
>>>>>>> Running this command, I see this in the output:
>>>>>>>
>>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>>>  18599/ta
>>>>>>>
>>>>>>> So it seems like it is listening on the correct port.
>>>>>>> Additionally, I was able to connect to the Tester machine from our
>>>>>>> Test Engine host using telnet. It printed the PID of the process once the
>>>>>>> connection was opened.
>>>>>>>
>>>>>>> I tried running the "ta" application manually on the command line,
>>>>>>> and it didn't print anything at all.
>>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>>
>>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>     Hi Adam,
>>>>>>>
>>>>>>>      > On the tester host (which appears to be the Peer agent),
>>>>>>> there
>>>>>>>     are four processes that I see running, which look like the test
>>>>>>>     agent processes.
>>>>>>>
>>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>>
>>>>>>>     Is there any firewall in the network or on test hosts which
>>>>>>> could
>>>>>>>     block incoming TCP connection to the port 23571
>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>     where you run test engine?
>>>>>>>
>>>>>>>     If behaviour the same on the next try and you see that test
>>>>>>> agent is
>>>>>>>     kept running, could you check using
>>>>>>>
>>>>>>>     # netstat -tnlp
>>>>>>>
>>>>>>>     that Test Agent is listening on the port and try to establish
>>>>>>> TCP
>>>>>>>     connection from test agent using
>>>>>>>
>>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>
>>>>>>>     and check if TCP connection could be established.
>>>>>>>
>>>>>>>     Another idea is to login Tester under root as testing does, get
>>>>>>>     start TA command from the log and try it by hands without -n and
>>>>>>>     remove extra escaping.
>>>>>>>
>>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>
>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>
>>>>>>>     Hopefully in this case test agent directory remains in the /tmp
>>>>>>> and
>>>>>>>     you don't need to copy it as testing does.
>>>>>>>     May be output could shed some light on what's going on.
>>>>>>>
>>>>>>>     Andrew.
>>>>>>>
>>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>
>>>>>>>     Hi Andrew,
>>>>>>>
>>>>>>>     This is the output that I see in the terminal when this failure
>>>>>>>     occurs, after the test agent binaries build and the test engine
>>>>>>>     starts:
>>>>>>>
>>>>>>>     Platform default build - pass
>>>>>>>     Simple RCF consistency check succeeded
>>>>>>>     --->>> Starting Logger...done
>>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>
>>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>>
>>>>>>>     On the tester host (which appears to be the Peer agent), there
>>>>>>> are
>>>>>>>     four processes that I see running, which look like the test
>>>>>>> agent
>>>>>>>     processes.
>>>>>>>
>>>>>>>     ta.Peer is an empty file. I've attached the log.txt from this
>>>>>>> run.
>>>>>>>
>>>>>>>      - Adam
>>>>>>>
>>>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>         Hi Adam,
>>>>>>>
>>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>         says that copy_timeout is in seconds and implementation in
>>>>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>>>>         Theoretically select() could be interrupted by signal, but I
>>>>>>>         think it is unlikely here.
>>>>>>>
>>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>>         does it happen when tests are in progress, e.g. in the
>>>>>>> middle
>>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>>>         likely either host with test agent dies or test agent itself
>>>>>>>         crashes. It would be easier for me if classify it if you
>>>>>>> share
>>>>>>>         text log (log.txt, full or just corresponding fragment with
>>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>>         depending on which agent has problems could shed some light.
>>>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>>>
>>>>>>>         Andrew.
>>>>>>>
>>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>
>>>>>>>         Hi Andrew,
>>>>>>>
>>>>>>>         I've set up a test rig repository here, and have created
>>>>>>>         configurations for our development testbed based off of the
>>>>>>>         examples.
>>>>>>>         We've been able to get the test suite to run manually on
>>>>>>>         Mellanox CX5 devices once.
>>>>>>>         However, we are running into an issue where, when RCF
>>>>>>> starts,
>>>>>>>         the RCF connection times out very frequently. We aren't sure
>>>>>>>         why this is the case.
>>>>>>>         It works sometimes, but most of the time when we try to run
>>>>>>>         the test engine, it encounters this issue.
>>>>>>>         I've tried changing the RCF port by setting
>>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>>>>         machines. Neither seems to fix the issue.
>>>>>>>
>>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>>         before I try to run the test suite.
>>>>>>>         I assume the unit for this variable is seconds?
>>>>>>>
>>>>>>>         Thanks,
>>>>>>>         Adam
>>>>>>>
>>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>>
>>>>>>>             Hi Andrew,
>>>>>>>
>>>>>>>             Thanks, I've cloned the example repository and will
>>>>>>> start
>>>>>>>             setting up a configuration for our development testbed
>>>>>>>             today. I'll let you know if I run into any difficulties
>>>>>>>             or have any questions.
>>>>>>>
>>>>>>>              - Adam
>>>>>>>
>>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                 Hi Adam,
>>>>>>>
>>>>>>>                 I've published
>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>                 Hopefully it will help to define your test rigs and
>>>>>>>                 successfully run some tests manually. Feel free to
>>>>>>>                 ask any questions and I'll answer here and try to
>>>>>>>                 update documentation.
>>>>>>>
>>>>>>>                 Meanwhile I'll prepare missing bits for steps (2)
>>>>>>> and
>>>>>>>                 (3).
>>>>>>>                 Hopefully everything is in place for step (4), but
>>>>>>> we
>>>>>>>                 need to make steps (2) and (3) first.
>>>>>>>
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>
>>>>>>>                 Hi Adam,
>>>>>>>
>>>>>>>                 > I've conferred with the rest of the team, and we
>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>                 option B.
>>>>>>>
>>>>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>>>>                 almost ready right now, but I need to double-check
>>>>>>>                 it before publishing.
>>>>>>>
>>>>>>>                 Regards,
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>
>>>>>>>                 Hi Andrew,
>>>>>>>
>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>                 conversation. Others in the community might find
>>>>>>>                 this conversation valuable.
>>>>>>>
>>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>>                 Jenkins integration will be very useful for us, as
>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>                 option B.
>>>>>>>                 If you would like to know anything about our
>>>>>>>                 testbeds that would help you with creating an
>>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>>                 questions you have.
>>>>>>>
>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>                 hardware testing on, with differing network
>>>>>>>                 hardware and CPU architecture. I figured this might
>>>>>>>                 be an important detail.
>>>>>>>
>>>>>>>                 Thanks,
>>>>>>>                 Adam
>>>>>>>
>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                     Greatings Adam,
>>>>>>>
>>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>>                     it up.
>>>>>>>
>>>>>>>                     As I understand the final goal is to run it on
>>>>>>>                     regular basis. So, we need to make it properly
>>>>>>>                     from the very beginning.
>>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>>
>>>>>>>                     1. Create site-specific repository (we call it
>>>>>>>                     ts-rigs) which contains information about test
>>>>>>>                     rigs and other site-specific information like
>>>>>>>                     where to send mails, where to store logs etc.
>>>>>>>                     It is required for manual execution as well,
>>>>>>>                     since test rigs description is essential. I'll
>>>>>>>                     return to the topic below.
>>>>>>>
>>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>>                     server with few CGI scripts which help a lot to
>>>>>>>                     save disk space.
>>>>>>>
>>>>>>>                     3. Setup Bublik web application which provides
>>>>>>>                     web interface to view testing results. Same as
>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>> <https://ts-factory.io/bublik>
>>>>>>>
>>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>>                     save logs in log storage (2) and import it to
>>>>>>>                     bublik (3).
>>>>>>>
>>>>>>>                     Last few month we spent on our homework to make
>>>>>>>                     it simpler to bring up automated execution
>>>>>>>                     using Jenkins -
>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>                     available tomorrow.
>>>>>>>
>>>>>>>                     Let's return to the step (1).
>>>>>>>
>>>>>>>                     Unfortunately there is no publicly available
>>>>>>>                     example of the ts-rigs repository since
>>>>>>>                     sensitive site-specific information is located
>>>>>>>                     there. But I'm ready to help you to create it
>>>>>>>                     for UNH. I see two options here:
>>>>>>>
>>>>>>>                     (A) I'll ask questions and based on your
>>>>>>>                     answers will create the first draft with my
>>>>>>>                     comments.
>>>>>>>
>>>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>>>                     on it.
>>>>>>>
>>>>>>>                     Of course, I'll help to debug and finally bring
>>>>>>>                     it up in any case.
>>>>>>>
>>>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>>>                     a bit more generic and will help other
>>>>>>>                     potential users to bring it up.
>>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>>                     What do you think?
>>>>>>>
>>>>>>>                     Thanks,
>>>>>>>                     Andrew.
>>>>>>>
>>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>
>>>>>>>                     Greetings Adam,
>>>>>>>
>>>>>>>
>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>                     would be happy to help
>>>>>>>
>>>>>>>                     Thanks,
>>>>>>>                     Konstantin
>>>>>>>
>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>>
>>>>>>>                     
>>>>>>>                     Greetings Konstantin,
>>>>>>>
>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>
>>>>>>>                     I have some questions about how to set the
>>>>>>>                     test suite arguments.
>>>>>>>
>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>                     run, however the prelude of the test suite
>>>>>>>                     fails to run.
>>>>>>>
>>>>>>>
>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>
>>>>>>>                     The documentation mentions that there are
>>>>>>>                     several test parameters for the test suite,
>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>                     to run many of the tests.
>>>>>>>
>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>                     are instructions on how to create new
>>>>>>>                     parameters for test suites in the Tester
>>>>>>>                     configuration, but there is nothing in the
>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>                     set the arguments for the parameters when
>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>                     config, or if I should be setting these in
>>>>>>>                     some other way.
>>>>>>>
>>>>>>>                     How should these values be set?
>>>>>>>
>>>>>>>                     I'm also not sure what environment
>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>                     which are optional.
>>>>>>>
>>>>>>>                     Regards,
>>>>>>>                     Adam
>>>>>>>
>>>>>>>                     --                     *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                 --                 *Adam Hassick*
>>>>>>>                 Senior Developer
>>>>>>>                 UNH InterOperability Lab
>>>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>                 +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             --             *Adam Hassick*
>>>>>>>             Senior Developer
>>>>>>>             UNH InterOperability Lab
>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>             +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         --         *Adam Hassick*
>>>>>>>         Senior Developer
>>>>>>>         UNH InterOperability Lab
>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>         +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>     --     *Adam Hassick*
>>>>>>>     Senior Developer
>>>>>>>     UNH InterOperability Lab
>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>     +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Adam Hassick*
>>>>>>> Senior Developer
>>>>>>> UNH InterOperability Lab
>>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>> <ahassick@iol.unh.edu>
>>>>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>> +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Adam Hassick*
>>>>>> Senior Developer
>>>>>> UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu
>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>


-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 126174 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 79294 bytes --]

Hi Adam,

I've pushed new TE tag v1.20.0 which supported a new command-line option 
--tester-dial=NUM where NUM is from 0 to 100. it allows to choose 
percentage of tests to run. If you want stable set, you should pass 
--tester-random-seed=0 (or other integer). It is the first sketch and we 
have plans to improve it, but feedback would be welcome.

 > Is it needed on the tester?

It is hard to say if it is strictly required for simple tests. However, 
it is better to update Tester as well, since performance tests run DPDK 
on Tester as well.

 > Are there any other manual setup steps for these devices that I might 
be missing?

I don't remember anything else.

I think it is better to get down to details and take a look at logs. I'm 
ready to help with it and explain what's happening there. May be it will 
help to understand if it is a problem with setup/configuration.

Text logs are not very convenient. Ideally logs should be imported to 
bublik, however, manual runs do not provide all required artifacts right 
now (Jenkins jobs generate all required artifacts).
Other option is 'tmp_raw_log' file (should be packed to make it smaller) 
which could be converted to various log formats.
Would it be OK for you if I import your logs to bublik at ts-factory.io? 
Or is it a problem that it is publicly available?
Would it help if we add authentication and access control there?

Andrew.

On 9/8/23 17:57, Adam Hassick wrote:
> Hi Andrew,
>
> I have a couple questions about needed setup of the NICs for the 
> ethdev test suite.
>
> Our MCX5s and XL710s are failing the checkup tests. The pass rate 
> appears to be much worse on the XL710s (40 of 73 tests failed, 3 
> passed unexpectedly).
>
> For the XL710s, I've updated the driver and NVM versions to match the 
> minimum supported versions in the compatibility matrix found on the 
> DPDK documentation. This did not change the failure rate much.
> For the MCX5s, I've installed the latest LTS version of the OFED 
> bifurcated driver on the DUT. Is it needed on the tester?
>
> Are there any other manual setup steps for these devices that I might 
> be missing?
>
> Thanks,
> Adam
>
> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu> wrote:
>
>     Hi Andrew,
>
>     Yes, I copied the X710 configs to set up XL710 configs. I changed
>     the environment variable names from the X710 suffix to XL710
>     suffix in the script, and forgot to change them in the
>     corresponding environment file.
>     That fixed the issue.
>
>     I got the checkup tests working on the XL710 now. Most of them are
>     failing, which leads me to believe this is an issue with our
>     testbed. Based on the DPDK documentation for i40e, the firmware
>     and driver versions are much older than what DPDK 22.11 LTS and
>     main prefer, so I'll try updating those.
>
>     For now I'm working on getting the XL710 checkup tests passing,
>     and will pick up getting the E810 configured properly next. I'll
>     let you know if I run into any more issues in relation to the test
>     engine.
>
>     Thanks,
>     Adam
>
>     On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko
>     <andrew.rybchenko@oktetlabs.ru> wrote:
>
>         Hi Adam,
>
>         On 9/5/23 18:01, Adam Hassick wrote:
>>         Hi Andrew,
>>
>>         The compilation warning issue is now resolved. Again, thank
>>         you guys for fixing this for us. I can run the tests on the
>>         Mellanox CX5s again, however I'm running into a couple new
>>         issues with running the prologues on the Intel cards.
>>
>>         When running testing on the Intel XL710s, I see this error
>>         appear in the log:
>>
>>             ERROR  prologue  Environment LIB  14:16:13.650
>>             Too few networks in available configuration (0) in
>>             comparison with required (1)
>>
>>
>>         This seems like a trivial configuration error, perhaps this
>>         is something I need to set up in ts-rigs. I briefly searched
>>         through the examples there and didn't see any mention of how
>>         to set up a network.
>>         I will attach this log just in case you need more information.
>
>         Unfortunately logs are insufficient to understand it. I've
>         pushed new tag to TE v1.19.0 which add log message with TE_*
>         environment variables.
>         Most likely something is wrong with variables which are used
>         as conditions when available networks are defined in
>         ts-conf/cs/inc.net_cfg_pci_fns.yml:
>         TE_PCI_INSTANCE_IUT_TST1
>         TE_PCI_INSTANCE_IUT_TST1a
>         TE_PCI_INSTANCE_TST1a_IUT
>         TE_PCI_INSTANCE_TST1_IUT
>         My guess it that you change naming a bit, but script like
>         ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>
>>         There is a different error when running on the Intel E810s.
>>         It appears to me like it starts DPDK, does some configuration
>>         inside DPDK and on the device, and then fails to bring the
>>         device back up. Since this error seems very non-trivial, I
>>         will also attach this log.
>
>         This one is a bit simpler. Few lines after the first ERROR in
>         log I see the following:
>         WARN  RCF  DPDK  13:06:00.144
>         ice_program_hw_rx_queue(): currently package doesn't support
>         RXDID (22)
>         ice_rx_queue_start(): fail to program RX queue 0
>         ice_dev_start(): fail to start Rx queue 0
>         Device with port_id=0 already stopped
>
>         It is stdout/stderr from test agent which runs DPDK. Same logs
>         in plain format are available in ta.DPDK file.
>         I'm not an expert here, but I vaguely remember that E810
>         requires correct firmware and DDP to be loaded.
>         There is some information in dpdk/doc/guides/nics/ice.rst.
>
>         You can try to add --dev-args=safe-mode-support=1 command-line
>         option described there.
>
>         Hope it helps,
>         Andrew.
>
>>
>>         Thanks,
>>         Adam
>>
>>         On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko
>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>             Hi Adam,
>>
>>             On 8/31/23 22:38, Adam Hassick wrote:
>>>             Hi Andrew,
>>>
>>>             I have one additional question as well: Does the test
>>>             engine support running tests on two ARMv8 test agents?
>>>
>>>                 1. We'll sort out warnings this week. Thanks for
>>>                 heads up.
>>>
>>>
>>>             Great. Let me know when that's fixed.
>>
>>             Done. We also fixed a number of warnings in TE.
>>             Also we fixed root test package name to be consistent
>>             with the repository name.
>>
>>>                 Support for old LTS branches was dropped some time
>>>                 ago, but in the future it is definitely possible to
>>>                 keep it for new LTS branches. I think 22.11 is
>>>                 supported, but I'm not sure about older LTS releases.
>>>
>>>
>>>             Good to know.
>>>
>>>                 2. You can add command-line option --sanity to run
>>>                 tests marked with TEST_HARNESS_SANITY requirement
>>>                 (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>                 TEST_HARNESS_SANITY dpdk-ethdev-ts to see which
>>>                 tests are marked). Yes, there is a space for
>>>                 terminology improvement here. We'll do it.
>>>
>>
>>             Done. Now it is called --checkup.
>>
>>>
>>>                 Also it takes a lot of time because of failures and
>>>                 tests which wait for some timeout.
>>>
>>>
>>>             That makes sense to me. We'll use the time to complete
>>>             tests on virtio or the Intel devices as a reference for
>>>             how long the tests really take to complete.
>>>             We will explore the possibility of periodically running
>>>             the sanity tests for patches.
>>
>>             I'll double-check and let you know how long entire TS
>>             runs on Intel X710, E810, Mellanox CX5 and virtio net.
>>             Just to ensure that time observed in your case looks the
>>             same.
>>
>>>
>>>                 The test harness can provide coverage reports based
>>>                 on gcov, but I'm not sure what you mean by a "dial"
>>>                 to control test coverage. Provided reports are
>>>                 rather for human to analyze.
>>>
>>>
>>>             The general idea is to have some kind of parameter on
>>>             the test suite, which could be an integer ranging from
>>>             zero to ten, that controls how many tests are run based
>>>             on how important the test is.
>>>
>>>             Similar to how some command line interfaces provide a
>>>             verbosity level parameter (some number of "-v"
>>>             arguments) to control the importance of the information
>>>             in the log.
>>>             The verbosity level zero only prints very important log
>>>             messages, while ten prints everything.
>>>
>>>             In much the same manner as above, this "dial" parameter
>>>             controls what tests are run and with what parameters
>>>             based on how important those tests and test parameter
>>>             combinations are.
>>>             Coverage Level zero tells the suite to run a very basic
>>>             set of important tests, with minimal parameterization.
>>>             This mode would take only ~5-10 minutes to run.
>>>             In contrast, Coverage Level ten includes all the edge
>>>             cases, every combination of test parameters, everything
>>>             the test suite can do, which takes the normal several
>>>             hours to run.
>>>             The values 1 - 9 are between those two extremes,
>>>             allowing the user to get a gradient of test coverage in
>>>             the results and to limit the running time.
>>>
>>>             Then we could, for example, run the "run.sh" with a
>>>             level of 2 or 3 for incoming patches that need quick
>>>             results, and with a level of 10 for the less often run
>>>             periodic tests performed on main or LTS branches.
>>
>>             Understood now. Thanks a lot for the idea. We'll discuss
>>             it and come back.
>>
>>>                 3. Yes, really many tests on Mellanox CX5 NICs
>>>                 report unexpected testing results. Unfortunately it
>>>                 is time consuming to fill in expectations database
>>>                 since it is necessary to analyze testing results and
>>>                 classify if it is a bug or just acceptable behaviour
>>>                 aspect.
>>>
>>>                 Bublik allows to compare results of two runs. It is
>>>                 useful for human, but still not good for automation.
>>>
>>>                 I have local patch for mlx5 driver which reports Tx
>>>                 ring size maximum. It makes pass rate higher. It is
>>>                 a problem for test harness that mlx5 does not report
>>>                 limits right now.
>>>
>>>                 Pass rate on Intel X710 is about 92% on my test rig.
>>>                 Pass rate on virtio net is 99% right now and could
>>>                 be done 100% easily (just one thing to fix in
>>>                 expectations).
>>>
>>>                 I think logs storage setup is essential for logs
>>>                 analysis. Of course, you can request HTML logs when
>>>                 you run tests (--log-html=html) or generate after
>>>                 run using dpdk-ethdev-ts/scripts/html-log.sh and
>>>                 open index.html in a browser, but logs storage makes
>>>                 it more convenient.
>>>
>>>
>>>             We are interested in setting up Bublik, potentially as
>>>             an externally-facing component, once we have our process
>>>             of running the test suite stabilized.
>>>             Once we are able to run the test suite again, I'll see
>>>             what the pass rate is on our other hardware.
>>>             Good to know that it isn't an issue with our dev testbed
>>>             causing the high fail rate.
>>>
>>>             For Intel hardware, we have an XL710 and an Intel E810-C
>>>             in our development testbed. Although they are slightly
>>>             different devices, ideally the pass rate will be
>>>             identical or similar. I have yet to set up a VM pair for
>>>             virtio, but we will soon.
>>>
>>>                 Latest version of test-environment has examples of
>>>                 our CGI scripts which we use for log storage (see
>>>                 tools/log_server/README.md).
>>>
>>>                 Also all bits for Jenkins setup are available. See
>>>                 dpdk-ethdev-ts/jenkins/README.md and examples of
>>>                 jenkins files in ts-rigs-sample.
>>>
>>>
>>>             Jenkins integration, setting up production rig
>>>             configurations, and permanent log storage will be our
>>>             next steps once I am able to run the tests again.
>>>             Unless there is an easy way to have meson not pass
>>>             "-Werror" into GCC. Then I would be able to run the test
>>>             suite.
>>
>>             Hopefully it is resolved now.
>>
>>             I thought a bit more about your usecase for Jenkins. I'm
>>             not 100% sure that existing pipelines are convenient for
>>             your usecase.
>>             Fill free to ask questions when you are on it.
>>
>>             Thanks,
>>             Andrew.
>>
>>>
>>>             Thanks,
>>>             Adam
>>>
>>>
>>>                 On 8/29/23 17:02, Adam Hassick wrote:
>>>>                 Hi Andrew,
>>>>
>>>>                 That fix seems to have resolved the issue, thanks
>>>>                 for the quick turnaround time on that patch.
>>>>                 Now that we have the RCF timeout issue resolved,
>>>>                 there are a few other questions and issues that we
>>>>                 have about the tests themselves.
>>>>
>>>>                 1. The test suite fails to build with a couple
>>>>                 warnings.
>>>>
>>>>                 Below is the stderr log from compilation:
>>>>
>>>>                     FAILED:
>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>                     -I../../lib
>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>                     -fdiagnostics-color=always -pipe
>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>                     -fPIC -MD -MQ
>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>                     -MF
>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>                     -o
>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>                     ‘test_create_traffic_generator_params’:
>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>                     not a string literal and no format arguments
>>>>                     [-Werror=format-security]
>>>>                     5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>                     |     ^~
>>>>                     cc1: all warnings being treated as errors
>>>>                     ninja: build stopped: subcommand failed.
>>>>                     ninja: Entering directory `.'
>>>>                     FAILED:
>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>                     -I../../lib
>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>                     -fdiagnostics-color=always -pipe
>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>                     -fPIC -MD -MQ
>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>                     -MF
>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>                     -o
>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>                     ‘test_create_traffic_generator_params’:
>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>                     not a string literal and no format arguments
>>>>                     [-Werror=format-security]
>>>>                     5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>                     |     ^~
>>>>                     cc1: all warnings being treated as errors
>>>>
>>>>
>>>>                 This error wasn't occurring last week, which was
>>>>                 the last time I ran the tests.
>>>>                 The TE host and the DUT have GCC v9.4.0 installed,
>>>>                 and the tester has GCC v11.4.0 installed, if this
>>>>                 information is helpful.
>>>>
>>>>                 2. On the Mellanox CX5s, there are over 6,000 tests
>>>>                 run, which collectively take around 9 hours. Is it
>>>>                 possible, and would it make sense, to lower the
>>>>                 test coverage and have the test suite run faster?
>>>>
>>>>                 For some context, we run immediate testing on
>>>>                 incoming patches for DPDK main and development
>>>>                 branches, as well as periodic test runs on the
>>>>                 main, stable, and LTS branches.
>>>>                 For us to consider including this test suite as
>>>>                 part of our immediate testing on patches, we would
>>>>                 have to reduce the test coverage to the most
>>>>                 important tests.
>>>>                 This is primarily to reduce the testing time to,
>>>>                 for example, less than 30 minutes. Testing on
>>>>                 patches can't take too long because the lab can
>>>>                 receive numerous patches each day, which each
>>>>                 require individual testing runs.
>>>>
>>>>                 At what frequency we run these tests, and on what,
>>>>                 still needs to be discussed with the DPDK
>>>>                 community, but it would be nice to know if the test
>>>>                 suite had a "dial" to control the testing coverage.
>>>>
>>>>                 3. We see a lot of test failures on our Mellanox
>>>>                 CX5 NICs. Around 2,300 of ~6,600 tests passed. Is
>>>>                 there anything we can do to diagnose these test
>>>>                 failures?
>>>>
>>>>                 Thanks,
>>>>                 Adam
>>>>
>>>>
>>>>                 On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko
>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>                     Hi Adam,
>>>>
>>>>                     I've pushed the fix in main branch and a new
>>>>                     tag v1.18.1. It should solve the problem with
>>>>                     IPv6 address from DNS.
>>>>
>>>>                     Andrew.
>>>>
>>>>                     On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>                     Hi Adam,
>>>>>
>>>>>                     > Does the test engine prefer to use IPv6 over
>>>>>                     IPv4 for initiating the RCF connection to the
>>>>>                     test bed hosts? And if so, is there a way to
>>>>>                     force it to use IPv4?
>>>>>
>>>>>                     Brilliant idea. If DNS returns both IPv4 and
>>>>>                     IPv6 addresses in your case, I guess it is the
>>>>>                     root cause of the problem.
>>>>>                     Of course, it is TE problem since I see really
>>>>>                     weird code in
>>>>>                     lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>
>>>>>                     I've pushed fix to the branch
>>>>>                     user/arybchik/fix_ipv4_only in
>>>>>                     ts-factory/test-environment repository.
>>>>>                     Please, try.
>>>>>
>>>>>                     It is late night fix with minimal testing and
>>>>>                     no review. I'll pass it through review process
>>>>>                     tomorrow and
>>>>>                     hopefully it will be released in one-two days.
>>>>>
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>                     Hi Andrew,
>>>>>>
>>>>>>                     We have yet to notice a distinct pattern with
>>>>>>                     the failures. Sometimes, the RCF will start
>>>>>>                     and connect without issue a few times in a
>>>>>>                     row before failing to connect again. Once the
>>>>>>                     issue begins to occur, neither rebooting all
>>>>>>                     of the hosts (test engine VM, tester, IUT) or
>>>>>>                     deleting all of the build directories
>>>>>>                     (suites, agents, inst) and rebooting the
>>>>>>                     hosts afterward resolves the issue. When it
>>>>>>                     begins working again seems very arbitrary to us.
>>>>>>
>>>>>>                     I do usually try to terminate the test engine
>>>>>>                     with Ctrl+C, but when it hangs while trying
>>>>>>                     to start RCF, that does not work.
>>>>>>
>>>>>>                     Does the test engine prefer to use IPv6 over
>>>>>>                     IPv4 for initiating the RCF connection to the
>>>>>>                     test bed hosts? And if so, is there a way to
>>>>>>                     force it to use IPv4?
>>>>>>
>>>>>>                      - Adam
>>>>>>
>>>>>>                     On Fri, Aug 25, 2023 at 1:35 PM Andrew
>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                         > I'll double-check test engine on Ubuntu
>>>>>>                         20.04 and Ubuntu 22.04.
>>>>>>
>>>>>>                         Done. It works fine for me without any
>>>>>>                         issues.
>>>>>>
>>>>>>                         Have you noticed any pattern when it
>>>>>>                         works or does not work?
>>>>>>                         May be it is a problem of not clean state
>>>>>>                         after termination?
>>>>>>                         Does it work fine the first time after
>>>>>>                         DUTs reboot?
>>>>>>                         How do you terminate testing? It should
>>>>>>                         be done using Ctrl+C in terminal where
>>>>>>                         you execute run.sh command.
>>>>>>                          In this case it should shutdown
>>>>>>                         gracefully and close all test agents and
>>>>>>                         engine applications.
>>>>>>
>>>>>>                         (I'm trying to understand why you've seen
>>>>>>                         many test agent processes. It should not
>>>>>>                         happen.)
>>>>>>
>>>>>>                         Andrew.
>>>>>>
>>>>>>                         On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>                         On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>                         Hi Andrew,
>>>>>>>>
>>>>>>>>                         Two of our systems (the Test Engine
>>>>>>>>                         runner and the DUT host) are running
>>>>>>>>                         Ubuntu 20.04 LTS, however this morning
>>>>>>>>                         I noticed that the tester system (the
>>>>>>>>                         one having issues) is running Ubuntu
>>>>>>>>                         22.04 LTS.
>>>>>>>>                         This could be the source of the
>>>>>>>>                         problem. I encountered a dependency
>>>>>>>>                         issue trying to run the Test Engine on
>>>>>>>>                         22.04 LTS, so I downgraded the system.
>>>>>>>>                         Since the tester is also the host
>>>>>>>>                         having connection issues, I will try
>>>>>>>>                         downgrading that system to 20.04, and
>>>>>>>>                         see if that changes anything.
>>>>>>>
>>>>>>>                         Unlikely, but who knows. We run tests
>>>>>>>                         (DUTs) on Ubuntu 20.04, Ubuntu 22.04,
>>>>>>>                         Ubuntu 22.10, Ubuntu 23.04, Debian 11
>>>>>>>                         and Fedora 38 every night.
>>>>>>>                         Right now Debian 11 is used for test
>>>>>>>                         engine in nightly regressions.
>>>>>>>
>>>>>>>                         I'll double-check test engine on Ubuntu
>>>>>>>                         20.04 and Ubuntu 22.04.
>>>>>>>
>>>>>>>>                         I did try passing in the "--vg-rcf"
>>>>>>>>                         argument to the run.sh script of the
>>>>>>>>                         test suite after installing valgrind,
>>>>>>>>                         but there was no additional output that
>>>>>>>>                         I saw.
>>>>>>>
>>>>>>>                         Sorry, I should valgrind output should
>>>>>>>                         be in valgrind.te_rcf (direction where
>>>>>>>                         you run test engine).
>>>>>>>
>>>>>>>>
>>>>>>>>                         I will try pulling in the changes
>>>>>>>>                         you've pushed up, and will see if that
>>>>>>>>                         fixes anything.
>>>>>>>>
>>>>>>>>                         Thanks,
>>>>>>>>                         Adam
>>>>>>>>
>>>>>>>>                         On Fri, Aug 25, 2023 at 9:57 AM Andrew
>>>>>>>>                         Rybchenko
>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>                             Hello Adam,
>>>>>>>>
>>>>>>>>                             On 8/24/23 23:54, Andrew Rybchenko
>>>>>>>>                             wrote:
>>>>>>>>>                             I'd like to try to repeat the
>>>>>>>>>                             problem locally. Which Linux
>>>>>>>>>                             distro is running on test engine
>>>>>>>>>                             and agents?
>>>>>>>>>
>>>>>>>>>                             In fact I know one problem with
>>>>>>>>>                             Debian 12 and Fedora 38 and we have
>>>>>>>>>                             patch in review to fix it,
>>>>>>>>>                             however, the behaviour is
>>>>>>>>>                             different in
>>>>>>>>>                             this case, so it is unlike the
>>>>>>>>>                             same problem.
>>>>>>>>
>>>>>>>>                             I've just published a new tag which
>>>>>>>>                             fixes known test engine side
>>>>>>>>                             problems on Debian 12 and Fedora 38.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>                             One more idea is to install
>>>>>>>>>                             valgrind on the test engine host and
>>>>>>>>>                             run with option --vg-rcf to check
>>>>>>>>>                             if something weird is happening.
>>>>>>>>>
>>>>>>>>>                             What I don't understand right now
>>>>>>>>>                             is why I see just one failed attempt
>>>>>>>>>                             to connect in your log.txt and
>>>>>>>>>                             then Logger shutdown after 9 minutes.
>>>>>>>>>
>>>>>>>>>                             Andrew.
>>>>>>>>>
>>>>>>>>>                             On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>>>                              > Is there any firewall in the
>>>>>>>>>>                             network or on test hosts which
>>>>>>>>>>                             could block incoming TCP
>>>>>>>>>>                             connection to the port 23571
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             from the host where you run test
>>>>>>>>>>                             engine?
>>>>>>>>>>
>>>>>>>>>>                             Our test engine host and the
>>>>>>>>>>                             testbed are on the same subnet.
>>>>>>>>>>                             The connection does work sometimes.
>>>>>>>>>>
>>>>>>>>>>                              > If behaviour the same on the
>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>                             agent is kept running, could you
>>>>>>>>>>                             check using
>>>>>>>>>>                              >
>>>>>>>>>>                              > # netstat -tnlp
>>>>>>>>>>                              >
>>>>>>>>>>                              > that Test Agent is listening
>>>>>>>>>>                             on the port and try to establish
>>>>>>>>>>                             TCP connection from test agent using
>>>>>>>>>>                              >
>>>>>>>>>>                              > $ telnet
>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             23571
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>
>>>>>>>>>>                              >
>>>>>>>>>>                              > and check if TCP connection
>>>>>>>>>>                             could be established.
>>>>>>>>>>
>>>>>>>>>>                             I was able to replicate the same
>>>>>>>>>>                             behavior again, where it hangs
>>>>>>>>>>                             while RCF is trying to start.
>>>>>>>>>>                             Running this command, I see this
>>>>>>>>>>                             in the output:
>>>>>>>>>>
>>>>>>>>>>                             tcp        0    0 0.0.0.0:23571
>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>                             <http://0.0.0.0:23571> 0.0.0.0:*
>>>>>>>>>>                             LISTEN  18599/ta
>>>>>>>>>>
>>>>>>>>>>                             So it seems like it is listening
>>>>>>>>>>                             on the correct port.
>>>>>>>>>>                             Additionally, I was able to
>>>>>>>>>>                             connect to the Tester machine
>>>>>>>>>>                             from our Test Engine host using
>>>>>>>>>>                             telnet. It printed the PID of the
>>>>>>>>>>                             process once the connection was
>>>>>>>>>>                             opened.
>>>>>>>>>>
>>>>>>>>>>                             I tried running the "ta"
>>>>>>>>>>                             application manually on the
>>>>>>>>>>                             command line, and it didn't print
>>>>>>>>>>                             anything at all.
>>>>>>>>>>                             Maybe the issue is something on
>>>>>>>>>>                             the Test Engine side.
>>>>>>>>>>
>>>>>>>>>>                             On Thu, Aug 24, 2023 at 2:35 PM
>>>>>>>>>>                             Andrew Rybchenko
>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>                             wrote:
>>>>>>>>>>
>>>>>>>>>>                                 Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                                  > On the tester host (which
>>>>>>>>>>                             appears to be the Peer agent), there
>>>>>>>>>>                                 are four processes that I see
>>>>>>>>>>                             running, which look like the test
>>>>>>>>>>                                 agent processes.
>>>>>>>>>>
>>>>>>>>>>                                 Before the next try I'd
>>>>>>>>>>                             recommend to kill these processes.
>>>>>>>>>>
>>>>>>>>>>                                 Is there any firewall in the
>>>>>>>>>>                             network or on test hosts which could
>>>>>>>>>>                                 block incoming TCP connection
>>>>>>>>>>                             to the port 23571
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             from the host
>>>>>>>>>>                                 where you run test engine?
>>>>>>>>>>
>>>>>>>>>>                                 If behaviour the same on the
>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>                             agent is
>>>>>>>>>>                                 kept running, could you check
>>>>>>>>>>                             using
>>>>>>>>>>
>>>>>>>>>>                                 # netstat -tnlp
>>>>>>>>>>
>>>>>>>>>>                                 that Test Agent is listening
>>>>>>>>>>                             on the port and try to establish TCP
>>>>>>>>>>                                 connection from test agent using
>>>>>>>>>>
>>>>>>>>>>                                 $ telnet
>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             23571
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                 and check if TCP connection
>>>>>>>>>>                             could be established.
>>>>>>>>>>
>>>>>>>>>>                                 Another idea is to login
>>>>>>>>>>                             Tester under root as testing
>>>>>>>>>>                             does, get
>>>>>>>>>>                                 start TA command from the log
>>>>>>>>>>                             and try it by hands without -n and
>>>>>>>>>>                                 remove extra escaping.
>>>>>>>>>>
>>>>>>>>>>                                 # sudo
>>>>>>>>>>                             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>
>>>>>>>>>>                             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>                             /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>                             Peer 23571
>>>>>>>>>>                             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>
>>>>>>>>>>                                 Hopefully in this case test
>>>>>>>>>>                             agent directory remains in the
>>>>>>>>>>                             /tmp and
>>>>>>>>>>                                 you don't need to copy it as
>>>>>>>>>>                             testing does.
>>>>>>>>>>                                 May be output could shed some
>>>>>>>>>>                             light on what's going on.
>>>>>>>>>>
>>>>>>>>>>                                 Andrew.
>>>>>>>>>>
>>>>>>>>>>                                 On 8/24/23 17:30, Adam
>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                                 This is the output that I
>>>>>>>>>>>                             see in the terminal when this
>>>>>>>>>>>                             failure
>>>>>>>>>>>                                 occurs, after the test agent
>>>>>>>>>>>                             binaries build and the test engine
>>>>>>>>>>>                                 starts:
>>>>>>>>>>>
>>>>>>>>>>>                                 Platform default build - pass
>>>>>>>>>>>                                 Simple RCF consistency check
>>>>>>>>>>>                             succeeded
>>>>>>>>>>>                             --->>> Starting Logger...done
>>>>>>>>>>>                             --->>> Starting
>>>>>>>>>>>                             RCF...rcf_net_engine_connect():
>>>>>>>>>>>                             Connection timed
>>>>>>>>>>>                                 out
>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                 Then, it hangs here until I
>>>>>>>>>>>                             kill the "te_rcf" and "te_tee"
>>>>>>>>>>>                                 processes. I let it hang for
>>>>>>>>>>>                             around 9 minutes.
>>>>>>>>>>>
>>>>>>>>>>>                                 On the tester host (which
>>>>>>>>>>>                             appears to be the Peer agent),
>>>>>>>>>>>                             there are
>>>>>>>>>>>                                 four processes that I see
>>>>>>>>>>>                             running, which look like the
>>>>>>>>>>>                             test agent
>>>>>>>>>>>                                 processes.
>>>>>>>>>>>
>>>>>>>>>>>                                 ta.Peer is an empty file.
>>>>>>>>>>>                             I've attached the log.txt from
>>>>>>>>>>>                             this run.
>>>>>>>>>>>
>>>>>>>>>>>                                  - Adam
>>>>>>>>>>>
>>>>>>>>>>>                                 On Thu, Aug 24, 2023 at
>>>>>>>>>>>                             4:22 AM Andrew Rybchenko
>>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>                             wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                     Yes, TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>                             is in seconds. I've double-checked
>>>>>>>>>>>                                     that it goes to
>>>>>>>>>>>                             'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>>>>>                             Description in in
>>>>>>>>>>>                             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>
>>>>>>>>>>>                                     says that copy_timeout
>>>>>>>>>>>                             is in seconds and implementation in
>>>>>>>>>>>                             lib/rcfunix/rcfunix.c passes the
>>>>>>>>>>>                             value to select() tv_sec.
>>>>>>>>>>>                             Theoretically select() could be
>>>>>>>>>>>                             interrupted by signal, but I
>>>>>>>>>>>                                     think it is unlikely here.
>>>>>>>>>>>
>>>>>>>>>>>                                     I'm not sure that I
>>>>>>>>>>>                             understand what do you mean by RCF
>>>>>>>>>>>                             connection timeout. Does it
>>>>>>>>>>>                             happen on TE startup when RCF
>>>>>>>>>>>                                     starts test agents. If
>>>>>>>>>>>                             so, TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>>                             help. Or
>>>>>>>>>>>                                     does it happen when
>>>>>>>>>>>                             tests are in progress, e.g. in
>>>>>>>>>>>                             the middle
>>>>>>>>>>>                                     of a test. If so,
>>>>>>>>>>>                             TE_RCFUNIX_TIMEOUT is unrelated
>>>>>>>>>>>                             and most
>>>>>>>>>>>                                     likely either host with
>>>>>>>>>>>                             test agent dies or test agent
>>>>>>>>>>>                             itself
>>>>>>>>>>>                             crashes. It would be easier for
>>>>>>>>>>>                             me if classify it if you share
>>>>>>>>>>>                                     text log (log.txt, full
>>>>>>>>>>>                             or just corresponding fragment with
>>>>>>>>>>>                                     some context). Also
>>>>>>>>>>>                             content of ta.DPDK or ta.Peer file
>>>>>>>>>>>                             depending on which agent has
>>>>>>>>>>>                             problems could shed some light.
>>>>>>>>>>>                             Corresponding files contain
>>>>>>>>>>>                             stdout/stderr of test agents.
>>>>>>>>>>>
>>>>>>>>>>>                             Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                     On 8/23/23 17:45, Adam
>>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                                     I've set up a test rig
>>>>>>>>>>>>                             repository here, and have created
>>>>>>>>>>>>                             configurations for our
>>>>>>>>>>>>                             development testbed based off
>>>>>>>>>>>>                             of the
>>>>>>>>>>>>                             examples.
>>>>>>>>>>>>                                     We've been able to get
>>>>>>>>>>>>                             the test suite to run manually on
>>>>>>>>>>>>                             Mellanox CX5 devices once.
>>>>>>>>>>>>                             However, we are running into an
>>>>>>>>>>>>                             issue where, when RCF starts,
>>>>>>>>>>>>                                     the RCF connection
>>>>>>>>>>>>                             times out very frequently. We
>>>>>>>>>>>>                             aren't sure
>>>>>>>>>>>>                                     why this is the case.
>>>>>>>>>>>>                                     It works sometimes, but
>>>>>>>>>>>>                             most of the time when we try to
>>>>>>>>>>>>                             run
>>>>>>>>>>>>                                     the test engine, it
>>>>>>>>>>>>                             encounters this issue.
>>>>>>>>>>>>                                     I've tried changing the
>>>>>>>>>>>>                             RCF port by setting
>>>>>>>>>>>>                             "TE_RCF_PORT=<some port
>>>>>>>>>>>>                             number>" and rebooting the testbed
>>>>>>>>>>>>                             machines. Neither seems to fix
>>>>>>>>>>>>                             the issue.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     It also seems like the
>>>>>>>>>>>>                             timeout takes far longer than 60
>>>>>>>>>>>>                             seconds, even when running
>>>>>>>>>>>>                             "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>                                     before I try to run the
>>>>>>>>>>>>                             test suite.
>>>>>>>>>>>>                                     I assume the unit for
>>>>>>>>>>>>                             this variable is seconds?
>>>>>>>>>>>>
>>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On Mon, Aug 21, 2023 at
>>>>>>>>>>>>                             10:19 AM Adam Hassick
>>>>>>>>>>>>                                     <ahassick@iol.unh.edu
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                             Thanks, I've cloned the example
>>>>>>>>>>>>                             repository and will start
>>>>>>>>>>>>                             setting up a configuration for
>>>>>>>>>>>>                             our development testbed
>>>>>>>>>>>>                             today. I'll let you know if I
>>>>>>>>>>>>                             run into any difficulties
>>>>>>>>>>>>                                         or have any questions.
>>>>>>>>>>>>
>>>>>>>>>>>>                                          - Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                                         On Sun, Aug 20,
>>>>>>>>>>>>                             2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                             I've published
>>>>>>>>>>>>                             https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>
>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Hopefully it
>>>>>>>>>>>>                             will help to define your test
>>>>>>>>>>>>                             rigs and
>>>>>>>>>>>>                                             successfully
>>>>>>>>>>>>                             run some tests manually. Feel
>>>>>>>>>>>>                             free to
>>>>>>>>>>>>                                             ask any
>>>>>>>>>>>>                             questions and I'll answer here
>>>>>>>>>>>>                             and try to
>>>>>>>>>>>>                                             update
>>>>>>>>>>>>                             documentation.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Meanwhile I'll
>>>>>>>>>>>>                             prepare missing bits for steps
>>>>>>>>>>>>                             (2) and
>>>>>>>>>>>>                                             (3).
>>>>>>>>>>>>                                             Hopefully
>>>>>>>>>>>>                             everything is in place for step
>>>>>>>>>>>>                             (4), but we
>>>>>>>>>>>>                                             need to make
>>>>>>>>>>>>                             steps (2) and (3) first.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             On 8/18/23
>>>>>>>>>>>>                             21:40, Andrew Rybchenko wrote:
>>>>>>>>>>>>>                             Hi Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             > I've
>>>>>>>>>>>>>                             conferred with the rest of the
>>>>>>>>>>>>>                             team, and we
>>>>>>>>>>>>>                                             think it would
>>>>>>>>>>>>>                             be best to move forward with
>>>>>>>>>>>>>                             mainly
>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             OK, I'll
>>>>>>>>>>>>>                             provide the sample on Monday
>>>>>>>>>>>>>                             for you. It is
>>>>>>>>>>>>>                                             almost ready
>>>>>>>>>>>>>                             right now, but I need to
>>>>>>>>>>>>>                             double-check
>>>>>>>>>>>>>                                             it before
>>>>>>>>>>>>>                             publishing.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Regards,
>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             On 8/17/23
>>>>>>>>>>>>>                             20:03, Adam Hassick wrote:
>>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             I'm adding
>>>>>>>>>>>>>>                             the CI mailing list to this
>>>>>>>>>>>>>>                                             conversation.
>>>>>>>>>>>>>>                             Others in the community might
>>>>>>>>>>>>>>                             find
>>>>>>>>>>>>>>                                             this
>>>>>>>>>>>>>>                             conversation valuable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             We do want to
>>>>>>>>>>>>>>                             run testing on a regular
>>>>>>>>>>>>>>                             basis. The
>>>>>>>>>>>>>>                                             Jenkins
>>>>>>>>>>>>>>                             integration will be very
>>>>>>>>>>>>>>                             useful for us, as
>>>>>>>>>>>>>>                                             most of our
>>>>>>>>>>>>>>                             CI is orchestrated by Jenkins.
>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>                             conferred with the rest of
>>>>>>>>>>>>>>                             the team, and we
>>>>>>>>>>>>>>                                             think it
>>>>>>>>>>>>>>                             would be best to move forward
>>>>>>>>>>>>>>                             with mainly
>>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>>                                             If you would
>>>>>>>>>>>>>>                             like to know anything about our
>>>>>>>>>>>>>>                                             testbeds that
>>>>>>>>>>>>>>                             would help you with creating an
>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>                             ts-rigs repo, I'd be happy to
>>>>>>>>>>>>>>                             answer any
>>>>>>>>>>>>>>                                             questions you
>>>>>>>>>>>>>>                             have.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             We have
>>>>>>>>>>>>>>                             multiple test rigs (we call
>>>>>>>>>>>>>>                             these
>>>>>>>>>>>>>>                                             "DUT-tester
>>>>>>>>>>>>>>                             pairs") that we run our existing
>>>>>>>>>>>>>>                                             hardware
>>>>>>>>>>>>>>                             testing on, with differing
>>>>>>>>>>>>>>                             network
>>>>>>>>>>>>>>                                             hardware and
>>>>>>>>>>>>>>                             CPU architecture. I figured
>>>>>>>>>>>>>>                             this might
>>>>>>>>>>>>>>                                             be an
>>>>>>>>>>>>>>                             important detail.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On Thu, Aug
>>>>>>>>>>>>>>                             17, 2023 at 11:44 AM Andrew
>>>>>>>>>>>>>>                             Rybchenko
>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Greatings
>>>>>>>>>>>>>>                             Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 I'm happy
>>>>>>>>>>>>>>                             to hear that you're trying to
>>>>>>>>>>>>>>                             bring
>>>>>>>>>>>>>>                                                 it up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 As I
>>>>>>>>>>>>>>                             understand the final goal is
>>>>>>>>>>>>>>                             to run it on
>>>>>>>>>>>>>>                                                 regular
>>>>>>>>>>>>>>                             basis. So, we need to make it
>>>>>>>>>>>>>>                             properly
>>>>>>>>>>>>>>                                                 from the
>>>>>>>>>>>>>>                             very beginning.
>>>>>>>>>>>>>>                                                 Bring up
>>>>>>>>>>>>>>                             of all features consists of 4
>>>>>>>>>>>>>>                             steps:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 1. Create
>>>>>>>>>>>>>>                             site-specific repository (we
>>>>>>>>>>>>>>                             call it
>>>>>>>>>>>>>>                                                 ts-rigs)
>>>>>>>>>>>>>>                             which contains information
>>>>>>>>>>>>>>                             about test
>>>>>>>>>>>>>>                                                 rigs and
>>>>>>>>>>>>>>                             other site-specific
>>>>>>>>>>>>>>                             information like
>>>>>>>>>>>>>>                                                 where to
>>>>>>>>>>>>>>                             send mails, where to store
>>>>>>>>>>>>>>                             logs etc.
>>>>>>>>>>>>>>                                                 It is
>>>>>>>>>>>>>>                             required for manual execution
>>>>>>>>>>>>>>                             as well,
>>>>>>>>>>>>>>                                                 since
>>>>>>>>>>>>>>                             test rigs description is
>>>>>>>>>>>>>>                             essential. I'll
>>>>>>>>>>>>>>                                                 return to
>>>>>>>>>>>>>>                             the topic below.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 2. Setup
>>>>>>>>>>>>>>                             logs storage for automated runs.
>>>>>>>>>>>>>>                                                 Basically
>>>>>>>>>>>>>>                             it is a disk space plus
>>>>>>>>>>>>>>                             apache2 web
>>>>>>>>>>>>>>                                                 server
>>>>>>>>>>>>>>                             with few CGI scripts which
>>>>>>>>>>>>>>                             help a lot to
>>>>>>>>>>>>>>                                                 save disk
>>>>>>>>>>>>>>                             space.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 3. Setup
>>>>>>>>>>>>>>                             Bublik web application which
>>>>>>>>>>>>>>                             provides
>>>>>>>>>>>>>>                                                 web
>>>>>>>>>>>>>>                             interface to view testing
>>>>>>>>>>>>>>                             results. Same as
>>>>>>>>>>>>>>                             https://ts-factory.io/bublik
>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 4. Setup
>>>>>>>>>>>>>>                             Jenkins to run tests on
>>>>>>>>>>>>>>                             regularly,
>>>>>>>>>>>>>>                                                 save logs
>>>>>>>>>>>>>>                             in log storage (2) and import
>>>>>>>>>>>>>>                             it to
>>>>>>>>>>>>>>                                                 bublik (3).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Last few
>>>>>>>>>>>>>>                             month we spent on our
>>>>>>>>>>>>>>                             homework to make
>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>                             simpler to bring up automated
>>>>>>>>>>>>>>                             execution
>>>>>>>>>>>>>>                                                 using
>>>>>>>>>>>>>>                             Jenkins -
>>>>>>>>>>>>>>                             https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>                             Corresponding bits in
>>>>>>>>>>>>>>                             dpdk-ethdev-ts will be
>>>>>>>>>>>>>>                                                 available
>>>>>>>>>>>>>>                             tomorrow.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Let's
>>>>>>>>>>>>>>                             return to the step (1).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>                             Unfortunately there is no
>>>>>>>>>>>>>>                             publicly available
>>>>>>>>>>>>>>                                                 example
>>>>>>>>>>>>>>                             of the ts-rigs repository since
>>>>>>>>>>>>>>                                                 sensitive
>>>>>>>>>>>>>>                             site-specific information is
>>>>>>>>>>>>>>                             located
>>>>>>>>>>>>>>                                                 there.
>>>>>>>>>>>>>>                             But I'm ready to help you to
>>>>>>>>>>>>>>                             create it
>>>>>>>>>>>>>>                                                 for UNH.
>>>>>>>>>>>>>>                             I see two options here:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 (A) I'll
>>>>>>>>>>>>>>                             ask questions and based on your
>>>>>>>>>>>>>>                                                 answers
>>>>>>>>>>>>>>                             will create the first draft
>>>>>>>>>>>>>>                             with my
>>>>>>>>>>>>>>                                                 comments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 (B) I'll
>>>>>>>>>>>>>>                             make a template/example
>>>>>>>>>>>>>>                             ts-rigs repo,
>>>>>>>>>>>>>>                                                 publish
>>>>>>>>>>>>>>                             it and you'll create UNH
>>>>>>>>>>>>>>                             ts-rigs based
>>>>>>>>>>>>>>                                                 on it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Of
>>>>>>>>>>>>>>                             course, I'll help to debug
>>>>>>>>>>>>>>                             and finally bring
>>>>>>>>>>>>>>                                                 it up in
>>>>>>>>>>>>>>                             any case.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 (A) is a
>>>>>>>>>>>>>>                             bit simpler for me and you,
>>>>>>>>>>>>>>                             but (B) is
>>>>>>>>>>>>>>                                                 a bit
>>>>>>>>>>>>>>                             more generic and will help other
>>>>>>>>>>>>>>                                                 potential
>>>>>>>>>>>>>>                             users to bring it up.
>>>>>>>>>>>>>>                                                 We can
>>>>>>>>>>>>>>                             combine (A)+(B). I.e. start
>>>>>>>>>>>>>>                             from (A).
>>>>>>>>>>>>>>                                                 What do
>>>>>>>>>>>>>>                             you think?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>                             8/17/23 15:18, Konstantin
>>>>>>>>>>>>>>                             Ushakov wrote:
>>>>>>>>>>>>>>>                             Greetings Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Thanks
>>>>>>>>>>>>>>>                             for contacting us. I copy
>>>>>>>>>>>>>>>                             Andrew who
>>>>>>>>>>>>>>>                                                 would be
>>>>>>>>>>>>>>>                             happy to help
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>                                                 Konstantin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             On 16 Aug 2023, at 21:50,
>>>>>>>>>>>>>>>>                             Adam Hassick
>>>>>>>>>>>>>>>>                             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I am in
>>>>>>>>>>>>>>>>                             the process of setting up
>>>>>>>>>>>>>>>>                             the DPDK
>>>>>>>>>>>>>>>>                                                 Poll
>>>>>>>>>>>>>>>>                             Mode Driver test suite as
>>>>>>>>>>>>>>>>                             an addition to
>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>                             testing coverage for DPDK
>>>>>>>>>>>>>>>>                             at the UNH lab.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I have
>>>>>>>>>>>>>>>>                             some questions about how to
>>>>>>>>>>>>>>>>                             set the
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                             suite arguments.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I have
>>>>>>>>>>>>>>>>                             been able to configure the
>>>>>>>>>>>>>>>>                             Test Engine
>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>                             connect to the hosts in the
>>>>>>>>>>>>>>>>                             testbed. The
>>>>>>>>>>>>>>>>                                                 RCF,
>>>>>>>>>>>>>>>>                             Configurator, and Tester
>>>>>>>>>>>>>>>>                             all begin to
>>>>>>>>>>>>>>>>                                                 run,
>>>>>>>>>>>>>>>>                             however the prelude of the
>>>>>>>>>>>>>>>>                             test suite
>>>>>>>>>>>>>>>>                                                 fails
>>>>>>>>>>>>>>>>                             to run.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 The
>>>>>>>>>>>>>>>>                             documentation mentions that
>>>>>>>>>>>>>>>>                             there are
>>>>>>>>>>>>>>>>                                                 several
>>>>>>>>>>>>>>>>                             test parameters for the
>>>>>>>>>>>>>>>>                             test suite,
>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>                             for the IUT test link MAC,
>>>>>>>>>>>>>>>>                             etc. These
>>>>>>>>>>>>>>>>                                                 seem
>>>>>>>>>>>>>>>>                             like they would need to be
>>>>>>>>>>>>>>>>                             set somewhere
>>>>>>>>>>>>>>>>                                                 to run
>>>>>>>>>>>>>>>>                             many of the tests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I see
>>>>>>>>>>>>>>>>                             in the Test Engine
>>>>>>>>>>>>>>>>                             documentation, there
>>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>>                             instructions on how to
>>>>>>>>>>>>>>>>                             create new
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             parameters for test suites
>>>>>>>>>>>>>>>>                             in the Tester
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             configuration, but there is
>>>>>>>>>>>>>>>>                             nothing in the
>>>>>>>>>>>>>>>>                                                 user
>>>>>>>>>>>>>>>>                             guide or in the Tester
>>>>>>>>>>>>>>>>                             guide for how to
>>>>>>>>>>>>>>>>                                                 set the
>>>>>>>>>>>>>>>>                             arguments for the
>>>>>>>>>>>>>>>>                             parameters when
>>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>>                             the test suite that I can
>>>>>>>>>>>>>>>>                             find. I'm
>>>>>>>>>>>>>>>>                                                 not
>>>>>>>>>>>>>>>>                             sure if I need to write my
>>>>>>>>>>>>>>>>                             own Tester
>>>>>>>>>>>>>>>>                                                 config,
>>>>>>>>>>>>>>>>                             or if I should be setting
>>>>>>>>>>>>>>>>                             these in
>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>                             other way.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 How
>>>>>>>>>>>>>>>>                             should these values be set?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>                             also not sure what environment
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             variables/arguments are
>>>>>>>>>>>>>>>>                             strictly necessary or
>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>                             are optional.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 Regards,
>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>                             Hassick*
>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>                             Developer
>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>                             (603) 475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>                                             Senior Developer
>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>                             475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                         -- *Adam Hassick*
>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                     --         *Adam Hassick*
>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                 -- *Adam Hassick*
>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             -- 
>>>>>>>>>>                             *Adam Hassick*
>>>>>>>>>>                             Senior Developer
>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                         -- 
>>>>>>>>                         *Adam Hassick*
>>>>>>>>                         Senior Developer
>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                         +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                     -- 
>>>>>>                     *Adam Hassick*
>>>>>>                     Senior Developer
>>>>>>                     UNH InterOperability Lab
>>>>>>                     ahassick@iol.unh.edu
>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                     +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>                 -- 
>>>>                 *Adam Hassick*
>>>>                 Senior Developer
>>>>                 UNH InterOperability Lab
>>>>                 ahassick@iol.unh.edu
>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>                 +1 (603) 475-8248
>>>
>>
>>
>>
>>         -- 
>>         *Adam Hassick*
>>         Senior Developer
>>         UNH InterOperability Lab
>>         ahassick@iol.unh.edu
>>         iol.unh.edu <https://www.iol.unh.edu/>
>>         +1 (603) 475-8248
>
>
>
>     -- 
>     *Adam Hassick*
>     Senior Developer
>     UNH InterOperability Lab
>     ahassick@iol.unh.edu
>     iol.unh.edu <https://www.iol.unh.edu/>
>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 161478 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 83142 bytes --]

Hi Adam,

I've uploaded fresh testing results to ts-factory.io [1] to be on the 
same page.

I think I know why your and mine results on Intel 710 series NICs differ 
so much. Testing results expectations database (dpdk-ethdev-ts/trc/*) is 
filled in in terms of TRC tags.  I.e. expectations depends on TRC tags 
discovered by helper scripts when testing is started. These tags 
identify various aspects of what is tested. Ideally expectations should 
be written in terms of root cause of the expected behaviour. If it is a 
driver expectations, driver tag should be used. If it is HW limitation, 
tags with PCI IDs should be used. However, it is not always easy to 
classify it correctly if you're not involved in driver development. So, 
in order case expectations for 710's Intel are filled in in terms of PCI 
IDs. I guess PCI ID differ in your case and that's why expectations 
filled in for my NIC do not apply to your runs.

Just try to add the following option when you run on your 710's Intel in 
order to mimic mine and see if it helps to achieve better pass rate.
--trc-tag=pci-8086-1572

BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for 
--tester-dial option. It should have better coverage now.

Andrew.

[1] 
https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1

On 9/13/23 18:45, Andrew Rybchenko wrote:
> Hi Adam,
>
> I've pushed new TE tag v1.20.0 which supported a new command-line 
> option --tester-dial=NUM where NUM is from 0 to 100. it allows to 
> choose percentage of tests to run. If you want stable set, you should 
> pass --tester-random-seed=0 (or other integer). It is the first sketch 
> and we have plans to improve it, but feedback would be welcome.
>
> > Is it needed on the tester?
>
> It is hard to say if it is strictly required for simple tests. 
> However, it is better to update Tester as well, since performance 
> tests run DPDK on Tester as well.
>
> > Are there any other manual setup steps for these devices that I 
> might be missing?
>
> I don't remember anything else.
>
> I think it is better to get down to details and take a look at logs. 
> I'm ready to help with it and explain what's happening there. May be 
> it will help to understand if it is a problem with setup/configuration.
>
> Text logs are not very convenient. Ideally logs should be imported to 
> bublik, however, manual runs do not provide all required artifacts 
> right now (Jenkins jobs generate all required artifacts).
> Other option is 'tmp_raw_log' file (should be packed to make it 
> smaller) which could be converted to various log formats.
> Would it be OK for you if I import your logs to bublik at 
> ts-factory.io? Or is it a problem that it is publicly available?
> Would it help if we add authentication and access control there?
>
> Andrew.
>
> On 9/8/23 17:57, Adam Hassick wrote:
>> Hi Andrew,
>>
>> I have a couple questions about needed setup of the NICs for the 
>> ethdev test suite.
>>
>> Our MCX5s and XL710s are failing the checkup tests. The pass rate 
>> appears to be much worse on the XL710s (40 of 73 tests failed, 3 
>> passed unexpectedly).
>>
>> For the XL710s, I've updated the driver and NVM versions to match the 
>> minimum supported versions in the compatibility matrix found on the 
>> DPDK documentation. This did not change the failure rate much.
>> For the MCX5s, I've installed the latest LTS version of the OFED 
>> bifurcated driver on the DUT. Is it needed on the tester?
>>
>> Are there any other manual setup steps for these devices that I might 
>> be missing?
>>
>> Thanks,
>> Adam
>>
>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu> 
>> wrote:
>>
>>     Hi Andrew,
>>
>>     Yes, I copied the X710 configs to set up XL710 configs. I changed
>>     the environment variable names from the X710 suffix to XL710
>>     suffix in the script, and forgot to change them in the
>>     corresponding environment file.
>>     That fixed the issue.
>>
>>     I got the checkup tests working on the XL710 now. Most of them
>>     are failing, which leads me to believe this is an issue with our
>>     testbed. Based on the DPDK documentation for i40e, the firmware
>>     and driver versions are much older than what DPDK 22.11 LTS and
>>     main prefer, so I'll try updating those.
>>
>>     For now I'm working on getting the XL710 checkup tests passing,
>>     and will pick up getting the E810 configured properly next. I'll
>>     let you know if I run into any more issues in relation to the
>>     test engine.
>>
>>     Thanks,
>>     Adam
>>
>>     On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         Hi Adam,
>>
>>         On 9/5/23 18:01, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         The compilation warning issue is now resolved. Again, thank
>>>         you guys for fixing this for us. I can run the tests on the
>>>         Mellanox CX5s again, however I'm running into a couple new
>>>         issues with running the prologues on the Intel cards.
>>>
>>>         When running testing on the Intel XL710s, I see this error
>>>         appear in the log:
>>>
>>>             ERROR  prologue  Environment LIB  14:16:13.650
>>>             Too few networks in available configuration (0) in
>>>             comparison with required (1)
>>>
>>>
>>>         This seems like a trivial configuration error, perhaps this
>>>         is something I need to set up in ts-rigs. I briefly searched
>>>         through the examples there and didn't see any mention of how
>>>         to set up a network.
>>>         I will attach this log just in case you need more information.
>>
>>         Unfortunately logs are insufficient to understand it. I've
>>         pushed new tag to TE v1.19.0 which add log message with TE_*
>>         environment variables.
>>         Most likely something is wrong with variables which are used
>>         as conditions when available networks are defined in
>>         ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>         TE_PCI_INSTANCE_IUT_TST1
>>         TE_PCI_INSTANCE_IUT_TST1a
>>         TE_PCI_INSTANCE_TST1a_IUT
>>         TE_PCI_INSTANCE_TST1_IUT
>>         My guess it that you change naming a bit, but script like
>>         ts-rigs-sample/scripts/iut.h1-x710 is not included or not
>>         updated.
>>
>>>         There is a different error when running on the Intel E810s.
>>>         It appears to me like it starts DPDK, does some
>>>         configuration inside DPDK and on the device, and then fails
>>>         to bring the device back up. Since this error seems very
>>>         non-trivial, I will also attach this log.
>>
>>         This one is a bit simpler. Few lines after the first ERROR in
>>         log I see the following:
>>         WARN  RCF  DPDK  13:06:00.144
>>         ice_program_hw_rx_queue(): currently package doesn't support
>>         RXDID (22)
>>         ice_rx_queue_start(): fail to program RX queue 0
>>         ice_dev_start(): fail to start Rx queue 0
>>         Device with port_id=0 already stopped
>>
>>         It is stdout/stderr from test agent which runs DPDK. Same
>>         logs in plain format are available in ta.DPDK file.
>>         I'm not an expert here, but I vaguely remember that E810
>>         requires correct firmware and DDP to be loaded.
>>         There is some information in dpdk/doc/guides/nics/ice.rst.
>>
>>         You can try to add --dev-args=safe-mode-support=1
>>         command-line option described there.
>>
>>         Hope it helps,
>>         Andrew.
>>
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             Hi Adam,
>>>
>>>             On 8/31/23 22:38, Adam Hassick wrote:
>>>>             Hi Andrew,
>>>>
>>>>             I have one additional question as well: Does the test
>>>>             engine support running tests on two ARMv8 test agents?
>>>>
>>>>                 1. We'll sort out warnings this week. Thanks for
>>>>                 heads up.
>>>>
>>>>
>>>>             Great. Let me know when that's fixed.
>>>
>>>             Done. We also fixed a number of warnings in TE.
>>>             Also we fixed root test package name to be consistent
>>>             with the repository name.
>>>
>>>>                 Support for old LTS branches was dropped some time
>>>>                 ago, but in the future it is definitely possible to
>>>>                 keep it for new LTS branches. I think 22.11 is
>>>>                 supported, but I'm not sure about older LTS releases.
>>>>
>>>>
>>>>             Good to know.
>>>>
>>>>                 2. You can add command-line option --sanity to run
>>>>                 tests marked with TEST_HARNESS_SANITY requirement
>>>>                 (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>>                 TEST_HARNESS_SANITY dpdk-ethdev-ts to see which
>>>>                 tests are marked). Yes, there is a space for
>>>>                 terminology improvement here. We'll do it.
>>>>
>>>
>>>             Done. Now it is called --checkup.
>>>
>>>>
>>>>                 Also it takes a lot of time because of failures and
>>>>                 tests which wait for some timeout.
>>>>
>>>>
>>>>             That makes sense to me. We'll use the time to complete
>>>>             tests on virtio or the Intel devices as a reference for
>>>>             how long the tests really take to complete.
>>>>             We will explore the possibility of periodically running
>>>>             the sanity tests for patches.
>>>
>>>             I'll double-check and let you know how long entire TS
>>>             runs on Intel X710, E810, Mellanox CX5 and virtio net.
>>>             Just to ensure that time observed in your case looks the
>>>             same.
>>>
>>>>
>>>>                 The test harness can provide coverage reports based
>>>>                 on gcov, but I'm not sure what you mean by a "dial"
>>>>                 to control test coverage. Provided reports are
>>>>                 rather for human to analyze.
>>>>
>>>>
>>>>             The general idea is to have some kind of parameter on
>>>>             the test suite, which could be an integer ranging from
>>>>             zero to ten, that controls how many tests are run based
>>>>             on how important the test is.
>>>>
>>>>             Similar to how some command line interfaces provide a
>>>>             verbosity level parameter (some number of "-v"
>>>>             arguments) to control the importance of the information
>>>>             in the log.
>>>>             The verbosity level zero only prints very important log
>>>>             messages, while ten prints everything.
>>>>
>>>>             In much the same manner as above, this "dial" parameter
>>>>             controls what tests are run and with what parameters
>>>>             based on how important those tests and test parameter
>>>>             combinations are.
>>>>             Coverage Level zero tells the suite to run a very basic
>>>>             set of important tests, with minimal parameterization.
>>>>             This mode would take only ~5-10 minutes to run.
>>>>             In contrast, Coverage Level ten includes all the edge
>>>>             cases, every combination of test parameters, everything
>>>>             the test suite can do, which takes the normal several
>>>>             hours to run.
>>>>             The values 1 - 9 are between those two extremes,
>>>>             allowing the user to get a gradient of test coverage in
>>>>             the results and to limit the running time.
>>>>
>>>>             Then we could, for example, run the "run.sh" with a
>>>>             level of 2 or 3 for incoming patches that need quick
>>>>             results, and with a level of 10 for the less often run
>>>>             periodic tests performed on main or LTS branches.
>>>
>>>             Understood now. Thanks a lot for the idea. We'll discuss
>>>             it and come back.
>>>
>>>>                 3. Yes, really many tests on Mellanox CX5 NICs
>>>>                 report unexpected testing results. Unfortunately it
>>>>                 is time consuming to fill in expectations database
>>>>                 since it is necessary to analyze testing results
>>>>                 and classify if it is a bug or just acceptable
>>>>                 behaviour aspect.
>>>>
>>>>                 Bublik allows to compare results of two runs. It is
>>>>                 useful for human, but still not good for automation.
>>>>
>>>>                 I have local patch for mlx5 driver which reports Tx
>>>>                 ring size maximum. It makes pass rate higher. It is
>>>>                 a problem for test harness that mlx5 does not
>>>>                 report limits right now.
>>>>
>>>>                 Pass rate on Intel X710 is about 92% on my test
>>>>                 rig. Pass rate on virtio net is 99% right now and
>>>>                 could be done 100% easily (just one thing to fix in
>>>>                 expectations).
>>>>
>>>>                 I think logs storage setup is essential for logs
>>>>                 analysis. Of course, you can request HTML logs when
>>>>                 you run tests (--log-html=html) or generate after
>>>>                 run using dpdk-ethdev-ts/scripts/html-log.sh and
>>>>                 open index.html in a browser, but logs storage
>>>>                 makes it more convenient.
>>>>
>>>>
>>>>             We are interested in setting up Bublik, potentially as
>>>>             an externally-facing component, once we have our
>>>>             process of running the test suite stabilized.
>>>>             Once we are able to run the test suite again, I'll see
>>>>             what the pass rate is on our other hardware.
>>>>             Good to know that it isn't an issue with our dev
>>>>             testbed causing the high fail rate.
>>>>
>>>>             For Intel hardware, we have an XL710 and an Intel
>>>>             E810-C in our development testbed. Although they are
>>>>             slightly different devices, ideally the pass rate will
>>>>             be identical or similar. I have yet to set up a VM pair
>>>>             for virtio, but we will soon.
>>>>
>>>>                 Latest version of test-environment has examples of
>>>>                 our CGI scripts which we use for log storage (see
>>>>                 tools/log_server/README.md).
>>>>
>>>>                 Also all bits for Jenkins setup are available. See
>>>>                 dpdk-ethdev-ts/jenkins/README.md and examples of
>>>>                 jenkins files in ts-rigs-sample.
>>>>
>>>>
>>>>             Jenkins integration, setting up production rig
>>>>             configurations, and permanent log storage will be our
>>>>             next steps once I am able to run the tests again.
>>>>             Unless there is an easy way to have meson not pass
>>>>             "-Werror" into GCC. Then I would be able to run the
>>>>             test suite.
>>>
>>>             Hopefully it is resolved now.
>>>
>>>             I thought a bit more about your usecase for Jenkins. I'm
>>>             not 100% sure that existing pipelines are convenient for
>>>             your usecase.
>>>             Fill free to ask questions when you are on it.
>>>
>>>             Thanks,
>>>             Andrew.
>>>
>>>>
>>>>             Thanks,
>>>>             Adam
>>>>
>>>>
>>>>                 On 8/29/23 17:02, Adam Hassick wrote:
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 That fix seems to have resolved the issue, thanks
>>>>>                 for the quick turnaround time on that patch.
>>>>>                 Now that we have the RCF timeout issue resolved,
>>>>>                 there are a few other questions and issues that we
>>>>>                 have about the tests themselves.
>>>>>
>>>>>                 1. The test suite fails to build with a couple
>>>>>                 warnings.
>>>>>
>>>>>                 Below is the stderr log from compilation:
>>>>>
>>>>>                     FAILED:
>>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>                     -I../../lib
>>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>                     -fdiagnostics-color=always -pipe
>>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>                     -fPIC -MD -MQ
>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>                     -MF
>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>                     -o
>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>>                     ‘test_create_traffic_generator_params’:
>>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>>                     not a string literal and no format arguments
>>>>>                     [-Werror=format-security]
>>>>>                     5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>                     |     ^~
>>>>>                     cc1: all warnings being treated as errors
>>>>>                     ninja: build stopped: subcommand failed.
>>>>>                     ninja: Entering directory `.'
>>>>>                     FAILED:
>>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>                     -I../../lib
>>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>                     -fdiagnostics-color=always -pipe
>>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>                     -fPIC -MD -MQ
>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>                     -MF
>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>                     -o
>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>>                     ‘test_create_traffic_generator_params’:
>>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>>                     not a string literal and no format arguments
>>>>>                     [-Werror=format-security]
>>>>>                     5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>                     |     ^~
>>>>>                     cc1: all warnings being treated as errors
>>>>>
>>>>>
>>>>>                 This error wasn't occurring last week, which was
>>>>>                 the last time I ran the tests.
>>>>>                 The TE host and the DUT have GCC v9.4.0 installed,
>>>>>                 and the tester has GCC v11.4.0 installed, if this
>>>>>                 information is helpful.
>>>>>
>>>>>                 2. On the Mellanox CX5s, there are over 6,000
>>>>>                 tests run, which collectively take around 9 hours.
>>>>>                 Is it possible, and would it make sense, to lower
>>>>>                 the test coverage and have the test suite run faster?
>>>>>
>>>>>                 For some context, we run immediate testing on
>>>>>                 incoming patches for DPDK main and development
>>>>>                 branches, as well as periodic test runs on the
>>>>>                 main, stable, and LTS branches.
>>>>>                 For us to consider including this test suite as
>>>>>                 part of our immediate testing on patches, we would
>>>>>                 have to reduce the test coverage to the most
>>>>>                 important tests.
>>>>>                 This is primarily to reduce the testing time to,
>>>>>                 for example, less than 30 minutes. Testing on
>>>>>                 patches can't take too long because the lab can
>>>>>                 receive numerous patches each day, which each
>>>>>                 require individual testing runs.
>>>>>
>>>>>                 At what frequency we run these tests, and on what,
>>>>>                 still needs to be discussed with the DPDK
>>>>>                 community, but it would be nice to know if the
>>>>>                 test suite had a "dial" to control the testing
>>>>>                 coverage.
>>>>>
>>>>>                 3. We see a lot of test failures on our Mellanox
>>>>>                 CX5 NICs. Around 2,300 of ~6,600 tests passed. Is
>>>>>                 there anything we can do to diagnose these test
>>>>>                 failures?
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>
>>>>>                 On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>                     Hi Adam,
>>>>>
>>>>>                     I've pushed the fix in main branch and a new
>>>>>                     tag v1.18.1. It should solve the problem with
>>>>>                     IPv6 address from DNS.
>>>>>
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     > Does the test engine prefer to use IPv6
>>>>>>                     over IPv4 for initiating the RCF connection
>>>>>>                     to the test bed hosts? And if so, is there a
>>>>>>                     way to force it to use IPv4?
>>>>>>
>>>>>>                     Brilliant idea. If DNS returns both IPv4 and
>>>>>>                     IPv6 addresses in your case, I guess it is
>>>>>>                     the root cause of the problem.
>>>>>>                     Of course, it is TE problem since I see
>>>>>>                     really weird code in
>>>>>>                     lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>>
>>>>>>                     I've pushed fix to the branch
>>>>>>                     user/arybchik/fix_ipv4_only in
>>>>>>                     ts-factory/test-environment repository.
>>>>>>                     Please, try.
>>>>>>
>>>>>>                     It is late night fix with minimal testing and
>>>>>>                     no review. I'll pass it through review
>>>>>>                     process tomorrow and
>>>>>>                     hopefully it will be released in one-two days.
>>>>>>
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>                     Hi Andrew,
>>>>>>>
>>>>>>>                     We have yet to notice a distinct pattern
>>>>>>>                     with the failures. Sometimes, the RCF will
>>>>>>>                     start and connect without issue a few times
>>>>>>>                     in a row before failing to connect again.
>>>>>>>                     Once the issue begins to occur, neither
>>>>>>>                     rebooting all of the hosts (test engine VM,
>>>>>>>                     tester, IUT) or deleting all of the build
>>>>>>>                     directories (suites, agents, inst) and
>>>>>>>                     rebooting the hosts afterward resolves the
>>>>>>>                     issue. When it begins working again seems
>>>>>>>                     very arbitrary to us.
>>>>>>>
>>>>>>>                     I do usually try to terminate the test
>>>>>>>                     engine with Ctrl+C, but when it hangs while
>>>>>>>                     trying to start RCF, that does not work.
>>>>>>>
>>>>>>>                     Does the test engine prefer to use IPv6 over
>>>>>>>                     IPv4 for initiating the RCF connection to
>>>>>>>                     the test bed hosts? And if so, is there a
>>>>>>>                     way to force it to use IPv4?
>>>>>>>
>>>>>>>                      - Adam
>>>>>>>
>>>>>>>                     On Fri, Aug 25, 2023 at 1:35 PM Andrew
>>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>                         > I'll double-check test engine on
>>>>>>>                         Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>
>>>>>>>                         Done. It works fine for me without any
>>>>>>>                         issues.
>>>>>>>
>>>>>>>                         Have you noticed any pattern when it
>>>>>>>                         works or does not work?
>>>>>>>                         May be it is a problem of not clean
>>>>>>>                         state after termination?
>>>>>>>                         Does it work fine the first time after
>>>>>>>                         DUTs reboot?
>>>>>>>                         How do you terminate testing? It should
>>>>>>>                         be done using Ctrl+C in terminal where
>>>>>>>                         you execute run.sh command.
>>>>>>>                          In this case it should shutdown
>>>>>>>                         gracefully and close all test agents and
>>>>>>>                         engine applications.
>>>>>>>
>>>>>>>                         (I'm trying to understand why you've
>>>>>>>                         seen many test agent processes. It
>>>>>>>                         should not happen.)
>>>>>>>
>>>>>>>                         Andrew.
>>>>>>>
>>>>>>>                         On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>>                         On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>                         Hi Andrew,
>>>>>>>>>
>>>>>>>>>                         Two of our systems (the Test Engine
>>>>>>>>>                         runner and the DUT host) are running
>>>>>>>>>                         Ubuntu 20.04 LTS, however this morning
>>>>>>>>>                         I noticed that the tester system (the
>>>>>>>>>                         one having issues) is running Ubuntu
>>>>>>>>>                         22.04 LTS.
>>>>>>>>>                         This could be the source of the
>>>>>>>>>                         problem. I encountered a dependency
>>>>>>>>>                         issue trying to run the Test Engine on
>>>>>>>>>                         22.04 LTS, so I downgraded the system.
>>>>>>>>>                         Since the tester is also the host
>>>>>>>>>                         having connection issues, I will try
>>>>>>>>>                         downgrading that system to 20.04, and
>>>>>>>>>                         see if that changes anything.
>>>>>>>>
>>>>>>>>                         Unlikely, but who knows. We run tests
>>>>>>>>                         (DUTs) on Ubuntu 20.04, Ubuntu 22.04,
>>>>>>>>                         Ubuntu 22.10, Ubuntu 23.04, Debian 11
>>>>>>>>                         and Fedora 38 every night.
>>>>>>>>                         Right now Debian 11 is used for test
>>>>>>>>                         engine in nightly regressions.
>>>>>>>>
>>>>>>>>                         I'll double-check test engine on Ubuntu
>>>>>>>>                         20.04 and Ubuntu 22.04.
>>>>>>>>
>>>>>>>>>                         I did try passing in the "--vg-rcf"
>>>>>>>>>                         argument to the run.sh script of the
>>>>>>>>>                         test suite after installing valgrind,
>>>>>>>>>                         but there was no additional output
>>>>>>>>>                         that I saw.
>>>>>>>>
>>>>>>>>                         Sorry, I should valgrind output should
>>>>>>>>                         be in valgrind.te_rcf (direction where
>>>>>>>>                         you run test engine).
>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         I will try pulling in the changes
>>>>>>>>>                         you've pushed up, and will see if that
>>>>>>>>>                         fixes anything.
>>>>>>>>>
>>>>>>>>>                         Thanks,
>>>>>>>>>                         Adam
>>>>>>>>>
>>>>>>>>>                         On Fri, Aug 25, 2023 at 9:57 AM Andrew
>>>>>>>>>                         Rybchenko
>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>
>>>>>>>>>                             Hello Adam,
>>>>>>>>>
>>>>>>>>>                             On 8/24/23 23:54, Andrew Rybchenko
>>>>>>>>>                             wrote:
>>>>>>>>>>                             I'd like to try to repeat the
>>>>>>>>>>                             problem locally. Which Linux
>>>>>>>>>>                             distro is running on test engine
>>>>>>>>>>                             and agents?
>>>>>>>>>>
>>>>>>>>>>                             In fact I know one problem with
>>>>>>>>>>                             Debian 12 and Fedora 38 and we have
>>>>>>>>>>                             patch in review to fix it,
>>>>>>>>>>                             however, the behaviour is
>>>>>>>>>>                             different in
>>>>>>>>>>                             this case, so it is unlike the
>>>>>>>>>>                             same problem.
>>>>>>>>>
>>>>>>>>>                             I've just published a new tag
>>>>>>>>>                             which fixes known test engine side
>>>>>>>>>                             problems on Debian 12 and Fedora 38.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             One more idea is to install
>>>>>>>>>>                             valgrind on the test engine host and
>>>>>>>>>>                             run with option --vg-rcf to check
>>>>>>>>>>                             if something weird is happening.
>>>>>>>>>>
>>>>>>>>>>                             What I don't understand right now
>>>>>>>>>>                             is why I see just one failed attempt
>>>>>>>>>>                             to connect in your log.txt and
>>>>>>>>>>                             then Logger shutdown after 9
>>>>>>>>>>                             minutes.
>>>>>>>>>>
>>>>>>>>>>                             Andrew.
>>>>>>>>>>
>>>>>>>>>>                             On 8/24/23 23:29, Adam Hassick
>>>>>>>>>>                             wrote:
>>>>>>>>>>>                              > Is there any firewall in the
>>>>>>>>>>>                             network or on test hosts which
>>>>>>>>>>>                             could block incoming TCP
>>>>>>>>>>>                             connection to the port 23571
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             from the host where you run test
>>>>>>>>>>>                             engine?
>>>>>>>>>>>
>>>>>>>>>>>                             Our test engine host and the
>>>>>>>>>>>                             testbed are on the same subnet.
>>>>>>>>>>>                             The connection does work sometimes.
>>>>>>>>>>>
>>>>>>>>>>>                              > If behaviour the same on the
>>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>>                             agent is kept running, could you
>>>>>>>>>>>                             check using
>>>>>>>>>>>                              >
>>>>>>>>>>>                              > # netstat -tnlp
>>>>>>>>>>>                              >
>>>>>>>>>>>                              > that Test Agent is listening
>>>>>>>>>>>                             on the port and try to establish
>>>>>>>>>>>                             TCP connection from test agent
>>>>>>>>>>>                             using
>>>>>>>>>>>                              >
>>>>>>>>>>>                              > $ telnet
>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             23571
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>
>>>>>>>>>>>                              >
>>>>>>>>>>>                              > and check if TCP connection
>>>>>>>>>>>                             could be established.
>>>>>>>>>>>
>>>>>>>>>>>                             I was able to replicate the same
>>>>>>>>>>>                             behavior again, where it hangs
>>>>>>>>>>>                             while RCF is trying to start.
>>>>>>>>>>>                             Running this command, I see this
>>>>>>>>>>>                             in the output:
>>>>>>>>>>>
>>>>>>>>>>>                             tcp        0    0 0.0.0.0:23571
>>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>>                             <http://0.0.0.0:23571> 0.0.0.0:*
>>>>>>>>>>>                             LISTEN  18599/ta
>>>>>>>>>>>
>>>>>>>>>>>                             So it seems like it is listening
>>>>>>>>>>>                             on the correct port.
>>>>>>>>>>>                             Additionally, I was able to
>>>>>>>>>>>                             connect to the Tester machine
>>>>>>>>>>>                             from our Test Engine host using
>>>>>>>>>>>                             telnet. It printed the PID of
>>>>>>>>>>>                             the process once the connection
>>>>>>>>>>>                             was opened.
>>>>>>>>>>>
>>>>>>>>>>>                             I tried running the "ta"
>>>>>>>>>>>                             application manually on the
>>>>>>>>>>>                             command line, and it didn't
>>>>>>>>>>>                             print anything at all.
>>>>>>>>>>>                             Maybe the issue is something on
>>>>>>>>>>>                             the Test Engine side.
>>>>>>>>>>>
>>>>>>>>>>>                             On Thu, Aug 24, 2023 at 2:35 PM
>>>>>>>>>>>                             Andrew Rybchenko
>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>                             wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                 Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                  > On the tester host (which
>>>>>>>>>>>                             appears to be the Peer agent),
>>>>>>>>>>>                             there
>>>>>>>>>>>                                 are four processes that I
>>>>>>>>>>>                             see running, which look like the
>>>>>>>>>>>                             test
>>>>>>>>>>>                                 agent processes.
>>>>>>>>>>>
>>>>>>>>>>>                                 Before the next try I'd
>>>>>>>>>>>                             recommend to kill these processes.
>>>>>>>>>>>
>>>>>>>>>>>                                 Is there any firewall in the
>>>>>>>>>>>                             network or on test hosts which
>>>>>>>>>>>                             could
>>>>>>>>>>>                                 block incoming TCP
>>>>>>>>>>>                             connection to the port 23571
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             from the host
>>>>>>>>>>>                                 where you run test engine?
>>>>>>>>>>>
>>>>>>>>>>>                                 If behaviour the same on the
>>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>>                             agent is
>>>>>>>>>>>                                 kept running, could you
>>>>>>>>>>>                             check using
>>>>>>>>>>>
>>>>>>>>>>>                                 # netstat -tnlp
>>>>>>>>>>>
>>>>>>>>>>>                                 that Test Agent is listening
>>>>>>>>>>>                             on the port and try to establish
>>>>>>>>>>>                             TCP
>>>>>>>>>>>                                 connection from test agent
>>>>>>>>>>>                             using
>>>>>>>>>>>
>>>>>>>>>>>                                 $ telnet
>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             23571
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                 and check if TCP connection
>>>>>>>>>>>                             could be established.
>>>>>>>>>>>
>>>>>>>>>>>                                 Another idea is to login
>>>>>>>>>>>                             Tester under root as testing
>>>>>>>>>>>                             does, get
>>>>>>>>>>>                                 start TA command from the
>>>>>>>>>>>                             log and try it by hands without
>>>>>>>>>>>                             -n and
>>>>>>>>>>>                                 remove extra escaping.
>>>>>>>>>>>
>>>>>>>>>>>                                 # sudo
>>>>>>>>>>>                             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>
>>>>>>>>>>>                             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>                             /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>                             Peer 23571
>>>>>>>>>>>                             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>
>>>>>>>>>>>                                 Hopefully in this case test
>>>>>>>>>>>                             agent directory remains in the
>>>>>>>>>>>                             /tmp and
>>>>>>>>>>>                                 you don't need to copy it as
>>>>>>>>>>>                             testing does.
>>>>>>>>>>>                                 May be output could shed
>>>>>>>>>>>                             some light on what's going on.
>>>>>>>>>>>
>>>>>>>>>>>                                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                 On 8/24/23 17:30, Adam
>>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                                 This is the output that I
>>>>>>>>>>>>                             see in the terminal when this
>>>>>>>>>>>>                             failure
>>>>>>>>>>>>                                 occurs, after the test
>>>>>>>>>>>>                             agent binaries build and the
>>>>>>>>>>>>                             test engine
>>>>>>>>>>>>                                 starts:
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Platform default build - pass
>>>>>>>>>>>>                                 Simple RCF consistency
>>>>>>>>>>>>                             check succeeded
>>>>>>>>>>>>                             --->>> Starting Logger...done
>>>>>>>>>>>>                             --->>> Starting
>>>>>>>>>>>>                             RCF...rcf_net_engine_connect():
>>>>>>>>>>>>                             Connection timed
>>>>>>>>>>>>                                 out
>>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Then, it hangs here until I
>>>>>>>>>>>>                             kill the "te_rcf" and "te_tee"
>>>>>>>>>>>>                                 processes. I let it hang
>>>>>>>>>>>>                             for around 9 minutes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 On the tester host (which
>>>>>>>>>>>>                             appears to be the Peer agent),
>>>>>>>>>>>>                             there are
>>>>>>>>>>>>                                 four processes that I see
>>>>>>>>>>>>                             running, which look like the
>>>>>>>>>>>>                             test agent
>>>>>>>>>>>>                                 processes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 ta.Peer is an empty file.
>>>>>>>>>>>>                             I've attached the log.txt from
>>>>>>>>>>>>                             this run.
>>>>>>>>>>>>
>>>>>>>>>>>>                                  - Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                                 On Thu, Aug 24, 2023 at
>>>>>>>>>>>>                             4:22 AM Andrew Rybchenko
>>>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Yes, TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>                             is in seconds. I've double-checked
>>>>>>>>>>>>                                     that it goes to
>>>>>>>>>>>>                             'copy_timeout' in
>>>>>>>>>>>>                             ts-conf/rcf.conf.
>>>>>>>>>>>>                             Description in in
>>>>>>>>>>>>                             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>
>>>>>>>>>>>>                                     says that copy_timeout
>>>>>>>>>>>>                             is in seconds and
>>>>>>>>>>>>                             implementation in
>>>>>>>>>>>>                             lib/rcfunix/rcfunix.c passes
>>>>>>>>>>>>                             the value to select() tv_sec.
>>>>>>>>>>>>                             Theoretically select() could be
>>>>>>>>>>>>                             interrupted by signal, but I
>>>>>>>>>>>>                                     think it is unlikely here.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     I'm not sure that I
>>>>>>>>>>>>                             understand what do you mean by RCF
>>>>>>>>>>>>                             connection timeout. Does it
>>>>>>>>>>>>                             happen on TE startup when RCF
>>>>>>>>>>>>                                     starts test agents. If
>>>>>>>>>>>>                             so, TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>>>                             help. Or
>>>>>>>>>>>>                                     does it happen when
>>>>>>>>>>>>                             tests are in progress, e.g. in
>>>>>>>>>>>>                             the middle
>>>>>>>>>>>>                                     of a test. If so,
>>>>>>>>>>>>                             TE_RCFUNIX_TIMEOUT is unrelated
>>>>>>>>>>>>                             and most
>>>>>>>>>>>>                                     likely either host with
>>>>>>>>>>>>                             test agent dies or test agent
>>>>>>>>>>>>                             itself
>>>>>>>>>>>>                             crashes. It would be easier for
>>>>>>>>>>>>                             me if classify it if you share
>>>>>>>>>>>>                                     text log (log.txt, full
>>>>>>>>>>>>                             or just corresponding fragment
>>>>>>>>>>>>                             with
>>>>>>>>>>>>                                     some context). Also
>>>>>>>>>>>>                             content of ta.DPDK or ta.Peer file
>>>>>>>>>>>>                             depending on which agent has
>>>>>>>>>>>>                             problems could shed some light.
>>>>>>>>>>>>                             Corresponding files contain
>>>>>>>>>>>>                             stdout/stderr of test agents.
>>>>>>>>>>>>
>>>>>>>>>>>>                             Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On 8/23/23 17:45, Adam
>>>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     I've set up a test rig
>>>>>>>>>>>>>                             repository here, and have created
>>>>>>>>>>>>>                             configurations for our
>>>>>>>>>>>>>                             development testbed based off
>>>>>>>>>>>>>                             of the
>>>>>>>>>>>>>                             examples.
>>>>>>>>>>>>>                                     We've been able to get
>>>>>>>>>>>>>                             the test suite to run manually on
>>>>>>>>>>>>>                             Mellanox CX5 devices once.
>>>>>>>>>>>>>                             However, we are running into
>>>>>>>>>>>>>                             an issue where, when RCF starts,
>>>>>>>>>>>>>                                     the RCF connection
>>>>>>>>>>>>>                             times out very frequently. We
>>>>>>>>>>>>>                             aren't sure
>>>>>>>>>>>>>                                     why this is the case.
>>>>>>>>>>>>>                                     It works sometimes,
>>>>>>>>>>>>>                             but most of the time when we
>>>>>>>>>>>>>                             try to run
>>>>>>>>>>>>>                                     the test engine, it
>>>>>>>>>>>>>                             encounters this issue.
>>>>>>>>>>>>>                                     I've tried changing
>>>>>>>>>>>>>                             the RCF port by setting
>>>>>>>>>>>>>                             "TE_RCF_PORT=<some port
>>>>>>>>>>>>>                             number>" and rebooting the
>>>>>>>>>>>>>                             testbed
>>>>>>>>>>>>>                             machines. Neither seems to fix
>>>>>>>>>>>>>                             the issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     It also seems like the
>>>>>>>>>>>>>                             timeout takes far longer than 60
>>>>>>>>>>>>>                             seconds, even when running
>>>>>>>>>>>>>                             "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>                                     before I try to run
>>>>>>>>>>>>>                             the test suite.
>>>>>>>>>>>>>                                     I assume the unit for
>>>>>>>>>>>>>                             this variable is seconds?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     On Mon, Aug 21, 2023
>>>>>>>>>>>>>                             at 10:19 AM Adam Hassick
>>>>>>>>>>>>>                                     <ahassick@iol.unh.edu
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                             Thanks, I've cloned the
>>>>>>>>>>>>>                             example repository and will start
>>>>>>>>>>>>>                             setting up a configuration for
>>>>>>>>>>>>>                             our development testbed
>>>>>>>>>>>>>                             today. I'll let you know if I
>>>>>>>>>>>>>                             run into any difficulties
>>>>>>>>>>>>>                                         or have any
>>>>>>>>>>>>>                             questions.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                          - Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         On Sun, Aug 20,
>>>>>>>>>>>>>                             2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             I've published
>>>>>>>>>>>>>                             https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>
>>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Hopefully it
>>>>>>>>>>>>>                             will help to define your test
>>>>>>>>>>>>>                             rigs and
>>>>>>>>>>>>>                                             successfully
>>>>>>>>>>>>>                             run some tests manually. Feel
>>>>>>>>>>>>>                             free to
>>>>>>>>>>>>>                                             ask any
>>>>>>>>>>>>>                             questions and I'll answer here
>>>>>>>>>>>>>                             and try to
>>>>>>>>>>>>>                                             update
>>>>>>>>>>>>>                             documentation.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Meanwhile I'll
>>>>>>>>>>>>>                             prepare missing bits for steps
>>>>>>>>>>>>>                             (2) and
>>>>>>>>>>>>>                                             (3).
>>>>>>>>>>>>>                                             Hopefully
>>>>>>>>>>>>>                             everything is in place for
>>>>>>>>>>>>>                             step (4), but we
>>>>>>>>>>>>>                                             need to make
>>>>>>>>>>>>>                             steps (2) and (3) first.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             On 8/18/23
>>>>>>>>>>>>>                             21:40, Andrew Rybchenko wrote:
>>>>>>>>>>>>>>                             Hi Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             > I've
>>>>>>>>>>>>>>                             conferred with the rest of
>>>>>>>>>>>>>>                             the team, and we
>>>>>>>>>>>>>>                                             think it
>>>>>>>>>>>>>>                             would be best to move forward
>>>>>>>>>>>>>>                             with mainly
>>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             OK, I'll
>>>>>>>>>>>>>>                             provide the sample on Monday
>>>>>>>>>>>>>>                             for you. It is
>>>>>>>>>>>>>>                                             almost ready
>>>>>>>>>>>>>>                             right now, but I need to
>>>>>>>>>>>>>>                             double-check
>>>>>>>>>>>>>>                                             it before
>>>>>>>>>>>>>>                             publishing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Regards,
>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On 8/17/23
>>>>>>>>>>>>>>                             20:03, Adam Hassick wrote:
>>>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             I'm adding
>>>>>>>>>>>>>>>                             the CI mailing list to this
>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>                             conversation. Others in the
>>>>>>>>>>>>>>>                             community might find
>>>>>>>>>>>>>>>                                             this
>>>>>>>>>>>>>>>                             conversation valuable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             We do want
>>>>>>>>>>>>>>>                             to run testing on a regular
>>>>>>>>>>>>>>>                             basis. The
>>>>>>>>>>>>>>>                                             Jenkins
>>>>>>>>>>>>>>>                             integration will be very
>>>>>>>>>>>>>>>                             useful for us, as
>>>>>>>>>>>>>>>                                             most of our
>>>>>>>>>>>>>>>                             CI is orchestrated by Jenkins.
>>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>>                             conferred with the rest of
>>>>>>>>>>>>>>>                             the team, and we
>>>>>>>>>>>>>>>                                             think it
>>>>>>>>>>>>>>>                             would be best to move
>>>>>>>>>>>>>>>                             forward with mainly
>>>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>>>                                             If you would
>>>>>>>>>>>>>>>                             like to know anything about our
>>>>>>>>>>>>>>>                                             testbeds
>>>>>>>>>>>>>>>                             that would help you with
>>>>>>>>>>>>>>>                             creating an
>>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>>                             ts-rigs repo, I'd be happy
>>>>>>>>>>>>>>>                             to answer any
>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>                             you have.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             We have
>>>>>>>>>>>>>>>                             multiple test rigs (we call
>>>>>>>>>>>>>>>                             these
>>>>>>>>>>>>>>>                                             "DUT-tester
>>>>>>>>>>>>>>>                             pairs") that we run our
>>>>>>>>>>>>>>>                             existing
>>>>>>>>>>>>>>>                                             hardware
>>>>>>>>>>>>>>>                             testing on, with differing
>>>>>>>>>>>>>>>                             network
>>>>>>>>>>>>>>>                                             hardware and
>>>>>>>>>>>>>>>                             CPU architecture. I figured
>>>>>>>>>>>>>>>                             this might
>>>>>>>>>>>>>>>                                             be an
>>>>>>>>>>>>>>>                             important detail.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             On Thu, Aug
>>>>>>>>>>>>>>>                             17, 2023 at 11:44 AM Andrew
>>>>>>>>>>>>>>>                             Rybchenko
>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                             Greatings Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>                             happy to hear that you're
>>>>>>>>>>>>>>>                             trying to bring
>>>>>>>>>>>>>>>                                                 it up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 As I
>>>>>>>>>>>>>>>                             understand the final goal is
>>>>>>>>>>>>>>>                             to run it on
>>>>>>>>>>>>>>>                                                 regular
>>>>>>>>>>>>>>>                             basis. So, we need to make
>>>>>>>>>>>>>>>                             it properly
>>>>>>>>>>>>>>>                                                 from the
>>>>>>>>>>>>>>>                             very beginning.
>>>>>>>>>>>>>>>                                                 Bring up
>>>>>>>>>>>>>>>                             of all features consists of
>>>>>>>>>>>>>>>                             4 steps:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 1.
>>>>>>>>>>>>>>>                             Create site-specific
>>>>>>>>>>>>>>>                             repository (we call it
>>>>>>>>>>>>>>>                                                 ts-rigs)
>>>>>>>>>>>>>>>                             which contains information
>>>>>>>>>>>>>>>                             about test
>>>>>>>>>>>>>>>                                                 rigs and
>>>>>>>>>>>>>>>                             other site-specific
>>>>>>>>>>>>>>>                             information like
>>>>>>>>>>>>>>>                                                 where to
>>>>>>>>>>>>>>>                             send mails, where to store
>>>>>>>>>>>>>>>                             logs etc.
>>>>>>>>>>>>>>>                                                 It is
>>>>>>>>>>>>>>>                             required for manual
>>>>>>>>>>>>>>>                             execution as well,
>>>>>>>>>>>>>>>                                                 since
>>>>>>>>>>>>>>>                             test rigs description is
>>>>>>>>>>>>>>>                             essential. I'll
>>>>>>>>>>>>>>>                                                 return
>>>>>>>>>>>>>>>                             to the topic below.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 2. Setup
>>>>>>>>>>>>>>>                             logs storage for automated
>>>>>>>>>>>>>>>                             runs.
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                             Basically it is a disk space
>>>>>>>>>>>>>>>                             plus apache2 web
>>>>>>>>>>>>>>>                                                 server
>>>>>>>>>>>>>>>                             with few CGI scripts which
>>>>>>>>>>>>>>>                             help a lot to
>>>>>>>>>>>>>>>                                                 save
>>>>>>>>>>>>>>>                             disk space.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 3. Setup
>>>>>>>>>>>>>>>                             Bublik web application which
>>>>>>>>>>>>>>>                             provides
>>>>>>>>>>>>>>>                                                 web
>>>>>>>>>>>>>>>                             interface to view testing
>>>>>>>>>>>>>>>                             results. Same as
>>>>>>>>>>>>>>>                             https://ts-factory.io/bublik
>>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 4. Setup
>>>>>>>>>>>>>>>                             Jenkins to run tests on
>>>>>>>>>>>>>>>                             regularly,
>>>>>>>>>>>>>>>                                                 save
>>>>>>>>>>>>>>>                             logs in log storage (2) and
>>>>>>>>>>>>>>>                             import it to
>>>>>>>>>>>>>>>                                                 bublik (3).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Last few
>>>>>>>>>>>>>>>                             month we spent on our
>>>>>>>>>>>>>>>                             homework to make
>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>                             simpler to bring up
>>>>>>>>>>>>>>>                             automated execution
>>>>>>>>>>>>>>>                                                 using
>>>>>>>>>>>>>>>                             Jenkins -
>>>>>>>>>>>>>>>                             https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                             Corresponding bits in
>>>>>>>>>>>>>>>                             dpdk-ethdev-ts will be
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                             available tomorrow.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Let's
>>>>>>>>>>>>>>>                             return to the step (1).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                             Unfortunately there is no
>>>>>>>>>>>>>>>                             publicly available
>>>>>>>>>>>>>>>                                                 example
>>>>>>>>>>>>>>>                             of the ts-rigs repository since
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                             sensitive site-specific
>>>>>>>>>>>>>>>                             information is located
>>>>>>>>>>>>>>>                                                 there.
>>>>>>>>>>>>>>>                             But I'm ready to help you to
>>>>>>>>>>>>>>>                             create it
>>>>>>>>>>>>>>>                                                 for UNH.
>>>>>>>>>>>>>>>                             I see two options here:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 (A) I'll
>>>>>>>>>>>>>>>                             ask questions and based on your
>>>>>>>>>>>>>>>                                                 answers
>>>>>>>>>>>>>>>                             will create the first draft
>>>>>>>>>>>>>>>                             with my
>>>>>>>>>>>>>>>                                                 comments.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 (B) I'll
>>>>>>>>>>>>>>>                             make a template/example
>>>>>>>>>>>>>>>                             ts-rigs repo,
>>>>>>>>>>>>>>>                                                 publish
>>>>>>>>>>>>>>>                             it and you'll create UNH
>>>>>>>>>>>>>>>                             ts-rigs based
>>>>>>>>>>>>>>>                                                 on it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Of
>>>>>>>>>>>>>>>                             course, I'll help to debug
>>>>>>>>>>>>>>>                             and finally bring
>>>>>>>>>>>>>>>                                                 it up in
>>>>>>>>>>>>>>>                             any case.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 (A) is a
>>>>>>>>>>>>>>>                             bit simpler for me and you,
>>>>>>>>>>>>>>>                             but (B) is
>>>>>>>>>>>>>>>                                                 a bit
>>>>>>>>>>>>>>>                             more generic and will help
>>>>>>>>>>>>>>>                             other
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                             potential users to bring it up.
>>>>>>>>>>>>>>>                                                 We can
>>>>>>>>>>>>>>>                             combine (A)+(B). I.e. start
>>>>>>>>>>>>>>>                             from (A).
>>>>>>>>>>>>>>>                                                 What do
>>>>>>>>>>>>>>>                             you think?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>                             8/17/23 15:18, Konstantin
>>>>>>>>>>>>>>>                             Ushakov wrote:
>>>>>>>>>>>>>>>>                             Greetings Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 Thanks
>>>>>>>>>>>>>>>>                             for contacting us. I copy
>>>>>>>>>>>>>>>>                             Andrew who
>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>                             be happy to help
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>>                                                 Konstantin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                             On 16 Aug 2023, at 21:50,
>>>>>>>>>>>>>>>>>                             Adam Hassick
>>>>>>>>>>>>>>>>>                             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 I am
>>>>>>>>>>>>>>>>>                             in the process of setting
>>>>>>>>>>>>>>>>>                             up the DPDK
>>>>>>>>>>>>>>>>>                                                 Poll
>>>>>>>>>>>>>>>>>                             Mode Driver test suite as
>>>>>>>>>>>>>>>>>                             an addition to
>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>                             testing coverage for DPDK
>>>>>>>>>>>>>>>>>                             at the UNH lab.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 I have
>>>>>>>>>>>>>>>>>                             some questions about how
>>>>>>>>>>>>>>>>>                             to set the
>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>                             suite arguments.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 I have
>>>>>>>>>>>>>>>>>                             been able to configure the
>>>>>>>>>>>>>>>>>                             Test Engine
>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>                             connect to the hosts in
>>>>>>>>>>>>>>>>>                             the testbed. The
>>>>>>>>>>>>>>>>>                                                 RCF,
>>>>>>>>>>>>>>>>>                             Configurator, and Tester
>>>>>>>>>>>>>>>>>                             all begin to
>>>>>>>>>>>>>>>>>                                                 run,
>>>>>>>>>>>>>>>>>                             however the prelude of the
>>>>>>>>>>>>>>>>>                             test suite
>>>>>>>>>>>>>>>>>                                                 fails
>>>>>>>>>>>>>>>>>                             to run.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 The
>>>>>>>>>>>>>>>>>                             documentation mentions
>>>>>>>>>>>>>>>>>                             that there are
>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>                             several test parameters
>>>>>>>>>>>>>>>>>                             for the test suite,
>>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>>                             for the IUT test link MAC,
>>>>>>>>>>>>>>>>>                             etc. These
>>>>>>>>>>>>>>>>>                                                 seem
>>>>>>>>>>>>>>>>>                             like they would need to be
>>>>>>>>>>>>>>>>>                             set somewhere
>>>>>>>>>>>>>>>>>                                                 to run
>>>>>>>>>>>>>>>>>                             many of the tests.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 I see
>>>>>>>>>>>>>>>>>                             in the Test Engine
>>>>>>>>>>>>>>>>>                             documentation, there
>>>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>>>                             instructions on how to
>>>>>>>>>>>>>>>>>                             create new
>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>                             parameters for test suites
>>>>>>>>>>>>>>>>>                             in the Tester
>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>                             configuration, but there
>>>>>>>>>>>>>>>>>                             is nothing in the
>>>>>>>>>>>>>>>>>                                                 user
>>>>>>>>>>>>>>>>>                             guide or in the Tester
>>>>>>>>>>>>>>>>>                             guide for how to
>>>>>>>>>>>>>>>>>                                                 set
>>>>>>>>>>>>>>>>>                             the arguments for the
>>>>>>>>>>>>>>>>>                             parameters when
>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>                             running the test suite
>>>>>>>>>>>>>>>>>                             that I can find. I'm
>>>>>>>>>>>>>>>>>                                                 not
>>>>>>>>>>>>>>>>>                             sure if I need to write my
>>>>>>>>>>>>>>>>>                             own Tester
>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>                             config, or if I should be
>>>>>>>>>>>>>>>>>                             setting these in
>>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>>                             other way.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 How
>>>>>>>>>>>>>>>>>                             should these values be set?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>>                             also not sure what
>>>>>>>>>>>>>>>>>                             environment
>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>                             variables/arguments are
>>>>>>>>>>>>>>>>>                             strictly necessary or
>>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>>                             are optional.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 Regards,
>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>>                             Hassick*
>>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>>                             Developer
>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>>                             (603) 475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>                             Developer
>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>                             475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         -- *Adam Hassick*
>>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     --         *Adam Hassick*
>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                 -- *Adam Hassick*
>>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             -- 
>>>>>>>>>>>                             *Adam Hassick*
>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>                             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         -- 
>>>>>>>>>                         *Adam Hassick*
>>>>>>>>>                         Senior Developer
>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                     -- 
>>>>>>>                     *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 -- 
>>>>>                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 167755 bytes --]

Re: Setting up DPDK PMD Test Suite

[Konstantin Ushakov]

[-- Attachment #1: Type: text/plain, Size: 91484 bytes --]

Hi Andrew,

should we always auto-assign the tags or you don’t do it since it 
slows down (by some seconds) the TE startup?

Hi Adam,

I think I second the question from Andrew - happy to help you with the 
triage so that we get to the same baseline. Do you have a good way for 
us to share the logs? I.e. say upload to ts-factory if we add strict 
permissions system so it’s not publishing or any other way.

Thanks,
Konstantin


On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:

> Hi Adam,
>
> I've uploaded fresh testing results to ts-factory.io [1] to be on the 
> same page.
>
> I think I know why your and mine results on Intel 710 series NICs 
> differ so much. Testing results expectations database 
> (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags.  I.e. 
> expectations depends on TRC tags discovered by helper scripts when 
> testing is started. These tags identify various aspects of what is 
> tested. Ideally expectations should be written in terms of root cause 
> of the expected behaviour. If it is a driver expectations, driver tag 
> should be used. If it is HW limitation, tags with PCI IDs should be 
> used. However, it is not always easy to classify it correctly if 
> you're not involved in driver development. So, in order case 
> expectations for 710's Intel are filled in in terms of PCI IDs. I 
> guess PCI ID differ in your case and that's why expectations filled in 
> for my NIC do not apply to your runs.
>
> Just try to add the following option when you run on your 710's Intel 
> in order to mimic mine and see if it helps to achieve better pass 
> rate.
> --trc-tag=pci-8086-1572
>
> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for 
> --tester-dial option. It should have better coverage now.
>
> Andrew.
>
> [1] 
> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>
> On 9/13/23 18:45, Andrew Rybchenko wrote:
>> Hi Adam,
>>
>> I've pushed new TE tag v1.20.0 which supported a new command-line 
>> option --tester-dial=NUM where NUM is from 0 to 100. it allows to 
>> choose percentage of tests to run. If you want stable set, you should 
>> pass --tester-random-seed=0 (or other integer). It is the first 
>> sketch and we have plans to improve it, but feedback would be 
>> welcome.
>>
>>> Is it needed on the tester?
>>
>> It is hard to say if it is strictly required for simple tests. 
>> However, it is better to update Tester as well, since performance 
>> tests run DPDK on Tester as well.
>>
>>> Are there any other manual setup steps for these devices that I 
>>> might be missing?
>>
>> I don't remember anything else.
>>
>> I think it is better to get down to details and take a look at logs. 
>> I'm ready to help with it and explain what's happening there. May be 
>> it will help to understand if it is a problem with 
>> setup/configuration.
>>
>> Text logs are not very convenient. Ideally logs should be imported to 
>> bublik, however, manual runs do not provide all required artifacts 
>> right now (Jenkins jobs generate all required artifacts).
>> Other option is 'tmp_raw_log' file (should be packed to make it 
>> smaller) which could be converted to various log formats.
>> Would it be OK for you if I import your logs to bublik at 
>> ts-factory.io? Or is it a problem that it is publicly available?
>> Would it help if we add authentication and access control there?
>>
>> Andrew.
>>
>> On 9/8/23 17:57, Adam Hassick wrote:
>>> Hi Andrew,
>>>
>>> I have a couple questions about needed setup of the NICs for the 
>>> ethdev test suite.
>>>
>>> Our MCX5s and XL710s are failing the checkup tests. The pass rate 
>>> appears to be much worse on the XL710s (40 of 73 tests failed, 3 
>>> passed unexpectedly).
>>>
>>> For the XL710s, I've updated the driver and NVM versions to match 
>>> the minimum supported versions in the compatibility matrix found on 
>>> the DPDK documentation. This did not change the failure rate much.
>>> For the MCX5s, I've installed the latest LTS version of the OFED 
>>> bifurcated driver on the DUT. Is it needed on the tester?
>>>
>>> Are there any other manual setup steps for these devices that I 
>>> might be missing?
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick 
>>> <ahassick@iol.unh.edu> wrote:
>>>
>>>     Hi Andrew,
>>>
>>>     Yes, I copied the X710 configs to set up XL710 configs. I 
>>> changed
>>>     the environment variable names from the X710 suffix to XL710
>>>     suffix in the script, and forgot to change them in the
>>>     corresponding environment file.
>>>     That fixed the issue.
>>>
>>>     I got the checkup tests working on the XL710 now. Most of them
>>>     are failing, which leads me to believe this is an issue with our
>>>     testbed. Based on the DPDK documentation for i40e, the firmware
>>>     and driver versions are much older than what DPDK 22.11 LTS and
>>>     main prefer, so I'll try updating those.
>>>
>>>     For now I'm working on getting the XL710 checkup tests passing,
>>>     and will pick up getting the E810 configured properly next. I'll
>>>     let you know if I run into any more issues in relation to the
>>>     test engine.
>>>
>>>     Thanks,
>>>     Adam
>>>
>>>     On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>         Hi Adam,
>>>
>>>         On 9/5/23 18:01, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         The compilation warning issue is now resolved. Again, thank
>>>>         you guys for fixing this for us. I can run the tests on the
>>>>         Mellanox CX5s again, however I'm running into a couple new
>>>>         issues with running the prologues on the Intel cards.
>>>>
>>>>         When running testing on the Intel XL710s, I see this error
>>>>         appear in the log:
>>>>
>>>>             ERROR  prologue  Environment LIB  14:16:13.650
>>>>             Too few networks in available configuration (0) in
>>>>             comparison with required (1)
>>>>
>>>>
>>>>         This seems like a trivial configuration error, perhaps this
>>>>         is something I need to set up in ts-rigs. I briefly 
>>>> searched
>>>>         through the examples there and didn't see any mention of 
>>>> how
>>>>         to set up a network.
>>>>         I will attach this log just in case you need more 
>>>> information.
>>>
>>>         Unfortunately logs are insufficient to understand it. I've
>>>         pushed new tag to TE v1.19.0 which add log message with TE_*
>>>         environment variables.
>>>         Most likely something is wrong with variables which are used
>>>         as conditions when available networks are defined in
>>>         ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>         TE_PCI_INSTANCE_IUT_TST1
>>>         TE_PCI_INSTANCE_IUT_TST1a
>>>         TE_PCI_INSTANCE_TST1a_IUT
>>>         TE_PCI_INSTANCE_TST1_IUT
>>>         My guess it that you change naming a bit, but script like
>>>         ts-rigs-sample/scripts/iut.h1-x710 is not included or not
>>>         updated.
>>>
>>>>         There is a different error when running on the Intel E810s.
>>>>         It appears to me like it starts DPDK, does some
>>>>         configuration inside DPDK and on the device, and then fails
>>>>         to bring the device back up. Since this error seems very
>>>>         non-trivial, I will also attach this log.
>>>
>>>         This one is a bit simpler. Few lines after the first ERROR 
>>> in
>>>         log I see the following:
>>>         WARN  RCF  DPDK  13:06:00.144
>>>         ice_program_hw_rx_queue(): currently package doesn't support
>>>         RXDID (22)
>>>         ice_rx_queue_start(): fail to program RX queue 0
>>>         ice_dev_start(): fail to start Rx queue 0
>>>         Device with port_id=0 already stopped
>>>
>>>         It is stdout/stderr from test agent which runs DPDK. Same
>>>         logs in plain format are available in ta.DPDK file.
>>>         I'm not an expert here, but I vaguely remember that E810
>>>         requires correct firmware and DDP to be loaded.
>>>         There is some information in dpdk/doc/guides/nics/ice.rst.
>>>
>>>         You can try to add --dev-args=safe-mode-support=1
>>>         command-line option described there.
>>>
>>>         Hope it helps,
>>>         Andrew.
>>>
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko
>>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>             Hi Adam,
>>>>
>>>>             On 8/31/23 22:38, Adam Hassick wrote:
>>>>>             Hi Andrew,
>>>>>
>>>>>             I have one additional question as well: Does the test
>>>>>             engine support running tests on two ARMv8 test agents?
>>>>>
>>>>>                 1. We'll sort out warnings this week. Thanks for
>>>>>                 heads up.
>>>>>
>>>>>
>>>>>             Great. Let me know when that's fixed.
>>>>
>>>>             Done. We also fixed a number of warnings in TE.
>>>>             Also we fixed root test package name to be consistent
>>>>             with the repository name.
>>>>
>>>>>                 Support for old LTS branches was dropped some time
>>>>>                 ago, but in the future it is definitely possible 
>>>>> to
>>>>>                 keep it for new LTS branches. I think 22.11 is
>>>>>                 supported, but I'm not sure about older LTS 
>>>>> releases.
>>>>>
>>>>>
>>>>>             Good to know.
>>>>>
>>>>>                 2. You can add command-line option --sanity to run
>>>>>                 tests marked with TEST_HARNESS_SANITY requirement
>>>>>                 (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>>>                 TEST_HARNESS_SANITY dpdk-ethdev-ts to see which
>>>>>                 tests are marked). Yes, there is a space for
>>>>>                 terminology improvement here. We'll do it.
>>>>>
>>>>
>>>>             Done. Now it is called --checkup.
>>>>
>>>>>
>>>>>                 Also it takes a lot of time because of failures 
>>>>> and
>>>>>                 tests which wait for some timeout.
>>>>>
>>>>>
>>>>>             That makes sense to me. We'll use the time to complete
>>>>>             tests on virtio or the Intel devices as a reference 
>>>>> for
>>>>>             how long the tests really take to complete.
>>>>>             We will explore the possibility of periodically 
>>>>> running
>>>>>             the sanity tests for patches.
>>>>
>>>>             I'll double-check and let you know how long entire TS
>>>>             runs on Intel X710, E810, Mellanox CX5 and virtio net.
>>>>             Just to ensure that time observed in your case looks 
>>>> the
>>>>             same.
>>>>
>>>>>
>>>>>                 The test harness can provide coverage reports 
>>>>> based
>>>>>                 on gcov, but I'm not sure what you mean by a 
>>>>> "dial"
>>>>>                 to control test coverage. Provided reports are
>>>>>                 rather for human to analyze.
>>>>>
>>>>>
>>>>>             The general idea is to have some kind of parameter on
>>>>>             the test suite, which could be an integer ranging from
>>>>>             zero to ten, that controls how many tests are run 
>>>>> based
>>>>>             on how important the test is.
>>>>>
>>>>>             Similar to how some command line interfaces provide a
>>>>>             verbosity level parameter (some number of "-v"
>>>>>             arguments) to control the importance of the 
>>>>> information
>>>>>             in the log.
>>>>>             The verbosity level zero only prints very important 
>>>>> log
>>>>>             messages, while ten prints everything.
>>>>>
>>>>>             In much the same manner as above, this "dial" 
>>>>> parameter
>>>>>             controls what tests are run and with what parameters
>>>>>             based on how important those tests and test parameter
>>>>>             combinations are.
>>>>>             Coverage Level zero tells the suite to run a very 
>>>>> basic
>>>>>             set of important tests, with minimal parameterization.
>>>>>             This mode would take only ~5-10 minutes to run.
>>>>>             In contrast, Coverage Level ten includes all the edge
>>>>>             cases, every combination of test parameters, 
>>>>> everything
>>>>>             the test suite can do, which takes the normal several
>>>>>             hours to run.
>>>>>             The values 1 - 9 are between those two extremes,
>>>>>             allowing the user to get a gradient of test coverage 
>>>>> in
>>>>>             the results and to limit the running time.
>>>>>
>>>>>             Then we could, for example, run the "run.sh" with a
>>>>>             level of 2 or 3 for incoming patches that need quick
>>>>>             results, and with a level of 10 for the less often run
>>>>>             periodic tests performed on main or LTS branches.
>>>>
>>>>             Understood now. Thanks a lot for the idea. We'll 
>>>> discuss
>>>>             it and come back.
>>>>
>>>>>                 3. Yes, really many tests on Mellanox CX5 NICs
>>>>>                 report unexpected testing results. Unfortunately 
>>>>> it
>>>>>                 is time consuming to fill in expectations database
>>>>>                 since it is necessary to analyze testing results
>>>>>                 and classify if it is a bug or just acceptable
>>>>>                 behaviour aspect.
>>>>>
>>>>>                 Bublik allows to compare results of two runs. It 
>>>>> is
>>>>>                 useful for human, but still not good for 
>>>>> automation.
>>>>>
>>>>>                 I have local patch for mlx5 driver which reports 
>>>>> Tx
>>>>>                 ring size maximum. It makes pass rate higher. It 
>>>>> is
>>>>>                 a problem for test harness that mlx5 does not
>>>>>                 report limits right now.
>>>>>
>>>>>                 Pass rate on Intel X710 is about 92% on my test
>>>>>                 rig. Pass rate on virtio net is 99% right now and
>>>>>                 could be done 100% easily (just one thing to fix 
>>>>> in
>>>>>                 expectations).
>>>>>
>>>>>                 I think logs storage setup is essential for logs
>>>>>                 analysis. Of course, you can request HTML logs 
>>>>> when
>>>>>                 you run tests (--log-html=html) or generate after
>>>>>                 run using dpdk-ethdev-ts/scripts/html-log.sh and
>>>>>                 open index.html in a browser, but logs storage
>>>>>                 makes it more convenient.
>>>>>
>>>>>
>>>>>             We are interested in setting up Bublik, potentially as
>>>>>             an externally-facing component, once we have our
>>>>>             process of running the test suite stabilized.
>>>>>             Once we are able to run the test suite again, I'll see
>>>>>             what the pass rate is on our other hardware.
>>>>>             Good to know that it isn't an issue with our dev
>>>>>             testbed causing the high fail rate.
>>>>>
>>>>>             For Intel hardware, we have an XL710 and an Intel
>>>>>             E810-C in our development testbed. Although they are
>>>>>             slightly different devices, ideally the pass rate will
>>>>>             be identical or similar. I have yet to set up a VM 
>>>>> pair
>>>>>             for virtio, but we will soon.
>>>>>
>>>>>                 Latest version of test-environment has examples of
>>>>>                 our CGI scripts which we use for log storage (see
>>>>>                 tools/log_server/README.md).
>>>>>
>>>>>                 Also all bits for Jenkins setup are available. See
>>>>>                 dpdk-ethdev-ts/jenkins/README.md and examples of
>>>>>                 jenkins files in ts-rigs-sample.
>>>>>
>>>>>
>>>>>             Jenkins integration, setting up production rig
>>>>>             configurations, and permanent log storage will be our
>>>>>             next steps once I am able to run the tests again.
>>>>>             Unless there is an easy way to have meson not pass
>>>>>             "-Werror" into GCC. Then I would be able to run the
>>>>>             test suite.
>>>>
>>>>             Hopefully it is resolved now.
>>>>
>>>>             I thought a bit more about your usecase for Jenkins. 
>>>> I'm
>>>>             not 100% sure that existing pipelines are convenient 
>>>> for
>>>>             your usecase.
>>>>             Fill free to ask questions when you are on it.
>>>>
>>>>             Thanks,
>>>>             Andrew.
>>>>
>>>>>
>>>>>             Thanks,
>>>>>             Adam
>>>>>
>>>>>
>>>>>                 On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 That fix seems to have resolved the issue, thanks
>>>>>>                 for the quick turnaround time on that patch.
>>>>>>                 Now that we have the RCF timeout issue resolved,
>>>>>>                 there are a few other questions and issues that 
>>>>>> we
>>>>>>                 have about the tests themselves.
>>>>>>
>>>>>>                 1. The test suite fails to build with a couple
>>>>>>                 warnings.
>>>>>>
>>>>>>                 Below is the stderr log from compilation:
>>>>>>
>>>>>>                     FAILED:
>>>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>>                     -I../../lib
>>>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>                     -fdiagnostics-color=always -pipe
>>>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>>                     -fPIC -MD -MQ
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -MF
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>                     -o
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>>>                     ‘test_create_traffic_generator_params’:
>>>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>>>                     not a string literal and no format arguments
>>>>>>                     [-Werror=format-security]
>>>>>>                     5577 |     rc = te_kvpair_add(result, buf, 
>>>>>> mode);
>>>>>>                     |     ^~
>>>>>>                     cc1: all warnings being treated as errors
>>>>>>                     ninja: build stopped: subcommand failed.
>>>>>>                     ninja: Entering directory `.'
>>>>>>                     FAILED:
>>>>>>                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>                     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>>                     -I../../lib
>>>>>>                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>                     -fdiagnostics-color=always -pipe
>>>>>>                     -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>                     -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>>                     -fPIC -MD -MQ
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -MF
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>                     -o
>>>>>>                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>                     ../../lib/dpdk_pmd_ts.c: In function
>>>>>>                     ‘test_create_traffic_generator_params’:
>>>>>>                     ../../lib/dpdk_pmd_ts.c:5577:5: error: format
>>>>>>                     not a string literal and no format arguments
>>>>>>                     [-Werror=format-security]
>>>>>>                     5577 |     rc = te_kvpair_add(result, buf, 
>>>>>> mode);
>>>>>>                     |     ^~
>>>>>>                     cc1: all warnings being treated as errors
>>>>>>
>>>>>>
>>>>>>                 This error wasn't occurring last week, which was
>>>>>>                 the last time I ran the tests.
>>>>>>                 The TE host and the DUT have GCC v9.4.0 
>>>>>> installed,
>>>>>>                 and the tester has GCC v11.4.0 installed, if this
>>>>>>                 information is helpful.
>>>>>>
>>>>>>                 2. On the Mellanox CX5s, there are over 6,000
>>>>>>                 tests run, which collectively take around 9 
>>>>>> hours.
>>>>>>                 Is it possible, and would it make sense, to lower
>>>>>>                 the test coverage and have the test suite run 
>>>>>> faster?
>>>>>>
>>>>>>                 For some context, we run immediate testing on
>>>>>>                 incoming patches for DPDK main and development
>>>>>>                 branches, as well as periodic test runs on the
>>>>>>                 main, stable, and LTS branches.
>>>>>>                 For us to consider including this test suite as
>>>>>>                 part of our immediate testing on patches, we 
>>>>>> would
>>>>>>                 have to reduce the test coverage to the most
>>>>>>                 important tests.
>>>>>>                 This is primarily to reduce the testing time to,
>>>>>>                 for example, less than 30 minutes. Testing on
>>>>>>                 patches can't take too long because the lab can
>>>>>>                 receive numerous patches each day, which each
>>>>>>                 require individual testing runs.
>>>>>>
>>>>>>                 At what frequency we run these tests, and on 
>>>>>> what,
>>>>>>                 still needs to be discussed with the DPDK
>>>>>>                 community, but it would be nice to know if the
>>>>>>                 test suite had a "dial" to control the testing
>>>>>>                 coverage.
>>>>>>
>>>>>>                 3. We see a lot of test failures on our Mellanox
>>>>>>                 CX5 NICs. Around 2,300 of ~6,600 tests passed. Is
>>>>>>                 there anything we can do to diagnose these test
>>>>>>                 failures?
>>>>>>
>>>>>>                 Thanks,
>>>>>>                 Adam
>>>>>>
>>>>>>
>>>>>>                 On Tue, Aug 29, 2023 at 8:07 AM Andrew 
>>>>>> Rybchenko
>>>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     I've pushed the fix in main branch and a new
>>>>>>                     tag v1.18.1. It should solve the problem with
>>>>>>                     IPv6 address from DNS.
>>>>>>
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>                     Hi Adam,
>>>>>>>
>>>>>>>                     > Does the test engine prefer to use IPv6
>>>>>>>                     over IPv4 for initiating the RCF connection
>>>>>>>                     to the test bed hosts? And if so, is there a
>>>>>>>                     way to force it to use IPv4?
>>>>>>>
>>>>>>>                     Brilliant idea. If DNS returns both IPv4 and
>>>>>>>                     IPv6 addresses in your case, I guess it is
>>>>>>>                     the root cause of the problem.
>>>>>>>                     Of course, it is TE problem since I see
>>>>>>>                     really weird code in
>>>>>>>                     lib/comm_net_engine/comm_net_engine.c line 
>>>>>>> 135.
>>>>>>>
>>>>>>>                     I've pushed fix to the branch
>>>>>>>                     user/arybchik/fix_ipv4_only in
>>>>>>>                     ts-factory/test-environment repository.
>>>>>>>                     Please, try.
>>>>>>>
>>>>>>>                     It is late night fix with minimal testing 
>>>>>>> and
>>>>>>>                     no review. I'll pass it through review
>>>>>>>                     process tomorrow and
>>>>>>>                     hopefully it will be released in one-two 
>>>>>>> days.
>>>>>>>
>>>>>>>                     Andrew.
>>>>>>>
>>>>>>>                     On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>                     Hi Andrew,
>>>>>>>>
>>>>>>>>                     We have yet to notice a distinct pattern
>>>>>>>>                     with the failures. Sometimes, the RCF will
>>>>>>>>                     start and connect without issue a few times
>>>>>>>>                     in a row before failing to connect again.
>>>>>>>>                     Once the issue begins to occur, neither
>>>>>>>>                     rebooting all of the hosts (test engine VM,
>>>>>>>>                     tester, IUT) or deleting all of the build
>>>>>>>>                     directories (suites, agents, inst) and
>>>>>>>>                     rebooting the hosts afterward resolves the
>>>>>>>>                     issue. When it begins working again seems
>>>>>>>>                     very arbitrary to us.
>>>>>>>>
>>>>>>>>                     I do usually try to terminate the test
>>>>>>>>                     engine with Ctrl+C, but when it hangs while
>>>>>>>>                     trying to start RCF, that does not work.
>>>>>>>>
>>>>>>>>                     Does the test engine prefer to use IPv6 
>>>>>>>> over
>>>>>>>>                     IPv4 for initiating the RCF connection to
>>>>>>>>                     the test bed hosts? And if so, is there a
>>>>>>>>                     way to force it to use IPv4?
>>>>>>>>
>>>>>>>>                      - Adam
>>>>>>>>
>>>>>>>>                     On Fri, Aug 25, 2023 at 1:35 PM Andrew
>>>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>                         > I'll double-check test engine on
>>>>>>>>                         Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>
>>>>>>>>                         Done. It works fine for me without any
>>>>>>>>                         issues.
>>>>>>>>
>>>>>>>>                         Have you noticed any pattern when it
>>>>>>>>                         works or does not work?
>>>>>>>>                         May be it is a problem of not clean
>>>>>>>>                         state after termination?
>>>>>>>>                         Does it work fine the first time after
>>>>>>>>                         DUTs reboot?
>>>>>>>>                         How do you terminate testing? It should
>>>>>>>>                         be done using Ctrl+C in terminal where
>>>>>>>>                         you execute run.sh command.
>>>>>>>>                          In this case it should shutdown
>>>>>>>>                         gracefully and close all test agents 
>>>>>>>> and
>>>>>>>>                         engine applications.
>>>>>>>>
>>>>>>>>                         (I'm trying to understand why you've
>>>>>>>>                         seen many test agent processes. It
>>>>>>>>                         should not happen.)
>>>>>>>>
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/25/23 17:41, Andrew Rybchenko 
>>>>>>>> wrote:
>>>>>>>>>                         On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>>                         Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                         Two of our systems (the Test Engine
>>>>>>>>>>                         runner and the DUT host) are running
>>>>>>>>>>                         Ubuntu 20.04 LTS, however this 
>>>>>>>>>> morning
>>>>>>>>>>                         I noticed that the tester system (the
>>>>>>>>>>                         one having issues) is running Ubuntu
>>>>>>>>>>                         22.04 LTS.
>>>>>>>>>>                         This could be the source of the
>>>>>>>>>>                         problem. I encountered a dependency
>>>>>>>>>>                         issue trying to run the Test Engine 
>>>>>>>>>> on
>>>>>>>>>>                         22.04 LTS, so I downgraded the 
>>>>>>>>>> system.
>>>>>>>>>>                         Since the tester is also the host
>>>>>>>>>>                         having connection issues, I will try
>>>>>>>>>>                         downgrading that system to 20.04, and
>>>>>>>>>>                         see if that changes anything.
>>>>>>>>>
>>>>>>>>>                         Unlikely, but who knows. We run tests
>>>>>>>>>                         (DUTs) on Ubuntu 20.04, Ubuntu 22.04,
>>>>>>>>>                         Ubuntu 22.10, Ubuntu 23.04, Debian 11
>>>>>>>>>                         and Fedora 38 every night.
>>>>>>>>>                         Right now Debian 11 is used for test
>>>>>>>>>                         engine in nightly regressions.
>>>>>>>>>
>>>>>>>>>                         I'll double-check test engine on 
>>>>>>>>> Ubuntu
>>>>>>>>>                         20.04 and Ubuntu 22.04.
>>>>>>>>>
>>>>>>>>>>                         I did try passing in the "--vg-rcf"
>>>>>>>>>>                         argument to the run.sh script of the
>>>>>>>>>>                         test suite after installing valgrind,
>>>>>>>>>>                         but there was no additional output
>>>>>>>>>>                         that I saw.
>>>>>>>>>
>>>>>>>>>                         Sorry, I should valgrind output should
>>>>>>>>>                         be in valgrind.te_rcf (direction where
>>>>>>>>>                         you run test engine).
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         I will try pulling in the changes
>>>>>>>>>>                         you've pushed up, and will see if 
>>>>>>>>>> that
>>>>>>>>>>                         fixes anything.
>>>>>>>>>>
>>>>>>>>>>                         Thanks,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         On Fri, Aug 25, 2023 at 9:57 AM 
>>>>>>>>>> Andrew
>>>>>>>>>>                         Rybchenko
>>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> 
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>                             Hello Adam,
>>>>>>>>>>
>>>>>>>>>>                             On 8/24/23 23:54, Andrew 
>>>>>>>>>> Rybchenko
>>>>>>>>>>                             wrote:
>>>>>>>>>>>                             I'd like to try to repeat the
>>>>>>>>>>>                             problem locally. Which Linux
>>>>>>>>>>>                             distro is running on test engine
>>>>>>>>>>>                             and agents?
>>>>>>>>>>>
>>>>>>>>>>>                             In fact I know one problem with
>>>>>>>>>>>                             Debian 12 and Fedora 38 and we 
>>>>>>>>>>> have
>>>>>>>>>>>                             patch in review to fix it,
>>>>>>>>>>>                             however, the behaviour is
>>>>>>>>>>>                             different in
>>>>>>>>>>>                             this case, so it is unlike the
>>>>>>>>>>>                             same problem.
>>>>>>>>>>
>>>>>>>>>>                             I've just published a new tag
>>>>>>>>>>                             which fixes known test engine 
>>>>>>>>>> side
>>>>>>>>>>                             problems on Debian 12 and Fedora 
>>>>>>>>>> 38.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             One more idea is to install
>>>>>>>>>>>                             valgrind on the test engine host 
>>>>>>>>>>> and
>>>>>>>>>>>                             run with option --vg-rcf to 
>>>>>>>>>>> check
>>>>>>>>>>>                             if something weird is happening.
>>>>>>>>>>>
>>>>>>>>>>>                             What I don't understand right 
>>>>>>>>>>> now
>>>>>>>>>>>                             is why I see just one failed 
>>>>>>>>>>> attempt
>>>>>>>>>>>                             to connect in your log.txt and
>>>>>>>>>>>                             then Logger shutdown after 9
>>>>>>>>>>>                             minutes.
>>>>>>>>>>>
>>>>>>>>>>>                             Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                             On 8/24/23 23:29, Adam Hassick
>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>                              > Is there any firewall in 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             network or on test hosts which
>>>>>>>>>>>>                             could block incoming TCP
>>>>>>>>>>>>                             connection to the port 23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             from the host where you run 
>>>>>>>>>>>> test
>>>>>>>>>>>>                             engine?
>>>>>>>>>>>>
>>>>>>>>>>>>                             Our test engine host and the
>>>>>>>>>>>>                             testbed are on the same subnet.
>>>>>>>>>>>>                             The connection does work 
>>>>>>>>>>>> sometimes.
>>>>>>>>>>>>
>>>>>>>>>>>>                              > If behaviour the same on 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>>>                             agent is kept running, could 
>>>>>>>>>>>> you
>>>>>>>>>>>>                             check using
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > # netstat -tnlp
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > that Test Agent is 
>>>>>>>>>>>> listening
>>>>>>>>>>>>                             on the port and try to 
>>>>>>>>>>>> establish
>>>>>>>>>>>>                             TCP connection from test agent
>>>>>>>>>>>>                             using
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > $ telnet
>>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>                              >
>>>>>>>>>>>>                              > and check if TCP connection
>>>>>>>>>>>>                             could be established.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I was able to replicate the 
>>>>>>>>>>>> same
>>>>>>>>>>>>                             behavior again, where it hangs
>>>>>>>>>>>>                             while RCF is trying to start.
>>>>>>>>>>>>                             Running this command, I see 
>>>>>>>>>>>> this
>>>>>>>>>>>>                             in the output:
>>>>>>>>>>>>
>>>>>>>>>>>>                             tcp        0    0 
>>>>>>>>>>>> 0.0.0.0:23571
>>>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>>>                             <http://0.0.0.0:23571>
>>>>>>>>>>>>                             <http://0.0.0.0:23571> 
>>>>>>>>>>>> 0.0.0.0:*
>>>>>>>>>>>>                             LISTEN  18599/ta
>>>>>>>>>>>>
>>>>>>>>>>>>                             So it seems like it is 
>>>>>>>>>>>> listening
>>>>>>>>>>>>                             on the correct port.
>>>>>>>>>>>>                             Additionally, I was able to
>>>>>>>>>>>>                             connect to the Tester machine
>>>>>>>>>>>>                             from our Test Engine host using
>>>>>>>>>>>>                             telnet. It printed the PID of
>>>>>>>>>>>>                             the process once the connection
>>>>>>>>>>>>                             was opened.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I tried running the "ta"
>>>>>>>>>>>>                             application manually on the
>>>>>>>>>>>>                             command line, and it didn't
>>>>>>>>>>>>                             print anything at all.
>>>>>>>>>>>>                             Maybe the issue is something on
>>>>>>>>>>>>                             the Test Engine side.
>>>>>>>>>>>>
>>>>>>>>>>>>                             On Thu, Aug 24, 2023 at 
>>>>>>>>>>>> 2:35 PM
>>>>>>>>>>>>                             Andrew Rybchenko
>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                  > On the tester host 
>>>>>>>>>>>> (which
>>>>>>>>>>>>                             appears to be the Peer agent),
>>>>>>>>>>>>                             there
>>>>>>>>>>>>                                 are four processes that 
>>>>>>>>>>>> I
>>>>>>>>>>>>                             see running, which look like 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             test
>>>>>>>>>>>>                                 agent processes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Before the next try I'd
>>>>>>>>>>>>                             recommend to kill these 
>>>>>>>>>>>> processes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Is there any firewall in 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             network or on test hosts which
>>>>>>>>>>>>                             could
>>>>>>>>>>>>                                 block incoming TCP
>>>>>>>>>>>>                             connection to the port 23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             from the host
>>>>>>>>>>>>                                 where you run test 
>>>>>>>>>>>> engine?
>>>>>>>>>>>>
>>>>>>>>>>>>                                 If behaviour the same on 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             next try and you see that test
>>>>>>>>>>>>                             agent is
>>>>>>>>>>>>                                 kept running, could you
>>>>>>>>>>>>                             check using
>>>>>>>>>>>>
>>>>>>>>>>>>                                 # netstat -tnlp
>>>>>>>>>>>>
>>>>>>>>>>>>                                 that Test Agent is 
>>>>>>>>>>>> listening
>>>>>>>>>>>>                             on the port and try to 
>>>>>>>>>>>> establish
>>>>>>>>>>>>                             TCP
>>>>>>>>>>>>                                 connection from test 
>>>>>>>>>>>> agent
>>>>>>>>>>>>                             using
>>>>>>>>>>>>
>>>>>>>>>>>>                                 $ telnet
>>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             23571
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                 and check if TCP 
>>>>>>>>>>>> connection
>>>>>>>>>>>>                             could be established.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Another idea is to login
>>>>>>>>>>>>                             Tester under root as testing
>>>>>>>>>>>>                             does, get
>>>>>>>>>>>>                                 start TA command from 
>>>>>>>>>>>> the
>>>>>>>>>>>>                             log and try it by hands without
>>>>>>>>>>>>                             -n and
>>>>>>>>>>>>                                 remove extra escaping.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 # sudo
>>>>>>>>>>>>                             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>
>>>>>>>>>>>>                             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>                             /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>                             Peer 23571
>>>>>>>>>>>>                             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Hopefully in this case 
>>>>>>>>>>>> test
>>>>>>>>>>>>                             agent directory remains in the
>>>>>>>>>>>>                             /tmp and
>>>>>>>>>>>>                                 you don't need to copy 
>>>>>>>>>>>> it as
>>>>>>>>>>>>                             testing does.
>>>>>>>>>>>>                                 May be output could shed
>>>>>>>>>>>>                             some light on what's going on.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 On 8/24/23 17:30, Adam
>>>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 This is the output that 
>>>>>>>>>>>>> I
>>>>>>>>>>>>>                             see in the terminal when this
>>>>>>>>>>>>>                             failure
>>>>>>>>>>>>>                                 occurs, after the test
>>>>>>>>>>>>>                             agent binaries build and the
>>>>>>>>>>>>>                             test engine
>>>>>>>>>>>>>                                 starts:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 Platform default build 
>>>>>>>>>>>>> - pass
>>>>>>>>>>>>>                                 Simple RCF consistency
>>>>>>>>>>>>>                             check succeeded
>>>>>>>>>>>>>                             --->>> Starting Logger...done
>>>>>>>>>>>>>                             --->>> Starting
>>>>>>>>>>>>>                             RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>                             Connection timed
>>>>>>>>>>>>>                                 out
>>>>>>>>>>>>>                             iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 Then, it hangs here 
>>>>>>>>>>>>> until I
>>>>>>>>>>>>>                             kill the "te_rcf" and "te_tee"
>>>>>>>>>>>>>                                 processes. I let it 
>>>>>>>>>>>>> hang
>>>>>>>>>>>>>                             for around 9 minutes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 On the tester host 
>>>>>>>>>>>>> (which
>>>>>>>>>>>>>                             appears to be the Peer agent),
>>>>>>>>>>>>>                             there are
>>>>>>>>>>>>>                                 four processes that I 
>>>>>>>>>>>>> see
>>>>>>>>>>>>>                             running, which look like the
>>>>>>>>>>>>>                             test agent
>>>>>>>>>>>>>                                 processes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 ta.Peer is an empty 
>>>>>>>>>>>>> file.
>>>>>>>>>>>>>                             I've attached the log.txt from
>>>>>>>>>>>>>                             this run.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                  - Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 On Thu, Aug 24, 2023 at
>>>>>>>>>>>>>                             4:22 AM Andrew Rybchenko
>>>>>>>>>>>>>                                 
>>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Yes, 
>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>                             is in seconds. I've 
>>>>>>>>>>>>> double-checked
>>>>>>>>>>>>>                                     that it goes to
>>>>>>>>>>>>>                             'copy_timeout' in
>>>>>>>>>>>>>                             ts-conf/rcf.conf.
>>>>>>>>>>>>>                             Description in in
>>>>>>>>>>>>>                             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     says that 
>>>>>>>>>>>>> copy_timeout
>>>>>>>>>>>>>                             is in seconds and
>>>>>>>>>>>>>                             implementation in
>>>>>>>>>>>>>                             lib/rcfunix/rcfunix.c passes
>>>>>>>>>>>>>                             the value to select() tv_sec.
>>>>>>>>>>>>>                             Theoretically select() could 
>>>>>>>>>>>>> be
>>>>>>>>>>>>>                             interrupted by signal, but I
>>>>>>>>>>>>>                                     think it is 
>>>>>>>>>>>>> unlikely here.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     I'm not sure 
>>>>>>>>>>>>> that I
>>>>>>>>>>>>>                             understand what do you mean by 
>>>>>>>>>>>>> RCF
>>>>>>>>>>>>>                             connection timeout. Does it
>>>>>>>>>>>>>                             happen on TE startup when RCF
>>>>>>>>>>>>>                                     starts test 
>>>>>>>>>>>>> agents. If
>>>>>>>>>>>>>                             so, TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>>>>                             help. Or
>>>>>>>>>>>>>                                     does it happen 
>>>>>>>>>>>>> when
>>>>>>>>>>>>>                             tests are in progress, e.g. in
>>>>>>>>>>>>>                             the middle
>>>>>>>>>>>>>                                     of a test. If 
>>>>>>>>>>>>> so,
>>>>>>>>>>>>>                             TE_RCFUNIX_TIMEOUT is 
>>>>>>>>>>>>> unrelated
>>>>>>>>>>>>>                             and most
>>>>>>>>>>>>>                                     likely either 
>>>>>>>>>>>>> host with
>>>>>>>>>>>>>                             test agent dies or test agent
>>>>>>>>>>>>>                             itself
>>>>>>>>>>>>>                             crashes. It would be easier 
>>>>>>>>>>>>> for
>>>>>>>>>>>>>                             me if classify it if you share
>>>>>>>>>>>>>                                     text log 
>>>>>>>>>>>>> (log.txt, full
>>>>>>>>>>>>>                             or just corresponding fragment
>>>>>>>>>>>>>                             with
>>>>>>>>>>>>>                                     some context). 
>>>>>>>>>>>>> Also
>>>>>>>>>>>>>                             content of ta.DPDK or ta.Peer 
>>>>>>>>>>>>> file
>>>>>>>>>>>>>                             depending on which agent has
>>>>>>>>>>>>>                             problems could shed some 
>>>>>>>>>>>>> light.
>>>>>>>>>>>>>                             Corresponding files contain
>>>>>>>>>>>>>                             stdout/stderr of test agents.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                             Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     On 8/23/23 
>>>>>>>>>>>>> 17:45, Adam
>>>>>>>>>>>>>                             Hassick wrote:
>>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I've set up a 
>>>>>>>>>>>>>> test rig
>>>>>>>>>>>>>>                             repository here, and have 
>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>                             configurations for our
>>>>>>>>>>>>>>                             development testbed based off
>>>>>>>>>>>>>>                             of the
>>>>>>>>>>>>>>                             examples.
>>>>>>>>>>>>>>                                     We've been 
>>>>>>>>>>>>>> able to get
>>>>>>>>>>>>>>                             the test suite to run 
>>>>>>>>>>>>>> manually on
>>>>>>>>>>>>>>                             Mellanox CX5 devices once.
>>>>>>>>>>>>>>                             However, we are running into
>>>>>>>>>>>>>>                             an issue where, when RCF 
>>>>>>>>>>>>>> starts,
>>>>>>>>>>>>>>                                     the RCF 
>>>>>>>>>>>>>> connection
>>>>>>>>>>>>>>                             times out very frequently. We
>>>>>>>>>>>>>>                             aren't sure
>>>>>>>>>>>>>>                                     why this is 
>>>>>>>>>>>>>> the case.
>>>>>>>>>>>>>>                                     It works 
>>>>>>>>>>>>>> sometimes,
>>>>>>>>>>>>>>                             but most of the time when we
>>>>>>>>>>>>>>                             try to run
>>>>>>>>>>>>>>                                     the test 
>>>>>>>>>>>>>> engine, it
>>>>>>>>>>>>>>                             encounters this issue.
>>>>>>>>>>>>>>                                     I've tried 
>>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>                             the RCF port by setting
>>>>>>>>>>>>>>                             "TE_RCF_PORT=<some port
>>>>>>>>>>>>>>                             number>" and rebooting the
>>>>>>>>>>>>>>                             testbed
>>>>>>>>>>>>>>                             machines. Neither seems to 
>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>                             the issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     It also seems 
>>>>>>>>>>>>>> like the
>>>>>>>>>>>>>>                             timeout takes far longer than 
>>>>>>>>>>>>>> 60
>>>>>>>>>>>>>>                             seconds, even when running
>>>>>>>>>>>>>>                             "export 
>>>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>                                     before I try 
>>>>>>>>>>>>>> to run
>>>>>>>>>>>>>>                             the test suite.
>>>>>>>>>>>>>>                                     I assume the 
>>>>>>>>>>>>>> unit for
>>>>>>>>>>>>>>                             this variable is seconds?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     On Mon, Aug 
>>>>>>>>>>>>>> 21, 2023
>>>>>>>>>>>>>>                             at 10:19 AM Adam Hassick
>>>>>>>>>>>>>>                                     
>>>>>>>>>>>>>> <ahassick@iol.unh.edu
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         Hi 
>>>>>>>>>>>>>> Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                             Thanks, I've cloned the
>>>>>>>>>>>>>>                             example repository and will 
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>                             setting up a configuration 
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>                             our development testbed
>>>>>>>>>>>>>>                             today. I'll let you know if I
>>>>>>>>>>>>>>                             run into any difficulties
>>>>>>>>>>>>>>                                         or 
>>>>>>>>>>>>>> have any
>>>>>>>>>>>>>>                             questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                          - 
>>>>>>>>>>>>>> Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         On 
>>>>>>>>>>>>>> Sun, Aug 20,
>>>>>>>>>>>>>>                             2023 at 4:40 AM Andrew 
>>>>>>>>>>>>>> Rybchenko
>>>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Hi Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> I've published
>>>>>>>>>>>>>>                            https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>                             <https://github.com/ts-factory/ts-rigs-sample> 
>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Hopefully it
>>>>>>>>>>>>>>                             will help to define your test
>>>>>>>>>>>>>>                             rigs and
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> successfully
>>>>>>>>>>>>>>                             run some tests manually. Feel
>>>>>>>>>>>>>>                             free to
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> ask any
>>>>>>>>>>>>>>                             questions and I'll answer 
>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>                             and try to
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>                             documentation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Meanwhile I'll
>>>>>>>>>>>>>>                             prepare missing bits for 
>>>>>>>>>>>>>> steps
>>>>>>>>>>>>>>                             (2) and
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> (3).
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Hopefully
>>>>>>>>>>>>>>                             everything is in place for
>>>>>>>>>>>>>>                             step (4), but we
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> need to make
>>>>>>>>>>>>>>                             steps (2) and (3) first.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>> On 8/18/23
>>>>>>>>>>>>>>                             21:40, Andrew Rybchenko 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>                             Hi Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> > I've
>>>>>>>>>>>>>>>                             conferred with the rest of
>>>>>>>>>>>>>>>                             the team, and we
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> think it
>>>>>>>>>>>>>>>                             would be best to move 
>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>                             with mainly
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> option B.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> OK, I'll
>>>>>>>>>>>>>>>                             provide the sample on Monday
>>>>>>>>>>>>>>>                             for you. It is
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> almost ready
>>>>>>>>>>>>>>>                             right now, but I need to
>>>>>>>>>>>>>>>                             double-check
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> it before
>>>>>>>>>>>>>>>                             publishing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>> On 8/17/23
>>>>>>>>>>>>>>>                             20:03, Adam Hassick wrote:
>>>>>>>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> I'm adding
>>>>>>>>>>>>>>>>                             the CI mailing list to this
>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>                             conversation. Others in the
>>>>>>>>>>>>>>>>                             community might find
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>                             conversation valuable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> We do want
>>>>>>>>>>>>>>>>                             to run testing on a regular
>>>>>>>>>>>>>>>>                             basis. The
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Jenkins
>>>>>>>>>>>>>>>>                             integration will be very
>>>>>>>>>>>>>>>>                             useful for us, as
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> most of our
>>>>>>>>>>>>>>>>                             CI is orchestrated by 
>>>>>>>>>>>>>>>> Jenkins.
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>                             conferred with the rest of
>>>>>>>>>>>>>>>>                             the team, and we
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> think it
>>>>>>>>>>>>>>>>                             would be best to move
>>>>>>>>>>>>>>>>                             forward with mainly
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> option B.
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> If you would
>>>>>>>>>>>>>>>>                             like to know anything about 
>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> testbeds
>>>>>>>>>>>>>>>>                             that would help you with
>>>>>>>>>>>>>>>>                             creating an
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> example
>>>>>>>>>>>>>>>>                             ts-rigs repo, I'd be happy
>>>>>>>>>>>>>>>>                             to answer any
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> questions
>>>>>>>>>>>>>>>>                             you have.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> We have
>>>>>>>>>>>>>>>>                             multiple test rigs (we call
>>>>>>>>>>>>>>>>                             these
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> "DUT-tester
>>>>>>>>>>>>>>>>                             pairs") that we run our
>>>>>>>>>>>>>>>>                             existing
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> hardware
>>>>>>>>>>>>>>>>                             testing on, with differing
>>>>>>>>>>>>>>>>                             network
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> hardware and
>>>>>>>>>>>>>>>>                             CPU architecture. I figured
>>>>>>>>>>>>>>>>                             this might
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> be an
>>>>>>>>>>>>>>>>                             important detail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> On Thu, Aug
>>>>>>>>>>>>>>>>                             17, 2023 at 11:44 AM 
>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>                             Rybchenko
>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Greatings Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>                             happy to hear that you're
>>>>>>>>>>>>>>>>                             trying to bring
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> it up.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> As I
>>>>>>>>>>>>>>>>                             understand the final goal 
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>                             to run it on
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> regular
>>>>>>>>>>>>>>>>                             basis. So, we need to make
>>>>>>>>>>>>>>>>                             it properly
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>                             very beginning.
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Bring up
>>>>>>>>>>>>>>>>                             of all features consists of
>>>>>>>>>>>>>>>>                             4 steps:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>                             Create site-specific
>>>>>>>>>>>>>>>>                             repository (we call it
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> ts-rigs)
>>>>>>>>>>>>>>>>                             which contains information
>>>>>>>>>>>>>>>>                             about test
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> rigs and
>>>>>>>>>>>>>>>>                             other site-specific
>>>>>>>>>>>>>>>>                             information like
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> where to
>>>>>>>>>>>>>>>>                             send mails, where to store
>>>>>>>>>>>>>>>>                             logs etc.
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> It is
>>>>>>>>>>>>>>>>                             required for manual
>>>>>>>>>>>>>>>>                             execution as well,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>                             test rigs description is
>>>>>>>>>>>>>>>>                             essential. I'll
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> return
>>>>>>>>>>>>>>>>                             to the topic below.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 2. Setup
>>>>>>>>>>>>>>>>                             logs storage for automated
>>>>>>>>>>>>>>>>                             runs.
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Basically it is a disk 
>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>>                             plus apache2 web
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>                             with few CGI scripts which
>>>>>>>>>>>>>>>>                             help a lot to
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>                             disk space.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 3. Setup
>>>>>>>>>>>>>>>>                             Bublik web application 
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>                             provides
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> web
>>>>>>>>>>>>>>>>                             interface to view testing
>>>>>>>>>>>>>>>>                             results. Same as
>>>>>>>>>>>>>>>>                            https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> 4. Setup
>>>>>>>>>>>>>>>>                             Jenkins to run tests on
>>>>>>>>>>>>>>>>                             regularly,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>                             logs in log storage (2) and
>>>>>>>>>>>>>>>>                             import it to
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> bublik (3).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Last few
>>>>>>>>>>>>>>>>                             month we spent on our
>>>>>>>>>>>>>>>>                             homework to make
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>                             simpler to bring up
>>>>>>>>>>>>>>>>                             automated execution
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>                             Jenkins -
>>>>>>>>>>>>>>>>                            https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Corresponding bits in
>>>>>>>>>>>>>>>>                             dpdk-ethdev-ts will be
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             available tomorrow.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Let's
>>>>>>>>>>>>>>>>                             return to the step (1).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             Unfortunately there is no
>>>>>>>>>>>>>>>>                             publicly available
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> example
>>>>>>>>>>>>>>>>                             of the ts-rigs repository 
>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             sensitive site-specific
>>>>>>>>>>>>>>>>                             information is located
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>                             But I'm ready to help you 
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>                             create it
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> for UNH.
>>>>>>>>>>>>>>>>                             I see two options here:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> (A) I'll
>>>>>>>>>>>>>>>>                             ask questions and based on 
>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> answers
>>>>>>>>>>>>>>>>                             will create the first draft
>>>>>>>>>>>>>>>>                             with my
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> (B) I'll
>>>>>>>>>>>>>>>>                             make a template/example
>>>>>>>>>>>>>>>>                             ts-rigs repo,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> publish
>>>>>>>>>>>>>>>>                             it and you'll create UNH
>>>>>>>>>>>>>>>>                             ts-rigs based
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> on it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Of
>>>>>>>>>>>>>>>>                             course, I'll help to debug
>>>>>>>>>>>>>>>>                             and finally bring
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> it up in
>>>>>>>>>>>>>>>>                             any case.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> (A) is a
>>>>>>>>>>>>>>>>                             bit simpler for me and you,
>>>>>>>>>>>>>>>>                             but (B) is
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> a bit
>>>>>>>>>>>>>>>>                             more generic and will help
>>>>>>>>>>>>>>>>                             other
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                             potential users to bring it 
>>>>>>>>>>>>>>>> up.
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> We can
>>>>>>>>>>>>>>>>                             combine (A)+(B). I.e. start
>>>>>>>>>>>>>>>>                             from (A).
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> What do
>>>>>>>>>>>>>>>>                             you think?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>> On
>>>>>>>>>>>>>>>>                             8/17/23 15:18, Konstantin
>>>>>>>>>>>>>>>>                             Ushakov wrote:
>>>>>>>>>>>>>>>>>                             Greetings Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>                             for contacting us. I copy
>>>>>>>>>>>>>>>>>                             Andrew who
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>                             be happy to help
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>> Konstantin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                             On 16 Aug 2023, at 21:50,
>>>>>>>>>>>>>>>>>>                             Adam Hassick
>>>>>>>>>>>>>>>>>>                             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>>>>>>                             in the process of setting
>>>>>>>>>>>>>>>>>>                             up the DPDK
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Poll
>>>>>>>>>>>>>>>>>>                             Mode Driver test suite as
>>>>>>>>>>>>>>>>>>                             an addition to
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>>>                             testing coverage for DPDK
>>>>>>>>>>>>>>>>>>                             at the UNH lab.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>>                             some questions about how
>>>>>>>>>>>>>>>>>>                             to set the
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>                             suite arguments.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>>                             been able to configure 
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>                             Test Engine
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>                             connect to the hosts in
>>>>>>>>>>>>>>>>>>                             the testbed. The
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> RCF,
>>>>>>>>>>>>>>>>>>                             Configurator, and Tester
>>>>>>>>>>>>>>>>>>                             all begin to
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> run,
>>>>>>>>>>>>>>>>>>                             however the prelude of 
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>                             test suite
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>>>                             to run.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                            https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>                             documentation mentions
>>>>>>>>>>>>>>>>>>                             that there are
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             several test parameters
>>>>>>>>>>>>>>>>>>                             for the test suite,
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>                             for the IUT test link 
>>>>>>>>>>>>>>>>>> MAC,
>>>>>>>>>>>>>>>>>>                             etc. These
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>                             like they would need to 
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>                             set somewhere
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> to run
>>>>>>>>>>>>>>>>>>                             many of the tests.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>                             in the Test Engine
>>>>>>>>>>>>>>>>>>                             documentation, there
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>                             instructions on how to
>>>>>>>>>>>>>>>>>>                             create new
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             parameters for test 
>>>>>>>>>>>>>>>>>> suites
>>>>>>>>>>>>>>>>>>                             in the Tester
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             configuration, but there
>>>>>>>>>>>>>>>>>>                             is nothing in the
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>                             guide or in the Tester
>>>>>>>>>>>>>>>>>>                             guide for how to
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>                             the arguments for the
>>>>>>>>>>>>>>>>>>                             parameters when
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             running the test suite
>>>>>>>>>>>>>>>>>>                             that I can find. I'm
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>                             sure if I need to write 
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>                             own Tester
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             config, or if I should be
>>>>>>>>>>>>>>>>>>                             setting these in
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>                             other way.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>                             should these values be 
>>>>>>>>>>>>>>>>>> set?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>                             also not sure what
>>>>>>>>>>>>>>>>>>                             environment
>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>                             variables/arguments are
>>>>>>>>>>>>>>>>>>                             strictly necessary or
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>                             are optional.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Adam
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> *Adam
>>>>>>>>>>>>>>>>>>                             Hassick*
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> Senior
>>>>>>>>>>>>>>>>>>                             Developer
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> UNH
>>>>>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>>>>                             (603) 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> Senior
>>>>>>>>>>>>>>>>                             Developer
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> UNH
>>>>>>>>>>>>>>>>                             InterOperability Lab
>>>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>> +1 (603)
>>>>>>>>>>>>>>>>                             475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         -- 
>>>>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                         +1 
>>>>>>>>>>>>>> (603) 475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     -- 
>>>>>>>>>>>>>>         *Adam Hassick*
>>>>>>>>>>>>>>                                     Senior 
>>>>>>>>>>>>>> Developer
>>>>>>>>>>>>>>                                     UNH 
>>>>>>>>>>>>>> InterOperability Lab
>>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                     +1 (603) 
>>>>>>>>>>>>>> 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 -- *Adam Hassick*
>>>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>>>                                 UNH InterOperability 
>>>>>>>>>>>>> Lab
>>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>>>>                             <http://iol.unh.edu>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                             --
>>>>>>>>>>>>                             *Adam Hassick*
>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                             iol.unh.edu 
>>>>>>>>>>>> <http://iol.unh.edu>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         --
>>>>>>>>>>                         *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>>>>                         iol.unh.edu 
>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                     --
>>>>>>>>                     *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --
>>>>>>                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>                 ahassick@iol.unh.edu
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>         --
>>>>         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>     --
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>>
>>>
>>>
>>> -- 
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>

[-- Attachment #2: Type: text/html, Size: 77968 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 94622 bytes --]

On 9/18/23 09:23, Konstantin Ushakov wrote:
>
> Hi Andrew,
>
> should we always auto-assign the tags or you don’t do it since it 
> slows down (by some seconds) the TE startup?
>

Tags are auto-assigned, but I guess it differs in Adam's case since NIC 
is a bit different. Below test will help to understand if it is the root 
cause of very different expectations. If pass rate will be close to 
mine, I'll simply update TRC database to share expectations for mine NIC 
and NIC used by Adam.

> Hi Adam,
>
> I think I second the question from Andrew - happy to help you with the 
> triage so that we get to the same baseline. Do you have a good way for 
> us to share the logs? I.e. say upload to ts-factory if we add strict 
> permissions system so it’s not publishing or any other way.
>
> Thanks,
> Konstantin
>
>
> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>
>     Hi Adam,
>
>     I've uploaded fresh testing results to ts-factory.io [1] to be on
>     the same page.
>
>     I think I know why your and mine results on Intel 710 series NICs
>     differ so much. Testing results expectations database
>     (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags.  I.e.
>     expectations depends on TRC tags discovered by helper scripts when
>     testing is started. These tags identify various aspects of what is
>     tested. Ideally expectations should be written in terms of root
>     cause of the expected behaviour. If it is a driver expectations,
>     driver tag should be used. If it is HW limitation, tags with PCI
>     IDs should be used. However, it is not always easy to classify it
>     correctly if you're not involved in driver development. So, in
>     order case expectations for 710's Intel are filled in in terms of
>     PCI IDs. I guess PCI ID differ in your case and that's why
>     expectations filled in for my NIC do not apply to your runs.
>
>     Just try to add the following option when you run on your 710's
>     Intel in order to mimic mine and see if it helps to achieve better
>     pass rate.
>     --trc-tag=pci-8086-1572
>
>     BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests
>     for --tester-dial option. It should have better coverage now.
>
>     Andrew.
>
>     [1]
>     https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>
>     On 9/13/23 18:45, Andrew Rybchenko wrote:
>>     Hi Adam,
>>
>>     I've pushed new TE tag v1.20.0 which supported a new command-line
>>     option --tester-dial=NUM where NUM is from 0 to 100. it allows to
>>     choose percentage of tests to run. If you want stable set, you
>>     should pass --tester-random-seed=0 (or other integer). It is the
>>     first sketch and we have plans to improve it, but feedback would
>>     be welcome.
>>
>>     > Is it needed on the tester?
>>
>>     It is hard to say if it is strictly required for simple tests.
>>     However, it is better to update Tester as well, since performance
>>     tests run DPDK on Tester as well.
>>
>>     > Are there any other manual setup steps for these devices that I
>>     might be missing?
>>
>>     I don't remember anything else.
>>
>>     I think it is better to get down to details and take a look at
>>     logs. I'm ready to help with it and explain what's happening
>>     there. May be it will help to understand if it is a problem with
>>     setup/configuration.
>>
>>     Text logs are not very convenient. Ideally logs should be
>>     imported to bublik, however, manual runs do not provide all
>>     required artifacts right now (Jenkins jobs generate all required
>>     artifacts).
>>     Other option is 'tmp_raw_log' file (should be packed to make it
>>     smaller) which could be converted to various log formats.
>>     Would it be OK for you if I import your logs to bublik at
>>     ts-factory.io? Or is it a problem that it is publicly available?
>>     Would it help if we add authentication and access control there?
>>
>>     Andrew.
>>
>>     On 9/8/23 17:57, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     I have a couple questions about needed setup of the NICs for the
>>>     ethdev test suite.
>>>
>>>     Our MCX5s and XL710s are failing the checkup tests. The pass
>>>     rate appears to be much worse on the XL710s (40 of 73 tests
>>>     failed, 3 passed unexpectedly).
>>>
>>>     For the XL710s, I've updated the driver and NVM versions to
>>>     match the minimum supported versions in the compatibility matrix
>>>     found on the DPDK documentation. This did not change the failure
>>>     rate much.
>>>     For the MCX5s, I've installed the latest LTS version of the OFED
>>>     bifurcated driver on the DUT. Is it needed on the tester?
>>>
>>>     Are there any other manual setup steps for these devices that I
>>>     might be missing?
>>>
>>>     Thanks,
>>>     Adam
>>>
>>>     On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick
>>>     <ahassick@iol.unh.edu> wrote:
>>>
>>>         Hi Andrew,
>>>
>>>         Yes, I copied the X710 configs to set up XL710 configs. I
>>>         changed the environment variable names from the X710 suffix
>>>         to XL710 suffix in the script, and forgot to change them in
>>>         the corresponding environment file.
>>>         That fixed the issue.
>>>
>>>         I got the checkup tests working on the XL710 now. Most of
>>>         them are failing, which leads me to believe this is an issue
>>>         with our testbed. Based on the DPDK documentation for i40e,
>>>         the firmware and driver versions are much older than what
>>>         DPDK 22.11 LTS and main prefer, so I'll try updating those.
>>>
>>>         For now I'm working on getting the XL710 checkup tests
>>>         passing, and will pick up getting the E810 configured
>>>         properly next. I'll let you know if I run into any more
>>>         issues in relation to the test engine.
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             Hi Adam,
>>>
>>>             On 9/5/23 18:01, Adam Hassick wrote:
>>>>             Hi Andrew,
>>>>
>>>>             The compilation warning issue is now resolved. Again,
>>>>             thank you guys for fixing this for us. I can run the
>>>>             tests on the Mellanox CX5s again, however I'm running
>>>>             into a couple new issues with running the prologues on
>>>>             the Intel cards.
>>>>
>>>>             When running testing on the Intel XL710s, I see this
>>>>             error appear in the log:
>>>>
>>>>                 ERROR  prologue  Environment LIB  14:16:13.650
>>>>                 Too few networks in available configuration (0) in
>>>>                 comparison with required (1)
>>>>
>>>>
>>>>             This seems like a trivial configuration error, perhaps
>>>>             this is something I need to set up in ts-rigs. I
>>>>             briefly searched through the examples there and didn't
>>>>             see any mention of how to set up a network.
>>>>             I will attach this log just in case you need more
>>>>             information.
>>>
>>>             Unfortunately logs are insufficient to understand it.
>>>             I've pushed new tag to TE v1.19.0 which add log message
>>>             with TE_* environment variables.
>>>             Most likely something is wrong with variables which are
>>>             used as conditions when available networks are defined
>>>             in ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>             TE_PCI_INSTANCE_IUT_TST1
>>>             TE_PCI_INSTANCE_IUT_TST1a
>>>             TE_PCI_INSTANCE_TST1a_IUT
>>>             TE_PCI_INSTANCE_TST1_IUT
>>>             My guess it that you change naming a bit, but script
>>>             like ts-rigs-sample/scripts/iut.h1-x710 is not included
>>>             or not updated.
>>>
>>>>             There is a different error when running on the Intel
>>>>             E810s. It appears to me like it starts DPDK, does some
>>>>             configuration inside DPDK and on the device, and then
>>>>             fails to bring the device back up. Since this error
>>>>             seems very non-trivial, I will also attach this log.
>>>
>>>             This one is a bit simpler. Few lines after the first
>>>             ERROR in log I see the following:
>>>             WARN  RCF  DPDK  13:06:00.144
>>>             ice_program_hw_rx_queue(): currently package doesn't
>>>             support RXDID (22)
>>>             ice_rx_queue_start(): fail to program RX queue 0
>>>             ice_dev_start(): fail to start Rx queue 0
>>>             Device with port_id=0 already stopped
>>>
>>>             It is stdout/stderr from test agent which runs DPDK.
>>>             Same logs in plain format are available in ta.DPDK file.
>>>             I'm not an expert here, but I vaguely remember that E810
>>>             requires correct firmware and DDP to be loaded.
>>>             There is some information in dpdk/doc/guides/nics/ice.rst.
>>>
>>>             You can try to add --dev-args=safe-mode-support=1
>>>             command-line option described there.
>>>
>>>             Hope it helps,
>>>             Andrew.
>>>
>>>>
>>>>             Thanks,
>>>>             Adam
>>>>
>>>>             On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 On 8/31/23 22:38, Adam Hassick wrote:
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 I have one additional question as well: Does the
>>>>>                 test engine support running tests on two ARMv8
>>>>>                 test agents?
>>>>>
>>>>>                     1. We'll sort out warnings this week. Thanks
>>>>>                     for heads up.
>>>>>
>>>>>
>>>>>                 Great. Let me know when that's fixed.
>>>>
>>>>                 Done. We also fixed a number of warnings in TE.
>>>>                 Also we fixed root test package name to be
>>>>                 consistent with the repository name.
>>>>
>>>>>                     Support for old LTS branches was dropped some
>>>>>                     time ago, but in the future it is definitely
>>>>>                     possible to keep it for new LTS branches. I
>>>>>                     think 22.11 is supported, but I'm not sure
>>>>>                     about older LTS releases.
>>>>>
>>>>>
>>>>>                 Good to know.
>>>>>
>>>>>                     2. You can add command-line option --sanity to
>>>>>                     run tests marked with TEST_HARNESS_SANITY
>>>>>                     requirement (see dpdk-ethdev-ts/scripts/run.sh
>>>>>                     and grep TEST_HARNESS_SANITY dpdk-ethdev-ts to
>>>>>                     see which tests are marked). Yes, there is a
>>>>>                     space for terminology improvement here. We'll
>>>>>                     do it.
>>>>>
>>>>
>>>>                 Done. Now it is called --checkup.
>>>>
>>>>>
>>>>>                     Also it takes a lot of time because of
>>>>>                     failures and tests which wait for some timeout.
>>>>>
>>>>>
>>>>>                 That makes sense to me. We'll use the time to
>>>>>                 complete tests on virtio or the Intel devices as a
>>>>>                 reference for how long the tests really take to
>>>>>                 complete.
>>>>>                 We will explore the possibility of periodically
>>>>>                 running the sanity tests for patches.
>>>>
>>>>                 I'll double-check and let you know how long entire
>>>>                 TS runs on Intel X710, E810, Mellanox CX5 and
>>>>                 virtio net. Just to ensure that time observed in
>>>>                 your case looks the same.
>>>>
>>>>>
>>>>>                     The test harness can provide coverage reports
>>>>>                     based on gcov, but I'm not sure what you mean
>>>>>                     by a "dial" to control test coverage. Provided
>>>>>                     reports are rather for human to analyze.
>>>>>
>>>>>
>>>>>                 The general idea is to have some kind of parameter
>>>>>                 on the test suite, which could be an integer
>>>>>                 ranging from zero to ten, that controls how many
>>>>>                 tests are run based on how important the test is.
>>>>>
>>>>>                 Similar to how some command line interfaces
>>>>>                 provide a verbosity level parameter (some number
>>>>>                 of "-v" arguments) to control the importance of
>>>>>                 the information in the log.
>>>>>                 The verbosity level zero only prints very
>>>>>                 important log messages, while ten prints everything.
>>>>>
>>>>>                 In much the same manner as above, this "dial"
>>>>>                 parameter controls what tests are run and with
>>>>>                 what parameters based on how important those tests
>>>>>                 and test parameter combinations are.
>>>>>                 Coverage Level zero tells the suite to run a very
>>>>>                 basic set of important tests, with minimal
>>>>>                 parameterization. This mode would take only ~5-10
>>>>>                 minutes to run.
>>>>>                 In contrast, Coverage Level ten includes all the
>>>>>                 edge cases, every combination of test parameters,
>>>>>                 everything the test suite can do, which takes the
>>>>>                 normal several hours to run.
>>>>>                 The values 1 - 9 are between those two extremes,
>>>>>                 allowing the user to get a gradient of test
>>>>>                 coverage in the results and to limit the running time.
>>>>>
>>>>>                 Then we could, for example, run the "run.sh" with
>>>>>                 a level of 2 or 3 for incoming patches that need
>>>>>                 quick results, and with a level of 10 for the less
>>>>>                 often run periodic tests performed on main or LTS
>>>>>                 branches.
>>>>
>>>>                 Understood now. Thanks a lot for the idea. We'll
>>>>                 discuss it and come back.
>>>>
>>>>>                     3. Yes, really many tests on Mellanox CX5 NICs
>>>>>                     report unexpected testing results.
>>>>>                     Unfortunately it is time consuming to fill in
>>>>>                     expectations database since it is necessary to
>>>>>                     analyze testing results and classify if it is
>>>>>                     a bug or just acceptable behaviour aspect.
>>>>>
>>>>>                     Bublik allows to compare results of two runs.
>>>>>                     It is useful for human, but still not good for
>>>>>                     automation.
>>>>>
>>>>>                     I have local patch for mlx5 driver which
>>>>>                     reports Tx ring size maximum. It makes pass
>>>>>                     rate higher. It is a problem for test harness
>>>>>                     that mlx5 does not report limits right now.
>>>>>
>>>>>                     Pass rate on Intel X710 is about 92% on my
>>>>>                     test rig. Pass rate on virtio net is 99% right
>>>>>                     now and could be done 100% easily (just one
>>>>>                     thing to fix in expectations).
>>>>>
>>>>>                     I think logs storage setup is essential for
>>>>>                     logs analysis. Of course, you can request HTML
>>>>>                     logs when you run tests (--log-html=html) or
>>>>>                     generate after run using
>>>>>                     dpdk-ethdev-ts/scripts/html-log.sh and open
>>>>>                     index.html in a browser, but logs storage
>>>>>                     makes it more convenient.
>>>>>
>>>>>
>>>>>                 We are interested in setting up Bublik,
>>>>>                 potentially as an externally-facing component,
>>>>>                 once we have our process of running the test suite
>>>>>                 stabilized.
>>>>>                 Once we are able to run the test suite again, I'll
>>>>>                 see what the pass rate is on our other hardware.
>>>>>                 Good to know that it isn't an issue with our dev
>>>>>                 testbed causing the high fail rate.
>>>>>
>>>>>                 For Intel hardware, we have an XL710 and an Intel
>>>>>                 E810-C in our development testbed. Although they
>>>>>                 are slightly different devices, ideally the pass
>>>>>                 rate will be identical or similar. I have yet to
>>>>>                 set up a VM pair for virtio, but we will soon.
>>>>>
>>>>>                     Latest version of test-environment has
>>>>>                     examples of our CGI scripts which we use for
>>>>>                     log storage (see tools/log_server/README.md).
>>>>>
>>>>>                     Also all bits for Jenkins setup are available.
>>>>>                     See dpdk-ethdev-ts/jenkins/README.md and
>>>>>                     examples of jenkins files in ts-rigs-sample.
>>>>>
>>>>>
>>>>>                 Jenkins integration, setting up production rig
>>>>>                 configurations, and permanent log storage will be
>>>>>                 our next steps once I am able to run the tests again.
>>>>>                 Unless there is an easy way to have meson not pass
>>>>>                 "-Werror" into GCC. Then I would be able to run
>>>>>                 the test suite.
>>>>
>>>>                 Hopefully it is resolved now.
>>>>
>>>>                 I thought a bit more about your usecase for
>>>>                 Jenkins. I'm not 100% sure that existing pipelines
>>>>                 are convenient for your usecase.
>>>>                 Fill free to ask questions when you are on it.
>>>>
>>>>                 Thanks,
>>>>                 Andrew.
>>>>
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>
>>>>>                     On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>                     Hi Andrew,
>>>>>>
>>>>>>                     That fix seems to have resolved the issue,
>>>>>>                     thanks for the quick turnaround time on that
>>>>>>                     patch.
>>>>>>                     Now that we have the RCF timeout issue
>>>>>>                     resolved, there are a few other questions and
>>>>>>                     issues that we have about the tests themselves.
>>>>>>
>>>>>>                     1. The test suite fails to build with a
>>>>>>                     couple warnings.
>>>>>>
>>>>>>                     Below is the stderr log from compilation:
>>>>>>
>>>>>>                         FAILED:
>>>>>>                         lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>                         cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>>                         -I../../lib
>>>>>>                         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>                         -fdiagnostics-color=always -pipe
>>>>>>                         -D_FILE_OFFSET_BITS=64 -Wall
>>>>>>                         -Winvalid-pch -Werror -g -D_GNU_SOURCE
>>>>>>                         -O0 -ggdb -Wall -W -fPIC -MD -MQ
>>>>>>                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                         -MF
>>>>>>                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>                         -o
>>>>>>                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                         -c ../../lib/dpdk_pmd_ts.c
>>>>>>                         ../../lib/dpdk_pmd_ts.c: In function
>>>>>>                         ‘test_create_traffic_generator_params’:
>>>>>>                         ../../lib/dpdk_pmd_ts.c:5577:5: error:
>>>>>>                         format not a string literal and no format
>>>>>>                         arguments [-Werror=format-security]
>>>>>>                         5577 |     rc = te_kvpair_add(result,
>>>>>>                         buf, mode);
>>>>>>                         |     ^~
>>>>>>                         cc1: all warnings being treated as errors
>>>>>>                         ninja: build stopped: subcommand failed.
>>>>>>                         ninja: Entering directory `.'
>>>>>>                         FAILED:
>>>>>>                         lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>                         cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
>>>>>>                         -I../../lib
>>>>>>                         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>                         -fdiagnostics-color=always -pipe
>>>>>>                         -D_FILE_OFFSET_BITS=64 -Wall
>>>>>>                         -Winvalid-pch -Werror -g -D_GNU_SOURCE
>>>>>>                         -O0 -ggdb -Wall -W -fPIC -MD -MQ
>>>>>>                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                         -MF
>>>>>>                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>                         -o
>>>>>>                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>                         -c ../../lib/dpdk_pmd_ts.c
>>>>>>                         ../../lib/dpdk_pmd_ts.c: In function
>>>>>>                         ‘test_create_traffic_generator_params’:
>>>>>>                         ../../lib/dpdk_pmd_ts.c:5577:5: error:
>>>>>>                         format not a string literal and no format
>>>>>>                         arguments [-Werror=format-security]
>>>>>>                         5577 |     rc = te_kvpair_add(result,
>>>>>>                         buf, mode);
>>>>>>                         |     ^~
>>>>>>                         cc1: all warnings being treated as errors
>>>>>>
>>>>>>
>>>>>>                     This error wasn't occurring last week, which
>>>>>>                     was the last time I ran the tests.
>>>>>>                     The TE host and the DUT have GCC v9.4.0
>>>>>>                     installed, and the tester has GCC v11.4.0
>>>>>>                     installed, if this information is helpful.
>>>>>>
>>>>>>                     2. On the Mellanox CX5s, there are over 6,000
>>>>>>                     tests run, which collectively take around 9
>>>>>>                     hours. Is it possible, and would it make
>>>>>>                     sense, to lower the test coverage and have
>>>>>>                     the test suite run faster?
>>>>>>
>>>>>>                     For some context, we run immediate testing on
>>>>>>                     incoming patches for DPDK main and
>>>>>>                     development branches, as well as periodic
>>>>>>                     test runs on the main, stable, and LTS branches.
>>>>>>                     For us to consider including this test suite
>>>>>>                     as part of our immediate testing on patches,
>>>>>>                     we would have to reduce the test coverage to
>>>>>>                     the most important tests.
>>>>>>                     This is primarily to reduce the testing time
>>>>>>                     to, for example, less than 30 minutes.
>>>>>>                     Testing on patches can't take too long
>>>>>>                     because the lab can receive numerous patches
>>>>>>                     each day, which each require individual
>>>>>>                     testing runs.
>>>>>>
>>>>>>                     At what frequency we run these tests, and on
>>>>>>                     what, still needs to be discussed with the
>>>>>>                     DPDK community, but it would be nice to know
>>>>>>                     if the test suite had a "dial" to control the
>>>>>>                     testing coverage.
>>>>>>
>>>>>>                     3. We see a lot of test failures on our
>>>>>>                     Mellanox CX5 NICs. Around 2,300 of ~6,600
>>>>>>                     tests passed. Is there anything we can do to
>>>>>>                     diagnose these test failures?
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Adam
>>>>>>
>>>>>>
>>>>>>                     On Tue, Aug 29, 2023 at 8:07 AM Andrew
>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                         Hi Adam,
>>>>>>
>>>>>>                         I've pushed the fix in main branch and a
>>>>>>                         new tag v1.18.1. It should solve the
>>>>>>                         problem with IPv6 address from DNS.
>>>>>>
>>>>>>                         Andrew.
>>>>>>
>>>>>>                         On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>                         Hi Adam,
>>>>>>>
>>>>>>>                         > Does the test engine prefer to use
>>>>>>>                         IPv6 over IPv4 for initiating the RCF
>>>>>>>                         connection to the test bed hosts? And if
>>>>>>>                         so, is there a way to force it to use IPv4?
>>>>>>>
>>>>>>>                         Brilliant idea. If DNS returns both IPv4
>>>>>>>                         and IPv6 addresses in your case, I guess
>>>>>>>                         it is the root cause of the problem.
>>>>>>>                         Of course, it is TE problem since I see
>>>>>>>                         really weird code in
>>>>>>>                         lib/comm_net_engine/comm_net_engine.c
>>>>>>>                         line 135.
>>>>>>>
>>>>>>>                         I've pushed fix to the branch
>>>>>>>                         user/arybchik/fix_ipv4_only in
>>>>>>>                         ts-factory/test-environment repository.
>>>>>>>                         Please, try.
>>>>>>>
>>>>>>>                         It is late night fix with minimal
>>>>>>>                         testing and no review. I'll pass it
>>>>>>>                         through review process tomorrow and
>>>>>>>                         hopefully it will be released in one-two
>>>>>>>                         days.
>>>>>>>
>>>>>>>                         Andrew.
>>>>>>>
>>>>>>>                         On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>                         Hi Andrew,
>>>>>>>>
>>>>>>>>                         We have yet to notice a distinct
>>>>>>>>                         pattern with the failures. Sometimes,
>>>>>>>>                         the RCF will start and connect without
>>>>>>>>                         issue a few times in a row before
>>>>>>>>                         failing to connect again. Once the
>>>>>>>>                         issue begins to occur, neither
>>>>>>>>                         rebooting all of the hosts (test engine
>>>>>>>>                         VM, tester, IUT) or deleting all of the
>>>>>>>>                         build directories (suites, agents,
>>>>>>>>                         inst) and rebooting the hosts afterward
>>>>>>>>                         resolves the issue. When it begins
>>>>>>>>                         working again seems very arbitrary to us.
>>>>>>>>
>>>>>>>>                         I do usually try to terminate the test
>>>>>>>>                         engine with Ctrl+C, but when it hangs
>>>>>>>>                         while trying to start RCF, that does
>>>>>>>>                         not work.
>>>>>>>>
>>>>>>>>                         Does the test engine prefer to use IPv6
>>>>>>>>                         over IPv4 for initiating the RCF
>>>>>>>>                         connection to the test bed hosts? And
>>>>>>>>                         if so, is there a way to force it to
>>>>>>>>                         use IPv4?
>>>>>>>>
>>>>>>>>                          - Adam
>>>>>>>>
>>>>>>>>                         On Fri, Aug 25, 2023 at 1:35 PM Andrew
>>>>>>>>                         Rybchenko
>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>                             > I'll double-check test engine on
>>>>>>>>                             Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>
>>>>>>>>                             Done. It works fine for me without
>>>>>>>>                             any issues.
>>>>>>>>
>>>>>>>>                             Have you noticed any pattern when
>>>>>>>>                             it works or does not work?
>>>>>>>>                             May be it is a problem of not clean
>>>>>>>>                             state after termination?
>>>>>>>>                             Does it work fine the first time
>>>>>>>>                             after DUTs reboot?
>>>>>>>>                             How do you terminate testing? It
>>>>>>>>                             should be done using Ctrl+C in
>>>>>>>>                             terminal where you execute run.sh
>>>>>>>>                             command.
>>>>>>>>                              In this case it should shutdown
>>>>>>>>                             gracefully and close all test
>>>>>>>>                             agents and engine applications.
>>>>>>>>
>>>>>>>>                             (I'm trying to understand why
>>>>>>>>                             you've seen many test agent
>>>>>>>>                             processes. It should not happen.)
>>>>>>>>
>>>>>>>>                             Andrew.
>>>>>>>>
>>>>>>>>                             On 8/25/23 17:41, Andrew Rybchenko
>>>>>>>>                             wrote:
>>>>>>>>>                             On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                             Two of our systems (the Test
>>>>>>>>>>                             Engine runner and the DUT host)
>>>>>>>>>>                             are running Ubuntu 20.04 LTS,
>>>>>>>>>>                             however this morning I noticed
>>>>>>>>>>                             that the tester system (the one
>>>>>>>>>>                             having issues) is running Ubuntu
>>>>>>>>>>                             22.04 LTS.
>>>>>>>>>>                             This could be the source of the
>>>>>>>>>>                             problem. I encountered a
>>>>>>>>>>                             dependency issue trying to run
>>>>>>>>>>                             the Test Engine on 22.04 LTS, so
>>>>>>>>>>                             I downgraded the system. Since
>>>>>>>>>>                             the tester is also the host
>>>>>>>>>>                             having connection issues, I will
>>>>>>>>>>                             try downgrading that system to
>>>>>>>>>>                             20.04, and see if that changes
>>>>>>>>>>                             anything.
>>>>>>>>>
>>>>>>>>>                             Unlikely, but who knows. We run
>>>>>>>>>                             tests (DUTs) on Ubuntu 20.04,
>>>>>>>>>                             Ubuntu 22.04, Ubuntu 22.10, Ubuntu
>>>>>>>>>                             23.04, Debian 11 and Fedora 38
>>>>>>>>>                             every night.
>>>>>>>>>                             Right now Debian 11 is used for
>>>>>>>>>                             test engine in nightly regressions.
>>>>>>>>>
>>>>>>>>>                             I'll double-check test engine on
>>>>>>>>>                             Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>
>>>>>>>>>>                             I did try passing in the
>>>>>>>>>>                             "--vg-rcf" argument to the run.sh
>>>>>>>>>>                             script of the test suite after
>>>>>>>>>>                             installing valgrind, but there
>>>>>>>>>>                             was no additional output that I saw.
>>>>>>>>>
>>>>>>>>>                             Sorry, I should valgrind output
>>>>>>>>>                             should be in valgrind.te_rcf
>>>>>>>>>                             (direction where you run test engine).
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             I will try pulling in the changes
>>>>>>>>>>                             you've pushed up, and will see if
>>>>>>>>>>                             that fixes anything.
>>>>>>>>>>
>>>>>>>>>>                             Thanks,
>>>>>>>>>>                             Adam
>>>>>>>>>>
>>>>>>>>>>                             On Fri, Aug 25, 2023 at 9:57 AM
>>>>>>>>>>                             Andrew Rybchenko
>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>                             wrote:
>>>>>>>>>>
>>>>>>>>>>                                 Hello Adam,
>>>>>>>>>>
>>>>>>>>>>                                 On 8/24/23 23:54, Andrew
>>>>>>>>>>                                 Rybchenko wrote:
>>>>>>>>>>>                                 I'd like to try to repeat
>>>>>>>>>>>                                 the problem locally. Which
>>>>>>>>>>>                                 Linux distro is running on
>>>>>>>>>>>                                 test engine and agents?
>>>>>>>>>>>
>>>>>>>>>>>                                 In fact I know one problem
>>>>>>>>>>>                                 with Debian 12 and Fedora 38
>>>>>>>>>>>                                 and we have
>>>>>>>>>>>                                 patch in review to fix it,
>>>>>>>>>>>                                 however, the behaviour is
>>>>>>>>>>>                                 different in
>>>>>>>>>>>                                 this case, so it is unlike
>>>>>>>>>>>                                 the same problem.
>>>>>>>>>>
>>>>>>>>>>                                 I've just published a new tag
>>>>>>>>>>                                 which fixes known test engine
>>>>>>>>>>                                 side problems on Debian 12
>>>>>>>>>>                                 and Fedora 38.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                 One more idea is to install
>>>>>>>>>>>                                 valgrind on the test engine
>>>>>>>>>>>                                 host and
>>>>>>>>>>>                                 run with option --vg-rcf to
>>>>>>>>>>>                                 check if something weird is
>>>>>>>>>>>                                 happening.
>>>>>>>>>>>
>>>>>>>>>>>                                 What I don't understand
>>>>>>>>>>>                                 right now is why I see just
>>>>>>>>>>>                                 one failed attempt
>>>>>>>>>>>                                 to connect in your log.txt
>>>>>>>>>>>                                 and then Logger shutdown
>>>>>>>>>>>                                 after 9 minutes.
>>>>>>>>>>>
>>>>>>>>>>>                                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                 On 8/24/23 23:29, Adam
>>>>>>>>>>>                                 Hassick wrote:
>>>>>>>>>>>>                                  > Is there any firewall in
>>>>>>>>>>>>                                 the network or on test
>>>>>>>>>>>>                                 hosts which could block
>>>>>>>>>>>>                                 incoming TCP connection to
>>>>>>>>>>>>                                 the port 23571
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 from the host where you run
>>>>>>>>>>>>                                 test engine?
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Our test engine host and
>>>>>>>>>>>>                                 the testbed are on the same
>>>>>>>>>>>>                                 subnet. The connection does
>>>>>>>>>>>>                                 work sometimes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                  > If behaviour the same on
>>>>>>>>>>>>                                 the next try and you see
>>>>>>>>>>>>                                 that test agent is kept
>>>>>>>>>>>>                                 running, could you check using
>>>>>>>>>>>>                                  >
>>>>>>>>>>>>                                  > # netstat -tnlp
>>>>>>>>>>>>                                  >
>>>>>>>>>>>>                                  > that Test Agent is
>>>>>>>>>>>>                                 listening on the port and
>>>>>>>>>>>>                                 try to establish TCP
>>>>>>>>>>>>                                 connection from test agent
>>>>>>>>>>>>                                 using
>>>>>>>>>>>>                                  >
>>>>>>>>>>>>                                  > $ telnet
>>>>>>>>>>>>                                 iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 23571
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                  >
>>>>>>>>>>>>                                  > and check if TCP
>>>>>>>>>>>>                                 connection could be
>>>>>>>>>>>>                                 established.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 I was able to replicate the
>>>>>>>>>>>>                                 same behavior again, where
>>>>>>>>>>>>                                 it hangs while RCF is
>>>>>>>>>>>>                                 trying to start.
>>>>>>>>>>>>                                 Running this command, I see
>>>>>>>>>>>>                                 this in the output:
>>>>>>>>>>>>
>>>>>>>>>>>>                                 tcp        0    0
>>>>>>>>>>>>                                 0.0.0.0:23571
>>>>>>>>>>>>                                 <http://0.0.0.0:23571>
>>>>>>>>>>>>                                 <http://0.0.0.0:23571>
>>>>>>>>>>>>                                 <http://0.0.0.0:23571>
>>>>>>>>>>>>                                 0.0.0.0:* LISTEN  18599/ta
>>>>>>>>>>>>
>>>>>>>>>>>>                                 So it seems like it is
>>>>>>>>>>>>                                 listening on the correct port.
>>>>>>>>>>>>                                 Additionally, I was able to
>>>>>>>>>>>>                                 connect to the Tester
>>>>>>>>>>>>                                 machine from our Test
>>>>>>>>>>>>                                 Engine host using telnet.
>>>>>>>>>>>>                                 It printed the PID of the
>>>>>>>>>>>>                                 process once the connection
>>>>>>>>>>>>                                 was opened.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 I tried running the "ta"
>>>>>>>>>>>>                                 application manually on the
>>>>>>>>>>>>                                 command line, and it didn't
>>>>>>>>>>>>                                 print anything at all.
>>>>>>>>>>>>                                 Maybe the issue is
>>>>>>>>>>>>                                 something on the Test
>>>>>>>>>>>>                                 Engine side.
>>>>>>>>>>>>
>>>>>>>>>>>>                                 On Thu, Aug 24, 2023 at
>>>>>>>>>>>>                                 2:35 PM Andrew Rybchenko
>>>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>                                 wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                      > On the tester host
>>>>>>>>>>>>                                 (which appears to be the
>>>>>>>>>>>>                                 Peer agent), there
>>>>>>>>>>>>                                     are four processes that
>>>>>>>>>>>>                                 I see running, which look
>>>>>>>>>>>>                                 like the test
>>>>>>>>>>>>                                     agent processes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Before the next try I'd
>>>>>>>>>>>>                                 recommend to kill these
>>>>>>>>>>>>                                 processes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Is there any firewall
>>>>>>>>>>>>                                 in the network or on test
>>>>>>>>>>>>                                 hosts which could
>>>>>>>>>>>>                                     block incoming TCP
>>>>>>>>>>>>                                 connection to the port 23571
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 from the host
>>>>>>>>>>>>                                     where you run test engine?
>>>>>>>>>>>>
>>>>>>>>>>>>                                     If behaviour the same
>>>>>>>>>>>>                                 on the next try and you see
>>>>>>>>>>>>                                 that test agent is
>>>>>>>>>>>>                                     kept running, could you
>>>>>>>>>>>>                                 check using
>>>>>>>>>>>>
>>>>>>>>>>>>                                     # netstat -tnlp
>>>>>>>>>>>>
>>>>>>>>>>>>                                     that Test Agent is
>>>>>>>>>>>>                                 listening on the port and
>>>>>>>>>>>>                                 try to establish TCP
>>>>>>>>>>>>                                     connection from test
>>>>>>>>>>>>                                 agent using
>>>>>>>>>>>>
>>>>>>>>>>>>                                     $ telnet
>>>>>>>>>>>>                                 iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 23571
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>                                     and check if TCP
>>>>>>>>>>>>                                 connection could be
>>>>>>>>>>>>                                 established.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Another idea is to
>>>>>>>>>>>>                                 login Tester under root as
>>>>>>>>>>>>                                 testing does, get
>>>>>>>>>>>>                                     start TA command from
>>>>>>>>>>>>                                 the log and try it by hands
>>>>>>>>>>>>                                 without -n and
>>>>>>>>>>>>                                     remove extra escaping.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     # sudo
>>>>>>>>>>>>                                 PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>                                 LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>                                 /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>                                 Peer 23571
>>>>>>>>>>>>                                 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Hopefully in this case
>>>>>>>>>>>>                                 test agent directory
>>>>>>>>>>>>                                 remains in the /tmp and
>>>>>>>>>>>>                                     you don't need to copy
>>>>>>>>>>>>                                 it as testing does.
>>>>>>>>>>>>                                     May be output could
>>>>>>>>>>>>                                 shed some light on what's
>>>>>>>>>>>>                                 going on.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On 8/24/23 17:30, Adam
>>>>>>>>>>>>                                 Hassick wrote:
>>>>>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     This is the output
>>>>>>>>>>>>>                                 that I see in the terminal
>>>>>>>>>>>>>                                 when this failure
>>>>>>>>>>>>>                                     occurs, after the test
>>>>>>>>>>>>>                                 agent binaries build and
>>>>>>>>>>>>>                                 the test engine
>>>>>>>>>>>>>                                     starts:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Platform default build
>>>>>>>>>>>>>                                 - pass
>>>>>>>>>>>>>                                     Simple RCF consistency
>>>>>>>>>>>>>                                 check succeeded
>>>>>>>>>>>>>                                 --->>> Starting Logger...done
>>>>>>>>>>>>>                                 --->>> Starting
>>>>>>>>>>>>>                                 RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>                                 Connection timed
>>>>>>>>>>>>>                                     out
>>>>>>>>>>>>>                                 iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Then, it hangs here
>>>>>>>>>>>>>                                 until I kill the "te_rcf"
>>>>>>>>>>>>>                                 and "te_tee"
>>>>>>>>>>>>>                                     processes. I let it
>>>>>>>>>>>>>                                 hang for around 9 minutes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     On the tester host
>>>>>>>>>>>>>                                 (which appears to be the
>>>>>>>>>>>>>                                 Peer agent), there are
>>>>>>>>>>>>>                                     four processes that I
>>>>>>>>>>>>>                                 see running, which look
>>>>>>>>>>>>>                                 like the test agent
>>>>>>>>>>>>>                                     processes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     ta.Peer is an empty
>>>>>>>>>>>>>                                 file. I've attached the
>>>>>>>>>>>>>                                 log.txt from this run.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                      - Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     On Thu, Aug 24, 2023
>>>>>>>>>>>>>                                 at 4:22 AM Andrew Rybchenko
>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>                                 wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Hi Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Yes,
>>>>>>>>>>>>>                                 TE_RCFUNIX_TIMEOUT is in
>>>>>>>>>>>>>                                 seconds. I've double-checked
>>>>>>>>>>>>>                                         that it goes to
>>>>>>>>>>>>>                                 'copy_timeout' in
>>>>>>>>>>>>>                                 ts-conf/rcf.conf.
>>>>>>>>>>>>>                                 Description in in
>>>>>>>>>>>>>                                 doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>                                         says that
>>>>>>>>>>>>>                                 copy_timeout is in seconds
>>>>>>>>>>>>>                                 and implementation in
>>>>>>>>>>>>>                                 lib/rcfunix/rcfunix.c
>>>>>>>>>>>>>                                 passes the value to
>>>>>>>>>>>>>                                 select() tv_sec.
>>>>>>>>>>>>>                                 Theoretically select()
>>>>>>>>>>>>>                                 could be interrupted by
>>>>>>>>>>>>>                                 signal, but I
>>>>>>>>>>>>>                                         think it is
>>>>>>>>>>>>>                                 unlikely here.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         I'm not sure that
>>>>>>>>>>>>>                                 I understand what do you
>>>>>>>>>>>>>                                 mean by RCF
>>>>>>>>>>>>>                                 connection timeout. Does
>>>>>>>>>>>>>                                 it happen on TE startup
>>>>>>>>>>>>>                                 when RCF
>>>>>>>>>>>>>                                         starts test
>>>>>>>>>>>>>                                 agents. If so,
>>>>>>>>>>>>>                                 TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>>>>                                 help. Or
>>>>>>>>>>>>>                                         does it happen
>>>>>>>>>>>>>                                 when tests are in
>>>>>>>>>>>>>                                 progress, e.g. in the middle
>>>>>>>>>>>>>                                         of a test. If so,
>>>>>>>>>>>>>                                 TE_RCFUNIX_TIMEOUT is
>>>>>>>>>>>>>                                 unrelated and most
>>>>>>>>>>>>>                                         likely either host
>>>>>>>>>>>>>                                 with test agent dies or
>>>>>>>>>>>>>                                 test agent itself
>>>>>>>>>>>>>                                 crashes. It would be
>>>>>>>>>>>>>                                 easier for me if classify
>>>>>>>>>>>>>                                 it if you share
>>>>>>>>>>>>>                                         text log (log.txt,
>>>>>>>>>>>>>                                 full or just corresponding
>>>>>>>>>>>>>                                 fragment with
>>>>>>>>>>>>>                                         some context).
>>>>>>>>>>>>>                                 Also content of ta.DPDK or
>>>>>>>>>>>>>                                 ta.Peer file
>>>>>>>>>>>>>                                 depending on which agent
>>>>>>>>>>>>>                                 has problems could shed
>>>>>>>>>>>>>                                 some light.
>>>>>>>>>>>>>                                 Corresponding files
>>>>>>>>>>>>>                                 contain stdout/stderr of
>>>>>>>>>>>>>                                 test agents.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         On 8/23/23 17:45,
>>>>>>>>>>>>>                                 Adam Hassick wrote:
>>>>>>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         I've set up a
>>>>>>>>>>>>>>                                 test rig repository here,
>>>>>>>>>>>>>>                                 and have created
>>>>>>>>>>>>>>                                 configurations for our
>>>>>>>>>>>>>>                                 development testbed based
>>>>>>>>>>>>>>                                 off of the
>>>>>>>>>>>>>>                                 examples.
>>>>>>>>>>>>>>                                         We've been able
>>>>>>>>>>>>>>                                 to get the test suite to
>>>>>>>>>>>>>>                                 run manually on
>>>>>>>>>>>>>>                                 Mellanox CX5 devices once.
>>>>>>>>>>>>>>                                 However, we are running
>>>>>>>>>>>>>>                                 into an issue where, when
>>>>>>>>>>>>>>                                 RCF starts,
>>>>>>>>>>>>>>                                         the RCF
>>>>>>>>>>>>>>                                 connection times out very
>>>>>>>>>>>>>>                                 frequently. We aren't sure
>>>>>>>>>>>>>>                                         why this is the case.
>>>>>>>>>>>>>>                                         It works
>>>>>>>>>>>>>>                                 sometimes, but most of
>>>>>>>>>>>>>>                                 the time when we try to run
>>>>>>>>>>>>>>                                         the test engine,
>>>>>>>>>>>>>>                                 it encounters this issue.
>>>>>>>>>>>>>>                                         I've tried
>>>>>>>>>>>>>>                                 changing the RCF port by
>>>>>>>>>>>>>>                                 setting
>>>>>>>>>>>>>>                                 "TE_RCF_PORT=<some port
>>>>>>>>>>>>>>                                 number>" and rebooting
>>>>>>>>>>>>>>                                 the testbed
>>>>>>>>>>>>>>                                 machines. Neither seems
>>>>>>>>>>>>>>                                 to fix the issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         It also seems
>>>>>>>>>>>>>>                                 like the timeout takes
>>>>>>>>>>>>>>                                 far longer than 60
>>>>>>>>>>>>>>                                 seconds, even when
>>>>>>>>>>>>>>                                 running "export
>>>>>>>>>>>>>>                                 TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>                                         before I try to
>>>>>>>>>>>>>>                                 run the test suite.
>>>>>>>>>>>>>>                                         I assume the unit
>>>>>>>>>>>>>>                                 for this variable is seconds?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                 Thanks,
>>>>>>>>>>>>>>                                         Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         On Mon, Aug 21,
>>>>>>>>>>>>>>                                 2023 at 10:19 AM Adam Hassick
>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>                                 <ahassick@iol.unh.edu
>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>                                 wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                 Thanks, I've cloned the
>>>>>>>>>>>>>>                                 example repository and
>>>>>>>>>>>>>>                                 will start
>>>>>>>>>>>>>>                                 setting up a
>>>>>>>>>>>>>>                                 configuration for our
>>>>>>>>>>>>>>                                 development testbed
>>>>>>>>>>>>>>                                 today. I'll let you know
>>>>>>>>>>>>>>                                 if I run into any
>>>>>>>>>>>>>>                                 difficulties
>>>>>>>>>>>>>>                                             or have any
>>>>>>>>>>>>>>                                 questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                              - Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On Sun, Aug
>>>>>>>>>>>>>>                                 20, 2023 at 4:40 AM
>>>>>>>>>>>>>>                                 Andrew Rybchenko
>>>>>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>                                 wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Hi Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>                                 published
>>>>>>>>>>>>>>                                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>                                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>                                 <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>>                                                 Hopefully
>>>>>>>>>>>>>>                                 it will help to define
>>>>>>>>>>>>>>                                 your test rigs and
>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>                                 successfully run some
>>>>>>>>>>>>>>                                 tests manually. Feel free to
>>>>>>>>>>>>>>                                                 ask any
>>>>>>>>>>>>>>                                 questions and I'll answer
>>>>>>>>>>>>>>                                 here and try to
>>>>>>>>>>>>>>                                                 update
>>>>>>>>>>>>>>                                 documentation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Meanwhile
>>>>>>>>>>>>>>                                 I'll prepare missing bits
>>>>>>>>>>>>>>                                 for steps (2) and
>>>>>>>>>>>>>>                                                 (3).
>>>>>>>>>>>>>>                                                 Hopefully
>>>>>>>>>>>>>>                                 everything is in place
>>>>>>>>>>>>>>                                 for step (4), but we
>>>>>>>>>>>>>>                                                 need to
>>>>>>>>>>>>>>                                 make steps (2) and (3) first.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>                                 8/18/23 21:40, Andrew
>>>>>>>>>>>>>>                                 Rybchenko wrote:
>>>>>>>>>>>>>>>                                 Hi Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 > I've
>>>>>>>>>>>>>>>                                 conferred with the rest
>>>>>>>>>>>>>>>                                 of the team, and we
>>>>>>>>>>>>>>>                                                 think it
>>>>>>>>>>>>>>>                                 would be best to move
>>>>>>>>>>>>>>>                                 forward with mainly
>>>>>>>>>>>>>>>                                                 option B.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 OK, I'll
>>>>>>>>>>>>>>>                                 provide the sample on
>>>>>>>>>>>>>>>                                 Monday for you. It is
>>>>>>>>>>>>>>>                                                 almost
>>>>>>>>>>>>>>>                                 ready right now, but I
>>>>>>>>>>>>>>>                                 need to double-check
>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>                                 before publishing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Regards,
>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>                                 8/17/23 20:03, Adam
>>>>>>>>>>>>>>>                                 Hassick wrote:
>>>>>>>>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>                                 adding the CI mailing
>>>>>>>>>>>>>>>>                                 list to this
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                 conversation. Others in
>>>>>>>>>>>>>>>>                                 the community might find
>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>                                 conversation valuable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 We do
>>>>>>>>>>>>>>>>                                 want to run testing on
>>>>>>>>>>>>>>>>                                 a regular basis. The
>>>>>>>>>>>>>>>>                                                 Jenkins
>>>>>>>>>>>>>>>>                                 integration will be
>>>>>>>>>>>>>>>>                                 very useful for us, as
>>>>>>>>>>>>>>>>                                                 most of
>>>>>>>>>>>>>>>>                                 our CI is orchestrated
>>>>>>>>>>>>>>>>                                 by Jenkins.
>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>                                 conferred with the rest
>>>>>>>>>>>>>>>>                                 of the team, and we
>>>>>>>>>>>>>>>>                                                 think
>>>>>>>>>>>>>>>>                                 it would be best to
>>>>>>>>>>>>>>>>                                 move forward with mainly
>>>>>>>>>>>>>>>>                                                 option B.
>>>>>>>>>>>>>>>>                                                 If you
>>>>>>>>>>>>>>>>                                 would like to know
>>>>>>>>>>>>>>>>                                 anything about our
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                 testbeds that would
>>>>>>>>>>>>>>>>                                 help you with creating an
>>>>>>>>>>>>>>>>                                                 example
>>>>>>>>>>>>>>>>                                 ts-rigs repo, I'd be
>>>>>>>>>>>>>>>>                                 happy to answer any
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                 questions you have.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 We have
>>>>>>>>>>>>>>>>                                 multiple test rigs (we
>>>>>>>>>>>>>>>>                                 call these
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                 "DUT-tester pairs")
>>>>>>>>>>>>>>>>                                 that we run our existing
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                 hardware testing on,
>>>>>>>>>>>>>>>>                                 with differing network
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                 hardware and CPU
>>>>>>>>>>>>>>>>                                 architecture. I figured
>>>>>>>>>>>>>>>>                                 this might
>>>>>>>>>>>>>>>>                                                 be an
>>>>>>>>>>>>>>>>                                 important detail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 On Thu,
>>>>>>>>>>>>>>>>                                 Aug 17, 2023 at
>>>>>>>>>>>>>>>>                                 11:44 AM Andrew Rybchenko
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>                                 wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 Greatings Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>                                 happy to hear that
>>>>>>>>>>>>>>>>                                 you're trying to bring
>>>>>>>>>>>>>>>>                                                     it up.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     As
>>>>>>>>>>>>>>>>                                 I understand the final
>>>>>>>>>>>>>>>>                                 goal is to run it on
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 regular basis. So, we
>>>>>>>>>>>>>>>>                                 need to make it properly
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 from the very beginning.
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 Bring up of all
>>>>>>>>>>>>>>>>                                 features consists of 4
>>>>>>>>>>>>>>>>                                 steps:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     1.
>>>>>>>>>>>>>>>>                                 Create site-specific
>>>>>>>>>>>>>>>>                                 repository (we call it
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 ts-rigs) which contains
>>>>>>>>>>>>>>>>                                 information about test
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 rigs and other
>>>>>>>>>>>>>>>>                                 site-specific
>>>>>>>>>>>>>>>>                                 information like
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 where to send mails,
>>>>>>>>>>>>>>>>                                 where to store logs etc.
>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>                                 is required for manual
>>>>>>>>>>>>>>>>                                 execution as well,
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 since test rigs
>>>>>>>>>>>>>>>>                                 description is
>>>>>>>>>>>>>>>>                                 essential. I'll
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 return to the topic below.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     2.
>>>>>>>>>>>>>>>>                                 Setup logs storage for
>>>>>>>>>>>>>>>>                                 automated runs.
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 Basically it is a disk
>>>>>>>>>>>>>>>>                                 space plus apache2 web
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 server with few CGI
>>>>>>>>>>>>>>>>                                 scripts which help a lot to
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 save disk space.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     3.
>>>>>>>>>>>>>>>>                                 Setup Bublik web
>>>>>>>>>>>>>>>>                                 application which provides
>>>>>>>>>>>>>>>>                                                     web
>>>>>>>>>>>>>>>>                                 interface to view
>>>>>>>>>>>>>>>>                                 testing results. Same as
>>>>>>>>>>>>>>>>                                 https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>                                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>                                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     4.
>>>>>>>>>>>>>>>>                                 Setup Jenkins to run
>>>>>>>>>>>>>>>>                                 tests on regularly,
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 save logs in log
>>>>>>>>>>>>>>>>                                 storage (2) and import
>>>>>>>>>>>>>>>>                                 it to
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 bublik (3).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 Last few month we spent
>>>>>>>>>>>>>>>>                                 on our homework to make
>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>                                 simpler to bring up
>>>>>>>>>>>>>>>>                                 automated execution
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 using Jenkins -
>>>>>>>>>>>>>>>>                                 https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>                                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>                                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 Corresponding bits in
>>>>>>>>>>>>>>>>                                 dpdk-ethdev-ts will be
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 available tomorrow.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 Let's return to the
>>>>>>>>>>>>>>>>                                 step (1).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 Unfortunately there is
>>>>>>>>>>>>>>>>                                 no publicly available
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 example of the ts-rigs
>>>>>>>>>>>>>>>>                                 repository since
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 sensitive site-specific
>>>>>>>>>>>>>>>>                                 information is located
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 there. But I'm ready to
>>>>>>>>>>>>>>>>                                 help you to create it
>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>                                 UNH. I see two options
>>>>>>>>>>>>>>>>                                 here:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     (A)
>>>>>>>>>>>>>>>>                                 I'll ask questions and
>>>>>>>>>>>>>>>>                                 based on your
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 answers will create the
>>>>>>>>>>>>>>>>                                 first draft with my
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 comments.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     (B)
>>>>>>>>>>>>>>>>                                 I'll make a
>>>>>>>>>>>>>>>>                                 template/example
>>>>>>>>>>>>>>>>                                 ts-rigs repo,
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 publish it and you'll
>>>>>>>>>>>>>>>>                                 create UNH ts-rigs based
>>>>>>>>>>>>>>>>                                                     on it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Of
>>>>>>>>>>>>>>>>                                 course, I'll help to
>>>>>>>>>>>>>>>>                                 debug and finally bring
>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>                                 up in any case.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     (A)
>>>>>>>>>>>>>>>>                                 is a bit simpler for me
>>>>>>>>>>>>>>>>                                 and you, but (B) is
>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>                                 bit more generic and
>>>>>>>>>>>>>>>>                                 will help other
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 potential users to
>>>>>>>>>>>>>>>>                                 bring it up.
>>>>>>>>>>>>>>>>                                                     We
>>>>>>>>>>>>>>>>                                 can combine (A)+(B).
>>>>>>>>>>>>>>>>                                 I.e. start from (A).
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                 What do you think?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Thanks,
>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>                                 8/17/23 15:18,
>>>>>>>>>>>>>>>>                                 Konstantin Ushakov wrote:
>>>>>>>>>>>>>>>>>                                 Greetings Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                 Thanks for contacting
>>>>>>>>>>>>>>>>>                                 us. I copy Andrew who
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                 would be happy to help
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                 Thanks,
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                 Konstantin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                 On 16 Aug 2023, at
>>>>>>>>>>>>>>>>>>                                 21:50, Adam Hassick
>>>>>>>>>>>>>>>>>>                                 <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                 wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                     
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 Greetings Konstantin,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                 am in the process of
>>>>>>>>>>>>>>>>>>                                 setting up the DPDK
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 Poll Mode Driver test
>>>>>>>>>>>>>>>>>>                                 suite as an addition to
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 our testing coverage
>>>>>>>>>>>>>>>>>>                                 for DPDK at the UNH lab.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                 have some questions
>>>>>>>>>>>>>>>>>>                                 about how to set the
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 test suite arguments.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                 have been able to
>>>>>>>>>>>>>>>>>>                                 configure the Test Engine
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 to connect to the
>>>>>>>>>>>>>>>>>>                                 hosts in the testbed. The
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 RCF, Configurator,
>>>>>>>>>>>>>>>>>>                                 and Tester all begin to
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 run, however the
>>>>>>>>>>>>>>>>>>                                 prelude of the test suite
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 fails to run.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                 https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>                                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>                                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 The documentation
>>>>>>>>>>>>>>>>>>                                 mentions that there are
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 several test
>>>>>>>>>>>>>>>>>>                                 parameters for the
>>>>>>>>>>>>>>>>>>                                 test suite,
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 like for the IUT test
>>>>>>>>>>>>>>>>>>                                 link MAC, etc. These
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 seem like they would
>>>>>>>>>>>>>>>>>>                                 need to be set somewhere
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 to run many of the tests.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                 see in the Test
>>>>>>>>>>>>>>>>>>                                 Engine documentation,
>>>>>>>>>>>>>>>>>>                                 there
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 are instructions on
>>>>>>>>>>>>>>>>>>                                 how to create new
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 parameters for test
>>>>>>>>>>>>>>>>>>                                 suites in the Tester
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 configuration, but
>>>>>>>>>>>>>>>>>>                                 there is nothing in the
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 user guide or in the
>>>>>>>>>>>>>>>>>>                                 Tester guide for how to
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 set the arguments for
>>>>>>>>>>>>>>>>>>                                 the parameters when
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 running the test
>>>>>>>>>>>>>>>>>>                                 suite that I can
>>>>>>>>>>>>>>>>>>                                 find. I'm
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 not sure if I need to
>>>>>>>>>>>>>>>>>>                                 write my own Tester
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 config, or if I
>>>>>>>>>>>>>>>>>>                                 should be setting
>>>>>>>>>>>>>>>>>>                                 these in
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 some other way.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 How should these
>>>>>>>>>>>>>>>>>>                                 values be set?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 I'm also not sure
>>>>>>>>>>>>>>>>>>                                 what environment
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 variables/arguments
>>>>>>>>>>>>>>>>>>                                 are strictly necessary or
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 which are optional.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 Regards,
>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 --
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 *Adam Hassick*
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>>>>>>>>>                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>                                 Hassick*
>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>                                 Developer
>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>                                 InterOperability Lab
>>>>>>>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>>>>>>>                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>                                 (603) 475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             -- *Adam Hassick*
>>>>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>>>>>                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                             +1 (603) 475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         --         *Adam
>>>>>>>>>>>>>>                                 Hassick*
>>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>>                                 InterOperability Lab
>>>>>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>>>>>                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     -- *Adam Hassick*
>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>>>>                                 <http://iol.unh.edu>
>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                 -- 
>>>>>>>>>>>>                                 *Adam Hassick*
>>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>>>                                 <http://iol.unh.edu>
>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             --
>>>>>>>>>>                             *Adam Hassick*
>>>>>>>>>>                             Senior Developer
>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>>                             iol.unh.edu
>>>>>>>>>>                             <https://www.iol.unh.edu/>
>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                         --
>>>>>>>>                         *Adam Hassick*
>>>>>>>>                         Senior Developer
>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                         +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                     --
>>>>>>                     *Adam Hassick*
>>>>>>                     Senior Developer
>>>>>>                     UNH InterOperability Lab
>>>>>>                     ahassick@iol.unh.edu
>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                     +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>             --
>>>>             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         --
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>     --
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>

[-- Attachment #2: Type: text/html, Size: 180866 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 48911 bytes --]

Hi Andrew and Konstantin,

Thank you for adding the tester-dial feature, this opens up the possibility
for us to do CI integrated testing in the future.

Our Mellanox pass rate is similar to yours (about ~2400 passing, ~4400
failing), however our Intel pass rates are far worse.
I will try running tests on the XL710 with the trc-tags argument set and
see if it improves the pass rate.
Another thing I noticed in the results you uploaded is that the results are
tagged with vfio-pci and not i40e.
Though in the environment dump, the driver on the test machine and the DUT
are set to use the i40e driver. Is this important at all?

There isn't anything preventing us from pushing our results up to the
existing Bublik instance running at ts-factory.io that I can think of at
the moment.
We will have to work out how to submit our results to your Bublik instance
in a controlled and secure manner in that case.
As far as I know we won't need access controls for the results themselves.
I'll discuss this with Patrick and will let you know once we confirm that
it's fine.

Thanks,
Adam

On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> On 9/18/23 09:23, Konstantin Ushakov wrote:
>
> Hi Andrew,
>
> should we always auto-assign the tags or you don’t do it since it slows
> down (by some seconds) the TE startup?
>
>
> Tags are auto-assigned, but I guess it differs in Adam's case since NIC is
> a bit different. Below test will help to understand if it is the root cause
> of very different expectations. If pass rate will be close to mine, I'll
> simply update TRC database to share expectations for mine NIC and NIC used
> by Adam.
>
> Hi Adam,
>
> I think I second the question from Andrew - happy to help you with the
> triage so that we get to the same baseline. Do you have a good way for us
> to share the logs? I.e. say upload to ts-factory if we add strict
> permissions system so it’s not publishing or any other way.
>
> Thanks,
> Konstantin
>
> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>
> Hi Adam,
>
> I've uploaded fresh testing results to ts-factory.io [1] to be on the
> same page.
>
> I think I know why your and mine results on Intel 710 series NICs differ
> so much. Testing results expectations database (dpdk-ethdev-ts/trc/*) is
> filled in in terms of TRC tags.  I.e. expectations depends on TRC tags
> discovered by helper scripts when testing is started. These tags identify
> various aspects of what is tested. Ideally expectations should be written
> in terms of root cause of the expected behaviour. If it is a driver
> expectations, driver tag should be used. If it is HW limitation, tags with
> PCI IDs should be used. However, it is not always easy to classify it
> correctly if you're not involved in driver development. So, in order case
> expectations for 710's Intel are filled in in terms of PCI IDs. I guess PCI
> ID differ in your case and that's why expectations filled in for my NIC do
> not apply to your runs.
>
> Just try to add the following option when you run on your 710's Intel in
> order to mimic mine and see if it helps to achieve better pass rate.
> --trc-tag=pci-8086-1572
>
> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for
> --tester-dial option. It should have better coverage now.
>
> Andrew.
>
> [1]
> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>
> On 9/13/23 18:45, Andrew Rybchenko wrote:
>
> Hi Adam,
>
> I've pushed new TE tag v1.20.0 which supported a new command-line option
> --tester-dial=NUM where NUM is from 0 to 100. it allows to choose
> percentage of tests to run. If you want stable set, you should pass
> --tester-random-seed=0 (or other integer). It is the first sketch and we
> have plans to improve it, but feedback would be welcome.
>
> > Is it needed on the tester?
>
> It is hard to say if it is strictly required for simple tests. However, it
> is better to update Tester as well, since performance tests run DPDK on
> Tester as well.
>
> > Are there any other manual setup steps for these devices that I might be
> missing?
>
> I don't remember anything else.
>
> I think it is better to get down to details and take a look at logs. I'm
> ready to help with it and explain what's happening there. May be it will
> help to understand if it is a problem with setup/configuration.
>
> Text logs are not very convenient. Ideally logs should be imported to
> bublik, however, manual runs do not provide all required artifacts right
> now (Jenkins jobs generate all required artifacts).
> Other option is 'tmp_raw_log' file (should be packed to make it smaller)
> which could be converted to various log formats.
> Would it be OK for you if I import your logs to bublik at ts-factory.io?
> Or is it a problem that it is publicly available?
> Would it help if we add authentication and access control there?
>
> Andrew.
>
> On 9/8/23 17:57, Adam Hassick wrote:
>
> Hi Andrew,
>
> I have a couple questions about needed setup of the NICs for the ethdev
> test suite.
>
> Our MCX5s and XL710s are failing the checkup tests. The pass rate appears
> to be much worse on the XL710s (40 of 73 tests failed, 3 passed
> unexpectedly).
>
> For the XL710s, I've updated the driver and NVM versions to match the
> minimum supported versions in the compatibility matrix found on the DPDK
> documentation. This did not change the failure rate much.
> For the MCX5s, I've installed the latest LTS version of the OFED
> bifurcated driver on the DUT. Is it needed on the tester?
>
> Are there any other manual setup steps for these devices that I might be
> missing?
>
> Thanks,
> Adam
>
> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu> wrote:
>
>> Hi Andrew,
>>
>> Yes, I copied the X710 configs to set up XL710 configs. I changed the
>> environment variable names from the X710 suffix to XL710 suffix in the
>> script, and forgot to change them in the corresponding environment file.
>> That fixed the issue.
>>
>> I got the checkup tests working on the XL710 now. Most of them are
>> failing, which leads me to believe this is an issue with our testbed. Based
>> on the DPDK documentation for i40e, the firmware and driver versions are
>> much older than what DPDK 22.11 LTS and main prefer, so I'll try updating
>> those.
>>
>> For now I'm working on getting the XL710 checkup tests passing, and will
>> pick up getting the E810 configured properly next. I'll let you know if I
>> run into any more issues in relation to the test engine.
>>
>> Thanks,
>> Adam
>>
>> On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> On 9/5/23 18:01, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> The compilation warning issue is now resolved. Again, thank you guys for
>>> fixing this for us. I can run the tests on the Mellanox CX5s again, however
>>> I'm running into a couple new issues with running the prologues on the
>>> Intel cards.
>>>
>>> When running testing on the Intel XL710s, I see this error appear in the
>>> log:
>>>
>>> ERROR  prologue  Environment LIB  14:16:13.650
>>>> Too few networks in available configuration (0) in comparison with
>>>> required (1)
>>>>
>>>
>>> This seems like a trivial configuration error, perhaps this is something
>>> I need to set up in ts-rigs. I briefly searched through the examples there
>>> and didn't see any mention of how to set up a network.
>>> I will attach this log just in case you need more information.
>>>
>>>
>>> Unfortunately logs are insufficient to understand it. I've pushed new
>>> tag to TE v1.19.0 which add log message with TE_* environment variables.
>>> Most likely something is wrong with variables which are used as
>>> conditions when available networks are defined in
>>> ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>> TE_PCI_INSTANCE_IUT_TST1
>>> TE_PCI_INSTANCE_IUT_TST1a
>>> TE_PCI_INSTANCE_TST1a_IUT
>>> TE_PCI_INSTANCE_TST1_IUT
>>> My guess it that you change naming a bit, but script like
>>> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>>>
>>> There is a different error when running on the Intel E810s. It appears
>>> to me like it starts DPDK, does some configuration inside DPDK and on the
>>> device, and then fails to bring the device back up. Since this error seems
>>> very non-trivial, I will also attach this log.
>>>
>>>
>>> This one is a bit simpler. Few lines after the first ERROR in log I see
>>> the following:
>>> WARN  RCF  DPDK  13:06:00.144
>>> ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)
>>> ice_rx_queue_start(): fail to program RX queue 0
>>> ice_dev_start(): fail to start Rx queue 0
>>> Device with port_id=0 already stopped
>>>
>>> It is stdout/stderr from test agent which runs DPDK. Same logs in plain
>>> format are available in ta.DPDK file.
>>> I'm not an expert here, but I vaguely remember that E810 requires
>>> correct firmware and DDP to be loaded.
>>> There is some information in dpdk/doc/guides/nics/ice.rst.
>>>
>>> You can try to add --dev-args=safe-mode-support=1 command-line option
>>> described there.
>>>
>>> Hope it helps,
>>> Andrew.
>>>
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hi Adam,
>>>>
>>>> On 8/31/23 22:38, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> I have one additional question as well: Does the test engine support
>>>> running tests on two ARMv8 test agents?
>>>>
>>>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>>>
>>>>
>>>> Great. Let me know when that's fixed.
>>>>
>>>>
>>>> Done. We also fixed a number of warnings in TE.
>>>> Also we fixed root test package name to be consistent with the
>>>> repository name.
>>>>
>>>> Support for old LTS branches was dropped some time ago, but in the
>>>>> future it is definitely possible to keep it for new LTS branches. I think
>>>>> 22.11 is supported, but I'm not sure about older LTS releases.
>>>>>
>>>>
>>>> Good to know.
>>>>
>>>>
>>>>> 2. You can add command-line option --sanity to run tests marked with
>>>>> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>>> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
>>>>> there is a space for terminology improvement here. We'll do it.
>>>>>
>>>>
>>>> Done. Now it is called --checkup.
>>>>
>>>>
>>>>> Also it takes a lot of time because of failures and tests which wait
>>>>> for some timeout.
>>>>>
>>>>
>>>> That makes sense to me. We'll use the time to complete tests on virtio
>>>> or the Intel devices as a reference for how long the tests really take to
>>>> complete.
>>>> We will explore the possibility of periodically running the sanity
>>>> tests for patches.
>>>>
>>>>
>>>> I'll double-check and let you know how long entire TS runs on Intel
>>>> X710, E810, Mellanox CX5 and virtio net. Just to ensure that time observed
>>>> in your case looks the same.
>>>>
>>>>
>>>>
>>>>> The test harness can provide coverage reports based on gcov, but I'm
>>>>> not sure what you mean by a "dial" to control test coverage. Provided
>>>>> reports are rather for human to analyze.
>>>>>
>>>>
>>>> The general idea is to have some kind of parameter on the test suite,
>>>> which could be an integer ranging from zero to ten, that controls how many
>>>> tests are run based on how important the test is.
>>>>
>>>> Similar to how some command line interfaces provide a verbosity level
>>>> parameter (some number of "-v" arguments) to control the importance of the
>>>> information in the log.
>>>> The verbosity level zero only prints very important log messages, while
>>>> ten prints everything.
>>>>
>>>> In much the same manner as above, this "dial" parameter controls what
>>>> tests are run and with what parameters based on how important those tests
>>>> and test parameter combinations are.
>>>> Coverage Level zero tells the suite to run a very basic set of
>>>> important tests, with minimal parameterization. This mode would take only
>>>> ~5-10 minutes to run.
>>>> In contrast, Coverage Level ten includes all the edge cases, every
>>>> combination of test parameters, everything the test suite can do, which
>>>> takes the normal several hours to run.
>>>> The values 1 - 9 are between those two extremes, allowing the user to
>>>> get a gradient of test coverage in the results and to limit the running
>>>> time.
>>>>
>>>> Then we could, for example, run the "run.sh" with a level of 2 or 3 for
>>>> incoming patches that need quick results, and with a level of 10 for the
>>>> less often run periodic tests performed on main or LTS branches.
>>>>
>>>>
>>>> Understood now. Thanks a lot for the idea. We'll discuss it and come
>>>> back.
>>>>
>>>>
>>>>
>>>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>>>>> testing results. Unfortunately it is time consuming to fill in expectations
>>>>> database since it is necessary to analyze testing results and classify if
>>>>> it is a bug or just acceptable behaviour aspect.
>>>>>
>>>>> Bublik allows to compare results of two runs. It is useful for human,
>>>>> but still not good for automation.
>>>>>
>>>>> I have local patch for mlx5 driver which reports Tx ring size maximum.
>>>>> It makes pass rate higher. It is a problem for test harness that mlx5 does
>>>>> not report limits right now.
>>>>>
>>>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>>>>> virtio net is 99% right now and could be done 100% easily (just one thing
>>>>> to fix in expectations).
>>>>>
>>>>> I think logs storage setup is essential for logs analysis. Of course,
>>>>> you can request HTML logs when you run tests (--log-html=html) or generate
>>>>> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>>>>> browser, but logs storage makes it more convenient.
>>>>>
>>>>
>>>> We are interested in setting up Bublik, potentially as an
>>>> externally-facing component, once we have our process of running the test
>>>> suite stabilized.
>>>> Once we are able to run the test suite again, I'll see what the pass
>>>> rate is on our other hardware.
>>>> Good to know that it isn't an issue with our dev testbed causing the
>>>> high fail rate.
>>>>
>>>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>>>> development testbed. Although they are slightly different devices, ideally
>>>> the pass rate will be identical or similar. I have yet to set up a VM pair
>>>> for virtio, but we will soon.
>>>>
>>>> Latest version of test-environment has examples of our CGI scripts
>>>>> which we use for log storage (see tools/log_server/README.md).
>>>>>
>>>>> Also all bits for Jenkins setup are available. See
>>>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>>>> ts-rigs-sample.
>>>>>
>>>>
>>>> Jenkins integration, setting up production rig configurations, and
>>>> permanent log storage will be our next steps once I am able to run the
>>>> tests again.
>>>> Unless there is an easy way to have meson not pass "-Werror" into GCC.
>>>> Then I would be able to run the test suite.
>>>>
>>>>
>>>> Hopefully it is resolved now.
>>>>
>>>> I thought a bit more about your usecase for Jenkins. I'm not 100% sure
>>>> that existing pipelines are convenient for your usecase.
>>>> Fill free to ask questions when you are on it.
>>>>
>>>> Thanks,
>>>> Andrew.
>>>>
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>>
>>>>
>>>>>
>>>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> That fix seems to have resolved the issue, thanks for the quick
>>>>> turnaround time on that patch.
>>>>> Now that we have the RCF timeout issue resolved, there are a few other
>>>>> questions and issues that we have about the tests themselves.
>>>>>
>>>>> 1. The test suite fails to build with a couple warnings.
>>>>>
>>>>> Below is the stderr log from compilation:
>>>>>
>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>> ‘test_create_traffic_generator_params’:
>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>> and no format arguments [-Werror=format-security]
>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>> |     ^~
>>>>>> cc1: all warnings being treated as errors
>>>>>> ninja: build stopped: subcommand failed.
>>>>>> ninja: Entering directory `.'
>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>> ‘test_create_traffic_generator_params’:
>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>> and no format arguments [-Werror=format-security]
>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>> |     ^~
>>>>>> cc1: all warnings being treated as errors
>>>>>>
>>>>>
>>>>> This error wasn't occurring last week, which was the last time I ran
>>>>> the tests.
>>>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester has
>>>>> GCC v11.4.0 installed, if this information is helpful.
>>>>>
>>>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>>>> to lower the test coverage and have the test suite run faster?
>>>>>
>>>>> For some context, we run immediate testing on incoming patches for
>>>>> DPDK main and development branches, as well as periodic test runs on the
>>>>> main, stable, and LTS branches.
>>>>> For us to consider including this test suite as part of our immediate
>>>>> testing on patches, we would have to reduce the test coverage to the most
>>>>> important tests.
>>>>> This is primarily to reduce the testing time to, for example, less
>>>>> than 30 minutes. Testing on patches can't take too long because the lab can
>>>>> receive numerous patches each day, which each require individual testing
>>>>> runs.
>>>>>
>>>>> At what frequency we run these tests, and on what, still needs to be
>>>>> discussed with the DPDK community, but it would be nice to know if the test
>>>>> suite had a "dial" to control the testing coverage.
>>>>>
>>>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>>>> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose these
>>>>> test failures?
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>>
>>>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>> Hi Adam,
>>>>>>
>>>>>> I've pushed the fix in main branch and a new tag v1.18.1. It should
>>>>>> solve the problem with IPv6 address from DNS.
>>>>>>
>>>>>> Andrew.
>>>>>>
>>>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>
>>>>>> Hi Adam,
>>>>>>
>>>>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>> force it to use IPv4?
>>>>>>
>>>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your
>>>>>> case, I guess it is the root cause of the problem.
>>>>>> Of course, it is TE problem since I see really weird code in
>>>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>>
>>>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>>>> ts-factory/test-environment repository. Please, try.
>>>>>>
>>>>>> It is late night fix with minimal testing and no review. I'll pass it
>>>>>> through review process tomorrow and
>>>>>> hopefully it will be released in one-two days.
>>>>>>
>>>>>> Andrew.
>>>>>>
>>>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> We have yet to notice a distinct pattern with the failures.
>>>>>> Sometimes, the RCF will start and connect without issue a few times in a
>>>>>> row before failing to connect again. Once the issue begins to occur,
>>>>>> neither rebooting all of the hosts (test engine VM, tester, IUT) or
>>>>>> deleting all of the build directories (suites, agents, inst) and rebooting
>>>>>> the hosts afterward resolves the issue. When it begins working again seems
>>>>>> very arbitrary to us.
>>>>>>
>>>>>> I do usually try to terminate the test engine with Ctrl+C, but when
>>>>>> it hangs while trying to start RCF, that does not work.
>>>>>>
>>>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>>>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>>>>> to use IPv4?
>>>>>>
>>>>>>  - Adam
>>>>>>
>>>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>
>>>>>>> Done. It works fine for me without any issues.
>>>>>>>
>>>>>>> Have you noticed any pattern when it works or does not work?
>>>>>>> May be it is a problem of not clean state after termination?
>>>>>>> Does it work fine the first time after DUTs reboot?
>>>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>>>> terminal where you execute run.sh command.
>>>>>>>  In this case it should shutdown gracefully and close all test
>>>>>>> agents and engine applications.
>>>>>>>
>>>>>>> (I'm trying to understand why you've seen many test agent processes.
>>>>>>> It should not happen.)
>>>>>>>
>>>>>>> Andrew.
>>>>>>>
>>>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>
>>>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>>>> This could be the source of the problem. I encountered a dependency
>>>>>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>>>>>> system. Since the tester is also the host having connection issues, I will
>>>>>>> try downgrading that system to 20.04, and see if that changes anything.
>>>>>>>
>>>>>>>
>>>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>>>>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>>>>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>>>>>
>>>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>
>>>>>>> I did try passing in the "--vg-rcf" argument to the run.sh script of
>>>>>>> the test suite after installing valgrind, but there was no additional
>>>>>>> output that I saw.
>>>>>>>
>>>>>>>
>>>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>>>> (direction where you run test engine).
>>>>>>>
>>>>>>>
>>>>>>> I will try pulling in the changes you've pushed up, and will see if
>>>>>>> that fixes anything.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Adam
>>>>>>>
>>>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>> Hello Adam,
>>>>>>>>
>>>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>>
>>>>>>>> I'd like to try to repeat the problem locally. Which Linux distro
>>>>>>>> is running on test engine and agents?
>>>>>>>>
>>>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>>>>>> patch in review to fix it, however, the behaviour is different in
>>>>>>>> this case, so it is unlike the same problem.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've just published a new tag which fixes known test engine side
>>>>>>>> problems on Debian 12 and Fedora 38.
>>>>>>>>
>>>>>>>>
>>>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>>>> run with option --vg-rcf to check if something weird is happening.
>>>>>>>>
>>>>>>>> What I don't understand right now is why I see just one failed
>>>>>>>> attempt
>>>>>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>>>>>
>>>>>>>> Andrew.
>>>>>>>>
>>>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>
>>>>>>>>  > Is there any firewall in the network or on test hosts which
>>>>>>>> could block incoming TCP connection to the port 23571
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>> where you run test engine?
>>>>>>>>
>>>>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>>>>> connection does work sometimes.
>>>>>>>>
>>>>>>>>  > If behaviour the same on the next try and you see that test
>>>>>>>> agent is kept running, could you check using
>>>>>>>>  >
>>>>>>>>  > # netstat -tnlp
>>>>>>>>  >
>>>>>>>>  > that Test Agent is listening on the port and try to establish
>>>>>>>> TCP connection from test agent using
>>>>>>>>  >
>>>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>  >
>>>>>>>>  > and check if TCP connection could be established.
>>>>>>>>
>>>>>>>> I was able to replicate the same behavior again, where it hangs
>>>>>>>> while RCF is trying to start.
>>>>>>>> Running this command, I see this in the output:
>>>>>>>>
>>>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>>>>    18599/ta
>>>>>>>>
>>>>>>>> So it seems like it is listening on the correct port.
>>>>>>>> Additionally, I was able to connect to the Tester machine from our
>>>>>>>> Test Engine host using telnet. It printed the PID of the process once the
>>>>>>>> connection was opened.
>>>>>>>>
>>>>>>>> I tried running the "ta" application manually on the command line,
>>>>>>>> and it didn't print anything at all.
>>>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>>>
>>>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>>>> andrew.rybchenko@oktetlabs.ru
>>>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>     Hi Adam,
>>>>>>>>
>>>>>>>>      > On the tester host (which appears to be the Peer agent),
>>>>>>>> there
>>>>>>>>     are four processes that I see running, which look like the test
>>>>>>>>     agent processes.
>>>>>>>>
>>>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>>>
>>>>>>>>     Is there any firewall in the network or on test hosts which
>>>>>>>> could
>>>>>>>>     block incoming TCP connection to the port 23571
>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>     where you run test engine?
>>>>>>>>
>>>>>>>>     If behaviour the same on the next try and you see that test
>>>>>>>> agent is
>>>>>>>>     kept running, could you check using
>>>>>>>>
>>>>>>>>     # netstat -tnlp
>>>>>>>>
>>>>>>>>     that Test Agent is listening on the port and try to establish
>>>>>>>> TCP
>>>>>>>>     connection from test agent using
>>>>>>>>
>>>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>
>>>>>>>>     and check if TCP connection could be established.
>>>>>>>>
>>>>>>>>     Another idea is to login Tester under root as testing does, get
>>>>>>>>     start TA command from the log and try it by hands without -n and
>>>>>>>>     remove extra escaping.
>>>>>>>>
>>>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>
>>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>
>>>>>>>>     Hopefully in this case test agent directory remains in the /tmp
>>>>>>>> and
>>>>>>>>     you don't need to copy it as testing does.
>>>>>>>>     May be output could shed some light on what's going on.
>>>>>>>>
>>>>>>>>     Andrew.
>>>>>>>>
>>>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>
>>>>>>>>     Hi Andrew,
>>>>>>>>
>>>>>>>>     This is the output that I see in the terminal when this failure
>>>>>>>>     occurs, after the test agent binaries build and the test engine
>>>>>>>>     starts:
>>>>>>>>
>>>>>>>>     Platform default build - pass
>>>>>>>>     Simple RCF consistency check succeeded
>>>>>>>>     --->>> Starting Logger...done
>>>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>
>>>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>>>
>>>>>>>>     On the tester host (which appears to be the Peer agent), there
>>>>>>>> are
>>>>>>>>     four processes that I see running, which look like the test
>>>>>>>> agent
>>>>>>>>     processes.
>>>>>>>>
>>>>>>>>     ta.Peer is an empty file. I've attached the log.txt from this
>>>>>>>> run.
>>>>>>>>
>>>>>>>>      - Adam
>>>>>>>>
>>>>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>         Hi Adam,
>>>>>>>>
>>>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>         says that copy_timeout is in seconds and implementation in
>>>>>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>>>>>         Theoretically select() could be interrupted by signal, but I
>>>>>>>>         think it is unlikely here.
>>>>>>>>
>>>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>>>         does it happen when tests are in progress, e.g. in the
>>>>>>>> middle
>>>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>>>>         likely either host with test agent dies or test agent itself
>>>>>>>>         crashes. It would be easier for me if classify it if you
>>>>>>>> share
>>>>>>>>         text log (log.txt, full or just corresponding fragment with
>>>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>>>         depending on which agent has problems could shed some light.
>>>>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>>>>
>>>>>>>>         Andrew.
>>>>>>>>
>>>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>
>>>>>>>>         Hi Andrew,
>>>>>>>>
>>>>>>>>         I've set up a test rig repository here, and have created
>>>>>>>>         configurations for our development testbed based off of the
>>>>>>>>         examples.
>>>>>>>>         We've been able to get the test suite to run manually on
>>>>>>>>         Mellanox CX5 devices once.
>>>>>>>>         However, we are running into an issue where, when RCF
>>>>>>>> starts,
>>>>>>>>         the RCF connection times out very frequently. We aren't sure
>>>>>>>>         why this is the case.
>>>>>>>>         It works sometimes, but most of the time when we try to run
>>>>>>>>         the test engine, it encounters this issue.
>>>>>>>>         I've tried changing the RCF port by setting
>>>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>>>>>         machines. Neither seems to fix the issue.
>>>>>>>>
>>>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>         before I try to run the test suite.
>>>>>>>>         I assume the unit for this variable is seconds?
>>>>>>>>
>>>>>>>>         Thanks,
>>>>>>>>         Adam
>>>>>>>>
>>>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>>>
>>>>>>>>             Hi Andrew,
>>>>>>>>
>>>>>>>>             Thanks, I've cloned the example repository and will
>>>>>>>> start
>>>>>>>>             setting up a configuration for our development testbed
>>>>>>>>             today. I'll let you know if I run into any difficulties
>>>>>>>>             or have any questions.
>>>>>>>>
>>>>>>>>              - Adam
>>>>>>>>
>>>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                 Hi Adam,
>>>>>>>>
>>>>>>>>                 I've published
>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>                 Hopefully it will help to define your test rigs and
>>>>>>>>                 successfully run some tests manually. Feel free to
>>>>>>>>                 ask any questions and I'll answer here and try to
>>>>>>>>                 update documentation.
>>>>>>>>
>>>>>>>>                 Meanwhile I'll prepare missing bits for steps (2)
>>>>>>>> and
>>>>>>>>                 (3).
>>>>>>>>                 Hopefully everything is in place for step (4), but
>>>>>>>> we
>>>>>>>>                 need to make steps (2) and (3) first.
>>>>>>>>
>>>>>>>>                 Andrew.
>>>>>>>>
>>>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>
>>>>>>>>                 Hi Adam,
>>>>>>>>
>>>>>>>>                 > I've conferred with the rest of the team, and we
>>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>>                 option B.
>>>>>>>>
>>>>>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>>>>>                 almost ready right now, but I need to double-check
>>>>>>>>                 it before publishing.
>>>>>>>>
>>>>>>>>                 Regards,
>>>>>>>>                 Andrew.
>>>>>>>>
>>>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>
>>>>>>>>                 Hi Andrew,
>>>>>>>>
>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>                 conversation. Others in the community might find
>>>>>>>>                 this conversation valuable.
>>>>>>>>
>>>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>>>                 Jenkins integration will be very useful for us, as
>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>>                 option B.
>>>>>>>>                 If you would like to know anything about our
>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>>>                 questions you have.
>>>>>>>>
>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>                 hardware and CPU architecture. I figured this might
>>>>>>>>                 be an important detail.
>>>>>>>>
>>>>>>>>                 Thanks,
>>>>>>>>                 Adam
>>>>>>>>
>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                     Greatings Adam,
>>>>>>>>
>>>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>>>                     it up.
>>>>>>>>
>>>>>>>>                     As I understand the final goal is to run it on
>>>>>>>>                     regular basis. So, we need to make it properly
>>>>>>>>                     from the very beginning.
>>>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>>>
>>>>>>>>                     1. Create site-specific repository (we call it
>>>>>>>>                     ts-rigs) which contains information about test
>>>>>>>>                     rigs and other site-specific information like
>>>>>>>>                     where to send mails, where to store logs etc.
>>>>>>>>                     It is required for manual execution as well,
>>>>>>>>                     since test rigs description is essential. I'll
>>>>>>>>                     return to the topic below.
>>>>>>>>
>>>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>>>                     server with few CGI scripts which help a lot to
>>>>>>>>                     save disk space.
>>>>>>>>
>>>>>>>>                     3. Setup Bublik web application which provides
>>>>>>>>                     web interface to view testing results. Same as
>>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>> <https://ts-factory.io/bublik>
>>>>>>>>
>>>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>>>                     save logs in log storage (2) and import it to
>>>>>>>>                     bublik (3).
>>>>>>>>
>>>>>>>>                     Last few month we spent on our homework to make
>>>>>>>>                     it simpler to bring up automated execution
>>>>>>>>                     using Jenkins -
>>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>                     available tomorrow.
>>>>>>>>
>>>>>>>>                     Let's return to the step (1).
>>>>>>>>
>>>>>>>>                     Unfortunately there is no publicly available
>>>>>>>>                     example of the ts-rigs repository since
>>>>>>>>                     sensitive site-specific information is located
>>>>>>>>                     there. But I'm ready to help you to create it
>>>>>>>>                     for UNH. I see two options here:
>>>>>>>>
>>>>>>>>                     (A) I'll ask questions and based on your
>>>>>>>>                     answers will create the first draft with my
>>>>>>>>                     comments.
>>>>>>>>
>>>>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>>>>                     on it.
>>>>>>>>
>>>>>>>>                     Of course, I'll help to debug and finally bring
>>>>>>>>                     it up in any case.
>>>>>>>>
>>>>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>>>>                     a bit more generic and will help other
>>>>>>>>                     potential users to bring it up.
>>>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>>>                     What do you think?
>>>>>>>>
>>>>>>>>                     Thanks,
>>>>>>>>                     Andrew.
>>>>>>>>
>>>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>
>>>>>>>>                     Greetings Adam,
>>>>>>>>
>>>>>>>>
>>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>>                     would be happy to help
>>>>>>>>
>>>>>>>>                     Thanks,
>>>>>>>>                     Konstantin
>>>>>>>>
>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>>>
>>>>>>>>                     
>>>>>>>>                     Greetings Konstantin,
>>>>>>>>
>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>>
>>>>>>>>                     I have some questions about how to set the
>>>>>>>>                     test suite arguments.
>>>>>>>>
>>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>                     fails to run.
>>>>>>>>
>>>>>>>>
>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>
>>>>>>>>                     The documentation mentions that there are
>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>>                     to run many of the tests.
>>>>>>>>
>>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>>                     are instructions on how to create new
>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>                     some other way.
>>>>>>>>
>>>>>>>>                     How should these values be set?
>>>>>>>>
>>>>>>>>                     I'm also not sure what environment
>>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>>                     which are optional.
>>>>>>>>
>>>>>>>>                     Regards,
>>>>>>>>                     Adam
>>>>>>>>
>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 --                 *Adam Hassick*
>>>>>>>>                 Senior Developer
>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             --             *Adam Hassick*
>>>>>>>>             Senior Developer
>>>>>>>>             UNH InterOperability Lab
>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>             +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>         --         *Adam Hassick*
>>>>>>>>         Senior Developer
>>>>>>>>         UNH InterOperability Lab
>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>         +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>     --     *Adam Hassick*
>>>>>>>>     Senior Developer
>>>>>>>>     UNH InterOperability Lab
>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Adam Hassick*
>>>>>>>> Senior Developer
>>>>>>>> UNH InterOperability Lab
>>>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>> +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Adam Hassick*
>>>>>>> Senior Developer
>>>>>>> UNH InterOperability Lab
>>>>>>> ahassick@iol.unh.edu
>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Adam Hassick*
>>>>>> Senior Developer
>>>>>> UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu
>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 172081 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 113160 bytes --]

On 9/18/23 17:44, Adam Hassick wrote:
> Hi Andrew and Konstantin,
>
> Thank you for adding the tester-dial feature, this opens up the 
> possibility for us to do CI integrated testing in the future.
>
> Our Mellanox pass rate is similar to yours (about ~2400 passing, ~4400 
> failing), however our Intel pass rates are far worse.
> I will try running tests on the XL710 with the trc-tags argument set 
> and see if it improves the pass rate.
> Another thing I noticed in the results you uploaded is that the 
> results are tagged with vfio-pci and not i40e.
> Though in the environment dump, the driver on the test machine and the 
> DUT are set to use the i40e driver. Is this important at all?

I think it is a misunderstanding here. There are two kinds of driver in 
configuration: net driver and so-called DPDK driver.
Net driver is a Linux kernel network device driver used on Tester side.
DPDK driver is a Linux kernel driver to bind device to to use it with 
DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
In the case of bifurcated driver (like mlx5_core) it is the same in both 
cases.
In non-bifurcated case DPDK driver is some UIO driver(vfio-pci, 
uio-pci-generic or igb_uio).
Some expectations depend on used UIO. For example, uio-pci-generic do 
not support many interrupts (used by usecases/rx_intr test cases).
That's why we care corresponding TRC tag.

TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's Intel case. 
Or uio-pci-generic if IOMMU is turned off on corresponding machines and 
Linux distro does not support VFIO no IOMMU mode.

Andrew.

> There isn't anything preventing us from pushing our results up to the 
> existing Bublik instance running at ts-factory.io 
> <http://ts-factory.io> that I can think of at the moment.
> We will have to work out how to submit our results to your Bublik 
> instance in a controlled and secure manner in that case.
> As far as I know we won't need access controls for the results 
> themselves. I'll discuss this with Patrick and will let you know once 
> we confirm that it's fine.
>
> Thanks,
> Adam
>
> On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     On 9/18/23 09:23, Konstantin Ushakov wrote:
>>
>>     Hi Andrew,
>>
>>     should we always auto-assign the tags or you don’t do it since it
>>     slows down (by some seconds) the TE startup?
>>
>
>     Tags are auto-assigned, but I guess it differs in Adam's case
>     since NIC is a bit different. Below test will help to understand
>     if it is the root cause of very different expectations. If pass
>     rate will be close to mine, I'll simply update TRC database to
>     share expectations for mine NIC and NIC used by Adam.
>
>>     Hi Adam,
>>
>>     I think I second the question from Andrew - happy to help you
>>     with the triage so that we get to the same baseline. Do you have
>>     a good way for us to share the logs? I.e. say upload to
>>     ts-factory if we add strict permissions system so it’s not
>>     publishing or any other way.
>>
>>     Thanks,
>>     Konstantin
>>
>>
>>     On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>
>>         Hi Adam,
>>
>>         I've uploaded fresh testing results to ts-factory.io
>>         <http://ts-factory.io> [1] to be on the same page.
>>
>>         I think I know why your and mine results on Intel 710 series
>>         NICs differ so much. Testing results expectations database
>>         (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags. 
>>         I.e. expectations depends on TRC tags discovered by helper
>>         scripts when testing is started. These tags identify various
>>         aspects of what is tested. Ideally expectations should be
>>         written in terms of root cause of the expected behaviour. If
>>         it is a driver expectations, driver tag should be used. If it
>>         is HW limitation, tags with PCI IDs should be used. However,
>>         it is not always easy to classify it correctly if you're not
>>         involved in driver development. So, in order case
>>         expectations for 710's Intel are filled in in terms of PCI
>>         IDs. I guess PCI ID differ in your case and that's why
>>         expectations filled in for my NIC do not apply to your runs.
>>
>>         Just try to add the following option when you run on your
>>         710's Intel in order to mimic mine and see if it helps to
>>         achieve better pass rate.
>>         --trc-tag=pci-8086-1572
>>
>>         BTW, fresh TE tag v1.21.0 has improved algorithm to choose
>>         tests for --tester-dial option. It should have better
>>         coverage now.
>>
>>         Andrew.
>>
>>         [1]
>>         https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>         <https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1>
>>
>>         On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>         Hi Adam,
>>>
>>>         I've pushed new TE tag v1.20.0 which supported a new
>>>         command-line option --tester-dial=NUM where NUM is from 0 to
>>>         100. it allows to choose percentage of tests to run. If you
>>>         want stable set, you should pass --tester-random-seed=0 (or
>>>         other integer). It is the first sketch and we have plans to
>>>         improve it, but feedback would be welcome.
>>>
>>>         > Is it needed on the tester?
>>>
>>>         It is hard to say if it is strictly required for simple
>>>         tests. However, it is better to update Tester as well, since
>>>         performance tests run DPDK on Tester as well.
>>>
>>>         > Are there any other manual setup steps for these devices
>>>         that I might be missing?
>>>
>>>         I don't remember anything else.
>>>
>>>         I think it is better to get down to details and take a look
>>>         at logs. I'm ready to help with it and explain what's
>>>         happening there. May be it will help to understand if it is
>>>         a problem with setup/configuration.
>>>
>>>         Text logs are not very convenient. Ideally logs should be
>>>         imported to bublik, however, manual runs do not provide all
>>>         required artifacts right now (Jenkins jobs generate all
>>>         required artifacts).
>>>         Other option is 'tmp_raw_log' file (should be packed to make
>>>         it smaller) which could be converted to various log formats.
>>>         Would it be OK for you if I import your logs to bublik at
>>>         ts-factory.io <http://ts-factory.io>? Or is it a problem
>>>         that it is publicly available?
>>>         Would it help if we add authentication and access control there?
>>>
>>>         Andrew.
>>>
>>>         On 9/8/23 17:57, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         I have a couple questions about needed setup of the NICs
>>>>         for the ethdev test suite.
>>>>
>>>>         Our MCX5s and XL710s are failing the checkup tests. The
>>>>         pass rate appears to be much worse on the XL710s (40 of 73
>>>>         tests failed, 3 passed unexpectedly).
>>>>
>>>>         For the XL710s, I've updated the driver and NVM versions to
>>>>         match the minimum supported versions in the compatibility
>>>>         matrix found on the DPDK documentation. This did not change
>>>>         the failure rate much.
>>>>         For the MCX5s, I've installed the latest LTS version of the
>>>>         OFED bifurcated driver on the DUT. Is it needed on the tester?
>>>>
>>>>         Are there any other manual setup steps for these devices
>>>>         that I might be missing?
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick
>>>>         <ahassick@iol.unh.edu> wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             Yes, I copied the X710 configs to set up XL710 configs.
>>>>             I changed the environment variable names from the X710
>>>>             suffix to XL710 suffix in the script, and forgot to
>>>>             change them in the corresponding environment file.
>>>>             That fixed the issue.
>>>>
>>>>             I got the checkup tests working on the XL710 now. Most
>>>>             of them are failing, which leads me to believe this is
>>>>             an issue with our testbed. Based on the DPDK
>>>>             documentation for i40e, the firmware and driver
>>>>             versions are much older than what DPDK 22.11 LTS and
>>>>             main prefer, so I'll try updating those.
>>>>
>>>>             For now I'm working on getting the XL710 checkup tests
>>>>             passing, and will pick up getting the E810 configured
>>>>             properly next. I'll let you know if I run into any more
>>>>             issues in relation to the test engine.
>>>>
>>>>             Thanks,
>>>>             Adam
>>>>
>>>>             On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 On 9/5/23 18:01, Adam Hassick wrote:
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 The compilation warning issue is now resolved.
>>>>>                 Again, thank you guys for fixing this for us. I
>>>>>                 can run the tests on the Mellanox CX5s again,
>>>>>                 however I'm running into a couple new issues with
>>>>>                 running the prologues on the Intel cards.
>>>>>
>>>>>                 When running testing on the Intel XL710s, I see
>>>>>                 this error appear in the log:
>>>>>
>>>>>                     ERROR  prologue  Environment LIB  14:16:13.650
>>>>>                     Too few networks in available configuration
>>>>>                     (0) in comparison with required (1)
>>>>>
>>>>>
>>>>>                 This seems like a trivial configuration error,
>>>>>                 perhaps this is something I need to set up in
>>>>>                 ts-rigs. I briefly searched through the examples
>>>>>                 there and didn't see any mention of how to set up
>>>>>                 a network.
>>>>>                 I will attach this log just in case you need more
>>>>>                 information.
>>>>
>>>>                 Unfortunately logs are insufficient to understand
>>>>                 it. I've pushed new tag to TE v1.19.0 which add log
>>>>                 message with TE_* environment variables.
>>>>                 Most likely something is wrong with variables which
>>>>                 are used as conditions when available networks are
>>>>                 defined in ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>                 TE_PCI_INSTANCE_IUT_TST1
>>>>                 TE_PCI_INSTANCE_IUT_TST1a
>>>>                 TE_PCI_INSTANCE_TST1a_IUT
>>>>                 TE_PCI_INSTANCE_TST1_IUT
>>>>                 My guess it that you change naming a bit, but
>>>>                 script like ts-rigs-sample/scripts/iut.h1-x710 is
>>>>                 not included or not updated.
>>>>
>>>>>                 There is a different error when running on the
>>>>>                 Intel E810s. It appears to me like it starts DPDK,
>>>>>                 does some configuration inside DPDK and on the
>>>>>                 device, and then fails to bring the device back
>>>>>                 up. Since this error seems very non-trivial, I
>>>>>                 will also attach this log.
>>>>
>>>>                 This one is a bit simpler. Few lines after the
>>>>                 first ERROR in log I see the following:
>>>>                 WARN  RCF  DPDK  13:06:00.144
>>>>                 ice_program_hw_rx_queue(): currently package
>>>>                 doesn't support RXDID (22)
>>>>                 ice_rx_queue_start(): fail to program RX queue 0
>>>>                 ice_dev_start(): fail to start Rx queue 0
>>>>                 Device with port_id=0 already stopped
>>>>
>>>>                 It is stdout/stderr from test agent which runs
>>>>                 DPDK. Same logs in plain format are available in
>>>>                 ta.DPDK file.
>>>>                 I'm not an expert here, but I vaguely remember that
>>>>                 E810 requires correct firmware and DDP to be loaded.
>>>>                 There is some information in
>>>>                 dpdk/doc/guides/nics/ice.rst.
>>>>
>>>>                 You can try to add --dev-args=safe-mode-support=1
>>>>                 command-line option described there.
>>>>
>>>>                 Hope it helps,
>>>>                 Andrew.
>>>>
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>                 On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>                     Hi Adam,
>>>>>
>>>>>                     On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>                     Hi Andrew,
>>>>>>
>>>>>>                     I have one additional question as well: Does
>>>>>>                     the test engine support running tests on two
>>>>>>                     ARMv8 test agents?
>>>>>>
>>>>>>                         1. We'll sort out warnings this week.
>>>>>>                         Thanks for heads up.
>>>>>>
>>>>>>
>>>>>>                     Great. Let me know when that's fixed.
>>>>>
>>>>>                     Done. We also fixed a number of warnings in TE.
>>>>>                     Also we fixed root test package name to be
>>>>>                     consistent with the repository name.
>>>>>
>>>>>>                         Support for old LTS branches was dropped
>>>>>>                         some time ago, but in the future it is
>>>>>>                         definitely possible to keep it for new
>>>>>>                         LTS branches. I think 22.11 is supported,
>>>>>>                         but I'm not sure about older LTS releases.
>>>>>>
>>>>>>
>>>>>>                     Good to know.
>>>>>>
>>>>>>                         2. You can add command-line option
>>>>>>                         --sanity to run tests marked with
>>>>>>                         TEST_HARNESS_SANITY requirement (see
>>>>>>                         dpdk-ethdev-ts/scripts/run.sh and grep
>>>>>>                         TEST_HARNESS_SANITY dpdk-ethdev-ts to see
>>>>>>                         which tests are marked). Yes, there is a
>>>>>>                         space for terminology improvement here.
>>>>>>                         We'll do it.
>>>>>>
>>>>>
>>>>>                     Done. Now it is called --checkup.
>>>>>
>>>>>>
>>>>>>                         Also it takes a lot of time because of
>>>>>>                         failures and tests which wait for some
>>>>>>                         timeout.
>>>>>>
>>>>>>
>>>>>>                     That makes sense to me. We'll use the time to
>>>>>>                     complete tests on virtio or the Intel devices
>>>>>>                     as a reference for how long the tests really
>>>>>>                     take to complete.
>>>>>>                     We will explore the possibility of
>>>>>>                     periodically running the sanity tests for
>>>>>>                     patches.
>>>>>
>>>>>                     I'll double-check and let you know how long
>>>>>                     entire TS runs on Intel X710, E810, Mellanox
>>>>>                     CX5 and virtio net. Just to ensure that time
>>>>>                     observed in your case looks the same.
>>>>>
>>>>>>
>>>>>>                         The test harness can provide coverage
>>>>>>                         reports based on gcov, but I'm not sure
>>>>>>                         what you mean by a "dial" to control test
>>>>>>                         coverage. Provided reports are rather for
>>>>>>                         human to analyze.
>>>>>>
>>>>>>
>>>>>>                     The general idea is to have some kind of
>>>>>>                     parameter on the test suite, which could be
>>>>>>                     an integer ranging from zero to ten, that
>>>>>>                     controls how many tests are run based on how
>>>>>>                     important the test is.
>>>>>>
>>>>>>                     Similar to how some command line interfaces
>>>>>>                     provide a verbosity level parameter (some
>>>>>>                     number of "-v" arguments) to control the
>>>>>>                     importance of the information in the log.
>>>>>>                     The verbosity level zero only prints very
>>>>>>                     important log messages, while ten prints
>>>>>>                     everything.
>>>>>>
>>>>>>                     In much the same manner as above, this "dial"
>>>>>>                     parameter controls what tests are run and
>>>>>>                     with what parameters based on how important
>>>>>>                     those tests and test parameter combinations are.
>>>>>>                     Coverage Level zero tells the suite to run a
>>>>>>                     very basic set of important tests, with
>>>>>>                     minimal parameterization. This mode would
>>>>>>                     take only ~5-10 minutes to run.
>>>>>>                     In contrast, Coverage Level ten includes all
>>>>>>                     the edge cases, every combination of test
>>>>>>                     parameters, everything the test suite can do,
>>>>>>                     which takes the normal several hours to run.
>>>>>>                     The values 1 - 9 are between those two
>>>>>>                     extremes, allowing the user to get a gradient
>>>>>>                     of test coverage in the results and to limit
>>>>>>                     the running time.
>>>>>>
>>>>>>                     Then we could, for example, run the "run.sh"
>>>>>>                     with a level of 2 or 3 for incoming patches
>>>>>>                     that need quick results, and with a level of
>>>>>>                     10 for the less often run periodic tests
>>>>>>                     performed on main or LTS branches.
>>>>>
>>>>>                     Understood now. Thanks a lot for the idea.
>>>>>                     We'll discuss it and come back.
>>>>>
>>>>>>                         3. Yes, really many tests on Mellanox CX5
>>>>>>                         NICs report unexpected testing results.
>>>>>>                         Unfortunately it is time consuming to
>>>>>>                         fill in expectations database since it is
>>>>>>                         necessary to analyze testing results and
>>>>>>                         classify if it is a bug or just
>>>>>>                         acceptable behaviour aspect.
>>>>>>
>>>>>>                         Bublik allows to compare results of two
>>>>>>                         runs. It is useful for human, but still
>>>>>>                         not good for automation.
>>>>>>
>>>>>>                         I have local patch for mlx5 driver which
>>>>>>                         reports Tx ring size maximum. It makes
>>>>>>                         pass rate higher. It is a problem for
>>>>>>                         test harness that mlx5 does not report
>>>>>>                         limits right now.
>>>>>>
>>>>>>                         Pass rate on Intel X710 is about 92% on
>>>>>>                         my test rig. Pass rate on virtio net is
>>>>>>                         99% right now and could be done 100%
>>>>>>                         easily (just one thing to fix in
>>>>>>                         expectations).
>>>>>>
>>>>>>                         I think logs storage setup is essential
>>>>>>                         for logs analysis. Of course, you can
>>>>>>                         request HTML logs when you run tests
>>>>>>                         (--log-html=html) or generate after run
>>>>>>                         using dpdk-ethdev-ts/scripts/html-log.sh
>>>>>>                         and open index.html in a browser, but
>>>>>>                         logs storage makes it more convenient.
>>>>>>
>>>>>>
>>>>>>                     We are interested in setting up Bublik,
>>>>>>                     potentially as an externally-facing
>>>>>>                     component, once we have our process of
>>>>>>                     running the test suite stabilized.
>>>>>>                     Once we are able to run the test suite again,
>>>>>>                     I'll see what the pass rate is on our other
>>>>>>                     hardware.
>>>>>>                     Good to know that it isn't an issue with our
>>>>>>                     dev testbed causing the high fail rate.
>>>>>>
>>>>>>                     For Intel hardware, we have an XL710 and an
>>>>>>                     Intel E810-C in our development testbed.
>>>>>>                     Although they are slightly different devices,
>>>>>>                     ideally the pass rate will be identical or
>>>>>>                     similar. I have yet to set up a VM pair for
>>>>>>                     virtio, but we will soon.
>>>>>>
>>>>>>                         Latest version of test-environment has
>>>>>>                         examples of our CGI scripts which we use
>>>>>>                         for log storage (see
>>>>>>                         tools/log_server/README.md).
>>>>>>
>>>>>>                         Also all bits for Jenkins setup are
>>>>>>                         available. See
>>>>>>                         dpdk-ethdev-ts/jenkins/README.md and
>>>>>>                         examples of jenkins files in ts-rigs-sample.
>>>>>>
>>>>>>
>>>>>>                     Jenkins integration, setting up production
>>>>>>                     rig configurations, and permanent log storage
>>>>>>                     will be our next steps once I am able to run
>>>>>>                     the tests again.
>>>>>>                     Unless there is an easy way to have meson not
>>>>>>                     pass "-Werror" into GCC. Then I would be able
>>>>>>                     to run the test suite.
>>>>>
>>>>>                     Hopefully it is resolved now.
>>>>>
>>>>>                     I thought a bit more about your usecase for
>>>>>                     Jenkins. I'm not 100% sure that existing
>>>>>                     pipelines are convenient for your usecase.
>>>>>                     Fill free to ask questions when you are on it.
>>>>>
>>>>>                     Thanks,
>>>>>                     Andrew.
>>>>>
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Adam
>>>>>>
>>>>>>
>>>>>>                         On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>>                         Hi Andrew,
>>>>>>>
>>>>>>>                         That fix seems to have resolved the
>>>>>>>                         issue, thanks for the quick turnaround
>>>>>>>                         time on that patch.
>>>>>>>                         Now that we have the RCF timeout issue
>>>>>>>                         resolved, there are a few other
>>>>>>>                         questions and issues that we have about
>>>>>>>                         the tests themselves.
>>>>>>>
>>>>>>>                         1. The test suite fails to build with a
>>>>>>>                         couple warnings.
>>>>>>>
>>>>>>>                         Below is the stderr log from compilation:
>>>>>>>
>>>>>>>                             FAILED:
>>>>>>>                             lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>                             cc -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>                             -Ilib -I../../lib
>>>>>>>                             -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>                             -fdiagnostics-color=always -pipe
>>>>>>>                             -D_FILE_OFFSET_BITS=64 -Wall
>>>>>>>                             -Winvalid-pch -Werror -g
>>>>>>>                             -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>>>                             -fPIC -MD -MQ
>>>>>>>                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>                             -MF
>>>>>>>                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>                             -o
>>>>>>>                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>                             -c ../../lib/dpdk_pmd_ts.c
>>>>>>>                             ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>                             ‘test_create_traffic_generator_params’:
>>>>>>>                             ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>                             error: format not a string literal
>>>>>>>                             and no format arguments
>>>>>>>                             [-Werror=format-security]
>>>>>>>                             5577 |     rc =
>>>>>>>                             te_kvpair_add(result, buf, mode);
>>>>>>>                             |     ^~
>>>>>>>                             cc1: all warnings being treated as
>>>>>>>                             errors
>>>>>>>                             ninja: build stopped: subcommand failed.
>>>>>>>                             ninja: Entering directory `.'
>>>>>>>                             FAILED:
>>>>>>>                             lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>                             cc -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>                             -Ilib -I../../lib
>>>>>>>                             -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>                             -fdiagnostics-color=always -pipe
>>>>>>>                             -D_FILE_OFFSET_BITS=64 -Wall
>>>>>>>                             -Winvalid-pch -Werror -g
>>>>>>>                             -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>>>>>>                             -fPIC -MD -MQ
>>>>>>>                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>                             -MF
>>>>>>>                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>                             -o
>>>>>>>                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>                             -c ../../lib/dpdk_pmd_ts.c
>>>>>>>                             ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>                             ‘test_create_traffic_generator_params’:
>>>>>>>                             ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>                             error: format not a string literal
>>>>>>>                             and no format arguments
>>>>>>>                             [-Werror=format-security]
>>>>>>>                             5577 |     rc =
>>>>>>>                             te_kvpair_add(result, buf, mode);
>>>>>>>                             |     ^~
>>>>>>>                             cc1: all warnings being treated as
>>>>>>>                             errors
>>>>>>>
>>>>>>>
>>>>>>>                         This error wasn't occurring last week,
>>>>>>>                         which was the last time I ran the tests.
>>>>>>>                         The TE host and the DUT have GCC v9.4.0
>>>>>>>                         installed, and the tester has GCC
>>>>>>>                         v11.4.0 installed, if this information
>>>>>>>                         is helpful.
>>>>>>>
>>>>>>>                         2. On the Mellanox CX5s, there are over
>>>>>>>                         6,000 tests run, which collectively take
>>>>>>>                         around 9 hours. Is it possible, and
>>>>>>>                         would it make sense, to lower the test
>>>>>>>                         coverage and have the test suite run faster?
>>>>>>>
>>>>>>>                         For some context, we run immediate
>>>>>>>                         testing on incoming patches for DPDK
>>>>>>>                         main and development branches, as well
>>>>>>>                         as periodic test runs on the main,
>>>>>>>                         stable, and LTS branches.
>>>>>>>                         For us to consider including this test
>>>>>>>                         suite as part of our immediate testing
>>>>>>>                         on patches, we would have to reduce the
>>>>>>>                         test coverage to the most important tests.
>>>>>>>                         This is primarily to reduce the testing
>>>>>>>                         time to, for example, less than 30
>>>>>>>                         minutes. Testing on patches can't take
>>>>>>>                         too long because the lab can receive
>>>>>>>                         numerous patches each day, which each
>>>>>>>                         require individual testing runs.
>>>>>>>
>>>>>>>                         At what frequency we run these tests,
>>>>>>>                         and on what, still needs to be discussed
>>>>>>>                         with the DPDK community, but it would be
>>>>>>>                         nice to know if the test suite had a
>>>>>>>                         "dial" to control the testing coverage.
>>>>>>>
>>>>>>>                         3. We see a lot of test failures on our
>>>>>>>                         Mellanox CX5 NICs. Around 2,300 of
>>>>>>>                         ~6,600 tests passed. Is there anything
>>>>>>>                         we can do to diagnose these test failures?
>>>>>>>
>>>>>>>                         Thanks,
>>>>>>>                         Adam
>>>>>>>
>>>>>>>
>>>>>>>                         On Tue, Aug 29, 2023 at 8:07 AM Andrew
>>>>>>>                         Rybchenko
>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>                             Hi Adam,
>>>>>>>
>>>>>>>                             I've pushed the fix in main branch
>>>>>>>                             and a new tag v1.18.1. It should
>>>>>>>                             solve the problem with IPv6 address
>>>>>>>                             from DNS.
>>>>>>>
>>>>>>>                             Andrew.
>>>>>>>
>>>>>>>                             On 8/29/23 00:05, Andrew Rybchenko
>>>>>>>                             wrote:
>>>>>>>>                             Hi Adam,
>>>>>>>>
>>>>>>>>                             > Does the test engine prefer to
>>>>>>>>                             use IPv6 over IPv4 for initiating
>>>>>>>>                             the RCF connection to the test bed
>>>>>>>>                             hosts? And if so, is there a way to
>>>>>>>>                             force it to use IPv4?
>>>>>>>>
>>>>>>>>                             Brilliant idea. If DNS returns both
>>>>>>>>                             IPv4 and IPv6 addresses in your
>>>>>>>>                             case, I guess it is the root cause
>>>>>>>>                             of the problem.
>>>>>>>>                             Of course, it is TE problem since I
>>>>>>>>                             see really weird code in
>>>>>>>>                             lib/comm_net_engine/comm_net_engine.c
>>>>>>>>                             line 135.
>>>>>>>>
>>>>>>>>                             I've pushed fix to the branch
>>>>>>>>                             user/arybchik/fix_ipv4_only in
>>>>>>>>                             ts-factory/test-environment
>>>>>>>>                             repository. Please, try.
>>>>>>>>
>>>>>>>>                             It is late night fix with minimal
>>>>>>>>                             testing and no review. I'll pass it
>>>>>>>>                             through review process tomorrow and
>>>>>>>>                             hopefully it will be released in
>>>>>>>>                             one-two days.
>>>>>>>>
>>>>>>>>                             Andrew.
>>>>>>>>
>>>>>>>>                             On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>>                             Hi Andrew,
>>>>>>>>>
>>>>>>>>>                             We have yet to notice a distinct
>>>>>>>>>                             pattern with the failures.
>>>>>>>>>                             Sometimes, the RCF will start and
>>>>>>>>>                             connect without issue a few times
>>>>>>>>>                             in a row before failing to connect
>>>>>>>>>                             again. Once the issue begins to
>>>>>>>>>                             occur, neither rebooting all of
>>>>>>>>>                             the hosts (test engine VM, tester,
>>>>>>>>>                             IUT) or deleting all of the build
>>>>>>>>>                             directories (suites, agents, inst)
>>>>>>>>>                             and rebooting the hosts afterward
>>>>>>>>>                             resolves the issue. When it begins
>>>>>>>>>                             working again seems very arbitrary
>>>>>>>>>                             to us.
>>>>>>>>>
>>>>>>>>>                             I do usually try to terminate the
>>>>>>>>>                             test engine with Ctrl+C, but when
>>>>>>>>>                             it hangs while trying to start
>>>>>>>>>                             RCF, that does not work.
>>>>>>>>>
>>>>>>>>>                             Does the test engine prefer to use
>>>>>>>>>                             IPv6 over IPv4 for initiating the
>>>>>>>>>                             RCF connection to the test bed
>>>>>>>>>                             hosts? And if so, is there a way
>>>>>>>>>                             to force it to use IPv4?
>>>>>>>>>
>>>>>>>>>                              - Adam
>>>>>>>>>
>>>>>>>>>                             On Fri, Aug 25, 2023 at 1:35 PM
>>>>>>>>>                             Andrew Rybchenko
>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>
>>>>>>>>>                                 > I'll double-check test
>>>>>>>>>                                 engine on Ubuntu 20.04 and
>>>>>>>>>                                 Ubuntu 22.04.
>>>>>>>>>
>>>>>>>>>                                 Done. It works fine for me
>>>>>>>>>                                 without any issues.
>>>>>>>>>
>>>>>>>>>                                 Have you noticed any pattern
>>>>>>>>>                                 when it works or does not work?
>>>>>>>>>                                 May be it is a problem of not
>>>>>>>>>                                 clean state after termination?
>>>>>>>>>                                 Does it work fine the first
>>>>>>>>>                                 time after DUTs reboot?
>>>>>>>>>                                 How do you terminate testing?
>>>>>>>>>                                 It should be done using Ctrl+C
>>>>>>>>>                                 in terminal where you execute
>>>>>>>>>                                 run.sh command.
>>>>>>>>>                                  In this case it should
>>>>>>>>>                                 shutdown gracefully and close
>>>>>>>>>                                 all test agents and engine
>>>>>>>>>                                 applications.
>>>>>>>>>
>>>>>>>>>                                 (I'm trying to understand why
>>>>>>>>>                                 you've seen many test agent
>>>>>>>>>                                 processes. It should not happen.)
>>>>>>>>>
>>>>>>>>>                                 Andrew.
>>>>>>>>>
>>>>>>>>>                                 On 8/25/23 17:41, Andrew
>>>>>>>>>                                 Rybchenko wrote:
>>>>>>>>>>                                 On 8/25/23 17:06, Adam
>>>>>>>>>>                                 Hassick wrote:
>>>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                                 Two of our systems (the Test
>>>>>>>>>>>                                 Engine runner and the DUT
>>>>>>>>>>>                                 host) are running Ubuntu
>>>>>>>>>>>                                 20.04 LTS, however this
>>>>>>>>>>>                                 morning I noticed that the
>>>>>>>>>>>                                 tester system (the one
>>>>>>>>>>>                                 having issues) is running
>>>>>>>>>>>                                 Ubuntu 22.04 LTS.
>>>>>>>>>>>                                 This could be the source of
>>>>>>>>>>>                                 the problem. I encountered a
>>>>>>>>>>>                                 dependency issue trying to
>>>>>>>>>>>                                 run the Test Engine on 22.04
>>>>>>>>>>>                                 LTS, so I downgraded the
>>>>>>>>>>>                                 system. Since the tester is
>>>>>>>>>>>                                 also the host having
>>>>>>>>>>>                                 connection issues, I will
>>>>>>>>>>>                                 try downgrading that system
>>>>>>>>>>>                                 to 20.04, and see if that
>>>>>>>>>>>                                 changes anything.
>>>>>>>>>>
>>>>>>>>>>                                 Unlikely, but who knows. We
>>>>>>>>>>                                 run tests (DUTs) on Ubuntu
>>>>>>>>>>                                 20.04, Ubuntu 22.04, Ubuntu
>>>>>>>>>>                                 22.10, Ubuntu 23.04, Debian
>>>>>>>>>>                                 11 and Fedora 38 every night.
>>>>>>>>>>                                 Right now Debian 11 is used
>>>>>>>>>>                                 for test engine in nightly
>>>>>>>>>>                                 regressions.
>>>>>>>>>>
>>>>>>>>>>                                 I'll double-check test engine
>>>>>>>>>>                                 on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>>
>>>>>>>>>>>                                 I did try passing in the
>>>>>>>>>>>                                 "--vg-rcf" argument to the
>>>>>>>>>>>                                 run.sh script of the test
>>>>>>>>>>>                                 suite after installing
>>>>>>>>>>>                                 valgrind, but there was no
>>>>>>>>>>>                                 additional output that I saw.
>>>>>>>>>>
>>>>>>>>>>                                 Sorry, I should valgrind
>>>>>>>>>>                                 output should be in
>>>>>>>>>>                                 valgrind.te_rcf (direction
>>>>>>>>>>                                 where you run test engine).
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                 I will try pulling in the
>>>>>>>>>>>                                 changes you've pushed up,
>>>>>>>>>>>                                 and will see if that fixes
>>>>>>>>>>>                                 anything.
>>>>>>>>>>>
>>>>>>>>>>>                                 Thanks,
>>>>>>>>>>>                                 Adam
>>>>>>>>>>>
>>>>>>>>>>>                                 On Fri, Aug 25, 2023 at
>>>>>>>>>>>                                 9:57 AM Andrew Rybchenko
>>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>                                 wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                     Hello Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                     On 8/24/23 23:54, Andrew
>>>>>>>>>>>                                     Rybchenko wrote:
>>>>>>>>>>>>                                     I'd like to try to
>>>>>>>>>>>>                                     repeat the problem
>>>>>>>>>>>>                                     locally. Which Linux
>>>>>>>>>>>>                                     distro is running on
>>>>>>>>>>>>                                     test engine and agents?
>>>>>>>>>>>>
>>>>>>>>>>>>                                     In fact I know one
>>>>>>>>>>>>                                     problem with Debian 12
>>>>>>>>>>>>                                     and Fedora 38 and we have
>>>>>>>>>>>>                                     patch in review to fix
>>>>>>>>>>>>                                     it, however, the
>>>>>>>>>>>>                                     behaviour is different in
>>>>>>>>>>>>                                     this case, so it is
>>>>>>>>>>>>                                     unlike the same problem.
>>>>>>>>>>>
>>>>>>>>>>>                                     I've just published a
>>>>>>>>>>>                                     new tag which fixes
>>>>>>>>>>>                                     known test engine side
>>>>>>>>>>>                                     problems on Debian 12
>>>>>>>>>>>                                     and Fedora 38.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                     One more idea is to
>>>>>>>>>>>>                                     install valgrind on the
>>>>>>>>>>>>                                     test engine host and
>>>>>>>>>>>>                                     run with option
>>>>>>>>>>>>                                     --vg-rcf to check if
>>>>>>>>>>>>                                     something weird is
>>>>>>>>>>>>                                     happening.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     What I don't understand
>>>>>>>>>>>>                                     right now is why I see
>>>>>>>>>>>>                                     just one failed attempt
>>>>>>>>>>>>                                     to connect in your
>>>>>>>>>>>>                                     log.txt and then Logger
>>>>>>>>>>>>                                     shutdown after 9 minutes.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On 8/24/23 23:29, Adam
>>>>>>>>>>>>                                     Hassick wrote:
>>>>>>>>>>>>>                                      > Is there any
>>>>>>>>>>>>>                                     firewall in the
>>>>>>>>>>>>>                                     network or on test
>>>>>>>>>>>>>                                     hosts which could
>>>>>>>>>>>>>                                     block incoming TCP
>>>>>>>>>>>>>                                     connection to the port
>>>>>>>>>>>>>                                     23571
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     from the host where
>>>>>>>>>>>>>                                     you run test engine?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Our test engine host
>>>>>>>>>>>>>                                     and the testbed are on
>>>>>>>>>>>>>                                     the same subnet. The
>>>>>>>>>>>>>                                     connection does work
>>>>>>>>>>>>>                                     sometimes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                      > If behaviour the
>>>>>>>>>>>>>                                     same on the next try
>>>>>>>>>>>>>                                     and you see that test
>>>>>>>>>>>>>                                     agent is kept running,
>>>>>>>>>>>>>                                     could you check using
>>>>>>>>>>>>>                                      >
>>>>>>>>>>>>>                                      > # netstat -tnlp
>>>>>>>>>>>>>                                      >
>>>>>>>>>>>>>                                      > that Test Agent is
>>>>>>>>>>>>>                                     listening on the port
>>>>>>>>>>>>>                                     and try to establish
>>>>>>>>>>>>>                                     TCP connection from
>>>>>>>>>>>>>                                     test agent using
>>>>>>>>>>>>>                                      >
>>>>>>>>>>>>>                                      > $ telnet
>>>>>>>>>>>>>                                     iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     23571
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                      >
>>>>>>>>>>>>>                                      > and check if TCP
>>>>>>>>>>>>>                                     connection could be
>>>>>>>>>>>>>                                     established.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     I was able to
>>>>>>>>>>>>>                                     replicate the same
>>>>>>>>>>>>>                                     behavior again, where
>>>>>>>>>>>>>                                     it hangs while RCF is
>>>>>>>>>>>>>                                     trying to start.
>>>>>>>>>>>>>                                     Running this command,
>>>>>>>>>>>>>                                     I see this in the output:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     tcp        0    0
>>>>>>>>>>>>>                                     0.0.0.0:23571
>>>>>>>>>>>>>                                     <http://0.0.0.0:23571>
>>>>>>>>>>>>>                                     <http://0.0.0.0:23571>
>>>>>>>>>>>>>                                     <http://0.0.0.0:23571>
>>>>>>>>>>>>>                                     0.0.0.0:* LISTEN  18599/ta
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     So it seems like it is
>>>>>>>>>>>>>                                     listening on the
>>>>>>>>>>>>>                                     correct port.
>>>>>>>>>>>>>                                     Additionally, I was
>>>>>>>>>>>>>                                     able to connect to the
>>>>>>>>>>>>>                                     Tester machine from
>>>>>>>>>>>>>                                     our Test Engine host
>>>>>>>>>>>>>                                     using telnet. It
>>>>>>>>>>>>>                                     printed the PID of the
>>>>>>>>>>>>>                                     process once the
>>>>>>>>>>>>>                                     connection was opened.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     I tried running the
>>>>>>>>>>>>>                                     "ta" application
>>>>>>>>>>>>>                                     manually on the
>>>>>>>>>>>>>                                     command line, and it
>>>>>>>>>>>>>                                     didn't print anything
>>>>>>>>>>>>>                                     at all.
>>>>>>>>>>>>>                                     Maybe the issue is
>>>>>>>>>>>>>                                     something on the Test
>>>>>>>>>>>>>                                     Engine side.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     On Thu, Aug 24, 2023
>>>>>>>>>>>>>                                     at 2:35 PM Andrew
>>>>>>>>>>>>>                                     Rybchenko
>>>>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Hi Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                          > On the tester
>>>>>>>>>>>>>                                     host (which appears to
>>>>>>>>>>>>>                                     be the Peer agent), there
>>>>>>>>>>>>>                                         are four processes
>>>>>>>>>>>>>                                     that I see running,
>>>>>>>>>>>>>                                     which look like the test
>>>>>>>>>>>>>                                         agent processes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Before the next
>>>>>>>>>>>>>                                     try I'd recommend to
>>>>>>>>>>>>>                                     kill these processes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Is there any
>>>>>>>>>>>>>                                     firewall in the
>>>>>>>>>>>>>                                     network or on test
>>>>>>>>>>>>>                                     hosts which could
>>>>>>>>>>>>>                                         block incoming TCP
>>>>>>>>>>>>>                                     connection to the port
>>>>>>>>>>>>>                                     23571
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     from the host
>>>>>>>>>>>>>                                         where you run test
>>>>>>>>>>>>>                                     engine?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         If behaviour the
>>>>>>>>>>>>>                                     same on the next try
>>>>>>>>>>>>>                                     and you see that test
>>>>>>>>>>>>>                                     agent is
>>>>>>>>>>>>>                                         kept running,
>>>>>>>>>>>>>                                     could you check using
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         # netstat -tnlp
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         that Test Agent is
>>>>>>>>>>>>>                                     listening on the port
>>>>>>>>>>>>>                                     and try to establish TCP
>>>>>>>>>>>>>                                         connection from
>>>>>>>>>>>>>                                     test agent using
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         $ telnet
>>>>>>>>>>>>>                                     iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     23571
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         and check if TCP
>>>>>>>>>>>>>                                     connection could be
>>>>>>>>>>>>>                                     established.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Another idea is to
>>>>>>>>>>>>>                                     login Tester under
>>>>>>>>>>>>>                                     root as testing does, get
>>>>>>>>>>>>>                                         start TA command
>>>>>>>>>>>>>                                     from the log and try
>>>>>>>>>>>>>                                     it by hands without -n and
>>>>>>>>>>>>>                                         remove extra escaping.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         # sudo
>>>>>>>>>>>>>                                     PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>                                     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>                                     /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>>                                     Peer 23571
>>>>>>>>>>>>>                                     host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Hopefully in this
>>>>>>>>>>>>>                                     case test agent
>>>>>>>>>>>>>                                     directory remains in
>>>>>>>>>>>>>                                     the /tmp and
>>>>>>>>>>>>>                                         you don't need to
>>>>>>>>>>>>>                                     copy it as testing does.
>>>>>>>>>>>>>                                         May be output
>>>>>>>>>>>>>                                     could shed some light
>>>>>>>>>>>>>                                     on what's going on.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         On 8/24/23 17:30,
>>>>>>>>>>>>>                                     Adam Hassick wrote:
>>>>>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         This is the
>>>>>>>>>>>>>>                                     output that I see in
>>>>>>>>>>>>>>                                     the terminal when
>>>>>>>>>>>>>>                                     this failure
>>>>>>>>>>>>>>                                         occurs, after the
>>>>>>>>>>>>>>                                     test agent binaries
>>>>>>>>>>>>>>                                     build and the test engine
>>>>>>>>>>>>>>                                         starts:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         Platform default
>>>>>>>>>>>>>>                                     build - pass
>>>>>>>>>>>>>>                                         Simple RCF
>>>>>>>>>>>>>>                                     consistency check
>>>>>>>>>>>>>>                                     succeeded
>>>>>>>>>>>>>>                                     --->>> Starting
>>>>>>>>>>>>>>                                     Logger...done
>>>>>>>>>>>>>>                                     --->>> Starting
>>>>>>>>>>>>>>                                     RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>>                                     Connection timed
>>>>>>>>>>>>>>                                         out
>>>>>>>>>>>>>>                                     iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         Then, it hangs
>>>>>>>>>>>>>>                                     here until I kill the
>>>>>>>>>>>>>>                                     "te_rcf" and "te_tee"
>>>>>>>>>>>>>>                                         processes. I let
>>>>>>>>>>>>>>                                     it hang for around 9
>>>>>>>>>>>>>>                                     minutes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         On the tester
>>>>>>>>>>>>>>                                     host (which appears
>>>>>>>>>>>>>>                                     to be the Peer
>>>>>>>>>>>>>>                                     agent), there are
>>>>>>>>>>>>>>                                         four processes
>>>>>>>>>>>>>>                                     that I see running,
>>>>>>>>>>>>>>                                     which look like the
>>>>>>>>>>>>>>                                     test agent
>>>>>>>>>>>>>>                                         processes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         ta.Peer is an
>>>>>>>>>>>>>>                                     empty file. I've
>>>>>>>>>>>>>>                                     attached the log.txt
>>>>>>>>>>>>>>                                     from this run.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                          - Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         On Thu, Aug 24,
>>>>>>>>>>>>>>                                     2023 at 4:22 AM
>>>>>>>>>>>>>>                                     Andrew Rybchenko
>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Yes,
>>>>>>>>>>>>>>                                     TE_RCFUNIX_TIMEOUT is
>>>>>>>>>>>>>>                                     in seconds. I've
>>>>>>>>>>>>>>                                     double-checked
>>>>>>>>>>>>>>                                             that it goes
>>>>>>>>>>>>>>                                     to 'copy_timeout' in
>>>>>>>>>>>>>>                                     ts-conf/rcf.conf.
>>>>>>>>>>>>>>                                     Description in in
>>>>>>>>>>>>>>                                     doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>>                                             says that
>>>>>>>>>>>>>>                                     copy_timeout is in
>>>>>>>>>>>>>>                                     seconds and
>>>>>>>>>>>>>>                                     implementation in
>>>>>>>>>>>>>>                                     lib/rcfunix/rcfunix.c
>>>>>>>>>>>>>>                                     passes the value to
>>>>>>>>>>>>>>                                     select() tv_sec.
>>>>>>>>>>>>>>                                     Theoretically
>>>>>>>>>>>>>>                                     select() could be
>>>>>>>>>>>>>>                                     interrupted by
>>>>>>>>>>>>>>                                     signal, but I
>>>>>>>>>>>>>>                                             think it is
>>>>>>>>>>>>>>                                     unlikely here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             I'm not sure
>>>>>>>>>>>>>>                                     that I understand
>>>>>>>>>>>>>>                                     what do you mean by RCF
>>>>>>>>>>>>>>                                     connection timeout.
>>>>>>>>>>>>>>                                     Does it happen on TE
>>>>>>>>>>>>>>                                     startup when RCF
>>>>>>>>>>>>>>                                             starts test
>>>>>>>>>>>>>>                                     agents. If so,
>>>>>>>>>>>>>>                                     TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>                                     could help. Or
>>>>>>>>>>>>>>                                             does it
>>>>>>>>>>>>>>                                     happen when tests are
>>>>>>>>>>>>>>                                     in progress, e.g. in
>>>>>>>>>>>>>>                                     the middle
>>>>>>>>>>>>>>                                             of a test. If
>>>>>>>>>>>>>>                                     so,
>>>>>>>>>>>>>>                                     TE_RCFUNIX_TIMEOUT is
>>>>>>>>>>>>>>                                     unrelated and most
>>>>>>>>>>>>>>                                             likely either
>>>>>>>>>>>>>>                                     host with test agent
>>>>>>>>>>>>>>                                     dies or test agent itself
>>>>>>>>>>>>>>                                     crashes. It would be
>>>>>>>>>>>>>>                                     easier for me if
>>>>>>>>>>>>>>                                     classify it if you share
>>>>>>>>>>>>>>                                             text log
>>>>>>>>>>>>>>                                     (log.txt, full or
>>>>>>>>>>>>>>                                     just corresponding
>>>>>>>>>>>>>>                                     fragment with
>>>>>>>>>>>>>>                                             some
>>>>>>>>>>>>>>                                     context). Also
>>>>>>>>>>>>>>                                     content of ta.DPDK or
>>>>>>>>>>>>>>                                     ta.Peer file
>>>>>>>>>>>>>>                                     depending on which
>>>>>>>>>>>>>>                                     agent has problems
>>>>>>>>>>>>>>                                     could shed some light.
>>>>>>>>>>>>>>                                     Corresponding files
>>>>>>>>>>>>>>                                     contain stdout/stderr
>>>>>>>>>>>>>>                                     of test agents.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On 8/23/23
>>>>>>>>>>>>>>                                     17:45, Adam Hassick
>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             I've set up
>>>>>>>>>>>>>>>                                     a test rig
>>>>>>>>>>>>>>>                                     repository here, and
>>>>>>>>>>>>>>>                                     have created
>>>>>>>>>>>>>>>                                     configurations for
>>>>>>>>>>>>>>>                                     our development
>>>>>>>>>>>>>>>                                     testbed based off of the
>>>>>>>>>>>>>>>                                     examples.
>>>>>>>>>>>>>>>                                             We've been
>>>>>>>>>>>>>>>                                     able to get the test
>>>>>>>>>>>>>>>                                     suite to run manually on
>>>>>>>>>>>>>>>                                     Mellanox CX5 devices
>>>>>>>>>>>>>>>                                     once.
>>>>>>>>>>>>>>>                                     However, we are
>>>>>>>>>>>>>>>                                     running into an
>>>>>>>>>>>>>>>                                     issue where, when
>>>>>>>>>>>>>>>                                     RCF starts,
>>>>>>>>>>>>>>>                                             the RCF
>>>>>>>>>>>>>>>                                     connection times out
>>>>>>>>>>>>>>>                                     very frequently. We
>>>>>>>>>>>>>>>                                     aren't sure
>>>>>>>>>>>>>>>                                             why this is
>>>>>>>>>>>>>>>                                     the case.
>>>>>>>>>>>>>>>                                             It works
>>>>>>>>>>>>>>>                                     sometimes, but most
>>>>>>>>>>>>>>>                                     of the time when we
>>>>>>>>>>>>>>>                                     try to run
>>>>>>>>>>>>>>>                                             the test
>>>>>>>>>>>>>>>                                     engine, it
>>>>>>>>>>>>>>>                                     encounters this issue.
>>>>>>>>>>>>>>>                                             I've tried
>>>>>>>>>>>>>>>                                     changing the RCF
>>>>>>>>>>>>>>>                                     port by setting
>>>>>>>>>>>>>>>                                     "TE_RCF_PORT=<some
>>>>>>>>>>>>>>>                                     port number>" and
>>>>>>>>>>>>>>>                                     rebooting the testbed
>>>>>>>>>>>>>>>                                     machines. Neither
>>>>>>>>>>>>>>>                                     seems to fix the issue.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             It also
>>>>>>>>>>>>>>>                                     seems like the
>>>>>>>>>>>>>>>                                     timeout takes far
>>>>>>>>>>>>>>>                                     longer than 60
>>>>>>>>>>>>>>>                                     seconds, even when
>>>>>>>>>>>>>>>                                     running "export
>>>>>>>>>>>>>>>                                     TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>>                                             before I try
>>>>>>>>>>>>>>>                                     to run the test suite.
>>>>>>>>>>>>>>>                                             I assume the
>>>>>>>>>>>>>>>                                     unit for this
>>>>>>>>>>>>>>>                                     variable is seconds?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             On Mon, Aug
>>>>>>>>>>>>>>>                                     21, 2023 at 10:19 AM
>>>>>>>>>>>>>>>                                     Adam Hassick
>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>                                     <ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Hi Andrew,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                     Thanks, I've cloned
>>>>>>>>>>>>>>>                                     the example
>>>>>>>>>>>>>>>                                     repository and will
>>>>>>>>>>>>>>>                                     start
>>>>>>>>>>>>>>>                                     setting up a
>>>>>>>>>>>>>>>                                     configuration for
>>>>>>>>>>>>>>>                                     our development testbed
>>>>>>>>>>>>>>>                                     today. I'll let you
>>>>>>>>>>>>>>>                                     know if I run into
>>>>>>>>>>>>>>>                                     any difficulties
>>>>>>>>>>>>>>>                                                 or have
>>>>>>>>>>>>>>>                                     any questions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                  - Adam
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On Sun,
>>>>>>>>>>>>>>>                                     Aug 20, 2023 at
>>>>>>>>>>>>>>>                                     4:40 AM Andrew Rybchenko
>>>>>>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                     Hi Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>                                     published
>>>>>>>>>>>>>>>                                     https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>>                                     <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>>                                     <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>                                     Hopefully it will
>>>>>>>>>>>>>>>                                     help to define your
>>>>>>>>>>>>>>>                                     test rigs and
>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>                                     successfully run
>>>>>>>>>>>>>>>                                     some tests manually.
>>>>>>>>>>>>>>>                                     Feel free to
>>>>>>>>>>>>>>>                                                     ask
>>>>>>>>>>>>>>>                                     any questions and
>>>>>>>>>>>>>>>                                     I'll answer here and
>>>>>>>>>>>>>>>                                     try to
>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>                                     update documentation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>                                     Meanwhile I'll
>>>>>>>>>>>>>>>                                     prepare missing bits
>>>>>>>>>>>>>>>                                     for steps (2) and
>>>>>>>>>>>>>>>                                                     (3).
>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>                                     Hopefully everything
>>>>>>>>>>>>>>>                                     is in place for step
>>>>>>>>>>>>>>>                                     (4), but we
>>>>>>>>>>>>>>>                                                     need
>>>>>>>>>>>>>>>                                     to make steps (2)
>>>>>>>>>>>>>>>                                     and (3) first.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>                                     8/18/23 21:40,
>>>>>>>>>>>>>>>                                     Andrew Rybchenko wrote:
>>>>>>>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     >
>>>>>>>>>>>>>>>>                                     I've conferred with
>>>>>>>>>>>>>>>>                                     the rest of the
>>>>>>>>>>>>>>>>                                     team, and we
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                     think it would be
>>>>>>>>>>>>>>>>                                     best to move
>>>>>>>>>>>>>>>>                                     forward with mainly
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                     option B.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     OK,
>>>>>>>>>>>>>>>>                                     I'll provide the
>>>>>>>>>>>>>>>>                                     sample on Monday
>>>>>>>>>>>>>>>>                                     for you. It is
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                     almost ready right
>>>>>>>>>>>>>>>>                                     now, but I need to
>>>>>>>>>>>>>>>>                                     double-check
>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>                                     before publishing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                     Regards,
>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>                                     8/17/23 20:03, Adam
>>>>>>>>>>>>>>>>                                     Hassick wrote:
>>>>>>>>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     I'm adding the CI
>>>>>>>>>>>>>>>>>                                     mailing list to this
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     conversation.
>>>>>>>>>>>>>>>>>                                     Others in the
>>>>>>>>>>>>>>>>>                                     community might find
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     this conversation
>>>>>>>>>>>>>>>>>                                     valuable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     We
>>>>>>>>>>>>>>>>>                                     do want to run
>>>>>>>>>>>>>>>>>                                     testing on a
>>>>>>>>>>>>>>>>>                                     regular basis. The
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     Jenkins
>>>>>>>>>>>>>>>>>                                     integration will
>>>>>>>>>>>>>>>>>                                     be very useful for
>>>>>>>>>>>>>>>>>                                     us, as
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     most of our CI is
>>>>>>>>>>>>>>>>>                                     orchestrated by
>>>>>>>>>>>>>>>>>                                     Jenkins.
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     I've conferred
>>>>>>>>>>>>>>>>>                                     with the rest of
>>>>>>>>>>>>>>>>>                                     the team, and we
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     think it would be
>>>>>>>>>>>>>>>>>                                     best to move
>>>>>>>>>>>>>>>>>                                     forward with mainly
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     option B.
>>>>>>>>>>>>>>>>>                                                     If
>>>>>>>>>>>>>>>>>                                     you would like to
>>>>>>>>>>>>>>>>>                                     know anything
>>>>>>>>>>>>>>>>>                                     about our
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     testbeds that
>>>>>>>>>>>>>>>>>                                     would help you
>>>>>>>>>>>>>>>>>                                     with creating an
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     example ts-rigs
>>>>>>>>>>>>>>>>>                                     repo, I'd be happy
>>>>>>>>>>>>>>>>>                                     to answer any
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     questions you have.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     We
>>>>>>>>>>>>>>>>>                                     have multiple test
>>>>>>>>>>>>>>>>>                                     rigs (we call these
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     "DUT-tester
>>>>>>>>>>>>>>>>>                                     pairs") that we
>>>>>>>>>>>>>>>>>                                     run our existing
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     hardware testing
>>>>>>>>>>>>>>>>>                                     on, with differing
>>>>>>>>>>>>>>>>>                                     network
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     hardware and CPU
>>>>>>>>>>>>>>>>>                                     architecture. I
>>>>>>>>>>>>>>>>>                                     figured this might
>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>                                     an important detail.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                     Thu, Aug 17, 2023
>>>>>>>>>>>>>>>>>                                     at 11:44 AM Andrew
>>>>>>>>>>>>>>>>>                                     Rybchenko
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Greatings Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     I'm happy to hear
>>>>>>>>>>>>>>>>>                                     that you're trying
>>>>>>>>>>>>>>>>>                                     to bring
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     it up.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     As I understand
>>>>>>>>>>>>>>>>>                                     the final goal is
>>>>>>>>>>>>>>>>>                                     to run it on
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     regular basis. So,
>>>>>>>>>>>>>>>>>                                     we need to make it
>>>>>>>>>>>>>>>>>                                     properly
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     from the very
>>>>>>>>>>>>>>>>>                                     beginning.
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Bring up of all
>>>>>>>>>>>>>>>>>                                     features consists
>>>>>>>>>>>>>>>>>                                     of 4 steps:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     1. Create
>>>>>>>>>>>>>>>>>                                     site-specific
>>>>>>>>>>>>>>>>>                                     repository (we call it
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     ts-rigs) which
>>>>>>>>>>>>>>>>>                                     contains
>>>>>>>>>>>>>>>>>                                     information about test
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     rigs and other
>>>>>>>>>>>>>>>>>                                     site-specific
>>>>>>>>>>>>>>>>>                                     information like
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     where to send
>>>>>>>>>>>>>>>>>                                     mails, where to
>>>>>>>>>>>>>>>>>                                     store logs etc.
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     It is required for
>>>>>>>>>>>>>>>>>                                     manual execution
>>>>>>>>>>>>>>>>>                                     as well,
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     since test rigs
>>>>>>>>>>>>>>>>>                                     description is
>>>>>>>>>>>>>>>>>                                     essential. I'll
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     return to the
>>>>>>>>>>>>>>>>>                                     topic below.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     2. Setup logs
>>>>>>>>>>>>>>>>>                                     storage for
>>>>>>>>>>>>>>>>>                                     automated runs.
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Basically it is a
>>>>>>>>>>>>>>>>>                                     disk space plus
>>>>>>>>>>>>>>>>>                                     apache2 web
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     server with few
>>>>>>>>>>>>>>>>>                                     CGI scripts which
>>>>>>>>>>>>>>>>>                                     help a lot to
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     save disk space.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     3. Setup Bublik
>>>>>>>>>>>>>>>>>                                     web application
>>>>>>>>>>>>>>>>>                                     which provides
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     web interface to
>>>>>>>>>>>>>>>>>                                     view testing
>>>>>>>>>>>>>>>>>                                     results. Same as
>>>>>>>>>>>>>>>>>                                     https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>>                                     <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>                                     <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     4. Setup Jenkins
>>>>>>>>>>>>>>>>>                                     to run tests on
>>>>>>>>>>>>>>>>>                                     regularly,
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     save logs in log
>>>>>>>>>>>>>>>>>                                     storage (2) and
>>>>>>>>>>>>>>>>>                                     import it to
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     bublik (3).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Last few month we
>>>>>>>>>>>>>>>>>                                     spent on our
>>>>>>>>>>>>>>>>>                                     homework to make
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     it simpler to
>>>>>>>>>>>>>>>>>                                     bring up automated
>>>>>>>>>>>>>>>>>                                     execution
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     using Jenkins -
>>>>>>>>>>>>>>>>>                                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>>                                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>                                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Corresponding bits
>>>>>>>>>>>>>>>>>                                     in dpdk-ethdev-ts
>>>>>>>>>>>>>>>>>                                     will be
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     available tomorrow.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Let's return to
>>>>>>>>>>>>>>>>>                                     the step (1).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Unfortunately
>>>>>>>>>>>>>>>>>                                     there is no
>>>>>>>>>>>>>>>>>                                     publicly available
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     example of the
>>>>>>>>>>>>>>>>>                                     ts-rigs repository
>>>>>>>>>>>>>>>>>                                     since
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     sensitive
>>>>>>>>>>>>>>>>>                                     site-specific
>>>>>>>>>>>>>>>>>                                     information is located
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     there. But I'm
>>>>>>>>>>>>>>>>>                                     ready to help you
>>>>>>>>>>>>>>>>>                                     to create it
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     for UNH. I see two
>>>>>>>>>>>>>>>>>                                     options here:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     (A) I'll ask
>>>>>>>>>>>>>>>>>                                     questions and
>>>>>>>>>>>>>>>>>                                     based on your
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     answers will
>>>>>>>>>>>>>>>>>                                     create the first
>>>>>>>>>>>>>>>>>                                     draft with my
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     comments.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     (B) I'll make a
>>>>>>>>>>>>>>>>>                                     template/example
>>>>>>>>>>>>>>>>>                                     ts-rigs repo,
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     publish it and
>>>>>>>>>>>>>>>>>                                     you'll create UNH
>>>>>>>>>>>>>>>>>                                     ts-rigs based
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     on it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Of course, I'll
>>>>>>>>>>>>>>>>>                                     help to debug and
>>>>>>>>>>>>>>>>>                                     finally bring
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     it up in any case.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     (A) is a bit
>>>>>>>>>>>>>>>>>                                     simpler for me and
>>>>>>>>>>>>>>>>>                                     you, but (B) is
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     a bit more generic
>>>>>>>>>>>>>>>>>                                     and will help other
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     potential users to
>>>>>>>>>>>>>>>>>                                     bring it up.
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     We can combine
>>>>>>>>>>>>>>>>>                                     (A)+(B). I.e.
>>>>>>>>>>>>>>>>>                                     start from (A).
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     What do you think?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                     On 8/17/23 15:18,
>>>>>>>>>>>>>>>>>                                     Konstantin Ushakov
>>>>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>>>>>                                     Greetings Adam,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                     Thanks for
>>>>>>>>>>>>>>>>>>                                     contacting us. I
>>>>>>>>>>>>>>>>>>                                     copy Andrew who
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                     would be happy to
>>>>>>>>>>>>>>>>>>                                     help
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                     Konstantin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                     On 16 Aug 2023,
>>>>>>>>>>>>>>>>>>>                                     at 21:50, Adam
>>>>>>>>>>>>>>>>>>>                                     Hassick
>>>>>>>>>>>>>>>>>>>                                     <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     Greetings
>>>>>>>>>>>>>>>>>>>                                     Konstantin,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     I am in the
>>>>>>>>>>>>>>>>>>>                                     process of
>>>>>>>>>>>>>>>>>>>                                     setting up the DPDK
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     Poll Mode Driver
>>>>>>>>>>>>>>>>>>>                                     test suite as an
>>>>>>>>>>>>>>>>>>>                                     addition to
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     our testing
>>>>>>>>>>>>>>>>>>>                                     coverage for
>>>>>>>>>>>>>>>>>>>                                     DPDK at the UNH lab.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     I have some
>>>>>>>>>>>>>>>>>>>                                     questions about
>>>>>>>>>>>>>>>>>>>                                     how to set the
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     test suite
>>>>>>>>>>>>>>>>>>>                                     arguments.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     I have been able
>>>>>>>>>>>>>>>>>>>                                     to configure the
>>>>>>>>>>>>>>>>>>>                                     Test Engine
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     to connect to
>>>>>>>>>>>>>>>>>>>                                     the hosts in the
>>>>>>>>>>>>>>>>>>>                                     testbed. The
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     RCF,
>>>>>>>>>>>>>>>>>>>                                     Configurator,
>>>>>>>>>>>>>>>>>>>                                     and Tester all
>>>>>>>>>>>>>>>>>>>                                     begin to
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     run, however the
>>>>>>>>>>>>>>>>>>>                                     prelude of the
>>>>>>>>>>>>>>>>>>>                                     test suite
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     fails to run.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>>                                     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>                                     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     The
>>>>>>>>>>>>>>>>>>>                                     documentation
>>>>>>>>>>>>>>>>>>>                                     mentions that
>>>>>>>>>>>>>>>>>>>                                     there are
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     several test
>>>>>>>>>>>>>>>>>>>                                     parameters for
>>>>>>>>>>>>>>>>>>>                                     the test suite,
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     like for the IUT
>>>>>>>>>>>>>>>>>>>                                     test link MAC,
>>>>>>>>>>>>>>>>>>>                                     etc. These
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     seem like they
>>>>>>>>>>>>>>>>>>>                                     would need to be
>>>>>>>>>>>>>>>>>>>                                     set somewhere
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     to run many of
>>>>>>>>>>>>>>>>>>>                                     the tests.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     I see in the
>>>>>>>>>>>>>>>>>>>                                     Test Engine
>>>>>>>>>>>>>>>>>>>                                     documentation, there
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     are instructions
>>>>>>>>>>>>>>>>>>>                                     on how to create new
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     parameters for
>>>>>>>>>>>>>>>>>>>                                     test suites in
>>>>>>>>>>>>>>>>>>>                                     the Tester
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     configuration,
>>>>>>>>>>>>>>>>>>>                                     but there is
>>>>>>>>>>>>>>>>>>>                                     nothing in the
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     user guide or in
>>>>>>>>>>>>>>>>>>>                                     the Tester guide
>>>>>>>>>>>>>>>>>>>                                     for how to
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     set the
>>>>>>>>>>>>>>>>>>>                                     arguments for
>>>>>>>>>>>>>>>>>>>                                     the parameters when
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     running the test
>>>>>>>>>>>>>>>>>>>                                     suite that I can
>>>>>>>>>>>>>>>>>>>                                     find. I'm
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     not sure if I
>>>>>>>>>>>>>>>>>>>                                     need to write my
>>>>>>>>>>>>>>>>>>>                                     own Tester
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     config, or if I
>>>>>>>>>>>>>>>>>>>                                     should be
>>>>>>>>>>>>>>>>>>>                                     setting these in
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     some other way.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     How should these
>>>>>>>>>>>>>>>>>>>                                     values be set?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     I'm also not
>>>>>>>>>>>>>>>>>>>                                     sure what
>>>>>>>>>>>>>>>>>>>                                     environment
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     variables/arguments
>>>>>>>>>>>>>>>>>>>                                     are strictly
>>>>>>>>>>>>>>>>>>>                                     necessary or
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     which are optional.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     Regards,
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     --
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     UNH
>>>>>>>>>>>>>>>>>>>                                     InterOperability Lab
>>>>>>>>>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                     UNH
>>>>>>>>>>>>>>>>>                                     InterOperability Lab
>>>>>>>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>>>>>>>                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>                                     (603) 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 -- *Adam
>>>>>>>>>>>>>>>                                     Hassick*
>>>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>>>>>                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                                 +1 (603)
>>>>>>>>>>>>>>>                                     475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             --        
>>>>>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>>>>>                                             Senior Developer
>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>                                     InterOperability Lab
>>>>>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>>>>>                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>                                     475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         -- *Adam Hassick*
>>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>>                                     InterOperability Lab
>>>>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>>>>                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     -- 
>>>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>>>                                     <http://iol.unh.edu>
>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                 --
>>>>>>>>>>>                                 *Adam Hassick*
>>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                             --
>>>>>>>>>                             *Adam Hassick*
>>>>>>>>>                             Senior Developer
>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>>                             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                         --
>>>>>>>                         *Adam Hassick*
>>>>>>>                         Senior Developer
>>>>>>>                         UNH InterOperability Lab
>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                         +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 --
>>>>>                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>             --
>>>>             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --
>>>>         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 200902 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 52078 bytes --]

Hi Andrew,

Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set anywhere in
the default configurations for the Intel X710. Do these default to vfio-pci?
We have IOMMU enabled on our development testbed, and should be able to
bind vfio-pci.
Here is the text log from a run on our Intel XL710 NICs, with the expected
result profile set to the X710. We haven't set up the Jenkins integration
yet, however if this is required to import the logs then we will prioritize
that.
 log.txt.tar.gz
<https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>

Thanks,
Adam

On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> On 9/18/23 17:44, Adam Hassick wrote:
>
> Hi Andrew and Konstantin,
>
> Thank you for adding the tester-dial feature, this opens up the
> possibility for us to do CI integrated testing in the future.
>
> Our Mellanox pass rate is similar to yours (about ~2400 passing, ~4400
> failing), however our Intel pass rates are far worse.
> I will try running tests on the XL710 with the trc-tags argument set and
> see if it improves the pass rate.
> Another thing I noticed in the results you uploaded is that the results
> are tagged with vfio-pci and not i40e.
> Though in the environment dump, the driver on the test machine and the DUT
> are set to use the i40e driver. Is this important at all?
>
>
> I think it is a misunderstanding here. There are two kinds of driver in
> configuration: net driver and so-called DPDK driver.
> Net driver is a Linux kernel network device driver used on Tester side.
> DPDK driver is a Linux kernel driver to bind device to to use it with
> DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
> In the case of bifurcated driver (like mlx5_core) it is the same in both
> cases.
> In non-bifurcated case DPDK driver is some UIO driver(vfio-pci,
> uio-pci-generic or igb_uio).
> Some expectations depend on used UIO. For example, uio-pci-generic do not
> support many interrupts (used by usecases/rx_intr test cases).
> That's why we care corresponding TRC tag.
>
> TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's Intel case. Or
> uio-pci-generic if IOMMU is turned off on corresponding machines and Linux
> distro does not support VFIO no IOMMU mode.
>
> Andrew.
>
> There isn't anything preventing us from pushing our results up to the
> existing Bublik instance running at ts-factory.io that I can think of at
> the moment.
> We will have to work out how to submit our results to your Bublik instance
> in a controlled and secure manner in that case.
> As far as I know we won't need access controls for the results themselves.
> I'll discuss this with Patrick and will let you know once we confirm that
> it's fine.
>
> Thanks,
> Adam
>
> On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> On 9/18/23 09:23, Konstantin Ushakov wrote:
>>
>> Hi Andrew,
>>
>> should we always auto-assign the tags or you don’t do it since it slows
>> down (by some seconds) the TE startup?
>>
>>
>> Tags are auto-assigned, but I guess it differs in Adam's case since NIC
>> is a bit different. Below test will help to understand if it is the root
>> cause of very different expectations. If pass rate will be close to mine,
>> I'll simply update TRC database to share expectations for mine NIC and NIC
>> used by Adam.
>>
>> Hi Adam,
>>
>> I think I second the question from Andrew - happy to help you with the
>> triage so that we get to the same baseline. Do you have a good way for us
>> to share the logs? I.e. say upload to ts-factory if we add strict
>> permissions system so it’s not publishing or any other way.
>>
>> Thanks,
>> Konstantin
>>
>> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>
>> Hi Adam,
>>
>> I've uploaded fresh testing results to ts-factory.io [1] to be on the
>> same page.
>>
>> I think I know why your and mine results on Intel 710 series NICs differ
>> so much. Testing results expectations database (dpdk-ethdev-ts/trc/*) is
>> filled in in terms of TRC tags.  I.e. expectations depends on TRC tags
>> discovered by helper scripts when testing is started. These tags identify
>> various aspects of what is tested. Ideally expectations should be written
>> in terms of root cause of the expected behaviour. If it is a driver
>> expectations, driver tag should be used. If it is HW limitation, tags with
>> PCI IDs should be used. However, it is not always easy to classify it
>> correctly if you're not involved in driver development. So, in order case
>> expectations for 710's Intel are filled in in terms of PCI IDs. I guess PCI
>> ID differ in your case and that's why expectations filled in for my NIC do
>> not apply to your runs.
>>
>> Just try to add the following option when you run on your 710's Intel in
>> order to mimic mine and see if it helps to achieve better pass rate.
>> --trc-tag=pci-8086-1572
>>
>> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for
>> --tester-dial option. It should have better coverage now.
>>
>> Andrew.
>>
>> [1]
>> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>
>> On 9/13/23 18:45, Andrew Rybchenko wrote:
>>
>> Hi Adam,
>>
>> I've pushed new TE tag v1.20.0 which supported a new command-line option
>> --tester-dial=NUM where NUM is from 0 to 100. it allows to choose
>> percentage of tests to run. If you want stable set, you should pass
>> --tester-random-seed=0 (or other integer). It is the first sketch and we
>> have plans to improve it, but feedback would be welcome.
>>
>> > Is it needed on the tester?
>>
>> It is hard to say if it is strictly required for simple tests. However,
>> it is better to update Tester as well, since performance tests run DPDK on
>> Tester as well.
>>
>> > Are there any other manual setup steps for these devices that I might
>> be missing?
>>
>> I don't remember anything else.
>>
>> I think it is better to get down to details and take a look at logs. I'm
>> ready to help with it and explain what's happening there. May be it will
>> help to understand if it is a problem with setup/configuration.
>>
>> Text logs are not very convenient. Ideally logs should be imported to
>> bublik, however, manual runs do not provide all required artifacts right
>> now (Jenkins jobs generate all required artifacts).
>> Other option is 'tmp_raw_log' file (should be packed to make it smaller)
>> which could be converted to various log formats.
>> Would it be OK for you if I import your logs to bublik at ts-factory.io?
>> Or is it a problem that it is publicly available?
>> Would it help if we add authentication and access control there?
>>
>> Andrew.
>>
>> On 9/8/23 17:57, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> I have a couple questions about needed setup of the NICs for the ethdev
>> test suite.
>>
>> Our MCX5s and XL710s are failing the checkup tests. The pass rate appears
>> to be much worse on the XL710s (40 of 73 tests failed, 3 passed
>> unexpectedly).
>>
>> For the XL710s, I've updated the driver and NVM versions to match the
>> minimum supported versions in the compatibility matrix found on the DPDK
>> documentation. This did not change the failure rate much.
>> For the MCX5s, I've installed the latest LTS version of the OFED
>> bifurcated driver on the DUT. Is it needed on the tester?
>>
>> Are there any other manual setup steps for these devices that I might be
>> missing?
>>
>> Thanks,
>> Adam
>>
>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> Yes, I copied the X710 configs to set up XL710 configs. I changed the
>>> environment variable names from the X710 suffix to XL710 suffix in the
>>> script, and forgot to change them in the corresponding environment file.
>>> That fixed the issue.
>>>
>>> I got the checkup tests working on the XL710 now. Most of them are
>>> failing, which leads me to believe this is an issue with our testbed. Based
>>> on the DPDK documentation for i40e, the firmware and driver versions are
>>> much older than what DPDK 22.11 LTS and main prefer, so I'll try updating
>>> those.
>>>
>>> For now I'm working on getting the XL710 checkup tests passing, and will
>>> pick up getting the E810 configured properly next. I'll let you know if I
>>> run into any more issues in relation to the test engine.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hi Adam,
>>>>
>>>> On 9/5/23 18:01, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> The compilation warning issue is now resolved. Again, thank you guys
>>>> for fixing this for us. I can run the tests on the Mellanox CX5s again,
>>>> however I'm running into a couple new issues with running the prologues on
>>>> the Intel cards.
>>>>
>>>> When running testing on the Intel XL710s, I see this error appear in
>>>> the log:
>>>>
>>>> ERROR  prologue  Environment LIB  14:16:13.650
>>>>> Too few networks in available configuration (0) in comparison with
>>>>> required (1)
>>>>>
>>>>
>>>> This seems like a trivial configuration error, perhaps this is
>>>> something I need to set up in ts-rigs. I briefly searched through the
>>>> examples there and didn't see any mention of how to set up a network.
>>>> I will attach this log just in case you need more information.
>>>>
>>>>
>>>> Unfortunately logs are insufficient to understand it. I've pushed new
>>>> tag to TE v1.19.0 which add log message with TE_* environment variables.
>>>> Most likely something is wrong with variables which are used as
>>>> conditions when available networks are defined in
>>>> ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>> TE_PCI_INSTANCE_IUT_TST1
>>>> TE_PCI_INSTANCE_IUT_TST1a
>>>> TE_PCI_INSTANCE_TST1a_IUT
>>>> TE_PCI_INSTANCE_TST1_IUT
>>>> My guess it that you change naming a bit, but script like
>>>> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>>>>
>>>> There is a different error when running on the Intel E810s. It appears
>>>> to me like it starts DPDK, does some configuration inside DPDK and on the
>>>> device, and then fails to bring the device back up. Since this error seems
>>>> very non-trivial, I will also attach this log.
>>>>
>>>>
>>>> This one is a bit simpler. Few lines after the first ERROR in log I see
>>>> the following:
>>>> WARN  RCF  DPDK  13:06:00.144
>>>> ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)
>>>> ice_rx_queue_start(): fail to program RX queue 0
>>>> ice_dev_start(): fail to start Rx queue 0
>>>> Device with port_id=0 already stopped
>>>>
>>>> It is stdout/stderr from test agent which runs DPDK. Same logs in plain
>>>> format are available in ta.DPDK file.
>>>> I'm not an expert here, but I vaguely remember that E810 requires
>>>> correct firmware and DDP to be loaded.
>>>> There is some information in dpdk/doc/guides/nics/ice.rst.
>>>>
>>>> You can try to add --dev-args=safe-mode-support=1 command-line option
>>>> described there.
>>>>
>>>> Hope it helps,
>>>> Andrew.
>>>>
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> Hi Adam,
>>>>>
>>>>> On 8/31/23 22:38, Adam Hassick wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> I have one additional question as well: Does the test engine support
>>>>> running tests on two ARMv8 test agents?
>>>>>
>>>>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>>>>
>>>>>
>>>>> Great. Let me know when that's fixed.
>>>>>
>>>>>
>>>>> Done. We also fixed a number of warnings in TE.
>>>>> Also we fixed root test package name to be consistent with the
>>>>> repository name.
>>>>>
>>>>> Support for old LTS branches was dropped some time ago, but in the
>>>>>> future it is definitely possible to keep it for new LTS branches. I think
>>>>>> 22.11 is supported, but I'm not sure about older LTS releases.
>>>>>>
>>>>>
>>>>> Good to know.
>>>>>
>>>>>
>>>>>> 2. You can add command-line option --sanity to run tests marked with
>>>>>> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>>>> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
>>>>>> there is a space for terminology improvement here. We'll do it.
>>>>>>
>>>>>
>>>>> Done. Now it is called --checkup.
>>>>>
>>>>>
>>>>>> Also it takes a lot of time because of failures and tests which wait
>>>>>> for some timeout.
>>>>>>
>>>>>
>>>>> That makes sense to me. We'll use the time to complete tests on virtio
>>>>> or the Intel devices as a reference for how long the tests really take to
>>>>> complete.
>>>>> We will explore the possibility of periodically running the sanity
>>>>> tests for patches.
>>>>>
>>>>>
>>>>> I'll double-check and let you know how long entire TS runs on Intel
>>>>> X710, E810, Mellanox CX5 and virtio net. Just to ensure that time observed
>>>>> in your case looks the same.
>>>>>
>>>>>
>>>>>
>>>>>> The test harness can provide coverage reports based on gcov, but I'm
>>>>>> not sure what you mean by a "dial" to control test coverage. Provided
>>>>>> reports are rather for human to analyze.
>>>>>>
>>>>>
>>>>> The general idea is to have some kind of parameter on the test suite,
>>>>> which could be an integer ranging from zero to ten, that controls how many
>>>>> tests are run based on how important the test is.
>>>>>
>>>>> Similar to how some command line interfaces provide a verbosity level
>>>>> parameter (some number of "-v" arguments) to control the importance of the
>>>>> information in the log.
>>>>> The verbosity level zero only prints very important log messages,
>>>>> while ten prints everything.
>>>>>
>>>>> In much the same manner as above, this "dial" parameter controls what
>>>>> tests are run and with what parameters based on how important those tests
>>>>> and test parameter combinations are.
>>>>> Coverage Level zero tells the suite to run a very basic set of
>>>>> important tests, with minimal parameterization. This mode would take only
>>>>> ~5-10 minutes to run.
>>>>> In contrast, Coverage Level ten includes all the edge cases, every
>>>>> combination of test parameters, everything the test suite can do, which
>>>>> takes the normal several hours to run.
>>>>> The values 1 - 9 are between those two extremes, allowing the user to
>>>>> get a gradient of test coverage in the results and to limit the running
>>>>> time.
>>>>>
>>>>> Then we could, for example, run the "run.sh" with a level of 2 or 3
>>>>> for incoming patches that need quick results, and with a level of 10 for
>>>>> the less often run periodic tests performed on main or LTS branches.
>>>>>
>>>>>
>>>>> Understood now. Thanks a lot for the idea. We'll discuss it and come
>>>>> back.
>>>>>
>>>>>
>>>>>
>>>>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>>>>>> testing results. Unfortunately it is time consuming to fill in expectations
>>>>>> database since it is necessary to analyze testing results and classify if
>>>>>> it is a bug or just acceptable behaviour aspect.
>>>>>>
>>>>>> Bublik allows to compare results of two runs. It is useful for human,
>>>>>> but still not good for automation.
>>>>>>
>>>>>> I have local patch for mlx5 driver which reports Tx ring size
>>>>>> maximum. It makes pass rate higher. It is a problem for test harness that
>>>>>> mlx5 does not report limits right now.
>>>>>>
>>>>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>>>>>> virtio net is 99% right now and could be done 100% easily (just one thing
>>>>>> to fix in expectations).
>>>>>>
>>>>>> I think logs storage setup is essential for logs analysis. Of course,
>>>>>> you can request HTML logs when you run tests (--log-html=html) or generate
>>>>>> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>>>>>> browser, but logs storage makes it more convenient.
>>>>>>
>>>>>
>>>>> We are interested in setting up Bublik, potentially as an
>>>>> externally-facing component, once we have our process of running the test
>>>>> suite stabilized.
>>>>> Once we are able to run the test suite again, I'll see what the pass
>>>>> rate is on our other hardware.
>>>>> Good to know that it isn't an issue with our dev testbed causing the
>>>>> high fail rate.
>>>>>
>>>>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>>>>> development testbed. Although they are slightly different devices, ideally
>>>>> the pass rate will be identical or similar. I have yet to set up a VM pair
>>>>> for virtio, but we will soon.
>>>>>
>>>>> Latest version of test-environment has examples of our CGI scripts
>>>>>> which we use for log storage (see tools/log_server/README.md).
>>>>>>
>>>>>> Also all bits for Jenkins setup are available. See
>>>>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>>>>> ts-rigs-sample.
>>>>>>
>>>>>
>>>>> Jenkins integration, setting up production rig configurations, and
>>>>> permanent log storage will be our next steps once I am able to run the
>>>>> tests again.
>>>>> Unless there is an easy way to have meson not pass "-Werror" into GCC.
>>>>> Then I would be able to run the test suite.
>>>>>
>>>>>
>>>>> Hopefully it is resolved now.
>>>>>
>>>>> I thought a bit more about your usecase for Jenkins. I'm not 100% sure
>>>>> that existing pipelines are convenient for your usecase.
>>>>> Fill free to ask questions when you are on it.
>>>>>
>>>>> Thanks,
>>>>> Andrew.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> That fix seems to have resolved the issue, thanks for the quick
>>>>>> turnaround time on that patch.
>>>>>> Now that we have the RCF timeout issue resolved, there are a few
>>>>>> other questions and issues that we have about the tests themselves.
>>>>>>
>>>>>> 1. The test suite fails to build with a couple warnings.
>>>>>>
>>>>>> Below is the stderr log from compilation:
>>>>>>
>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>>> and no format arguments [-Werror=format-security]
>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>> |     ^~
>>>>>>> cc1: all warnings being treated as errors
>>>>>>> ninja: build stopped: subcommand failed.
>>>>>>> ninja: Entering directory `.'
>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>>> and no format arguments [-Werror=format-security]
>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>> |     ^~
>>>>>>> cc1: all warnings being treated as errors
>>>>>>>
>>>>>>
>>>>>> This error wasn't occurring last week, which was the last time I ran
>>>>>> the tests.
>>>>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester has
>>>>>> GCC v11.4.0 installed, if this information is helpful.
>>>>>>
>>>>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>>>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>>>>> to lower the test coverage and have the test suite run faster?
>>>>>>
>>>>>> For some context, we run immediate testing on incoming patches for
>>>>>> DPDK main and development branches, as well as periodic test runs on the
>>>>>> main, stable, and LTS branches.
>>>>>> For us to consider including this test suite as part of our immediate
>>>>>> testing on patches, we would have to reduce the test coverage to the most
>>>>>> important tests.
>>>>>> This is primarily to reduce the testing time to, for example, less
>>>>>> than 30 minutes. Testing on patches can't take too long because the lab can
>>>>>> receive numerous patches each day, which each require individual testing
>>>>>> runs.
>>>>>>
>>>>>> At what frequency we run these tests, and on what, still needs to be
>>>>>> discussed with the DPDK community, but it would be nice to know if the test
>>>>>> suite had a "dial" to control the testing coverage.
>>>>>>
>>>>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>>>>> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose these
>>>>>> test failures?
>>>>>>
>>>>>> Thanks,
>>>>>> Adam
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>> Hi Adam,
>>>>>>>
>>>>>>> I've pushed the fix in main branch and a new tag v1.18.1. It should
>>>>>>> solve the problem with IPv6 address from DNS.
>>>>>>>
>>>>>>> Andrew.
>>>>>>>
>>>>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>
>>>>>>> Hi Adam,
>>>>>>>
>>>>>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>>> force it to use IPv4?
>>>>>>>
>>>>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your
>>>>>>> case, I guess it is the root cause of the problem.
>>>>>>> Of course, it is TE problem since I see really weird code in
>>>>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>>>
>>>>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>>>>> ts-factory/test-environment repository. Please, try.
>>>>>>>
>>>>>>> It is late night fix with minimal testing and no review. I'll pass
>>>>>>> it through review process tomorrow and
>>>>>>> hopefully it will be released in one-two days.
>>>>>>>
>>>>>>> Andrew.
>>>>>>>
>>>>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> We have yet to notice a distinct pattern with the failures.
>>>>>>> Sometimes, the RCF will start and connect without issue a few times in a
>>>>>>> row before failing to connect again. Once the issue begins to occur,
>>>>>>> neither rebooting all of the hosts (test engine VM, tester, IUT) or
>>>>>>> deleting all of the build directories (suites, agents, inst) and rebooting
>>>>>>> the hosts afterward resolves the issue. When it begins working again seems
>>>>>>> very arbitrary to us.
>>>>>>>
>>>>>>> I do usually try to terminate the test engine with Ctrl+C, but when
>>>>>>> it hangs while trying to start RCF, that does not work.
>>>>>>>
>>>>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>>>>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>>>>>> to use IPv4?
>>>>>>>
>>>>>>>  - Adam
>>>>>>>
>>>>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>
>>>>>>>> Done. It works fine for me without any issues.
>>>>>>>>
>>>>>>>> Have you noticed any pattern when it works or does not work?
>>>>>>>> May be it is a problem of not clean state after termination?
>>>>>>>> Does it work fine the first time after DUTs reboot?
>>>>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>>>>> terminal where you execute run.sh command.
>>>>>>>>  In this case it should shutdown gracefully and close all test
>>>>>>>> agents and engine applications.
>>>>>>>>
>>>>>>>> (I'm trying to understand why you've seen many test agent
>>>>>>>> processes. It should not happen.)
>>>>>>>>
>>>>>>>> Andrew.
>>>>>>>>
>>>>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>>
>>>>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>>>>> This could be the source of the problem. I encountered a dependency
>>>>>>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>>>>>>> system. Since the tester is also the host having connection issues, I will
>>>>>>>> try downgrading that system to 20.04, and see if that changes anything.
>>>>>>>>
>>>>>>>>
>>>>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>>>>>>> Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every
>>>>>>>> night.
>>>>>>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>>>>>>
>>>>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>
>>>>>>>> I did try passing in the "--vg-rcf" argument to the run.sh script
>>>>>>>> of the test suite after installing valgrind, but there was no additional
>>>>>>>> output that I saw.
>>>>>>>>
>>>>>>>>
>>>>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>>>>> (direction where you run test engine).
>>>>>>>>
>>>>>>>>
>>>>>>>> I will try pulling in the changes you've pushed up, and will see if
>>>>>>>> that fixes anything.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Adam
>>>>>>>>
>>>>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>> Hello Adam,
>>>>>>>>>
>>>>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>>>
>>>>>>>>> I'd like to try to repeat the problem locally. Which Linux distro
>>>>>>>>> is running on test engine and agents?
>>>>>>>>>
>>>>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>>>>>>> patch in review to fix it, however, the behaviour is different in
>>>>>>>>> this case, so it is unlike the same problem.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I've just published a new tag which fixes known test engine side
>>>>>>>>> problems on Debian 12 and Fedora 38.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>>>>> run with option --vg-rcf to check if something weird is happening.
>>>>>>>>>
>>>>>>>>> What I don't understand right now is why I see just one failed
>>>>>>>>> attempt
>>>>>>>>> to connect in your log.txt and then Logger shutdown after 9
>>>>>>>>> minutes.
>>>>>>>>>
>>>>>>>>> Andrew.
>>>>>>>>>
>>>>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>>  > Is there any firewall in the network or on test hosts which
>>>>>>>>> could block incoming TCP connection to the port 23571
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>> where you run test engine?
>>>>>>>>>
>>>>>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>>>>>> connection does work sometimes.
>>>>>>>>>
>>>>>>>>>  > If behaviour the same on the next try and you see that test
>>>>>>>>> agent is kept running, could you check using
>>>>>>>>>  >
>>>>>>>>>  > # netstat -tnlp
>>>>>>>>>  >
>>>>>>>>>  > that Test Agent is listening on the port and try to establish
>>>>>>>>> TCP connection from test agent using
>>>>>>>>>  >
>>>>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>  >
>>>>>>>>>  > and check if TCP connection could be established.
>>>>>>>>>
>>>>>>>>> I was able to replicate the same behavior again, where it hangs
>>>>>>>>> while RCF is trying to start.
>>>>>>>>> Running this command, I see this in the output:
>>>>>>>>>
>>>>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>>>>>    18599/ta
>>>>>>>>>
>>>>>>>>> So it seems like it is listening on the correct port.
>>>>>>>>> Additionally, I was able to connect to the Tester machine from our
>>>>>>>>> Test Engine host using telnet. It printed the PID of the process once the
>>>>>>>>> connection was opened.
>>>>>>>>>
>>>>>>>>> I tried running the "ta" application manually on the command line,
>>>>>>>>> and it didn't print anything at all.
>>>>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>>>>> andrew.rybchenko@oktetlabs.ru
>>>>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>     Hi Adam,
>>>>>>>>>
>>>>>>>>>      > On the tester host (which appears to be the Peer agent),
>>>>>>>>> there
>>>>>>>>>     are four processes that I see running, which look like the test
>>>>>>>>>     agent processes.
>>>>>>>>>
>>>>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>>>>
>>>>>>>>>     Is there any firewall in the network or on test hosts which
>>>>>>>>> could
>>>>>>>>>     block incoming TCP connection to the port 23571
>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>>     where you run test engine?
>>>>>>>>>
>>>>>>>>>     If behaviour the same on the next try and you see that test
>>>>>>>>> agent is
>>>>>>>>>     kept running, could you check using
>>>>>>>>>
>>>>>>>>>     # netstat -tnlp
>>>>>>>>>
>>>>>>>>>     that Test Agent is listening on the port and try to establish
>>>>>>>>> TCP
>>>>>>>>>     connection from test agent using
>>>>>>>>>
>>>>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>
>>>>>>>>>     and check if TCP connection could be established.
>>>>>>>>>
>>>>>>>>>     Another idea is to login Tester under root as testing does, get
>>>>>>>>>     start TA command from the log and try it by hands without -n
>>>>>>>>> and
>>>>>>>>>     remove extra escaping.
>>>>>>>>>
>>>>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>
>>>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>
>>>>>>>>>     Hopefully in this case test agent directory remains in the
>>>>>>>>> /tmp and
>>>>>>>>>     you don't need to copy it as testing does.
>>>>>>>>>     May be output could shed some light on what's going on.
>>>>>>>>>
>>>>>>>>>     Andrew.
>>>>>>>>>
>>>>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>>     Hi Andrew,
>>>>>>>>>
>>>>>>>>>     This is the output that I see in the terminal when this failure
>>>>>>>>>     occurs, after the test agent binaries build and the test engine
>>>>>>>>>     starts:
>>>>>>>>>
>>>>>>>>>     Platform default build - pass
>>>>>>>>>     Simple RCF consistency check succeeded
>>>>>>>>>     --->>> Starting Logger...done
>>>>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>>>>>> timed
>>>>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>
>>>>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>>>>
>>>>>>>>>     On the tester host (which appears to be the Peer agent), there
>>>>>>>>> are
>>>>>>>>>     four processes that I see running, which look like the test
>>>>>>>>> agent
>>>>>>>>>     processes.
>>>>>>>>>
>>>>>>>>>     ta.Peer is an empty file. I've attached the log.txt from this
>>>>>>>>> run.
>>>>>>>>>
>>>>>>>>>      - Adam
>>>>>>>>>
>>>>>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>         Hi Adam,
>>>>>>>>>
>>>>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>         says that copy_timeout is in seconds and implementation in
>>>>>>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>>>>>>         Theoretically select() could be interrupted by signal, but
>>>>>>>>> I
>>>>>>>>>         think it is unlikely here.
>>>>>>>>>
>>>>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help.
>>>>>>>>> Or
>>>>>>>>>         does it happen when tests are in progress, e.g. in the
>>>>>>>>> middle
>>>>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>>>>>         likely either host with test agent dies or test agent
>>>>>>>>> itself
>>>>>>>>>         crashes. It would be easier for me if classify it if you
>>>>>>>>> share
>>>>>>>>>         text log (log.txt, full or just corresponding fragment with
>>>>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>>>>         depending on which agent has problems could shed some
>>>>>>>>> light.
>>>>>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>>>>>
>>>>>>>>>         Andrew.
>>>>>>>>>
>>>>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>>         Hi Andrew,
>>>>>>>>>
>>>>>>>>>         I've set up a test rig repository here, and have created
>>>>>>>>>         configurations for our development testbed based off of the
>>>>>>>>>         examples.
>>>>>>>>>         We've been able to get the test suite to run manually on
>>>>>>>>>         Mellanox CX5 devices once.
>>>>>>>>>         However, we are running into an issue where, when RCF
>>>>>>>>> starts,
>>>>>>>>>         the RCF connection times out very frequently. We aren't
>>>>>>>>> sure
>>>>>>>>>         why this is the case.
>>>>>>>>>         It works sometimes, but most of the time when we try to run
>>>>>>>>>         the test engine, it encounters this issue.
>>>>>>>>>         I've tried changing the RCF port by setting
>>>>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>>>>>>         machines. Neither seems to fix the issue.
>>>>>>>>>
>>>>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>         before I try to run the test suite.
>>>>>>>>>         I assume the unit for this variable is seconds?
>>>>>>>>>
>>>>>>>>>         Thanks,
>>>>>>>>>         Adam
>>>>>>>>>
>>>>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>>>>
>>>>>>>>>             Hi Andrew,
>>>>>>>>>
>>>>>>>>>             Thanks, I've cloned the example repository and will
>>>>>>>>> start
>>>>>>>>>             setting up a configuration for our development testbed
>>>>>>>>>             today. I'll let you know if I run into any difficulties
>>>>>>>>>             or have any questions.
>>>>>>>>>
>>>>>>>>>              - Adam
>>>>>>>>>
>>>>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                 Hi Adam,
>>>>>>>>>
>>>>>>>>>                 I've published
>>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>                 Hopefully it will help to define your test rigs and
>>>>>>>>>                 successfully run some tests manually. Feel free to
>>>>>>>>>                 ask any questions and I'll answer here and try to
>>>>>>>>>                 update documentation.
>>>>>>>>>
>>>>>>>>>                 Meanwhile I'll prepare missing bits for steps (2)
>>>>>>>>> and
>>>>>>>>>                 (3).
>>>>>>>>>                 Hopefully everything is in place for step (4), but
>>>>>>>>> we
>>>>>>>>>                 need to make steps (2) and (3) first.
>>>>>>>>>
>>>>>>>>>                 Andrew.
>>>>>>>>>
>>>>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>>
>>>>>>>>>                 Hi Adam,
>>>>>>>>>
>>>>>>>>>                 > I've conferred with the rest of the team, and we
>>>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>>>                 option B.
>>>>>>>>>
>>>>>>>>>                 OK, I'll provide the sample on Monday for you. It
>>>>>>>>> is
>>>>>>>>>                 almost ready right now, but I need to double-check
>>>>>>>>>                 it before publishing.
>>>>>>>>>
>>>>>>>>>                 Regards,
>>>>>>>>>                 Andrew.
>>>>>>>>>
>>>>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>>                 Hi Andrew,
>>>>>>>>>
>>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>>                 conversation. Others in the community might find
>>>>>>>>>                 this conversation valuable.
>>>>>>>>>
>>>>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>>>>                 Jenkins integration will be very useful for us, as
>>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>>>                 option B.
>>>>>>>>>                 If you would like to know anything about our
>>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>>>>                 questions you have.
>>>>>>>>>
>>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>>                 hardware and CPU architecture. I figured this might
>>>>>>>>>                 be an important detail.
>>>>>>>>>
>>>>>>>>>                 Thanks,
>>>>>>>>>                 Adam
>>>>>>>>>
>>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                     Greatings Adam,
>>>>>>>>>
>>>>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>>>>                     it up.
>>>>>>>>>
>>>>>>>>>                     As I understand the final goal is to run it on
>>>>>>>>>                     regular basis. So, we need to make it properly
>>>>>>>>>                     from the very beginning.
>>>>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>>>>
>>>>>>>>>                     1. Create site-specific repository (we call it
>>>>>>>>>                     ts-rigs) which contains information about test
>>>>>>>>>                     rigs and other site-specific information like
>>>>>>>>>                     where to send mails, where to store logs etc.
>>>>>>>>>                     It is required for manual execution as well,
>>>>>>>>>                     since test rigs description is essential. I'll
>>>>>>>>>                     return to the topic below.
>>>>>>>>>
>>>>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>>>>                     server with few CGI scripts which help a lot to
>>>>>>>>>                     save disk space.
>>>>>>>>>
>>>>>>>>>                     3. Setup Bublik web application which provides
>>>>>>>>>                     web interface to view testing results. Same as
>>>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>>> <https://ts-factory.io/bublik>
>>>>>>>>>
>>>>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>>>>                     save logs in log storage (2) and import it to
>>>>>>>>>                     bublik (3).
>>>>>>>>>
>>>>>>>>>                     Last few month we spent on our homework to make
>>>>>>>>>                     it simpler to bring up automated execution
>>>>>>>>>                     using Jenkins -
>>>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>>                     available tomorrow.
>>>>>>>>>
>>>>>>>>>                     Let's return to the step (1).
>>>>>>>>>
>>>>>>>>>                     Unfortunately there is no publicly available
>>>>>>>>>                     example of the ts-rigs repository since
>>>>>>>>>                     sensitive site-specific information is located
>>>>>>>>>                     there. But I'm ready to help you to create it
>>>>>>>>>                     for UNH. I see two options here:
>>>>>>>>>
>>>>>>>>>                     (A) I'll ask questions and based on your
>>>>>>>>>                     answers will create the first draft with my
>>>>>>>>>                     comments.
>>>>>>>>>
>>>>>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>>>>>                     on it.
>>>>>>>>>
>>>>>>>>>                     Of course, I'll help to debug and finally bring
>>>>>>>>>                     it up in any case.
>>>>>>>>>
>>>>>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>>>>>                     a bit more generic and will help other
>>>>>>>>>                     potential users to bring it up.
>>>>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>>>>                     What do you think?
>>>>>>>>>
>>>>>>>>>                     Thanks,
>>>>>>>>>                     Andrew.
>>>>>>>>>
>>>>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>>
>>>>>>>>>                     Greetings Adam,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>>>                     would be happy to help
>>>>>>>>>
>>>>>>>>>                     Thanks,
>>>>>>>>>                     Konstantin
>>>>>>>>>
>>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>>>>
>>>>>>>>>                     
>>>>>>>>>                     Greetings Konstantin,
>>>>>>>>>
>>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>>>
>>>>>>>>>                     I have some questions about how to set the
>>>>>>>>>                     test suite arguments.
>>>>>>>>>
>>>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>>                     fails to run.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>
>>>>>>>>>                     The documentation mentions that there are
>>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>>>                     to run many of the tests.
>>>>>>>>>
>>>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>>>                     are instructions on how to create new
>>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>>                     some other way.
>>>>>>>>>
>>>>>>>>>                     How should these values be set?
>>>>>>>>>
>>>>>>>>>                     I'm also not sure what environment
>>>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>>>                     which are optional.
>>>>>>>>>
>>>>>>>>>                     Regards,
>>>>>>>>>                     Adam
>>>>>>>>>
>>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>>                     Senior Developer
>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                 --                 *Adam Hassick*
>>>>>>>>>                 Senior Developer
>>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>             --             *Adam Hassick*
>>>>>>>>>             Senior Developer
>>>>>>>>>             UNH InterOperability Lab
>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>             +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         --         *Adam Hassick*
>>>>>>>>>         Senior Developer
>>>>>>>>>         UNH InterOperability Lab
>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>     --     *Adam Hassick*
>>>>>>>>>     Senior Developer
>>>>>>>>>     UNH InterOperability Lab
>>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>     +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Adam Hassick*
>>>>>>>>> Senior Developer
>>>>>>>>> UNH InterOperability Lab
>>>>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Adam Hassick*
>>>>>>>> Senior Developer
>>>>>>>> UNH InterOperability Lab
>>>>>>>> ahassick@iol.unh.edu
>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Adam Hassick*
>>>>>>> Senior Developer
>>>>>>> UNH InterOperability Lab
>>>>>>> ahassick@iol.unh.edu
>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Adam Hassick*
>>>>>> Senior Developer
>>>>>> UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu
>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 191862 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 138198 bytes --]

Hi Adam,

 > Do these default to vfio-pci?

Yes, vfio-pci is the default.
However, it does not work in the case of Mellanox which uses bifurcated 
driver. It should mlx5_core for Mellanox NICs.

 > Here is the text log from a run on our Intel XL710 NICs, with the 
expected result profile set to the X710.

It is hard to analyze all tests using text logs, but I definitely see 
one common problem. Tests receive unexpected packets and fail because of it.
Tests are written very strict from this point of view and it brought 
fruits in the past when HW had bugs.
Are DUT and tester connected back-to-back on tested interfaces or via 
switch?
If via switch, is it possible to isolate it from everything else?
If back-to-back, it could be some embedded SW/FW which regenerates these 
packet.
I definitely see unexpected DHCP packets.

 > We haven't set up the Jenkins integration yet, however if this is 
required to import the logs then we will prioritize that.

Unfortunately manual runs do not generate all artifacts required to 
import logs. However, we have almost solved it right now. Hopefully 
we'll finalize it in a day or two. I'll let you know when these changes 
are available.

Regards,
Andrew.

On 10/4/23 16:48, Adam Hassick wrote:
> Hi Andrew,
>
> Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set 
> anywhere in the default configurations for the Intel X710. Do these 
> default to vfio-pci?
> We have IOMMU enabled on our development testbed, and should be able 
> to bind vfio-pci.
> Here is the text log from a run on our Intel XL710 NICs, with the 
> expected result profile set to the X710. We haven't set up the Jenkins 
> integration yet, however if this is required to import the logs then 
> we will prioritize that.
> log.txt.tar.gz 
> <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>
> Thanks,
> Adam
>
> On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     On 9/18/23 17:44, Adam Hassick wrote:
>>     Hi Andrew and Konstantin,
>>
>>     Thank you for adding the tester-dial feature, this opens up the
>>     possibility for us to do CI integrated testing in the future.
>>
>>     Our Mellanox pass rate is similar to yours (about ~2400 passing,
>>     ~4400 failing), however our Intel pass rates are far worse.
>>     I will try running tests on the XL710 with the trc-tags argument
>>     set and see if it improves the pass rate.
>>     Another thing I noticed in the results you uploaded is that the
>>     results are tagged with vfio-pci and not i40e.
>>     Though in the environment dump, the driver on the test machine
>>     and the DUT are set to use the i40e driver. Is this important at all?
>
>     I think it is a misunderstanding here. There are two kinds of
>     driver in configuration: net driver and so-called DPDK driver.
>     Net driver is a Linux kernel network device driver used on Tester
>     side.
>     DPDK driver is a Linux kernel driver to bind device to to use it
>     with DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
>     In the case of bifurcated driver (like mlx5_core) it is the same
>     in both cases.
>     In non-bifurcated case DPDK driver is some UIO driver(vfio-pci,
>     uio-pci-generic or igb_uio).
>     Some expectations depend on used UIO. For example, uio-pci-generic
>     do not support many interrupts (used by usecases/rx_intr test cases).
>     That's why we care corresponding TRC tag.
>
>     TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's Intel
>     case. Or uio-pci-generic if IOMMU is turned off on corresponding
>     machines and Linux distro does not support VFIO no IOMMU mode.
>
>     Andrew.
>
>>     There isn't anything preventing us from pushing our results up to
>>     the existing Bublik instance running at ts-factory.io
>>     <http://ts-factory.io> that I can think of at the moment.
>>     We will have to work out how to submit our results to your Bublik
>>     instance in a controlled and secure manner in that case.
>>     As far as I know we won't need access controls for the results
>>     themselves. I'll discuss this with Patrick and will let you know
>>     once we confirm that it's fine.
>>
>>     Thanks,
>>     Adam
>>
>>     On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>
>>>         Hi Andrew,
>>>
>>>         should we always auto-assign the tags or you don’t do it
>>>         since it slows down (by some seconds) the TE startup?
>>>
>>
>>         Tags are auto-assigned, but I guess it differs in Adam's case
>>         since NIC is a bit different. Below test will help to
>>         understand if it is the root cause of very different
>>         expectations. If pass rate will be close to mine, I'll simply
>>         update TRC database to share expectations for mine NIC and
>>         NIC used by Adam.
>>
>>>         Hi Adam,
>>>
>>>         I think I second the question from Andrew - happy to help
>>>         you with the triage so that we get to the same baseline. Do
>>>         you have a good way for us to share the logs? I.e. say
>>>         upload to ts-factory if we add strict permissions system so
>>>         it’s not publishing or any other way.
>>>
>>>         Thanks,
>>>         Konstantin
>>>
>>>
>>>         On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>
>>>             Hi Adam,
>>>
>>>             I've uploaded fresh testing results to ts-factory.io
>>>             <http://ts-factory.io> [1] to be on the same page.
>>>
>>>             I think I know why your and mine results on Intel 710
>>>             series NICs differ so much. Testing results expectations
>>>             database (dpdk-ethdev-ts/trc/*) is filled in in terms of
>>>             TRC tags.  I.e. expectations depends on TRC tags
>>>             discovered by helper scripts when testing is started.
>>>             These tags identify various aspects of what is tested.
>>>             Ideally expectations should be written in terms of root
>>>             cause of the expected behaviour. If it is a driver
>>>             expectations, driver tag should be used. If it is HW
>>>             limitation, tags with PCI IDs should be used. However,
>>>             it is not always easy to classify it correctly if you're
>>>             not involved in driver development. So, in order case
>>>             expectations for 710's Intel are filled in in terms of
>>>             PCI IDs. I guess PCI ID differ in your case and that's
>>>             why expectations filled in for my NIC do not apply to
>>>             your runs.
>>>
>>>             Just try to add the following option when you run on
>>>             your 710's Intel in order to mimic mine and see if it
>>>             helps to achieve better pass rate.
>>>             --trc-tag=pci-8086-1572
>>>
>>>             BTW, fresh TE tag v1.21.0 has improved algorithm to
>>>             choose tests for --tester-dial option. It should have
>>>             better coverage now.
>>>
>>>             Andrew.
>>>
>>>             [1]
>>>             https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>             <https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1>
>>>
>>>             On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>             Hi Adam,
>>>>
>>>>             I've pushed new TE tag v1.20.0 which supported a new
>>>>             command-line option --tester-dial=NUM where NUM is from
>>>>             0 to 100. it allows to choose percentage of tests to
>>>>             run. If you want stable set, you should pass
>>>>             --tester-random-seed=0 (or other integer). It is the
>>>>             first sketch and we have plans to improve it, but
>>>>             feedback would be welcome.
>>>>
>>>>             > Is it needed on the tester?
>>>>
>>>>             It is hard to say if it is strictly required for simple
>>>>             tests. However, it is better to update Tester as well,
>>>>             since performance tests run DPDK on Tester as well.
>>>>
>>>>             > Are there any other manual setup steps for these
>>>>             devices that I might be missing?
>>>>
>>>>             I don't remember anything else.
>>>>
>>>>             I think it is better to get down to details and take a
>>>>             look at logs. I'm ready to help with it and explain
>>>>             what's happening there. May be it will help to
>>>>             understand if it is a problem with setup/configuration.
>>>>
>>>>             Text logs are not very convenient. Ideally logs should
>>>>             be imported to bublik, however, manual runs do not
>>>>             provide all required artifacts right now (Jenkins jobs
>>>>             generate all required artifacts).
>>>>             Other option is 'tmp_raw_log' file (should be packed to
>>>>             make it smaller) which could be converted to various
>>>>             log formats.
>>>>             Would it be OK for you if I import your logs to bublik
>>>>             at ts-factory.io <http://ts-factory.io>? Or is it a
>>>>             problem that it is publicly available?
>>>>             Would it help if we add authentication and access
>>>>             control there?
>>>>
>>>>             Andrew.
>>>>
>>>>             On 9/8/23 17:57, Adam Hassick wrote:
>>>>>             Hi Andrew,
>>>>>
>>>>>             I have a couple questions about needed setup of the
>>>>>             NICs for the ethdev test suite.
>>>>>
>>>>>             Our MCX5s and XL710s are failing the checkup tests.
>>>>>             The pass rate appears to be much worse on the XL710s
>>>>>             (40 of 73 tests failed, 3 passed unexpectedly).
>>>>>
>>>>>             For the XL710s, I've updated the driver and NVM
>>>>>             versions to match the minimum supported versions in
>>>>>             the compatibility matrix found on the DPDK
>>>>>             documentation. This did not change the failure rate much.
>>>>>             For the MCX5s, I've installed the latest LTS version
>>>>>             of the OFED bifurcated driver on the DUT. Is it needed
>>>>>             on the tester?
>>>>>
>>>>>             Are there any other manual setup steps for these
>>>>>             devices that I might be missing?
>>>>>
>>>>>             Thanks,
>>>>>             Adam
>>>>>
>>>>>             On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick
>>>>>             <ahassick@iol.unh.edu> wrote:
>>>>>
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 Yes, I copied the X710 configs to set up XL710
>>>>>                 configs. I changed the environment variable names
>>>>>                 from the X710 suffix to XL710 suffix in the
>>>>>                 script, and forgot to change them in the
>>>>>                 corresponding environment file.
>>>>>                 That fixed the issue.
>>>>>
>>>>>                 I got the checkup tests working on the XL710 now.
>>>>>                 Most of them are failing, which leads me to
>>>>>                 believe this is an issue with our testbed. Based
>>>>>                 on the DPDK documentation for i40e, the firmware
>>>>>                 and driver versions are much older than what DPDK
>>>>>                 22.11 LTS and main prefer, so I'll try updating those.
>>>>>
>>>>>                 For now I'm working on getting the XL710 checkup
>>>>>                 tests passing, and will pick up getting the E810
>>>>>                 configured properly next. I'll let you know if I
>>>>>                 run into any more issues in relation to the test
>>>>>                 engine.
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>                 On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>                     Hi Adam,
>>>>>
>>>>>                     On 9/5/23 18:01, Adam Hassick wrote:
>>>>>>                     Hi Andrew,
>>>>>>
>>>>>>                     The compilation warning issue is now
>>>>>>                     resolved. Again, thank you guys for fixing
>>>>>>                     this for us. I can run the tests on the
>>>>>>                     Mellanox CX5s again, however I'm running into
>>>>>>                     a couple new issues with running the
>>>>>>                     prologues on the Intel cards.
>>>>>>
>>>>>>                     When running testing on the Intel XL710s, I
>>>>>>                     see this error appear in the log:
>>>>>>
>>>>>>                         ERROR  prologue  Environment LIB
>>>>>>                          14:16:13.650
>>>>>>                         Too few networks in available
>>>>>>                         configuration (0) in comparison with
>>>>>>                         required (1)
>>>>>>
>>>>>>
>>>>>>                     This seems like a trivial configuration
>>>>>>                     error, perhaps this is something I need to
>>>>>>                     set up in ts-rigs. I briefly searched through
>>>>>>                     the examples there and didn't see any mention
>>>>>>                     of how to set up a network.
>>>>>>                     I will attach this log just in case you need
>>>>>>                     more information.
>>>>>
>>>>>                     Unfortunately logs are insufficient to
>>>>>                     understand it. I've pushed new tag to TE
>>>>>                     v1.19.0 which add log message with TE_*
>>>>>                     environment variables.
>>>>>                     Most likely something is wrong with variables
>>>>>                     which are used as conditions when available
>>>>>                     networks are defined in
>>>>>                     ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>                     TE_PCI_INSTANCE_IUT_TST1
>>>>>                     TE_PCI_INSTANCE_IUT_TST1a
>>>>>                     TE_PCI_INSTANCE_TST1a_IUT
>>>>>                     TE_PCI_INSTANCE_TST1_IUT
>>>>>                     My guess it that you change naming a bit, but
>>>>>                     script like ts-rigs-sample/scripts/iut.h1-x710
>>>>>                     is not included or not updated.
>>>>>
>>>>>>                     There is a different error when running on
>>>>>>                     the Intel E810s. It appears to me like it
>>>>>>                     starts DPDK, does some configuration inside
>>>>>>                     DPDK and on the device, and then fails to
>>>>>>                     bring the device back up. Since this error
>>>>>>                     seems very non-trivial, I will also attach
>>>>>>                     this log.
>>>>>
>>>>>                     This one is a bit simpler. Few lines after the
>>>>>                     first ERROR in log I see the following:
>>>>>                     WARN  RCF  DPDK  13:06:00.144
>>>>>                     ice_program_hw_rx_queue(): currently package
>>>>>                     doesn't support RXDID (22)
>>>>>                     ice_rx_queue_start(): fail to program RX queue 0
>>>>>                     ice_dev_start(): fail to start Rx queue 0
>>>>>                     Device with port_id=0 already stopped
>>>>>
>>>>>                     It is stdout/stderr from test agent which runs
>>>>>                     DPDK. Same logs in plain format are available
>>>>>                     in ta.DPDK file.
>>>>>                     I'm not an expert here, but I vaguely remember
>>>>>                     that E810 requires correct firmware and DDP to
>>>>>                     be loaded.
>>>>>                     There is some information in
>>>>>                     dpdk/doc/guides/nics/ice.rst.
>>>>>
>>>>>                     You can try to add
>>>>>                     --dev-args=safe-mode-support=1 command-line
>>>>>                     option described there.
>>>>>
>>>>>                     Hope it helps,
>>>>>                     Andrew.
>>>>>
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Adam
>>>>>>
>>>>>>                     On Fri, Sep 1, 2023 at 3:59 AM Andrew
>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                         Hi Adam,
>>>>>>
>>>>>>                         On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>>                         Hi Andrew,
>>>>>>>
>>>>>>>                         I have one additional question as well:
>>>>>>>                         Does the test engine support running
>>>>>>>                         tests on two ARMv8 test agents?
>>>>>>>
>>>>>>>                             1. We'll sort out warnings this
>>>>>>>                             week. Thanks for heads up.
>>>>>>>
>>>>>>>
>>>>>>>                         Great. Let me know when that's fixed.
>>>>>>
>>>>>>                         Done. We also fixed a number of warnings
>>>>>>                         in TE.
>>>>>>                         Also we fixed root test package name to
>>>>>>                         be consistent with the repository name.
>>>>>>
>>>>>>>                             Support for old LTS branches was
>>>>>>>                             dropped some time ago, but in the
>>>>>>>                             future it is definitely possible to
>>>>>>>                             keep it for new LTS branches. I
>>>>>>>                             think 22.11 is supported, but I'm
>>>>>>>                             not sure about older LTS releases.
>>>>>>>
>>>>>>>
>>>>>>>                         Good to know.
>>>>>>>
>>>>>>>                             2. You can add command-line option
>>>>>>>                             --sanity to run tests marked with
>>>>>>>                             TEST_HARNESS_SANITY requirement (see
>>>>>>>                             dpdk-ethdev-ts/scripts/run.sh and
>>>>>>>                             grep TEST_HARNESS_SANITY
>>>>>>>                             dpdk-ethdev-ts to see which tests
>>>>>>>                             are marked). Yes, there is a space
>>>>>>>                             for terminology improvement here.
>>>>>>>                             We'll do it.
>>>>>>>
>>>>>>
>>>>>>                         Done. Now it is called --checkup.
>>>>>>
>>>>>>>
>>>>>>>                             Also it takes a lot of time because
>>>>>>>                             of failures and tests which wait for
>>>>>>>                             some timeout.
>>>>>>>
>>>>>>>
>>>>>>>                         That makes sense to me. We'll use the
>>>>>>>                         time to complete tests on virtio or the
>>>>>>>                         Intel devices as a reference for how
>>>>>>>                         long the tests really take to complete.
>>>>>>>                         We will explore the possibility of
>>>>>>>                         periodically running the sanity tests
>>>>>>>                         for patches.
>>>>>>
>>>>>>                         I'll double-check and let you know how
>>>>>>                         long entire TS runs on Intel X710, E810,
>>>>>>                         Mellanox CX5 and virtio net. Just to
>>>>>>                         ensure that time observed in your case
>>>>>>                         looks the same.
>>>>>>
>>>>>>>
>>>>>>>                             The test harness can provide
>>>>>>>                             coverage reports based on gcov, but
>>>>>>>                             I'm not sure what you mean by a
>>>>>>>                             "dial" to control test coverage.
>>>>>>>                             Provided reports are rather for
>>>>>>>                             human to analyze.
>>>>>>>
>>>>>>>
>>>>>>>                         The general idea is to have some kind of
>>>>>>>                         parameter on the test suite, which could
>>>>>>>                         be an integer ranging from zero to ten,
>>>>>>>                         that controls how many tests are run
>>>>>>>                         based on how important the test is.
>>>>>>>
>>>>>>>                         Similar to how some command line
>>>>>>>                         interfaces provide a verbosity level
>>>>>>>                         parameter (some number of "-v"
>>>>>>>                         arguments) to control the importance of
>>>>>>>                         the information in the log.
>>>>>>>                         The verbosity level zero only prints
>>>>>>>                         very important log messages, while ten
>>>>>>>                         prints everything.
>>>>>>>
>>>>>>>                         In much the same manner as above, this
>>>>>>>                         "dial" parameter controls what tests are
>>>>>>>                         run and with what parameters based on
>>>>>>>                         how important those tests and test
>>>>>>>                         parameter combinations are.
>>>>>>>                         Coverage Level zero tells the suite to
>>>>>>>                         run a very basic set of important tests,
>>>>>>>                         with minimal parameterization. This mode
>>>>>>>                         would take only ~5-10 minutes to run.
>>>>>>>                         In contrast, Coverage Level ten includes
>>>>>>>                         all the edge cases, every combination of
>>>>>>>                         test parameters, everything the test
>>>>>>>                         suite can do, which takes the normal
>>>>>>>                         several hours to run.
>>>>>>>                         The values 1 - 9 are between those two
>>>>>>>                         extremes, allowing the user to get a
>>>>>>>                         gradient of test coverage in the results
>>>>>>>                         and to limit the running time.
>>>>>>>
>>>>>>>                         Then we could, for example, run the
>>>>>>>                         "run.sh" with a level of 2 or 3 for
>>>>>>>                         incoming patches that need quick
>>>>>>>                         results, and with a level of 10 for the
>>>>>>>                         less often run periodic tests performed
>>>>>>>                         on main or LTS branches.
>>>>>>
>>>>>>                         Understood now. Thanks a lot for the
>>>>>>                         idea. We'll discuss it and come back.
>>>>>>
>>>>>>>                             3. Yes, really many tests on
>>>>>>>                             Mellanox CX5 NICs report unexpected
>>>>>>>                             testing results. Unfortunately it is
>>>>>>>                             time consuming to fill in
>>>>>>>                             expectations database since it is
>>>>>>>                             necessary to analyze testing results
>>>>>>>                             and classify if it is a bug or just
>>>>>>>                             acceptable behaviour aspect.
>>>>>>>
>>>>>>>                             Bublik allows to compare results of
>>>>>>>                             two runs. It is useful for human,
>>>>>>>                             but still not good for automation.
>>>>>>>
>>>>>>>                             I have local patch for mlx5 driver
>>>>>>>                             which reports Tx ring size maximum.
>>>>>>>                             It makes pass rate higher. It is a
>>>>>>>                             problem for test harness that mlx5
>>>>>>>                             does not report limits right now.
>>>>>>>
>>>>>>>                             Pass rate on Intel X710 is about 92%
>>>>>>>                             on my test rig. Pass rate on virtio
>>>>>>>                             net is 99% right now and could be
>>>>>>>                             done 100% easily (just one thing to
>>>>>>>                             fix in expectations).
>>>>>>>
>>>>>>>                             I think logs storage setup is
>>>>>>>                             essential for logs analysis. Of
>>>>>>>                             course, you can request HTML logs
>>>>>>>                             when you run tests (--log-html=html)
>>>>>>>                             or generate after run using
>>>>>>>                             dpdk-ethdev-ts/scripts/html-log.sh
>>>>>>>                             and open index.html in a browser,
>>>>>>>                             but logs storage makes it more
>>>>>>>                             convenient.
>>>>>>>
>>>>>>>
>>>>>>>                         We are interested in setting up Bublik,
>>>>>>>                         potentially as an externally-facing
>>>>>>>                         component, once we have our process of
>>>>>>>                         running the test suite stabilized.
>>>>>>>                         Once we are able to run the test suite
>>>>>>>                         again, I'll see what the pass rate is on
>>>>>>>                         our other hardware.
>>>>>>>                         Good to know that it isn't an issue with
>>>>>>>                         our dev testbed causing the high fail rate.
>>>>>>>
>>>>>>>                         For Intel hardware, we have an XL710 and
>>>>>>>                         an Intel E810-C in our development
>>>>>>>                         testbed. Although they are slightly
>>>>>>>                         different devices, ideally the pass rate
>>>>>>>                         will be identical or similar. I have yet
>>>>>>>                         to set up a VM pair for virtio, but we
>>>>>>>                         will soon.
>>>>>>>
>>>>>>>                             Latest version of test-environment
>>>>>>>                             has examples of our CGI scripts
>>>>>>>                             which we use for log storage (see
>>>>>>>                             tools/log_server/README.md).
>>>>>>>
>>>>>>>                             Also all bits for Jenkins setup are
>>>>>>>                             available. See
>>>>>>>                             dpdk-ethdev-ts/jenkins/README.md and
>>>>>>>                             examples of jenkins files in
>>>>>>>                             ts-rigs-sample.
>>>>>>>
>>>>>>>
>>>>>>>                         Jenkins integration, setting up
>>>>>>>                         production rig configurations, and
>>>>>>>                         permanent log storage will be our next
>>>>>>>                         steps once I am able to run the tests again.
>>>>>>>                         Unless there is an easy way to have
>>>>>>>                         meson not pass "-Werror" into GCC. Then
>>>>>>>                         I would be able to run the test suite.
>>>>>>
>>>>>>                         Hopefully it is resolved now.
>>>>>>
>>>>>>                         I thought a bit more about your usecase
>>>>>>                         for Jenkins. I'm not 100% sure that
>>>>>>                         existing pipelines are convenient for
>>>>>>                         your usecase.
>>>>>>                         Fill free to ask questions when you are
>>>>>>                         on it.
>>>>>>
>>>>>>                         Thanks,
>>>>>>                         Andrew.
>>>>>>
>>>>>>>
>>>>>>>                         Thanks,
>>>>>>>                         Adam
>>>>>>>
>>>>>>>
>>>>>>>                             On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>>>                             Hi Andrew,
>>>>>>>>
>>>>>>>>                             That fix seems to have resolved the
>>>>>>>>                             issue, thanks for the quick
>>>>>>>>                             turnaround time on that patch.
>>>>>>>>                             Now that we have the RCF timeout
>>>>>>>>                             issue resolved, there are a few
>>>>>>>>                             other questions and issues that we
>>>>>>>>                             have about the tests themselves.
>>>>>>>>
>>>>>>>>                             1. The test suite fails to build
>>>>>>>>                             with a couple warnings.
>>>>>>>>
>>>>>>>>                             Below is the stderr log from
>>>>>>>>                             compilation:
>>>>>>>>
>>>>>>>>                                 FAILED:
>>>>>>>>                                 lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>                                 cc
>>>>>>>>                                 -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>                                 -Ilib -I../../lib
>>>>>>>>                                 -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>                                 -fdiagnostics-color=always
>>>>>>>>                                 -pipe -D_FILE_OFFSET_BITS=64
>>>>>>>>                                 -Wall -Winvalid-pch -Werror -g
>>>>>>>>                                 -D_GNU_SOURCE -O0 -ggdb -Wall
>>>>>>>>                                 -W -fPIC -MD -MQ
>>>>>>>>                                 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>                                 -MF
>>>>>>>>                                 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>                                 -o
>>>>>>>>                                 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>                                 -c ../../lib/dpdk_pmd_ts.c
>>>>>>>>                                 ../../lib/dpdk_pmd_ts.c: In
>>>>>>>>                                 function
>>>>>>>>                                 ‘test_create_traffic_generator_params’:
>>>>>>>>                                 ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>                                 error: format not a string
>>>>>>>>                                 literal and no format arguments
>>>>>>>>                                 [-Werror=format-security]
>>>>>>>>                                 5577 |     rc =
>>>>>>>>                                 te_kvpair_add(result, buf, mode);
>>>>>>>>                                 |     ^~
>>>>>>>>                                 cc1: all warnings being treated
>>>>>>>>                                 as errors
>>>>>>>>                                 ninja: build stopped:
>>>>>>>>                                 subcommand failed.
>>>>>>>>                                 ninja: Entering directory `.'
>>>>>>>>                                 FAILED:
>>>>>>>>                                 lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>                                 cc
>>>>>>>>                                 -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>                                 -Ilib -I../../lib
>>>>>>>>                                 -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>                                 -fdiagnostics-color=always
>>>>>>>>                                 -pipe -D_FILE_OFFSET_BITS=64
>>>>>>>>                                 -Wall -Winvalid-pch -Werror -g
>>>>>>>>                                 -D_GNU_SOURCE -O0 -ggdb -Wall
>>>>>>>>                                 -W -fPIC -MD -MQ
>>>>>>>>                                 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>                                 -MF
>>>>>>>>                                 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>                                 -o
>>>>>>>>                                 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>                                 -c ../../lib/dpdk_pmd_ts.c
>>>>>>>>                                 ../../lib/dpdk_pmd_ts.c: In
>>>>>>>>                                 function
>>>>>>>>                                 ‘test_create_traffic_generator_params’:
>>>>>>>>                                 ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>                                 error: format not a string
>>>>>>>>                                 literal and no format arguments
>>>>>>>>                                 [-Werror=format-security]
>>>>>>>>                                 5577 |     rc =
>>>>>>>>                                 te_kvpair_add(result, buf, mode);
>>>>>>>>                                 |     ^~
>>>>>>>>                                 cc1: all warnings being treated
>>>>>>>>                                 as errors
>>>>>>>>
>>>>>>>>
>>>>>>>>                             This error wasn't occurring last
>>>>>>>>                             week, which was the last time I ran
>>>>>>>>                             the tests.
>>>>>>>>                             The TE host and the DUT have GCC
>>>>>>>>                             v9.4.0 installed, and the tester
>>>>>>>>                             has GCC v11.4.0 installed, if this
>>>>>>>>                             information is helpful.
>>>>>>>>
>>>>>>>>                             2. On the Mellanox CX5s, there are
>>>>>>>>                             over 6,000 tests run, which
>>>>>>>>                             collectively take around 9 hours.
>>>>>>>>                             Is it possible, and would it make
>>>>>>>>                             sense, to lower the test coverage
>>>>>>>>                             and have the test suite run faster?
>>>>>>>>
>>>>>>>>                             For some context, we run immediate
>>>>>>>>                             testing on incoming patches for
>>>>>>>>                             DPDK main and development branches,
>>>>>>>>                             as well as periodic test runs on
>>>>>>>>                             the main, stable, and LTS branches.
>>>>>>>>                             For us to consider including this
>>>>>>>>                             test suite as part of our immediate
>>>>>>>>                             testing on patches, we would have
>>>>>>>>                             to reduce the test coverage to the
>>>>>>>>                             most important tests.
>>>>>>>>                             This is primarily to reduce the
>>>>>>>>                             testing time to, for example, less
>>>>>>>>                             than 30 minutes. Testing on patches
>>>>>>>>                             can't take too long because the lab
>>>>>>>>                             can receive numerous patches each
>>>>>>>>                             day, which each require individual
>>>>>>>>                             testing runs.
>>>>>>>>
>>>>>>>>                             At what frequency we run these
>>>>>>>>                             tests, and on what, still needs to
>>>>>>>>                             be discussed with the DPDK
>>>>>>>>                             community, but it would be nice to
>>>>>>>>                             know if the test suite had a "dial"
>>>>>>>>                             to control the testing coverage.
>>>>>>>>
>>>>>>>>                             3. We see a lot of test failures on
>>>>>>>>                             our Mellanox CX5 NICs. Around 2,300
>>>>>>>>                             of ~6,600 tests passed. Is there
>>>>>>>>                             anything we can do to diagnose
>>>>>>>>                             these test failures?
>>>>>>>>
>>>>>>>>                             Thanks,
>>>>>>>>                             Adam
>>>>>>>>
>>>>>>>>
>>>>>>>>                             On Tue, Aug 29, 2023 at 8:07 AM
>>>>>>>>                             Andrew Rybchenko
>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>                                 Hi Adam,
>>>>>>>>
>>>>>>>>                                 I've pushed the fix in main
>>>>>>>>                                 branch and a new tag v1.18.1.
>>>>>>>>                                 It should solve the problem
>>>>>>>>                                 with IPv6 address from DNS.
>>>>>>>>
>>>>>>>>                                 Andrew.
>>>>>>>>
>>>>>>>>                                 On 8/29/23 00:05, Andrew
>>>>>>>>                                 Rybchenko wrote:
>>>>>>>>>                                 Hi Adam,
>>>>>>>>>
>>>>>>>>>                                 > Does the test engine prefer
>>>>>>>>>                                 to use IPv6 over IPv4 for
>>>>>>>>>                                 initiating the RCF connection
>>>>>>>>>                                 to the test bed hosts? And if
>>>>>>>>>                                 so, is there a way to force it
>>>>>>>>>                                 to use IPv4?
>>>>>>>>>
>>>>>>>>>                                 Brilliant idea. If DNS returns
>>>>>>>>>                                 both IPv4 and IPv6 addresses
>>>>>>>>>                                 in your case, I guess it is
>>>>>>>>>                                 the root cause of the problem.
>>>>>>>>>                                 Of course, it is TE problem
>>>>>>>>>                                 since I see really weird code
>>>>>>>>>                                 in
>>>>>>>>>                                 lib/comm_net_engine/comm_net_engine.c
>>>>>>>>>                                 line 135.
>>>>>>>>>
>>>>>>>>>                                 I've pushed fix to the branch
>>>>>>>>>                                 user/arybchik/fix_ipv4_only in
>>>>>>>>>                                 ts-factory/test-environment
>>>>>>>>>                                 repository. Please, try.
>>>>>>>>>
>>>>>>>>>                                 It is late night fix with
>>>>>>>>>                                 minimal testing and no review.
>>>>>>>>>                                 I'll pass it through review
>>>>>>>>>                                 process tomorrow and
>>>>>>>>>                                 hopefully it will be released
>>>>>>>>>                                 in one-two days.
>>>>>>>>>
>>>>>>>>>                                 Andrew.
>>>>>>>>>
>>>>>>>>>                                 On 8/28/23 18:02, Adam Hassick
>>>>>>>>>                                 wrote:
>>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                                 We have yet to notice a
>>>>>>>>>>                                 distinct pattern with the
>>>>>>>>>>                                 failures. Sometimes, the RCF
>>>>>>>>>>                                 will start and connect
>>>>>>>>>>                                 without issue a few times in
>>>>>>>>>>                                 a row before failing to
>>>>>>>>>>                                 connect again. Once the issue
>>>>>>>>>>                                 begins to occur, neither
>>>>>>>>>>                                 rebooting all of the hosts
>>>>>>>>>>                                 (test engine VM, tester, IUT)
>>>>>>>>>>                                 or deleting all of the build
>>>>>>>>>>                                 directories (suites, agents,
>>>>>>>>>>                                 inst) and rebooting the hosts
>>>>>>>>>>                                 afterward resolves the issue.
>>>>>>>>>>                                 When it begins working again
>>>>>>>>>>                                 seems very arbitrary to us.
>>>>>>>>>>
>>>>>>>>>>                                 I do usually try to terminate
>>>>>>>>>>                                 the test engine with Ctrl+C,
>>>>>>>>>>                                 but when it hangs while
>>>>>>>>>>                                 trying to start RCF, that
>>>>>>>>>>                                 does not work.
>>>>>>>>>>
>>>>>>>>>>                                 Does the test engine prefer
>>>>>>>>>>                                 to use IPv6 over IPv4 for
>>>>>>>>>>                                 initiating the RCF connection
>>>>>>>>>>                                 to the test bed hosts? And if
>>>>>>>>>>                                 so, is there a way to force
>>>>>>>>>>                                 it to use IPv4?
>>>>>>>>>>
>>>>>>>>>>                                  - Adam
>>>>>>>>>>
>>>>>>>>>>                                 On Fri, Aug 25, 2023 at
>>>>>>>>>>                                 1:35 PM Andrew Rybchenko
>>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>                                 wrote:
>>>>>>>>>>
>>>>>>>>>>                                     > I'll double-check test
>>>>>>>>>>                                     engine on Ubuntu 20.04
>>>>>>>>>>                                     and Ubuntu 22.04.
>>>>>>>>>>
>>>>>>>>>>                                     Done. It works fine for
>>>>>>>>>>                                     me without any issues.
>>>>>>>>>>
>>>>>>>>>>                                     Have you noticed any
>>>>>>>>>>                                     pattern when it works or
>>>>>>>>>>                                     does not work?
>>>>>>>>>>                                     May be it is a problem of
>>>>>>>>>>                                     not clean state after
>>>>>>>>>>                                     termination?
>>>>>>>>>>                                     Does it work fine the
>>>>>>>>>>                                     first time after DUTs reboot?
>>>>>>>>>>                                     How do you terminate
>>>>>>>>>>                                     testing? It should be
>>>>>>>>>>                                     done using Ctrl+C in
>>>>>>>>>>                                     terminal where you
>>>>>>>>>>                                     execute run.sh command.
>>>>>>>>>>                                      In this case it should
>>>>>>>>>>                                     shutdown gracefully and
>>>>>>>>>>                                     close all test agents and
>>>>>>>>>>                                     engine applications.
>>>>>>>>>>
>>>>>>>>>>                                     (I'm trying to understand
>>>>>>>>>>                                     why you've seen many test
>>>>>>>>>>                                     agent processes. It
>>>>>>>>>>                                     should not happen.)
>>>>>>>>>>
>>>>>>>>>>                                     Andrew.
>>>>>>>>>>
>>>>>>>>>>                                     On 8/25/23 17:41, Andrew
>>>>>>>>>>                                     Rybchenko wrote:
>>>>>>>>>>>                                     On 8/25/23 17:06, Adam
>>>>>>>>>>>                                     Hassick wrote:
>>>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Two of our systems (the
>>>>>>>>>>>>                                     Test Engine runner and
>>>>>>>>>>>>                                     the DUT host) are
>>>>>>>>>>>>                                     running Ubuntu 20.04
>>>>>>>>>>>>                                     LTS, however this
>>>>>>>>>>>>                                     morning I noticed that
>>>>>>>>>>>>                                     the tester system (the
>>>>>>>>>>>>                                     one having issues) is
>>>>>>>>>>>>                                     running Ubuntu 22.04 LTS.
>>>>>>>>>>>>                                     This could be the
>>>>>>>>>>>>                                     source of the problem.
>>>>>>>>>>>>                                     I encountered a
>>>>>>>>>>>>                                     dependency issue trying
>>>>>>>>>>>>                                     to run the Test Engine
>>>>>>>>>>>>                                     on 22.04 LTS, so I
>>>>>>>>>>>>                                     downgraded the system.
>>>>>>>>>>>>                                     Since the tester is
>>>>>>>>>>>>                                     also the host having
>>>>>>>>>>>>                                     connection issues, I
>>>>>>>>>>>>                                     will try downgrading
>>>>>>>>>>>>                                     that system to 20.04,
>>>>>>>>>>>>                                     and see if that changes
>>>>>>>>>>>>                                     anything.
>>>>>>>>>>>
>>>>>>>>>>>                                     Unlikely, but who knows.
>>>>>>>>>>>                                     We run tests (DUTs) on
>>>>>>>>>>>                                     Ubuntu 20.04, Ubuntu
>>>>>>>>>>>                                     22.04, Ubuntu 22.10,
>>>>>>>>>>>                                     Ubuntu 23.04, Debian 11
>>>>>>>>>>>                                     and Fedora 38 every night.
>>>>>>>>>>>                                     Right now Debian 11 is
>>>>>>>>>>>                                     used for test engine in
>>>>>>>>>>>                                     nightly regressions.
>>>>>>>>>>>
>>>>>>>>>>>                                     I'll double-check test
>>>>>>>>>>>                                     engine on Ubuntu 20.04
>>>>>>>>>>>                                     and Ubuntu 22.04.
>>>>>>>>>>>
>>>>>>>>>>>>                                     I did try passing in
>>>>>>>>>>>>                                     the "--vg-rcf" argument
>>>>>>>>>>>>                                     to the run.sh script of
>>>>>>>>>>>>                                     the test suite after
>>>>>>>>>>>>                                     installing valgrind,
>>>>>>>>>>>>                                     but there was no
>>>>>>>>>>>>                                     additional output that
>>>>>>>>>>>>                                     I saw.
>>>>>>>>>>>
>>>>>>>>>>>                                     Sorry, I should valgrind
>>>>>>>>>>>                                     output should be in
>>>>>>>>>>>                                     valgrind.te_rcf
>>>>>>>>>>>                                     (direction where you run
>>>>>>>>>>>                                     test engine).
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                     I will try pulling in
>>>>>>>>>>>>                                     the changes you've
>>>>>>>>>>>>                                     pushed up, and will see
>>>>>>>>>>>>                                     if that fixes anything.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On Fri, Aug 25, 2023 at
>>>>>>>>>>>>                                     9:57 AM Andrew
>>>>>>>>>>>>                                     Rybchenko
>>>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                         Hello Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                         On 8/24/23 23:54,
>>>>>>>>>>>>                                         Andrew Rybchenko wrote:
>>>>>>>>>>>>>                                         I'd like to try to
>>>>>>>>>>>>>                                         repeat the problem
>>>>>>>>>>>>>                                         locally. Which
>>>>>>>>>>>>>                                         Linux distro is
>>>>>>>>>>>>>                                         running on test
>>>>>>>>>>>>>                                         engine and agents?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         In fact I know one
>>>>>>>>>>>>>                                         problem with
>>>>>>>>>>>>>                                         Debian 12 and
>>>>>>>>>>>>>                                         Fedora 38 and we have
>>>>>>>>>>>>>                                         patch in review to
>>>>>>>>>>>>>                                         fix it, however,
>>>>>>>>>>>>>                                         the behaviour is
>>>>>>>>>>>>>                                         different in
>>>>>>>>>>>>>                                         this case, so it
>>>>>>>>>>>>>                                         is unlike the same
>>>>>>>>>>>>>                                         problem.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         I've just published
>>>>>>>>>>>>                                         a new tag which
>>>>>>>>>>>>                                         fixes known test
>>>>>>>>>>>>                                         engine side
>>>>>>>>>>>>                                         problems on Debian
>>>>>>>>>>>>                                         12 and Fedora 38.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         One more idea is
>>>>>>>>>>>>>                                         to install
>>>>>>>>>>>>>                                         valgrind on the
>>>>>>>>>>>>>                                         test engine host and
>>>>>>>>>>>>>                                         run with option
>>>>>>>>>>>>>                                         --vg-rcf to check
>>>>>>>>>>>>>                                         if something weird
>>>>>>>>>>>>>                                         is happening.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         What I don't
>>>>>>>>>>>>>                                         understand right
>>>>>>>>>>>>>                                         now is why I see
>>>>>>>>>>>>>                                         just one failed
>>>>>>>>>>>>>                                         attempt
>>>>>>>>>>>>>                                         to connect in your
>>>>>>>>>>>>>                                         log.txt and then
>>>>>>>>>>>>>                                         Logger shutdown
>>>>>>>>>>>>>                                         after 9 minutes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         On 8/24/23 23:29,
>>>>>>>>>>>>>                                         Adam Hassick wrote:
>>>>>>>>>>>>>>                                          > Is there any
>>>>>>>>>>>>>>                                         firewall in the
>>>>>>>>>>>>>>                                         network or on
>>>>>>>>>>>>>>                                         test hosts which
>>>>>>>>>>>>>>                                         could block
>>>>>>>>>>>>>>                                         incoming TCP
>>>>>>>>>>>>>>                                         connection to the
>>>>>>>>>>>>>>                                         port 23571
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         from the host
>>>>>>>>>>>>>>                                         where you run
>>>>>>>>>>>>>>                                         test engine?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         Our test engine
>>>>>>>>>>>>>>                                         host and the
>>>>>>>>>>>>>>                                         testbed are on
>>>>>>>>>>>>>>                                         the same subnet.
>>>>>>>>>>>>>>                                         The connection
>>>>>>>>>>>>>>                                         does work sometimes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                          > If behaviour
>>>>>>>>>>>>>>                                         the same on the
>>>>>>>>>>>>>>                                         next try and you
>>>>>>>>>>>>>>                                         see that test
>>>>>>>>>>>>>>                                         agent is kept
>>>>>>>>>>>>>>                                         running, could
>>>>>>>>>>>>>>                                         you check using
>>>>>>>>>>>>>>                                          >
>>>>>>>>>>>>>>                                          > # netstat -tnlp
>>>>>>>>>>>>>>                                          >
>>>>>>>>>>>>>>                                          > that Test
>>>>>>>>>>>>>>                                         Agent is
>>>>>>>>>>>>>>                                         listening on the
>>>>>>>>>>>>>>                                         port and try to
>>>>>>>>>>>>>>                                         establish TCP
>>>>>>>>>>>>>>                                         connection from
>>>>>>>>>>>>>>                                         test agent using
>>>>>>>>>>>>>>                                          >
>>>>>>>>>>>>>>                                          > $ telnet
>>>>>>>>>>>>>>                                         iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         23571
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                          >
>>>>>>>>>>>>>>                                          > and check if
>>>>>>>>>>>>>>                                         TCP connection
>>>>>>>>>>>>>>                                         could be established.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         I was able to
>>>>>>>>>>>>>>                                         replicate the
>>>>>>>>>>>>>>                                         same behavior
>>>>>>>>>>>>>>                                         again, where it
>>>>>>>>>>>>>>                                         hangs while RCF
>>>>>>>>>>>>>>                                         is trying to start.
>>>>>>>>>>>>>>                                         Running this
>>>>>>>>>>>>>>                                         command, I see
>>>>>>>>>>>>>>                                         this in the output:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         tcp        0    0
>>>>>>>>>>>>>>                                         0.0.0.0:23571
>>>>>>>>>>>>>>                                         <http://0.0.0.0:23571>
>>>>>>>>>>>>>>                                         <http://0.0.0.0:23571>
>>>>>>>>>>>>>>                                         <http://0.0.0.0:23571>
>>>>>>>>>>>>>>                                         0.0.0.0:* LISTEN
>>>>>>>>>>>>>>                                          18599/ta
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         So it seems like
>>>>>>>>>>>>>>                                         it is listening
>>>>>>>>>>>>>>                                         on the correct port.
>>>>>>>>>>>>>>                                         Additionally, I
>>>>>>>>>>>>>>                                         was able to
>>>>>>>>>>>>>>                                         connect to the
>>>>>>>>>>>>>>                                         Tester machine
>>>>>>>>>>>>>>                                         from our Test
>>>>>>>>>>>>>>                                         Engine host using
>>>>>>>>>>>>>>                                         telnet. It
>>>>>>>>>>>>>>                                         printed the PID
>>>>>>>>>>>>>>                                         of the process
>>>>>>>>>>>>>>                                         once the
>>>>>>>>>>>>>>                                         connection was
>>>>>>>>>>>>>>                                         opened.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         I tried running
>>>>>>>>>>>>>>                                         the "ta"
>>>>>>>>>>>>>>                                         application
>>>>>>>>>>>>>>                                         manually on the
>>>>>>>>>>>>>>                                         command line, and
>>>>>>>>>>>>>>                                         it didn't print
>>>>>>>>>>>>>>                                         anything at all.
>>>>>>>>>>>>>>                                         Maybe the issue
>>>>>>>>>>>>>>                                         is something on
>>>>>>>>>>>>>>                                         the Test Engine side.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         On Thu, Aug 24,
>>>>>>>>>>>>>>                                         2023 at 2:35 PM
>>>>>>>>>>>>>>                                         Andrew Rybchenko
>>>>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                              > On the
>>>>>>>>>>>>>>                                         tester host
>>>>>>>>>>>>>>                                         (which appears to
>>>>>>>>>>>>>>                                         be the Peer
>>>>>>>>>>>>>>                                         agent), there
>>>>>>>>>>>>>>                                             are four
>>>>>>>>>>>>>>                                         processes that I
>>>>>>>>>>>>>>                                         see running,
>>>>>>>>>>>>>>                                         which look like
>>>>>>>>>>>>>>                                         the test
>>>>>>>>>>>>>>                                             agent processes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Before the
>>>>>>>>>>>>>>                                         next try I'd
>>>>>>>>>>>>>>                                         recommend to kill
>>>>>>>>>>>>>>                                         these processes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Is there any
>>>>>>>>>>>>>>                                         firewall in the
>>>>>>>>>>>>>>                                         network or on
>>>>>>>>>>>>>>                                         test hosts which
>>>>>>>>>>>>>>                                         could
>>>>>>>>>>>>>>                                             block
>>>>>>>>>>>>>>                                         incoming TCP
>>>>>>>>>>>>>>                                         connection to the
>>>>>>>>>>>>>>                                         port 23571
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         from the host
>>>>>>>>>>>>>>                                             where you run
>>>>>>>>>>>>>>                                         test engine?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             If behaviour
>>>>>>>>>>>>>>                                         the same on the
>>>>>>>>>>>>>>                                         next try and you
>>>>>>>>>>>>>>                                         see that test
>>>>>>>>>>>>>>                                         agent is
>>>>>>>>>>>>>>                                             kept running,
>>>>>>>>>>>>>>                                         could you check using
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             # netstat -tnlp
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             that Test
>>>>>>>>>>>>>>                                         Agent is
>>>>>>>>>>>>>>                                         listening on the
>>>>>>>>>>>>>>                                         port and try to
>>>>>>>>>>>>>>                                         establish TCP
>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>                                         from test agent using
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             $ telnet
>>>>>>>>>>>>>>                                         iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         23571
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             and check if
>>>>>>>>>>>>>>                                         TCP connection
>>>>>>>>>>>>>>                                         could be established.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Another idea
>>>>>>>>>>>>>>                                         is to login
>>>>>>>>>>>>>>                                         Tester under root
>>>>>>>>>>>>>>                                         as testing does, get
>>>>>>>>>>>>>>                                             start TA
>>>>>>>>>>>>>>                                         command from the
>>>>>>>>>>>>>>                                         log and try it by
>>>>>>>>>>>>>>                                         hands without -n and
>>>>>>>>>>>>>>                                             remove extra
>>>>>>>>>>>>>>                                         escaping.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             # sudo
>>>>>>>>>>>>>>                                         PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>                                         LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>                                         /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>>>                                         Peer 23571
>>>>>>>>>>>>>>                                         host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Hopefully in
>>>>>>>>>>>>>>                                         this case test
>>>>>>>>>>>>>>                                         agent directory
>>>>>>>>>>>>>>                                         remains in the
>>>>>>>>>>>>>>                                         /tmp and
>>>>>>>>>>>>>>                                             you don't
>>>>>>>>>>>>>>                                         need to copy it
>>>>>>>>>>>>>>                                         as testing does.
>>>>>>>>>>>>>>                                             May be output
>>>>>>>>>>>>>>                                         could shed some
>>>>>>>>>>>>>>                                         light on what's
>>>>>>>>>>>>>>                                         going on.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On 8/24/23
>>>>>>>>>>>>>>                                         17:30, Adam
>>>>>>>>>>>>>>                                         Hassick wrote:
>>>>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             This is the
>>>>>>>>>>>>>>>                                         output that I
>>>>>>>>>>>>>>>                                         see in the
>>>>>>>>>>>>>>>                                         terminal when
>>>>>>>>>>>>>>>                                         this failure
>>>>>>>>>>>>>>>                                             occurs,
>>>>>>>>>>>>>>>                                         after the test
>>>>>>>>>>>>>>>                                         agent binaries
>>>>>>>>>>>>>>>                                         build and the
>>>>>>>>>>>>>>>                                         test engine
>>>>>>>>>>>>>>>                                             starts:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             Platform
>>>>>>>>>>>>>>>                                         default build - pass
>>>>>>>>>>>>>>>                                             Simple RCF
>>>>>>>>>>>>>>>                                         consistency
>>>>>>>>>>>>>>>                                         check succeeded
>>>>>>>>>>>>>>>                                         --->>> Starting
>>>>>>>>>>>>>>>                                         Logger...done
>>>>>>>>>>>>>>>                                         --->>> Starting
>>>>>>>>>>>>>>>                                         RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>>>                                         Connection timed
>>>>>>>>>>>>>>>                                             out
>>>>>>>>>>>>>>>                                         iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             Then, it
>>>>>>>>>>>>>>>                                         hangs here until
>>>>>>>>>>>>>>>                                         I kill the
>>>>>>>>>>>>>>>                                         "te_rcf" and
>>>>>>>>>>>>>>>                                         "te_tee"
>>>>>>>>>>>>>>>                                             processes. I
>>>>>>>>>>>>>>>                                         let it hang for
>>>>>>>>>>>>>>>                                         around 9 minutes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             On the
>>>>>>>>>>>>>>>                                         tester host
>>>>>>>>>>>>>>>                                         (which appears
>>>>>>>>>>>>>>>                                         to be the Peer
>>>>>>>>>>>>>>>                                         agent), there are
>>>>>>>>>>>>>>>                                             four
>>>>>>>>>>>>>>>                                         processes that I
>>>>>>>>>>>>>>>                                         see running,
>>>>>>>>>>>>>>>                                         which look like
>>>>>>>>>>>>>>>                                         the test agent
>>>>>>>>>>>>>>>                                             processes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             ta.Peer is
>>>>>>>>>>>>>>>                                         an empty file.
>>>>>>>>>>>>>>>                                         I've attached
>>>>>>>>>>>>>>>                                         the log.txt from
>>>>>>>>>>>>>>>                                         this run.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                              - Adam
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             On Thu, Aug
>>>>>>>>>>>>>>>                                         24, 2023 at
>>>>>>>>>>>>>>>                                         4:22 AM Andrew
>>>>>>>>>>>>>>>                                         Rybchenko
>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Hi Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Yes,
>>>>>>>>>>>>>>>                                         TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>                                         is in seconds.
>>>>>>>>>>>>>>>                                         I've double-checked
>>>>>>>>>>>>>>>                                                 that it
>>>>>>>>>>>>>>>                                         goes to
>>>>>>>>>>>>>>>                                         'copy_timeout'
>>>>>>>>>>>>>>>                                         in ts-conf/rcf.conf.
>>>>>>>>>>>>>>>                                         Description in
>>>>>>>>>>>>>>>                                         in
>>>>>>>>>>>>>>>                                         doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>>>                                                 says
>>>>>>>>>>>>>>>                                         that
>>>>>>>>>>>>>>>                                         copy_timeout is
>>>>>>>>>>>>>>>                                         in seconds and
>>>>>>>>>>>>>>>                                         implementation in
>>>>>>>>>>>>>>>                                         lib/rcfunix/rcfunix.c
>>>>>>>>>>>>>>>                                         passes the value
>>>>>>>>>>>>>>>                                         to select() tv_sec.
>>>>>>>>>>>>>>>                                         Theoretically
>>>>>>>>>>>>>>>                                         select() could
>>>>>>>>>>>>>>>                                         be interrupted
>>>>>>>>>>>>>>>                                         by signal, but I
>>>>>>>>>>>>>>>                                                 think it
>>>>>>>>>>>>>>>                                         is unlikely here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 I'm not
>>>>>>>>>>>>>>>                                         sure that I
>>>>>>>>>>>>>>>                                         understand what
>>>>>>>>>>>>>>>                                         do you mean by RCF
>>>>>>>>>>>>>>>                                         connection
>>>>>>>>>>>>>>>                                         timeout. Does it
>>>>>>>>>>>>>>>                                         happen on TE
>>>>>>>>>>>>>>>                                         startup when RCF
>>>>>>>>>>>>>>>                                                 starts
>>>>>>>>>>>>>>>                                         test agents. If
>>>>>>>>>>>>>>>                                         so,
>>>>>>>>>>>>>>>                                         TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>                                         could help. Or
>>>>>>>>>>>>>>>                                                 does it
>>>>>>>>>>>>>>>                                         happen when
>>>>>>>>>>>>>>>                                         tests are in
>>>>>>>>>>>>>>>                                         progress, e.g.
>>>>>>>>>>>>>>>                                         in the middle
>>>>>>>>>>>>>>>                                                 of a
>>>>>>>>>>>>>>>                                         test. If so,
>>>>>>>>>>>>>>>                                         TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>                                         is unrelated and
>>>>>>>>>>>>>>>                                         most
>>>>>>>>>>>>>>>                                                 likely
>>>>>>>>>>>>>>>                                         either host with
>>>>>>>>>>>>>>>                                         test agent dies
>>>>>>>>>>>>>>>                                         or test agent itself
>>>>>>>>>>>>>>>                                         crashes. It
>>>>>>>>>>>>>>>                                         would be easier
>>>>>>>>>>>>>>>                                         for me if
>>>>>>>>>>>>>>>                                         classify it if
>>>>>>>>>>>>>>>                                         you share
>>>>>>>>>>>>>>>                                                 text log
>>>>>>>>>>>>>>>                                         (log.txt, full
>>>>>>>>>>>>>>>                                         or just
>>>>>>>>>>>>>>>                                         corresponding
>>>>>>>>>>>>>>>                                         fragment with
>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>                                         context). Also
>>>>>>>>>>>>>>>                                         content of
>>>>>>>>>>>>>>>                                         ta.DPDK or
>>>>>>>>>>>>>>>                                         ta.Peer file
>>>>>>>>>>>>>>>                                         depending on
>>>>>>>>>>>>>>>                                         which agent has
>>>>>>>>>>>>>>>                                         problems could
>>>>>>>>>>>>>>>                                         shed some light.
>>>>>>>>>>>>>>>                                         Corresponding
>>>>>>>>>>>>>>>                                         files contain
>>>>>>>>>>>>>>>                                         stdout/stderr of
>>>>>>>>>>>>>>>                                         test agents.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>                                         8/23/23 17:45,
>>>>>>>>>>>>>>>                                         Adam Hassick wrote:
>>>>>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>                                         set up a test
>>>>>>>>>>>>>>>>                                         rig repository
>>>>>>>>>>>>>>>>                                         here, and have
>>>>>>>>>>>>>>>>                                         created
>>>>>>>>>>>>>>>>                                         configurations
>>>>>>>>>>>>>>>>                                         for our
>>>>>>>>>>>>>>>>                                         development
>>>>>>>>>>>>>>>>                                         testbed based
>>>>>>>>>>>>>>>>                                         off of the
>>>>>>>>>>>>>>>>                                         examples.
>>>>>>>>>>>>>>>>                                                 We've
>>>>>>>>>>>>>>>>                                         been able to
>>>>>>>>>>>>>>>>                                         get the test
>>>>>>>>>>>>>>>>                                         suite to run
>>>>>>>>>>>>>>>>                                         manually on
>>>>>>>>>>>>>>>>                                         Mellanox CX5
>>>>>>>>>>>>>>>>                                         devices once.
>>>>>>>>>>>>>>>>                                         However, we are
>>>>>>>>>>>>>>>>                                         running into an
>>>>>>>>>>>>>>>>                                         issue where,
>>>>>>>>>>>>>>>>                                         when RCF starts,
>>>>>>>>>>>>>>>>                                                 the RCF
>>>>>>>>>>>>>>>>                                         connection
>>>>>>>>>>>>>>>>                                         times out very
>>>>>>>>>>>>>>>>                                         frequently. We
>>>>>>>>>>>>>>>>                                         aren't sure
>>>>>>>>>>>>>>>>                                                 why
>>>>>>>>>>>>>>>>                                         this is the case.
>>>>>>>>>>>>>>>>                                                 It
>>>>>>>>>>>>>>>>                                         works
>>>>>>>>>>>>>>>>                                         sometimes, but
>>>>>>>>>>>>>>>>                                         most of the
>>>>>>>>>>>>>>>>                                         time when we
>>>>>>>>>>>>>>>>                                         try to run
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                         test engine, it
>>>>>>>>>>>>>>>>                                         encounters this
>>>>>>>>>>>>>>>>                                         issue.
>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>                                         tried changing
>>>>>>>>>>>>>>>>                                         the RCF port by
>>>>>>>>>>>>>>>>                                         setting
>>>>>>>>>>>>>>>>                                         "TE_RCF_PORT=<some
>>>>>>>>>>>>>>>>                                         port number>"
>>>>>>>>>>>>>>>>                                         and rebooting
>>>>>>>>>>>>>>>>                                         the testbed
>>>>>>>>>>>>>>>>                                         machines.
>>>>>>>>>>>>>>>>                                         Neither seems
>>>>>>>>>>>>>>>>                                         to fix the issue.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 It also
>>>>>>>>>>>>>>>>                                         seems like the
>>>>>>>>>>>>>>>>                                         timeout takes
>>>>>>>>>>>>>>>>                                         far longer than 60
>>>>>>>>>>>>>>>>                                         seconds, even
>>>>>>>>>>>>>>>>                                         when running
>>>>>>>>>>>>>>>>                                         "export
>>>>>>>>>>>>>>>>                                         TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>>>                                                 before
>>>>>>>>>>>>>>>>                                         I try to run
>>>>>>>>>>>>>>>>                                         the test suite.
>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>                                         assume the unit
>>>>>>>>>>>>>>>>                                         for this
>>>>>>>>>>>>>>>>                                         variable is
>>>>>>>>>>>>>>>>                                         seconds?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 On Mon,
>>>>>>>>>>>>>>>>                                         Aug 21, 2023 at
>>>>>>>>>>>>>>>>                                         10:19 AM Adam
>>>>>>>>>>>>>>>>                                         Hassick
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                         <ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>                                         Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                         Thanks, I've
>>>>>>>>>>>>>>>>                                         cloned the
>>>>>>>>>>>>>>>>                                         example
>>>>>>>>>>>>>>>>                                         repository and
>>>>>>>>>>>>>>>>                                         will start
>>>>>>>>>>>>>>>>                                         setting up a
>>>>>>>>>>>>>>>>                                         configuration
>>>>>>>>>>>>>>>>                                         for our
>>>>>>>>>>>>>>>>                                         development testbed
>>>>>>>>>>>>>>>>                                         today. I'll let
>>>>>>>>>>>>>>>>                                         you know if I
>>>>>>>>>>>>>>>>                                         run into any
>>>>>>>>>>>>>>>>                                         difficulties
>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>                                         have any questions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                      - Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>                                         Sun, Aug 20,
>>>>>>>>>>>>>>>>                                         2023 at 4:40 AM
>>>>>>>>>>>>>>>>                                         Andrew Rybchenko
>>>>>>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         Hi Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         I've published
>>>>>>>>>>>>>>>>                                         https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>>>                                         <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>>>                                         <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         Hopefully it
>>>>>>>>>>>>>>>>                                         will help to
>>>>>>>>>>>>>>>>                                         define your
>>>>>>>>>>>>>>>>                                         test rigs and
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         successfully
>>>>>>>>>>>>>>>>                                         run some tests
>>>>>>>>>>>>>>>>                                         manually. Feel
>>>>>>>>>>>>>>>>                                         free to
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         ask any
>>>>>>>>>>>>>>>>                                         questions and
>>>>>>>>>>>>>>>>                                         I'll answer
>>>>>>>>>>>>>>>>                                         here and try to
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         update
>>>>>>>>>>>>>>>>                                         documentation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         Meanwhile I'll
>>>>>>>>>>>>>>>>                                         prepare missing
>>>>>>>>>>>>>>>>                                         bits for steps
>>>>>>>>>>>>>>>>                                         (2) and
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         (3).
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         Hopefully
>>>>>>>>>>>>>>>>                                         everything is
>>>>>>>>>>>>>>>>                                         in place for
>>>>>>>>>>>>>>>>                                         step (4), but we
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         need to make
>>>>>>>>>>>>>>>>                                         steps (2) and
>>>>>>>>>>>>>>>>                                         (3) first.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>                                         On 8/18/23
>>>>>>>>>>>>>>>>                                         21:40, Andrew
>>>>>>>>>>>>>>>>                                         Rybchenko wrote:
>>>>>>>>>>>>>>>>>                                         Hi Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         > I've
>>>>>>>>>>>>>>>>>                                         conferred with
>>>>>>>>>>>>>>>>>                                         the rest of
>>>>>>>>>>>>>>>>>                                         the team, and we
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         think it would
>>>>>>>>>>>>>>>>>                                         be best to
>>>>>>>>>>>>>>>>>                                         move forward
>>>>>>>>>>>>>>>>>                                         with mainly
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         option B.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         OK, I'll
>>>>>>>>>>>>>>>>>                                         provide the
>>>>>>>>>>>>>>>>>                                         sample on
>>>>>>>>>>>>>>>>>                                         Monday for
>>>>>>>>>>>>>>>>>                                         you. It is
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         almost ready
>>>>>>>>>>>>>>>>>                                         right now, but
>>>>>>>>>>>>>>>>>                                         I need to
>>>>>>>>>>>>>>>>>                                         double-check
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         it before
>>>>>>>>>>>>>>>>>                                         publishing.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         Regards,
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                         On 8/17/23
>>>>>>>>>>>>>>>>>                                         20:03, Adam
>>>>>>>>>>>>>>>>>                                         Hassick wrote:
>>>>>>>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         I'm adding
>>>>>>>>>>>>>>>>>>                                         the CI
>>>>>>>>>>>>>>>>>>                                         mailing list
>>>>>>>>>>>>>>>>>>                                         to this
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         conversation.
>>>>>>>>>>>>>>>>>>                                         Others in the
>>>>>>>>>>>>>>>>>>                                         community
>>>>>>>>>>>>>>>>>>                                         might find
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         this
>>>>>>>>>>>>>>>>>>                                         conversation
>>>>>>>>>>>>>>>>>>                                         valuable.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         We do want to
>>>>>>>>>>>>>>>>>>                                         run testing
>>>>>>>>>>>>>>>>>>                                         on a regular
>>>>>>>>>>>>>>>>>>                                         basis. The
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         Jenkins
>>>>>>>>>>>>>>>>>>                                         integration
>>>>>>>>>>>>>>>>>>                                         will be very
>>>>>>>>>>>>>>>>>>                                         useful for us, as
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         most of our
>>>>>>>>>>>>>>>>>>                                         CI is
>>>>>>>>>>>>>>>>>>                                         orchestrated
>>>>>>>>>>>>>>>>>>                                         by Jenkins.
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         I've
>>>>>>>>>>>>>>>>>>                                         conferred
>>>>>>>>>>>>>>>>>>                                         with the rest
>>>>>>>>>>>>>>>>>>                                         of the team,
>>>>>>>>>>>>>>>>>>                                         and we
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         think it
>>>>>>>>>>>>>>>>>>                                         would be best
>>>>>>>>>>>>>>>>>>                                         to move
>>>>>>>>>>>>>>>>>>                                         forward with
>>>>>>>>>>>>>>>>>>                                         mainly
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         option B.
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         If you would
>>>>>>>>>>>>>>>>>>                                         like to know
>>>>>>>>>>>>>>>>>>                                         anything
>>>>>>>>>>>>>>>>>>                                         about our
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         testbeds that
>>>>>>>>>>>>>>>>>>                                         would help
>>>>>>>>>>>>>>>>>>                                         you with
>>>>>>>>>>>>>>>>>>                                         creating an
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         example
>>>>>>>>>>>>>>>>>>                                         ts-rigs repo,
>>>>>>>>>>>>>>>>>>                                         I'd be happy
>>>>>>>>>>>>>>>>>>                                         to answer any
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         questions you
>>>>>>>>>>>>>>>>>>                                         have.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         We have
>>>>>>>>>>>>>>>>>>                                         multiple test
>>>>>>>>>>>>>>>>>>                                         rigs (we call
>>>>>>>>>>>>>>>>>>                                         these
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         "DUT-tester
>>>>>>>>>>>>>>>>>>                                         pairs") that
>>>>>>>>>>>>>>>>>>                                         we run our
>>>>>>>>>>>>>>>>>>                                         existing
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         hardware
>>>>>>>>>>>>>>>>>>                                         testing on,
>>>>>>>>>>>>>>>>>>                                         with
>>>>>>>>>>>>>>>>>>                                         differing network
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         hardware and
>>>>>>>>>>>>>>>>>>                                         CPU
>>>>>>>>>>>>>>>>>>                                         architecture.
>>>>>>>>>>>>>>>>>>                                         I figured
>>>>>>>>>>>>>>>>>>                                         this might
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         be an
>>>>>>>>>>>>>>>>>>                                         important detail.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         Adam
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         On Thu, Aug
>>>>>>>>>>>>>>>>>>                                         17, 2023 at
>>>>>>>>>>>>>>>>>>                                         11:44 AM
>>>>>>>>>>>>>>>>>>                                         Andrew Rybchenko
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>                                         <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Greatings Adam,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         I'm happy to
>>>>>>>>>>>>>>>>>>                                         hear that
>>>>>>>>>>>>>>>>>>                                         you're trying
>>>>>>>>>>>>>>>>>>                                         to bring
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         it up.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         As I
>>>>>>>>>>>>>>>>>>                                         understand
>>>>>>>>>>>>>>>>>>                                         the final
>>>>>>>>>>>>>>>>>>                                         goal is to
>>>>>>>>>>>>>>>>>>                                         run it on
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         regular
>>>>>>>>>>>>>>>>>>                                         basis. So, we
>>>>>>>>>>>>>>>>>>                                         need to make
>>>>>>>>>>>>>>>>>>                                         it properly
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         from the very
>>>>>>>>>>>>>>>>>>                                         beginning.
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Bring up of
>>>>>>>>>>>>>>>>>>                                         all features
>>>>>>>>>>>>>>>>>>                                         consists of 4
>>>>>>>>>>>>>>>>>>                                         steps:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         1. Create
>>>>>>>>>>>>>>>>>>                                         site-specific
>>>>>>>>>>>>>>>>>>                                         repository
>>>>>>>>>>>>>>>>>>                                         (we call it
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         ts-rigs)
>>>>>>>>>>>>>>>>>>                                         which
>>>>>>>>>>>>>>>>>>                                         contains
>>>>>>>>>>>>>>>>>>                                         information
>>>>>>>>>>>>>>>>>>                                         about test
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         rigs and
>>>>>>>>>>>>>>>>>>                                         other
>>>>>>>>>>>>>>>>>>                                         site-specific
>>>>>>>>>>>>>>>>>>                                         information like
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         where to send
>>>>>>>>>>>>>>>>>>                                         mails, where
>>>>>>>>>>>>>>>>>>                                         to store logs
>>>>>>>>>>>>>>>>>>                                         etc.
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         It is
>>>>>>>>>>>>>>>>>>                                         required for
>>>>>>>>>>>>>>>>>>                                         manual
>>>>>>>>>>>>>>>>>>                                         execution as
>>>>>>>>>>>>>>>>>>                                         well,
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         since test
>>>>>>>>>>>>>>>>>>                                         rigs
>>>>>>>>>>>>>>>>>>                                         description
>>>>>>>>>>>>>>>>>>                                         is essential.
>>>>>>>>>>>>>>>>>>                                         I'll
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         return to the
>>>>>>>>>>>>>>>>>>                                         topic below.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         2. Setup logs
>>>>>>>>>>>>>>>>>>                                         storage for
>>>>>>>>>>>>>>>>>>                                         automated runs.
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Basically it
>>>>>>>>>>>>>>>>>>                                         is a disk
>>>>>>>>>>>>>>>>>>                                         space plus
>>>>>>>>>>>>>>>>>>                                         apache2 web
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         server with
>>>>>>>>>>>>>>>>>>                                         few CGI
>>>>>>>>>>>>>>>>>>                                         scripts which
>>>>>>>>>>>>>>>>>>                                         help a lot to
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         save disk space.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         3. Setup
>>>>>>>>>>>>>>>>>>                                         Bublik web
>>>>>>>>>>>>>>>>>>                                         application
>>>>>>>>>>>>>>>>>>                                         which provides
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         web interface
>>>>>>>>>>>>>>>>>>                                         to view
>>>>>>>>>>>>>>>>>>                                         testing
>>>>>>>>>>>>>>>>>>                                         results. Same as
>>>>>>>>>>>>>>>>>>                                         https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>>>                                         <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>                                         <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         4. Setup
>>>>>>>>>>>>>>>>>>                                         Jenkins to
>>>>>>>>>>>>>>>>>>                                         run tests on
>>>>>>>>>>>>>>>>>>                                         regularly,
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         save logs in
>>>>>>>>>>>>>>>>>>                                         log storage
>>>>>>>>>>>>>>>>>>                                         (2) and
>>>>>>>>>>>>>>>>>>                                         import it to
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         bublik (3).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Last few
>>>>>>>>>>>>>>>>>>                                         month we
>>>>>>>>>>>>>>>>>>                                         spent on our
>>>>>>>>>>>>>>>>>>                                         homework to make
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         it simpler to
>>>>>>>>>>>>>>>>>>                                         bring up
>>>>>>>>>>>>>>>>>>                                         automated
>>>>>>>>>>>>>>>>>>                                         execution
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         using Jenkins -
>>>>>>>>>>>>>>>>>>                                         https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>>>                                         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>                                         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Corresponding
>>>>>>>>>>>>>>>>>>                                         bits in
>>>>>>>>>>>>>>>>>>                                         dpdk-ethdev-ts
>>>>>>>>>>>>>>>>>>                                         will be
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         available
>>>>>>>>>>>>>>>>>>                                         tomorrow.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Let's return
>>>>>>>>>>>>>>>>>>                                         to the step (1).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Unfortunately
>>>>>>>>>>>>>>>>>>                                         there is no
>>>>>>>>>>>>>>>>>>                                         publicly
>>>>>>>>>>>>>>>>>>                                         available
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         example of
>>>>>>>>>>>>>>>>>>                                         the ts-rigs
>>>>>>>>>>>>>>>>>>                                         repository since
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         sensitive
>>>>>>>>>>>>>>>>>>                                         site-specific
>>>>>>>>>>>>>>>>>>                                         information
>>>>>>>>>>>>>>>>>>                                         is located
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         there. But
>>>>>>>>>>>>>>>>>>                                         I'm ready to
>>>>>>>>>>>>>>>>>>                                         help you to
>>>>>>>>>>>>>>>>>>                                         create it
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         for UNH. I
>>>>>>>>>>>>>>>>>>                                         see two
>>>>>>>>>>>>>>>>>>                                         options here:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         (A) I'll ask
>>>>>>>>>>>>>>>>>>                                         questions and
>>>>>>>>>>>>>>>>>>                                         based on your
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         answers will
>>>>>>>>>>>>>>>>>>                                         create the
>>>>>>>>>>>>>>>>>>                                         first draft
>>>>>>>>>>>>>>>>>>                                         with my
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         comments.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         (B) I'll make
>>>>>>>>>>>>>>>>>>                                         a
>>>>>>>>>>>>>>>>>>                                         template/example
>>>>>>>>>>>>>>>>>>                                         ts-rigs repo,
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         publish it
>>>>>>>>>>>>>>>>>>                                         and you'll
>>>>>>>>>>>>>>>>>>                                         create UNH
>>>>>>>>>>>>>>>>>>                                         ts-rigs based
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         on it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Of course,
>>>>>>>>>>>>>>>>>>                                         I'll help to
>>>>>>>>>>>>>>>>>>                                         debug and
>>>>>>>>>>>>>>>>>>                                         finally bring
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         it up in any
>>>>>>>>>>>>>>>>>>                                         case.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         (A) is a bit
>>>>>>>>>>>>>>>>>>                                         simpler for
>>>>>>>>>>>>>>>>>>                                         me and you,
>>>>>>>>>>>>>>>>>>                                         but (B) is
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         a bit more
>>>>>>>>>>>>>>>>>>                                         generic and
>>>>>>>>>>>>>>>>>>                                         will help other
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         potential
>>>>>>>>>>>>>>>>>>                                         users to
>>>>>>>>>>>>>>>>>>                                         bring it up.
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         We can
>>>>>>>>>>>>>>>>>>                                         combine
>>>>>>>>>>>>>>>>>>                                         (A)+(B). I.e.
>>>>>>>>>>>>>>>>>>                                         start from (A).
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         What do you
>>>>>>>>>>>>>>>>>>                                         think?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                         On 8/17/23
>>>>>>>>>>>>>>>>>>                                         15:18,
>>>>>>>>>>>>>>>>>>                                         Konstantin
>>>>>>>>>>>>>>>>>>                                         Ushakov wrote:
>>>>>>>>>>>>>>>>>>>                                         Greetings Adam,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                         Thanks for
>>>>>>>>>>>>>>>>>>>                                         contacting
>>>>>>>>>>>>>>>>>>>                                         us. I copy
>>>>>>>>>>>>>>>>>>>                                         Andrew who
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                         would be
>>>>>>>>>>>>>>>>>>>                                         happy to help
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                         Konstantin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                         On 16 Aug
>>>>>>>>>>>>>>>>>>>>                                         2023, at
>>>>>>>>>>>>>>>>>>>>                                         21:50, Adam
>>>>>>>>>>>>>>>>>>>>                                         Hassick
>>>>>>>>>>>>>>>>>>>>                                         <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         Greetings
>>>>>>>>>>>>>>>>>>>>                                         Konstantin,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         I am in the
>>>>>>>>>>>>>>>>>>>>                                         process of
>>>>>>>>>>>>>>>>>>>>                                         setting up
>>>>>>>>>>>>>>>>>>>>                                         the DPDK
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         Poll Mode
>>>>>>>>>>>>>>>>>>>>                                         Driver test
>>>>>>>>>>>>>>>>>>>>                                         suite as an
>>>>>>>>>>>>>>>>>>>>                                         addition to
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         our testing
>>>>>>>>>>>>>>>>>>>>                                         coverage
>>>>>>>>>>>>>>>>>>>>                                         for DPDK at
>>>>>>>>>>>>>>>>>>>>                                         the UNH lab.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         I have some
>>>>>>>>>>>>>>>>>>>>                                         questions
>>>>>>>>>>>>>>>>>>>>                                         about how
>>>>>>>>>>>>>>>>>>>>                                         to set the
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         test suite
>>>>>>>>>>>>>>>>>>>>                                         arguments.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         I have been
>>>>>>>>>>>>>>>>>>>>                                         able to
>>>>>>>>>>>>>>>>>>>>                                         configure
>>>>>>>>>>>>>>>>>>>>                                         the Test Engine
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         to connect
>>>>>>>>>>>>>>>>>>>>                                         to the
>>>>>>>>>>>>>>>>>>>>                                         hosts in
>>>>>>>>>>>>>>>>>>>>                                         the
>>>>>>>>>>>>>>>>>>>>                                         testbed. The
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         RCF,
>>>>>>>>>>>>>>>>>>>>                                         Configurator,
>>>>>>>>>>>>>>>>>>>>                                         and Tester
>>>>>>>>>>>>>>>>>>>>                                         all begin to
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         run,
>>>>>>>>>>>>>>>>>>>>                                         however the
>>>>>>>>>>>>>>>>>>>>                                         prelude of
>>>>>>>>>>>>>>>>>>>>                                         the test suite
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         fails to run.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                         https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>>>                                         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>                                         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         The
>>>>>>>>>>>>>>>>>>>>                                         documentation
>>>>>>>>>>>>>>>>>>>>                                         mentions
>>>>>>>>>>>>>>>>>>>>                                         that there are
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         several
>>>>>>>>>>>>>>>>>>>>                                         test
>>>>>>>>>>>>>>>>>>>>                                         parameters
>>>>>>>>>>>>>>>>>>>>                                         for the
>>>>>>>>>>>>>>>>>>>>                                         test suite,
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         like for
>>>>>>>>>>>>>>>>>>>>                                         the IUT
>>>>>>>>>>>>>>>>>>>>                                         test link
>>>>>>>>>>>>>>>>>>>>                                         MAC, etc. These
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         seem like
>>>>>>>>>>>>>>>>>>>>                                         they would
>>>>>>>>>>>>>>>>>>>>                                         need to be
>>>>>>>>>>>>>>>>>>>>                                         set somewhere
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         to run many
>>>>>>>>>>>>>>>>>>>>                                         of the tests.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         I see in
>>>>>>>>>>>>>>>>>>>>                                         the Test
>>>>>>>>>>>>>>>>>>>>                                         Engine
>>>>>>>>>>>>>>>>>>>>                                         documentation,
>>>>>>>>>>>>>>>>>>>>                                         there
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         are
>>>>>>>>>>>>>>>>>>>>                                         instructions
>>>>>>>>>>>>>>>>>>>>                                         on how to
>>>>>>>>>>>>>>>>>>>>                                         create new
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         parameters
>>>>>>>>>>>>>>>>>>>>                                         for test
>>>>>>>>>>>>>>>>>>>>                                         suites in
>>>>>>>>>>>>>>>>>>>>                                         the Tester
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         configuration,
>>>>>>>>>>>>>>>>>>>>                                         but there
>>>>>>>>>>>>>>>>>>>>                                         is nothing
>>>>>>>>>>>>>>>>>>>>                                         in the
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         user guide
>>>>>>>>>>>>>>>>>>>>                                         or in the
>>>>>>>>>>>>>>>>>>>>                                         Tester
>>>>>>>>>>>>>>>>>>>>                                         guide for
>>>>>>>>>>>>>>>>>>>>                                         how to
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         set the
>>>>>>>>>>>>>>>>>>>>                                         arguments
>>>>>>>>>>>>>>>>>>>>                                         for the
>>>>>>>>>>>>>>>>>>>>                                         parameters when
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         running the
>>>>>>>>>>>>>>>>>>>>                                         test suite
>>>>>>>>>>>>>>>>>>>>                                         that I can
>>>>>>>>>>>>>>>>>>>>                                         find. I'm
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         not sure if
>>>>>>>>>>>>>>>>>>>>                                         I need to
>>>>>>>>>>>>>>>>>>>>                                         write my
>>>>>>>>>>>>>>>>>>>>                                         own Tester
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         config, or
>>>>>>>>>>>>>>>>>>>>                                         if I should
>>>>>>>>>>>>>>>>>>>>                                         be setting
>>>>>>>>>>>>>>>>>>>>                                         these in
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         some other way.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         How should
>>>>>>>>>>>>>>>>>>>>                                         these
>>>>>>>>>>>>>>>>>>>>                                         values be set?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         I'm also
>>>>>>>>>>>>>>>>>>>>                                         not sure
>>>>>>>>>>>>>>>>>>>>                                         what
>>>>>>>>>>>>>>>>>>>>                                         environment
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         variables/arguments
>>>>>>>>>>>>>>>>>>>>                                         are
>>>>>>>>>>>>>>>>>>>>                                         strictly
>>>>>>>>>>>>>>>>>>>>                                         necessary or
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         which are
>>>>>>>>>>>>>>>>>>>>                                         optional.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         Regards,
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         Adam
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         --
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         Senior
>>>>>>>>>>>>>>>>>>>>                                         Developer
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>>>>>>>>                                         InterOperability
>>>>>>>>>>>>>>>>>>>>                                         Lab
>>>>>>>>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>>>>>>>>                                         <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>>                                         +1 (603)
>>>>>>>>>>>>>>>>>>>>                                         475-8248
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         --
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>>>>>>                                         InterOperability
>>>>>>>>>>>>>>>>>>                                         Lab
>>>>>>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>>>>>>                                         <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>>>>                                         InterOperability
>>>>>>>>>>>>>>>>                                         Lab
>>>>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>>>>                                         <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>                                         (603) 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>                                         Hassick*
>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>                                         Developer
>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>                                         InterOperability
>>>>>>>>>>>>>>>>                                         Lab
>>>>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>>>>                                         <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>                                         (603) 475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             -- *Adam
>>>>>>>>>>>>>>>                                         Hassick*
>>>>>>>>>>>>>>>                                             Senior Developer
>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>                                         InterOperability Lab
>>>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>>>                                         <http://iol.unh.edu>
>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>                                         475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                         -- 
>>>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>>                                         InterOperability Lab
>>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>>                                         <http://iol.unh.edu>
>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                     --
>>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                 --
>>>>>>>>>>                                 *Adam Hassick*
>>>>>>>>>>                                 Senior Developer
>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                             --
>>>>>>>>                             *Adam Hassick*
>>>>>>>>                             Senior Developer
>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>                             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                             +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                     --
>>>>>>                     *Adam Hassick*
>>>>>>                     Senior Developer
>>>>>>                     UNH InterOperability Lab
>>>>>>                     ahassick@iol.unh.edu
>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                     +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>                 --
>>>>>                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>             --
>>>>>             *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>             ahassick@iol.unh.edu
>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>>
>>>
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 221057 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 56419 bytes --]

Hi Andrew,

Thank you for taking a look at our log. Netplan was attempting to run DHCP
on our test links, and additionally I discovered that the NIC firmware was
transmitting LLDP packets, causing tests to fail in the same way. Now that
these problems have been fixed, our pass rate on the XL710 is approximately
91%. Now that our test results are in line with yours, we can begin looking
into setting up the production environment.

First, is it possible to run the test agent on ARM hosts? Our ARM testbeds
have the best topology for running this test suite, with separate tester
and DUT servers.

We are testing this test suite on two x86 development servers using the
test suite's recommended server topology. In contrast, our existing x86
production testbeds which run DTS have a single server topology. This
single server has both the tester NIC and the device under test NIC
installed, with NUMA node separation between TRex and DPDK. We're going to
test running the two test agent processes on the single-server testbeds if
we cannot run this on ARM. Is there any reason you can think of that would
prevent this setup from working?

Once we figure out where this can live in production, then we will begin
setting up log storage, Jenkins integration, and Bublik.

Thanks,
Adam

On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> > Do these default to vfio-pci?
>
> Yes, vfio-pci is the default.
> However, it does not work in the case of Mellanox which uses bifurcated
> driver. It should mlx5_core for Mellanox NICs.
>
> > Here is the text log from a run on our Intel XL710 NICs, with the
> expected result profile set to the X710.
>
> It is hard to analyze all tests using text logs, but I definitely see one
> common problem. Tests receive unexpected packets and fail because of it.
> Tests are written very strict from this point of view and it brought
> fruits in the past when HW had bugs.
> Are DUT and tester connected back-to-back on tested interfaces or via
> switch?
> If via switch, is it possible to isolate it from everything else?
> If back-to-back, it could be some embedded SW/FW which regenerates these
> packet.
> I definitely see unexpected DHCP packets.
>
> > We haven't set up the Jenkins integration yet, however if this is
> required to import the logs then we will prioritize that.
>
> Unfortunately manual runs do not generate all artifacts required to import
> logs. However, we have almost solved it right now. Hopefully we'll finalize
> it in a day or two. I'll let you know when these changes are available.
>
> Regards,
> Andrew.
>
> On 10/4/23 16:48, Adam Hassick wrote:
>
> Hi Andrew,
>
> Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set anywhere in
> the default configurations for the Intel X710. Do these default to vfio-pci?
> We have IOMMU enabled on our development testbed, and should be able to
> bind vfio-pci.
> Here is the text log from a run on our Intel XL710 NICs, with the expected
> result profile set to the X710. We haven't set up the Jenkins integration
> yet, however if this is required to import the logs then we will prioritize
> that.
>  log.txt.tar.gz
> <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>
> Thanks,
> Adam
>
> On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> On 9/18/23 17:44, Adam Hassick wrote:
>>
>> Hi Andrew and Konstantin,
>>
>> Thank you for adding the tester-dial feature, this opens up the
>> possibility for us to do CI integrated testing in the future.
>>
>> Our Mellanox pass rate is similar to yours (about ~2400 passing, ~4400
>> failing), however our Intel pass rates are far worse.
>> I will try running tests on the XL710 with the trc-tags argument set and
>> see if it improves the pass rate.
>> Another thing I noticed in the results you uploaded is that the results
>> are tagged with vfio-pci and not i40e.
>> Though in the environment dump, the driver on the test machine and the
>> DUT are set to use the i40e driver. Is this important at all?
>>
>>
>> I think it is a misunderstanding here. There are two kinds of driver in
>> configuration: net driver and so-called DPDK driver.
>> Net driver is a Linux kernel network device driver used on Tester side.
>> DPDK driver is a Linux kernel driver to bind device to to use it with
>> DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
>> In the case of bifurcated driver (like mlx5_core) it is the same in both
>> cases.
>> In non-bifurcated case DPDK driver is some UIO driver(vfio-pci,
>> uio-pci-generic or igb_uio).
>> Some expectations depend on used UIO. For example, uio-pci-generic do not
>> support many interrupts (used by usecases/rx_intr test cases).
>> That's why we care corresponding TRC tag.
>>
>> TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's Intel case. Or
>> uio-pci-generic if IOMMU is turned off on corresponding machines and Linux
>> distro does not support VFIO no IOMMU mode.
>>
>> Andrew.
>>
>> There isn't anything preventing us from pushing our results up to the
>> existing Bublik instance running at ts-factory.io that I can think of at
>> the moment.
>> We will have to work out how to submit our results to your Bublik
>> instance in a controlled and secure manner in that case.
>> As far as I know we won't need access controls for the results
>> themselves. I'll discuss this with Patrick and will let you know once we
>> confirm that it's fine.
>>
>> Thanks,
>> Adam
>>
>> On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>
>>> Hi Andrew,
>>>
>>> should we always auto-assign the tags or you don’t do it since it slows
>>> down (by some seconds) the TE startup?
>>>
>>>
>>> Tags are auto-assigned, but I guess it differs in Adam's case since NIC
>>> is a bit different. Below test will help to understand if it is the root
>>> cause of very different expectations. If pass rate will be close to mine,
>>> I'll simply update TRC database to share expectations for mine NIC and NIC
>>> used by Adam.
>>>
>>> Hi Adam,
>>>
>>> I think I second the question from Andrew - happy to help you with the
>>> triage so that we get to the same baseline. Do you have a good way for us
>>> to share the logs? I.e. say upload to ts-factory if we add strict
>>> permissions system so it’s not publishing or any other way.
>>>
>>> Thanks,
>>> Konstantin
>>>
>>> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>
>>> Hi Adam,
>>>
>>> I've uploaded fresh testing results to ts-factory.io [1] to be on the
>>> same page.
>>>
>>> I think I know why your and mine results on Intel 710 series NICs differ
>>> so much. Testing results expectations database (dpdk-ethdev-ts/trc/*) is
>>> filled in in terms of TRC tags.  I.e. expectations depends on TRC tags
>>> discovered by helper scripts when testing is started. These tags identify
>>> various aspects of what is tested. Ideally expectations should be written
>>> in terms of root cause of the expected behaviour. If it is a driver
>>> expectations, driver tag should be used. If it is HW limitation, tags with
>>> PCI IDs should be used. However, it is not always easy to classify it
>>> correctly if you're not involved in driver development. So, in order case
>>> expectations for 710's Intel are filled in in terms of PCI IDs. I guess PCI
>>> ID differ in your case and that's why expectations filled in for my NIC do
>>> not apply to your runs.
>>>
>>> Just try to add the following option when you run on your 710's Intel in
>>> order to mimic mine and see if it helps to achieve better pass rate.
>>> --trc-tag=pci-8086-1572
>>>
>>> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for
>>> --tester-dial option. It should have better coverage now.
>>>
>>> Andrew.
>>>
>>> [1]
>>> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>
>>> On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>
>>> Hi Adam,
>>>
>>> I've pushed new TE tag v1.20.0 which supported a new command-line option
>>> --tester-dial=NUM where NUM is from 0 to 100. it allows to choose
>>> percentage of tests to run. If you want stable set, you should pass
>>> --tester-random-seed=0 (or other integer). It is the first sketch and we
>>> have plans to improve it, but feedback would be welcome.
>>>
>>> > Is it needed on the tester?
>>>
>>> It is hard to say if it is strictly required for simple tests. However,
>>> it is better to update Tester as well, since performance tests run DPDK on
>>> Tester as well.
>>>
>>> > Are there any other manual setup steps for these devices that I might
>>> be missing?
>>>
>>> I don't remember anything else.
>>>
>>> I think it is better to get down to details and take a look at logs. I'm
>>> ready to help with it and explain what's happening there. May be it will
>>> help to understand if it is a problem with setup/configuration.
>>>
>>> Text logs are not very convenient. Ideally logs should be imported to
>>> bublik, however, manual runs do not provide all required artifacts right
>>> now (Jenkins jobs generate all required artifacts).
>>> Other option is 'tmp_raw_log' file (should be packed to make it smaller)
>>> which could be converted to various log formats.
>>> Would it be OK for you if I import your logs to bublik at ts-factory.io?
>>> Or is it a problem that it is publicly available?
>>> Would it help if we add authentication and access control there?
>>>
>>> Andrew.
>>>
>>> On 9/8/23 17:57, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> I have a couple questions about needed setup of the NICs for the ethdev
>>> test suite.
>>>
>>> Our MCX5s and XL710s are failing the checkup tests. The pass rate
>>> appears to be much worse on the XL710s (40 of 73 tests failed, 3 passed
>>> unexpectedly).
>>>
>>> For the XL710s, I've updated the driver and NVM versions to match the
>>> minimum supported versions in the compatibility matrix found on the DPDK
>>> documentation. This did not change the failure rate much.
>>> For the MCX5s, I've installed the latest LTS version of the OFED
>>> bifurcated driver on the DUT. Is it needed on the tester?
>>>
>>> Are there any other manual setup steps for these devices that I might be
>>> missing?
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu>
>>> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> Yes, I copied the X710 configs to set up XL710 configs. I changed the
>>>> environment variable names from the X710 suffix to XL710 suffix in the
>>>> script, and forgot to change them in the corresponding environment file.
>>>> That fixed the issue.
>>>>
>>>> I got the checkup tests working on the XL710 now. Most of them are
>>>> failing, which leads me to believe this is an issue with our testbed. Based
>>>> on the DPDK documentation for i40e, the firmware and driver versions are
>>>> much older than what DPDK 22.11 LTS and main prefer, so I'll try updating
>>>> those.
>>>>
>>>> For now I'm working on getting the XL710 checkup tests passing, and
>>>> will pick up getting the E810 configured properly next. I'll let you know
>>>> if I run into any more issues in relation to the test engine.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> Hi Adam,
>>>>>
>>>>> On 9/5/23 18:01, Adam Hassick wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> The compilation warning issue is now resolved. Again, thank you guys
>>>>> for fixing this for us. I can run the tests on the Mellanox CX5s again,
>>>>> however I'm running into a couple new issues with running the prologues on
>>>>> the Intel cards.
>>>>>
>>>>> When running testing on the Intel XL710s, I see this error appear in
>>>>> the log:
>>>>>
>>>>> ERROR  prologue  Environment LIB  14:16:13.650
>>>>>> Too few networks in available configuration (0) in comparison with
>>>>>> required (1)
>>>>>>
>>>>>
>>>>> This seems like a trivial configuration error, perhaps this is
>>>>> something I need to set up in ts-rigs. I briefly searched through the
>>>>> examples there and didn't see any mention of how to set up a network.
>>>>> I will attach this log just in case you need more information.
>>>>>
>>>>>
>>>>> Unfortunately logs are insufficient to understand it. I've pushed new
>>>>> tag to TE v1.19.0 which add log message with TE_* environment variables.
>>>>> Most likely something is wrong with variables which are used as
>>>>> conditions when available networks are defined in
>>>>> ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>> TE_PCI_INSTANCE_IUT_TST1
>>>>> TE_PCI_INSTANCE_IUT_TST1a
>>>>> TE_PCI_INSTANCE_TST1a_IUT
>>>>> TE_PCI_INSTANCE_TST1_IUT
>>>>> My guess it that you change naming a bit, but script like
>>>>> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>>>>>
>>>>> There is a different error when running on the Intel E810s. It appears
>>>>> to me like it starts DPDK, does some configuration inside DPDK and on the
>>>>> device, and then fails to bring the device back up. Since this error seems
>>>>> very non-trivial, I will also attach this log.
>>>>>
>>>>>
>>>>> This one is a bit simpler. Few lines after the first ERROR in log I
>>>>> see the following:
>>>>> WARN  RCF  DPDK  13:06:00.144
>>>>> ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)
>>>>> ice_rx_queue_start(): fail to program RX queue 0
>>>>> ice_dev_start(): fail to start Rx queue 0
>>>>> Device with port_id=0 already stopped
>>>>>
>>>>> It is stdout/stderr from test agent which runs DPDK. Same logs in
>>>>> plain format are available in ta.DPDK file.
>>>>> I'm not an expert here, but I vaguely remember that E810 requires
>>>>> correct firmware and DDP to be loaded.
>>>>> There is some information in dpdk/doc/guides/nics/ice.rst.
>>>>>
>>>>> You can try to add --dev-args=safe-mode-support=1 command-line option
>>>>> described there.
>>>>>
>>>>> Hope it helps,
>>>>> Andrew.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>> Hi Adam,
>>>>>>
>>>>>> On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> I have one additional question as well: Does the test engine support
>>>>>> running tests on two ARMv8 test agents?
>>>>>>
>>>>>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>>>>>
>>>>>>
>>>>>> Great. Let me know when that's fixed.
>>>>>>
>>>>>>
>>>>>> Done. We also fixed a number of warnings in TE.
>>>>>> Also we fixed root test package name to be consistent with the
>>>>>> repository name.
>>>>>>
>>>>>> Support for old LTS branches was dropped some time ago, but in the
>>>>>>> future it is definitely possible to keep it for new LTS branches. I think
>>>>>>> 22.11 is supported, but I'm not sure about older LTS releases.
>>>>>>>
>>>>>>
>>>>>> Good to know.
>>>>>>
>>>>>>
>>>>>>> 2. You can add command-line option --sanity to run tests marked with
>>>>>>> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
>>>>>>> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
>>>>>>> there is a space for terminology improvement here. We'll do it.
>>>>>>>
>>>>>>
>>>>>> Done. Now it is called --checkup.
>>>>>>
>>>>>>
>>>>>>> Also it takes a lot of time because of failures and tests which wait
>>>>>>> for some timeout.
>>>>>>>
>>>>>>
>>>>>> That makes sense to me. We'll use the time to complete tests on
>>>>>> virtio or the Intel devices as a reference for how long the tests really
>>>>>> take to complete.
>>>>>> We will explore the possibility of periodically running the sanity
>>>>>> tests for patches.
>>>>>>
>>>>>>
>>>>>> I'll double-check and let you know how long entire TS runs on Intel
>>>>>> X710, E810, Mellanox CX5 and virtio net. Just to ensure that time observed
>>>>>> in your case looks the same.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> The test harness can provide coverage reports based on gcov, but I'm
>>>>>>> not sure what you mean by a "dial" to control test coverage. Provided
>>>>>>> reports are rather for human to analyze.
>>>>>>>
>>>>>>
>>>>>> The general idea is to have some kind of parameter on the test suite,
>>>>>> which could be an integer ranging from zero to ten, that controls how many
>>>>>> tests are run based on how important the test is.
>>>>>>
>>>>>> Similar to how some command line interfaces provide a verbosity level
>>>>>> parameter (some number of "-v" arguments) to control the importance of the
>>>>>> information in the log.
>>>>>> The verbosity level zero only prints very important log messages,
>>>>>> while ten prints everything.
>>>>>>
>>>>>> In much the same manner as above, this "dial" parameter controls what
>>>>>> tests are run and with what parameters based on how important those tests
>>>>>> and test parameter combinations are.
>>>>>> Coverage Level zero tells the suite to run a very basic set of
>>>>>> important tests, with minimal parameterization. This mode would take only
>>>>>> ~5-10 minutes to run.
>>>>>> In contrast, Coverage Level ten includes all the edge cases, every
>>>>>> combination of test parameters, everything the test suite can do, which
>>>>>> takes the normal several hours to run.
>>>>>> The values 1 - 9 are between those two extremes, allowing the user to
>>>>>> get a gradient of test coverage in the results and to limit the running
>>>>>> time.
>>>>>>
>>>>>> Then we could, for example, run the "run.sh" with a level of 2 or 3
>>>>>> for incoming patches that need quick results, and with a level of 10 for
>>>>>> the less often run periodic tests performed on main or LTS branches.
>>>>>>
>>>>>>
>>>>>> Understood now. Thanks a lot for the idea. We'll discuss it and come
>>>>>> back.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>>>>>>> testing results. Unfortunately it is time consuming to fill in expectations
>>>>>>> database since it is necessary to analyze testing results and classify if
>>>>>>> it is a bug or just acceptable behaviour aspect.
>>>>>>>
>>>>>>> Bublik allows to compare results of two runs. It is useful for
>>>>>>> human, but still not good for automation.
>>>>>>>
>>>>>>> I have local patch for mlx5 driver which reports Tx ring size
>>>>>>> maximum. It makes pass rate higher. It is a problem for test harness that
>>>>>>> mlx5 does not report limits right now.
>>>>>>>
>>>>>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>>>>>>> virtio net is 99% right now and could be done 100% easily (just one thing
>>>>>>> to fix in expectations).
>>>>>>>
>>>>>>> I think logs storage setup is essential for logs analysis. Of
>>>>>>> course, you can request HTML logs when you run tests (--log-html=html) or
>>>>>>> generate after run using dpdk-ethdev-ts/scripts/html-log.sh and open
>>>>>>> index.html in a browser, but logs storage makes it more convenient.
>>>>>>>
>>>>>>
>>>>>> We are interested in setting up Bublik, potentially as an
>>>>>> externally-facing component, once we have our process of running the test
>>>>>> suite stabilized.
>>>>>> Once we are able to run the test suite again, I'll see what the pass
>>>>>> rate is on our other hardware.
>>>>>> Good to know that it isn't an issue with our dev testbed causing the
>>>>>> high fail rate.
>>>>>>
>>>>>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>>>>>> development testbed. Although they are slightly different devices, ideally
>>>>>> the pass rate will be identical or similar. I have yet to set up a VM pair
>>>>>> for virtio, but we will soon.
>>>>>>
>>>>>> Latest version of test-environment has examples of our CGI scripts
>>>>>>> which we use for log storage (see tools/log_server/README.md).
>>>>>>>
>>>>>>> Also all bits for Jenkins setup are available. See
>>>>>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>>>>>> ts-rigs-sample.
>>>>>>>
>>>>>>
>>>>>> Jenkins integration, setting up production rig configurations, and
>>>>>> permanent log storage will be our next steps once I am able to run the
>>>>>> tests again.
>>>>>> Unless there is an easy way to have meson not pass "-Werror" into
>>>>>> GCC. Then I would be able to run the test suite.
>>>>>>
>>>>>>
>>>>>> Hopefully it is resolved now.
>>>>>>
>>>>>> I thought a bit more about your usecase for Jenkins. I'm not 100%
>>>>>> sure that existing pipelines are convenient for your usecase.
>>>>>> Fill free to ask questions when you are on it.
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Adam
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>>
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> That fix seems to have resolved the issue, thanks for the quick
>>>>>>> turnaround time on that patch.
>>>>>>> Now that we have the RCF timeout issue resolved, there are a few
>>>>>>> other questions and issues that we have about the tests themselves.
>>>>>>>
>>>>>>> 1. The test suite fails to build with a couple warnings.
>>>>>>>
>>>>>>> Below is the stderr log from compilation:
>>>>>>>
>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>>>> and no format arguments [-Werror=format-security]
>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>> |     ^~
>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>> ninja: build stopped: subcommand failed.
>>>>>>>> ninja: Entering directory `.'
>>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>>>> and no format arguments [-Werror=format-security]
>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>> |     ^~
>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>>
>>>>>>>
>>>>>>> This error wasn't occurring last week, which was the last time I ran
>>>>>>> the tests.
>>>>>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester
>>>>>>> has GCC v11.4.0 installed, if this information is helpful.
>>>>>>>
>>>>>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>>>>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>>>>>> to lower the test coverage and have the test suite run faster?
>>>>>>>
>>>>>>> For some context, we run immediate testing on incoming patches for
>>>>>>> DPDK main and development branches, as well as periodic test runs on the
>>>>>>> main, stable, and LTS branches.
>>>>>>> For us to consider including this test suite as part of our
>>>>>>> immediate testing on patches, we would have to reduce the test coverage to
>>>>>>> the most important tests.
>>>>>>> This is primarily to reduce the testing time to, for example, less
>>>>>>> than 30 minutes. Testing on patches can't take too long because the lab can
>>>>>>> receive numerous patches each day, which each require individual testing
>>>>>>> runs.
>>>>>>>
>>>>>>> At what frequency we run these tests, and on what, still needs to be
>>>>>>> discussed with the DPDK community, but it would be nice to know if the test
>>>>>>> suite had a "dial" to control the testing coverage.
>>>>>>>
>>>>>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>>>>>> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose these
>>>>>>> test failures?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Adam
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>> Hi Adam,
>>>>>>>>
>>>>>>>> I've pushed the fix in main branch and a new tag v1.18.1. It should
>>>>>>>> solve the problem with IPv6 address from DNS.
>>>>>>>>
>>>>>>>> Andrew.
>>>>>>>>
>>>>>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>>
>>>>>>>> Hi Adam,
>>>>>>>>
>>>>>>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>>>> force it to use IPv4?
>>>>>>>>
>>>>>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your
>>>>>>>> case, I guess it is the root cause of the problem.
>>>>>>>> Of course, it is TE problem since I see really weird code in
>>>>>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>>>>
>>>>>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>>>>>> ts-factory/test-environment repository. Please, try.
>>>>>>>>
>>>>>>>> It is late night fix with minimal testing and no review. I'll pass
>>>>>>>> it through review process tomorrow and
>>>>>>>> hopefully it will be released in one-two days.
>>>>>>>>
>>>>>>>> Andrew.
>>>>>>>>
>>>>>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> We have yet to notice a distinct pattern with the failures.
>>>>>>>> Sometimes, the RCF will start and connect without issue a few times in a
>>>>>>>> row before failing to connect again. Once the issue begins to occur,
>>>>>>>> neither rebooting all of the hosts (test engine VM, tester, IUT) or
>>>>>>>> deleting all of the build directories (suites, agents, inst) and rebooting
>>>>>>>> the hosts afterward resolves the issue. When it begins working again seems
>>>>>>>> very arbitrary to us.
>>>>>>>>
>>>>>>>> I do usually try to terminate the test engine with Ctrl+C, but when
>>>>>>>> it hangs while trying to start RCF, that does not work.
>>>>>>>>
>>>>>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>>>> force it to use IPv4?
>>>>>>>>
>>>>>>>>  - Adam
>>>>>>>>
>>>>>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>
>>>>>>>>> Done. It works fine for me without any issues.
>>>>>>>>>
>>>>>>>>> Have you noticed any pattern when it works or does not work?
>>>>>>>>> May be it is a problem of not clean state after termination?
>>>>>>>>> Does it work fine the first time after DUTs reboot?
>>>>>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>>>>>> terminal where you execute run.sh command.
>>>>>>>>>  In this case it should shutdown gracefully and close all test
>>>>>>>>> agents and engine applications.
>>>>>>>>>
>>>>>>>>> (I'm trying to understand why you've seen many test agent
>>>>>>>>> processes. It should not happen.)
>>>>>>>>>
>>>>>>>>> Andrew.
>>>>>>>>>
>>>>>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>>>
>>>>>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>> Hi Andrew,
>>>>>>>>>
>>>>>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>>>>>> This could be the source of the problem. I encountered a
>>>>>>>>> dependency issue trying to run the Test Engine on 22.04 LTS, so I
>>>>>>>>> downgraded the system. Since the tester is also the host having connection
>>>>>>>>> issues, I will try downgrading that system to 20.04, and see if that
>>>>>>>>> changes anything.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>>>>>>>> Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every
>>>>>>>>> night.
>>>>>>>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>>>>>>>
>>>>>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>
>>>>>>>>> I did try passing in the "--vg-rcf" argument to the run.sh script
>>>>>>>>> of the test suite after installing valgrind, but there was no additional
>>>>>>>>> output that I saw.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>>>>>> (direction where you run test engine).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I will try pulling in the changes you've pushed up, and will see
>>>>>>>>> if that fixes anything.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Adam
>>>>>>>>>
>>>>>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Adam,
>>>>>>>>>>
>>>>>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>>>>
>>>>>>>>>> I'd like to try to repeat the problem locally. Which Linux distro
>>>>>>>>>> is running on test engine and agents?
>>>>>>>>>>
>>>>>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we
>>>>>>>>>> have
>>>>>>>>>> patch in review to fix it, however, the behaviour is different in
>>>>>>>>>> this case, so it is unlike the same problem.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I've just published a new tag which fixes known test engine side
>>>>>>>>>> problems on Debian 12 and Fedora 38.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>>>>>> run with option --vg-rcf to check if something weird is happening.
>>>>>>>>>>
>>>>>>>>>> What I don't understand right now is why I see just one failed
>>>>>>>>>> attempt
>>>>>>>>>> to connect in your log.txt and then Logger shutdown after 9
>>>>>>>>>> minutes.
>>>>>>>>>>
>>>>>>>>>> Andrew.
>>>>>>>>>>
>>>>>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>>>
>>>>>>>>>>  > Is there any firewall in the network or on test hosts which
>>>>>>>>>> could block incoming TCP connection to the port 23571
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>>> where you run test engine?
>>>>>>>>>>
>>>>>>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>>>>>>> connection does work sometimes.
>>>>>>>>>>
>>>>>>>>>>  > If behaviour the same on the next try and you see that test
>>>>>>>>>> agent is kept running, could you check using
>>>>>>>>>>  >
>>>>>>>>>>  > # netstat -tnlp
>>>>>>>>>>  >
>>>>>>>>>>  > that Test Agent is listening on the port and try to establish
>>>>>>>>>> TCP connection from test agent using
>>>>>>>>>>  >
>>>>>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>  >
>>>>>>>>>>  > and check if TCP connection could be established.
>>>>>>>>>>
>>>>>>>>>> I was able to replicate the same behavior again, where it hangs
>>>>>>>>>> while RCF is trying to start.
>>>>>>>>>> Running this command, I see this in the output:
>>>>>>>>>>
>>>>>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>>>>>>      18599/ta
>>>>>>>>>>
>>>>>>>>>> So it seems like it is listening on the correct port.
>>>>>>>>>> Additionally, I was able to connect to the Tester machine from
>>>>>>>>>> our Test Engine host using telnet. It printed the PID of the process once
>>>>>>>>>> the connection was opened.
>>>>>>>>>>
>>>>>>>>>> I tried running the "ta" application manually on the command
>>>>>>>>>> line, and it didn't print anything at all.
>>>>>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>>>>>> andrew.rybchenko@oktetlabs.ru
>>>>>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>     Hi Adam,
>>>>>>>>>>
>>>>>>>>>>      > On the tester host (which appears to be the Peer agent),
>>>>>>>>>> there
>>>>>>>>>>     are four processes that I see running, which look like the
>>>>>>>>>> test
>>>>>>>>>>     agent processes.
>>>>>>>>>>
>>>>>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>>>>>
>>>>>>>>>>     Is there any firewall in the network or on test hosts which
>>>>>>>>>> could
>>>>>>>>>>     block incoming TCP connection to the port 23571
>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>>>     where you run test engine?
>>>>>>>>>>
>>>>>>>>>>     If behaviour the same on the next try and you see that test
>>>>>>>>>> agent is
>>>>>>>>>>     kept running, could you check using
>>>>>>>>>>
>>>>>>>>>>     # netstat -tnlp
>>>>>>>>>>
>>>>>>>>>>     that Test Agent is listening on the port and try to establish
>>>>>>>>>> TCP
>>>>>>>>>>     connection from test agent using
>>>>>>>>>>
>>>>>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>
>>>>>>>>>>     and check if TCP connection could be established.
>>>>>>>>>>
>>>>>>>>>>     Another idea is to login Tester under root as testing does,
>>>>>>>>>> get
>>>>>>>>>>     start TA command from the log and try it by hands without -n
>>>>>>>>>> and
>>>>>>>>>>     remove extra escaping.
>>>>>>>>>>
>>>>>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>
>>>>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>
>>>>>>>>>>     Hopefully in this case test agent directory remains in the
>>>>>>>>>> /tmp and
>>>>>>>>>>     you don't need to copy it as testing does.
>>>>>>>>>>     May be output could shed some light on what's going on.
>>>>>>>>>>
>>>>>>>>>>     Andrew.
>>>>>>>>>>
>>>>>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>>
>>>>>>>>>>     Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>     This is the output that I see in the terminal when this
>>>>>>>>>> failure
>>>>>>>>>>     occurs, after the test agent binaries build and the test
>>>>>>>>>> engine
>>>>>>>>>>     starts:
>>>>>>>>>>
>>>>>>>>>>     Platform default build - pass
>>>>>>>>>>     Simple RCF consistency check succeeded
>>>>>>>>>>     --->>> Starting Logger...done
>>>>>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>>>>>>> timed
>>>>>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>
>>>>>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>>>>>
>>>>>>>>>>     On the tester host (which appears to be the Peer agent),
>>>>>>>>>> there are
>>>>>>>>>>     four processes that I see running, which look like the test
>>>>>>>>>> agent
>>>>>>>>>>     processes.
>>>>>>>>>>
>>>>>>>>>>     ta.Peer is an empty file. I've attached the log.txt from this
>>>>>>>>>> run.
>>>>>>>>>>
>>>>>>>>>>      - Adam
>>>>>>>>>>
>>>>>>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>         Hi Adam,
>>>>>>>>>>
>>>>>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>         says that copy_timeout is in seconds and implementation in
>>>>>>>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>>>>>>>         Theoretically select() could be interrupted by signal,
>>>>>>>>>> but I
>>>>>>>>>>         think it is unlikely here.
>>>>>>>>>>
>>>>>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help.
>>>>>>>>>> Or
>>>>>>>>>>         does it happen when tests are in progress, e.g. in the
>>>>>>>>>> middle
>>>>>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>>>>>>         likely either host with test agent dies or test agent
>>>>>>>>>> itself
>>>>>>>>>>         crashes. It would be easier for me if classify it if you
>>>>>>>>>> share
>>>>>>>>>>         text log (log.txt, full or just corresponding fragment
>>>>>>>>>> with
>>>>>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>>>>>         depending on which agent has problems could shed some
>>>>>>>>>> light.
>>>>>>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>>>>>>
>>>>>>>>>>         Andrew.
>>>>>>>>>>
>>>>>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>>
>>>>>>>>>>         Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>         I've set up a test rig repository here, and have created
>>>>>>>>>>         configurations for our development testbed based off of
>>>>>>>>>> the
>>>>>>>>>>         examples.
>>>>>>>>>>         We've been able to get the test suite to run manually on
>>>>>>>>>>         Mellanox CX5 devices once.
>>>>>>>>>>         However, we are running into an issue where, when RCF
>>>>>>>>>> starts,
>>>>>>>>>>         the RCF connection times out very frequently. We aren't
>>>>>>>>>> sure
>>>>>>>>>>         why this is the case.
>>>>>>>>>>         It works sometimes, but most of the time when we try to
>>>>>>>>>> run
>>>>>>>>>>         the test engine, it encounters this issue.
>>>>>>>>>>         I've tried changing the RCF port by setting
>>>>>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>>>>>>>         machines. Neither seems to fix the issue.
>>>>>>>>>>
>>>>>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>         before I try to run the test suite.
>>>>>>>>>>         I assume the unit for this variable is seconds?
>>>>>>>>>>
>>>>>>>>>>         Thanks,
>>>>>>>>>>         Adam
>>>>>>>>>>
>>>>>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>>>>>
>>>>>>>>>>             Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>             Thanks, I've cloned the example repository and will
>>>>>>>>>> start
>>>>>>>>>>             setting up a configuration for our development testbed
>>>>>>>>>>             today. I'll let you know if I run into any
>>>>>>>>>> difficulties
>>>>>>>>>>             or have any questions.
>>>>>>>>>>
>>>>>>>>>>              - Adam
>>>>>>>>>>
>>>>>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                 I've published
>>>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>                 Hopefully it will help to define your test rigs
>>>>>>>>>> and
>>>>>>>>>>                 successfully run some tests manually. Feel free to
>>>>>>>>>>                 ask any questions and I'll answer here and try to
>>>>>>>>>>                 update documentation.
>>>>>>>>>>
>>>>>>>>>>                 Meanwhile I'll prepare missing bits for steps (2)
>>>>>>>>>> and
>>>>>>>>>>                 (3).
>>>>>>>>>>                 Hopefully everything is in place for step (4),
>>>>>>>>>> but we
>>>>>>>>>>                 need to make steps (2) and (3) first.
>>>>>>>>>>
>>>>>>>>>>                 Andrew.
>>>>>>>>>>
>>>>>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>>>
>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                 > I've conferred with the rest of the team, and we
>>>>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>>>>                 option B.
>>>>>>>>>>
>>>>>>>>>>                 OK, I'll provide the sample on Monday for you. It
>>>>>>>>>> is
>>>>>>>>>>                 almost ready right now, but I need to double-check
>>>>>>>>>>                 it before publishing.
>>>>>>>>>>
>>>>>>>>>>                 Regards,
>>>>>>>>>>                 Andrew.
>>>>>>>>>>
>>>>>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>>
>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>>>                 conversation. Others in the community might find
>>>>>>>>>>                 this conversation valuable.
>>>>>>>>>>
>>>>>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>>>>>                 Jenkins integration will be very useful for us, as
>>>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>>>>>                 think it would be best to move forward with mainly
>>>>>>>>>>                 option B.
>>>>>>>>>>                 If you would like to know anything about our
>>>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>>>>>                 questions you have.
>>>>>>>>>>
>>>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>>>                 hardware and CPU architecture. I figured this
>>>>>>>>>> might
>>>>>>>>>>                 be an important detail.
>>>>>>>>>>
>>>>>>>>>>                 Thanks,
>>>>>>>>>>                 Adam
>>>>>>>>>>
>>>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>                     Greatings Adam,
>>>>>>>>>>
>>>>>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>>>>>                     it up.
>>>>>>>>>>
>>>>>>>>>>                     As I understand the final goal is to run it on
>>>>>>>>>>                     regular basis. So, we need to make it properly
>>>>>>>>>>                     from the very beginning.
>>>>>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>>>>>
>>>>>>>>>>                     1. Create site-specific repository (we call it
>>>>>>>>>>                     ts-rigs) which contains information about test
>>>>>>>>>>                     rigs and other site-specific information like
>>>>>>>>>>                     where to send mails, where to store logs etc.
>>>>>>>>>>                     It is required for manual execution as well,
>>>>>>>>>>                     since test rigs description is essential. I'll
>>>>>>>>>>                     return to the topic below.
>>>>>>>>>>
>>>>>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>>>>>                     server with few CGI scripts which help a lot
>>>>>>>>>> to
>>>>>>>>>>                     save disk space.
>>>>>>>>>>
>>>>>>>>>>                     3. Setup Bublik web application which provides
>>>>>>>>>>                     web interface to view testing results. Same as
>>>>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>>>> <https://ts-factory.io/bublik>
>>>>>>>>>>
>>>>>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>>>>>                     save logs in log storage (2) and import it to
>>>>>>>>>>                     bublik (3).
>>>>>>>>>>
>>>>>>>>>>                     Last few month we spent on our homework to
>>>>>>>>>> make
>>>>>>>>>>                     it simpler to bring up automated execution
>>>>>>>>>>                     using Jenkins -
>>>>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>>>                     available tomorrow.
>>>>>>>>>>
>>>>>>>>>>                     Let's return to the step (1).
>>>>>>>>>>
>>>>>>>>>>                     Unfortunately there is no publicly available
>>>>>>>>>>                     example of the ts-rigs repository since
>>>>>>>>>>                     sensitive site-specific information is located
>>>>>>>>>>                     there. But I'm ready to help you to create it
>>>>>>>>>>                     for UNH. I see two options here:
>>>>>>>>>>
>>>>>>>>>>                     (A) I'll ask questions and based on your
>>>>>>>>>>                     answers will create the first draft with my
>>>>>>>>>>                     comments.
>>>>>>>>>>
>>>>>>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>>>>>>                     on it.
>>>>>>>>>>
>>>>>>>>>>                     Of course, I'll help to debug and finally
>>>>>>>>>> bring
>>>>>>>>>>                     it up in any case.
>>>>>>>>>>
>>>>>>>>>>                     (A) is a bit simpler for me and you, but (B)
>>>>>>>>>> is
>>>>>>>>>>                     a bit more generic and will help other
>>>>>>>>>>                     potential users to bring it up.
>>>>>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>>>>>                     What do you think?
>>>>>>>>>>
>>>>>>>>>>                     Thanks,
>>>>>>>>>>                     Andrew.
>>>>>>>>>>
>>>>>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>>>
>>>>>>>>>>                     Greetings Adam,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>>>>                     would be happy to help
>>>>>>>>>>
>>>>>>>>>>                     Thanks,
>>>>>>>>>>                     Konstantin
>>>>>>>>>>
>>>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>                     Greetings Konstantin,
>>>>>>>>>>
>>>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>>>>
>>>>>>>>>>                     I have some questions about how to set the
>>>>>>>>>>                     test suite arguments.
>>>>>>>>>>
>>>>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>>>                     fails to run.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>
>>>>>>>>>>                     The documentation mentions that there are
>>>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>>>>                     to run many of the tests.
>>>>>>>>>>
>>>>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>>>>                     are instructions on how to create new
>>>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>>>                     some other way.
>>>>>>>>>>
>>>>>>>>>>                     How should these values be set?
>>>>>>>>>>
>>>>>>>>>>                     I'm also not sure what environment
>>>>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>>>>                     which are optional.
>>>>>>>>>>
>>>>>>>>>>                     Regards,
>>>>>>>>>>                     Adam
>>>>>>>>>>
>>>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>>>                     Senior Developer
>>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 --                 *Adam Hassick*
>>>>>>>>>>                 Senior Developer
>>>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>> <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>             --             *Adam Hassick*
>>>>>>>>>>             Senior Developer
>>>>>>>>>>             UNH InterOperability Lab
>>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>         --         *Adam Hassick*
>>>>>>>>>>         Senior Developer
>>>>>>>>>>         UNH InterOperability Lab
>>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>         +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     --     *Adam Hassick*
>>>>>>>>>>     Senior Developer
>>>>>>>>>>     UNH InterOperability Lab
>>>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>     +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Adam Hassick*
>>>>>>>>>> Senior Developer
>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Adam Hassick*
>>>>>>>>> Senior Developer
>>>>>>>>> UNH InterOperability Lab
>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Adam Hassick*
>>>>>>>> Senior Developer
>>>>>>>> UNH InterOperability Lab
>>>>>>>> ahassick@iol.unh.edu
>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Adam Hassick*
>>>>>>> Senior Developer
>>>>>>> UNH InterOperability Lab
>>>>>>> ahassick@iol.unh.edu
>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 212419 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 178151 bytes --]

Hi Adam,

On 10/10/23 17:09, Adam Hassick wrote:
> Hi Andrew,
>
> Thank you for taking a look at our log. Netplan was attempting to run 
> DHCP on our test links,

Got it. Our automatic checks that network manager is not running on test 
interfaces, but we do not care about netplan yet.

> and additionally I discovered that the NIC firmware was transmitting 
> LLDP packets, causing tests to fail in the same way.

In theory testing prologue tries to disable FW LLDP using 
disable-fw-lldp interface private flag, but may be NIC in your case does 
not support it or name differs.

> Now that these problems have been fixed, our pass rate on the XL710 is 
> approximately 91%. Now that our test results are in line with yours, 
> we can begin looking into setting up the production environment.

Great. I'll update TRC database to inherit XL710 expectations from X710. 
It will make unnecessary to pass additional TRC tag.
The change is already published in snapshot-20231011 tag.

> First, is it possible to run the test agent on ARM hosts? Our ARM 
> testbeds have the best topology for running this test suite, with 
> separate tester and DUT servers.

It is definitely possible. Other projects run testing on ARM DUTs right 
now. Also dpdk-ethdev-ts was used on ARM hosts some time ago.
Of course, it could be surprises, but I expect it to go smoothly.

> We are testing this test suite on two x86 development servers using 
> the test suite's recommended server topology. In contrast, our 
> existing x86 production testbeds which run DTS have a single server 
> topology. This single server has both the tester NIC and the device 
> under test NIC installed, with NUMA node separation between TRex and 
> DPDK. We're going to test running the two test agent processes on the 
> single-server testbeds if we cannot run this on ARM. Is there any 
> reason you can think of that would prevent this setup from working?

Right now I see no theoretical problems with it. So, let's try and let 
me know if something goes wrong.
It should be  no problems with regular mode. It could be problems with 
AF_XDP mode testing, but it is a separate story.

> Once we figure out where this can live in production, then we will 
> begin setting up log storage, Jenkins integration, and Bublik.

Great.

I've created setup on ts-factory.io which allows to publish logs. Let me 
know if you'd like to try it and I'll provide credentials, script and 
short instruction.

Regards,
Andrew.

>
> Thanks,
> Adam
>
> On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     > Do these default to vfio-pci?
>
>     Yes, vfio-pci is the default.
>     However, it does not work in the case of Mellanox which uses
>     bifurcated driver. It should mlx5_core for Mellanox NICs.
>
>     > Here is the text log from a run on our Intel XL710 NICs, with
>     the expected result profile set to the X710.
>
>     It is hard to analyze all tests using text logs, but I definitely
>     see one common problem. Tests receive unexpected packets and fail
>     because of it.
>     Tests are written very strict from this point of view and it
>     brought fruits in the past when HW had bugs.
>     Are DUT and tester connected back-to-back on tested interfaces or
>     via switch?
>     If via switch, is it possible to isolate it from everything else?
>     If back-to-back, it could be some embedded SW/FW which regenerates
>     these packet.
>     I definitely see unexpected DHCP packets.
>
>     > We haven't set up the Jenkins integration yet, however if this
>     is required to import the logs then we will prioritize that.
>
>     Unfortunately manual runs do not generate all artifacts required
>     to import logs. However, we have almost solved it right now.
>     Hopefully we'll finalize it in a day or two. I'll let you know
>     when these changes are available.
>
>     Regards,
>     Andrew.
>
>     On 10/4/23 16:48, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set
>>     anywhere in the default configurations for the Intel X710. Do
>>     these default to vfio-pci?
>>     We have IOMMU enabled on our development testbed, and should be
>>     able to bind vfio-pci.
>>     Here is the text log from a run on our Intel XL710 NICs, with the
>>     expected result profile set to the X710. We haven't set up the
>>     Jenkins integration yet, however if this is required to import
>>     the logs then we will prioritize that.
>>     log.txt.tar.gz
>>     <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>>
>>     Thanks,
>>     Adam
>>
>>     On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         On 9/18/23 17:44, Adam Hassick wrote:
>>>         Hi Andrew and Konstantin,
>>>
>>>         Thank you for adding the tester-dial feature, this opens up
>>>         the possibility for us to do CI integrated testing in the
>>>         future.
>>>
>>>         Our Mellanox pass rate is similar to yours (about ~2400
>>>         passing, ~4400 failing), however our Intel pass rates are
>>>         far worse.
>>>         I will try running tests on the XL710 with the trc-tags
>>>         argument set and see if it improves the pass rate.
>>>         Another thing I noticed in the results you uploaded is that
>>>         the results are tagged with vfio-pci and not i40e.
>>>         Though in the environment dump, the driver on the test
>>>         machine and the DUT are set to use the i40e driver. Is this
>>>         important at all?
>>
>>         I think it is a misunderstanding here. There are two kinds of
>>         driver in configuration: net driver and so-called DPDK driver.
>>         Net driver is a Linux kernel network device driver used on
>>         Tester side.
>>         DPDK driver is a Linux kernel driver to bind device to to use
>>         it with DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
>>         In the case of bifurcated driver (like mlx5_core) it is the
>>         same in both cases.
>>         In non-bifurcated case DPDK driver is some UIO
>>         driver(vfio-pci, uio-pci-generic or igb_uio).
>>         Some expectations depend on used UIO. For example,
>>         uio-pci-generic do not support many interrupts (used by
>>         usecases/rx_intr test cases).
>>         That's why we care corresponding TRC tag.
>>
>>         TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's
>>         Intel case. Or uio-pci-generic if IOMMU is turned off on
>>         corresponding machines and Linux distro does not support VFIO
>>         no IOMMU mode.
>>
>>         Andrew.
>>
>>>         There isn't anything preventing us from pushing our results
>>>         up to the existing Bublik instance running at ts-factory.io
>>>         <http://ts-factory.io> that I can think of at the moment.
>>>         We will have to work out how to submit our results to your
>>>         Bublik instance in a controlled and secure manner in that case.
>>>         As far as I know we won't need access controls for the
>>>         results themselves. I'll discuss this with Patrick and will
>>>         let you know once we confirm that it's fine.
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             should we always auto-assign the tags or you don’t do
>>>>             it since it slows down (by some seconds) the TE startup?
>>>>
>>>
>>>             Tags are auto-assigned, but I guess it differs in Adam's
>>>             case since NIC is a bit different. Below test will help
>>>             to understand if it is the root cause of very different
>>>             expectations. If pass rate will be close to mine, I'll
>>>             simply update TRC database to share expectations for
>>>             mine NIC and NIC used by Adam.
>>>
>>>>             Hi Adam,
>>>>
>>>>             I think I second the question from Andrew - happy to
>>>>             help you with the triage so that we get to the same
>>>>             baseline. Do you have a good way for us to share the
>>>>             logs? I.e. say upload to ts-factory if we add strict
>>>>             permissions system so it’s not publishing or any other way.
>>>>
>>>>             Thanks,
>>>>             Konstantin
>>>>
>>>>
>>>>             On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 I've uploaded fresh testing results to
>>>>                 ts-factory.io <http://ts-factory.io> [1] to be on
>>>>                 the same page.
>>>>
>>>>                 I think I know why your and mine results on Intel
>>>>                 710 series NICs differ so much. Testing results
>>>>                 expectations database (dpdk-ethdev-ts/trc/*) is
>>>>                 filled in in terms of TRC tags.  I.e. expectations
>>>>                 depends on TRC tags discovered by helper scripts
>>>>                 when testing is started. These tags identify
>>>>                 various aspects of what is tested. Ideally
>>>>                 expectations should be written in terms of root
>>>>                 cause of the expected behaviour. If it is a driver
>>>>                 expectations, driver tag should be used. If it is
>>>>                 HW limitation, tags with PCI IDs should be used.
>>>>                 However, it is not always easy to classify it
>>>>                 correctly if you're not involved in driver
>>>>                 development. So, in order case expectations for
>>>>                 710's Intel are filled in in terms of PCI IDs. I
>>>>                 guess PCI ID differ in your case and that's why
>>>>                 expectations filled in for my NIC do not apply to
>>>>                 your runs.
>>>>
>>>>                 Just try to add the following option when you run
>>>>                 on your 710's Intel in order to mimic mine and see
>>>>                 if it helps to achieve better pass rate.
>>>>                 --trc-tag=pci-8086-1572
>>>>
>>>>                 BTW, fresh TE tag v1.21.0 has improved algorithm to
>>>>                 choose tests for --tester-dial option. It should
>>>>                 have better coverage now.
>>>>
>>>>                 Andrew.
>>>>
>>>>                 [1]
>>>>                 https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>>                 <https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1>
>>>>
>>>>                 On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>>                 Hi Adam,
>>>>>
>>>>>                 I've pushed new TE tag v1.20.0 which supported a
>>>>>                 new command-line option --tester-dial=NUM where
>>>>>                 NUM is from 0 to 100. it allows to choose
>>>>>                 percentage of tests to run. If you want stable
>>>>>                 set, you should pass --tester-random-seed=0 (or
>>>>>                 other integer). It is the first sketch and we have
>>>>>                 plans to improve it, but feedback would be welcome.
>>>>>
>>>>>                 > Is it needed on the tester?
>>>>>
>>>>>                 It is hard to say if it is strictly required for
>>>>>                 simple tests. However, it is better to update
>>>>>                 Tester as well, since performance tests run DPDK
>>>>>                 on Tester as well.
>>>>>
>>>>>                 > Are there any other manual setup steps for these
>>>>>                 devices that I might be missing?
>>>>>
>>>>>                 I don't remember anything else.
>>>>>
>>>>>                 I think it is better to get down to details and
>>>>>                 take a look at logs. I'm ready to help with it and
>>>>>                 explain what's happening there. May be it will
>>>>>                 help to understand if it is a problem with
>>>>>                 setup/configuration.
>>>>>
>>>>>                 Text logs are not very convenient. Ideally logs
>>>>>                 should be imported to bublik, however, manual runs
>>>>>                 do not provide all required artifacts right now
>>>>>                 (Jenkins jobs generate all required artifacts).
>>>>>                 Other option is 'tmp_raw_log' file (should be
>>>>>                 packed to make it smaller) which could be
>>>>>                 converted to various log formats.
>>>>>                 Would it be OK for you if I import your logs to
>>>>>                 bublik at ts-factory.io <http://ts-factory.io>? Or
>>>>>                 is it a problem that it is publicly available?
>>>>>                 Would it help if we add authentication and access
>>>>>                 control there?
>>>>>
>>>>>                 Andrew.
>>>>>
>>>>>                 On 9/8/23 17:57, Adam Hassick wrote:
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 I have a couple questions about needed setup of
>>>>>>                 the NICs for the ethdev test suite.
>>>>>>
>>>>>>                 Our MCX5s and XL710s are failing the checkup
>>>>>>                 tests. The pass rate appears to be much worse on
>>>>>>                 the XL710s (40 of 73 tests failed, 3 passed
>>>>>>                 unexpectedly).
>>>>>>
>>>>>>                 For the XL710s, I've updated the driver and NVM
>>>>>>                 versions to match the minimum supported versions
>>>>>>                 in the compatibility matrix found on the DPDK
>>>>>>                 documentation. This did not change the failure
>>>>>>                 rate much.
>>>>>>                 For the MCX5s, I've installed the latest LTS
>>>>>>                 version of the OFED bifurcated driver on the DUT.
>>>>>>                 Is it needed on the tester?
>>>>>>
>>>>>>                 Are there any other manual setup steps for these
>>>>>>                 devices that I might be missing?
>>>>>>
>>>>>>                 Thanks,
>>>>>>                 Adam
>>>>>>
>>>>>>                 On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick
>>>>>>                 <ahassick@iol.unh.edu> wrote:
>>>>>>
>>>>>>                     Hi Andrew,
>>>>>>
>>>>>>                     Yes, I copied the X710 configs to set up
>>>>>>                     XL710 configs. I changed the environment
>>>>>>                     variable names from the X710 suffix to XL710
>>>>>>                     suffix in the script, and forgot to change
>>>>>>                     them in the corresponding environment file.
>>>>>>                     That fixed the issue.
>>>>>>
>>>>>>                     I got the checkup tests working on the XL710
>>>>>>                     now. Most of them are failing, which leads me
>>>>>>                     to believe this is an issue with our testbed.
>>>>>>                     Based on the DPDK documentation for i40e, the
>>>>>>                     firmware and driver versions are much older
>>>>>>                     than what DPDK 22.11 LTS and main prefer, so
>>>>>>                     I'll try updating those.
>>>>>>
>>>>>>                     For now I'm working on getting the XL710
>>>>>>                     checkup tests passing, and will pick up
>>>>>>                     getting the E810 configured properly next.
>>>>>>                     I'll let you know if I run into any more
>>>>>>                     issues in relation to the test engine.
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Adam
>>>>>>
>>>>>>                     On Wed, Sep 6, 2023 at 7:36 AM Andrew
>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                         Hi Adam,
>>>>>>
>>>>>>                         On 9/5/23 18:01, Adam Hassick wrote:
>>>>>>>                         Hi Andrew,
>>>>>>>
>>>>>>>                         The compilation warning issue is now
>>>>>>>                         resolved. Again, thank you guys for
>>>>>>>                         fixing this for us. I can run the tests
>>>>>>>                         on the Mellanox CX5s again, however I'm
>>>>>>>                         running into a couple new issues with
>>>>>>>                         running the prologues on the Intel cards.
>>>>>>>
>>>>>>>                         When running testing on the Intel
>>>>>>>                         XL710s, I see this error appear in the log:
>>>>>>>
>>>>>>>                             ERROR  prologue  Environment LIB
>>>>>>>                              14:16:13.650
>>>>>>>                             Too few networks in available
>>>>>>>                             configuration (0) in comparison with
>>>>>>>                             required (1)
>>>>>>>
>>>>>>>
>>>>>>>                         This seems like a trivial configuration
>>>>>>>                         error, perhaps this is something I need
>>>>>>>                         to set up in ts-rigs. I briefly searched
>>>>>>>                         through the examples there and didn't
>>>>>>>                         see any mention of how to set up a network.
>>>>>>>                         I will attach this log just in case you
>>>>>>>                         need more information.
>>>>>>
>>>>>>                         Unfortunately logs are insufficient to
>>>>>>                         understand it. I've pushed new tag to TE
>>>>>>                         v1.19.0 which add log message with TE_*
>>>>>>                         environment variables.
>>>>>>                         Most likely something is wrong with
>>>>>>                         variables which are used as conditions
>>>>>>                         when available networks are defined in
>>>>>>                         ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>>                         TE_PCI_INSTANCE_IUT_TST1
>>>>>>                         TE_PCI_INSTANCE_IUT_TST1a
>>>>>>                         TE_PCI_INSTANCE_TST1a_IUT
>>>>>>                         TE_PCI_INSTANCE_TST1_IUT
>>>>>>                         My guess it that you change naming a bit,
>>>>>>                         but script like
>>>>>>                         ts-rigs-sample/scripts/iut.h1-x710 is not
>>>>>>                         included or not updated.
>>>>>>
>>>>>>>                         There is a different error when running
>>>>>>>                         on the Intel E810s. It appears to me
>>>>>>>                         like it starts DPDK, does some
>>>>>>>                         configuration inside DPDK and on the
>>>>>>>                         device, and then fails to bring the
>>>>>>>                         device back up. Since this error seems
>>>>>>>                         very non-trivial, I will also attach
>>>>>>>                         this log.
>>>>>>
>>>>>>                         This one is a bit simpler. Few lines
>>>>>>                         after the first ERROR in log I see the
>>>>>>                         following:
>>>>>>                         WARN  RCF  DPDK 13:06:00.144
>>>>>>                         ice_program_hw_rx_queue(): currently
>>>>>>                         package doesn't support RXDID (22)
>>>>>>                         ice_rx_queue_start(): fail to program RX
>>>>>>                         queue 0
>>>>>>                         ice_dev_start(): fail to start Rx queue 0
>>>>>>                         Device with port_id=0 already stopped
>>>>>>
>>>>>>                         It is stdout/stderr from test agent which
>>>>>>                         runs DPDK. Same logs in plain format are
>>>>>>                         available in ta.DPDK file.
>>>>>>                         I'm not an expert here, but I vaguely
>>>>>>                         remember that E810 requires correct
>>>>>>                         firmware and DDP to be loaded.
>>>>>>                         There is some information in
>>>>>>                         dpdk/doc/guides/nics/ice.rst.
>>>>>>
>>>>>>                         You can try to add
>>>>>>                         --dev-args=safe-mode-support=1
>>>>>>                         command-line option described there.
>>>>>>
>>>>>>                         Hope it helps,
>>>>>>                         Andrew.
>>>>>>
>>>>>>>
>>>>>>>                         Thanks,
>>>>>>>                         Adam
>>>>>>>
>>>>>>>                         On Fri, Sep 1, 2023 at 3:59 AM Andrew
>>>>>>>                         Rybchenko
>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>                             Hi Adam,
>>>>>>>
>>>>>>>                             On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>>>                             Hi Andrew,
>>>>>>>>
>>>>>>>>                             I have one additional question as
>>>>>>>>                             well: Does the test engine support
>>>>>>>>                             running tests on two ARMv8 test agents?
>>>>>>>>
>>>>>>>>                                 1. We'll sort out warnings this
>>>>>>>>                                 week. Thanks for heads up.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             Great. Let me know when that's fixed.
>>>>>>>
>>>>>>>                             Done. We also fixed a number of
>>>>>>>                             warnings in TE.
>>>>>>>                             Also we fixed root test package name
>>>>>>>                             to be consistent with the repository
>>>>>>>                             name.
>>>>>>>
>>>>>>>>                                 Support for old LTS branches
>>>>>>>>                                 was dropped some time ago, but
>>>>>>>>                                 in the future it is definitely
>>>>>>>>                                 possible to keep it for new LTS
>>>>>>>>                                 branches. I think 22.11 is
>>>>>>>>                                 supported, but I'm not sure
>>>>>>>>                                 about older LTS releases.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             Good to know.
>>>>>>>>
>>>>>>>>                                 2. You can add command-line
>>>>>>>>                                 option --sanity to run tests
>>>>>>>>                                 marked with TEST_HARNESS_SANITY
>>>>>>>>                                 requirement (see
>>>>>>>>                                 dpdk-ethdev-ts/scripts/run.sh
>>>>>>>>                                 and grep TEST_HARNESS_SANITY
>>>>>>>>                                 dpdk-ethdev-ts to see which
>>>>>>>>                                 tests are marked). Yes, there
>>>>>>>>                                 is a space for terminology
>>>>>>>>                                 improvement here. We'll do it.
>>>>>>>>
>>>>>>>
>>>>>>>                             Done. Now it is called --checkup.
>>>>>>>
>>>>>>>>
>>>>>>>>                                 Also it takes a lot of time
>>>>>>>>                                 because of failures and tests
>>>>>>>>                                 which wait for some timeout.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             That makes sense to me. We'll use
>>>>>>>>                             the time to complete tests on
>>>>>>>>                             virtio or the Intel devices as a
>>>>>>>>                             reference for how long the tests
>>>>>>>>                             really take to complete.
>>>>>>>>                             We will explore the possibility of
>>>>>>>>                             periodically running the sanity
>>>>>>>>                             tests for patches.
>>>>>>>
>>>>>>>                             I'll double-check and let you know
>>>>>>>                             how long entire TS runs on Intel
>>>>>>>                             X710, E810, Mellanox CX5 and virtio
>>>>>>>                             net. Just to ensure that time
>>>>>>>                             observed in your case looks the same.
>>>>>>>
>>>>>>>>
>>>>>>>>                                 The test harness can provide
>>>>>>>>                                 coverage reports based on gcov,
>>>>>>>>                                 but I'm not sure what you mean
>>>>>>>>                                 by a "dial" to control test
>>>>>>>>                                 coverage. Provided reports are
>>>>>>>>                                 rather for human to analyze.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             The general idea is to have some
>>>>>>>>                             kind of parameter on the test
>>>>>>>>                             suite, which could be an integer
>>>>>>>>                             ranging from zero to ten, that
>>>>>>>>                             controls how many tests are run
>>>>>>>>                             based on how important the test is.
>>>>>>>>
>>>>>>>>                             Similar to how some command line
>>>>>>>>                             interfaces provide a verbosity
>>>>>>>>                             level parameter (some number of
>>>>>>>>                             "-v" arguments) to control the
>>>>>>>>                             importance of the information in
>>>>>>>>                             the log.
>>>>>>>>                             The verbosity level zero only
>>>>>>>>                             prints very important log messages,
>>>>>>>>                             while ten prints everything.
>>>>>>>>
>>>>>>>>                             In much the same manner as above,
>>>>>>>>                             this "dial" parameter controls what
>>>>>>>>                             tests are run and with what
>>>>>>>>                             parameters based on how important
>>>>>>>>                             those tests and test parameter
>>>>>>>>                             combinations are.
>>>>>>>>                             Coverage Level zero tells the suite
>>>>>>>>                             to run a very basic set of
>>>>>>>>                             important tests, with minimal
>>>>>>>>                             parameterization. This mode would
>>>>>>>>                             take only ~5-10 minutes to run.
>>>>>>>>                             In contrast, Coverage Level ten
>>>>>>>>                             includes all the edge cases, every
>>>>>>>>                             combination of test parameters,
>>>>>>>>                             everything the test suite can do,
>>>>>>>>                             which takes the normal several
>>>>>>>>                             hours to run.
>>>>>>>>                             The values 1 - 9 are between those
>>>>>>>>                             two extremes, allowing the user to
>>>>>>>>                             get a gradient of test coverage in
>>>>>>>>                             the results and to limit the
>>>>>>>>                             running time.
>>>>>>>>
>>>>>>>>                             Then we could, for example, run the
>>>>>>>>                             "run.sh" with a level of 2 or 3 for
>>>>>>>>                             incoming patches that need quick
>>>>>>>>                             results, and with a level of 10 for
>>>>>>>>                             the less often run periodic tests
>>>>>>>>                             performed on main or LTS branches.
>>>>>>>
>>>>>>>                             Understood now. Thanks a lot for the
>>>>>>>                             idea. We'll discuss it and come back.
>>>>>>>
>>>>>>>>                                 3. Yes, really many tests on
>>>>>>>>                                 Mellanox CX5 NICs report
>>>>>>>>                                 unexpected testing results.
>>>>>>>>                                 Unfortunately it is time
>>>>>>>>                                 consuming to fill in
>>>>>>>>                                 expectations database since it
>>>>>>>>                                 is necessary to analyze testing
>>>>>>>>                                 results and classify if it is a
>>>>>>>>                                 bug or just acceptable
>>>>>>>>                                 behaviour aspect.
>>>>>>>>
>>>>>>>>                                 Bublik allows to compare
>>>>>>>>                                 results of two runs. It is
>>>>>>>>                                 useful for human, but still not
>>>>>>>>                                 good for automation.
>>>>>>>>
>>>>>>>>                                 I have local patch for mlx5
>>>>>>>>                                 driver which reports Tx ring
>>>>>>>>                                 size maximum. It makes pass
>>>>>>>>                                 rate higher. It is a problem
>>>>>>>>                                 for test harness that mlx5 does
>>>>>>>>                                 not report limits right now.
>>>>>>>>
>>>>>>>>                                 Pass rate on Intel X710 is
>>>>>>>>                                 about 92% on my test rig. Pass
>>>>>>>>                                 rate on virtio net is 99% right
>>>>>>>>                                 now and could be done 100%
>>>>>>>>                                 easily (just one thing to fix
>>>>>>>>                                 in expectations).
>>>>>>>>
>>>>>>>>                                 I think logs storage setup is
>>>>>>>>                                 essential for logs analysis. Of
>>>>>>>>                                 course, you can request HTML
>>>>>>>>                                 logs when you run tests
>>>>>>>>                                 (--log-html=html) or generate
>>>>>>>>                                 after run using
>>>>>>>>                                 dpdk-ethdev-ts/scripts/html-log.sh
>>>>>>>>                                 and open index.html in a
>>>>>>>>                                 browser, but logs storage makes
>>>>>>>>                                 it more convenient.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             We are interested in setting up
>>>>>>>>                             Bublik, potentially as an
>>>>>>>>                             externally-facing component, once
>>>>>>>>                             we have our process of running the
>>>>>>>>                             test suite stabilized.
>>>>>>>>                             Once we are able to run the test
>>>>>>>>                             suite again, I'll see what the pass
>>>>>>>>                             rate is on our other hardware.
>>>>>>>>                             Good to know that it isn't an issue
>>>>>>>>                             with our dev testbed causing the
>>>>>>>>                             high fail rate.
>>>>>>>>
>>>>>>>>                             For Intel hardware, we have an
>>>>>>>>                             XL710 and an Intel E810-C in our
>>>>>>>>                             development testbed. Although they
>>>>>>>>                             are slightly different devices,
>>>>>>>>                             ideally the pass rate will be
>>>>>>>>                             identical or similar. I have yet to
>>>>>>>>                             set up a VM pair for virtio, but we
>>>>>>>>                             will soon.
>>>>>>>>
>>>>>>>>                                 Latest version of
>>>>>>>>                                 test-environment has examples
>>>>>>>>                                 of our CGI scripts which we use
>>>>>>>>                                 for log storage (see
>>>>>>>>                                 tools/log_server/README.md).
>>>>>>>>
>>>>>>>>                                 Also all bits for Jenkins setup
>>>>>>>>                                 are available. See
>>>>>>>>                                 dpdk-ethdev-ts/jenkins/README.md
>>>>>>>>                                 and examples of jenkins files
>>>>>>>>                                 in ts-rigs-sample.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             Jenkins integration, setting up
>>>>>>>>                             production rig configurations, and
>>>>>>>>                             permanent log storage will be our
>>>>>>>>                             next steps once I am able to run
>>>>>>>>                             the tests again.
>>>>>>>>                             Unless there is an easy way to have
>>>>>>>>                             meson not pass "-Werror" into GCC.
>>>>>>>>                             Then I would be able to run the
>>>>>>>>                             test suite.
>>>>>>>
>>>>>>>                             Hopefully it is resolved now.
>>>>>>>
>>>>>>>                             I thought a bit more about your
>>>>>>>                             usecase for Jenkins. I'm not 100%
>>>>>>>                             sure that existing pipelines are
>>>>>>>                             convenient for your usecase.
>>>>>>>                             Fill free to ask questions when you
>>>>>>>                             are on it.
>>>>>>>
>>>>>>>                             Thanks,
>>>>>>>                             Andrew.
>>>>>>>
>>>>>>>>
>>>>>>>>                             Thanks,
>>>>>>>>                             Adam
>>>>>>>>
>>>>>>>>
>>>>>>>>                                 On 8/29/23 17:02, Adam Hassick
>>>>>>>>                                 wrote:
>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>
>>>>>>>>>                                 That fix seems to have
>>>>>>>>>                                 resolved the issue, thanks for
>>>>>>>>>                                 the quick turnaround time on
>>>>>>>>>                                 that patch.
>>>>>>>>>                                 Now that we have the RCF
>>>>>>>>>                                 timeout issue resolved, there
>>>>>>>>>                                 are a few other questions and
>>>>>>>>>                                 issues that we have about the
>>>>>>>>>                                 tests themselves.
>>>>>>>>>
>>>>>>>>>                                 1. The test suite fails to
>>>>>>>>>                                 build with a couple warnings.
>>>>>>>>>
>>>>>>>>>                                 Below is the stderr log from
>>>>>>>>>                                 compilation:
>>>>>>>>>
>>>>>>>>>                                     FAILED:
>>>>>>>>>                                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>                                     cc
>>>>>>>>>                                     -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>                                     -Ilib -I../../lib
>>>>>>>>>                                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>                                     -fdiagnostics-color=always
>>>>>>>>>                                     -pipe
>>>>>>>>>                                     -D_FILE_OFFSET_BITS=64
>>>>>>>>>                                     -Wall -Winvalid-pch
>>>>>>>>>                                     -Werror -g -D_GNU_SOURCE
>>>>>>>>>                                     -O0 -ggdb -Wall -W -fPIC
>>>>>>>>>                                     -MD -MQ
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -MF
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>                                     -o
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>                                     In function
>>>>>>>>>                                     ‘test_create_traffic_generator_params’:
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>                                     error: format not a string
>>>>>>>>>                                     literal and no format
>>>>>>>>>                                     arguments
>>>>>>>>>                                     [-Werror=format-security]
>>>>>>>>>                                     5577 |     rc =
>>>>>>>>>                                     te_kvpair_add(result, buf,
>>>>>>>>>                                     mode);
>>>>>>>>>                                     |     ^~
>>>>>>>>>                                     cc1: all warnings being
>>>>>>>>>                                     treated as errors
>>>>>>>>>                                     ninja: build stopped:
>>>>>>>>>                                     subcommand failed.
>>>>>>>>>                                     ninja: Entering directory `.'
>>>>>>>>>                                     FAILED:
>>>>>>>>>                                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>                                     cc
>>>>>>>>>                                     -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>                                     -Ilib -I../../lib
>>>>>>>>>                                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>                                     -fdiagnostics-color=always
>>>>>>>>>                                     -pipe
>>>>>>>>>                                     -D_FILE_OFFSET_BITS=64
>>>>>>>>>                                     -Wall -Winvalid-pch
>>>>>>>>>                                     -Werror -g -D_GNU_SOURCE
>>>>>>>>>                                     -O0 -ggdb -Wall -W -fPIC
>>>>>>>>>                                     -MD -MQ
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -MF
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>                                     -o
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>                                     In function
>>>>>>>>>                                     ‘test_create_traffic_generator_params’:
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>                                     error: format not a string
>>>>>>>>>                                     literal and no format
>>>>>>>>>                                     arguments
>>>>>>>>>                                     [-Werror=format-security]
>>>>>>>>>                                     5577 |     rc =
>>>>>>>>>                                     te_kvpair_add(result, buf,
>>>>>>>>>                                     mode);
>>>>>>>>>                                     |     ^~
>>>>>>>>>                                     cc1: all warnings being
>>>>>>>>>                                     treated as errors
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 This error wasn't occurring
>>>>>>>>>                                 last week, which was the last
>>>>>>>>>                                 time I ran the tests.
>>>>>>>>>                                 The TE host and the DUT have
>>>>>>>>>                                 GCC v9.4.0 installed, and the
>>>>>>>>>                                 tester has GCC v11.4.0
>>>>>>>>>                                 installed, if this information
>>>>>>>>>                                 is helpful.
>>>>>>>>>
>>>>>>>>>                                 2. On the Mellanox CX5s, there
>>>>>>>>>                                 are over 6,000 tests run,
>>>>>>>>>                                 which collectively take around
>>>>>>>>>                                 9 hours. Is it possible, and
>>>>>>>>>                                 would it make sense, to lower
>>>>>>>>>                                 the test coverage and have the
>>>>>>>>>                                 test suite run faster?
>>>>>>>>>
>>>>>>>>>                                 For some context, we run
>>>>>>>>>                                 immediate testing on incoming
>>>>>>>>>                                 patches for DPDK main and
>>>>>>>>>                                 development branches, as well
>>>>>>>>>                                 as periodic test runs on the
>>>>>>>>>                                 main, stable, and LTS branches.
>>>>>>>>>                                 For us to consider including
>>>>>>>>>                                 this test suite as part of our
>>>>>>>>>                                 immediate testing on patches,
>>>>>>>>>                                 we would have to reduce the
>>>>>>>>>                                 test coverage to the most
>>>>>>>>>                                 important tests.
>>>>>>>>>                                 This is primarily to reduce
>>>>>>>>>                                 the testing time to, for
>>>>>>>>>                                 example, less than 30 minutes.
>>>>>>>>>                                 Testing on patches can't take
>>>>>>>>>                                 too long because the lab can
>>>>>>>>>                                 receive numerous patches each
>>>>>>>>>                                 day, which each require
>>>>>>>>>                                 individual testing runs.
>>>>>>>>>
>>>>>>>>>                                 At what frequency we run these
>>>>>>>>>                                 tests, and on what, still
>>>>>>>>>                                 needs to be discussed with the
>>>>>>>>>                                 DPDK community, but it would
>>>>>>>>>                                 be nice to know if the test
>>>>>>>>>                                 suite had a "dial" to control
>>>>>>>>>                                 the testing coverage.
>>>>>>>>>
>>>>>>>>>                                 3. We see a lot of test
>>>>>>>>>                                 failures on our Mellanox CX5
>>>>>>>>>                                 NICs. Around 2,300 of ~6,600
>>>>>>>>>                                 tests passed. Is there
>>>>>>>>>                                 anything we can do to diagnose
>>>>>>>>>                                 these test failures?
>>>>>>>>>
>>>>>>>>>                                 Thanks,
>>>>>>>>>                                 Adam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 On Tue, Aug 29, 2023 at
>>>>>>>>>                                 8:07 AM Andrew Rybchenko
>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>                                 wrote:
>>>>>>>>>
>>>>>>>>>                                     Hi Adam,
>>>>>>>>>
>>>>>>>>>                                     I've pushed the fix in
>>>>>>>>>                                     main branch and a new tag
>>>>>>>>>                                     v1.18.1. It should solve
>>>>>>>>>                                     the problem with IPv6
>>>>>>>>>                                     address from DNS.
>>>>>>>>>
>>>>>>>>>                                     Andrew.
>>>>>>>>>
>>>>>>>>>                                     On 8/29/23 00:05, Andrew
>>>>>>>>>                                     Rybchenko wrote:
>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                                     > Does the test engine
>>>>>>>>>>                                     prefer to use IPv6 over
>>>>>>>>>>                                     IPv4 for initiating the
>>>>>>>>>>                                     RCF connection to the
>>>>>>>>>>                                     test bed hosts? And if
>>>>>>>>>>                                     so, is there a way to
>>>>>>>>>>                                     force it to use IPv4?
>>>>>>>>>>
>>>>>>>>>>                                     Brilliant idea. If DNS
>>>>>>>>>>                                     returns both IPv4 and
>>>>>>>>>>                                     IPv6 addresses in your
>>>>>>>>>>                                     case, I guess it is the
>>>>>>>>>>                                     root cause of the problem.
>>>>>>>>>>                                     Of course, it is TE
>>>>>>>>>>                                     problem since I see
>>>>>>>>>>                                     really weird code in
>>>>>>>>>>                                     lib/comm_net_engine/comm_net_engine.c
>>>>>>>>>>                                     line 135.
>>>>>>>>>>
>>>>>>>>>>                                     I've pushed fix to the
>>>>>>>>>>                                     branch
>>>>>>>>>>                                     user/arybchik/fix_ipv4_only
>>>>>>>>>>                                     in
>>>>>>>>>>                                     ts-factory/test-environment
>>>>>>>>>>                                     repository. Please, try.
>>>>>>>>>>
>>>>>>>>>>                                     It is late night fix with
>>>>>>>>>>                                     minimal testing and no
>>>>>>>>>>                                     review. I'll pass it
>>>>>>>>>>                                     through review process
>>>>>>>>>>                                     tomorrow and
>>>>>>>>>>                                     hopefully it will be
>>>>>>>>>>                                     released in one-two days.
>>>>>>>>>>
>>>>>>>>>>                                     Andrew.
>>>>>>>>>>
>>>>>>>>>>                                     On 8/28/23 18:02, Adam
>>>>>>>>>>                                     Hassick wrote:
>>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                                     We have yet to notice a
>>>>>>>>>>>                                     distinct pattern with
>>>>>>>>>>>                                     the failures. Sometimes,
>>>>>>>>>>>                                     the RCF will start and
>>>>>>>>>>>                                     connect without issue a
>>>>>>>>>>>                                     few times in a row
>>>>>>>>>>>                                     before failing to
>>>>>>>>>>>                                     connect again. Once the
>>>>>>>>>>>                                     issue begins to occur,
>>>>>>>>>>>                                     neither rebooting all of
>>>>>>>>>>>                                     the hosts (test engine
>>>>>>>>>>>                                     VM, tester, IUT) or
>>>>>>>>>>>                                     deleting all of the
>>>>>>>>>>>                                     build directories
>>>>>>>>>>>                                     (suites, agents, inst)
>>>>>>>>>>>                                     and rebooting the hosts
>>>>>>>>>>>                                     afterward resolves the
>>>>>>>>>>>                                     issue. When it begins
>>>>>>>>>>>                                     working again seems very
>>>>>>>>>>>                                     arbitrary to us.
>>>>>>>>>>>
>>>>>>>>>>>                                     I do usually try to
>>>>>>>>>>>                                     terminate the test
>>>>>>>>>>>                                     engine with Ctrl+C, but
>>>>>>>>>>>                                     when it hangs while
>>>>>>>>>>>                                     trying to start RCF,
>>>>>>>>>>>                                     that does not work.
>>>>>>>>>>>
>>>>>>>>>>>                                     Does the test engine
>>>>>>>>>>>                                     prefer to use IPv6 over
>>>>>>>>>>>                                     IPv4 for initiating the
>>>>>>>>>>>                                     RCF connection to the
>>>>>>>>>>>                                     test bed hosts? And if
>>>>>>>>>>>                                     so, is there a way to
>>>>>>>>>>>                                     force it to use IPv4?
>>>>>>>>>>>
>>>>>>>>>>>                                      - Adam
>>>>>>>>>>>
>>>>>>>>>>>                                     On Fri, Aug 25, 2023 at
>>>>>>>>>>>                                     1:35 PM Andrew Rybchenko
>>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                         > I'll double-check
>>>>>>>>>>>                                         test engine on
>>>>>>>>>>>                                         Ubuntu 20.04 and
>>>>>>>>>>>                                         Ubuntu 22.04.
>>>>>>>>>>>
>>>>>>>>>>>                                         Done. It works fine
>>>>>>>>>>>                                         for me without any
>>>>>>>>>>>                                         issues.
>>>>>>>>>>>
>>>>>>>>>>>                                         Have you noticed any
>>>>>>>>>>>                                         pattern when it
>>>>>>>>>>>                                         works or does not work?
>>>>>>>>>>>                                         May be it is a
>>>>>>>>>>>                                         problem of not clean
>>>>>>>>>>>                                         state after termination?
>>>>>>>>>>>                                         Does it work fine
>>>>>>>>>>>                                         the first time after
>>>>>>>>>>>                                         DUTs reboot?
>>>>>>>>>>>                                         How do you terminate
>>>>>>>>>>>                                         testing? It should
>>>>>>>>>>>                                         be done using Ctrl+C
>>>>>>>>>>>                                         in terminal where
>>>>>>>>>>>                                         you execute run.sh
>>>>>>>>>>>                                         command.
>>>>>>>>>>>                                          In this case it
>>>>>>>>>>>                                         should shutdown
>>>>>>>>>>>                                         gracefully and close
>>>>>>>>>>>                                         all test agents and
>>>>>>>>>>>                                         engine applications.
>>>>>>>>>>>
>>>>>>>>>>>                                         (I'm trying to
>>>>>>>>>>>                                         understand why
>>>>>>>>>>>                                         you've seen many
>>>>>>>>>>>                                         test agent
>>>>>>>>>>>                                         processes. It should
>>>>>>>>>>>                                         not happen.)
>>>>>>>>>>>
>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                         On 8/25/23 17:41,
>>>>>>>>>>>                                         Andrew Rybchenko wrote:
>>>>>>>>>>>>                                         On 8/25/23 17:06,
>>>>>>>>>>>>                                         Adam Hassick wrote:
>>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Two of our systems
>>>>>>>>>>>>>                                         (the Test Engine
>>>>>>>>>>>>>                                         runner and the DUT
>>>>>>>>>>>>>                                         host) are running
>>>>>>>>>>>>>                                         Ubuntu 20.04 LTS,
>>>>>>>>>>>>>                                         however this
>>>>>>>>>>>>>                                         morning I noticed
>>>>>>>>>>>>>                                         that the tester
>>>>>>>>>>>>>                                         system (the one
>>>>>>>>>>>>>                                         having issues) is
>>>>>>>>>>>>>                                         running Ubuntu
>>>>>>>>>>>>>                                         22.04 LTS.
>>>>>>>>>>>>>                                         This could be the
>>>>>>>>>>>>>                                         source of the
>>>>>>>>>>>>>                                         problem. I
>>>>>>>>>>>>>                                         encountered a
>>>>>>>>>>>>>                                         dependency issue
>>>>>>>>>>>>>                                         trying to run the
>>>>>>>>>>>>>                                         Test Engine on
>>>>>>>>>>>>>                                         22.04 LTS, so I
>>>>>>>>>>>>>                                         downgraded the
>>>>>>>>>>>>>                                         system. Since the
>>>>>>>>>>>>>                                         tester is also the
>>>>>>>>>>>>>                                         host having
>>>>>>>>>>>>>                                         connection issues,
>>>>>>>>>>>>>                                         I will try
>>>>>>>>>>>>>                                         downgrading that
>>>>>>>>>>>>>                                         system to 20.04,
>>>>>>>>>>>>>                                         and see if that
>>>>>>>>>>>>>                                         changes anything.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         Unlikely, but who
>>>>>>>>>>>>                                         knows. We run tests
>>>>>>>>>>>>                                         (DUTs) on Ubuntu
>>>>>>>>>>>>                                         20.04, Ubuntu
>>>>>>>>>>>>                                         22.04, Ubuntu
>>>>>>>>>>>>                                         22.10, Ubuntu
>>>>>>>>>>>>                                         23.04, Debian 11
>>>>>>>>>>>>                                         and Fedora 38 every
>>>>>>>>>>>>                                         night.
>>>>>>>>>>>>                                         Right now Debian 11
>>>>>>>>>>>>                                         is used for test
>>>>>>>>>>>>                                         engine in nightly
>>>>>>>>>>>>                                         regressions.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         I'll double-check
>>>>>>>>>>>>                                         test engine on
>>>>>>>>>>>>                                         Ubuntu 20.04 and
>>>>>>>>>>>>                                         Ubuntu 22.04.
>>>>>>>>>>>>
>>>>>>>>>>>>>                                         I did try passing
>>>>>>>>>>>>>                                         in the "--vg-rcf"
>>>>>>>>>>>>>                                         argument to the
>>>>>>>>>>>>>                                         run.sh script of
>>>>>>>>>>>>>                                         the test suite
>>>>>>>>>>>>>                                         after installing
>>>>>>>>>>>>>                                         valgrind, but
>>>>>>>>>>>>>                                         there was no
>>>>>>>>>>>>>                                         additional output
>>>>>>>>>>>>>                                         that I saw.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         Sorry, I should
>>>>>>>>>>>>                                         valgrind output
>>>>>>>>>>>>                                         should be in
>>>>>>>>>>>>                                         valgrind.te_rcf
>>>>>>>>>>>>                                         (direction where
>>>>>>>>>>>>                                         you run test engine).
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         I will try pulling
>>>>>>>>>>>>>                                         in the changes
>>>>>>>>>>>>>                                         you've pushed up,
>>>>>>>>>>>>>                                         and will see if
>>>>>>>>>>>>>                                         that fixes anything.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>                                         Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         On Fri, Aug 25,
>>>>>>>>>>>>>                                         2023 at 9:57 AM
>>>>>>>>>>>>>                                         Andrew Rybchenko
>>>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Hello Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             On 8/24/23
>>>>>>>>>>>>>                                             23:54, Andrew
>>>>>>>>>>>>>                                             Rybchenko wrote:
>>>>>>>>>>>>>>                                             I'd like to
>>>>>>>>>>>>>>                                             try to repeat
>>>>>>>>>>>>>>                                             the problem
>>>>>>>>>>>>>>                                             locally.
>>>>>>>>>>>>>>                                             Which Linux
>>>>>>>>>>>>>>                                             distro is
>>>>>>>>>>>>>>                                             running on
>>>>>>>>>>>>>>                                             test engine
>>>>>>>>>>>>>>                                             and agents?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             In fact I
>>>>>>>>>>>>>>                                             know one
>>>>>>>>>>>>>>                                             problem with
>>>>>>>>>>>>>>                                             Debian 12 and
>>>>>>>>>>>>>>                                             Fedora 38 and
>>>>>>>>>>>>>>                                             we have
>>>>>>>>>>>>>>                                             patch in
>>>>>>>>>>>>>>                                             review to fix
>>>>>>>>>>>>>>                                             it, however,
>>>>>>>>>>>>>>                                             the behaviour
>>>>>>>>>>>>>>                                             is different in
>>>>>>>>>>>>>>                                             this case, so
>>>>>>>>>>>>>>                                             it is unlike
>>>>>>>>>>>>>>                                             the same problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             I've just
>>>>>>>>>>>>>                                             published a
>>>>>>>>>>>>>                                             new tag which
>>>>>>>>>>>>>                                             fixes known
>>>>>>>>>>>>>                                             test engine
>>>>>>>>>>>>>                                             side problems
>>>>>>>>>>>>>                                             on Debian 12
>>>>>>>>>>>>>                                             and Fedora 38.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             One more idea
>>>>>>>>>>>>>>                                             is to install
>>>>>>>>>>>>>>                                             valgrind on
>>>>>>>>>>>>>>                                             the test
>>>>>>>>>>>>>>                                             engine host and
>>>>>>>>>>>>>>                                             run with
>>>>>>>>>>>>>>                                             option
>>>>>>>>>>>>>>                                             --vg-rcf to
>>>>>>>>>>>>>>                                             check if
>>>>>>>>>>>>>>                                             something
>>>>>>>>>>>>>>                                             weird is
>>>>>>>>>>>>>>                                             happening.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             What I don't
>>>>>>>>>>>>>>                                             understand
>>>>>>>>>>>>>>                                             right now is
>>>>>>>>>>>>>>                                             why I see
>>>>>>>>>>>>>>                                             just one
>>>>>>>>>>>>>>                                             failed attempt
>>>>>>>>>>>>>>                                             to connect in
>>>>>>>>>>>>>>                                             your log.txt
>>>>>>>>>>>>>>                                             and then
>>>>>>>>>>>>>>                                             Logger
>>>>>>>>>>>>>>                                             shutdown
>>>>>>>>>>>>>>                                             after 9 minutes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On 8/24/23
>>>>>>>>>>>>>>                                             23:29, Adam
>>>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>>>                                              > Is there
>>>>>>>>>>>>>>>                                             any firewall
>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>                                             network or
>>>>>>>>>>>>>>>                                             on test
>>>>>>>>>>>>>>>                                             hosts which
>>>>>>>>>>>>>>>                                             could block
>>>>>>>>>>>>>>>                                             incoming TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             to the port
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             from the
>>>>>>>>>>>>>>>                                             host where
>>>>>>>>>>>>>>>                                             you run test
>>>>>>>>>>>>>>>                                             engine?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             Our test
>>>>>>>>>>>>>>>                                             engine host
>>>>>>>>>>>>>>>                                             and the
>>>>>>>>>>>>>>>                                             testbed are
>>>>>>>>>>>>>>>                                             on the same
>>>>>>>>>>>>>>>                                             subnet. The
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             does work
>>>>>>>>>>>>>>>                                             sometimes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                              > If
>>>>>>>>>>>>>>>                                             behaviour
>>>>>>>>>>>>>>>                                             the same on
>>>>>>>>>>>>>>>                                             the next try
>>>>>>>>>>>>>>>                                             and you see
>>>>>>>>>>>>>>>                                             that test
>>>>>>>>>>>>>>>                                             agent is
>>>>>>>>>>>>>>>                                             kept
>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>                                             could you
>>>>>>>>>>>>>>>                                             check using
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > # netstat
>>>>>>>>>>>>>>>                                             -tnlp
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > that Test
>>>>>>>>>>>>>>>                                             Agent is
>>>>>>>>>>>>>>>                                             listening on
>>>>>>>>>>>>>>>                                             the port and
>>>>>>>>>>>>>>>                                             try to
>>>>>>>>>>>>>>>                                             establish
>>>>>>>>>>>>>>>                                             TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             from test
>>>>>>>>>>>>>>>                                             agent using
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > $ telnet
>>>>>>>>>>>>>>>                                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > and check
>>>>>>>>>>>>>>>                                             if TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             could be
>>>>>>>>>>>>>>>                                             established.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             I was able
>>>>>>>>>>>>>>>                                             to replicate
>>>>>>>>>>>>>>>                                             the same
>>>>>>>>>>>>>>>                                             behavior
>>>>>>>>>>>>>>>                                             again, where
>>>>>>>>>>>>>>>                                             it hangs
>>>>>>>>>>>>>>>                                             while RCF is
>>>>>>>>>>>>>>>                                             trying to start.
>>>>>>>>>>>>>>>                                             Running this
>>>>>>>>>>>>>>>                                             command, I
>>>>>>>>>>>>>>>                                             see this in
>>>>>>>>>>>>>>>                                             the output:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             tcp        0
>>>>>>>>>>>>>>>                                                0
>>>>>>>>>>>>>>>                                             0.0.0.0:23571
>>>>>>>>>>>>>>>                                             <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>                                             <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>                                             <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>                                             0.0.0.0:*
>>>>>>>>>>>>>>>                                             LISTEN  18599/ta
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             So it seems
>>>>>>>>>>>>>>>                                             like it is
>>>>>>>>>>>>>>>                                             listening on
>>>>>>>>>>>>>>>                                             the correct
>>>>>>>>>>>>>>>                                             port.
>>>>>>>>>>>>>>>                                             Additionally,
>>>>>>>>>>>>>>>                                             I was able
>>>>>>>>>>>>>>>                                             to connect
>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>                                             machine from
>>>>>>>>>>>>>>>                                             our Test
>>>>>>>>>>>>>>>                                             Engine host
>>>>>>>>>>>>>>>                                             using
>>>>>>>>>>>>>>>                                             telnet. It
>>>>>>>>>>>>>>>                                             printed the
>>>>>>>>>>>>>>>                                             PID of the
>>>>>>>>>>>>>>>                                             process once
>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             was opened.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             I tried
>>>>>>>>>>>>>>>                                             running the
>>>>>>>>>>>>>>>                                             "ta"
>>>>>>>>>>>>>>>                                             application
>>>>>>>>>>>>>>>                                             manually on
>>>>>>>>>>>>>>>                                             the command
>>>>>>>>>>>>>>>                                             line, and it
>>>>>>>>>>>>>>>                                             didn't print
>>>>>>>>>>>>>>>                                             anything at all.
>>>>>>>>>>>>>>>                                             Maybe the
>>>>>>>>>>>>>>>                                             issue is
>>>>>>>>>>>>>>>                                             something on
>>>>>>>>>>>>>>>                                             the Test
>>>>>>>>>>>>>>>                                             Engine side.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             On Thu, Aug
>>>>>>>>>>>>>>>                                             24, 2023 at
>>>>>>>>>>>>>>>                                             2:35 PM
>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Hi Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                  > On
>>>>>>>>>>>>>>>                                             the tester
>>>>>>>>>>>>>>>                                             host (which
>>>>>>>>>>>>>>>                                             appears to
>>>>>>>>>>>>>>>                                             be the Peer
>>>>>>>>>>>>>>>                                             agent), there
>>>>>>>>>>>>>>>                                                 are four
>>>>>>>>>>>>>>>                                             processes
>>>>>>>>>>>>>>>                                             that I see
>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>                                             which look
>>>>>>>>>>>>>>>                                             like the test
>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>                                             processes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Before
>>>>>>>>>>>>>>>                                             the next try
>>>>>>>>>>>>>>>                                             I'd
>>>>>>>>>>>>>>>                                             recommend to
>>>>>>>>>>>>>>>                                             kill these
>>>>>>>>>>>>>>>                                             processes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Is there
>>>>>>>>>>>>>>>                                             any firewall
>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>                                             network or
>>>>>>>>>>>>>>>                                             on test
>>>>>>>>>>>>>>>                                             hosts which
>>>>>>>>>>>>>>>                                             could
>>>>>>>>>>>>>>>                                                 block
>>>>>>>>>>>>>>>                                             incoming TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             to the port
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             from the host
>>>>>>>>>>>>>>>                                                 where
>>>>>>>>>>>>>>>                                             you run test
>>>>>>>>>>>>>>>                                             engine?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 If
>>>>>>>>>>>>>>>                                             behaviour
>>>>>>>>>>>>>>>                                             the same on
>>>>>>>>>>>>>>>                                             the next try
>>>>>>>>>>>>>>>                                             and you see
>>>>>>>>>>>>>>>                                             that test
>>>>>>>>>>>>>>>                                             agent is
>>>>>>>>>>>>>>>                                                 kept
>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>                                             could you
>>>>>>>>>>>>>>>                                             check using
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 #
>>>>>>>>>>>>>>>                                             netstat -tnlp
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>                                             Test Agent
>>>>>>>>>>>>>>>                                             is listening
>>>>>>>>>>>>>>>                                             on the port
>>>>>>>>>>>>>>>                                             and try to
>>>>>>>>>>>>>>>                                             establish TCP
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             from test
>>>>>>>>>>>>>>>                                             agent using
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 $ telnet
>>>>>>>>>>>>>>>                                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>                                             check if TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             could be
>>>>>>>>>>>>>>>                                             established.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Another
>>>>>>>>>>>>>>>                                             idea is to
>>>>>>>>>>>>>>>                                             login Tester
>>>>>>>>>>>>>>>                                             under root
>>>>>>>>>>>>>>>                                             as testing
>>>>>>>>>>>>>>>                                             does, get
>>>>>>>>>>>>>>>                                                 start TA
>>>>>>>>>>>>>>>                                             command from
>>>>>>>>>>>>>>>                                             the log and
>>>>>>>>>>>>>>>                                             try it by
>>>>>>>>>>>>>>>                                             hands
>>>>>>>>>>>>>>>                                             without -n and
>>>>>>>>>>>>>>>                                                 remove
>>>>>>>>>>>>>>>                                             extra escaping.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 # sudo
>>>>>>>>>>>>>>>                                             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>                                             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>                                             /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>>>>                                             Peer 23571
>>>>>>>>>>>>>>>                                             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                                             Hopefully in
>>>>>>>>>>>>>>>                                             this case
>>>>>>>>>>>>>>>                                             test agent
>>>>>>>>>>>>>>>                                             directory
>>>>>>>>>>>>>>>                                             remains in
>>>>>>>>>>>>>>>                                             the /tmp and
>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>                                             don't need
>>>>>>>>>>>>>>>                                             to copy it
>>>>>>>>>>>>>>>                                             as testing does.
>>>>>>>>>>>>>>>                                                 May be
>>>>>>>>>>>>>>>                                             output could
>>>>>>>>>>>>>>>                                             shed some
>>>>>>>>>>>>>>>                                             light on
>>>>>>>>>>>>>>>                                             what's going on.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>                                             8/24/23
>>>>>>>>>>>>>>>                                             17:30, Adam
>>>>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 This is
>>>>>>>>>>>>>>>>                                             the output
>>>>>>>>>>>>>>>>                                             that I see
>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>                                             terminal
>>>>>>>>>>>>>>>>                                             when this
>>>>>>>>>>>>>>>>                                             failure
>>>>>>>>>>>>>>>>                                                 occurs,
>>>>>>>>>>>>>>>>                                             after the
>>>>>>>>>>>>>>>>                                             test agent
>>>>>>>>>>>>>>>>                                             binaries
>>>>>>>>>>>>>>>>                                             build and
>>>>>>>>>>>>>>>>                                             the test engine
>>>>>>>>>>>>>>>>                                                 starts:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                             Platform
>>>>>>>>>>>>>>>>                                             default
>>>>>>>>>>>>>>>>                                             build - pass
>>>>>>>>>>>>>>>>                                                 Simple
>>>>>>>>>>>>>>>>                                             RCF
>>>>>>>>>>>>>>>>                                             consistency
>>>>>>>>>>>>>>>>                                             check succeeded
>>>>>>>>>>>>>>>>                                             --->>>
>>>>>>>>>>>>>>>>                                             Starting
>>>>>>>>>>>>>>>>                                             Logger...done
>>>>>>>>>>>>>>>>                                             --->>>
>>>>>>>>>>>>>>>>                                             Starting
>>>>>>>>>>>>>>>>                                             RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>>>>                                             Connection
>>>>>>>>>>>>>>>>                                             timed
>>>>>>>>>>>>>>>>                                                 out
>>>>>>>>>>>>>>>>                                             iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 Then,
>>>>>>>>>>>>>>>>                                             it hangs
>>>>>>>>>>>>>>>>                                             here until
>>>>>>>>>>>>>>>>                                             I kill the
>>>>>>>>>>>>>>>>                                             "te_rcf"
>>>>>>>>>>>>>>>>                                             and "te_tee"
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                             processes.
>>>>>>>>>>>>>>>>                                             I let it
>>>>>>>>>>>>>>>>                                             hang for
>>>>>>>>>>>>>>>>                                             around 9
>>>>>>>>>>>>>>>>                                             minutes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 On the
>>>>>>>>>>>>>>>>                                             tester host
>>>>>>>>>>>>>>>>                                             (which
>>>>>>>>>>>>>>>>                                             appears to
>>>>>>>>>>>>>>>>                                             be the Peer
>>>>>>>>>>>>>>>>                                             agent),
>>>>>>>>>>>>>>>>                                             there are
>>>>>>>>>>>>>>>>                                                 four
>>>>>>>>>>>>>>>>                                             processes
>>>>>>>>>>>>>>>>                                             that I see
>>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>>                                             which look
>>>>>>>>>>>>>>>>                                             like the
>>>>>>>>>>>>>>>>                                             test agent
>>>>>>>>>>>>>>>>                                                 processes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 ta.Peer
>>>>>>>>>>>>>>>>                                             is an empty
>>>>>>>>>>>>>>>>                                             file. I've
>>>>>>>>>>>>>>>>                                             attached
>>>>>>>>>>>>>>>>                                             the log.txt
>>>>>>>>>>>>>>>>                                             from this run.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                  - Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 On Thu,
>>>>>>>>>>>>>>>>                                             Aug 24,
>>>>>>>>>>>>>>>>                                             2023 at
>>>>>>>>>>>>>>>>                                             4:22 AM
>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>                                             Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             Yes,
>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>                                             is in
>>>>>>>>>>>>>>>>                                             seconds.
>>>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>>>                                             double-checked
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             that it
>>>>>>>>>>>>>>>>                                             goes to
>>>>>>>>>>>>>>>>                                             'copy_timeout'
>>>>>>>>>>>>>>>>                                             in
>>>>>>>>>>>>>>>>                                             ts-conf/rcf.conf.
>>>>>>>>>>>>>>>>                                             Description
>>>>>>>>>>>>>>>>                                             in in
>>>>>>>>>>>>>>>>                                             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             says that
>>>>>>>>>>>>>>>>                                             copy_timeout
>>>>>>>>>>>>>>>>                                             is in
>>>>>>>>>>>>>>>>                                             seconds and
>>>>>>>>>>>>>>>>                                             implementation
>>>>>>>>>>>>>>>>                                             in
>>>>>>>>>>>>>>>>                                             lib/rcfunix/rcfunix.c
>>>>>>>>>>>>>>>>                                             passes the
>>>>>>>>>>>>>>>>                                             value to
>>>>>>>>>>>>>>>>                                             select()
>>>>>>>>>>>>>>>>                                             tv_sec.
>>>>>>>>>>>>>>>>                                             Theoretically
>>>>>>>>>>>>>>>>                                             select()
>>>>>>>>>>>>>>>>                                             could be
>>>>>>>>>>>>>>>>                                             interrupted
>>>>>>>>>>>>>>>>                                             by signal,
>>>>>>>>>>>>>>>>                                             but I
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             think it is
>>>>>>>>>>>>>>>>                                             unlikely here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>                                             not sure
>>>>>>>>>>>>>>>>                                             that I
>>>>>>>>>>>>>>>>                                             understand
>>>>>>>>>>>>>>>>                                             what do you
>>>>>>>>>>>>>>>>                                             mean by RCF
>>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>>                                             timeout.
>>>>>>>>>>>>>>>>                                             Does it
>>>>>>>>>>>>>>>>                                             happen on
>>>>>>>>>>>>>>>>                                             TE startup
>>>>>>>>>>>>>>>>                                             when RCF
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             starts test
>>>>>>>>>>>>>>>>                                             agents. If
>>>>>>>>>>>>>>>>                                             so,
>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>                                             could help. Or
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             does it
>>>>>>>>>>>>>>>>                                             happen when
>>>>>>>>>>>>>>>>                                             tests are
>>>>>>>>>>>>>>>>                                             in
>>>>>>>>>>>>>>>>                                             progress,
>>>>>>>>>>>>>>>>                                             e.g. in the
>>>>>>>>>>>>>>>>                                             middle
>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>                                             a test. If
>>>>>>>>>>>>>>>>                                             so,
>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>                                             unrelated
>>>>>>>>>>>>>>>>                                             and most
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             likely
>>>>>>>>>>>>>>>>                                             either host
>>>>>>>>>>>>>>>>                                             with test
>>>>>>>>>>>>>>>>                                             agent dies
>>>>>>>>>>>>>>>>                                             or test
>>>>>>>>>>>>>>>>                                             agent itself
>>>>>>>>>>>>>>>>                                             crashes. It
>>>>>>>>>>>>>>>>                                             would be
>>>>>>>>>>>>>>>>                                             easier for
>>>>>>>>>>>>>>>>                                             me if
>>>>>>>>>>>>>>>>                                             classify it
>>>>>>>>>>>>>>>>                                             if you share
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             text log
>>>>>>>>>>>>>>>>                                             (log.txt,
>>>>>>>>>>>>>>>>                                             full or
>>>>>>>>>>>>>>>>                                             just
>>>>>>>>>>>>>>>>                                             corresponding
>>>>>>>>>>>>>>>>                                             fragment with
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             some
>>>>>>>>>>>>>>>>                                             context).
>>>>>>>>>>>>>>>>                                             Also
>>>>>>>>>>>>>>>>                                             content of
>>>>>>>>>>>>>>>>                                             ta.DPDK or
>>>>>>>>>>>>>>>>                                             ta.Peer file
>>>>>>>>>>>>>>>>                                             depending
>>>>>>>>>>>>>>>>                                             on which
>>>>>>>>>>>>>>>>                                             agent has
>>>>>>>>>>>>>>>>                                             problems
>>>>>>>>>>>>>>>>                                             could shed
>>>>>>>>>>>>>>>>                                             some light.
>>>>>>>>>>>>>>>>                                             Corresponding
>>>>>>>>>>>>>>>>                                             files
>>>>>>>>>>>>>>>>                                             contain
>>>>>>>>>>>>>>>>                                             stdout/stderr
>>>>>>>>>>>>>>>>                                             of test agents.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>                                             8/23/23
>>>>>>>>>>>>>>>>                                             17:45, Adam
>>>>>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             I've set
>>>>>>>>>>>>>>>>>                                             up a test
>>>>>>>>>>>>>>>>>                                             rig
>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>                                             here, and
>>>>>>>>>>>>>>>>>                                             have created
>>>>>>>>>>>>>>>>>                                             configurations
>>>>>>>>>>>>>>>>>                                             for our
>>>>>>>>>>>>>>>>>                                             development
>>>>>>>>>>>>>>>>>                                             testbed
>>>>>>>>>>>>>>>>>                                             based off
>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>                                             examples.
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             We've been
>>>>>>>>>>>>>>>>>                                             able to
>>>>>>>>>>>>>>>>>                                             get the
>>>>>>>>>>>>>>>>>                                             test suite
>>>>>>>>>>>>>>>>>                                             to run
>>>>>>>>>>>>>>>>>                                             manually on
>>>>>>>>>>>>>>>>>                                             Mellanox
>>>>>>>>>>>>>>>>>                                             CX5
>>>>>>>>>>>>>>>>>                                             devices once.
>>>>>>>>>>>>>>>>>                                             However,
>>>>>>>>>>>>>>>>>                                             we are
>>>>>>>>>>>>>>>>>                                             running
>>>>>>>>>>>>>>>>>                                             into an
>>>>>>>>>>>>>>>>>                                             issue
>>>>>>>>>>>>>>>>>                                             where,
>>>>>>>>>>>>>>>>>                                             when RCF
>>>>>>>>>>>>>>>>>                                             starts,
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             the RCF
>>>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>>>                                             times out
>>>>>>>>>>>>>>>>>                                             very
>>>>>>>>>>>>>>>>>                                             frequently.
>>>>>>>>>>>>>>>>>                                             We aren't sure
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             why this
>>>>>>>>>>>>>>>>>                                             is the case.
>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>                                             works
>>>>>>>>>>>>>>>>>                                             sometimes,
>>>>>>>>>>>>>>>>>                                             but most
>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>                                             time when
>>>>>>>>>>>>>>>>>                                             we try to run
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             the test
>>>>>>>>>>>>>>>>>                                             engine, it
>>>>>>>>>>>>>>>>>                                             encounters
>>>>>>>>>>>>>>>>>                                             this issue.
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             I've tried
>>>>>>>>>>>>>>>>>                                             changing
>>>>>>>>>>>>>>>>>                                             the RCF
>>>>>>>>>>>>>>>>>                                             port by
>>>>>>>>>>>>>>>>>                                             setting
>>>>>>>>>>>>>>>>>                                             "TE_RCF_PORT=<some
>>>>>>>>>>>>>>>>>                                             port
>>>>>>>>>>>>>>>>>                                             number>"
>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>                                             rebooting
>>>>>>>>>>>>>>>>>                                             the testbed
>>>>>>>>>>>>>>>>>                                             machines.
>>>>>>>>>>>>>>>>>                                             Neither
>>>>>>>>>>>>>>>>>                                             seems to
>>>>>>>>>>>>>>>>>                                             fix the issue.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>                                             also seems
>>>>>>>>>>>>>>>>>                                             like the
>>>>>>>>>>>>>>>>>                                             timeout
>>>>>>>>>>>>>>>>>                                             takes far
>>>>>>>>>>>>>>>>>                                             longer than 60
>>>>>>>>>>>>>>>>>                                             seconds,
>>>>>>>>>>>>>>>>>                                             even when
>>>>>>>>>>>>>>>>>                                             running
>>>>>>>>>>>>>>>>>                                             "export
>>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             before I
>>>>>>>>>>>>>>>>>                                             try to run
>>>>>>>>>>>>>>>>>                                             the test
>>>>>>>>>>>>>>>>>                                             suite.
>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>                                             assume the
>>>>>>>>>>>>>>>>>                                             unit for
>>>>>>>>>>>>>>>>>                                             this
>>>>>>>>>>>>>>>>>                                             variable
>>>>>>>>>>>>>>>>>                                             is seconds?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                             Mon, Aug
>>>>>>>>>>>>>>>>>                                             21, 2023
>>>>>>>>>>>>>>>>>                                             at
>>>>>>>>>>>>>>>>>                                             10:19 AM
>>>>>>>>>>>>>>>>>                                             Adam Hassick
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             <ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>>>>                                             cloned the
>>>>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>                                             and will start
>>>>>>>>>>>>>>>>>                                             setting up
>>>>>>>>>>>>>>>>>                                             a
>>>>>>>>>>>>>>>>>                                             configuration
>>>>>>>>>>>>>>>>>                                             for our
>>>>>>>>>>>>>>>>>                                             development
>>>>>>>>>>>>>>>>>                                             testbed
>>>>>>>>>>>>>>>>>                                             today.
>>>>>>>>>>>>>>>>>                                             I'll let
>>>>>>>>>>>>>>>>>                                             you know
>>>>>>>>>>>>>>>>>                                             if I run
>>>>>>>>>>>>>>>>>                                             into any
>>>>>>>>>>>>>>>>>                                             difficulties
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             or have
>>>>>>>>>>>>>>>>>                                             any questions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                              - Adam
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             On Sun,
>>>>>>>>>>>>>>>>>                                             Aug 20,
>>>>>>>>>>>>>>>>>                                             2023 at
>>>>>>>>>>>>>>>>>                                             4:40 AM
>>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             I've published
>>>>>>>>>>>>>>>>>                                             https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Hopefully
>>>>>>>>>>>>>>>>>                                             it will
>>>>>>>>>>>>>>>>>                                             help to
>>>>>>>>>>>>>>>>>                                             define
>>>>>>>>>>>>>>>>>                                             your test
>>>>>>>>>>>>>>>>>                                             rigs and
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             successfully
>>>>>>>>>>>>>>>>>                                             run some
>>>>>>>>>>>>>>>>>                                             tests
>>>>>>>>>>>>>>>>>                                             manually.
>>>>>>>>>>>>>>>>>                                             Feel free to
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             ask any
>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>                                             and I'll
>>>>>>>>>>>>>>>>>                                             answer
>>>>>>>>>>>>>>>>>                                             here and
>>>>>>>>>>>>>>>>>                                             try to
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             update
>>>>>>>>>>>>>>>>>                                             documentation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Meanwhile
>>>>>>>>>>>>>>>>>                                             I'll
>>>>>>>>>>>>>>>>>                                             prepare
>>>>>>>>>>>>>>>>>                                             missing
>>>>>>>>>>>>>>>>>                                             bits for
>>>>>>>>>>>>>>>>>                                             steps (2) and
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             (3).
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Hopefully
>>>>>>>>>>>>>>>>>                                             everything
>>>>>>>>>>>>>>>>>                                             is in
>>>>>>>>>>>>>>>>>                                             place for
>>>>>>>>>>>>>>>>>                                             step (4),
>>>>>>>>>>>>>>>>>                                             but we
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             need to
>>>>>>>>>>>>>>>>>                                             make steps
>>>>>>>>>>>>>>>>>                                             (2) and
>>>>>>>>>>>>>>>>>                                             (3) first.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             On 8/18/23
>>>>>>>>>>>>>>>>>                                             21:40,
>>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             > I've
>>>>>>>>>>>>>>>>>>                                             conferred
>>>>>>>>>>>>>>>>>>                                             with the
>>>>>>>>>>>>>>>>>>                                             rest of
>>>>>>>>>>>>>>>>>>                                             the team,
>>>>>>>>>>>>>>>>>>                                             and we
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             think it
>>>>>>>>>>>>>>>>>>                                             would be
>>>>>>>>>>>>>>>>>>                                             best to
>>>>>>>>>>>>>>>>>>                                             move
>>>>>>>>>>>>>>>>>>                                             forward
>>>>>>>>>>>>>>>>>>                                             with mainly
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             OK, I'll
>>>>>>>>>>>>>>>>>>                                             provide
>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>                                             sample on
>>>>>>>>>>>>>>>>>>                                             Monday
>>>>>>>>>>>>>>>>>>                                             for you.
>>>>>>>>>>>>>>>>>>                                             It is
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             almost
>>>>>>>>>>>>>>>>>>                                             ready
>>>>>>>>>>>>>>>>>>                                             right
>>>>>>>>>>>>>>>>>>                                             now, but
>>>>>>>>>>>>>>>>>>                                             I need to
>>>>>>>>>>>>>>>>>>                                             double-check
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             it before
>>>>>>>>>>>>>>>>>>                                             publishing.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             Regards,
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             On
>>>>>>>>>>>>>>>>>>                                             8/17/23
>>>>>>>>>>>>>>>>>>                                             20:03,
>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>                                             Hassick
>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             I'm
>>>>>>>>>>>>>>>>>>>                                             adding
>>>>>>>>>>>>>>>>>>>                                             the CI
>>>>>>>>>>>>>>>>>>>                                             mailing
>>>>>>>>>>>>>>>>>>>                                             list to this
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             conversation.
>>>>>>>>>>>>>>>>>>>                                             Others
>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>                                             community
>>>>>>>>>>>>>>>>>>>                                             might find
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             this
>>>>>>>>>>>>>>>>>>>                                             conversation
>>>>>>>>>>>>>>>>>>>                                             valuable.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             We do
>>>>>>>>>>>>>>>>>>>                                             want to
>>>>>>>>>>>>>>>>>>>                                             run
>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>                                             on a
>>>>>>>>>>>>>>>>>>>                                             regular
>>>>>>>>>>>>>>>>>>>                                             basis. The
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Jenkins
>>>>>>>>>>>>>>>>>>>                                             integration
>>>>>>>>>>>>>>>>>>>                                             will be
>>>>>>>>>>>>>>>>>>>                                             very
>>>>>>>>>>>>>>>>>>>                                             useful
>>>>>>>>>>>>>>>>>>>                                             for us, as
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             most of
>>>>>>>>>>>>>>>>>>>                                             our CI
>>>>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>>>>                                             orchestrated
>>>>>>>>>>>>>>>>>>>                                             by Jenkins.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>>>>>>                                             conferred
>>>>>>>>>>>>>>>>>>>                                             with the
>>>>>>>>>>>>>>>>>>>                                             rest of
>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>                                             team, and we
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             think it
>>>>>>>>>>>>>>>>>>>                                             would be
>>>>>>>>>>>>>>>>>>>                                             best to
>>>>>>>>>>>>>>>>>>>                                             move
>>>>>>>>>>>>>>>>>>>                                             forward
>>>>>>>>>>>>>>>>>>>                                             with mainly
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             If you
>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>                                             like to
>>>>>>>>>>>>>>>>>>>                                             know
>>>>>>>>>>>>>>>>>>>                                             anything
>>>>>>>>>>>>>>>>>>>                                             about our
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             testbeds
>>>>>>>>>>>>>>>>>>>                                             that
>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>                                             help you
>>>>>>>>>>>>>>>>>>>                                             with
>>>>>>>>>>>>>>>>>>>                                             creating an
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             repo,
>>>>>>>>>>>>>>>>>>>                                             I'd be
>>>>>>>>>>>>>>>>>>>                                             happy to
>>>>>>>>>>>>>>>>>>>                                             answer any
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>>>                                             you have.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             We have
>>>>>>>>>>>>>>>>>>>                                             multiple
>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>                                             rigs (we
>>>>>>>>>>>>>>>>>>>                                             call these
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             "DUT-tester
>>>>>>>>>>>>>>>>>>>                                             pairs")
>>>>>>>>>>>>>>>>>>>                                             that we
>>>>>>>>>>>>>>>>>>>                                             run our
>>>>>>>>>>>>>>>>>>>                                             existing
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             hardware
>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>                                             on, with
>>>>>>>>>>>>>>>>>>>                                             differing
>>>>>>>>>>>>>>>>>>>                                             network
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             hardware
>>>>>>>>>>>>>>>>>>>                                             and CPU
>>>>>>>>>>>>>>>>>>>                                             architecture.
>>>>>>>>>>>>>>>>>>>                                             I
>>>>>>>>>>>>>>>>>>>                                             figured
>>>>>>>>>>>>>>>>>>>                                             this might
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             be an
>>>>>>>>>>>>>>>>>>>                                             important
>>>>>>>>>>>>>>>>>>>                                             detail.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             On Thu,
>>>>>>>>>>>>>>>>>>>                                             Aug 17,
>>>>>>>>>>>>>>>>>>>                                             2023 at
>>>>>>>>>>>>>>>>>>>                                             11:44 AM
>>>>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Greatings
>>>>>>>>>>>>>>>>>>>                                             Adam,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             I'm
>>>>>>>>>>>>>>>>>>>                                             happy to
>>>>>>>>>>>>>>>>>>>                                             hear
>>>>>>>>>>>>>>>>>>>                                             that
>>>>>>>>>>>>>>>>>>>                                             you're
>>>>>>>>>>>>>>>>>>>                                             trying
>>>>>>>>>>>>>>>>>>>                                             to bring
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             it up.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             As I
>>>>>>>>>>>>>>>>>>>                                             understand
>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>                                             final
>>>>>>>>>>>>>>>>>>>                                             goal is
>>>>>>>>>>>>>>>>>>>                                             to run it on
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             regular
>>>>>>>>>>>>>>>>>>>                                             basis.
>>>>>>>>>>>>>>>>>>>                                             So, we
>>>>>>>>>>>>>>>>>>>                                             need to
>>>>>>>>>>>>>>>>>>>                                             make it
>>>>>>>>>>>>>>>>>>>                                             properly
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             from the
>>>>>>>>>>>>>>>>>>>                                             very
>>>>>>>>>>>>>>>>>>>                                             beginning.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Bring up
>>>>>>>>>>>>>>>>>>>                                             of all
>>>>>>>>>>>>>>>>>>>                                             features
>>>>>>>>>>>>>>>>>>>                                             consists
>>>>>>>>>>>>>>>>>>>                                             of 4 steps:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             1.
>>>>>>>>>>>>>>>>>>>                                             Create
>>>>>>>>>>>>>>>>>>>                                             site-specific
>>>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>>>                                             (we call it
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             ts-rigs)
>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>                                             contains
>>>>>>>>>>>>>>>>>>>                                             information
>>>>>>>>>>>>>>>>>>>                                             about test
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             rigs and
>>>>>>>>>>>>>>>>>>>                                             other
>>>>>>>>>>>>>>>>>>>                                             site-specific
>>>>>>>>>>>>>>>>>>>                                             information
>>>>>>>>>>>>>>>>>>>                                             like
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             where to
>>>>>>>>>>>>>>>>>>>                                             send
>>>>>>>>>>>>>>>>>>>                                             mails,
>>>>>>>>>>>>>>>>>>>                                             where to
>>>>>>>>>>>>>>>>>>>                                             store
>>>>>>>>>>>>>>>>>>>                                             logs etc.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             It is
>>>>>>>>>>>>>>>>>>>                                             required
>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>                                             manual
>>>>>>>>>>>>>>>>>>>                                             execution
>>>>>>>>>>>>>>>>>>>                                             as well,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             since
>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>                                             rigs
>>>>>>>>>>>>>>>>>>>                                             description
>>>>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>>>>                                             essential.
>>>>>>>>>>>>>>>>>>>                                             I'll
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             return
>>>>>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>>>>>                                             topic below.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             2. Setup
>>>>>>>>>>>>>>>>>>>                                             logs
>>>>>>>>>>>>>>>>>>>                                             storage
>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>                                             automated
>>>>>>>>>>>>>>>>>>>                                             runs.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Basically
>>>>>>>>>>>>>>>>>>>                                             it is a
>>>>>>>>>>>>>>>>>>>                                             disk
>>>>>>>>>>>>>>>>>>>                                             space
>>>>>>>>>>>>>>>>>>>                                             plus
>>>>>>>>>>>>>>>>>>>                                             apache2 web
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             server
>>>>>>>>>>>>>>>>>>>                                             with few
>>>>>>>>>>>>>>>>>>>                                             CGI
>>>>>>>>>>>>>>>>>>>                                             scripts
>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>                                             help a
>>>>>>>>>>>>>>>>>>>                                             lot to
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             save
>>>>>>>>>>>>>>>>>>>                                             disk space.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             3. Setup
>>>>>>>>>>>>>>>>>>>                                             Bublik
>>>>>>>>>>>>>>>>>>>                                             web
>>>>>>>>>>>>>>>>>>>                                             application
>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>                                             provides
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             web
>>>>>>>>>>>>>>>>>>>                                             interface
>>>>>>>>>>>>>>>>>>>                                             to view
>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>                                             results.
>>>>>>>>>>>>>>>>>>>                                             Same as
>>>>>>>>>>>>>>>>>>>                                             https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             4. Setup
>>>>>>>>>>>>>>>>>>>                                             Jenkins
>>>>>>>>>>>>>>>>>>>                                             to run
>>>>>>>>>>>>>>>>>>>                                             tests on
>>>>>>>>>>>>>>>>>>>                                             regularly,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             save
>>>>>>>>>>>>>>>>>>>                                             logs in
>>>>>>>>>>>>>>>>>>>                                             log
>>>>>>>>>>>>>>>>>>>                                             storage
>>>>>>>>>>>>>>>>>>>                                             (2) and
>>>>>>>>>>>>>>>>>>>                                             import it to
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             bublik (3).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Last few
>>>>>>>>>>>>>>>>>>>                                             month we
>>>>>>>>>>>>>>>>>>>                                             spent on
>>>>>>>>>>>>>>>>>>>                                             our
>>>>>>>>>>>>>>>>>>>                                             homework
>>>>>>>>>>>>>>>>>>>                                             to make
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             it
>>>>>>>>>>>>>>>>>>>                                             simpler
>>>>>>>>>>>>>>>>>>>                                             to bring
>>>>>>>>>>>>>>>>>>>                                             up
>>>>>>>>>>>>>>>>>>>                                             automated
>>>>>>>>>>>>>>>>>>>                                             execution
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             using
>>>>>>>>>>>>>>>>>>>                                             Jenkins -
>>>>>>>>>>>>>>>>>>>                                             https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Corresponding
>>>>>>>>>>>>>>>>>>>                                             bits in
>>>>>>>>>>>>>>>>>>>                                             dpdk-ethdev-ts
>>>>>>>>>>>>>>>>>>>                                             will be
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             available
>>>>>>>>>>>>>>>>>>>                                             tomorrow.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Let's
>>>>>>>>>>>>>>>>>>>                                             return
>>>>>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>>>>>                                             step (1).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Unfortunately
>>>>>>>>>>>>>>>>>>>                                             there is
>>>>>>>>>>>>>>>>>>>                                             no
>>>>>>>>>>>>>>>>>>>                                             publicly
>>>>>>>>>>>>>>>>>>>                                             available
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>>>                                             since
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             sensitive
>>>>>>>>>>>>>>>>>>>                                             site-specific
>>>>>>>>>>>>>>>>>>>                                             information
>>>>>>>>>>>>>>>>>>>                                             is located
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             there.
>>>>>>>>>>>>>>>>>>>                                             But I'm
>>>>>>>>>>>>>>>>>>>                                             ready to
>>>>>>>>>>>>>>>>>>>                                             help you
>>>>>>>>>>>>>>>>>>>                                             to create it
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             for UNH.
>>>>>>>>>>>>>>>>>>>                                             I see
>>>>>>>>>>>>>>>>>>>                                             two
>>>>>>>>>>>>>>>>>>>                                             options
>>>>>>>>>>>>>>>>>>>                                             here:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             (A) I'll
>>>>>>>>>>>>>>>>>>>                                             ask
>>>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>>>                                             based on
>>>>>>>>>>>>>>>>>>>                                             your
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             answers
>>>>>>>>>>>>>>>>>>>                                             will
>>>>>>>>>>>>>>>>>>>                                             create
>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>                                             first
>>>>>>>>>>>>>>>>>>>                                             draft
>>>>>>>>>>>>>>>>>>>                                             with my
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             comments.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             (B) I'll
>>>>>>>>>>>>>>>>>>>                                             make a
>>>>>>>>>>>>>>>>>>>                                             template/example
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             repo,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             publish
>>>>>>>>>>>>>>>>>>>                                             it and
>>>>>>>>>>>>>>>>>>>                                             you'll
>>>>>>>>>>>>>>>>>>>                                             create
>>>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             based
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             on it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Of
>>>>>>>>>>>>>>>>>>>                                             course,
>>>>>>>>>>>>>>>>>>>                                             I'll
>>>>>>>>>>>>>>>>>>>                                             help to
>>>>>>>>>>>>>>>>>>>                                             debug
>>>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>>>                                             finally
>>>>>>>>>>>>>>>>>>>                                             bring
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             it up in
>>>>>>>>>>>>>>>>>>>                                             any case.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             (A) is a
>>>>>>>>>>>>>>>>>>>                                             bit
>>>>>>>>>>>>>>>>>>>                                             simpler
>>>>>>>>>>>>>>>>>>>                                             for me
>>>>>>>>>>>>>>>>>>>                                             and you,
>>>>>>>>>>>>>>>>>>>                                             but (B) is
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             a bit
>>>>>>>>>>>>>>>>>>>                                             more
>>>>>>>>>>>>>>>>>>>                                             generic
>>>>>>>>>>>>>>>>>>>                                             and will
>>>>>>>>>>>>>>>>>>>                                             help other
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             potential
>>>>>>>>>>>>>>>>>>>                                             users to
>>>>>>>>>>>>>>>>>>>                                             bring it up.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             We can
>>>>>>>>>>>>>>>>>>>                                             combine
>>>>>>>>>>>>>>>>>>>                                             (A)+(B).
>>>>>>>>>>>>>>>>>>>                                             I.e.
>>>>>>>>>>>>>>>>>>>                                             start
>>>>>>>>>>>>>>>>>>>                                             from (A).
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             What do
>>>>>>>>>>>>>>>>>>>                                             you think?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             On
>>>>>>>>>>>>>>>>>>>                                             8/17/23
>>>>>>>>>>>>>>>>>>>                                             15:18,
>>>>>>>>>>>>>>>>>>>                                             Konstantin
>>>>>>>>>>>>>>>>>>>                                             Ushakov
>>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>>                                             Greetings
>>>>>>>>>>>>>>>>>>>>                                             Adam,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             Thanks
>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>                                             contacting
>>>>>>>>>>>>>>>>>>>>                                             us. I
>>>>>>>>>>>>>>>>>>>>                                             copy
>>>>>>>>>>>>>>>>>>>>                                             Andrew who
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>>                                             be
>>>>>>>>>>>>>>>>>>>>                                             happy
>>>>>>>>>>>>>>>>>>>>                                             to help
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             Konstantin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                             On 16
>>>>>>>>>>>>>>>>>>>>>                                             Aug
>>>>>>>>>>>>>>>>>>>>>                                             2023,
>>>>>>>>>>>>>>>>>>>>>                                             at
>>>>>>>>>>>>>>>>>>>>>                                             21:50,
>>>>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>>>>                                             Hassick
>>>>>>>>>>>>>>>>>>>>>                                             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Greetings
>>>>>>>>>>>>>>>>>>>>>                                             Konstantin,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I am
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             process
>>>>>>>>>>>>>>>>>>>>>                                             of
>>>>>>>>>>>>>>>>>>>>>                                             setting
>>>>>>>>>>>>>>>>>>>>>                                             up the
>>>>>>>>>>>>>>>>>>>>>                                             DPDK
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Poll
>>>>>>>>>>>>>>>>>>>>>                                             Mode
>>>>>>>>>>>>>>>>>>>>>                                             Driver
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite
>>>>>>>>>>>>>>>>>>>>>                                             as an
>>>>>>>>>>>>>>>>>>>>>                                             addition
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             our
>>>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>>>                                             coverage
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             DPDK
>>>>>>>>>>>>>>>>>>>>>                                             at the
>>>>>>>>>>>>>>>>>>>>>                                             UNH lab.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I have
>>>>>>>>>>>>>>>>>>>>>                                             some
>>>>>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>>>>>                                             about
>>>>>>>>>>>>>>>>>>>>>                                             how to
>>>>>>>>>>>>>>>>>>>>>                                             set the
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite
>>>>>>>>>>>>>>>>>>>>>                                             arguments.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I have
>>>>>>>>>>>>>>>>>>>>>                                             been
>>>>>>>>>>>>>>>>>>>>>                                             able
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             configure
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             Test
>>>>>>>>>>>>>>>>>>>>>                                             Engine
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             connect
>>>>>>>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>>>>>>>                                             hosts
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             testbed.
>>>>>>>>>>>>>>>>>>>>>                                             The
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             RCF,
>>>>>>>>>>>>>>>>>>>>>                                             Configurator,
>>>>>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                             all
>>>>>>>>>>>>>>>>>>>>>                                             begin to
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             run,
>>>>>>>>>>>>>>>>>>>>>                                             however
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             prelude
>>>>>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>>>>>                                             test suite
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             fails
>>>>>>>>>>>>>>>>>>>>>                                             to run.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             The
>>>>>>>>>>>>>>>>>>>>>                                             documentation
>>>>>>>>>>>>>>>>>>>>>                                             mentions
>>>>>>>>>>>>>>>>>>>>>                                             that
>>>>>>>>>>>>>>>>>>>>>                                             there are
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             several
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             parameters
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite,
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             like
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             IUT
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             link
>>>>>>>>>>>>>>>>>>>>>                                             MAC,
>>>>>>>>>>>>>>>>>>>>>                                             etc. These
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             seem
>>>>>>>>>>>>>>>>>>>>>                                             like
>>>>>>>>>>>>>>>>>>>>>                                             they
>>>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>>>                                             need
>>>>>>>>>>>>>>>>>>>>>                                             to be
>>>>>>>>>>>>>>>>>>>>>                                             set
>>>>>>>>>>>>>>>>>>>>>                                             somewhere
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             to run
>>>>>>>>>>>>>>>>>>>>>                                             many
>>>>>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>>>>>                                             tests.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I see
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             Test
>>>>>>>>>>>>>>>>>>>>>                                             Engine
>>>>>>>>>>>>>>>>>>>>>                                             documentation,
>>>>>>>>>>>>>>>>>>>>>                                             there
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             are
>>>>>>>>>>>>>>>>>>>>>                                             instructions
>>>>>>>>>>>>>>>>>>>>>                                             on how
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             create new
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             parameters
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suites
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             configuration,
>>>>>>>>>>>>>>>>>>>>>                                             but
>>>>>>>>>>>>>>>>>>>>>                                             there
>>>>>>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>>>>>>                                             nothing
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             user
>>>>>>>>>>>>>>>>>>>>>                                             guide
>>>>>>>>>>>>>>>>>>>>>                                             or in
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                             guide
>>>>>>>>>>>>>>>>>>>>>                                             for how to
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             set
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             arguments
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             parameters
>>>>>>>>>>>>>>>>>>>>>                                             when
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             running
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite
>>>>>>>>>>>>>>>>>>>>>                                             that I
>>>>>>>>>>>>>>>>>>>>>                                             can
>>>>>>>>>>>>>>>>>>>>>                                             find. I'm
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             not
>>>>>>>>>>>>>>>>>>>>>                                             sure
>>>>>>>>>>>>>>>>>>>>>                                             if I
>>>>>>>>>>>>>>>>>>>>>                                             need
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             write
>>>>>>>>>>>>>>>>>>>>>                                             my own
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             config,
>>>>>>>>>>>>>>>>>>>>>                                             or if
>>>>>>>>>>>>>>>>>>>>>                                             I
>>>>>>>>>>>>>>>>>>>>>                                             should
>>>>>>>>>>>>>>>>>>>>>                                             be
>>>>>>>>>>>>>>>>>>>>>                                             setting
>>>>>>>>>>>>>>>>>>>>>                                             these in
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             some
>>>>>>>>>>>>>>>>>>>>>                                             other way.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             How
>>>>>>>>>>>>>>>>>>>>>                                             should
>>>>>>>>>>>>>>>>>>>>>                                             these
>>>>>>>>>>>>>>>>>>>>>                                             values
>>>>>>>>>>>>>>>>>>>>>                                             be set?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I'm
>>>>>>>>>>>>>>>>>>>>>                                             also
>>>>>>>>>>>>>>>>>>>>>                                             not
>>>>>>>>>>>>>>>>>>>>>                                             sure
>>>>>>>>>>>>>>>>>>>>>                                             what
>>>>>>>>>>>>>>>>>>>>>                                             environment
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             variables/arguments
>>>>>>>>>>>>>>>>>>>>>                                             are
>>>>>>>>>>>>>>>>>>>>>                                             strictly
>>>>>>>>>>>>>>>>>>>>>                                             necessary
>>>>>>>>>>>>>>>>>>>>>                                             or
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>>>                                             are
>>>>>>>>>>>>>>>>>>>>>                                             optional.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Regards,
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             *Adam
>>>>>>>>>>>>>>>>>>>>>                                             Hassick*
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             +1
>>>>>>>>>>>>>>>>>>>>>                                             (603)
>>>>>>>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             *Adam
>>>>>>>>>>>>>>>>>>>                                             Hassick*
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             -- *Adam
>>>>>>>>>>>>>>>>>                                             Hassick*
>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>                                             (603) 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>                                             (603) 475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             -- 
>>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>>                                             Senior Developer
>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         --
>>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>                                         InterOperability Lab
>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                     --
>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 --
>>>>>>>>>                                 *Adam Hassick*
>>>>>>>>>                                 Senior Developer
>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                         --
>>>>>>>                         *Adam Hassick*
>>>>>>>                         Senior Developer
>>>>>>>                         UNH InterOperability Lab
>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                         +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>                     --
>>>>>>                     *Adam Hassick*
>>>>>>                     Senior Developer
>>>>>>                     UNH InterOperability Lab
>>>>>>                     ahassick@iol.unh.edu
>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --
>>>>>>                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>                 ahassick@iol.unh.edu
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>
>>>>
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 241836 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 177277 bytes --]

Hi Adam,

 > Now that our test results are in line with yours, we can begin 
looking into setting up the production environment.

Please, let me know if you need any help with it or waiting for an input 
from me.

Regards,
Andrew.

On 10/10/23 17:09, Adam Hassick wrote:
> Hi Andrew,
>
> Thank you for taking a look at our log. Netplan was attempting to run 
> DHCP on our test links, and additionally I discovered that the NIC 
> firmware was transmitting LLDP packets, causing tests to fail in the 
> same way. Now that these problems have been fixed, our pass rate on 
> the XL710 is approximately 91%. Now that our test results are in line 
> with yours, we can begin looking into setting up the production 
> environment.
>
> First, is it possible to run the test agent on ARM hosts? Our ARM 
> testbeds have the best topology for running this test suite, with 
> separate tester and DUT servers.
>
> We are testing this test suite on two x86 development servers using 
> the test suite's recommended server topology. In contrast, our 
> existing x86 production testbeds which run DTS have a single server 
> topology. This single server has both the tester NIC and the device 
> under test NIC installed, with NUMA node separation between TRex and 
> DPDK. We're going to test running the two test agent processes on the 
> single-server testbeds if we cannot run this on ARM. Is there any 
> reason you can think of that would prevent this setup from working?
>
> Once we figure out where this can live in production, then we will 
> begin setting up log storage, Jenkins integration, and Bublik.
>
> Thanks,
> Adam
>
> On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     > Do these default to vfio-pci?
>
>     Yes, vfio-pci is the default.
>     However, it does not work in the case of Mellanox which uses
>     bifurcated driver. It should mlx5_core for Mellanox NICs.
>
>     > Here is the text log from a run on our Intel XL710 NICs, with
>     the expected result profile set to the X710.
>
>     It is hard to analyze all tests using text logs, but I definitely
>     see one common problem. Tests receive unexpected packets and fail
>     because of it.
>     Tests are written very strict from this point of view and it
>     brought fruits in the past when HW had bugs.
>     Are DUT and tester connected back-to-back on tested interfaces or
>     via switch?
>     If via switch, is it possible to isolate it from everything else?
>     If back-to-back, it could be some embedded SW/FW which regenerates
>     these packet.
>     I definitely see unexpected DHCP packets.
>
>     > We haven't set up the Jenkins integration yet, however if this
>     is required to import the logs then we will prioritize that.
>
>     Unfortunately manual runs do not generate all artifacts required
>     to import logs. However, we have almost solved it right now.
>     Hopefully we'll finalize it in a day or two. I'll let you know
>     when these changes are available.
>
>     Regards,
>     Andrew.
>
>     On 10/4/23 16:48, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set
>>     anywhere in the default configurations for the Intel X710. Do
>>     these default to vfio-pci?
>>     We have IOMMU enabled on our development testbed, and should be
>>     able to bind vfio-pci.
>>     Here is the text log from a run on our Intel XL710 NICs, with the
>>     expected result profile set to the X710. We haven't set up the
>>     Jenkins integration yet, however if this is required to import
>>     the logs then we will prioritize that.
>>     log.txt.tar.gz
>>     <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>>
>>     Thanks,
>>     Adam
>>
>>     On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         On 9/18/23 17:44, Adam Hassick wrote:
>>>         Hi Andrew and Konstantin,
>>>
>>>         Thank you for adding the tester-dial feature, this opens up
>>>         the possibility for us to do CI integrated testing in the
>>>         future.
>>>
>>>         Our Mellanox pass rate is similar to yours (about ~2400
>>>         passing, ~4400 failing), however our Intel pass rates are
>>>         far worse.
>>>         I will try running tests on the XL710 with the trc-tags
>>>         argument set and see if it improves the pass rate.
>>>         Another thing I noticed in the results you uploaded is that
>>>         the results are tagged with vfio-pci and not i40e.
>>>         Though in the environment dump, the driver on the test
>>>         machine and the DUT are set to use the i40e driver. Is this
>>>         important at all?
>>
>>         I think it is a misunderstanding here. There are two kinds of
>>         driver in configuration: net driver and so-called DPDK driver.
>>         Net driver is a Linux kernel network device driver used on
>>         Tester side.
>>         DPDK driver is a Linux kernel driver to bind device to to use
>>         it with DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
>>         In the case of bifurcated driver (like mlx5_core) it is the
>>         same in both cases.
>>         In non-bifurcated case DPDK driver is some UIO
>>         driver(vfio-pci, uio-pci-generic or igb_uio).
>>         Some expectations depend on used UIO. For example,
>>         uio-pci-generic do not support many interrupts (used by
>>         usecases/rx_intr test cases).
>>         That's why we care corresponding TRC tag.
>>
>>         TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's
>>         Intel case. Or uio-pci-generic if IOMMU is turned off on
>>         corresponding machines and Linux distro does not support VFIO
>>         no IOMMU mode.
>>
>>         Andrew.
>>
>>>         There isn't anything preventing us from pushing our results
>>>         up to the existing Bublik instance running at ts-factory.io
>>>         <http://ts-factory.io> that I can think of at the moment.
>>>         We will have to work out how to submit our results to your
>>>         Bublik instance in a controlled and secure manner in that case.
>>>         As far as I know we won't need access controls for the
>>>         results themselves. I'll discuss this with Patrick and will
>>>         let you know once we confirm that it's fine.
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             should we always auto-assign the tags or you don’t do
>>>>             it since it slows down (by some seconds) the TE startup?
>>>>
>>>
>>>             Tags are auto-assigned, but I guess it differs in Adam's
>>>             case since NIC is a bit different. Below test will help
>>>             to understand if it is the root cause of very different
>>>             expectations. If pass rate will be close to mine, I'll
>>>             simply update TRC database to share expectations for
>>>             mine NIC and NIC used by Adam.
>>>
>>>>             Hi Adam,
>>>>
>>>>             I think I second the question from Andrew - happy to
>>>>             help you with the triage so that we get to the same
>>>>             baseline. Do you have a good way for us to share the
>>>>             logs? I.e. say upload to ts-factory if we add strict
>>>>             permissions system so it’s not publishing or any other way.
>>>>
>>>>             Thanks,
>>>>             Konstantin
>>>>
>>>>
>>>>             On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 I've uploaded fresh testing results to
>>>>                 ts-factory.io <http://ts-factory.io> [1] to be on
>>>>                 the same page.
>>>>
>>>>                 I think I know why your and mine results on Intel
>>>>                 710 series NICs differ so much. Testing results
>>>>                 expectations database (dpdk-ethdev-ts/trc/*) is
>>>>                 filled in in terms of TRC tags.  I.e. expectations
>>>>                 depends on TRC tags discovered by helper scripts
>>>>                 when testing is started. These tags identify
>>>>                 various aspects of what is tested. Ideally
>>>>                 expectations should be written in terms of root
>>>>                 cause of the expected behaviour. If it is a driver
>>>>                 expectations, driver tag should be used. If it is
>>>>                 HW limitation, tags with PCI IDs should be used.
>>>>                 However, it is not always easy to classify it
>>>>                 correctly if you're not involved in driver
>>>>                 development. So, in order case expectations for
>>>>                 710's Intel are filled in in terms of PCI IDs. I
>>>>                 guess PCI ID differ in your case and that's why
>>>>                 expectations filled in for my NIC do not apply to
>>>>                 your runs.
>>>>
>>>>                 Just try to add the following option when you run
>>>>                 on your 710's Intel in order to mimic mine and see
>>>>                 if it helps to achieve better pass rate.
>>>>                 --trc-tag=pci-8086-1572
>>>>
>>>>                 BTW, fresh TE tag v1.21.0 has improved algorithm to
>>>>                 choose tests for --tester-dial option. It should
>>>>                 have better coverage now.
>>>>
>>>>                 Andrew.
>>>>
>>>>                 [1]
>>>>                 https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>>                 <https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1>
>>>>
>>>>                 On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>>                 Hi Adam,
>>>>>
>>>>>                 I've pushed new TE tag v1.20.0 which supported a
>>>>>                 new command-line option --tester-dial=NUM where
>>>>>                 NUM is from 0 to 100. it allows to choose
>>>>>                 percentage of tests to run. If you want stable
>>>>>                 set, you should pass --tester-random-seed=0 (or
>>>>>                 other integer). It is the first sketch and we have
>>>>>                 plans to improve it, but feedback would be welcome.
>>>>>
>>>>>                 > Is it needed on the tester?
>>>>>
>>>>>                 It is hard to say if it is strictly required for
>>>>>                 simple tests. However, it is better to update
>>>>>                 Tester as well, since performance tests run DPDK
>>>>>                 on Tester as well.
>>>>>
>>>>>                 > Are there any other manual setup steps for these
>>>>>                 devices that I might be missing?
>>>>>
>>>>>                 I don't remember anything else.
>>>>>
>>>>>                 I think it is better to get down to details and
>>>>>                 take a look at logs. I'm ready to help with it and
>>>>>                 explain what's happening there. May be it will
>>>>>                 help to understand if it is a problem with
>>>>>                 setup/configuration.
>>>>>
>>>>>                 Text logs are not very convenient. Ideally logs
>>>>>                 should be imported to bublik, however, manual runs
>>>>>                 do not provide all required artifacts right now
>>>>>                 (Jenkins jobs generate all required artifacts).
>>>>>                 Other option is 'tmp_raw_log' file (should be
>>>>>                 packed to make it smaller) which could be
>>>>>                 converted to various log formats.
>>>>>                 Would it be OK for you if I import your logs to
>>>>>                 bublik at ts-factory.io <http://ts-factory.io>? Or
>>>>>                 is it a problem that it is publicly available?
>>>>>                 Would it help if we add authentication and access
>>>>>                 control there?
>>>>>
>>>>>                 Andrew.
>>>>>
>>>>>                 On 9/8/23 17:57, Adam Hassick wrote:
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 I have a couple questions about needed setup of
>>>>>>                 the NICs for the ethdev test suite.
>>>>>>
>>>>>>                 Our MCX5s and XL710s are failing the checkup
>>>>>>                 tests. The pass rate appears to be much worse on
>>>>>>                 the XL710s (40 of 73 tests failed, 3 passed
>>>>>>                 unexpectedly).
>>>>>>
>>>>>>                 For the XL710s, I've updated the driver and NVM
>>>>>>                 versions to match the minimum supported versions
>>>>>>                 in the compatibility matrix found on the DPDK
>>>>>>                 documentation. This did not change the failure
>>>>>>                 rate much.
>>>>>>                 For the MCX5s, I've installed the latest LTS
>>>>>>                 version of the OFED bifurcated driver on the DUT.
>>>>>>                 Is it needed on the tester?
>>>>>>
>>>>>>                 Are there any other manual setup steps for these
>>>>>>                 devices that I might be missing?
>>>>>>
>>>>>>                 Thanks,
>>>>>>                 Adam
>>>>>>
>>>>>>                 On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick
>>>>>>                 <ahassick@iol.unh.edu> wrote:
>>>>>>
>>>>>>                     Hi Andrew,
>>>>>>
>>>>>>                     Yes, I copied the X710 configs to set up
>>>>>>                     XL710 configs. I changed the environment
>>>>>>                     variable names from the X710 suffix to XL710
>>>>>>                     suffix in the script, and forgot to change
>>>>>>                     them in the corresponding environment file.
>>>>>>                     That fixed the issue.
>>>>>>
>>>>>>                     I got the checkup tests working on the XL710
>>>>>>                     now. Most of them are failing, which leads me
>>>>>>                     to believe this is an issue with our testbed.
>>>>>>                     Based on the DPDK documentation for i40e, the
>>>>>>                     firmware and driver versions are much older
>>>>>>                     than what DPDK 22.11 LTS and main prefer, so
>>>>>>                     I'll try updating those.
>>>>>>
>>>>>>                     For now I'm working on getting the XL710
>>>>>>                     checkup tests passing, and will pick up
>>>>>>                     getting the E810 configured properly next.
>>>>>>                     I'll let you know if I run into any more
>>>>>>                     issues in relation to the test engine.
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Adam
>>>>>>
>>>>>>                     On Wed, Sep 6, 2023 at 7:36 AM Andrew
>>>>>>                     Rybchenko <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                         Hi Adam,
>>>>>>
>>>>>>                         On 9/5/23 18:01, Adam Hassick wrote:
>>>>>>>                         Hi Andrew,
>>>>>>>
>>>>>>>                         The compilation warning issue is now
>>>>>>>                         resolved. Again, thank you guys for
>>>>>>>                         fixing this for us. I can run the tests
>>>>>>>                         on the Mellanox CX5s again, however I'm
>>>>>>>                         running into a couple new issues with
>>>>>>>                         running the prologues on the Intel cards.
>>>>>>>
>>>>>>>                         When running testing on the Intel
>>>>>>>                         XL710s, I see this error appear in the log:
>>>>>>>
>>>>>>>                             ERROR  prologue  Environment LIB
>>>>>>>                              14:16:13.650
>>>>>>>                             Too few networks in available
>>>>>>>                             configuration (0) in comparison with
>>>>>>>                             required (1)
>>>>>>>
>>>>>>>
>>>>>>>                         This seems like a trivial configuration
>>>>>>>                         error, perhaps this is something I need
>>>>>>>                         to set up in ts-rigs. I briefly searched
>>>>>>>                         through the examples there and didn't
>>>>>>>                         see any mention of how to set up a network.
>>>>>>>                         I will attach this log just in case you
>>>>>>>                         need more information.
>>>>>>
>>>>>>                         Unfortunately logs are insufficient to
>>>>>>                         understand it. I've pushed new tag to TE
>>>>>>                         v1.19.0 which add log message with TE_*
>>>>>>                         environment variables.
>>>>>>                         Most likely something is wrong with
>>>>>>                         variables which are used as conditions
>>>>>>                         when available networks are defined in
>>>>>>                         ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>>                         TE_PCI_INSTANCE_IUT_TST1
>>>>>>                         TE_PCI_INSTANCE_IUT_TST1a
>>>>>>                         TE_PCI_INSTANCE_TST1a_IUT
>>>>>>                         TE_PCI_INSTANCE_TST1_IUT
>>>>>>                         My guess it that you change naming a bit,
>>>>>>                         but script like
>>>>>>                         ts-rigs-sample/scripts/iut.h1-x710 is not
>>>>>>                         included or not updated.
>>>>>>
>>>>>>>                         There is a different error when running
>>>>>>>                         on the Intel E810s. It appears to me
>>>>>>>                         like it starts DPDK, does some
>>>>>>>                         configuration inside DPDK and on the
>>>>>>>                         device, and then fails to bring the
>>>>>>>                         device back up. Since this error seems
>>>>>>>                         very non-trivial, I will also attach
>>>>>>>                         this log.
>>>>>>
>>>>>>                         This one is a bit simpler. Few lines
>>>>>>                         after the first ERROR in log I see the
>>>>>>                         following:
>>>>>>                         WARN  RCF  DPDK 13:06:00.144
>>>>>>                         ice_program_hw_rx_queue(): currently
>>>>>>                         package doesn't support RXDID (22)
>>>>>>                         ice_rx_queue_start(): fail to program RX
>>>>>>                         queue 0
>>>>>>                         ice_dev_start(): fail to start Rx queue 0
>>>>>>                         Device with port_id=0 already stopped
>>>>>>
>>>>>>                         It is stdout/stderr from test agent which
>>>>>>                         runs DPDK. Same logs in plain format are
>>>>>>                         available in ta.DPDK file.
>>>>>>                         I'm not an expert here, but I vaguely
>>>>>>                         remember that E810 requires correct
>>>>>>                         firmware and DDP to be loaded.
>>>>>>                         There is some information in
>>>>>>                         dpdk/doc/guides/nics/ice.rst.
>>>>>>
>>>>>>                         You can try to add
>>>>>>                         --dev-args=safe-mode-support=1
>>>>>>                         command-line option described there.
>>>>>>
>>>>>>                         Hope it helps,
>>>>>>                         Andrew.
>>>>>>
>>>>>>>
>>>>>>>                         Thanks,
>>>>>>>                         Adam
>>>>>>>
>>>>>>>                         On Fri, Sep 1, 2023 at 3:59 AM Andrew
>>>>>>>                         Rybchenko
>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>                             Hi Adam,
>>>>>>>
>>>>>>>                             On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>>>                             Hi Andrew,
>>>>>>>>
>>>>>>>>                             I have one additional question as
>>>>>>>>                             well: Does the test engine support
>>>>>>>>                             running tests on two ARMv8 test agents?
>>>>>>>>
>>>>>>>>                                 1. We'll sort out warnings this
>>>>>>>>                                 week. Thanks for heads up.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             Great. Let me know when that's fixed.
>>>>>>>
>>>>>>>                             Done. We also fixed a number of
>>>>>>>                             warnings in TE.
>>>>>>>                             Also we fixed root test package name
>>>>>>>                             to be consistent with the repository
>>>>>>>                             name.
>>>>>>>
>>>>>>>>                                 Support for old LTS branches
>>>>>>>>                                 was dropped some time ago, but
>>>>>>>>                                 in the future it is definitely
>>>>>>>>                                 possible to keep it for new LTS
>>>>>>>>                                 branches. I think 22.11 is
>>>>>>>>                                 supported, but I'm not sure
>>>>>>>>                                 about older LTS releases.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             Good to know.
>>>>>>>>
>>>>>>>>                                 2. You can add command-line
>>>>>>>>                                 option --sanity to run tests
>>>>>>>>                                 marked with TEST_HARNESS_SANITY
>>>>>>>>                                 requirement (see
>>>>>>>>                                 dpdk-ethdev-ts/scripts/run.sh
>>>>>>>>                                 and grep TEST_HARNESS_SANITY
>>>>>>>>                                 dpdk-ethdev-ts to see which
>>>>>>>>                                 tests are marked). Yes, there
>>>>>>>>                                 is a space for terminology
>>>>>>>>                                 improvement here. We'll do it.
>>>>>>>>
>>>>>>>
>>>>>>>                             Done. Now it is called --checkup.
>>>>>>>
>>>>>>>>
>>>>>>>>                                 Also it takes a lot of time
>>>>>>>>                                 because of failures and tests
>>>>>>>>                                 which wait for some timeout.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             That makes sense to me. We'll use
>>>>>>>>                             the time to complete tests on
>>>>>>>>                             virtio or the Intel devices as a
>>>>>>>>                             reference for how long the tests
>>>>>>>>                             really take to complete.
>>>>>>>>                             We will explore the possibility of
>>>>>>>>                             periodically running the sanity
>>>>>>>>                             tests for patches.
>>>>>>>
>>>>>>>                             I'll double-check and let you know
>>>>>>>                             how long entire TS runs on Intel
>>>>>>>                             X710, E810, Mellanox CX5 and virtio
>>>>>>>                             net. Just to ensure that time
>>>>>>>                             observed in your case looks the same.
>>>>>>>
>>>>>>>>
>>>>>>>>                                 The test harness can provide
>>>>>>>>                                 coverage reports based on gcov,
>>>>>>>>                                 but I'm not sure what you mean
>>>>>>>>                                 by a "dial" to control test
>>>>>>>>                                 coverage. Provided reports are
>>>>>>>>                                 rather for human to analyze.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             The general idea is to have some
>>>>>>>>                             kind of parameter on the test
>>>>>>>>                             suite, which could be an integer
>>>>>>>>                             ranging from zero to ten, that
>>>>>>>>                             controls how many tests are run
>>>>>>>>                             based on how important the test is.
>>>>>>>>
>>>>>>>>                             Similar to how some command line
>>>>>>>>                             interfaces provide a verbosity
>>>>>>>>                             level parameter (some number of
>>>>>>>>                             "-v" arguments) to control the
>>>>>>>>                             importance of the information in
>>>>>>>>                             the log.
>>>>>>>>                             The verbosity level zero only
>>>>>>>>                             prints very important log messages,
>>>>>>>>                             while ten prints everything.
>>>>>>>>
>>>>>>>>                             In much the same manner as above,
>>>>>>>>                             this "dial" parameter controls what
>>>>>>>>                             tests are run and with what
>>>>>>>>                             parameters based on how important
>>>>>>>>                             those tests and test parameter
>>>>>>>>                             combinations are.
>>>>>>>>                             Coverage Level zero tells the suite
>>>>>>>>                             to run a very basic set of
>>>>>>>>                             important tests, with minimal
>>>>>>>>                             parameterization. This mode would
>>>>>>>>                             take only ~5-10 minutes to run.
>>>>>>>>                             In contrast, Coverage Level ten
>>>>>>>>                             includes all the edge cases, every
>>>>>>>>                             combination of test parameters,
>>>>>>>>                             everything the test suite can do,
>>>>>>>>                             which takes the normal several
>>>>>>>>                             hours to run.
>>>>>>>>                             The values 1 - 9 are between those
>>>>>>>>                             two extremes, allowing the user to
>>>>>>>>                             get a gradient of test coverage in
>>>>>>>>                             the results and to limit the
>>>>>>>>                             running time.
>>>>>>>>
>>>>>>>>                             Then we could, for example, run the
>>>>>>>>                             "run.sh" with a level of 2 or 3 for
>>>>>>>>                             incoming patches that need quick
>>>>>>>>                             results, and with a level of 10 for
>>>>>>>>                             the less often run periodic tests
>>>>>>>>                             performed on main or LTS branches.
>>>>>>>
>>>>>>>                             Understood now. Thanks a lot for the
>>>>>>>                             idea. We'll discuss it and come back.
>>>>>>>
>>>>>>>>                                 3. Yes, really many tests on
>>>>>>>>                                 Mellanox CX5 NICs report
>>>>>>>>                                 unexpected testing results.
>>>>>>>>                                 Unfortunately it is time
>>>>>>>>                                 consuming to fill in
>>>>>>>>                                 expectations database since it
>>>>>>>>                                 is necessary to analyze testing
>>>>>>>>                                 results and classify if it is a
>>>>>>>>                                 bug or just acceptable
>>>>>>>>                                 behaviour aspect.
>>>>>>>>
>>>>>>>>                                 Bublik allows to compare
>>>>>>>>                                 results of two runs. It is
>>>>>>>>                                 useful for human, but still not
>>>>>>>>                                 good for automation.
>>>>>>>>
>>>>>>>>                                 I have local patch for mlx5
>>>>>>>>                                 driver which reports Tx ring
>>>>>>>>                                 size maximum. It makes pass
>>>>>>>>                                 rate higher. It is a problem
>>>>>>>>                                 for test harness that mlx5 does
>>>>>>>>                                 not report limits right now.
>>>>>>>>
>>>>>>>>                                 Pass rate on Intel X710 is
>>>>>>>>                                 about 92% on my test rig. Pass
>>>>>>>>                                 rate on virtio net is 99% right
>>>>>>>>                                 now and could be done 100%
>>>>>>>>                                 easily (just one thing to fix
>>>>>>>>                                 in expectations).
>>>>>>>>
>>>>>>>>                                 I think logs storage setup is
>>>>>>>>                                 essential for logs analysis. Of
>>>>>>>>                                 course, you can request HTML
>>>>>>>>                                 logs when you run tests
>>>>>>>>                                 (--log-html=html) or generate
>>>>>>>>                                 after run using
>>>>>>>>                                 dpdk-ethdev-ts/scripts/html-log.sh
>>>>>>>>                                 and open index.html in a
>>>>>>>>                                 browser, but logs storage makes
>>>>>>>>                                 it more convenient.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             We are interested in setting up
>>>>>>>>                             Bublik, potentially as an
>>>>>>>>                             externally-facing component, once
>>>>>>>>                             we have our process of running the
>>>>>>>>                             test suite stabilized.
>>>>>>>>                             Once we are able to run the test
>>>>>>>>                             suite again, I'll see what the pass
>>>>>>>>                             rate is on our other hardware.
>>>>>>>>                             Good to know that it isn't an issue
>>>>>>>>                             with our dev testbed causing the
>>>>>>>>                             high fail rate.
>>>>>>>>
>>>>>>>>                             For Intel hardware, we have an
>>>>>>>>                             XL710 and an Intel E810-C in our
>>>>>>>>                             development testbed. Although they
>>>>>>>>                             are slightly different devices,
>>>>>>>>                             ideally the pass rate will be
>>>>>>>>                             identical or similar. I have yet to
>>>>>>>>                             set up a VM pair for virtio, but we
>>>>>>>>                             will soon.
>>>>>>>>
>>>>>>>>                                 Latest version of
>>>>>>>>                                 test-environment has examples
>>>>>>>>                                 of our CGI scripts which we use
>>>>>>>>                                 for log storage (see
>>>>>>>>                                 tools/log_server/README.md).
>>>>>>>>
>>>>>>>>                                 Also all bits for Jenkins setup
>>>>>>>>                                 are available. See
>>>>>>>>                                 dpdk-ethdev-ts/jenkins/README.md
>>>>>>>>                                 and examples of jenkins files
>>>>>>>>                                 in ts-rigs-sample.
>>>>>>>>
>>>>>>>>
>>>>>>>>                             Jenkins integration, setting up
>>>>>>>>                             production rig configurations, and
>>>>>>>>                             permanent log storage will be our
>>>>>>>>                             next steps once I am able to run
>>>>>>>>                             the tests again.
>>>>>>>>                             Unless there is an easy way to have
>>>>>>>>                             meson not pass "-Werror" into GCC.
>>>>>>>>                             Then I would be able to run the
>>>>>>>>                             test suite.
>>>>>>>
>>>>>>>                             Hopefully it is resolved now.
>>>>>>>
>>>>>>>                             I thought a bit more about your
>>>>>>>                             usecase for Jenkins. I'm not 100%
>>>>>>>                             sure that existing pipelines are
>>>>>>>                             convenient for your usecase.
>>>>>>>                             Fill free to ask questions when you
>>>>>>>                             are on it.
>>>>>>>
>>>>>>>                             Thanks,
>>>>>>>                             Andrew.
>>>>>>>
>>>>>>>>
>>>>>>>>                             Thanks,
>>>>>>>>                             Adam
>>>>>>>>
>>>>>>>>
>>>>>>>>                                 On 8/29/23 17:02, Adam Hassick
>>>>>>>>                                 wrote:
>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>
>>>>>>>>>                                 That fix seems to have
>>>>>>>>>                                 resolved the issue, thanks for
>>>>>>>>>                                 the quick turnaround time on
>>>>>>>>>                                 that patch.
>>>>>>>>>                                 Now that we have the RCF
>>>>>>>>>                                 timeout issue resolved, there
>>>>>>>>>                                 are a few other questions and
>>>>>>>>>                                 issues that we have about the
>>>>>>>>>                                 tests themselves.
>>>>>>>>>
>>>>>>>>>                                 1. The test suite fails to
>>>>>>>>>                                 build with a couple warnings.
>>>>>>>>>
>>>>>>>>>                                 Below is the stderr log from
>>>>>>>>>                                 compilation:
>>>>>>>>>
>>>>>>>>>                                     FAILED:
>>>>>>>>>                                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>                                     cc
>>>>>>>>>                                     -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>                                     -Ilib -I../../lib
>>>>>>>>>                                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>                                     -fdiagnostics-color=always
>>>>>>>>>                                     -pipe
>>>>>>>>>                                     -D_FILE_OFFSET_BITS=64
>>>>>>>>>                                     -Wall -Winvalid-pch
>>>>>>>>>                                     -Werror -g -D_GNU_SOURCE
>>>>>>>>>                                     -O0 -ggdb -Wall -W -fPIC
>>>>>>>>>                                     -MD -MQ
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -MF
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>                                     -o
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>                                     In function
>>>>>>>>>                                     ‘test_create_traffic_generator_params’:
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>                                     error: format not a string
>>>>>>>>>                                     literal and no format
>>>>>>>>>                                     arguments
>>>>>>>>>                                     [-Werror=format-security]
>>>>>>>>>                                     5577 |     rc =
>>>>>>>>>                                     te_kvpair_add(result, buf,
>>>>>>>>>                                     mode);
>>>>>>>>>                                     |     ^~
>>>>>>>>>                                     cc1: all warnings being
>>>>>>>>>                                     treated as errors
>>>>>>>>>                                     ninja: build stopped:
>>>>>>>>>                                     subcommand failed.
>>>>>>>>>                                     ninja: Entering directory `.'
>>>>>>>>>                                     FAILED:
>>>>>>>>>                                     lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>                                     cc
>>>>>>>>>                                     -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>                                     -Ilib -I../../lib
>>>>>>>>>                                     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>                                     -fdiagnostics-color=always
>>>>>>>>>                                     -pipe
>>>>>>>>>                                     -D_FILE_OFFSET_BITS=64
>>>>>>>>>                                     -Wall -Winvalid-pch
>>>>>>>>>                                     -Werror -g -D_GNU_SOURCE
>>>>>>>>>                                     -O0 -ggdb -Wall -W -fPIC
>>>>>>>>>                                     -MD -MQ
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -MF
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>                                     -o
>>>>>>>>>                                     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>                                     -c ../../lib/dpdk_pmd_ts.c
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>                                     In function
>>>>>>>>>                                     ‘test_create_traffic_generator_params’:
>>>>>>>>>                                     ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>                                     error: format not a string
>>>>>>>>>                                     literal and no format
>>>>>>>>>                                     arguments
>>>>>>>>>                                     [-Werror=format-security]
>>>>>>>>>                                     5577 |     rc =
>>>>>>>>>                                     te_kvpair_add(result, buf,
>>>>>>>>>                                     mode);
>>>>>>>>>                                     |     ^~
>>>>>>>>>                                     cc1: all warnings being
>>>>>>>>>                                     treated as errors
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 This error wasn't occurring
>>>>>>>>>                                 last week, which was the last
>>>>>>>>>                                 time I ran the tests.
>>>>>>>>>                                 The TE host and the DUT have
>>>>>>>>>                                 GCC v9.4.0 installed, and the
>>>>>>>>>                                 tester has GCC v11.4.0
>>>>>>>>>                                 installed, if this information
>>>>>>>>>                                 is helpful.
>>>>>>>>>
>>>>>>>>>                                 2. On the Mellanox CX5s, there
>>>>>>>>>                                 are over 6,000 tests run,
>>>>>>>>>                                 which collectively take around
>>>>>>>>>                                 9 hours. Is it possible, and
>>>>>>>>>                                 would it make sense, to lower
>>>>>>>>>                                 the test coverage and have the
>>>>>>>>>                                 test suite run faster?
>>>>>>>>>
>>>>>>>>>                                 For some context, we run
>>>>>>>>>                                 immediate testing on incoming
>>>>>>>>>                                 patches for DPDK main and
>>>>>>>>>                                 development branches, as well
>>>>>>>>>                                 as periodic test runs on the
>>>>>>>>>                                 main, stable, and LTS branches.
>>>>>>>>>                                 For us to consider including
>>>>>>>>>                                 this test suite as part of our
>>>>>>>>>                                 immediate testing on patches,
>>>>>>>>>                                 we would have to reduce the
>>>>>>>>>                                 test coverage to the most
>>>>>>>>>                                 important tests.
>>>>>>>>>                                 This is primarily to reduce
>>>>>>>>>                                 the testing time to, for
>>>>>>>>>                                 example, less than 30 minutes.
>>>>>>>>>                                 Testing on patches can't take
>>>>>>>>>                                 too long because the lab can
>>>>>>>>>                                 receive numerous patches each
>>>>>>>>>                                 day, which each require
>>>>>>>>>                                 individual testing runs.
>>>>>>>>>
>>>>>>>>>                                 At what frequency we run these
>>>>>>>>>                                 tests, and on what, still
>>>>>>>>>                                 needs to be discussed with the
>>>>>>>>>                                 DPDK community, but it would
>>>>>>>>>                                 be nice to know if the test
>>>>>>>>>                                 suite had a "dial" to control
>>>>>>>>>                                 the testing coverage.
>>>>>>>>>
>>>>>>>>>                                 3. We see a lot of test
>>>>>>>>>                                 failures on our Mellanox CX5
>>>>>>>>>                                 NICs. Around 2,300 of ~6,600
>>>>>>>>>                                 tests passed. Is there
>>>>>>>>>                                 anything we can do to diagnose
>>>>>>>>>                                 these test failures?
>>>>>>>>>
>>>>>>>>>                                 Thanks,
>>>>>>>>>                                 Adam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 On Tue, Aug 29, 2023 at
>>>>>>>>>                                 8:07 AM Andrew Rybchenko
>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>                                 wrote:
>>>>>>>>>
>>>>>>>>>                                     Hi Adam,
>>>>>>>>>
>>>>>>>>>                                     I've pushed the fix in
>>>>>>>>>                                     main branch and a new tag
>>>>>>>>>                                     v1.18.1. It should solve
>>>>>>>>>                                     the problem with IPv6
>>>>>>>>>                                     address from DNS.
>>>>>>>>>
>>>>>>>>>                                     Andrew.
>>>>>>>>>
>>>>>>>>>                                     On 8/29/23 00:05, Andrew
>>>>>>>>>                                     Rybchenko wrote:
>>>>>>>>>>                                     Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                                     > Does the test engine
>>>>>>>>>>                                     prefer to use IPv6 over
>>>>>>>>>>                                     IPv4 for initiating the
>>>>>>>>>>                                     RCF connection to the
>>>>>>>>>>                                     test bed hosts? And if
>>>>>>>>>>                                     so, is there a way to
>>>>>>>>>>                                     force it to use IPv4?
>>>>>>>>>>
>>>>>>>>>>                                     Brilliant idea. If DNS
>>>>>>>>>>                                     returns both IPv4 and
>>>>>>>>>>                                     IPv6 addresses in your
>>>>>>>>>>                                     case, I guess it is the
>>>>>>>>>>                                     root cause of the problem.
>>>>>>>>>>                                     Of course, it is TE
>>>>>>>>>>                                     problem since I see
>>>>>>>>>>                                     really weird code in
>>>>>>>>>>                                     lib/comm_net_engine/comm_net_engine.c
>>>>>>>>>>                                     line 135.
>>>>>>>>>>
>>>>>>>>>>                                     I've pushed fix to the
>>>>>>>>>>                                     branch
>>>>>>>>>>                                     user/arybchik/fix_ipv4_only
>>>>>>>>>>                                     in
>>>>>>>>>>                                     ts-factory/test-environment
>>>>>>>>>>                                     repository. Please, try.
>>>>>>>>>>
>>>>>>>>>>                                     It is late night fix with
>>>>>>>>>>                                     minimal testing and no
>>>>>>>>>>                                     review. I'll pass it
>>>>>>>>>>                                     through review process
>>>>>>>>>>                                     tomorrow and
>>>>>>>>>>                                     hopefully it will be
>>>>>>>>>>                                     released in one-two days.
>>>>>>>>>>
>>>>>>>>>>                                     Andrew.
>>>>>>>>>>
>>>>>>>>>>                                     On 8/28/23 18:02, Adam
>>>>>>>>>>                                     Hassick wrote:
>>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                                     We have yet to notice a
>>>>>>>>>>>                                     distinct pattern with
>>>>>>>>>>>                                     the failures. Sometimes,
>>>>>>>>>>>                                     the RCF will start and
>>>>>>>>>>>                                     connect without issue a
>>>>>>>>>>>                                     few times in a row
>>>>>>>>>>>                                     before failing to
>>>>>>>>>>>                                     connect again. Once the
>>>>>>>>>>>                                     issue begins to occur,
>>>>>>>>>>>                                     neither rebooting all of
>>>>>>>>>>>                                     the hosts (test engine
>>>>>>>>>>>                                     VM, tester, IUT) or
>>>>>>>>>>>                                     deleting all of the
>>>>>>>>>>>                                     build directories
>>>>>>>>>>>                                     (suites, agents, inst)
>>>>>>>>>>>                                     and rebooting the hosts
>>>>>>>>>>>                                     afterward resolves the
>>>>>>>>>>>                                     issue. When it begins
>>>>>>>>>>>                                     working again seems very
>>>>>>>>>>>                                     arbitrary to us.
>>>>>>>>>>>
>>>>>>>>>>>                                     I do usually try to
>>>>>>>>>>>                                     terminate the test
>>>>>>>>>>>                                     engine with Ctrl+C, but
>>>>>>>>>>>                                     when it hangs while
>>>>>>>>>>>                                     trying to start RCF,
>>>>>>>>>>>                                     that does not work.
>>>>>>>>>>>
>>>>>>>>>>>                                     Does the test engine
>>>>>>>>>>>                                     prefer to use IPv6 over
>>>>>>>>>>>                                     IPv4 for initiating the
>>>>>>>>>>>                                     RCF connection to the
>>>>>>>>>>>                                     test bed hosts? And if
>>>>>>>>>>>                                     so, is there a way to
>>>>>>>>>>>                                     force it to use IPv4?
>>>>>>>>>>>
>>>>>>>>>>>                                      - Adam
>>>>>>>>>>>
>>>>>>>>>>>                                     On Fri, Aug 25, 2023 at
>>>>>>>>>>>                                     1:35 PM Andrew Rybchenko
>>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                         > I'll double-check
>>>>>>>>>>>                                         test engine on
>>>>>>>>>>>                                         Ubuntu 20.04 and
>>>>>>>>>>>                                         Ubuntu 22.04.
>>>>>>>>>>>
>>>>>>>>>>>                                         Done. It works fine
>>>>>>>>>>>                                         for me without any
>>>>>>>>>>>                                         issues.
>>>>>>>>>>>
>>>>>>>>>>>                                         Have you noticed any
>>>>>>>>>>>                                         pattern when it
>>>>>>>>>>>                                         works or does not work?
>>>>>>>>>>>                                         May be it is a
>>>>>>>>>>>                                         problem of not clean
>>>>>>>>>>>                                         state after termination?
>>>>>>>>>>>                                         Does it work fine
>>>>>>>>>>>                                         the first time after
>>>>>>>>>>>                                         DUTs reboot?
>>>>>>>>>>>                                         How do you terminate
>>>>>>>>>>>                                         testing? It should
>>>>>>>>>>>                                         be done using Ctrl+C
>>>>>>>>>>>                                         in terminal where
>>>>>>>>>>>                                         you execute run.sh
>>>>>>>>>>>                                         command.
>>>>>>>>>>>                                          In this case it
>>>>>>>>>>>                                         should shutdown
>>>>>>>>>>>                                         gracefully and close
>>>>>>>>>>>                                         all test agents and
>>>>>>>>>>>                                         engine applications.
>>>>>>>>>>>
>>>>>>>>>>>                                         (I'm trying to
>>>>>>>>>>>                                         understand why
>>>>>>>>>>>                                         you've seen many
>>>>>>>>>>>                                         test agent
>>>>>>>>>>>                                         processes. It should
>>>>>>>>>>>                                         not happen.)
>>>>>>>>>>>
>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                         On 8/25/23 17:41,
>>>>>>>>>>>                                         Andrew Rybchenko wrote:
>>>>>>>>>>>>                                         On 8/25/23 17:06,
>>>>>>>>>>>>                                         Adam Hassick wrote:
>>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Two of our systems
>>>>>>>>>>>>>                                         (the Test Engine
>>>>>>>>>>>>>                                         runner and the DUT
>>>>>>>>>>>>>                                         host) are running
>>>>>>>>>>>>>                                         Ubuntu 20.04 LTS,
>>>>>>>>>>>>>                                         however this
>>>>>>>>>>>>>                                         morning I noticed
>>>>>>>>>>>>>                                         that the tester
>>>>>>>>>>>>>                                         system (the one
>>>>>>>>>>>>>                                         having issues) is
>>>>>>>>>>>>>                                         running Ubuntu
>>>>>>>>>>>>>                                         22.04 LTS.
>>>>>>>>>>>>>                                         This could be the
>>>>>>>>>>>>>                                         source of the
>>>>>>>>>>>>>                                         problem. I
>>>>>>>>>>>>>                                         encountered a
>>>>>>>>>>>>>                                         dependency issue
>>>>>>>>>>>>>                                         trying to run the
>>>>>>>>>>>>>                                         Test Engine on
>>>>>>>>>>>>>                                         22.04 LTS, so I
>>>>>>>>>>>>>                                         downgraded the
>>>>>>>>>>>>>                                         system. Since the
>>>>>>>>>>>>>                                         tester is also the
>>>>>>>>>>>>>                                         host having
>>>>>>>>>>>>>                                         connection issues,
>>>>>>>>>>>>>                                         I will try
>>>>>>>>>>>>>                                         downgrading that
>>>>>>>>>>>>>                                         system to 20.04,
>>>>>>>>>>>>>                                         and see if that
>>>>>>>>>>>>>                                         changes anything.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         Unlikely, but who
>>>>>>>>>>>>                                         knows. We run tests
>>>>>>>>>>>>                                         (DUTs) on Ubuntu
>>>>>>>>>>>>                                         20.04, Ubuntu
>>>>>>>>>>>>                                         22.04, Ubuntu
>>>>>>>>>>>>                                         22.10, Ubuntu
>>>>>>>>>>>>                                         23.04, Debian 11
>>>>>>>>>>>>                                         and Fedora 38 every
>>>>>>>>>>>>                                         night.
>>>>>>>>>>>>                                         Right now Debian 11
>>>>>>>>>>>>                                         is used for test
>>>>>>>>>>>>                                         engine in nightly
>>>>>>>>>>>>                                         regressions.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         I'll double-check
>>>>>>>>>>>>                                         test engine on
>>>>>>>>>>>>                                         Ubuntu 20.04 and
>>>>>>>>>>>>                                         Ubuntu 22.04.
>>>>>>>>>>>>
>>>>>>>>>>>>>                                         I did try passing
>>>>>>>>>>>>>                                         in the "--vg-rcf"
>>>>>>>>>>>>>                                         argument to the
>>>>>>>>>>>>>                                         run.sh script of
>>>>>>>>>>>>>                                         the test suite
>>>>>>>>>>>>>                                         after installing
>>>>>>>>>>>>>                                         valgrind, but
>>>>>>>>>>>>>                                         there was no
>>>>>>>>>>>>>                                         additional output
>>>>>>>>>>>>>                                         that I saw.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         Sorry, I should
>>>>>>>>>>>>                                         valgrind output
>>>>>>>>>>>>                                         should be in
>>>>>>>>>>>>                                         valgrind.te_rcf
>>>>>>>>>>>>                                         (direction where
>>>>>>>>>>>>                                         you run test engine).
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         I will try pulling
>>>>>>>>>>>>>                                         in the changes
>>>>>>>>>>>>>                                         you've pushed up,
>>>>>>>>>>>>>                                         and will see if
>>>>>>>>>>>>>                                         that fixes anything.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>>>                                         Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         On Fri, Aug 25,
>>>>>>>>>>>>>                                         2023 at 9:57 AM
>>>>>>>>>>>>>                                         Andrew Rybchenko
>>>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Hello Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             On 8/24/23
>>>>>>>>>>>>>                                             23:54, Andrew
>>>>>>>>>>>>>                                             Rybchenko wrote:
>>>>>>>>>>>>>>                                             I'd like to
>>>>>>>>>>>>>>                                             try to repeat
>>>>>>>>>>>>>>                                             the problem
>>>>>>>>>>>>>>                                             locally.
>>>>>>>>>>>>>>                                             Which Linux
>>>>>>>>>>>>>>                                             distro is
>>>>>>>>>>>>>>                                             running on
>>>>>>>>>>>>>>                                             test engine
>>>>>>>>>>>>>>                                             and agents?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             In fact I
>>>>>>>>>>>>>>                                             know one
>>>>>>>>>>>>>>                                             problem with
>>>>>>>>>>>>>>                                             Debian 12 and
>>>>>>>>>>>>>>                                             Fedora 38 and
>>>>>>>>>>>>>>                                             we have
>>>>>>>>>>>>>>                                             patch in
>>>>>>>>>>>>>>                                             review to fix
>>>>>>>>>>>>>>                                             it, however,
>>>>>>>>>>>>>>                                             the behaviour
>>>>>>>>>>>>>>                                             is different in
>>>>>>>>>>>>>>                                             this case, so
>>>>>>>>>>>>>>                                             it is unlike
>>>>>>>>>>>>>>                                             the same problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             I've just
>>>>>>>>>>>>>                                             published a
>>>>>>>>>>>>>                                             new tag which
>>>>>>>>>>>>>                                             fixes known
>>>>>>>>>>>>>                                             test engine
>>>>>>>>>>>>>                                             side problems
>>>>>>>>>>>>>                                             on Debian 12
>>>>>>>>>>>>>                                             and Fedora 38.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             One more idea
>>>>>>>>>>>>>>                                             is to install
>>>>>>>>>>>>>>                                             valgrind on
>>>>>>>>>>>>>>                                             the test
>>>>>>>>>>>>>>                                             engine host and
>>>>>>>>>>>>>>                                             run with
>>>>>>>>>>>>>>                                             option
>>>>>>>>>>>>>>                                             --vg-rcf to
>>>>>>>>>>>>>>                                             check if
>>>>>>>>>>>>>>                                             something
>>>>>>>>>>>>>>                                             weird is
>>>>>>>>>>>>>>                                             happening.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             What I don't
>>>>>>>>>>>>>>                                             understand
>>>>>>>>>>>>>>                                             right now is
>>>>>>>>>>>>>>                                             why I see
>>>>>>>>>>>>>>                                             just one
>>>>>>>>>>>>>>                                             failed attempt
>>>>>>>>>>>>>>                                             to connect in
>>>>>>>>>>>>>>                                             your log.txt
>>>>>>>>>>>>>>                                             and then
>>>>>>>>>>>>>>                                             Logger
>>>>>>>>>>>>>>                                             shutdown
>>>>>>>>>>>>>>                                             after 9 minutes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On 8/24/23
>>>>>>>>>>>>>>                                             23:29, Adam
>>>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>>>                                              > Is there
>>>>>>>>>>>>>>>                                             any firewall
>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>                                             network or
>>>>>>>>>>>>>>>                                             on test
>>>>>>>>>>>>>>>                                             hosts which
>>>>>>>>>>>>>>>                                             could block
>>>>>>>>>>>>>>>                                             incoming TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             to the port
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             from the
>>>>>>>>>>>>>>>                                             host where
>>>>>>>>>>>>>>>                                             you run test
>>>>>>>>>>>>>>>                                             engine?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             Our test
>>>>>>>>>>>>>>>                                             engine host
>>>>>>>>>>>>>>>                                             and the
>>>>>>>>>>>>>>>                                             testbed are
>>>>>>>>>>>>>>>                                             on the same
>>>>>>>>>>>>>>>                                             subnet. The
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             does work
>>>>>>>>>>>>>>>                                             sometimes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                              > If
>>>>>>>>>>>>>>>                                             behaviour
>>>>>>>>>>>>>>>                                             the same on
>>>>>>>>>>>>>>>                                             the next try
>>>>>>>>>>>>>>>                                             and you see
>>>>>>>>>>>>>>>                                             that test
>>>>>>>>>>>>>>>                                             agent is
>>>>>>>>>>>>>>>                                             kept
>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>                                             could you
>>>>>>>>>>>>>>>                                             check using
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > # netstat
>>>>>>>>>>>>>>>                                             -tnlp
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > that Test
>>>>>>>>>>>>>>>                                             Agent is
>>>>>>>>>>>>>>>                                             listening on
>>>>>>>>>>>>>>>                                             the port and
>>>>>>>>>>>>>>>                                             try to
>>>>>>>>>>>>>>>                                             establish
>>>>>>>>>>>>>>>                                             TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             from test
>>>>>>>>>>>>>>>                                             agent using
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > $ telnet
>>>>>>>>>>>>>>>                                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                              >
>>>>>>>>>>>>>>>                                              > and check
>>>>>>>>>>>>>>>                                             if TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             could be
>>>>>>>>>>>>>>>                                             established.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             I was able
>>>>>>>>>>>>>>>                                             to replicate
>>>>>>>>>>>>>>>                                             the same
>>>>>>>>>>>>>>>                                             behavior
>>>>>>>>>>>>>>>                                             again, where
>>>>>>>>>>>>>>>                                             it hangs
>>>>>>>>>>>>>>>                                             while RCF is
>>>>>>>>>>>>>>>                                             trying to start.
>>>>>>>>>>>>>>>                                             Running this
>>>>>>>>>>>>>>>                                             command, I
>>>>>>>>>>>>>>>                                             see this in
>>>>>>>>>>>>>>>                                             the output:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             tcp        0
>>>>>>>>>>>>>>>                                                0
>>>>>>>>>>>>>>>                                             0.0.0.0:23571
>>>>>>>>>>>>>>>                                             <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>                                             <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>                                             <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>                                             0.0.0.0:*
>>>>>>>>>>>>>>>                                             LISTEN  18599/ta
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             So it seems
>>>>>>>>>>>>>>>                                             like it is
>>>>>>>>>>>>>>>                                             listening on
>>>>>>>>>>>>>>>                                             the correct
>>>>>>>>>>>>>>>                                             port.
>>>>>>>>>>>>>>>                                             Additionally,
>>>>>>>>>>>>>>>                                             I was able
>>>>>>>>>>>>>>>                                             to connect
>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>                                             machine from
>>>>>>>>>>>>>>>                                             our Test
>>>>>>>>>>>>>>>                                             Engine host
>>>>>>>>>>>>>>>                                             using
>>>>>>>>>>>>>>>                                             telnet. It
>>>>>>>>>>>>>>>                                             printed the
>>>>>>>>>>>>>>>                                             PID of the
>>>>>>>>>>>>>>>                                             process once
>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             was opened.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             I tried
>>>>>>>>>>>>>>>                                             running the
>>>>>>>>>>>>>>>                                             "ta"
>>>>>>>>>>>>>>>                                             application
>>>>>>>>>>>>>>>                                             manually on
>>>>>>>>>>>>>>>                                             the command
>>>>>>>>>>>>>>>                                             line, and it
>>>>>>>>>>>>>>>                                             didn't print
>>>>>>>>>>>>>>>                                             anything at all.
>>>>>>>>>>>>>>>                                             Maybe the
>>>>>>>>>>>>>>>                                             issue is
>>>>>>>>>>>>>>>                                             something on
>>>>>>>>>>>>>>>                                             the Test
>>>>>>>>>>>>>>>                                             Engine side.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             On Thu, Aug
>>>>>>>>>>>>>>>                                             24, 2023 at
>>>>>>>>>>>>>>>                                             2:35 PM
>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Hi Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                  > On
>>>>>>>>>>>>>>>                                             the tester
>>>>>>>>>>>>>>>                                             host (which
>>>>>>>>>>>>>>>                                             appears to
>>>>>>>>>>>>>>>                                             be the Peer
>>>>>>>>>>>>>>>                                             agent), there
>>>>>>>>>>>>>>>                                                 are four
>>>>>>>>>>>>>>>                                             processes
>>>>>>>>>>>>>>>                                             that I see
>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>                                             which look
>>>>>>>>>>>>>>>                                             like the test
>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>                                             processes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Before
>>>>>>>>>>>>>>>                                             the next try
>>>>>>>>>>>>>>>                                             I'd
>>>>>>>>>>>>>>>                                             recommend to
>>>>>>>>>>>>>>>                                             kill these
>>>>>>>>>>>>>>>                                             processes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Is there
>>>>>>>>>>>>>>>                                             any firewall
>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>                                             network or
>>>>>>>>>>>>>>>                                             on test
>>>>>>>>>>>>>>>                                             hosts which
>>>>>>>>>>>>>>>                                             could
>>>>>>>>>>>>>>>                                                 block
>>>>>>>>>>>>>>>                                             incoming TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             to the port
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             from the host
>>>>>>>>>>>>>>>                                                 where
>>>>>>>>>>>>>>>                                             you run test
>>>>>>>>>>>>>>>                                             engine?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 If
>>>>>>>>>>>>>>>                                             behaviour
>>>>>>>>>>>>>>>                                             the same on
>>>>>>>>>>>>>>>                                             the next try
>>>>>>>>>>>>>>>                                             and you see
>>>>>>>>>>>>>>>                                             that test
>>>>>>>>>>>>>>>                                             agent is
>>>>>>>>>>>>>>>                                                 kept
>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>                                             could you
>>>>>>>>>>>>>>>                                             check using
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 #
>>>>>>>>>>>>>>>                                             netstat -tnlp
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>                                             Test Agent
>>>>>>>>>>>>>>>                                             is listening
>>>>>>>>>>>>>>>                                             on the port
>>>>>>>>>>>>>>>                                             and try to
>>>>>>>>>>>>>>>                                             establish TCP
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             from test
>>>>>>>>>>>>>>>                                             agent using
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 $ telnet
>>>>>>>>>>>>>>>                                             iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             23571
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>                                             check if TCP
>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>                                             could be
>>>>>>>>>>>>>>>                                             established.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Another
>>>>>>>>>>>>>>>                                             idea is to
>>>>>>>>>>>>>>>                                             login Tester
>>>>>>>>>>>>>>>                                             under root
>>>>>>>>>>>>>>>                                             as testing
>>>>>>>>>>>>>>>                                             does, get
>>>>>>>>>>>>>>>                                                 start TA
>>>>>>>>>>>>>>>                                             command from
>>>>>>>>>>>>>>>                                             the log and
>>>>>>>>>>>>>>>                                             try it by
>>>>>>>>>>>>>>>                                             hands
>>>>>>>>>>>>>>>                                             without -n and
>>>>>>>>>>>>>>>                                                 remove
>>>>>>>>>>>>>>>                                             extra escaping.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 # sudo
>>>>>>>>>>>>>>>                                             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>                                             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>                                             /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>>>>                                             Peer 23571
>>>>>>>>>>>>>>>                                             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>                                             Hopefully in
>>>>>>>>>>>>>>>                                             this case
>>>>>>>>>>>>>>>                                             test agent
>>>>>>>>>>>>>>>                                             directory
>>>>>>>>>>>>>>>                                             remains in
>>>>>>>>>>>>>>>                                             the /tmp and
>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>                                             don't need
>>>>>>>>>>>>>>>                                             to copy it
>>>>>>>>>>>>>>>                                             as testing does.
>>>>>>>>>>>>>>>                                                 May be
>>>>>>>>>>>>>>>                                             output could
>>>>>>>>>>>>>>>                                             shed some
>>>>>>>>>>>>>>>                                             light on
>>>>>>>>>>>>>>>                                             what's going on.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>                                             8/24/23
>>>>>>>>>>>>>>>                                             17:30, Adam
>>>>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 This is
>>>>>>>>>>>>>>>>                                             the output
>>>>>>>>>>>>>>>>                                             that I see
>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>                                             terminal
>>>>>>>>>>>>>>>>                                             when this
>>>>>>>>>>>>>>>>                                             failure
>>>>>>>>>>>>>>>>                                                 occurs,
>>>>>>>>>>>>>>>>                                             after the
>>>>>>>>>>>>>>>>                                             test agent
>>>>>>>>>>>>>>>>                                             binaries
>>>>>>>>>>>>>>>>                                             build and
>>>>>>>>>>>>>>>>                                             the test engine
>>>>>>>>>>>>>>>>                                                 starts:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                             Platform
>>>>>>>>>>>>>>>>                                             default
>>>>>>>>>>>>>>>>                                             build - pass
>>>>>>>>>>>>>>>>                                                 Simple
>>>>>>>>>>>>>>>>                                             RCF
>>>>>>>>>>>>>>>>                                             consistency
>>>>>>>>>>>>>>>>                                             check succeeded
>>>>>>>>>>>>>>>>                                             --->>>
>>>>>>>>>>>>>>>>                                             Starting
>>>>>>>>>>>>>>>>                                             Logger...done
>>>>>>>>>>>>>>>>                                             --->>>
>>>>>>>>>>>>>>>>                                             Starting
>>>>>>>>>>>>>>>>                                             RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>>>>                                             Connection
>>>>>>>>>>>>>>>>                                             timed
>>>>>>>>>>>>>>>>                                                 out
>>>>>>>>>>>>>>>>                                             iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 Then,
>>>>>>>>>>>>>>>>                                             it hangs
>>>>>>>>>>>>>>>>                                             here until
>>>>>>>>>>>>>>>>                                             I kill the
>>>>>>>>>>>>>>>>                                             "te_rcf"
>>>>>>>>>>>>>>>>                                             and "te_tee"
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                             processes.
>>>>>>>>>>>>>>>>                                             I let it
>>>>>>>>>>>>>>>>                                             hang for
>>>>>>>>>>>>>>>>                                             around 9
>>>>>>>>>>>>>>>>                                             minutes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 On the
>>>>>>>>>>>>>>>>                                             tester host
>>>>>>>>>>>>>>>>                                             (which
>>>>>>>>>>>>>>>>                                             appears to
>>>>>>>>>>>>>>>>                                             be the Peer
>>>>>>>>>>>>>>>>                                             agent),
>>>>>>>>>>>>>>>>                                             there are
>>>>>>>>>>>>>>>>                                                 four
>>>>>>>>>>>>>>>>                                             processes
>>>>>>>>>>>>>>>>                                             that I see
>>>>>>>>>>>>>>>>                                             running,
>>>>>>>>>>>>>>>>                                             which look
>>>>>>>>>>>>>>>>                                             like the
>>>>>>>>>>>>>>>>                                             test agent
>>>>>>>>>>>>>>>>                                                 processes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 ta.Peer
>>>>>>>>>>>>>>>>                                             is an empty
>>>>>>>>>>>>>>>>                                             file. I've
>>>>>>>>>>>>>>>>                                             attached
>>>>>>>>>>>>>>>>                                             the log.txt
>>>>>>>>>>>>>>>>                                             from this run.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                  - Adam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 On Thu,
>>>>>>>>>>>>>>>>                                             Aug 24,
>>>>>>>>>>>>>>>>                                             2023 at
>>>>>>>>>>>>>>>>                                             4:22 AM
>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>                                             Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             Yes,
>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>                                             is in
>>>>>>>>>>>>>>>>                                             seconds.
>>>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>>>                                             double-checked
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             that it
>>>>>>>>>>>>>>>>                                             goes to
>>>>>>>>>>>>>>>>                                             'copy_timeout'
>>>>>>>>>>>>>>>>                                             in
>>>>>>>>>>>>>>>>                                             ts-conf/rcf.conf.
>>>>>>>>>>>>>>>>                                             Description
>>>>>>>>>>>>>>>>                                             in in
>>>>>>>>>>>>>>>>                                             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             says that
>>>>>>>>>>>>>>>>                                             copy_timeout
>>>>>>>>>>>>>>>>                                             is in
>>>>>>>>>>>>>>>>                                             seconds and
>>>>>>>>>>>>>>>>                                             implementation
>>>>>>>>>>>>>>>>                                             in
>>>>>>>>>>>>>>>>                                             lib/rcfunix/rcfunix.c
>>>>>>>>>>>>>>>>                                             passes the
>>>>>>>>>>>>>>>>                                             value to
>>>>>>>>>>>>>>>>                                             select()
>>>>>>>>>>>>>>>>                                             tv_sec.
>>>>>>>>>>>>>>>>                                             Theoretically
>>>>>>>>>>>>>>>>                                             select()
>>>>>>>>>>>>>>>>                                             could be
>>>>>>>>>>>>>>>>                                             interrupted
>>>>>>>>>>>>>>>>                                             by signal,
>>>>>>>>>>>>>>>>                                             but I
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             think it is
>>>>>>>>>>>>>>>>                                             unlikely here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>                                             not sure
>>>>>>>>>>>>>>>>                                             that I
>>>>>>>>>>>>>>>>                                             understand
>>>>>>>>>>>>>>>>                                             what do you
>>>>>>>>>>>>>>>>                                             mean by RCF
>>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>>                                             timeout.
>>>>>>>>>>>>>>>>                                             Does it
>>>>>>>>>>>>>>>>                                             happen on
>>>>>>>>>>>>>>>>                                             TE startup
>>>>>>>>>>>>>>>>                                             when RCF
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             starts test
>>>>>>>>>>>>>>>>                                             agents. If
>>>>>>>>>>>>>>>>                                             so,
>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>                                             could help. Or
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             does it
>>>>>>>>>>>>>>>>                                             happen when
>>>>>>>>>>>>>>>>                                             tests are
>>>>>>>>>>>>>>>>                                             in
>>>>>>>>>>>>>>>>                                             progress,
>>>>>>>>>>>>>>>>                                             e.g. in the
>>>>>>>>>>>>>>>>                                             middle
>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>                                             a test. If
>>>>>>>>>>>>>>>>                                             so,
>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>                                             unrelated
>>>>>>>>>>>>>>>>                                             and most
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             likely
>>>>>>>>>>>>>>>>                                             either host
>>>>>>>>>>>>>>>>                                             with test
>>>>>>>>>>>>>>>>                                             agent dies
>>>>>>>>>>>>>>>>                                             or test
>>>>>>>>>>>>>>>>                                             agent itself
>>>>>>>>>>>>>>>>                                             crashes. It
>>>>>>>>>>>>>>>>                                             would be
>>>>>>>>>>>>>>>>                                             easier for
>>>>>>>>>>>>>>>>                                             me if
>>>>>>>>>>>>>>>>                                             classify it
>>>>>>>>>>>>>>>>                                             if you share
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             text log
>>>>>>>>>>>>>>>>                                             (log.txt,
>>>>>>>>>>>>>>>>                                             full or
>>>>>>>>>>>>>>>>                                             just
>>>>>>>>>>>>>>>>                                             corresponding
>>>>>>>>>>>>>>>>                                             fragment with
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                             some
>>>>>>>>>>>>>>>>                                             context).
>>>>>>>>>>>>>>>>                                             Also
>>>>>>>>>>>>>>>>                                             content of
>>>>>>>>>>>>>>>>                                             ta.DPDK or
>>>>>>>>>>>>>>>>                                             ta.Peer file
>>>>>>>>>>>>>>>>                                             depending
>>>>>>>>>>>>>>>>                                             on which
>>>>>>>>>>>>>>>>                                             agent has
>>>>>>>>>>>>>>>>                                             problems
>>>>>>>>>>>>>>>>                                             could shed
>>>>>>>>>>>>>>>>                                             some light.
>>>>>>>>>>>>>>>>                                             Corresponding
>>>>>>>>>>>>>>>>                                             files
>>>>>>>>>>>>>>>>                                             contain
>>>>>>>>>>>>>>>>                                             stdout/stderr
>>>>>>>>>>>>>>>>                                             of test agents.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>                                             8/23/23
>>>>>>>>>>>>>>>>                                             17:45, Adam
>>>>>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             I've set
>>>>>>>>>>>>>>>>>                                             up a test
>>>>>>>>>>>>>>>>>                                             rig
>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>                                             here, and
>>>>>>>>>>>>>>>>>                                             have created
>>>>>>>>>>>>>>>>>                                             configurations
>>>>>>>>>>>>>>>>>                                             for our
>>>>>>>>>>>>>>>>>                                             development
>>>>>>>>>>>>>>>>>                                             testbed
>>>>>>>>>>>>>>>>>                                             based off
>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>                                             examples.
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             We've been
>>>>>>>>>>>>>>>>>                                             able to
>>>>>>>>>>>>>>>>>                                             get the
>>>>>>>>>>>>>>>>>                                             test suite
>>>>>>>>>>>>>>>>>                                             to run
>>>>>>>>>>>>>>>>>                                             manually on
>>>>>>>>>>>>>>>>>                                             Mellanox
>>>>>>>>>>>>>>>>>                                             CX5
>>>>>>>>>>>>>>>>>                                             devices once.
>>>>>>>>>>>>>>>>>                                             However,
>>>>>>>>>>>>>>>>>                                             we are
>>>>>>>>>>>>>>>>>                                             running
>>>>>>>>>>>>>>>>>                                             into an
>>>>>>>>>>>>>>>>>                                             issue
>>>>>>>>>>>>>>>>>                                             where,
>>>>>>>>>>>>>>>>>                                             when RCF
>>>>>>>>>>>>>>>>>                                             starts,
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             the RCF
>>>>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>>>>                                             times out
>>>>>>>>>>>>>>>>>                                             very
>>>>>>>>>>>>>>>>>                                             frequently.
>>>>>>>>>>>>>>>>>                                             We aren't sure
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             why this
>>>>>>>>>>>>>>>>>                                             is the case.
>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>                                             works
>>>>>>>>>>>>>>>>>                                             sometimes,
>>>>>>>>>>>>>>>>>                                             but most
>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>                                             time when
>>>>>>>>>>>>>>>>>                                             we try to run
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             the test
>>>>>>>>>>>>>>>>>                                             engine, it
>>>>>>>>>>>>>>>>>                                             encounters
>>>>>>>>>>>>>>>>>                                             this issue.
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             I've tried
>>>>>>>>>>>>>>>>>                                             changing
>>>>>>>>>>>>>>>>>                                             the RCF
>>>>>>>>>>>>>>>>>                                             port by
>>>>>>>>>>>>>>>>>                                             setting
>>>>>>>>>>>>>>>>>                                             "TE_RCF_PORT=<some
>>>>>>>>>>>>>>>>>                                             port
>>>>>>>>>>>>>>>>>                                             number>"
>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>                                             rebooting
>>>>>>>>>>>>>>>>>                                             the testbed
>>>>>>>>>>>>>>>>>                                             machines.
>>>>>>>>>>>>>>>>>                                             Neither
>>>>>>>>>>>>>>>>>                                             seems to
>>>>>>>>>>>>>>>>>                                             fix the issue.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>                                             also seems
>>>>>>>>>>>>>>>>>                                             like the
>>>>>>>>>>>>>>>>>                                             timeout
>>>>>>>>>>>>>>>>>                                             takes far
>>>>>>>>>>>>>>>>>                                             longer than 60
>>>>>>>>>>>>>>>>>                                             seconds,
>>>>>>>>>>>>>>>>>                                             even when
>>>>>>>>>>>>>>>>>                                             running
>>>>>>>>>>>>>>>>>                                             "export
>>>>>>>>>>>>>>>>>                                             TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             before I
>>>>>>>>>>>>>>>>>                                             try to run
>>>>>>>>>>>>>>>>>                                             the test
>>>>>>>>>>>>>>>>>                                             suite.
>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>                                             assume the
>>>>>>>>>>>>>>>>>                                             unit for
>>>>>>>>>>>>>>>>>                                             this
>>>>>>>>>>>>>>>>>                                             variable
>>>>>>>>>>>>>>>>>                                             is seconds?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                             Mon, Aug
>>>>>>>>>>>>>>>>>                                             21, 2023
>>>>>>>>>>>>>>>>>                                             at
>>>>>>>>>>>>>>>>>                                             10:19 AM
>>>>>>>>>>>>>>>>>                                             Adam Hassick
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             <ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>>>>                                             cloned the
>>>>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>                                             and will start
>>>>>>>>>>>>>>>>>                                             setting up
>>>>>>>>>>>>>>>>>                                             a
>>>>>>>>>>>>>>>>>                                             configuration
>>>>>>>>>>>>>>>>>                                             for our
>>>>>>>>>>>>>>>>>                                             development
>>>>>>>>>>>>>>>>>                                             testbed
>>>>>>>>>>>>>>>>>                                             today.
>>>>>>>>>>>>>>>>>                                             I'll let
>>>>>>>>>>>>>>>>>                                             you know
>>>>>>>>>>>>>>>>>                                             if I run
>>>>>>>>>>>>>>>>>                                             into any
>>>>>>>>>>>>>>>>>                                             difficulties
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             or have
>>>>>>>>>>>>>>>>>                                             any questions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                              - Adam
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             On Sun,
>>>>>>>>>>>>>>>>>                                             Aug 20,
>>>>>>>>>>>>>>>>>                                             2023 at
>>>>>>>>>>>>>>>>>                                             4:40 AM
>>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             I've published
>>>>>>>>>>>>>>>>>                                             https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Hopefully
>>>>>>>>>>>>>>>>>                                             it will
>>>>>>>>>>>>>>>>>                                             help to
>>>>>>>>>>>>>>>>>                                             define
>>>>>>>>>>>>>>>>>                                             your test
>>>>>>>>>>>>>>>>>                                             rigs and
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             successfully
>>>>>>>>>>>>>>>>>                                             run some
>>>>>>>>>>>>>>>>>                                             tests
>>>>>>>>>>>>>>>>>                                             manually.
>>>>>>>>>>>>>>>>>                                             Feel free to
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             ask any
>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>                                             and I'll
>>>>>>>>>>>>>>>>>                                             answer
>>>>>>>>>>>>>>>>>                                             here and
>>>>>>>>>>>>>>>>>                                             try to
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             update
>>>>>>>>>>>>>>>>>                                             documentation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Meanwhile
>>>>>>>>>>>>>>>>>                                             I'll
>>>>>>>>>>>>>>>>>                                             prepare
>>>>>>>>>>>>>>>>>                                             missing
>>>>>>>>>>>>>>>>>                                             bits for
>>>>>>>>>>>>>>>>>                                             steps (2) and
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             (3).
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Hopefully
>>>>>>>>>>>>>>>>>                                             everything
>>>>>>>>>>>>>>>>>                                             is in
>>>>>>>>>>>>>>>>>                                             place for
>>>>>>>>>>>>>>>>>                                             step (4),
>>>>>>>>>>>>>>>>>                                             but we
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             need to
>>>>>>>>>>>>>>>>>                                             make steps
>>>>>>>>>>>>>>>>>                                             (2) and
>>>>>>>>>>>>>>>>>                                             (3) first.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>                                             On 8/18/23
>>>>>>>>>>>>>>>>>                                             21:40,
>>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             > I've
>>>>>>>>>>>>>>>>>>                                             conferred
>>>>>>>>>>>>>>>>>>                                             with the
>>>>>>>>>>>>>>>>>>                                             rest of
>>>>>>>>>>>>>>>>>>                                             the team,
>>>>>>>>>>>>>>>>>>                                             and we
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             think it
>>>>>>>>>>>>>>>>>>                                             would be
>>>>>>>>>>>>>>>>>>                                             best to
>>>>>>>>>>>>>>>>>>                                             move
>>>>>>>>>>>>>>>>>>                                             forward
>>>>>>>>>>>>>>>>>>                                             with mainly
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             OK, I'll
>>>>>>>>>>>>>>>>>>                                             provide
>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>                                             sample on
>>>>>>>>>>>>>>>>>>                                             Monday
>>>>>>>>>>>>>>>>>>                                             for you.
>>>>>>>>>>>>>>>>>>                                             It is
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             almost
>>>>>>>>>>>>>>>>>>                                             ready
>>>>>>>>>>>>>>>>>>                                             right
>>>>>>>>>>>>>>>>>>                                             now, but
>>>>>>>>>>>>>>>>>>                                             I need to
>>>>>>>>>>>>>>>>>>                                             double-check
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             it before
>>>>>>>>>>>>>>>>>>                                             publishing.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             Regards,
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                             On
>>>>>>>>>>>>>>>>>>                                             8/17/23
>>>>>>>>>>>>>>>>>>                                             20:03,
>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>                                             Hassick
>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             I'm
>>>>>>>>>>>>>>>>>>>                                             adding
>>>>>>>>>>>>>>>>>>>                                             the CI
>>>>>>>>>>>>>>>>>>>                                             mailing
>>>>>>>>>>>>>>>>>>>                                             list to this
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             conversation.
>>>>>>>>>>>>>>>>>>>                                             Others
>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>                                             community
>>>>>>>>>>>>>>>>>>>                                             might find
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             this
>>>>>>>>>>>>>>>>>>>                                             conversation
>>>>>>>>>>>>>>>>>>>                                             valuable.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             We do
>>>>>>>>>>>>>>>>>>>                                             want to
>>>>>>>>>>>>>>>>>>>                                             run
>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>                                             on a
>>>>>>>>>>>>>>>>>>>                                             regular
>>>>>>>>>>>>>>>>>>>                                             basis. The
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Jenkins
>>>>>>>>>>>>>>>>>>>                                             integration
>>>>>>>>>>>>>>>>>>>                                             will be
>>>>>>>>>>>>>>>>>>>                                             very
>>>>>>>>>>>>>>>>>>>                                             useful
>>>>>>>>>>>>>>>>>>>                                             for us, as
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             most of
>>>>>>>>>>>>>>>>>>>                                             our CI
>>>>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>>>>                                             orchestrated
>>>>>>>>>>>>>>>>>>>                                             by Jenkins.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             I've
>>>>>>>>>>>>>>>>>>>                                             conferred
>>>>>>>>>>>>>>>>>>>                                             with the
>>>>>>>>>>>>>>>>>>>                                             rest of
>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>                                             team, and we
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             think it
>>>>>>>>>>>>>>>>>>>                                             would be
>>>>>>>>>>>>>>>>>>>                                             best to
>>>>>>>>>>>>>>>>>>>                                             move
>>>>>>>>>>>>>>>>>>>                                             forward
>>>>>>>>>>>>>>>>>>>                                             with mainly
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             option B.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             If you
>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>                                             like to
>>>>>>>>>>>>>>>>>>>                                             know
>>>>>>>>>>>>>>>>>>>                                             anything
>>>>>>>>>>>>>>>>>>>                                             about our
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             testbeds
>>>>>>>>>>>>>>>>>>>                                             that
>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>                                             help you
>>>>>>>>>>>>>>>>>>>                                             with
>>>>>>>>>>>>>>>>>>>                                             creating an
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             repo,
>>>>>>>>>>>>>>>>>>>                                             I'd be
>>>>>>>>>>>>>>>>>>>                                             happy to
>>>>>>>>>>>>>>>>>>>                                             answer any
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>>>                                             you have.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             We have
>>>>>>>>>>>>>>>>>>>                                             multiple
>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>                                             rigs (we
>>>>>>>>>>>>>>>>>>>                                             call these
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             "DUT-tester
>>>>>>>>>>>>>>>>>>>                                             pairs")
>>>>>>>>>>>>>>>>>>>                                             that we
>>>>>>>>>>>>>>>>>>>                                             run our
>>>>>>>>>>>>>>>>>>>                                             existing
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             hardware
>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>                                             on, with
>>>>>>>>>>>>>>>>>>>                                             differing
>>>>>>>>>>>>>>>>>>>                                             network
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             hardware
>>>>>>>>>>>>>>>>>>>                                             and CPU
>>>>>>>>>>>>>>>>>>>                                             architecture.
>>>>>>>>>>>>>>>>>>>                                             I
>>>>>>>>>>>>>>>>>>>                                             figured
>>>>>>>>>>>>>>>>>>>                                             this might
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             be an
>>>>>>>>>>>>>>>>>>>                                             important
>>>>>>>>>>>>>>>>>>>                                             detail.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             On Thu,
>>>>>>>>>>>>>>>>>>>                                             Aug 17,
>>>>>>>>>>>>>>>>>>>                                             2023 at
>>>>>>>>>>>>>>>>>>>                                             11:44 AM
>>>>>>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>>                                             <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Greatings
>>>>>>>>>>>>>>>>>>>                                             Adam,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             I'm
>>>>>>>>>>>>>>>>>>>                                             happy to
>>>>>>>>>>>>>>>>>>>                                             hear
>>>>>>>>>>>>>>>>>>>                                             that
>>>>>>>>>>>>>>>>>>>                                             you're
>>>>>>>>>>>>>>>>>>>                                             trying
>>>>>>>>>>>>>>>>>>>                                             to bring
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             it up.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             As I
>>>>>>>>>>>>>>>>>>>                                             understand
>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>                                             final
>>>>>>>>>>>>>>>>>>>                                             goal is
>>>>>>>>>>>>>>>>>>>                                             to run it on
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             regular
>>>>>>>>>>>>>>>>>>>                                             basis.
>>>>>>>>>>>>>>>>>>>                                             So, we
>>>>>>>>>>>>>>>>>>>                                             need to
>>>>>>>>>>>>>>>>>>>                                             make it
>>>>>>>>>>>>>>>>>>>                                             properly
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             from the
>>>>>>>>>>>>>>>>>>>                                             very
>>>>>>>>>>>>>>>>>>>                                             beginning.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Bring up
>>>>>>>>>>>>>>>>>>>                                             of all
>>>>>>>>>>>>>>>>>>>                                             features
>>>>>>>>>>>>>>>>>>>                                             consists
>>>>>>>>>>>>>>>>>>>                                             of 4 steps:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             1.
>>>>>>>>>>>>>>>>>>>                                             Create
>>>>>>>>>>>>>>>>>>>                                             site-specific
>>>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>>>                                             (we call it
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             ts-rigs)
>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>                                             contains
>>>>>>>>>>>>>>>>>>>                                             information
>>>>>>>>>>>>>>>>>>>                                             about test
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             rigs and
>>>>>>>>>>>>>>>>>>>                                             other
>>>>>>>>>>>>>>>>>>>                                             site-specific
>>>>>>>>>>>>>>>>>>>                                             information
>>>>>>>>>>>>>>>>>>>                                             like
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             where to
>>>>>>>>>>>>>>>>>>>                                             send
>>>>>>>>>>>>>>>>>>>                                             mails,
>>>>>>>>>>>>>>>>>>>                                             where to
>>>>>>>>>>>>>>>>>>>                                             store
>>>>>>>>>>>>>>>>>>>                                             logs etc.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             It is
>>>>>>>>>>>>>>>>>>>                                             required
>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>                                             manual
>>>>>>>>>>>>>>>>>>>                                             execution
>>>>>>>>>>>>>>>>>>>                                             as well,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             since
>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>                                             rigs
>>>>>>>>>>>>>>>>>>>                                             description
>>>>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>>>>                                             essential.
>>>>>>>>>>>>>>>>>>>                                             I'll
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             return
>>>>>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>>>>>                                             topic below.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             2. Setup
>>>>>>>>>>>>>>>>>>>                                             logs
>>>>>>>>>>>>>>>>>>>                                             storage
>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>                                             automated
>>>>>>>>>>>>>>>>>>>                                             runs.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Basically
>>>>>>>>>>>>>>>>>>>                                             it is a
>>>>>>>>>>>>>>>>>>>                                             disk
>>>>>>>>>>>>>>>>>>>                                             space
>>>>>>>>>>>>>>>>>>>                                             plus
>>>>>>>>>>>>>>>>>>>                                             apache2 web
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             server
>>>>>>>>>>>>>>>>>>>                                             with few
>>>>>>>>>>>>>>>>>>>                                             CGI
>>>>>>>>>>>>>>>>>>>                                             scripts
>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>                                             help a
>>>>>>>>>>>>>>>>>>>                                             lot to
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             save
>>>>>>>>>>>>>>>>>>>                                             disk space.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             3. Setup
>>>>>>>>>>>>>>>>>>>                                             Bublik
>>>>>>>>>>>>>>>>>>>                                             web
>>>>>>>>>>>>>>>>>>>                                             application
>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>                                             provides
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             web
>>>>>>>>>>>>>>>>>>>                                             interface
>>>>>>>>>>>>>>>>>>>                                             to view
>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>                                             results.
>>>>>>>>>>>>>>>>>>>                                             Same as
>>>>>>>>>>>>>>>>>>>                                             https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             4. Setup
>>>>>>>>>>>>>>>>>>>                                             Jenkins
>>>>>>>>>>>>>>>>>>>                                             to run
>>>>>>>>>>>>>>>>>>>                                             tests on
>>>>>>>>>>>>>>>>>>>                                             regularly,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             save
>>>>>>>>>>>>>>>>>>>                                             logs in
>>>>>>>>>>>>>>>>>>>                                             log
>>>>>>>>>>>>>>>>>>>                                             storage
>>>>>>>>>>>>>>>>>>>                                             (2) and
>>>>>>>>>>>>>>>>>>>                                             import it to
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             bublik (3).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Last few
>>>>>>>>>>>>>>>>>>>                                             month we
>>>>>>>>>>>>>>>>>>>                                             spent on
>>>>>>>>>>>>>>>>>>>                                             our
>>>>>>>>>>>>>>>>>>>                                             homework
>>>>>>>>>>>>>>>>>>>                                             to make
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             it
>>>>>>>>>>>>>>>>>>>                                             simpler
>>>>>>>>>>>>>>>>>>>                                             to bring
>>>>>>>>>>>>>>>>>>>                                             up
>>>>>>>>>>>>>>>>>>>                                             automated
>>>>>>>>>>>>>>>>>>>                                             execution
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             using
>>>>>>>>>>>>>>>>>>>                                             Jenkins -
>>>>>>>>>>>>>>>>>>>                                             https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>                                             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Corresponding
>>>>>>>>>>>>>>>>>>>                                             bits in
>>>>>>>>>>>>>>>>>>>                                             dpdk-ethdev-ts
>>>>>>>>>>>>>>>>>>>                                             will be
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             available
>>>>>>>>>>>>>>>>>>>                                             tomorrow.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Let's
>>>>>>>>>>>>>>>>>>>                                             return
>>>>>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>>>>>                                             step (1).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Unfortunately
>>>>>>>>>>>>>>>>>>>                                             there is
>>>>>>>>>>>>>>>>>>>                                             no
>>>>>>>>>>>>>>>>>>>                                             publicly
>>>>>>>>>>>>>>>>>>>                                             available
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             example
>>>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             repository
>>>>>>>>>>>>>>>>>>>                                             since
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             sensitive
>>>>>>>>>>>>>>>>>>>                                             site-specific
>>>>>>>>>>>>>>>>>>>                                             information
>>>>>>>>>>>>>>>>>>>                                             is located
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             there.
>>>>>>>>>>>>>>>>>>>                                             But I'm
>>>>>>>>>>>>>>>>>>>                                             ready to
>>>>>>>>>>>>>>>>>>>                                             help you
>>>>>>>>>>>>>>>>>>>                                             to create it
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             for UNH.
>>>>>>>>>>>>>>>>>>>                                             I see
>>>>>>>>>>>>>>>>>>>                                             two
>>>>>>>>>>>>>>>>>>>                                             options
>>>>>>>>>>>>>>>>>>>                                             here:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             (A) I'll
>>>>>>>>>>>>>>>>>>>                                             ask
>>>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>>>                                             based on
>>>>>>>>>>>>>>>>>>>                                             your
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             answers
>>>>>>>>>>>>>>>>>>>                                             will
>>>>>>>>>>>>>>>>>>>                                             create
>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>                                             first
>>>>>>>>>>>>>>>>>>>                                             draft
>>>>>>>>>>>>>>>>>>>                                             with my
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             comments.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             (B) I'll
>>>>>>>>>>>>>>>>>>>                                             make a
>>>>>>>>>>>>>>>>>>>                                             template/example
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             repo,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             publish
>>>>>>>>>>>>>>>>>>>                                             it and
>>>>>>>>>>>>>>>>>>>                                             you'll
>>>>>>>>>>>>>>>>>>>                                             create
>>>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>>>                                             ts-rigs
>>>>>>>>>>>>>>>>>>>                                             based
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             on it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Of
>>>>>>>>>>>>>>>>>>>                                             course,
>>>>>>>>>>>>>>>>>>>                                             I'll
>>>>>>>>>>>>>>>>>>>                                             help to
>>>>>>>>>>>>>>>>>>>                                             debug
>>>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>>>                                             finally
>>>>>>>>>>>>>>>>>>>                                             bring
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             it up in
>>>>>>>>>>>>>>>>>>>                                             any case.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             (A) is a
>>>>>>>>>>>>>>>>>>>                                             bit
>>>>>>>>>>>>>>>>>>>                                             simpler
>>>>>>>>>>>>>>>>>>>                                             for me
>>>>>>>>>>>>>>>>>>>                                             and you,
>>>>>>>>>>>>>>>>>>>                                             but (B) is
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             a bit
>>>>>>>>>>>>>>>>>>>                                             more
>>>>>>>>>>>>>>>>>>>                                             generic
>>>>>>>>>>>>>>>>>>>                                             and will
>>>>>>>>>>>>>>>>>>>                                             help other
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             potential
>>>>>>>>>>>>>>>>>>>                                             users to
>>>>>>>>>>>>>>>>>>>                                             bring it up.
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             We can
>>>>>>>>>>>>>>>>>>>                                             combine
>>>>>>>>>>>>>>>>>>>                                             (A)+(B).
>>>>>>>>>>>>>>>>>>>                                             I.e.
>>>>>>>>>>>>>>>>>>>                                             start
>>>>>>>>>>>>>>>>>>>                                             from (A).
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             What do
>>>>>>>>>>>>>>>>>>>                                             you think?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                             On
>>>>>>>>>>>>>>>>>>>                                             8/17/23
>>>>>>>>>>>>>>>>>>>                                             15:18,
>>>>>>>>>>>>>>>>>>>                                             Konstantin
>>>>>>>>>>>>>>>>>>>                                             Ushakov
>>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>>                                             Greetings
>>>>>>>>>>>>>>>>>>>>                                             Adam,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             Thanks
>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>                                             contacting
>>>>>>>>>>>>>>>>>>>>                                             us. I
>>>>>>>>>>>>>>>>>>>>                                             copy
>>>>>>>>>>>>>>>>>>>>                                             Andrew who
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>>                                             be
>>>>>>>>>>>>>>>>>>>>                                             happy
>>>>>>>>>>>>>>>>>>>>                                             to help
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                             Konstantin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                             On 16
>>>>>>>>>>>>>>>>>>>>>                                             Aug
>>>>>>>>>>>>>>>>>>>>>                                             2023,
>>>>>>>>>>>>>>>>>>>>>                                             at
>>>>>>>>>>>>>>>>>>>>>                                             21:50,
>>>>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>>>>                                             Hassick
>>>>>>>>>>>>>>>>>>>>>                                             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Greetings
>>>>>>>>>>>>>>>>>>>>>                                             Konstantin,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I am
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             process
>>>>>>>>>>>>>>>>>>>>>                                             of
>>>>>>>>>>>>>>>>>>>>>                                             setting
>>>>>>>>>>>>>>>>>>>>>                                             up the
>>>>>>>>>>>>>>>>>>>>>                                             DPDK
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Poll
>>>>>>>>>>>>>>>>>>>>>                                             Mode
>>>>>>>>>>>>>>>>>>>>>                                             Driver
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite
>>>>>>>>>>>>>>>>>>>>>                                             as an
>>>>>>>>>>>>>>>>>>>>>                                             addition
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             our
>>>>>>>>>>>>>>>>>>>>>                                             testing
>>>>>>>>>>>>>>>>>>>>>                                             coverage
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             DPDK
>>>>>>>>>>>>>>>>>>>>>                                             at the
>>>>>>>>>>>>>>>>>>>>>                                             UNH lab.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I have
>>>>>>>>>>>>>>>>>>>>>                                             some
>>>>>>>>>>>>>>>>>>>>>                                             questions
>>>>>>>>>>>>>>>>>>>>>                                             about
>>>>>>>>>>>>>>>>>>>>>                                             how to
>>>>>>>>>>>>>>>>>>>>>                                             set the
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite
>>>>>>>>>>>>>>>>>>>>>                                             arguments.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I have
>>>>>>>>>>>>>>>>>>>>>                                             been
>>>>>>>>>>>>>>>>>>>>>                                             able
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             configure
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             Test
>>>>>>>>>>>>>>>>>>>>>                                             Engine
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             connect
>>>>>>>>>>>>>>>>>>>>>                                             to the
>>>>>>>>>>>>>>>>>>>>>                                             hosts
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             testbed.
>>>>>>>>>>>>>>>>>>>>>                                             The
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             RCF,
>>>>>>>>>>>>>>>>>>>>>                                             Configurator,
>>>>>>>>>>>>>>>>>>>>>                                             and
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                             all
>>>>>>>>>>>>>>>>>>>>>                                             begin to
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             run,
>>>>>>>>>>>>>>>>>>>>>                                             however
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             prelude
>>>>>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>>>>>                                             test suite
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             fails
>>>>>>>>>>>>>>>>>>>>>                                             to run.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>                                             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             The
>>>>>>>>>>>>>>>>>>>>>                                             documentation
>>>>>>>>>>>>>>>>>>>>>                                             mentions
>>>>>>>>>>>>>>>>>>>>>                                             that
>>>>>>>>>>>>>>>>>>>>>                                             there are
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             several
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             parameters
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite,
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             like
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             IUT
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             link
>>>>>>>>>>>>>>>>>>>>>                                             MAC,
>>>>>>>>>>>>>>>>>>>>>                                             etc. These
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             seem
>>>>>>>>>>>>>>>>>>>>>                                             like
>>>>>>>>>>>>>>>>>>>>>                                             they
>>>>>>>>>>>>>>>>>>>>>                                             would
>>>>>>>>>>>>>>>>>>>>>                                             need
>>>>>>>>>>>>>>>>>>>>>                                             to be
>>>>>>>>>>>>>>>>>>>>>                                             set
>>>>>>>>>>>>>>>>>>>>>                                             somewhere
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             to run
>>>>>>>>>>>>>>>>>>>>>                                             many
>>>>>>>>>>>>>>>>>>>>>                                             of the
>>>>>>>>>>>>>>>>>>>>>                                             tests.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I see
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             Test
>>>>>>>>>>>>>>>>>>>>>                                             Engine
>>>>>>>>>>>>>>>>>>>>>                                             documentation,
>>>>>>>>>>>>>>>>>>>>>                                             there
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             are
>>>>>>>>>>>>>>>>>>>>>                                             instructions
>>>>>>>>>>>>>>>>>>>>>                                             on how
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             create new
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             parameters
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suites
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             configuration,
>>>>>>>>>>>>>>>>>>>>>                                             but
>>>>>>>>>>>>>>>>>>>>>                                             there
>>>>>>>>>>>>>>>>>>>>>                                             is
>>>>>>>>>>>>>>>>>>>>>                                             nothing
>>>>>>>>>>>>>>>>>>>>>                                             in the
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             user
>>>>>>>>>>>>>>>>>>>>>                                             guide
>>>>>>>>>>>>>>>>>>>>>                                             or in
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                             guide
>>>>>>>>>>>>>>>>>>>>>                                             for how to
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             set
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             arguments
>>>>>>>>>>>>>>>>>>>>>                                             for
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             parameters
>>>>>>>>>>>>>>>>>>>>>                                             when
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             running
>>>>>>>>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>>>>>>>>                                             test
>>>>>>>>>>>>>>>>>>>>>                                             suite
>>>>>>>>>>>>>>>>>>>>>                                             that I
>>>>>>>>>>>>>>>>>>>>>                                             can
>>>>>>>>>>>>>>>>>>>>>                                             find. I'm
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             not
>>>>>>>>>>>>>>>>>>>>>                                             sure
>>>>>>>>>>>>>>>>>>>>>                                             if I
>>>>>>>>>>>>>>>>>>>>>                                             need
>>>>>>>>>>>>>>>>>>>>>                                             to
>>>>>>>>>>>>>>>>>>>>>                                             write
>>>>>>>>>>>>>>>>>>>>>                                             my own
>>>>>>>>>>>>>>>>>>>>>                                             Tester
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             config,
>>>>>>>>>>>>>>>>>>>>>                                             or if
>>>>>>>>>>>>>>>>>>>>>                                             I
>>>>>>>>>>>>>>>>>>>>>                                             should
>>>>>>>>>>>>>>>>>>>>>                                             be
>>>>>>>>>>>>>>>>>>>>>                                             setting
>>>>>>>>>>>>>>>>>>>>>                                             these in
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             some
>>>>>>>>>>>>>>>>>>>>>                                             other way.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             How
>>>>>>>>>>>>>>>>>>>>>                                             should
>>>>>>>>>>>>>>>>>>>>>                                             these
>>>>>>>>>>>>>>>>>>>>>                                             values
>>>>>>>>>>>>>>>>>>>>>                                             be set?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             I'm
>>>>>>>>>>>>>>>>>>>>>                                             also
>>>>>>>>>>>>>>>>>>>>>                                             not
>>>>>>>>>>>>>>>>>>>>>                                             sure
>>>>>>>>>>>>>>>>>>>>>                                             what
>>>>>>>>>>>>>>>>>>>>>                                             environment
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             variables/arguments
>>>>>>>>>>>>>>>>>>>>>                                             are
>>>>>>>>>>>>>>>>>>>>>                                             strictly
>>>>>>>>>>>>>>>>>>>>>                                             necessary
>>>>>>>>>>>>>>>>>>>>>                                             or
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             which
>>>>>>>>>>>>>>>>>>>>>                                             are
>>>>>>>>>>>>>>>>>>>>>                                             optional.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Regards,
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             *Adam
>>>>>>>>>>>>>>>>>>>>>                                             Hassick*
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>>                                             +1
>>>>>>>>>>>>>>>>>>>>>                                             (603)
>>>>>>>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             *Adam
>>>>>>>>>>>>>>>>>>>                                             Hassick*
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             -- *Adam
>>>>>>>>>>>>>>>>>                                             Hassick*
>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             Senior
>>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>                                             (603) 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>                                             Developer
>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>                                             (603) 475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                             -- 
>>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>>                                             Senior Developer
>>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>>                                             <http://iol.unh.edu>
>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                             +1 (603)
>>>>>>>>>>>>>>>                                             475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                         --
>>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>>                                         InterOperability Lab
>>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                     --
>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 --
>>>>>>>>>                                 *Adam Hassick*
>>>>>>>>>                                 Senior Developer
>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                         --
>>>>>>>                         *Adam Hassick*
>>>>>>>                         Senior Developer
>>>>>>>                         UNH InterOperability Lab
>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                         +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>                     --
>>>>>>                     *Adam Hassick*
>>>>>>                     Senior Developer
>>>>>>                     UNH InterOperability Lab
>>>>>>                     ahassick@iol.unh.edu
>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --
>>>>>>                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>                 ahassick@iol.unh.edu
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>
>>>>
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 229841 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 58964 bytes --]

Hi Andrew,

Sorry about the two week radio silence, we're still trying to sort out
logistics for deployment on our end.

> I've created setup on ts-factory.io which allows to publish logs. Let me
know if you'd like to try it and I'll provide credentials, script and short
instruction.

We're interested in publishing some test logs to the ts-factory Bublik
instance in the meantime.

Thanks,
Adam

On Mon, Oct 23, 2023 at 7:11 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> > Now that our test results are in line with yours, we can begin looking
> into setting up the production environment.
>
> Please, let me know if you need any help with it or waiting for an input
> from me.
>
> Regards,
> Andrew.
>
> On 10/10/23 17:09, Adam Hassick wrote:
>
> Hi Andrew,
>
> Thank you for taking a look at our log. Netplan was attempting to run DHCP
> on our test links, and additionally I discovered that the NIC firmware was
> transmitting LLDP packets, causing tests to fail in the same way. Now that
> these problems have been fixed, our pass rate on the XL710 is approximately
> 91%. Now that our test results are in line with yours, we can begin looking
> into setting up the production environment.
>
> First, is it possible to run the test agent on ARM hosts? Our ARM testbeds
> have the best topology for running this test suite, with separate tester
> and DUT servers.
>
> We are testing this test suite on two x86 development servers using the
> test suite's recommended server topology. In contrast, our existing x86
> production testbeds which run DTS have a single server topology. This
> single server has both the tester NIC and the device under test NIC
> installed, with NUMA node separation between TRex and DPDK. We're going to
> test running the two test agent processes on the single-server testbeds if
> we cannot run this on ARM. Is there any reason you can think of that would
> prevent this setup from working?
>
> Once we figure out where this can live in production, then we will begin
> setting up log storage, Jenkins integration, and Bublik.
>
> Thanks,
> Adam
>
> On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> > Do these default to vfio-pci?
>>
>> Yes, vfio-pci is the default.
>> However, it does not work in the case of Mellanox which uses bifurcated
>> driver. It should mlx5_core for Mellanox NICs.
>>
>> > Here is the text log from a run on our Intel XL710 NICs, with the
>> expected result profile set to the X710.
>>
>> It is hard to analyze all tests using text logs, but I definitely see one
>> common problem. Tests receive unexpected packets and fail because of it.
>> Tests are written very strict from this point of view and it brought
>> fruits in the past when HW had bugs.
>> Are DUT and tester connected back-to-back on tested interfaces or via
>> switch?
>> If via switch, is it possible to isolate it from everything else?
>> If back-to-back, it could be some embedded SW/FW which regenerates these
>> packet.
>> I definitely see unexpected DHCP packets.
>>
>> > We haven't set up the Jenkins integration yet, however if this is
>> required to import the logs then we will prioritize that.
>>
>> Unfortunately manual runs do not generate all artifacts required to
>> import logs. However, we have almost solved it right now. Hopefully we'll
>> finalize it in a day or two. I'll let you know when these changes are
>> available.
>>
>> Regards,
>> Andrew.
>>
>> On 10/4/23 16:48, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set anywhere
>> in the default configurations for the Intel X710. Do these default to
>> vfio-pci?
>> We have IOMMU enabled on our development testbed, and should be able to
>> bind vfio-pci.
>> Here is the text log from a run on our Intel XL710 NICs, with the
>> expected result profile set to the X710. We haven't set up the Jenkins
>> integration yet, however if this is required to import the logs then we
>> will prioritize that.
>>  log.txt.tar.gz
>> <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>>
>> Thanks,
>> Adam
>>
>> On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> On 9/18/23 17:44, Adam Hassick wrote:
>>>
>>> Hi Andrew and Konstantin,
>>>
>>> Thank you for adding the tester-dial feature, this opens up the
>>> possibility for us to do CI integrated testing in the future.
>>>
>>> Our Mellanox pass rate is similar to yours (about ~2400 passing, ~4400
>>> failing), however our Intel pass rates are far worse.
>>> I will try running tests on the XL710 with the trc-tags argument set and
>>> see if it improves the pass rate.
>>> Another thing I noticed in the results you uploaded is that the results
>>> are tagged with vfio-pci and not i40e.
>>> Though in the environment dump, the driver on the test machine and the
>>> DUT are set to use the i40e driver. Is this important at all?
>>>
>>>
>>> I think it is a misunderstanding here. There are two kinds of driver in
>>> configuration: net driver and so-called DPDK driver.
>>> Net driver is a Linux kernel network device driver used on Tester side.
>>> DPDK driver is a Linux kernel driver to bind device to to use it with
>>> DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
>>> In the case of bifurcated driver (like mlx5_core) it is the same in both
>>> cases.
>>> In non-bifurcated case DPDK driver is some UIO driver(vfio-pci,
>>> uio-pci-generic or igb_uio).
>>> Some expectations depend on used UIO. For example, uio-pci-generic do
>>> not support many interrupts (used by usecases/rx_intr test cases).
>>> That's why we care corresponding TRC tag.
>>>
>>> TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's Intel case.
>>> Or uio-pci-generic if IOMMU is turned off on corresponding machines and
>>> Linux distro does not support VFIO no IOMMU mode.
>>>
>>> Andrew.
>>>
>>> There isn't anything preventing us from pushing our results up to the
>>> existing Bublik instance running at ts-factory.io that I can think of
>>> at the moment.
>>> We will have to work out how to submit our results to your Bublik
>>> instance in a controlled and secure manner in that case.
>>> As far as I know we won't need access controls for the results
>>> themselves. I'll discuss this with Patrick and will let you know once we
>>> confirm that it's fine.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> should we always auto-assign the tags or you don’t do it since it slows
>>>> down (by some seconds) the TE startup?
>>>>
>>>>
>>>> Tags are auto-assigned, but I guess it differs in Adam's case since NIC
>>>> is a bit different. Below test will help to understand if it is the root
>>>> cause of very different expectations. If pass rate will be close to mine,
>>>> I'll simply update TRC database to share expectations for mine NIC and NIC
>>>> used by Adam.
>>>>
>>>> Hi Adam,
>>>>
>>>> I think I second the question from Andrew - happy to help you with the
>>>> triage so that we get to the same baseline. Do you have a good way for us
>>>> to share the logs? I.e. say upload to ts-factory if we add strict
>>>> permissions system so it’s not publishing or any other way.
>>>>
>>>> Thanks,
>>>> Konstantin
>>>>
>>>> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>>
>>>> Hi Adam,
>>>>
>>>> I've uploaded fresh testing results to ts-factory.io [1] to be on the
>>>> same page.
>>>>
>>>> I think I know why your and mine results on Intel 710 series NICs
>>>> differ so much. Testing results expectations database
>>>> (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags.  I.e.
>>>> expectations depends on TRC tags discovered by helper scripts when testing
>>>> is started. These tags identify various aspects of what is tested. Ideally
>>>> expectations should be written in terms of root cause of the expected
>>>> behaviour. If it is a driver expectations, driver tag should be used. If it
>>>> is HW limitation, tags with PCI IDs should be used. However, it is not
>>>> always easy to classify it correctly if you're not involved in driver
>>>> development. So, in order case expectations for 710's Intel are filled in
>>>> in terms of PCI IDs. I guess PCI ID differ in your case and that's why
>>>> expectations filled in for my NIC do not apply to your runs.
>>>>
>>>> Just try to add the following option when you run on your 710's Intel
>>>> in order to mimic mine and see if it helps to achieve better pass rate.
>>>> --trc-tag=pci-8086-1572
>>>>
>>>> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for
>>>> --tester-dial option. It should have better coverage now.
>>>>
>>>> Andrew.
>>>>
>>>> [1]
>>>> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>>
>>>> On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>
>>>> Hi Adam,
>>>>
>>>> I've pushed new TE tag v1.20.0 which supported a new command-line
>>>> option --tester-dial=NUM where NUM is from 0 to 100. it allows to choose
>>>> percentage of tests to run. If you want stable set, you should pass
>>>> --tester-random-seed=0 (or other integer). It is the first sketch and we
>>>> have plans to improve it, but feedback would be welcome.
>>>>
>>>> > Is it needed on the tester?
>>>>
>>>> It is hard to say if it is strictly required for simple tests. However,
>>>> it is better to update Tester as well, since performance tests run DPDK on
>>>> Tester as well.
>>>>
>>>> > Are there any other manual setup steps for these devices that I might
>>>> be missing?
>>>>
>>>> I don't remember anything else.
>>>>
>>>> I think it is better to get down to details and take a look at logs.
>>>> I'm ready to help with it and explain what's happening there. May be it
>>>> will help to understand if it is a problem with setup/configuration.
>>>>
>>>> Text logs are not very convenient. Ideally logs should be imported to
>>>> bublik, however, manual runs do not provide all required artifacts right
>>>> now (Jenkins jobs generate all required artifacts).
>>>> Other option is 'tmp_raw_log' file (should be packed to make it
>>>> smaller) which could be converted to various log formats.
>>>> Would it be OK for you if I import your logs to bublik at ts-factory.io?
>>>> Or is it a problem that it is publicly available?
>>>> Would it help if we add authentication and access control there?
>>>>
>>>> Andrew.
>>>>
>>>> On 9/8/23 17:57, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> I have a couple questions about needed setup of the NICs for the ethdev
>>>> test suite.
>>>>
>>>> Our MCX5s and XL710s are failing the checkup tests. The pass rate
>>>> appears to be much worse on the XL710s (40 of 73 tests failed, 3 passed
>>>> unexpectedly).
>>>>
>>>> For the XL710s, I've updated the driver and NVM versions to match the
>>>> minimum supported versions in the compatibility matrix found on the DPDK
>>>> documentation. This did not change the failure rate much.
>>>> For the MCX5s, I've installed the latest LTS version of the OFED
>>>> bifurcated driver on the DUT. Is it needed on the tester?
>>>>
>>>> Are there any other manual setup steps for these devices that I might
>>>> be missing?
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu>
>>>> wrote:
>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> Yes, I copied the X710 configs to set up XL710 configs. I changed the
>>>>> environment variable names from the X710 suffix to XL710 suffix in the
>>>>> script, and forgot to change them in the corresponding environment file.
>>>>> That fixed the issue.
>>>>>
>>>>> I got the checkup tests working on the XL710 now. Most of them are
>>>>> failing, which leads me to believe this is an issue with our testbed. Based
>>>>> on the DPDK documentation for i40e, the firmware and driver versions are
>>>>> much older than what DPDK 22.11 LTS and main prefer, so I'll try updating
>>>>> those.
>>>>>
>>>>> For now I'm working on getting the XL710 checkup tests passing, and
>>>>> will pick up getting the E810 configured properly next. I'll let you know
>>>>> if I run into any more issues in relation to the test engine.
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>> On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>> Hi Adam,
>>>>>>
>>>>>> On 9/5/23 18:01, Adam Hassick wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> The compilation warning issue is now resolved. Again, thank you guys
>>>>>> for fixing this for us. I can run the tests on the Mellanox CX5s again,
>>>>>> however I'm running into a couple new issues with running the prologues on
>>>>>> the Intel cards.
>>>>>>
>>>>>> When running testing on the Intel XL710s, I see this error appear in
>>>>>> the log:
>>>>>>
>>>>>> ERROR  prologue  Environment LIB  14:16:13.650
>>>>>>> Too few networks in available configuration (0) in comparison with
>>>>>>> required (1)
>>>>>>>
>>>>>>
>>>>>> This seems like a trivial configuration error, perhaps this is
>>>>>> something I need to set up in ts-rigs. I briefly searched through the
>>>>>> examples there and didn't see any mention of how to set up a network.
>>>>>> I will attach this log just in case you need more information.
>>>>>>
>>>>>>
>>>>>> Unfortunately logs are insufficient to understand it. I've pushed new
>>>>>> tag to TE v1.19.0 which add log message with TE_* environment variables.
>>>>>> Most likely something is wrong with variables which are used as
>>>>>> conditions when available networks are defined in
>>>>>> ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>> TE_PCI_INSTANCE_IUT_TST1
>>>>>> TE_PCI_INSTANCE_IUT_TST1a
>>>>>> TE_PCI_INSTANCE_TST1a_IUT
>>>>>> TE_PCI_INSTANCE_TST1_IUT
>>>>>> My guess it that you change naming a bit, but script like
>>>>>> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>>>>>>
>>>>>> There is a different error when running on the Intel E810s. It
>>>>>> appears to me like it starts DPDK, does some configuration inside DPDK and
>>>>>> on the device, and then fails to bring the device back up. Since this error
>>>>>> seems very non-trivial, I will also attach this log.
>>>>>>
>>>>>>
>>>>>> This one is a bit simpler. Few lines after the first ERROR in log I
>>>>>> see the following:
>>>>>> WARN  RCF  DPDK  13:06:00.144
>>>>>> ice_program_hw_rx_queue(): currently package doesn't support RXDID
>>>>>> (22)
>>>>>> ice_rx_queue_start(): fail to program RX queue 0
>>>>>> ice_dev_start(): fail to start Rx queue 0
>>>>>> Device with port_id=0 already stopped
>>>>>>
>>>>>> It is stdout/stderr from test agent which runs DPDK. Same logs in
>>>>>> plain format are available in ta.DPDK file.
>>>>>> I'm not an expert here, but I vaguely remember that E810 requires
>>>>>> correct firmware and DDP to be loaded.
>>>>>> There is some information in dpdk/doc/guides/nics/ice.rst.
>>>>>>
>>>>>> You can try to add --dev-args=safe-mode-support=1 command-line option
>>>>>> described there.
>>>>>>
>>>>>> Hope it helps,
>>>>>> Andrew.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Adam
>>>>>>
>>>>>> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>> Hi Adam,
>>>>>>>
>>>>>>> On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>>
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> I have one additional question as well: Does the test engine support
>>>>>>> running tests on two ARMv8 test agents?
>>>>>>>
>>>>>>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>>>>>>
>>>>>>>
>>>>>>> Great. Let me know when that's fixed.
>>>>>>>
>>>>>>>
>>>>>>> Done. We also fixed a number of warnings in TE.
>>>>>>> Also we fixed root test package name to be consistent with the
>>>>>>> repository name.
>>>>>>>
>>>>>>> Support for old LTS branches was dropped some time ago, but in the
>>>>>>>> future it is definitely possible to keep it for new LTS branches. I think
>>>>>>>> 22.11 is supported, but I'm not sure about older LTS releases.
>>>>>>>>
>>>>>>>
>>>>>>> Good to know.
>>>>>>>
>>>>>>>
>>>>>>>> 2. You can add command-line option --sanity to run tests marked
>>>>>>>> with TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and
>>>>>>>> grep TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked).
>>>>>>>> Yes, there is a space for terminology improvement here. We'll do it.
>>>>>>>>
>>>>>>>
>>>>>>> Done. Now it is called --checkup.
>>>>>>>
>>>>>>>
>>>>>>>> Also it takes a lot of time because of failures and tests which
>>>>>>>> wait for some timeout.
>>>>>>>>
>>>>>>>
>>>>>>> That makes sense to me. We'll use the time to complete tests on
>>>>>>> virtio or the Intel devices as a reference for how long the tests really
>>>>>>> take to complete.
>>>>>>> We will explore the possibility of periodically running the sanity
>>>>>>> tests for patches.
>>>>>>>
>>>>>>>
>>>>>>> I'll double-check and let you know how long entire TS runs on Intel
>>>>>>> X710, E810, Mellanox CX5 and virtio net. Just to ensure that time observed
>>>>>>> in your case looks the same.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> The test harness can provide coverage reports based on gcov, but
>>>>>>>> I'm not sure what you mean by a "dial" to control test coverage. Provided
>>>>>>>> reports are rather for human to analyze.
>>>>>>>>
>>>>>>>
>>>>>>> The general idea is to have some kind of parameter on the test
>>>>>>> suite, which could be an integer ranging from zero to ten, that controls
>>>>>>> how many tests are run based on how important the test is.
>>>>>>>
>>>>>>> Similar to how some command line interfaces provide a verbosity
>>>>>>> level parameter (some number of "-v" arguments) to control the importance
>>>>>>> of the information in the log.
>>>>>>> The verbosity level zero only prints very important log messages,
>>>>>>> while ten prints everything.
>>>>>>>
>>>>>>> In much the same manner as above, this "dial" parameter controls
>>>>>>> what tests are run and with what parameters based on how important those
>>>>>>> tests and test parameter combinations are.
>>>>>>> Coverage Level zero tells the suite to run a very basic set of
>>>>>>> important tests, with minimal parameterization. This mode would take only
>>>>>>> ~5-10 minutes to run.
>>>>>>> In contrast, Coverage Level ten includes all the edge cases, every
>>>>>>> combination of test parameters, everything the test suite can do, which
>>>>>>> takes the normal several hours to run.
>>>>>>> The values 1 - 9 are between those two extremes, allowing the user
>>>>>>> to get a gradient of test coverage in the results and to limit the running
>>>>>>> time.
>>>>>>>
>>>>>>> Then we could, for example, run the "run.sh" with a level of 2 or 3
>>>>>>> for incoming patches that need quick results, and with a level of 10 for
>>>>>>> the less often run periodic tests performed on main or LTS branches.
>>>>>>>
>>>>>>>
>>>>>>> Understood now. Thanks a lot for the idea. We'll discuss it and come
>>>>>>> back.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>>>>>>>> testing results. Unfortunately it is time consuming to fill in expectations
>>>>>>>> database since it is necessary to analyze testing results and classify if
>>>>>>>> it is a bug or just acceptable behaviour aspect.
>>>>>>>>
>>>>>>>> Bublik allows to compare results of two runs. It is useful for
>>>>>>>> human, but still not good for automation.
>>>>>>>>
>>>>>>>> I have local patch for mlx5 driver which reports Tx ring size
>>>>>>>> maximum. It makes pass rate higher. It is a problem for test harness that
>>>>>>>> mlx5 does not report limits right now.
>>>>>>>>
>>>>>>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>>>>>>>> virtio net is 99% right now and could be done 100% easily (just one thing
>>>>>>>> to fix in expectations).
>>>>>>>>
>>>>>>>> I think logs storage setup is essential for logs analysis. Of
>>>>>>>> course, you can request HTML logs when you run tests (--log-html=html) or
>>>>>>>> generate after run using dpdk-ethdev-ts/scripts/html-log.sh and open
>>>>>>>> index.html in a browser, but logs storage makes it more convenient.
>>>>>>>>
>>>>>>>
>>>>>>> We are interested in setting up Bublik, potentially as an
>>>>>>> externally-facing component, once we have our process of running the test
>>>>>>> suite stabilized.
>>>>>>> Once we are able to run the test suite again, I'll see what the pass
>>>>>>> rate is on our other hardware.
>>>>>>> Good to know that it isn't an issue with our dev testbed causing the
>>>>>>> high fail rate.
>>>>>>>
>>>>>>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>>>>>>> development testbed. Although they are slightly different devices, ideally
>>>>>>> the pass rate will be identical or similar. I have yet to set up a VM pair
>>>>>>> for virtio, but we will soon.
>>>>>>>
>>>>>>> Latest version of test-environment has examples of our CGI scripts
>>>>>>>> which we use for log storage (see tools/log_server/README.md).
>>>>>>>>
>>>>>>>> Also all bits for Jenkins setup are available. See
>>>>>>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>>>>>>> ts-rigs-sample.
>>>>>>>>
>>>>>>>
>>>>>>> Jenkins integration, setting up production rig configurations, and
>>>>>>> permanent log storage will be our next steps once I am able to run the
>>>>>>> tests again.
>>>>>>> Unless there is an easy way to have meson not pass "-Werror" into
>>>>>>> GCC. Then I would be able to run the test suite.
>>>>>>>
>>>>>>>
>>>>>>> Hopefully it is resolved now.
>>>>>>>
>>>>>>> I thought a bit more about your usecase for Jenkins. I'm not 100%
>>>>>>> sure that existing pipelines are convenient for your usecase.
>>>>>>> Fill free to ask questions when you are on it.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Adam
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> That fix seems to have resolved the issue, thanks for the quick
>>>>>>>> turnaround time on that patch.
>>>>>>>> Now that we have the RCF timeout issue resolved, there are a few
>>>>>>>> other questions and issues that we have about the tests themselves.
>>>>>>>>
>>>>>>>> 1. The test suite fails to build with a couple warnings.
>>>>>>>>
>>>>>>>> Below is the stderr log from compilation:
>>>>>>>>
>>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>>>>> and no format arguments [-Werror=format-security]
>>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>>> |     ^~
>>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>>> ninja: build stopped: subcommand failed.
>>>>>>>>> ninja: Entering directory `.'
>>>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>>>>>>>>> and no format arguments [-Werror=format-security]
>>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>>> |     ^~
>>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>>>
>>>>>>>>
>>>>>>>> This error wasn't occurring last week, which was the last time I
>>>>>>>> ran the tests.
>>>>>>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester
>>>>>>>> has GCC v11.4.0 installed, if this information is helpful.
>>>>>>>>
>>>>>>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>>>>>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>>>>>>> to lower the test coverage and have the test suite run faster?
>>>>>>>>
>>>>>>>> For some context, we run immediate testing on incoming patches for
>>>>>>>> DPDK main and development branches, as well as periodic test runs on the
>>>>>>>> main, stable, and LTS branches.
>>>>>>>> For us to consider including this test suite as part of our
>>>>>>>> immediate testing on patches, we would have to reduce the test coverage to
>>>>>>>> the most important tests.
>>>>>>>> This is primarily to reduce the testing time to, for example, less
>>>>>>>> than 30 minutes. Testing on patches can't take too long because the lab can
>>>>>>>> receive numerous patches each day, which each require individual testing
>>>>>>>> runs.
>>>>>>>>
>>>>>>>> At what frequency we run these tests, and on what, still needs to
>>>>>>>> be discussed with the DPDK community, but it would be nice to know if the
>>>>>>>> test suite had a "dial" to control the testing coverage.
>>>>>>>>
>>>>>>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>>>>>>> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose these
>>>>>>>> test failures?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Adam
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>> Hi Adam,
>>>>>>>>>
>>>>>>>>> I've pushed the fix in main branch and a new tag v1.18.1. It
>>>>>>>>> should solve the problem with IPv6 address from DNS.
>>>>>>>>>
>>>>>>>>> Andrew.
>>>>>>>>>
>>>>>>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>>>
>>>>>>>>> Hi Adam,
>>>>>>>>>
>>>>>>>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>>>>> force it to use IPv4?
>>>>>>>>>
>>>>>>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in
>>>>>>>>> your case, I guess it is the root cause of the problem.
>>>>>>>>> Of course, it is TE problem since I see really weird code in
>>>>>>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>>>>>
>>>>>>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>>>>>>> ts-factory/test-environment repository. Please, try.
>>>>>>>>>
>>>>>>>>> It is late night fix with minimal testing and no review. I'll pass
>>>>>>>>> it through review process tomorrow and
>>>>>>>>> hopefully it will be released in one-two days.
>>>>>>>>>
>>>>>>>>> Andrew.
>>>>>>>>>
>>>>>>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>> Hi Andrew,
>>>>>>>>>
>>>>>>>>> We have yet to notice a distinct pattern with the failures.
>>>>>>>>> Sometimes, the RCF will start and connect without issue a few times in a
>>>>>>>>> row before failing to connect again. Once the issue begins to occur,
>>>>>>>>> neither rebooting all of the hosts (test engine VM, tester, IUT) or
>>>>>>>>> deleting all of the build directories (suites, agents, inst) and rebooting
>>>>>>>>> the hosts afterward resolves the issue. When it begins working again seems
>>>>>>>>> very arbitrary to us.
>>>>>>>>>
>>>>>>>>> I do usually try to terminate the test engine with Ctrl+C, but
>>>>>>>>> when it hangs while trying to start RCF, that does not work.
>>>>>>>>>
>>>>>>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>>>>> force it to use IPv4?
>>>>>>>>>
>>>>>>>>>  - Adam
>>>>>>>>>
>>>>>>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>
>>>>>>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>>
>>>>>>>>>> Done. It works fine for me without any issues.
>>>>>>>>>>
>>>>>>>>>> Have you noticed any pattern when it works or does not work?
>>>>>>>>>> May be it is a problem of not clean state after termination?
>>>>>>>>>> Does it work fine the first time after DUTs reboot?
>>>>>>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>>>>>>> terminal where you execute run.sh command.
>>>>>>>>>>  In this case it should shutdown gracefully and close all test
>>>>>>>>>> agents and engine applications.
>>>>>>>>>>
>>>>>>>>>> (I'm trying to understand why you've seen many test agent
>>>>>>>>>> processes. It should not happen.)
>>>>>>>>>>
>>>>>>>>>> Andrew.
>>>>>>>>>>
>>>>>>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>>>>
>>>>>>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Andrew,
>>>>>>>>>>
>>>>>>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>>>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>>>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>>>>>>> This could be the source of the problem. I encountered a
>>>>>>>>>> dependency issue trying to run the Test Engine on 22.04 LTS, so I
>>>>>>>>>> downgraded the system. Since the tester is also the host having connection
>>>>>>>>>> issues, I will try downgrading that system to 20.04, and see if that
>>>>>>>>>> changes anything.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>>>>>>>>> Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every
>>>>>>>>>> night.
>>>>>>>>>> Right now Debian 11 is used for test engine in nightly
>>>>>>>>>> regressions.
>>>>>>>>>>
>>>>>>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>>
>>>>>>>>>> I did try passing in the "--vg-rcf" argument to the run.sh script
>>>>>>>>>> of the test suite after installing valgrind, but there was no additional
>>>>>>>>>> output that I saw.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>>>>>>> (direction where you run test engine).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I will try pulling in the changes you've pushed up, and will see
>>>>>>>>>> if that fixes anything.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Adam
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Adam,
>>>>>>>>>>>
>>>>>>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>>>>>
>>>>>>>>>>> I'd like to try to repeat the problem locally. Which Linux
>>>>>>>>>>> distro is running on test engine and agents?
>>>>>>>>>>>
>>>>>>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we
>>>>>>>>>>> have
>>>>>>>>>>> patch in review to fix it, however, the behaviour is different in
>>>>>>>>>>> this case, so it is unlike the same problem.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I've just published a new tag which fixes known test engine side
>>>>>>>>>>> problems on Debian 12 and Fedora 38.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>>>>>>> run with option --vg-rcf to check if something weird is
>>>>>>>>>>> happening.
>>>>>>>>>>>
>>>>>>>>>>> What I don't understand right now is why I see just one failed
>>>>>>>>>>> attempt
>>>>>>>>>>> to connect in your log.txt and then Logger shutdown after 9
>>>>>>>>>>> minutes.
>>>>>>>>>>>
>>>>>>>>>>> Andrew.
>>>>>>>>>>>
>>>>>>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>>>>
>>>>>>>>>>>  > Is there any firewall in the network or on test hosts which
>>>>>>>>>>> could block incoming TCP connection to the port 23571
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>>>> where you run test engine?
>>>>>>>>>>>
>>>>>>>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>>>>>>>> connection does work sometimes.
>>>>>>>>>>>
>>>>>>>>>>>  > If behaviour the same on the next try and you see that test
>>>>>>>>>>> agent is kept running, could you check using
>>>>>>>>>>>  >
>>>>>>>>>>>  > # netstat -tnlp
>>>>>>>>>>>  >
>>>>>>>>>>>  > that Test Agent is listening on the port and try to establish
>>>>>>>>>>> TCP connection from test agent using
>>>>>>>>>>>  >
>>>>>>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>  >
>>>>>>>>>>>  > and check if TCP connection could be established.
>>>>>>>>>>>
>>>>>>>>>>> I was able to replicate the same behavior again, where it hangs
>>>>>>>>>>> while RCF is trying to start.
>>>>>>>>>>> Running this command, I see this in the output:
>>>>>>>>>>>
>>>>>>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>>>>>>>      18599/ta
>>>>>>>>>>>
>>>>>>>>>>> So it seems like it is listening on the correct port.
>>>>>>>>>>> Additionally, I was able to connect to the Tester machine from
>>>>>>>>>>> our Test Engine host using telnet. It printed the PID of the process once
>>>>>>>>>>> the connection was opened.
>>>>>>>>>>>
>>>>>>>>>>> I tried running the "ta" application manually on the command
>>>>>>>>>>> line, and it didn't print anything at all.
>>>>>>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>>>>>>> andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>     Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>      > On the tester host (which appears to be the Peer agent),
>>>>>>>>>>> there
>>>>>>>>>>>     are four processes that I see running, which look like the
>>>>>>>>>>> test
>>>>>>>>>>>     agent processes.
>>>>>>>>>>>
>>>>>>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>>>>>>
>>>>>>>>>>>     Is there any firewall in the network or on test hosts which
>>>>>>>>>>> could
>>>>>>>>>>>     block incoming TCP connection to the port 23571
>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>>>>     where you run test engine?
>>>>>>>>>>>
>>>>>>>>>>>     If behaviour the same on the next try and you see that test
>>>>>>>>>>> agent is
>>>>>>>>>>>     kept running, could you check using
>>>>>>>>>>>
>>>>>>>>>>>     # netstat -tnlp
>>>>>>>>>>>
>>>>>>>>>>>     that Test Agent is listening on the port and try to
>>>>>>>>>>> establish TCP
>>>>>>>>>>>     connection from test agent using
>>>>>>>>>>>
>>>>>>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>
>>>>>>>>>>>     and check if TCP connection could be established.
>>>>>>>>>>>
>>>>>>>>>>>     Another idea is to login Tester under root as testing does,
>>>>>>>>>>> get
>>>>>>>>>>>     start TA command from the log and try it by hands without -n
>>>>>>>>>>> and
>>>>>>>>>>>     remove extra escaping.
>>>>>>>>>>>
>>>>>>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>
>>>>>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>
>>>>>>>>>>>     Hopefully in this case test agent directory remains in the
>>>>>>>>>>> /tmp and
>>>>>>>>>>>     you don't need to copy it as testing does.
>>>>>>>>>>>     May be output could shed some light on what's going on.
>>>>>>>>>>>
>>>>>>>>>>>     Andrew.
>>>>>>>>>>>
>>>>>>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>>>
>>>>>>>>>>>     Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>     This is the output that I see in the terminal when this
>>>>>>>>>>> failure
>>>>>>>>>>>     occurs, after the test agent binaries build and the test
>>>>>>>>>>> engine
>>>>>>>>>>>     starts:
>>>>>>>>>>>
>>>>>>>>>>>     Platform default build - pass
>>>>>>>>>>>     Simple RCF consistency check succeeded
>>>>>>>>>>>     --->>> Starting Logger...done
>>>>>>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>>>>>>>> timed
>>>>>>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>
>>>>>>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>>>>>>
>>>>>>>>>>>     On the tester host (which appears to be the Peer agent),
>>>>>>>>>>> there are
>>>>>>>>>>>     four processes that I see running, which look like the test
>>>>>>>>>>> agent
>>>>>>>>>>>     processes.
>>>>>>>>>>>
>>>>>>>>>>>     ta.Peer is an empty file. I've attached the log.txt from
>>>>>>>>>>> this run.
>>>>>>>>>>>
>>>>>>>>>>>      - Adam
>>>>>>>>>>>
>>>>>>>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>         Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>>>>>> double-checked
>>>>>>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>>>>>         Description in in
>>>>>>>>>>> doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>         says that copy_timeout is in seconds and implementation
>>>>>>>>>>> in
>>>>>>>>>>>         lib/rcfunix/rcfunix.c passes the value to select()
>>>>>>>>>>> tv_sec.
>>>>>>>>>>>         Theoretically select() could be interrupted by signal,
>>>>>>>>>>> but I
>>>>>>>>>>>         think it is unlikely here.
>>>>>>>>>>>
>>>>>>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>> help. Or
>>>>>>>>>>>         does it happen when tests are in progress, e.g. in the
>>>>>>>>>>> middle
>>>>>>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and
>>>>>>>>>>> most
>>>>>>>>>>>         likely either host with test agent dies or test agent
>>>>>>>>>>> itself
>>>>>>>>>>>         crashes. It would be easier for me if classify it if you
>>>>>>>>>>> share
>>>>>>>>>>>         text log (log.txt, full or just corresponding fragment
>>>>>>>>>>> with
>>>>>>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>>>>>>         depending on which agent has problems could shed some
>>>>>>>>>>> light.
>>>>>>>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>>>>>>>
>>>>>>>>>>>         Andrew.
>>>>>>>>>>>
>>>>>>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>>>
>>>>>>>>>>>         Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>         I've set up a test rig repository here, and have created
>>>>>>>>>>>         configurations for our development testbed based off of
>>>>>>>>>>> the
>>>>>>>>>>>         examples.
>>>>>>>>>>>         We've been able to get the test suite to run manually on
>>>>>>>>>>>         Mellanox CX5 devices once.
>>>>>>>>>>>         However, we are running into an issue where, when RCF
>>>>>>>>>>> starts,
>>>>>>>>>>>         the RCF connection times out very frequently. We aren't
>>>>>>>>>>> sure
>>>>>>>>>>>         why this is the case.
>>>>>>>>>>>         It works sometimes, but most of the time when we try to
>>>>>>>>>>> run
>>>>>>>>>>>         the test engine, it encounters this issue.
>>>>>>>>>>>         I've tried changing the RCF port by setting
>>>>>>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the
>>>>>>>>>>> testbed
>>>>>>>>>>>         machines. Neither seems to fix the issue.
>>>>>>>>>>>
>>>>>>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>         before I try to run the test suite.
>>>>>>>>>>>         I assume the unit for this variable is seconds?
>>>>>>>>>>>
>>>>>>>>>>>         Thanks,
>>>>>>>>>>>         Adam
>>>>>>>>>>>
>>>>>>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>             Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>             Thanks, I've cloned the example repository and will
>>>>>>>>>>> start
>>>>>>>>>>>             setting up a configuration for our development
>>>>>>>>>>> testbed
>>>>>>>>>>>             today. I'll let you know if I run into any
>>>>>>>>>>> difficulties
>>>>>>>>>>>             or have any questions.
>>>>>>>>>>>
>>>>>>>>>>>              - Adam
>>>>>>>>>>>
>>>>>>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                 I've published
>>>>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>                 Hopefully it will help to define your test rigs
>>>>>>>>>>> and
>>>>>>>>>>>                 successfully run some tests manually. Feel free
>>>>>>>>>>> to
>>>>>>>>>>>                 ask any questions and I'll answer here and try to
>>>>>>>>>>>                 update documentation.
>>>>>>>>>>>
>>>>>>>>>>>                 Meanwhile I'll prepare missing bits for steps
>>>>>>>>>>> (2) and
>>>>>>>>>>>                 (3).
>>>>>>>>>>>                 Hopefully everything is in place for step (4),
>>>>>>>>>>> but we
>>>>>>>>>>>                 need to make steps (2) and (3) first.
>>>>>>>>>>>
>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>>>>
>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                 > I've conferred with the rest of the team, and
>>>>>>>>>>> we
>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>> mainly
>>>>>>>>>>>                 option B.
>>>>>>>>>>>
>>>>>>>>>>>                 OK, I'll provide the sample on Monday for you.
>>>>>>>>>>> It is
>>>>>>>>>>>                 almost ready right now, but I need to
>>>>>>>>>>> double-check
>>>>>>>>>>>                 it before publishing.
>>>>>>>>>>>
>>>>>>>>>>>                 Regards,
>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>>>
>>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>>>>                 conversation. Others in the community might find
>>>>>>>>>>>                 this conversation valuable.
>>>>>>>>>>>
>>>>>>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>>>>>>                 Jenkins integration will be very useful for us,
>>>>>>>>>>> as
>>>>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>> mainly
>>>>>>>>>>>                 option B.
>>>>>>>>>>>                 If you would like to know anything about our
>>>>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>>>>>>                 questions you have.
>>>>>>>>>>>
>>>>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>>>>                 hardware and CPU architecture. I figured this
>>>>>>>>>>> might
>>>>>>>>>>>                 be an important detail.
>>>>>>>>>>>
>>>>>>>>>>>                 Thanks,
>>>>>>>>>>>                 Adam
>>>>>>>>>>>
>>>>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>                     Greatings Adam,
>>>>>>>>>>>
>>>>>>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>>>>>>                     it up.
>>>>>>>>>>>
>>>>>>>>>>>                     As I understand the final goal is to run it
>>>>>>>>>>> on
>>>>>>>>>>>                     regular basis. So, we need to make it
>>>>>>>>>>> properly
>>>>>>>>>>>                     from the very beginning.
>>>>>>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>>>>>>
>>>>>>>>>>>                     1. Create site-specific repository (we call
>>>>>>>>>>> it
>>>>>>>>>>>                     ts-rigs) which contains information about
>>>>>>>>>>> test
>>>>>>>>>>>                     rigs and other site-specific information like
>>>>>>>>>>>                     where to send mails, where to store logs etc.
>>>>>>>>>>>                     It is required for manual execution as well,
>>>>>>>>>>>                     since test rigs description is essential.
>>>>>>>>>>> I'll
>>>>>>>>>>>                     return to the topic below.
>>>>>>>>>>>
>>>>>>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>>>>>>                     server with few CGI scripts which help a lot
>>>>>>>>>>> to
>>>>>>>>>>>                     save disk space.
>>>>>>>>>>>
>>>>>>>>>>>                     3. Setup Bublik web application which
>>>>>>>>>>> provides
>>>>>>>>>>>                     web interface to view testing results. Same
>>>>>>>>>>> as
>>>>>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>>>>> <https://ts-factory.io/bublik>
>>>>>>>>>>>
>>>>>>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>>>>>>                     save logs in log storage (2) and import it to
>>>>>>>>>>>                     bublik (3).
>>>>>>>>>>>
>>>>>>>>>>>                     Last few month we spent on our homework to
>>>>>>>>>>> make
>>>>>>>>>>>                     it simpler to bring up automated execution
>>>>>>>>>>>                     using Jenkins -
>>>>>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>>>>                     available tomorrow.
>>>>>>>>>>>
>>>>>>>>>>>                     Let's return to the step (1).
>>>>>>>>>>>
>>>>>>>>>>>                     Unfortunately there is no publicly available
>>>>>>>>>>>                     example of the ts-rigs repository since
>>>>>>>>>>>                     sensitive site-specific information is
>>>>>>>>>>> located
>>>>>>>>>>>                     there. But I'm ready to help you to create it
>>>>>>>>>>>                     for UNH. I see two options here:
>>>>>>>>>>>
>>>>>>>>>>>                     (A) I'll ask questions and based on your
>>>>>>>>>>>                     answers will create the first draft with my
>>>>>>>>>>>                     comments.
>>>>>>>>>>>
>>>>>>>>>>>                     (B) I'll make a template/example ts-rigs
>>>>>>>>>>> repo,
>>>>>>>>>>>                     publish it and you'll create UNH ts-rigs
>>>>>>>>>>> based
>>>>>>>>>>>                     on it.
>>>>>>>>>>>
>>>>>>>>>>>                     Of course, I'll help to debug and finally
>>>>>>>>>>> bring
>>>>>>>>>>>                     it up in any case.
>>>>>>>>>>>
>>>>>>>>>>>                     (A) is a bit simpler for me and you, but (B)
>>>>>>>>>>> is
>>>>>>>>>>>                     a bit more generic and will help other
>>>>>>>>>>>                     potential users to bring it up.
>>>>>>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>>>>>>                     What do you think?
>>>>>>>>>>>
>>>>>>>>>>>                     Thanks,
>>>>>>>>>>>                     Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>>>>
>>>>>>>>>>>                     Greetings Adam,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>>>>>                     would be happy to help
>>>>>>>>>>>
>>>>>>>>>>>                     Thanks,
>>>>>>>>>>>                     Konstantin
>>>>>>>>>>>
>>>>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>                     <ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>                     
>>>>>>>>>>>                     Greetings Konstantin,
>>>>>>>>>>>
>>>>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>>>>>
>>>>>>>>>>>                     I have some questions about how to set the
>>>>>>>>>>>                     test suite arguments.
>>>>>>>>>>>
>>>>>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>>>>                     fails to run.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>
>>>>>>>>>>>                     The documentation mentions that there are
>>>>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>>>>>                     to run many of the tests.
>>>>>>>>>>>
>>>>>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>>>>>                     are instructions on how to create new
>>>>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>>>>                     some other way.
>>>>>>>>>>>
>>>>>>>>>>>                     How should these values be set?
>>>>>>>>>>>
>>>>>>>>>>>                     I'm also not sure what environment
>>>>>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>>>>>                     which are optional.
>>>>>>>>>>>
>>>>>>>>>>>                     Regards,
>>>>>>>>>>>                     Adam
>>>>>>>>>>>
>>>>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>>>>                     Senior Developer
>>>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                 --                 *Adam Hassick*
>>>>>>>>>>>                 Senior Developer
>>>>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>> <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>             --             *Adam Hassick*
>>>>>>>>>>>             Senior Developer
>>>>>>>>>>>             UNH InterOperability Lab
>>>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>             +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         --         *Adam Hassick*
>>>>>>>>>>>         Senior Developer
>>>>>>>>>>>         UNH InterOperability Lab
>>>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>         +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>     --     *Adam Hassick*
>>>>>>>>>>>     Senior Developer
>>>>>>>>>>>     UNH InterOperability Lab
>>>>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>     +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>> Senior Developer
>>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Adam Hassick*
>>>>>>>>>> Senior Developer
>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Adam Hassick*
>>>>>>>>> Senior Developer
>>>>>>>>> UNH InterOperability Lab
>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Adam Hassick*
>>>>>>>> Senior Developer
>>>>>>>> UNH InterOperability Lab
>>>>>>>> ahassick@iol.unh.edu
>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>> +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Adam Hassick*
>>>>>> Senior Developer
>>>>>> UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu
>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

[-- Attachment #2: Type: text/html, Size: 227159 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 232471 bytes --]

Hi Adam,

On 10/25/23 23:27, Adam Hassick wrote:
> Hi Andrew,
>
> Sorry about the two week radio silence, we're still trying to sort out 
> logistics for deployment on our end.
>
> > I've created setup on ts-factory.io <http://ts-factory.io> which 
> allows to publish logs. Let me know if you'd like to try it and I'll 
> provide credentials, script and short instruction.
>
> We're interested in publishing some test logs to the ts-factory Bublik 
> instance in the meantime.

Please, send me SSH public key which you'd like to use to upload logs.
I'll provide helper script and instructions to do it.

Andrew.
>
> Thanks,
> Adam
>
> On Mon, Oct 23, 2023 at 7:11 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     > Now that our test results are in line with yours, we can begin
>     looking into setting up the production environment.
>
>     Please, let me know if you need any help with it or waiting for an
>     input from me.
>
>     Regards,
>     Andrew.
>
>     On 10/10/23 17:09, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     Thank you for taking a look at our log. Netplan was attempting to
>>     run DHCP on our test links, and additionally I discovered that
>>     the NIC firmware was transmitting LLDP packets, causing tests to
>>     fail in the same way. Now that these problems have been fixed,
>>     our pass rate on the XL710 is approximately 91%. Now that our
>>     test results are in line with yours, we can begin looking into
>>     setting up the production environment.
>>
>>     First, is it possible to run the test agent on ARM hosts? Our ARM
>>     testbeds have the best topology for running this test suite, with
>>     separate tester and DUT servers.
>>
>>     We are testing this test suite on two x86 development servers
>>     using the test suite's recommended server topology. In contrast,
>>     our existing x86 production testbeds which run DTS have a single
>>     server topology. This single server has both the tester NIC and
>>     the device under test NIC installed, with NUMA node separation
>>     between TRex and DPDK. We're going to test running the two test
>>     agent processes on the single-server testbeds if we cannot run
>>     this on ARM. Is there any reason you can think of that would
>>     prevent this setup from working?
>>
>>     Once we figure out where this can live in production, then we
>>     will begin setting up log storage, Jenkins integration, and Bublik.
>>
>>     Thanks,
>>     Adam
>>
>>     On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         Hi Adam,
>>
>>         > Do these default to vfio-pci?
>>
>>         Yes, vfio-pci is the default.
>>         However, it does not work in the case of Mellanox which uses
>>         bifurcated driver. It should mlx5_core for Mellanox NICs.
>>
>>         > Here is the text log from a run on our Intel XL710 NICs,
>>         with the expected result profile set to the X710.
>>
>>         It is hard to analyze all tests using text logs, but I
>>         definitely see one common problem. Tests receive unexpected
>>         packets and fail because of it.
>>         Tests are written very strict from this point of view and it
>>         brought fruits in the past when HW had bugs.
>>         Are DUT and tester connected back-to-back on tested
>>         interfaces or via switch?
>>         If via switch, is it possible to isolate it from everything else?
>>         If back-to-back, it could be some embedded SW/FW which
>>         regenerates these packet.
>>         I definitely see unexpected DHCP packets.
>>
>>         > We haven't set up the Jenkins integration yet, however if
>>         this is required to import the logs then we will prioritize that.
>>
>>         Unfortunately manual runs do not generate all artifacts
>>         required to import logs. However, we have almost solved it
>>         right now. Hopefully we'll finalize it in a day or two. I'll
>>         let you know when these changes are available.
>>
>>         Regards,
>>         Andrew.
>>
>>         On 10/4/23 16:48, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER
>>>         set anywhere in the default configurations for the Intel
>>>         X710. Do these default to vfio-pci?
>>>         We have IOMMU enabled on our development testbed, and should
>>>         be able to bind vfio-pci.
>>>         Here is the text log from a run on our Intel XL710 NICs,
>>>         with the expected result profile set to the X710. We haven't
>>>         set up the Jenkins integration yet, however if this is
>>>         required to import the logs then we will prioritize that.
>>>         log.txt.tar.gz
>>>         <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             On 9/18/23 17:44, Adam Hassick wrote:
>>>>             Hi Andrew and Konstantin,
>>>>
>>>>             Thank you for adding the tester-dial feature, this
>>>>             opens up the possibility for us to do CI integrated
>>>>             testing in the future.
>>>>
>>>>             Our Mellanox pass rate is similar to yours (about ~2400
>>>>             passing, ~4400 failing), however our Intel pass rates
>>>>             are far worse.
>>>>             I will try running tests on the XL710 with the trc-tags
>>>>             argument set and see if it improves the pass rate.
>>>>             Another thing I noticed in the results you uploaded is
>>>>             that the results are tagged with vfio-pci and not i40e.
>>>>             Though in the environment dump, the driver on the test
>>>>             machine and the DUT are set to use the i40e driver. Is
>>>>             this important at all?
>>>
>>>             I think it is a misunderstanding here. There are two
>>>             kinds of driver in configuration: net driver and
>>>             so-called DPDK driver.
>>>             Net driver is a Linux kernel network device driver used
>>>             on Tester side.
>>>             DPDK driver is a Linux kernel driver to bind device to
>>>             to use it with DPDK. So, it is NOT a driver inside DPDK
>>>             (drivers/net/*).
>>>             In the case of bifurcated driver (like mlx5_core) it is
>>>             the same in both cases.
>>>             In non-bifurcated case DPDK driver is some UIO
>>>             driver(vfio-pci, uio-pci-generic or igb_uio).
>>>             Some expectations depend on used UIO. For example,
>>>             uio-pci-generic do not support many interrupts (used by
>>>             usecases/rx_intr test cases).
>>>             That's why we care corresponding TRC tag.
>>>
>>>             TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in
>>>             710's Intel case. Or uio-pci-generic if IOMMU is turned
>>>             off on corresponding machines and Linux distro does not
>>>             support VFIO no IOMMU mode.
>>>
>>>             Andrew.
>>>
>>>>             There isn't anything preventing us from pushing our
>>>>             results up to the existing Bublik instance running at
>>>>             ts-factory.io <http://ts-factory.io> that I can think
>>>>             of at the moment.
>>>>             We will have to work out how to submit our results to
>>>>             your Bublik instance in a controlled and secure manner
>>>>             in that case.
>>>>             As far as I know we won't need access controls for the
>>>>             results themselves. I'll discuss this with Patrick and
>>>>             will let you know once we confirm that it's fine.
>>>>
>>>>             Thanks,
>>>>             Adam
>>>>
>>>>             On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>                 On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>>>
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 should we always auto-assign the tags or you don’t
>>>>>                 do it since it slows down (by some seconds) the TE
>>>>>                 startup?
>>>>>
>>>>
>>>>                 Tags are auto-assigned, but I guess it differs in
>>>>                 Adam's case since NIC is a bit different. Below
>>>>                 test will help to understand if it is the root
>>>>                 cause of very different expectations. If pass rate
>>>>                 will be close to mine, I'll simply update TRC
>>>>                 database to share expectations for mine NIC and NIC
>>>>                 used by Adam.
>>>>
>>>>>                 Hi Adam,
>>>>>
>>>>>                 I think I second the question from Andrew - happy
>>>>>                 to help you with the triage so that we get to the
>>>>>                 same baseline. Do you have a good way for us to
>>>>>                 share the logs? I.e. say upload to ts-factory if
>>>>>                 we add strict permissions system so it’s not
>>>>>                 publishing or any other way.
>>>>>
>>>>>                 Thanks,
>>>>>                 Konstantin
>>>>>
>>>>>
>>>>>                 On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>>>
>>>>>                     Hi Adam,
>>>>>
>>>>>                     I've uploaded fresh testing results to
>>>>>                     ts-factory.io <http://ts-factory.io> [1] to be
>>>>>                     on the same page.
>>>>>
>>>>>                     I think I know why your and mine results on
>>>>>                     Intel 710 series NICs differ so much. Testing
>>>>>                     results expectations database
>>>>>                     (dpdk-ethdev-ts/trc/*) is filled in in terms
>>>>>                     of TRC tags.  I.e. expectations depends on TRC
>>>>>                     tags discovered by helper scripts when testing
>>>>>                     is started. These tags identify various
>>>>>                     aspects of what is tested. Ideally
>>>>>                     expectations should be written in terms of
>>>>>                     root cause of the expected behaviour. If it is
>>>>>                     a driver expectations, driver tag should be
>>>>>                     used. If it is HW limitation, tags with PCI
>>>>>                     IDs should be used. However, it is not always
>>>>>                     easy to classify it correctly if you're not
>>>>>                     involved in driver development. So, in order
>>>>>                     case expectations for 710's Intel are filled
>>>>>                     in in terms of PCI IDs. I guess PCI ID differ
>>>>>                     in your case and that's why expectations
>>>>>                     filled in for my NIC do not apply to your runs.
>>>>>
>>>>>                     Just try to add the following option when you
>>>>>                     run on your 710's Intel in order to mimic mine
>>>>>                     and see if it helps to achieve better pass rate.
>>>>>                     --trc-tag=pci-8086-1572
>>>>>
>>>>>                     BTW, fresh TE tag v1.21.0 has improved
>>>>>                     algorithm to choose tests for --tester-dial
>>>>>                     option. It should have better coverage now.
>>>>>
>>>>>                     Andrew.
>>>>>
>>>>>                     [1]
>>>>>                     https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>>>                     <https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1>
>>>>>
>>>>>                     On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     I've pushed new TE tag v1.20.0 which
>>>>>>                     supported a new command-line option
>>>>>>                     --tester-dial=NUM where NUM is from 0 to 100.
>>>>>>                     it allows to choose percentage of tests to
>>>>>>                     run. If you want stable set, you should pass
>>>>>>                     --tester-random-seed=0 (or other integer). It
>>>>>>                     is the first sketch and we have plans to
>>>>>>                     improve it, but feedback would be welcome.
>>>>>>
>>>>>>                     > Is it needed on the tester?
>>>>>>
>>>>>>                     It is hard to say if it is strictly required
>>>>>>                     for simple tests. However, it is better to
>>>>>>                     update Tester as well, since performance
>>>>>>                     tests run DPDK on Tester as well.
>>>>>>
>>>>>>                     > Are there any other manual setup steps for
>>>>>>                     these devices that I might be missing?
>>>>>>
>>>>>>                     I don't remember anything else.
>>>>>>
>>>>>>                     I think it is better to get down to details
>>>>>>                     and take a look at logs. I'm ready to help
>>>>>>                     with it and explain what's happening there.
>>>>>>                     May be it will help to understand if it is a
>>>>>>                     problem with setup/configuration.
>>>>>>
>>>>>>                     Text logs are not very convenient. Ideally
>>>>>>                     logs should be imported to bublik, however,
>>>>>>                     manual runs do not provide all required
>>>>>>                     artifacts right now (Jenkins jobs generate
>>>>>>                     all required artifacts).
>>>>>>                     Other option is 'tmp_raw_log' file (should be
>>>>>>                     packed to make it smaller) which could be
>>>>>>                     converted to various log formats.
>>>>>>                     Would it be OK for you if I import your logs
>>>>>>                     to bublik at ts-factory.io
>>>>>>                     <http://ts-factory.io>? Or is it a problem
>>>>>>                     that it is publicly available?
>>>>>>                     Would it help if we add authentication and
>>>>>>                     access control there?
>>>>>>
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 9/8/23 17:57, Adam Hassick wrote:
>>>>>>>                     Hi Andrew,
>>>>>>>
>>>>>>>                     I have a couple questions about needed setup
>>>>>>>                     of the NICs for the ethdev test suite.
>>>>>>>
>>>>>>>                     Our MCX5s and XL710s are failing the checkup
>>>>>>>                     tests. The pass rate appears to be much
>>>>>>>                     worse on the XL710s (40 of 73 tests failed,
>>>>>>>                     3 passed unexpectedly).
>>>>>>>
>>>>>>>                     For the XL710s, I've updated the driver and
>>>>>>>                     NVM versions to match the minimum supported
>>>>>>>                     versions in the compatibility matrix found
>>>>>>>                     on the DPDK documentation. This did not
>>>>>>>                     change the failure rate much.
>>>>>>>                     For the MCX5s, I've installed the latest LTS
>>>>>>>                     version of the OFED bifurcated driver on the
>>>>>>>                     DUT. Is it needed on the tester?
>>>>>>>
>>>>>>>                     Are there any other manual setup steps for
>>>>>>>                     these devices that I might be missing?
>>>>>>>
>>>>>>>                     Thanks,
>>>>>>>                     Adam
>>>>>>>
>>>>>>>                     On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick
>>>>>>>                     <ahassick@iol.unh.edu> wrote:
>>>>>>>
>>>>>>>                         Hi Andrew,
>>>>>>>
>>>>>>>                         Yes, I copied the X710 configs to set up
>>>>>>>                         XL710 configs. I changed the environment
>>>>>>>                         variable names from the X710 suffix to
>>>>>>>                         XL710 suffix in the script, and forgot
>>>>>>>                         to change them in the corresponding
>>>>>>>                         environment file.
>>>>>>>                         That fixed the issue.
>>>>>>>
>>>>>>>                         I got the checkup tests working on the
>>>>>>>                         XL710 now. Most of them are failing,
>>>>>>>                         which leads me to believe this is an
>>>>>>>                         issue with our testbed. Based on the
>>>>>>>                         DPDK documentation for i40e, the
>>>>>>>                         firmware and driver versions are much
>>>>>>>                         older than what DPDK 22.11 LTS and main
>>>>>>>                         prefer, so I'll try updating those.
>>>>>>>
>>>>>>>                         For now I'm working on getting the XL710
>>>>>>>                         checkup tests passing, and will pick up
>>>>>>>                         getting the E810 configured properly
>>>>>>>                         next. I'll let you know if I run into
>>>>>>>                         any more issues in relation to the test
>>>>>>>                         engine.
>>>>>>>
>>>>>>>                         Thanks,
>>>>>>>                         Adam
>>>>>>>
>>>>>>>                         On Wed, Sep 6, 2023 at 7:36 AM Andrew
>>>>>>>                         Rybchenko
>>>>>>>                         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>                             Hi Adam,
>>>>>>>
>>>>>>>                             On 9/5/23 18:01, Adam Hassick wrote:
>>>>>>>>                             Hi Andrew,
>>>>>>>>
>>>>>>>>                             The compilation warning issue is
>>>>>>>>                             now resolved. Again, thank you guys
>>>>>>>>                             for fixing this for us. I can run
>>>>>>>>                             the tests on the Mellanox CX5s
>>>>>>>>                             again, however I'm running into a
>>>>>>>>                             couple new issues with running the
>>>>>>>>                             prologues on the Intel cards.
>>>>>>>>
>>>>>>>>                             When running testing on the Intel
>>>>>>>>                             XL710s, I see this error appear in
>>>>>>>>                             the log:
>>>>>>>>
>>>>>>>>                                 ERROR  prologue  Environment
>>>>>>>>                                 LIB  14:16:13.650
>>>>>>>>                                 Too few networks in available
>>>>>>>>                                 configuration (0) in comparison
>>>>>>>>                                 with required (1)
>>>>>>>>
>>>>>>>>
>>>>>>>>                             This seems like a trivial
>>>>>>>>                             configuration error, perhaps this
>>>>>>>>                             is something I need to set up in
>>>>>>>>                             ts-rigs. I briefly searched through
>>>>>>>>                             the examples there and didn't see
>>>>>>>>                             any mention of how to set up a network.
>>>>>>>>                             I will attach this log just in case
>>>>>>>>                             you need more information.
>>>>>>>
>>>>>>>                             Unfortunately logs are insufficient
>>>>>>>                             to understand it. I've pushed new
>>>>>>>                             tag to TE v1.19.0 which add log
>>>>>>>                             message with TE_* environment variables.
>>>>>>>                             Most likely something is wrong with
>>>>>>>                             variables which are used as
>>>>>>>                             conditions when available networks
>>>>>>>                             are defined in
>>>>>>>                             ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>>>                             TE_PCI_INSTANCE_IUT_TST1
>>>>>>>                             TE_PCI_INSTANCE_IUT_TST1a
>>>>>>>                             TE_PCI_INSTANCE_TST1a_IUT
>>>>>>>                             TE_PCI_INSTANCE_TST1_IUT
>>>>>>>                             My guess it that you change naming a
>>>>>>>                             bit, but script like
>>>>>>>                             ts-rigs-sample/scripts/iut.h1-x710
>>>>>>>                             is not included or not updated.
>>>>>>>
>>>>>>>>                             There is a different error when
>>>>>>>>                             running on the Intel E810s. It
>>>>>>>>                             appears to me like it starts DPDK,
>>>>>>>>                             does some configuration inside DPDK
>>>>>>>>                             and on the device, and then fails
>>>>>>>>                             to bring the device back up. Since
>>>>>>>>                             this error seems very non-trivial,
>>>>>>>>                             I will also attach this log.
>>>>>>>
>>>>>>>                             This one is a bit simpler. Few lines
>>>>>>>                             after the first ERROR in log I see
>>>>>>>                             the following:
>>>>>>>                             WARN  RCF DPDK 13:06:00.144
>>>>>>>                             ice_program_hw_rx_queue(): currently
>>>>>>>                             package doesn't support RXDID (22)
>>>>>>>                             ice_rx_queue_start(): fail to
>>>>>>>                             program RX queue 0
>>>>>>>                             ice_dev_start(): fail to start Rx
>>>>>>>                             queue 0
>>>>>>>                             Device with port_id=0 already stopped
>>>>>>>
>>>>>>>                             It is stdout/stderr from test agent
>>>>>>>                             which runs DPDK. Same logs in plain
>>>>>>>                             format are available in ta.DPDK file.
>>>>>>>                             I'm not an expert here, but I
>>>>>>>                             vaguely remember that E810 requires
>>>>>>>                             correct firmware and DDP to be loaded.
>>>>>>>                             There is some information in
>>>>>>>                             dpdk/doc/guides/nics/ice.rst.
>>>>>>>
>>>>>>>                             You can try to add
>>>>>>>                             --dev-args=safe-mode-support=1
>>>>>>>                             command-line option described there.
>>>>>>>
>>>>>>>                             Hope it helps,
>>>>>>>                             Andrew.
>>>>>>>
>>>>>>>>
>>>>>>>>                             Thanks,
>>>>>>>>                             Adam
>>>>>>>>
>>>>>>>>                             On Fri, Sep 1, 2023 at 3:59 AM
>>>>>>>>                             Andrew Rybchenko
>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>                                 Hi Adam,
>>>>>>>>
>>>>>>>>                                 On 8/31/23 22:38, Adam Hassick
>>>>>>>>                                 wrote:
>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>
>>>>>>>>>                                 I have one additional question
>>>>>>>>>                                 as well: Does the test engine
>>>>>>>>>                                 support running tests on two
>>>>>>>>>                                 ARMv8 test agents?
>>>>>>>>>
>>>>>>>>>                                     1. We'll sort out warnings
>>>>>>>>>                                     this week. Thanks for
>>>>>>>>>                                     heads up.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 Great. Let me know when that's
>>>>>>>>>                                 fixed.
>>>>>>>>
>>>>>>>>                                 Done. We also fixed a number of
>>>>>>>>                                 warnings in TE.
>>>>>>>>                                 Also we fixed root test package
>>>>>>>>                                 name to be consistent with the
>>>>>>>>                                 repository name.
>>>>>>>>
>>>>>>>>>                                     Support for old LTS
>>>>>>>>>                                     branches was dropped some
>>>>>>>>>                                     time ago, but in the
>>>>>>>>>                                     future it is definitely
>>>>>>>>>                                     possible to keep it for
>>>>>>>>>                                     new LTS branches. I think
>>>>>>>>>                                     22.11 is supported, but
>>>>>>>>>                                     I'm not sure about older
>>>>>>>>>                                     LTS releases.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 Good to know.
>>>>>>>>>
>>>>>>>>>                                     2. You can add
>>>>>>>>>                                     command-line option
>>>>>>>>>                                     --sanity to run tests
>>>>>>>>>                                     marked with
>>>>>>>>>                                     TEST_HARNESS_SANITY
>>>>>>>>>                                     requirement (see
>>>>>>>>>                                     dpdk-ethdev-ts/scripts/run.sh
>>>>>>>>>                                     and grep
>>>>>>>>>                                     TEST_HARNESS_SANITY
>>>>>>>>>                                     dpdk-ethdev-ts to see
>>>>>>>>>                                     which tests are marked).
>>>>>>>>>                                     Yes, there is a space for
>>>>>>>>>                                     terminology improvement
>>>>>>>>>                                     here. We'll do it.
>>>>>>>>>
>>>>>>>>
>>>>>>>>                                 Done. Now it is called --checkup.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                     Also it takes a lot of
>>>>>>>>>                                     time because of failures
>>>>>>>>>                                     and tests which wait for
>>>>>>>>>                                     some timeout.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 That makes sense to me. We'll
>>>>>>>>>                                 use the time to complete tests
>>>>>>>>>                                 on virtio or the Intel devices
>>>>>>>>>                                 as a reference for how long
>>>>>>>>>                                 the tests really take to complete.
>>>>>>>>>                                 We will explore the
>>>>>>>>>                                 possibility of periodically
>>>>>>>>>                                 running the sanity tests for
>>>>>>>>>                                 patches.
>>>>>>>>
>>>>>>>>                                 I'll double-check and let you
>>>>>>>>                                 know how long entire TS runs on
>>>>>>>>                                 Intel X710, E810, Mellanox CX5
>>>>>>>>                                 and virtio net. Just to ensure
>>>>>>>>                                 that time observed in your case
>>>>>>>>                                 looks the same.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                     The test harness can
>>>>>>>>>                                     provide coverage reports
>>>>>>>>>                                     based on gcov, but I'm not
>>>>>>>>>                                     sure what you mean by a
>>>>>>>>>                                     "dial" to control test
>>>>>>>>>                                     coverage. Provided reports
>>>>>>>>>                                     are rather for human to
>>>>>>>>>                                     analyze.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 The general idea is to have
>>>>>>>>>                                 some kind of parameter on the
>>>>>>>>>                                 test suite, which could be an
>>>>>>>>>                                 integer ranging from zero to
>>>>>>>>>                                 ten, that controls how many
>>>>>>>>>                                 tests are run based on how
>>>>>>>>>                                 important the test is.
>>>>>>>>>
>>>>>>>>>                                 Similar to how some command
>>>>>>>>>                                 line interfaces provide a
>>>>>>>>>                                 verbosity level parameter
>>>>>>>>>                                 (some number of "-v"
>>>>>>>>>                                 arguments) to control the
>>>>>>>>>                                 importance of the information
>>>>>>>>>                                 in the log.
>>>>>>>>>                                 The verbosity level zero only
>>>>>>>>>                                 prints very important log
>>>>>>>>>                                 messages, while ten prints
>>>>>>>>>                                 everything.
>>>>>>>>>
>>>>>>>>>                                 In much the same manner as
>>>>>>>>>                                 above, this "dial" parameter
>>>>>>>>>                                 controls what tests are run
>>>>>>>>>                                 and with what parameters based
>>>>>>>>>                                 on how important those tests
>>>>>>>>>                                 and test parameter
>>>>>>>>>                                 combinations are.
>>>>>>>>>                                 Coverage Level zero tells the
>>>>>>>>>                                 suite to run a very basic set
>>>>>>>>>                                 of important tests, with
>>>>>>>>>                                 minimal parameterization. This
>>>>>>>>>                                 mode would take only ~5-10
>>>>>>>>>                                 minutes to run.
>>>>>>>>>                                 In contrast, Coverage Level
>>>>>>>>>                                 ten includes all the edge
>>>>>>>>>                                 cases, every combination of
>>>>>>>>>                                 test parameters, everything
>>>>>>>>>                                 the test suite can do, which
>>>>>>>>>                                 takes the normal several hours
>>>>>>>>>                                 to run.
>>>>>>>>>                                 The values 1 - 9 are between
>>>>>>>>>                                 those two extremes, allowing
>>>>>>>>>                                 the user to get a gradient of
>>>>>>>>>                                 test coverage in the results
>>>>>>>>>                                 and to limit the running time.
>>>>>>>>>
>>>>>>>>>                                 Then we could, for example,
>>>>>>>>>                                 run the "run.sh" with a level
>>>>>>>>>                                 of 2 or 3 for incoming patches
>>>>>>>>>                                 that need quick results, and
>>>>>>>>>                                 with a level of 10 for the
>>>>>>>>>                                 less often run periodic tests
>>>>>>>>>                                 performed on main or LTS branches.
>>>>>>>>
>>>>>>>>                                 Understood now. Thanks a lot
>>>>>>>>                                 for the idea. We'll discuss it
>>>>>>>>                                 and come back.
>>>>>>>>
>>>>>>>>>                                     3. Yes, really many tests
>>>>>>>>>                                     on Mellanox CX5 NICs
>>>>>>>>>                                     report unexpected testing
>>>>>>>>>                                     results. Unfortunately it
>>>>>>>>>                                     is time consuming to fill
>>>>>>>>>                                     in expectations database
>>>>>>>>>                                     since it is necessary to
>>>>>>>>>                                     analyze testing results
>>>>>>>>>                                     and classify if it is a
>>>>>>>>>                                     bug or just acceptable
>>>>>>>>>                                     behaviour aspect.
>>>>>>>>>
>>>>>>>>>                                     Bublik allows to compare
>>>>>>>>>                                     results of two runs. It is
>>>>>>>>>                                     useful for human, but
>>>>>>>>>                                     still not good for automation.
>>>>>>>>>
>>>>>>>>>                                     I have local patch for
>>>>>>>>>                                     mlx5 driver which reports
>>>>>>>>>                                     Tx ring size maximum. It
>>>>>>>>>                                     makes pass rate higher. It
>>>>>>>>>                                     is a problem for test
>>>>>>>>>                                     harness that mlx5 does not
>>>>>>>>>                                     report limits right now.
>>>>>>>>>
>>>>>>>>>                                     Pass rate on Intel X710 is
>>>>>>>>>                                     about 92% on my test rig.
>>>>>>>>>                                     Pass rate on virtio net is
>>>>>>>>>                                     99% right now and could be
>>>>>>>>>                                     done 100% easily (just one
>>>>>>>>>                                     thing to fix in expectations).
>>>>>>>>>
>>>>>>>>>                                     I think logs storage setup
>>>>>>>>>                                     is essential for logs
>>>>>>>>>                                     analysis. Of course, you
>>>>>>>>>                                     can request HTML logs when
>>>>>>>>>                                     you run tests
>>>>>>>>>                                     (--log-html=html) or
>>>>>>>>>                                     generate after run using
>>>>>>>>>                                     dpdk-ethdev-ts/scripts/html-log.sh
>>>>>>>>>                                     and open index.html in a
>>>>>>>>>                                     browser, but logs storage
>>>>>>>>>                                     makes it more convenient.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 We are interested in setting
>>>>>>>>>                                 up Bublik, potentially as an
>>>>>>>>>                                 externally-facing component,
>>>>>>>>>                                 once we have our process of
>>>>>>>>>                                 running the test suite stabilized.
>>>>>>>>>                                 Once we are able to run the
>>>>>>>>>                                 test suite again, I'll see
>>>>>>>>>                                 what the pass rate is on our
>>>>>>>>>                                 other hardware.
>>>>>>>>>                                 Good to know that it isn't an
>>>>>>>>>                                 issue with our dev testbed
>>>>>>>>>                                 causing the high fail rate.
>>>>>>>>>
>>>>>>>>>                                 For Intel hardware, we have an
>>>>>>>>>                                 XL710 and an Intel E810-C in
>>>>>>>>>                                 our development testbed.
>>>>>>>>>                                 Although they are slightly
>>>>>>>>>                                 different devices, ideally the
>>>>>>>>>                                 pass rate will be identical or
>>>>>>>>>                                 similar. I have yet to set up
>>>>>>>>>                                 a VM pair for virtio, but we
>>>>>>>>>                                 will soon.
>>>>>>>>>
>>>>>>>>>                                     Latest version of
>>>>>>>>>                                     test-environment has
>>>>>>>>>                                     examples of our CGI
>>>>>>>>>                                     scripts which we use for
>>>>>>>>>                                     log storage (see
>>>>>>>>>                                     tools/log_server/README.md).
>>>>>>>>>
>>>>>>>>>                                     Also all bits for Jenkins
>>>>>>>>>                                     setup are available. See
>>>>>>>>>                                     dpdk-ethdev-ts/jenkins/README.md
>>>>>>>>>                                     and examples of jenkins
>>>>>>>>>                                     files in ts-rigs-sample.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 Jenkins integration, setting
>>>>>>>>>                                 up production rig
>>>>>>>>>                                 configurations, and permanent
>>>>>>>>>                                 log storage will be our next
>>>>>>>>>                                 steps once I am able to run
>>>>>>>>>                                 the tests again.
>>>>>>>>>                                 Unless there is an easy way to
>>>>>>>>>                                 have meson not pass "-Werror"
>>>>>>>>>                                 into GCC. Then I would be able
>>>>>>>>>                                 to run the test suite.
>>>>>>>>
>>>>>>>>                                 Hopefully it is resolved now.
>>>>>>>>
>>>>>>>>                                 I thought a bit more about your
>>>>>>>>                                 usecase for Jenkins. I'm not
>>>>>>>>                                 100% sure that existing
>>>>>>>>                                 pipelines are convenient for
>>>>>>>>                                 your usecase.
>>>>>>>>                                 Fill free to ask questions when
>>>>>>>>                                 you are on it.
>>>>>>>>
>>>>>>>>                                 Thanks,
>>>>>>>>                                 Andrew.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 Thanks,
>>>>>>>>>                                 Adam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                     On 8/29/23 17:02, Adam
>>>>>>>>>                                     Hassick wrote:
>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                                     That fix seems to have
>>>>>>>>>>                                     resolved the issue,
>>>>>>>>>>                                     thanks for the quick
>>>>>>>>>>                                     turnaround time on that
>>>>>>>>>>                                     patch.
>>>>>>>>>>                                     Now that we have the RCF
>>>>>>>>>>                                     timeout issue resolved,
>>>>>>>>>>                                     there are a few other
>>>>>>>>>>                                     questions and issues that
>>>>>>>>>>                                     we have about the tests
>>>>>>>>>>                                     themselves.
>>>>>>>>>>
>>>>>>>>>>                                     1. The test suite fails
>>>>>>>>>>                                     to build with a couple
>>>>>>>>>>                                     warnings.
>>>>>>>>>>
>>>>>>>>>>                                     Below is the stderr log
>>>>>>>>>>                                     from compilation:
>>>>>>>>>>
>>>>>>>>>>                                         FAILED:
>>>>>>>>>>                                         lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>>                                         cc
>>>>>>>>>>                                         -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>>                                         -Ilib -I../../lib
>>>>>>>>>>                                         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>>                                         -fdiagnostics-color=always
>>>>>>>>>>                                         -pipe
>>>>>>>>>>                                         -D_FILE_OFFSET_BITS=64
>>>>>>>>>>                                         -Wall -Winvalid-pch
>>>>>>>>>>                                         -Werror -g
>>>>>>>>>>                                         -D_GNU_SOURCE -O0
>>>>>>>>>>                                         -ggdb -Wall -W -fPIC
>>>>>>>>>>                                         -MD -MQ
>>>>>>>>>>                                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>                                         -MF
>>>>>>>>>>                                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>>                                         -o
>>>>>>>>>>                                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>                                         -c
>>>>>>>>>>                                         ../../lib/dpdk_pmd_ts.c
>>>>>>>>>>                                         ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>>                                         In function
>>>>>>>>>>                                         ‘test_create_traffic_generator_params’:
>>>>>>>>>>                                         ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>>                                         error: format not a
>>>>>>>>>>                                         string literal and no
>>>>>>>>>>                                         format arguments
>>>>>>>>>>                                         [-Werror=format-security]
>>>>>>>>>>                                         5577 |     rc =
>>>>>>>>>>                                         te_kvpair_add(result,
>>>>>>>>>>                                         buf, mode);
>>>>>>>>>>                                         |     ^~
>>>>>>>>>>                                         cc1: all warnings
>>>>>>>>>>                                         being treated as errors
>>>>>>>>>>                                         ninja: build stopped:
>>>>>>>>>>                                         subcommand failed.
>>>>>>>>>>                                         ninja: Entering
>>>>>>>>>>                                         directory `.'
>>>>>>>>>>                                         FAILED:
>>>>>>>>>>                                         lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>>                                         cc
>>>>>>>>>>                                         -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>>                                         -Ilib -I../../lib
>>>>>>>>>>                                         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>>                                         -fdiagnostics-color=always
>>>>>>>>>>                                         -pipe
>>>>>>>>>>                                         -D_FILE_OFFSET_BITS=64
>>>>>>>>>>                                         -Wall -Winvalid-pch
>>>>>>>>>>                                         -Werror -g
>>>>>>>>>>                                         -D_GNU_SOURCE -O0
>>>>>>>>>>                                         -ggdb -Wall -W -fPIC
>>>>>>>>>>                                         -MD -MQ
>>>>>>>>>>                                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>                                         -MF
>>>>>>>>>>                                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>>                                         -o
>>>>>>>>>>                                         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>                                         -c
>>>>>>>>>>                                         ../../lib/dpdk_pmd_ts.c
>>>>>>>>>>                                         ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>>                                         In function
>>>>>>>>>>                                         ‘test_create_traffic_generator_params’:
>>>>>>>>>>                                         ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>>                                         error: format not a
>>>>>>>>>>                                         string literal and no
>>>>>>>>>>                                         format arguments
>>>>>>>>>>                                         [-Werror=format-security]
>>>>>>>>>>                                         5577 |     rc =
>>>>>>>>>>                                         te_kvpair_add(result,
>>>>>>>>>>                                         buf, mode);
>>>>>>>>>>                                         |     ^~
>>>>>>>>>>                                         cc1: all warnings
>>>>>>>>>>                                         being treated as errors
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     This error wasn't
>>>>>>>>>>                                     occurring last week,
>>>>>>>>>>                                     which was the last time I
>>>>>>>>>>                                     ran the tests.
>>>>>>>>>>                                     The TE host and the DUT
>>>>>>>>>>                                     have GCC v9.4.0
>>>>>>>>>>                                     installed, and the tester
>>>>>>>>>>                                     has GCC v11.4.0
>>>>>>>>>>                                     installed, if this
>>>>>>>>>>                                     information is helpful.
>>>>>>>>>>
>>>>>>>>>>                                     2. On the Mellanox CX5s,
>>>>>>>>>>                                     there are over 6,000
>>>>>>>>>>                                     tests run, which
>>>>>>>>>>                                     collectively take around
>>>>>>>>>>                                     9 hours. Is it possible,
>>>>>>>>>>                                     and would it make sense,
>>>>>>>>>>                                     to lower the test
>>>>>>>>>>                                     coverage and have the
>>>>>>>>>>                                     test suite run faster?
>>>>>>>>>>
>>>>>>>>>>                                     For some context, we run
>>>>>>>>>>                                     immediate testing on
>>>>>>>>>>                                     incoming patches for DPDK
>>>>>>>>>>                                     main and development
>>>>>>>>>>                                     branches, as well as
>>>>>>>>>>                                     periodic test runs on the
>>>>>>>>>>                                     main, stable, and LTS
>>>>>>>>>>                                     branches.
>>>>>>>>>>                                     For us to consider
>>>>>>>>>>                                     including this test suite
>>>>>>>>>>                                     as part of our immediate
>>>>>>>>>>                                     testing on patches, we
>>>>>>>>>>                                     would have to reduce the
>>>>>>>>>>                                     test coverage to the most
>>>>>>>>>>                                     important tests.
>>>>>>>>>>                                     This is primarily to
>>>>>>>>>>                                     reduce the testing time
>>>>>>>>>>                                     to, for example, less
>>>>>>>>>>                                     than 30 minutes. Testing
>>>>>>>>>>                                     on patches can't take too
>>>>>>>>>>                                     long because the lab can
>>>>>>>>>>                                     receive numerous patches
>>>>>>>>>>                                     each day, which each
>>>>>>>>>>                                     require individual
>>>>>>>>>>                                     testing runs.
>>>>>>>>>>
>>>>>>>>>>                                     At what frequency we run
>>>>>>>>>>                                     these tests, and on what,
>>>>>>>>>>                                     still needs to be
>>>>>>>>>>                                     discussed with the DPDK
>>>>>>>>>>                                     community, but it would
>>>>>>>>>>                                     be nice to know if the
>>>>>>>>>>                                     test suite had a "dial"
>>>>>>>>>>                                     to control the testing
>>>>>>>>>>                                     coverage.
>>>>>>>>>>
>>>>>>>>>>                                     3. We see a lot of test
>>>>>>>>>>                                     failures on our Mellanox
>>>>>>>>>>                                     CX5 NICs. Around 2,300 of
>>>>>>>>>>                                     ~6,600 tests passed. Is
>>>>>>>>>>                                     there anything we can do
>>>>>>>>>>                                     to diagnose these test
>>>>>>>>>>                                     failures?
>>>>>>>>>>
>>>>>>>>>>                                     Thanks,
>>>>>>>>>>                                     Adam
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     On Tue, Aug 29, 2023 at
>>>>>>>>>>                                     8:07 AM Andrew Rybchenko
>>>>>>>>>>                                     <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>                                     wrote:
>>>>>>>>>>
>>>>>>>>>>                                         Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                                         I've pushed the fix
>>>>>>>>>>                                         in main branch and a
>>>>>>>>>>                                         new tag v1.18.1. It
>>>>>>>>>>                                         should solve the
>>>>>>>>>>                                         problem with IPv6
>>>>>>>>>>                                         address from DNS.
>>>>>>>>>>
>>>>>>>>>>                                         Andrew.
>>>>>>>>>>
>>>>>>>>>>                                         On 8/29/23 00:05,
>>>>>>>>>>                                         Andrew Rybchenko wrote:
>>>>>>>>>>>                                         Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                         > Does the test
>>>>>>>>>>>                                         engine prefer to use
>>>>>>>>>>>                                         IPv6 over IPv4 for
>>>>>>>>>>>                                         initiating the RCF
>>>>>>>>>>>                                         connection to the
>>>>>>>>>>>                                         test bed hosts? And
>>>>>>>>>>>                                         if so, is there a
>>>>>>>>>>>                                         way to force it to
>>>>>>>>>>>                                         use IPv4?
>>>>>>>>>>>
>>>>>>>>>>>                                         Brilliant idea. If
>>>>>>>>>>>                                         DNS returns both
>>>>>>>>>>>                                         IPv4 and IPv6
>>>>>>>>>>>                                         addresses in your
>>>>>>>>>>>                                         case, I guess it is
>>>>>>>>>>>                                         the root cause of
>>>>>>>>>>>                                         the problem.
>>>>>>>>>>>                                         Of course, it is TE
>>>>>>>>>>>                                         problem since I see
>>>>>>>>>>>                                         really weird code in
>>>>>>>>>>>                                         lib/comm_net_engine/comm_net_engine.c
>>>>>>>>>>>                                         line 135.
>>>>>>>>>>>
>>>>>>>>>>>                                         I've pushed fix to
>>>>>>>>>>>                                         the branch
>>>>>>>>>>>                                         user/arybchik/fix_ipv4_only
>>>>>>>>>>>                                         in
>>>>>>>>>>>                                         ts-factory/test-environment
>>>>>>>>>>>                                         repository. Please, try.
>>>>>>>>>>>
>>>>>>>>>>>                                         It is late night fix
>>>>>>>>>>>                                         with minimal testing
>>>>>>>>>>>                                         and no review. I'll
>>>>>>>>>>>                                         pass it through
>>>>>>>>>>>                                         review process
>>>>>>>>>>>                                         tomorrow and
>>>>>>>>>>>                                         hopefully it will be
>>>>>>>>>>>                                         released in one-two
>>>>>>>>>>>                                         days.
>>>>>>>>>>>
>>>>>>>>>>>                                         Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                         On 8/28/23 18:02,
>>>>>>>>>>>                                         Adam Hassick wrote:
>>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                                         We have yet to
>>>>>>>>>>>>                                         notice a distinct
>>>>>>>>>>>>                                         pattern with the
>>>>>>>>>>>>                                         failures.
>>>>>>>>>>>>                                         Sometimes, the RCF
>>>>>>>>>>>>                                         will start and
>>>>>>>>>>>>                                         connect without
>>>>>>>>>>>>                                         issue a few times
>>>>>>>>>>>>                                         in a row before
>>>>>>>>>>>>                                         failing to connect
>>>>>>>>>>>>                                         again. Once the
>>>>>>>>>>>>                                         issue begins to
>>>>>>>>>>>>                                         occur, neither
>>>>>>>>>>>>                                         rebooting all of
>>>>>>>>>>>>                                         the hosts (test
>>>>>>>>>>>>                                         engine VM, tester,
>>>>>>>>>>>>                                         IUT) or deleting
>>>>>>>>>>>>                                         all of the build
>>>>>>>>>>>>                                         directories
>>>>>>>>>>>>                                         (suites, agents,
>>>>>>>>>>>>                                         inst) and rebooting
>>>>>>>>>>>>                                         the hosts afterward
>>>>>>>>>>>>                                         resolves the issue.
>>>>>>>>>>>>                                         When it begins
>>>>>>>>>>>>                                         working again seems
>>>>>>>>>>>>                                         very arbitrary to us.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         I do usually try to
>>>>>>>>>>>>                                         terminate the test
>>>>>>>>>>>>                                         engine with Ctrl+C,
>>>>>>>>>>>>                                         but when it hangs
>>>>>>>>>>>>                                         while trying to
>>>>>>>>>>>>                                         start RCF, that
>>>>>>>>>>>>                                         does not work.
>>>>>>>>>>>>
>>>>>>>>>>>>                                         Does the test
>>>>>>>>>>>>                                         engine prefer to
>>>>>>>>>>>>                                         use IPv6 over IPv4
>>>>>>>>>>>>                                         for initiating the
>>>>>>>>>>>>                                         RCF connection to
>>>>>>>>>>>>                                         the test bed hosts?
>>>>>>>>>>>>                                         And if so, is there
>>>>>>>>>>>>                                         a way to force it
>>>>>>>>>>>>                                         to use IPv4?
>>>>>>>>>>>>
>>>>>>>>>>>>                                          - Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                                         On Fri, Aug 25,
>>>>>>>>>>>>                                         2023 at 1:35 PM
>>>>>>>>>>>>                                         Andrew Rybchenko
>>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                             > I'll
>>>>>>>>>>>>                                             double-check
>>>>>>>>>>>>                                             test engine on
>>>>>>>>>>>>                                             Ubuntu 20.04
>>>>>>>>>>>>                                             and Ubuntu 22.04.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Done. It works
>>>>>>>>>>>>                                             fine for me
>>>>>>>>>>>>                                             without any issues.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Have you
>>>>>>>>>>>>                                             noticed any
>>>>>>>>>>>>                                             pattern when it
>>>>>>>>>>>>                                             works or does
>>>>>>>>>>>>                                             not work?
>>>>>>>>>>>>                                             May be it is a
>>>>>>>>>>>>                                             problem of not
>>>>>>>>>>>>                                             clean state
>>>>>>>>>>>>                                             after termination?
>>>>>>>>>>>>                                             Does it work
>>>>>>>>>>>>                                             fine the first
>>>>>>>>>>>>                                             time after DUTs
>>>>>>>>>>>>                                             reboot?
>>>>>>>>>>>>                                             How do you
>>>>>>>>>>>>                                             terminate
>>>>>>>>>>>>                                             testing? It
>>>>>>>>>>>>                                             should be done
>>>>>>>>>>>>                                             using Ctrl+C in
>>>>>>>>>>>>                                             terminal where
>>>>>>>>>>>>                                             you execute
>>>>>>>>>>>>                                             run.sh command.
>>>>>>>>>>>>                                              In this case
>>>>>>>>>>>>                                             it should
>>>>>>>>>>>>                                             shutdown
>>>>>>>>>>>>                                             gracefully and
>>>>>>>>>>>>                                             close all test
>>>>>>>>>>>>                                             agents and
>>>>>>>>>>>>                                             engine
>>>>>>>>>>>>                                             applications.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             (I'm trying to
>>>>>>>>>>>>                                             understand why
>>>>>>>>>>>>                                             you've seen
>>>>>>>>>>>>                                             many test agent
>>>>>>>>>>>>                                             processes. It
>>>>>>>>>>>>                                             should not happen.)
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             On 8/25/23
>>>>>>>>>>>>                                             17:41, Andrew
>>>>>>>>>>>>                                             Rybchenko wrote:
>>>>>>>>>>>>>                                             On 8/25/23
>>>>>>>>>>>>>                                             17:06, Adam
>>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Two of our
>>>>>>>>>>>>>>                                             systems (the
>>>>>>>>>>>>>>                                             Test Engine
>>>>>>>>>>>>>>                                             runner and
>>>>>>>>>>>>>>                                             the DUT host)
>>>>>>>>>>>>>>                                             are running
>>>>>>>>>>>>>>                                             Ubuntu 20.04
>>>>>>>>>>>>>>                                             LTS, however
>>>>>>>>>>>>>>                                             this morning
>>>>>>>>>>>>>>                                             I noticed
>>>>>>>>>>>>>>                                             that the
>>>>>>>>>>>>>>                                             tester system
>>>>>>>>>>>>>>                                             (the one
>>>>>>>>>>>>>>                                             having
>>>>>>>>>>>>>>                                             issues) is
>>>>>>>>>>>>>>                                             running
>>>>>>>>>>>>>>                                             Ubuntu 22.04 LTS.
>>>>>>>>>>>>>>                                             This could be
>>>>>>>>>>>>>>                                             the source of
>>>>>>>>>>>>>>                                             the problem.
>>>>>>>>>>>>>>                                             I encountered
>>>>>>>>>>>>>>                                             a dependency
>>>>>>>>>>>>>>                                             issue trying
>>>>>>>>>>>>>>                                             to run the
>>>>>>>>>>>>>>                                             Test Engine
>>>>>>>>>>>>>>                                             on 22.04 LTS,
>>>>>>>>>>>>>>                                             so I
>>>>>>>>>>>>>>                                             downgraded
>>>>>>>>>>>>>>                                             the system.
>>>>>>>>>>>>>>                                             Since the
>>>>>>>>>>>>>>                                             tester is
>>>>>>>>>>>>>>                                             also the host
>>>>>>>>>>>>>>                                             having
>>>>>>>>>>>>>>                                             connection
>>>>>>>>>>>>>>                                             issues, I
>>>>>>>>>>>>>>                                             will try
>>>>>>>>>>>>>>                                             downgrading
>>>>>>>>>>>>>>                                             that system
>>>>>>>>>>>>>>                                             to 20.04, and
>>>>>>>>>>>>>>                                             see if that
>>>>>>>>>>>>>>                                             changes anything.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Unlikely, but
>>>>>>>>>>>>>                                             who knows. We
>>>>>>>>>>>>>                                             run tests
>>>>>>>>>>>>>                                             (DUTs) on
>>>>>>>>>>>>>                                             Ubuntu 20.04,
>>>>>>>>>>>>>                                             Ubuntu 22.04,
>>>>>>>>>>>>>                                             Ubuntu 22.10,
>>>>>>>>>>>>>                                             Ubuntu 23.04,
>>>>>>>>>>>>>                                             Debian 11 and
>>>>>>>>>>>>>                                             Fedora 38
>>>>>>>>>>>>>                                             every night.
>>>>>>>>>>>>>                                             Right now
>>>>>>>>>>>>>                                             Debian 11 is
>>>>>>>>>>>>>                                             used for test
>>>>>>>>>>>>>                                             engine in
>>>>>>>>>>>>>                                             nightly
>>>>>>>>>>>>>                                             regressions.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             I'll
>>>>>>>>>>>>>                                             double-check
>>>>>>>>>>>>>                                             test engine on
>>>>>>>>>>>>>                                             Ubuntu 20.04
>>>>>>>>>>>>>                                             and Ubuntu 22.04.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             I did try
>>>>>>>>>>>>>>                                             passing in
>>>>>>>>>>>>>>                                             the
>>>>>>>>>>>>>>                                             "--vg-rcf"
>>>>>>>>>>>>>>                                             argument to
>>>>>>>>>>>>>>                                             the run.sh
>>>>>>>>>>>>>>                                             script of the
>>>>>>>>>>>>>>                                             test suite
>>>>>>>>>>>>>>                                             after
>>>>>>>>>>>>>>                                             installing
>>>>>>>>>>>>>>                                             valgrind, but
>>>>>>>>>>>>>>                                             there was no
>>>>>>>>>>>>>>                                             additional
>>>>>>>>>>>>>>                                             output that I
>>>>>>>>>>>>>>                                             saw.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Sorry, I
>>>>>>>>>>>>>                                             should
>>>>>>>>>>>>>                                             valgrind
>>>>>>>>>>>>>                                             output should
>>>>>>>>>>>>>                                             be in
>>>>>>>>>>>>>                                             valgrind.te_rcf
>>>>>>>>>>>>>                                             (direction
>>>>>>>>>>>>>                                             where you run
>>>>>>>>>>>>>                                             test engine).
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             I will try
>>>>>>>>>>>>>>                                             pulling in
>>>>>>>>>>>>>>                                             the changes
>>>>>>>>>>>>>>                                             you've pushed
>>>>>>>>>>>>>>                                             up, and will
>>>>>>>>>>>>>>                                             see if that
>>>>>>>>>>>>>>                                             fixes anything.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             Thanks,
>>>>>>>>>>>>>>                                             Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             On Fri, Aug
>>>>>>>>>>>>>>                                             25, 2023 at
>>>>>>>>>>>>>>                                             9:57 AM
>>>>>>>>>>>>>>                                             Andrew
>>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Hello Adam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>                                                 8/24/23
>>>>>>>>>>>>>>                                                 23:54,
>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>                                                 I'd like
>>>>>>>>>>>>>>>                                                 to try
>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>                                                 repeat
>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>                                                 problem
>>>>>>>>>>>>>>>                                                 locally.
>>>>>>>>>>>>>>>                                                 Which
>>>>>>>>>>>>>>>                                                 Linux
>>>>>>>>>>>>>>>                                                 distro
>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>                                                 on test
>>>>>>>>>>>>>>>                                                 engine
>>>>>>>>>>>>>>>                                                 and agents?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 In fact
>>>>>>>>>>>>>>>                                                 I know
>>>>>>>>>>>>>>>                                                 one
>>>>>>>>>>>>>>>                                                 problem
>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>                                                 Debian
>>>>>>>>>>>>>>>                                                 12 and
>>>>>>>>>>>>>>>                                                 Fedora
>>>>>>>>>>>>>>>                                                 38 and
>>>>>>>>>>>>>>>                                                 we have
>>>>>>>>>>>>>>>                                                 patch in
>>>>>>>>>>>>>>>                                                 review
>>>>>>>>>>>>>>>                                                 to fix
>>>>>>>>>>>>>>>                                                 it,
>>>>>>>>>>>>>>>                                                 however,
>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>                                                 behaviour
>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>                                                 different in
>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>                                                 case, so
>>>>>>>>>>>>>>>                                                 it is
>>>>>>>>>>>>>>>                                                 unlike
>>>>>>>>>>>>>>>                                                 the same
>>>>>>>>>>>>>>>                                                 problem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 I've just
>>>>>>>>>>>>>>                                                 published
>>>>>>>>>>>>>>                                                 a new tag
>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>                                                 fixes
>>>>>>>>>>>>>>                                                 known
>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>                                                 engine
>>>>>>>>>>>>>>                                                 side
>>>>>>>>>>>>>>                                                 problems
>>>>>>>>>>>>>>                                                 on Debian
>>>>>>>>>>>>>>                                                 12 and
>>>>>>>>>>>>>>                                                 Fedora 38.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 One more
>>>>>>>>>>>>>>>                                                 idea is
>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>                                                 install
>>>>>>>>>>>>>>>                                                 valgrind
>>>>>>>>>>>>>>>                                                 on the
>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>                                                 engine
>>>>>>>>>>>>>>>                                                 host and
>>>>>>>>>>>>>>>                                                 run with
>>>>>>>>>>>>>>>                                                 option
>>>>>>>>>>>>>>>                                                 --vg-rcf
>>>>>>>>>>>>>>>                                                 to check
>>>>>>>>>>>>>>>                                                 if
>>>>>>>>>>>>>>>                                                 something
>>>>>>>>>>>>>>>                                                 weird is
>>>>>>>>>>>>>>>                                                 happening.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 What I
>>>>>>>>>>>>>>>                                                 don't
>>>>>>>>>>>>>>>                                                 understand
>>>>>>>>>>>>>>>                                                 right
>>>>>>>>>>>>>>>                                                 now is
>>>>>>>>>>>>>>>                                                 why I
>>>>>>>>>>>>>>>                                                 see just
>>>>>>>>>>>>>>>                                                 one
>>>>>>>>>>>>>>>                                                 failed
>>>>>>>>>>>>>>>                                                 attempt
>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>                                                 connect
>>>>>>>>>>>>>>>                                                 in your
>>>>>>>>>>>>>>>                                                 log.txt
>>>>>>>>>>>>>>>                                                 and then
>>>>>>>>>>>>>>>                                                 Logger
>>>>>>>>>>>>>>>                                                 shutdown
>>>>>>>>>>>>>>>                                                 after 9
>>>>>>>>>>>>>>>                                                 minutes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>                                                 8/24/23
>>>>>>>>>>>>>>>                                                 23:29,
>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>                                                 Hassick
>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>                                                  > Is
>>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>>>>                                                 firewall
>>>>>>>>>>>>>>>>                                                 in the
>>>>>>>>>>>>>>>>                                                 network
>>>>>>>>>>>>>>>>                                                 or on
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 hosts
>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>                                                 block
>>>>>>>>>>>>>>>>                                                 incoming
>>>>>>>>>>>>>>>>                                                 TCP
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 to the
>>>>>>>>>>>>>>>>                                                 port
>>>>>>>>>>>>>>>>                                                 23571
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 host
>>>>>>>>>>>>>>>>                                                 where
>>>>>>>>>>>>>>>>                                                 you run
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 engine?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 Our
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 engine
>>>>>>>>>>>>>>>>                                                 host
>>>>>>>>>>>>>>>>                                                 and the
>>>>>>>>>>>>>>>>                                                 testbed
>>>>>>>>>>>>>>>>                                                 are on
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 same
>>>>>>>>>>>>>>>>                                                 subnet.
>>>>>>>>>>>>>>>>                                                 The
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 does
>>>>>>>>>>>>>>>>                                                 work
>>>>>>>>>>>>>>>>                                                 sometimes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                  > If
>>>>>>>>>>>>>>>>                                                 behaviour
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 same on
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 next
>>>>>>>>>>>>>>>>                                                 try and
>>>>>>>>>>>>>>>>                                                 you see
>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>>                                                 is kept
>>>>>>>>>>>>>>>>                                                 running,
>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>                                                 check using
>>>>>>>>>>>>>>>>                                                  >
>>>>>>>>>>>>>>>>                                                  > #
>>>>>>>>>>>>>>>>                                                 netstat
>>>>>>>>>>>>>>>>                                                 -tnlp
>>>>>>>>>>>>>>>>                                                  >
>>>>>>>>>>>>>>>>                                                  > that
>>>>>>>>>>>>>>>>                                                 Test
>>>>>>>>>>>>>>>>                                                 Agent
>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>                                                 listening
>>>>>>>>>>>>>>>>                                                 on the
>>>>>>>>>>>>>>>>                                                 port
>>>>>>>>>>>>>>>>                                                 and try
>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>                                                 establish
>>>>>>>>>>>>>>>>                                                 TCP
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 agent using
>>>>>>>>>>>>>>>>                                                  >
>>>>>>>>>>>>>>>>                                                  > $
>>>>>>>>>>>>>>>>                                                 telnet
>>>>>>>>>>>>>>>>                                                 iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 23571
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                  >
>>>>>>>>>>>>>>>>                                                  > and
>>>>>>>>>>>>>>>>                                                 check
>>>>>>>>>>>>>>>>                                                 if TCP
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>                                                 established.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I was
>>>>>>>>>>>>>>>>                                                 able to
>>>>>>>>>>>>>>>>                                                 replicate
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 same
>>>>>>>>>>>>>>>>                                                 behavior
>>>>>>>>>>>>>>>>                                                 again,
>>>>>>>>>>>>>>>>                                                 where
>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>                                                 hangs
>>>>>>>>>>>>>>>>                                                 while
>>>>>>>>>>>>>>>>                                                 RCF is
>>>>>>>>>>>>>>>>                                                 trying
>>>>>>>>>>>>>>>>                                                 to start.
>>>>>>>>>>>>>>>>                                                 Running
>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>                                                 command,
>>>>>>>>>>>>>>>>                                                 I see
>>>>>>>>>>>>>>>>                                                 this in
>>>>>>>>>>>>>>>>                                                 the output:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 tcp    
>>>>>>>>>>>>>>>>                                                    0  
>>>>>>>>>>>>>>>>                                                  0
>>>>>>>>>>>>>>>>                                                 0.0.0.0:23571
>>>>>>>>>>>>>>>>                                                 <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>>                                                 <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>>                                                 <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>>                                                 0.0.0.0:*
>>>>>>>>>>>>>>>>                                                 LISTEN
>>>>>>>>>>>>>>>>                                                  18599/ta
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 So it
>>>>>>>>>>>>>>>>                                                 seems
>>>>>>>>>>>>>>>>                                                 like it
>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>                                                 listening
>>>>>>>>>>>>>>>>                                                 on the
>>>>>>>>>>>>>>>>                                                 correct
>>>>>>>>>>>>>>>>                                                 port.
>>>>>>>>>>>>>>>>                                                 Additionally,
>>>>>>>>>>>>>>>>                                                 I was
>>>>>>>>>>>>>>>>                                                 able to
>>>>>>>>>>>>>>>>                                                 connect
>>>>>>>>>>>>>>>>                                                 to the
>>>>>>>>>>>>>>>>                                                 Tester
>>>>>>>>>>>>>>>>                                                 machine
>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>                                                 Test
>>>>>>>>>>>>>>>>                                                 Engine
>>>>>>>>>>>>>>>>                                                 host
>>>>>>>>>>>>>>>>                                                 using
>>>>>>>>>>>>>>>>                                                 telnet.
>>>>>>>>>>>>>>>>                                                 It
>>>>>>>>>>>>>>>>                                                 printed
>>>>>>>>>>>>>>>>                                                 the PID
>>>>>>>>>>>>>>>>                                                 of the
>>>>>>>>>>>>>>>>                                                 process
>>>>>>>>>>>>>>>>                                                 once
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 was opened.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 I tried
>>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 "ta"
>>>>>>>>>>>>>>>>                                                 application
>>>>>>>>>>>>>>>>                                                 manually
>>>>>>>>>>>>>>>>                                                 on the
>>>>>>>>>>>>>>>>                                                 command
>>>>>>>>>>>>>>>>                                                 line,
>>>>>>>>>>>>>>>>                                                 and it
>>>>>>>>>>>>>>>>                                                 didn't
>>>>>>>>>>>>>>>>                                                 print
>>>>>>>>>>>>>>>>                                                 anything
>>>>>>>>>>>>>>>>                                                 at all.
>>>>>>>>>>>>>>>>                                                 Maybe
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 issue
>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>                                                 something
>>>>>>>>>>>>>>>>                                                 on the
>>>>>>>>>>>>>>>>                                                 Test
>>>>>>>>>>>>>>>>                                                 Engine
>>>>>>>>>>>>>>>>                                                 side.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 On Thu,
>>>>>>>>>>>>>>>>                                                 Aug 24,
>>>>>>>>>>>>>>>>                                                 2023 at
>>>>>>>>>>>>>>>>                                                 2:35 PM
>>>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>>>>                                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>                                                 Adam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>                                                 On the
>>>>>>>>>>>>>>>>                                                 tester
>>>>>>>>>>>>>>>>                                                 host
>>>>>>>>>>>>>>>>                                                 (which
>>>>>>>>>>>>>>>>                                                 appears
>>>>>>>>>>>>>>>>                                                 to be
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 Peer
>>>>>>>>>>>>>>>>                                                 agent),
>>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>                                                 four
>>>>>>>>>>>>>>>>                                                 processes
>>>>>>>>>>>>>>>>                                                 that I
>>>>>>>>>>>>>>>>                                                 see
>>>>>>>>>>>>>>>>                                                 running,
>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>                                                 look
>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>                                                 the test
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>>                                                 processes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 Before
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 next
>>>>>>>>>>>>>>>>                                                 try I'd
>>>>>>>>>>>>>>>>                                                 recommend
>>>>>>>>>>>>>>>>                                                 to kill
>>>>>>>>>>>>>>>>                                                 these
>>>>>>>>>>>>>>>>                                                 processes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Is
>>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>>>>                                                 firewall
>>>>>>>>>>>>>>>>                                                 in the
>>>>>>>>>>>>>>>>                                                 network
>>>>>>>>>>>>>>>>                                                 or on
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 hosts
>>>>>>>>>>>>>>>>                                                 which could
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 block
>>>>>>>>>>>>>>>>                                                 incoming
>>>>>>>>>>>>>>>>                                                 TCP
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 to the
>>>>>>>>>>>>>>>>                                                 port 23571
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>                                                 the host
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 where
>>>>>>>>>>>>>>>>                                                 you run
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 engine?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     If
>>>>>>>>>>>>>>>>                                                 behaviour
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 same on
>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>                                                 next
>>>>>>>>>>>>>>>>                                                 try and
>>>>>>>>>>>>>>>>                                                 you see
>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 agent is
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 kept
>>>>>>>>>>>>>>>>                                                 running,
>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>                                                 check using
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     #
>>>>>>>>>>>>>>>>                                                 netstat
>>>>>>>>>>>>>>>>                                                 -tnlp
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>                                                 Test
>>>>>>>>>>>>>>>>                                                 Agent
>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>                                                 listening
>>>>>>>>>>>>>>>>                                                 on the
>>>>>>>>>>>>>>>>                                                 port
>>>>>>>>>>>>>>>>                                                 and try
>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>                                                 establish
>>>>>>>>>>>>>>>>                                                 TCP
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 agent using
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     $
>>>>>>>>>>>>>>>>                                                 telnet
>>>>>>>>>>>>>>>>                                                 iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 23571
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>                                                 check
>>>>>>>>>>>>>>>>                                                 if TCP
>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>                                                 established.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 Another
>>>>>>>>>>>>>>>>                                                 idea is
>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>                                                 login
>>>>>>>>>>>>>>>>                                                 Tester
>>>>>>>>>>>>>>>>                                                 under
>>>>>>>>>>>>>>>>                                                 root as
>>>>>>>>>>>>>>>>                                                 testing
>>>>>>>>>>>>>>>>                                                 does, get
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 start
>>>>>>>>>>>>>>>>                                                 TA
>>>>>>>>>>>>>>>>                                                 command
>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>                                                 the log
>>>>>>>>>>>>>>>>                                                 and try
>>>>>>>>>>>>>>>>                                                 it by
>>>>>>>>>>>>>>>>                                                 hands
>>>>>>>>>>>>>>>>                                                 without
>>>>>>>>>>>>>>>>                                                 -n and
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 remove
>>>>>>>>>>>>>>>>                                                 extra
>>>>>>>>>>>>>>>>                                                 escaping.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     #
>>>>>>>>>>>>>>>>                                                 sudo
>>>>>>>>>>>>>>>>                                                 PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>>                                                 LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>>                                                 /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>>>>>                                                 Peer
>>>>>>>>>>>>>>>>                                                 23571
>>>>>>>>>>>>>>>>                                                 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>                                                 Hopefully
>>>>>>>>>>>>>>>>                                                 in this
>>>>>>>>>>>>>>>>                                                 case
>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>>                                                 directory
>>>>>>>>>>>>>>>>                                                 remains
>>>>>>>>>>>>>>>>                                                 in the
>>>>>>>>>>>>>>>>                                                 /tmp and
>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>                                                 don't
>>>>>>>>>>>>>>>>                                                 need to
>>>>>>>>>>>>>>>>                                                 copy it
>>>>>>>>>>>>>>>>                                                 as
>>>>>>>>>>>>>>>>                                                 testing
>>>>>>>>>>>>>>>>                                                 does.
>>>>>>>>>>>>>>>>                                                     May
>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>                                                 output
>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>                                                 shed
>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>                                                 light
>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>                                                 what's
>>>>>>>>>>>>>>>>                                                 going on.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>                                                 8/24/23
>>>>>>>>>>>>>>>>                                                 17:30,
>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>                                                 Hassick
>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>                                                 Hi Andrew,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 This
>>>>>>>>>>>>>>>>>                                                 is the
>>>>>>>>>>>>>>>>>                                                 output
>>>>>>>>>>>>>>>>>                                                 that I
>>>>>>>>>>>>>>>>>                                                 see in
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 terminal
>>>>>>>>>>>>>>>>>                                                 when
>>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>>                                                 failure
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 occurs,
>>>>>>>>>>>>>>>>>                                                 after
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>>>                                                 binaries
>>>>>>>>>>>>>>>>>                                                 build
>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>                                                 engine
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 starts:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 Platform
>>>>>>>>>>>>>>>>>                                                 default
>>>>>>>>>>>>>>>>>                                                 build
>>>>>>>>>>>>>>>>>                                                 - pass
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 Simple
>>>>>>>>>>>>>>>>>                                                 RCF
>>>>>>>>>>>>>>>>>                                                 consistency
>>>>>>>>>>>>>>>>>                                                 check
>>>>>>>>>>>>>>>>>                                                 succeeded
>>>>>>>>>>>>>>>>>                                                 --->>>
>>>>>>>>>>>>>>>>>                                                 Starting
>>>>>>>>>>>>>>>>>                                                 Logger...done
>>>>>>>>>>>>>>>>>                                                 --->>>
>>>>>>>>>>>>>>>>>                                                 Starting
>>>>>>>>>>>>>>>>>                                                 RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>>>>>                                                 Connection
>>>>>>>>>>>>>>>>>                                                 timed
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 out
>>>>>>>>>>>>>>>>>                                                 iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 Then,
>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>                                                 hangs
>>>>>>>>>>>>>>>>>                                                 here
>>>>>>>>>>>>>>>>>                                                 until
>>>>>>>>>>>>>>>>>                                                 I kill
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 "te_rcf"
>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>                                                 "te_tee"
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 processes.
>>>>>>>>>>>>>>>>>                                                 I let
>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>                                                 hang
>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>                                                 around
>>>>>>>>>>>>>>>>>                                                 9 minutes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 tester
>>>>>>>>>>>>>>>>>                                                 host
>>>>>>>>>>>>>>>>>                                                 (which
>>>>>>>>>>>>>>>>>                                                 appears
>>>>>>>>>>>>>>>>>                                                 to be
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 Peer
>>>>>>>>>>>>>>>>>                                                 agent),
>>>>>>>>>>>>>>>>>                                                 there are
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 four
>>>>>>>>>>>>>>>>>                                                 processes
>>>>>>>>>>>>>>>>>                                                 that I
>>>>>>>>>>>>>>>>>                                                 see
>>>>>>>>>>>>>>>>>                                                 running,
>>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>>                                                 look
>>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 test agent
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 processes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 ta.Peer
>>>>>>>>>>>>>>>>>                                                 is an
>>>>>>>>>>>>>>>>>                                                 empty
>>>>>>>>>>>>>>>>>                                                 file.
>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>                                                 attached
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 log.txt
>>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>>                                                 this run.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                      -
>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                                 Thu,
>>>>>>>>>>>>>>>>>                                                 Aug
>>>>>>>>>>>>>>>>>                                                 24,
>>>>>>>>>>>>>>>>>                                                 2023
>>>>>>>>>>>>>>>>>                                                 at
>>>>>>>>>>>>>>>>>                                                 4:22��AM
>>>>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 Hi Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 Yes,
>>>>>>>>>>>>>>>>>                                                 TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>>                                                 is in
>>>>>>>>>>>>>>>>>                                                 seconds.
>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>                                                 double-checked
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>                                                 goes
>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>                                                 'copy_timeout'
>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>                                                 ts-conf/rcf.conf.
>>>>>>>>>>>>>>>>>                                                 Description
>>>>>>>>>>>>>>>>>                                                 in in
>>>>>>>>>>>>>>>>>                                                 doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 says
>>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>>                                                 copy_timeout
>>>>>>>>>>>>>>>>>                                                 is in
>>>>>>>>>>>>>>>>>                                                 seconds
>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>                                                 implementation
>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>                                                 lib/rcfunix/rcfunix.c
>>>>>>>>>>>>>>>>>                                                 passes
>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>                                                 value
>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>                                                 select()
>>>>>>>>>>>>>>>>>                                                 tv_sec.
>>>>>>>>>>>>>>>>>                                                 Theoretically
>>>>>>>>>>>>>>>>>                                                 select()
>>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>                                                 interrupted
>>>>>>>>>>>>>>>>>                                                 by
>>>>>>>>>>>>>>>>>                                                 signal,
>>>>>>>>>>>>>>>>>                                                 but I
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 think
>>>>>>>>>>>>>>>>>                                                 it is
>>>>>>>>>>>>>>>>>                                                 unlikely
>>>>>>>>>>>>>>>>>                                                 here.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>>                                                 not
>>>>>>>>>>>>>>>>>                                                 sure
>>>>>>>>>>>>>>>>>                                                 that I
>>>>>>>>>>>>>>>>>                                                 understand
>>>>>>>>>>>>>>>>>                                                 what
>>>>>>>>>>>>>>>>>                                                 do you
>>>>>>>>>>>>>>>>>                                                 mean
>>>>>>>>>>>>>>>>>                                                 by RCF
>>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>>                                                 timeout.
>>>>>>>>>>>>>>>>>                                                 Does
>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>                                                 happen
>>>>>>>>>>>>>>>>>                                                 on TE
>>>>>>>>>>>>>>>>>                                                 startup
>>>>>>>>>>>>>>>>>                                                 when RCF
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 starts
>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>                                                 agents.
>>>>>>>>>>>>>>>>>                                                 If so,
>>>>>>>>>>>>>>>>>                                                 TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>>                                                 help. Or
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 does
>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>                                                 happen
>>>>>>>>>>>>>>>>>                                                 when
>>>>>>>>>>>>>>>>>                                                 tests
>>>>>>>>>>>>>>>>>                                                 are in
>>>>>>>>>>>>>>>>>                                                 progress,
>>>>>>>>>>>>>>>>>                                                 e.g.
>>>>>>>>>>>>>>>>>                                                 in the
>>>>>>>>>>>>>>>>>                                                 middle
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 of a
>>>>>>>>>>>>>>>>>                                                 test.
>>>>>>>>>>>>>>>>>                                                 If so,
>>>>>>>>>>>>>>>>>                                                 TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>                                                 unrelated
>>>>>>>>>>>>>>>>>                                                 and most
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 likely
>>>>>>>>>>>>>>>>>                                                 either
>>>>>>>>>>>>>>>>>                                                 host
>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>>>                                                 dies
>>>>>>>>>>>>>>>>>                                                 or
>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>>>                                                 itself
>>>>>>>>>>>>>>>>>                                                 crashes.
>>>>>>>>>>>>>>>>>                                                 It
>>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>                                                 easier
>>>>>>>>>>>>>>>>>                                                 for me
>>>>>>>>>>>>>>>>>                                                 if
>>>>>>>>>>>>>>>>>                                                 classify
>>>>>>>>>>>>>>>>>                                                 it if
>>>>>>>>>>>>>>>>>                                                 you share
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 text
>>>>>>>>>>>>>>>>>                                                 log
>>>>>>>>>>>>>>>>>                                                 (log.txt,
>>>>>>>>>>>>>>>>>                                                 full
>>>>>>>>>>>>>>>>>                                                 or
>>>>>>>>>>>>>>>>>                                                 just
>>>>>>>>>>>>>>>>>                                                 corresponding
>>>>>>>>>>>>>>>>>                                                 fragment
>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>>                                                 context).
>>>>>>>>>>>>>>>>>                                                 Also
>>>>>>>>>>>>>>>>>                                                 content
>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>                                                 ta.DPDK
>>>>>>>>>>>>>>>>>                                                 or
>>>>>>>>>>>>>>>>>                                                 ta.Peer
>>>>>>>>>>>>>>>>>                                                 file
>>>>>>>>>>>>>>>>>                                                 depending
>>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>>                                                 agent
>>>>>>>>>>>>>>>>>                                                 has
>>>>>>>>>>>>>>>>>                                                 problems
>>>>>>>>>>>>>>>>>                                                 could
>>>>>>>>>>>>>>>>>                                                 shed
>>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>>                                                 light.
>>>>>>>>>>>>>>>>>                                                 Corresponding
>>>>>>>>>>>>>>>>>                                                 files
>>>>>>>>>>>>>>>>>                                                 contain
>>>>>>>>>>>>>>>>>                                                 stdout/stderr
>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>                                                 agents.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>                                                 8/23/23
>>>>>>>>>>>>>>>>>                                                 17:45,
>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>                                                 Hassick
>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>                                                 Hi
>>>>>>>>>>>>>>>>>>                                                 Andrew,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>>                                                 set
>>>>>>>>>>>>>>>>>>                                                 up a
>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>                                                 rig
>>>>>>>>>>>>>>>>>>                                                 repository
>>>>>>>>>>>>>>>>>>                                                 here,
>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>                                                 have
>>>>>>>>>>>>>>>>>>                                                 created
>>>>>>>>>>>>>>>>>>                                                 configurations
>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>>                                                 development
>>>>>>>>>>>>>>>>>>                                                 testbed
>>>>>>>>>>>>>>>>>>                                                 based
>>>>>>>>>>>>>>>>>>                                                 off
>>>>>>>>>>>>>>>>>>                                                 of the
>>>>>>>>>>>>>>>>>>                                                 examples.
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 We've
>>>>>>>>>>>>>>>>>>                                                 been
>>>>>>>>>>>>>>>>>>                                                 able
>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>                                                 get
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>                                                 suite
>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>                                                 manually
>>>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>>>                                                 Mellanox
>>>>>>>>>>>>>>>>>>                                                 CX5
>>>>>>>>>>>>>>>>>>                                                 devices
>>>>>>>>>>>>>>>>>>                                                 once.
>>>>>>>>>>>>>>>>>>                                                 However,
>>>>>>>>>>>>>>>>>>                                                 we
>>>>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>>>>                                                 into
>>>>>>>>>>>>>>>>>>                                                 an
>>>>>>>>>>>>>>>>>>                                                 issue
>>>>>>>>>>>>>>>>>>                                                 where,
>>>>>>>>>>>>>>>>>>                                                 when
>>>>>>>>>>>>>>>>>>                                                 RCF
>>>>>>>>>>>>>>>>>>                                                 starts,
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 RCF
>>>>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>>>>                                                 times
>>>>>>>>>>>>>>>>>>                                                 out
>>>>>>>>>>>>>>>>>>                                                 very
>>>>>>>>>>>>>>>>>>                                                 frequently.
>>>>>>>>>>>>>>>>>>                                                 We
>>>>>>>>>>>>>>>>>>                                                 aren't
>>>>>>>>>>>>>>>>>>                                                 sure
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 why
>>>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>                                                 the case.
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 It
>>>>>>>>>>>>>>>>>>                                                 works
>>>>>>>>>>>>>>>>>>                                                 sometimes,
>>>>>>>>>>>>>>>>>>                                                 but
>>>>>>>>>>>>>>>>>>                                                 most
>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 time
>>>>>>>>>>>>>>>>>>                                                 when
>>>>>>>>>>>>>>>>>>                                                 we
>>>>>>>>>>>>>>>>>>                                                 try
>>>>>>>>>>>>>>>>>>                                                 to run
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>                                                 engine,
>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>                                                 encounters
>>>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>>>                                                 issue.
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>>                                                 tried
>>>>>>>>>>>>>>>>>>                                                 changing
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 RCF
>>>>>>>>>>>>>>>>>>                                                 port
>>>>>>>>>>>>>>>>>>                                                 by
>>>>>>>>>>>>>>>>>>                                                 setting
>>>>>>>>>>>>>>>>>>                                                 "TE_RCF_PORT=<some
>>>>>>>>>>>>>>>>>>                                                 port
>>>>>>>>>>>>>>>>>>                                                 number>"
>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>                                                 rebooting
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 testbed
>>>>>>>>>>>>>>>>>>                                                 machines.
>>>>>>>>>>>>>>>>>>                                                 Neither
>>>>>>>>>>>>>>>>>>                                                 seems
>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>                                                 fix
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 issue.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 It
>>>>>>>>>>>>>>>>>>                                                 also
>>>>>>>>>>>>>>>>>>                                                 seems
>>>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 timeout
>>>>>>>>>>>>>>>>>>                                                 takes
>>>>>>>>>>>>>>>>>>                                                 far
>>>>>>>>>>>>>>>>>>                                                 longer
>>>>>>>>>>>>>>>>>>                                                 than 60
>>>>>>>>>>>>>>>>>>                                                 seconds,
>>>>>>>>>>>>>>>>>>                                                 even
>>>>>>>>>>>>>>>>>>                                                 when
>>>>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>>>>                                                 "export
>>>>>>>>>>>>>>>>>>                                                 TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 before
>>>>>>>>>>>>>>>>>>                                                 I try
>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>                                                 suite.
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>                                                 assume
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 unit
>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>>>                                                 variable
>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>                                                 seconds?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>>                                                 Mon,
>>>>>>>>>>>>>>>>>>                                                 Aug
>>>>>>>>>>>>>>>>>>                                                 21,
>>>>>>>>>>>>>>>>>>                                                 2023
>>>>>>>>>>>>>>>>>>                                                 at
>>>>>>>>>>>>>>>>>>                                                 10:19 AM
>>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>>                                                 Hassick
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 <ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                 Hi
>>>>>>>>>>>>>>>>>>                                                 Andrew,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>>                                                 cloned
>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>                                                 example
>>>>>>>>>>>>>>>>>>                                                 repository
>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>                                                 will
>>>>>>>>>>>>>>>>>>                                                 start
>>>>>>>>>>>>>>>>>>                                                 setting
>>>>>>>>>>>>>>>>>>                                                 up a
>>>>>>>>>>>>>>>>>>                                                 configuration
>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>>                                                 development
>>>>>>>>>>>>>>>>>>                                                 testbed
>>>>>>>>>>>>>>>>>>                                                 today.
>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>                                                 let
>>>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>>>                                                 know
>>>>>>>>>>>>>>>>>>                                                 if I
>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>                                                 into
>>>>>>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>>>>>>                                                 difficulties
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                 or
>>>>>>>>>>>>>>>>>>                                                 have
>>>>>>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>>>>>>                                                 questions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                  - Adam
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>>                                                 Sun,
>>>>>>>>>>>>>>>>>>                                                 Aug
>>>>>>>>>>>>>>>>>>                                                 20,
>>>>>>>>>>>>>>>>>>                                                 2023
>>>>>>>>>>>>>>>>>>                                                 at
>>>>>>>>>>>>>>>>>>                                                 4:40 AM
>>>>>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>>>>>>                                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 Hi Adam,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>>                                                 published
>>>>>>>>>>>>>>>>>>                                                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>>>>>                                                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>>>>>                                                 <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 Hopefully
>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>                                                 will
>>>>>>>>>>>>>>>>>>                                                 help
>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>                                                 define
>>>>>>>>>>>>>>>>>>                                                 your
>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>                                                 rigs and
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 successfully
>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>>>                                                 tests
>>>>>>>>>>>>>>>>>>                                                 manually.
>>>>>>>>>>>>>>>>>>                                                 Feel
>>>>>>>>>>>>>>>>>>                                                 free to
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 ask
>>>>>>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>>>>>>                                                 questions
>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>                                                 answer
>>>>>>>>>>>>>>>>>>                                                 here
>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>                                                 try to
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 update
>>>>>>>>>>>>>>>>>>                                                 documentation.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 Meanwhile
>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>                                                 prepare
>>>>>>>>>>>>>>>>>>                                                 missing
>>>>>>>>>>>>>>>>>>                                                 bits
>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>                                                 steps
>>>>>>>>>>>>>>>>>>                                                 (2) and
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 (3).
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 Hopefully
>>>>>>>>>>>>>>>>>>                                                 everything
>>>>>>>>>>>>>>>>>>                                                 is in
>>>>>>>>>>>>>>>>>>                                                 place
>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>                                                 step
>>>>>>>>>>>>>>>>>>                                                 (4),
>>>>>>>>>>>>>>>>>>                                                 but we
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 need
>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>                                                 make
>>>>>>>>>>>>>>>>>>                                                 steps
>>>>>>>>>>>>>>>>>>                                                 (2)
>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>                                                 (3)
>>>>>>>>>>>>>>>>>>                                                 first.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>>                                                 8/18/23
>>>>>>>>>>>>>>>>>>                                                 21:40,
>>>>>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>>                                                 Hi Adam,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 >
>>>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>>>                                                 conferred
>>>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>                                                 rest
>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>                                                 team,
>>>>>>>>>>>>>>>>>>>                                                 and we
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 think
>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>                                                 best
>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>                                                 move
>>>>>>>>>>>>>>>>>>>                                                 forward
>>>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>>>                                                 mainly
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 option
>>>>>>>>>>>>>>>>>>>                                                 B.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 OK,
>>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>>                                                 provide
>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>                                                 sample
>>>>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>>>>                                                 Monday
>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>                                                 you.
>>>>>>>>>>>>>>>>>>>                                                 It is
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 almost
>>>>>>>>>>>>>>>>>>>                                                 ready
>>>>>>>>>>>>>>>>>>>                                                 right
>>>>>>>>>>>>>>>>>>>                                                 now,
>>>>>>>>>>>>>>>>>>>                                                 but
>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>                                                 need
>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>                                                 double-check
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>                                                 before
>>>>>>>>>>>>>>>>>>>                                                 publishing.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 Regards,
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>>>                                                 8/17/23
>>>>>>>>>>>>>>>>>>>                                                 20:03,
>>>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>>>                                                 Hassick
>>>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>>>                                                 Hi
>>>>>>>>>>>>>>>>>>>>                                                 Andrew,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>>>>>                                                 adding
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 CI
>>>>>>>>>>>>>>>>>>>>                                                 mailing
>>>>>>>>>>>>>>>>>>>>                                                 list
>>>>>>>>>>>>>>>>>>>>                                                 to this
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 conversation.
>>>>>>>>>>>>>>>>>>>>                                                 Others
>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 community
>>>>>>>>>>>>>>>>>>>>                                                 might
>>>>>>>>>>>>>>>>>>>>                                                 find
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>>>>>                                                 conversation
>>>>>>>>>>>>>>>>>>>>                                                 valuable.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 We
>>>>>>>>>>>>>>>>>>>>                                                 do
>>>>>>>>>>>>>>>>>>>>                                                 want
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>>>                                                 testing
>>>>>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>>>>>                                                 a
>>>>>>>>>>>>>>>>>>>>                                                 regular
>>>>>>>>>>>>>>>>>>>>                                                 basis.
>>>>>>>>>>>>>>>>>>>>                                                 The
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 Jenkins
>>>>>>>>>>>>>>>>>>>>                                                 integration
>>>>>>>>>>>>>>>>>>>>                                                 will
>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>                                                 very
>>>>>>>>>>>>>>>>>>>>                                                 useful
>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>                                                 us, as
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 most
>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>>>>                                                 CI
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 orchestrated
>>>>>>>>>>>>>>>>>>>>                                                 by
>>>>>>>>>>>>>>>>>>>>                                                 Jenkins.
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 I've
>>>>>>>>>>>>>>>>>>>>                                                 conferred
>>>>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 rest
>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 team,
>>>>>>>>>>>>>>>>>>>>                                                 and we
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 think
>>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>                                                 best
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 move
>>>>>>>>>>>>>>>>>>>>                                                 forward
>>>>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>>>>                                                 mainly
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 option
>>>>>>>>>>>>>>>>>>>>                                                 B.
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 If
>>>>>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 know
>>>>>>>>>>>>>>>>>>>>                                                 anything
>>>>>>>>>>>>>>>>>>>>                                                 about
>>>>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 testbeds
>>>>>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>>>>>                                                 help
>>>>>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>>>>                                                 creating
>>>>>>>>>>>>>>>>>>>>                                                 an
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 example
>>>>>>>>>>>>>>>>>>>>                                                 ts-rigs
>>>>>>>>>>>>>>>>>>>>                                                 repo,
>>>>>>>>>>>>>>>>>>>>                                                 I'd
>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>                                                 happy
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 answer
>>>>>>>>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 questions
>>>>>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>>>>>                                                 have.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 We
>>>>>>>>>>>>>>>>>>>>                                                 have
>>>>>>>>>>>>>>>>>>>>                                                 multiple
>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>                                                 rigs
>>>>>>>>>>>>>>>>>>>>                                                 (we
>>>>>>>>>>>>>>>>>>>>                                                 call
>>>>>>>>>>>>>>>>>>>>                                                 these
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 "DUT-tester
>>>>>>>>>>>>>>>>>>>>                                                 pairs")
>>>>>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>>>>>                                                 we
>>>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>>>>                                                 existing
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 hardware
>>>>>>>>>>>>>>>>>>>>                                                 testing
>>>>>>>>>>>>>>>>>>>>                                                 on,
>>>>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>>>>                                                 differing
>>>>>>>>>>>>>>>>>>>>                                                 network
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 hardware
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 CPU
>>>>>>>>>>>>>>>>>>>>                                                 architecture.
>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>                                                 figured
>>>>>>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>>>>>>                                                 might
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>                                                 an
>>>>>>>>>>>>>>>>>>>>                                                 important
>>>>>>>>>>>>>>>>>>>>                                                 detail.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>>>>                                                 Thu,
>>>>>>>>>>>>>>>>>>>>                                                 Aug
>>>>>>>>>>>>>>>>>>>>                                                 17,
>>>>>>>>>>>>>>>>>>>>                                                 2023
>>>>>>>>>>>>>>>>>>>>                                                 at
>>>>>>>>>>>>>>>>>>>>                                                 11:44 AM
>>>>>>>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>>>                                                 <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Greatings
>>>>>>>>>>>>>>>>>>>>                                                 Adam,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>>>>>                                                 happy
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 hear
>>>>>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>>>>>                                                 you're
>>>>>>>>>>>>>>>>>>>>                                                 trying
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 bring
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 it up.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 As
>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>                                                 understand
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 final
>>>>>>>>>>>>>>>>>>>>                                                 goal
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>>>                                                 it on
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 regular
>>>>>>>>>>>>>>>>>>>>                                                 basis.
>>>>>>>>>>>>>>>>>>>>                                                 So,
>>>>>>>>>>>>>>>>>>>>                                                 we
>>>>>>>>>>>>>>>>>>>>                                                 need
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 make
>>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>>                                                 properly
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 very
>>>>>>>>>>>>>>>>>>>>                                                 beginning.
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Bring
>>>>>>>>>>>>>>>>>>>>                                                 up
>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>                                                 all
>>>>>>>>>>>>>>>>>>>>                                                 features
>>>>>>>>>>>>>>>>>>>>                                                 consists
>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>                                                 4
>>>>>>>>>>>>>>>>>>>>                                                 steps:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 1.
>>>>>>>>>>>>>>>>>>>>                                                 Create
>>>>>>>>>>>>>>>>>>>>                                                 site-specific
>>>>>>>>>>>>>>>>>>>>                                                 repository
>>>>>>>>>>>>>>>>>>>>                                                 (we
>>>>>>>>>>>>>>>>>>>>                                                 call it
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 ts-rigs)
>>>>>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>>>>>                                                 contains
>>>>>>>>>>>>>>>>>>>>                                                 information
>>>>>>>>>>>>>>>>>>>>                                                 about
>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 rigs
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 other
>>>>>>>>>>>>>>>>>>>>                                                 site-specific
>>>>>>>>>>>>>>>>>>>>                                                 information
>>>>>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 where
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 send
>>>>>>>>>>>>>>>>>>>>                                                 mails,
>>>>>>>>>>>>>>>>>>>>                                                 where
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 store
>>>>>>>>>>>>>>>>>>>>                                                 logs
>>>>>>>>>>>>>>>>>>>>                                                 etc.
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 It
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 required
>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>                                                 manual
>>>>>>>>>>>>>>>>>>>>                                                 execution
>>>>>>>>>>>>>>>>>>>>                                                 as
>>>>>>>>>>>>>>>>>>>>                                                 well,
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 since
>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>                                                 rigs
>>>>>>>>>>>>>>>>>>>>                                                 description
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 essential.
>>>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 return
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 topic
>>>>>>>>>>>>>>>>>>>>                                                 below.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 2.
>>>>>>>>>>>>>>>>>>>>                                                 Setup
>>>>>>>>>>>>>>>>>>>>                                                 logs
>>>>>>>>>>>>>>>>>>>>                                                 storage
>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>                                                 automated
>>>>>>>>>>>>>>>>>>>>                                                 runs.
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Basically
>>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 a
>>>>>>>>>>>>>>>>>>>>                                                 disk
>>>>>>>>>>>>>>>>>>>>                                                 space
>>>>>>>>>>>>>>>>>>>>                                                 plus
>>>>>>>>>>>>>>>>>>>>                                                 apache2
>>>>>>>>>>>>>>>>>>>>                                                 web
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 server
>>>>>>>>>>>>>>>>>>>>                                                 with
>>>>>>>>>>>>>>>>>>>>                                                 few
>>>>>>>>>>>>>>>>>>>>                                                 CGI
>>>>>>>>>>>>>>>>>>>>                                                 scripts
>>>>>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>>>>>                                                 help
>>>>>>>>>>>>>>>>>>>>                                                 a
>>>>>>>>>>>>>>>>>>>>                                                 lot to
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 save
>>>>>>>>>>>>>>>>>>>>                                                 disk
>>>>>>>>>>>>>>>>>>>>                                                 space.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 3.
>>>>>>>>>>>>>>>>>>>>                                                 Setup
>>>>>>>>>>>>>>>>>>>>                                                 Bublik
>>>>>>>>>>>>>>>>>>>>                                                 web
>>>>>>>>>>>>>>>>>>>>                                                 application
>>>>>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>>>>>                                                 provides
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 web
>>>>>>>>>>>>>>>>>>>>                                                 interface
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 view
>>>>>>>>>>>>>>>>>>>>                                                 testing
>>>>>>>>>>>>>>>>>>>>                                                 results.
>>>>>>>>>>>>>>>>>>>>                                                 Same as
>>>>>>>>>>>>>>>>>>>>                                                 https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>>>>>                                                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>>                                                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 4.
>>>>>>>>>>>>>>>>>>>>                                                 Setup
>>>>>>>>>>>>>>>>>>>>                                                 Jenkins
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>>>                                                 tests
>>>>>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>>>>>                                                 regularly,
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 save
>>>>>>>>>>>>>>>>>>>>                                                 logs
>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>                                                 log
>>>>>>>>>>>>>>>>>>>>                                                 storage
>>>>>>>>>>>>>>>>>>>>                                                 (2)
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 import
>>>>>>>>>>>>>>>>>>>>                                                 it to
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 bublik
>>>>>>>>>>>>>>>>>>>>                                                 (3).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Last
>>>>>>>>>>>>>>>>>>>>                                                 few
>>>>>>>>>>>>>>>>>>>>                                                 month
>>>>>>>>>>>>>>>>>>>>                                                 we
>>>>>>>>>>>>>>>>>>>>                                                 spent
>>>>>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>>>>                                                 homework
>>>>>>>>>>>>>>>>>>>>                                                 to make
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>>                                                 simpler
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 bring
>>>>>>>>>>>>>>>>>>>>                                                 up
>>>>>>>>>>>>>>>>>>>>                                                 automated
>>>>>>>>>>>>>>>>>>>>                                                 execution
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 using
>>>>>>>>>>>>>>>>>>>>                                                 Jenkins
>>>>>>>>>>>>>>>>>>>>                                                 -
>>>>>>>>>>>>>>>>>>>>                                                 https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>>>>>                                                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>>                                                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Corresponding
>>>>>>>>>>>>>>>>>>>>                                                 bits
>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>                                                 dpdk-ethdev-ts
>>>>>>>>>>>>>>>>>>>>                                                 will be
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 available
>>>>>>>>>>>>>>>>>>>>                                                 tomorrow.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Let's
>>>>>>>>>>>>>>>>>>>>                                                 return
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 step
>>>>>>>>>>>>>>>>>>>>                                                 (1).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Unfortunately
>>>>>>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 no
>>>>>>>>>>>>>>>>>>>>                                                 publicly
>>>>>>>>>>>>>>>>>>>>                                                 available
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 example
>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 ts-rigs
>>>>>>>>>>>>>>>>>>>>                                                 repository
>>>>>>>>>>>>>>>>>>>>                                                 since
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 sensitive
>>>>>>>>>>>>>>>>>>>>                                                 site-specific
>>>>>>>>>>>>>>>>>>>>                                                 information
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 located
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 there.
>>>>>>>>>>>>>>>>>>>>                                                 But
>>>>>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>>>>>                                                 ready
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 help
>>>>>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 create
>>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>                                                 UNH.
>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>                                                 see
>>>>>>>>>>>>>>>>>>>>                                                 two
>>>>>>>>>>>>>>>>>>>>                                                 options
>>>>>>>>>>>>>>>>>>>>                                                 here:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 (A)
>>>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>>>                                                 ask
>>>>>>>>>>>>>>>>>>>>                                                 questions
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 based
>>>>>>>>>>>>>>>>>>>>                                                 on your
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 answers
>>>>>>>>>>>>>>>>>>>>                                                 will
>>>>>>>>>>>>>>>>>>>>                                                 create
>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>                                                 first
>>>>>>>>>>>>>>>>>>>>                                                 draft
>>>>>>>>>>>>>>>>>>>>                                                 with my
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 comments.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 (B)
>>>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>>>                                                 make
>>>>>>>>>>>>>>>>>>>>                                                 a
>>>>>>>>>>>>>>>>>>>>                                                 template/example
>>>>>>>>>>>>>>>>>>>>                                                 ts-rigs
>>>>>>>>>>>>>>>>>>>>                                                 repo,
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 publish
>>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 you'll
>>>>>>>>>>>>>>>>>>>>                                                 create
>>>>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>>>>                                                 ts-rigs
>>>>>>>>>>>>>>>>>>>>                                                 based
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 on it.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Of
>>>>>>>>>>>>>>>>>>>>                                                 course,
>>>>>>>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>>>>>>>                                                 help
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 debug
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 finally
>>>>>>>>>>>>>>>>>>>>                                                 bring
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 it
>>>>>>>>>>>>>>>>>>>>                                                 up
>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>>>>>>>>                                                 case.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 (A)
>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>                                                 a
>>>>>>>>>>>>>>>>>>>>                                                 bit
>>>>>>>>>>>>>>>>>>>>                                                 simpler
>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>                                                 me
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 you,
>>>>>>>>>>>>>>>>>>>>                                                 but
>>>>>>>>>>>>>>>>>>>>                                                 (B) is
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 a
>>>>>>>>>>>>>>>>>>>>                                                 bit
>>>>>>>>>>>>>>>>>>>>                                                 more
>>>>>>>>>>>>>>>>>>>>                                                 generic
>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>                                                 will
>>>>>>>>>>>>>>>>>>>>                                                 help
>>>>>>>>>>>>>>>>>>>>                                                 other
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 potential
>>>>>>>>>>>>>>>>>>>>                                                 users
>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>                                                 bring
>>>>>>>>>>>>>>>>>>>>                                                 it up.
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 We
>>>>>>>>>>>>>>>>>>>>                                                 can
>>>>>>>>>>>>>>>>>>>>                                                 combine
>>>>>>>>>>>>>>>>>>>>                                                 (A)+(B).
>>>>>>>>>>>>>>>>>>>>                                                 I.e.
>>>>>>>>>>>>>>>>>>>>                                                 start
>>>>>>>>>>>>>>>>>>>>                                                 from
>>>>>>>>>>>>>>>>>>>>                                                 (A).
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 What
>>>>>>>>>>>>>>>>>>>>                                                 do
>>>>>>>>>>>>>>>>>>>>                                                 you
>>>>>>>>>>>>>>>>>>>>                                                 think?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>>>>                                                 8/17/23
>>>>>>>>>>>>>>>>>>>>                                                 15:18,
>>>>>>>>>>>>>>>>>>>>                                                 Konstantin
>>>>>>>>>>>>>>>>>>>>                                                 Ushakov
>>>>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>>>>                                                 Greetings
>>>>>>>>>>>>>>>>>>>>>                                                 Adam,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                 Thanks
>>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>>                                                 contacting
>>>>>>>>>>>>>>>>>>>>>                                                 us.
>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>                                                 copy
>>>>>>>>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>>>>>>>>                                                 who
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>>                                                 happy
>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>                                                 help
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                 Konstantin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>>>>>>>>>                                                 16
>>>>>>>>>>>>>>>>>>>>>>                                                 Aug
>>>>>>>>>>>>>>>>>>>>>>                                                 2023,
>>>>>>>>>>>>>>>>>>>>>>                                                 at
>>>>>>>>>>>>>>>>>>>>>>                                                 21:50,
>>>>>>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>>>>>>                                                 Hassick
>>>>>>>>>>>>>>>>>>>>>>                                                 <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 Greetings
>>>>>>>>>>>>>>>>>>>>>>                                                 Konstantin,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>>                                                 am
>>>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 process
>>>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>>>                                                 setting
>>>>>>>>>>>>>>>>>>>>>>                                                 up
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 DPDK
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 Poll
>>>>>>>>>>>>>>>>>>>>>>                                                 Mode
>>>>>>>>>>>>>>>>>>>>>>                                                 Driver
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 suite
>>>>>>>>>>>>>>>>>>>>>>                                                 as
>>>>>>>>>>>>>>>>>>>>>>                                                 an
>>>>>>>>>>>>>>>>>>>>>>                                                 addition
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>>>>>>>>                                                 testing
>>>>>>>>>>>>>>>>>>>>>>                                                 coverage
>>>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>>>                                                 DPDK
>>>>>>>>>>>>>>>>>>>>>>                                                 at
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>>>>>>                                                 lab.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>>                                                 have
>>>>>>>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>>>>>>>                                                 questions
>>>>>>>>>>>>>>>>>>>>>>                                                 about
>>>>>>>>>>>>>>>>>>>>>>                                                 how
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 set
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 suite
>>>>>>>>>>>>>>>>>>>>>>                                                 arguments.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>>                                                 have
>>>>>>>>>>>>>>>>>>>>>>                                                 been
>>>>>>>>>>>>>>>>>>>>>>                                                 able
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 configure
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 Test
>>>>>>>>>>>>>>>>>>>>>>                                                 Engine
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 connect
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 hosts
>>>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 testbed.
>>>>>>>>>>>>>>>>>>>>>>                                                 The
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 RCF,
>>>>>>>>>>>>>>>>>>>>>>                                                 Configurator,
>>>>>>>>>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>>>>>>>>>                                                 Tester
>>>>>>>>>>>>>>>>>>>>>>                                                 all
>>>>>>>>>>>>>>>>>>>>>>                                                 begin
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 run,
>>>>>>>>>>>>>>>>>>>>>>                                                 however
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 prelude
>>>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 suite
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 fails
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 run.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                 https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>>>>>                                                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>>                                                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 The
>>>>>>>>>>>>>>>>>>>>>>                                                 documentation
>>>>>>>>>>>>>>>>>>>>>>                                                 mentions
>>>>>>>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 several
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 parameters
>>>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 suite,
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 IUT
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 link
>>>>>>>>>>>>>>>>>>>>>>                                                 MAC,
>>>>>>>>>>>>>>>>>>>>>>                                                 etc.
>>>>>>>>>>>>>>>>>>>>>>                                                 These
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 seem
>>>>>>>>>>>>>>>>>>>>>>                                                 like
>>>>>>>>>>>>>>>>>>>>>>                                                 they
>>>>>>>>>>>>>>>>>>>>>>                                                 would
>>>>>>>>>>>>>>>>>>>>>>                                                 need
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>>>                                                 set
>>>>>>>>>>>>>>>>>>>>>>                                                 somewhere
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 run
>>>>>>>>>>>>>>>>>>>>>>                                                 many
>>>>>>>>>>>>>>>>>>>>>>                                                 of
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 tests.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>>                                                 see
>>>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 Test
>>>>>>>>>>>>>>>>>>>>>>                                                 Engine
>>>>>>>>>>>>>>>>>>>>>>                                                 documentation,
>>>>>>>>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>>>>>>>>                                                 instructions
>>>>>>>>>>>>>>>>>>>>>>                                                 on
>>>>>>>>>>>>>>>>>>>>>>                                                 how
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 create
>>>>>>>>>>>>>>>>>>>>>>                                                 new
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 parameters
>>>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 suites
>>>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 Tester
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 configuration,
>>>>>>>>>>>>>>>>>>>>>>                                                 but
>>>>>>>>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>>>>>>>>                                                 nothing
>>>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 user
>>>>>>>>>>>>>>>>>>>>>>                                                 guide
>>>>>>>>>>>>>>>>>>>>>>                                                 or
>>>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 Tester
>>>>>>>>>>>>>>>>>>>>>>                                                 guide
>>>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>>>                                                 how
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 set
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 arguments
>>>>>>>>>>>>>>>>>>>>>>                                                 for
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 parameters
>>>>>>>>>>>>>>>>>>>>>>                                                 when
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>>>>>>>>                                                 suite
>>>>>>>>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>>                                                 can
>>>>>>>>>>>>>>>>>>>>>>                                                 find.
>>>>>>>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 not
>>>>>>>>>>>>>>>>>>>>>>                                                 sure
>>>>>>>>>>>>>>>>>>>>>>                                                 if
>>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>>                                                 need
>>>>>>>>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>>>>>>>>                                                 write
>>>>>>>>>>>>>>>>>>>>>>                                                 my
>>>>>>>>>>>>>>>>>>>>>>                                                 own
>>>>>>>>>>>>>>>>>>>>>>                                                 Tester
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 config,
>>>>>>>>>>>>>>>>>>>>>>                                                 or
>>>>>>>>>>>>>>>>>>>>>>                                                 if
>>>>>>>>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>>>>>>>>                                                 should
>>>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>>>                                                 setting
>>>>>>>>>>>>>>>>>>>>>>                                                 these
>>>>>>>>>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 some
>>>>>>>>>>>>>>>>>>>>>>                                                 other
>>>>>>>>>>>>>>>>>>>>>>                                                 way.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 How
>>>>>>>>>>>>>>>>>>>>>>                                                 should
>>>>>>>>>>>>>>>>>>>>>>                                                 these
>>>>>>>>>>>>>>>>>>>>>>                                                 values
>>>>>>>>>>>>>>>>>>>>>>                                                 be
>>>>>>>>>>>>>>>>>>>>>>                                                 set?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 I'm
>>>>>>>>>>>>>>>>>>>>>>                                                 also
>>>>>>>>>>>>>>>>>>>>>>                                                 not
>>>>>>>>>>>>>>>>>>>>>>                                                 sure
>>>>>>>>>>>>>>>>>>>>>>                                                 what
>>>>>>>>>>>>>>>>>>>>>>                                                 environment
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 variables/arguments
>>>>>>>>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>>>>>>>>                                                 strictly
>>>>>>>>>>>>>>>>>>>>>>                                                 necessary
>>>>>>>>>>>>>>>>>>>>>>                                                 or
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 which
>>>>>>>>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>>>>>>>>                                                 optional.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 Regards,
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>>>>>>>                                                 Hassick*
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>>>>>>>                                                 Developer
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>>>>>>                                                 InterOperability
>>>>>>>>>>>>>>>>>>>>>>                                                 Lab
>>>>>>>>>>>>>>>>>>>>>>                                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>                                                 iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>>                                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>>>>>>>                                                 (603)
>>>>>>>>>>>>>>>>>>>>>>                                                 475-8248
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>>>>>                                                 Hassick*
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>>>>>                                                 Developer
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>>>>                                                 InterOperability
>>>>>>>>>>>>>>>>>>>>                                                 Lab
>>>>>>>>>>>>>>>>>>>>                                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                                 iol.unh.edu
>>>>>>>>>>>>>>>>>>>>                                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>>>>>                                                 (603)
>>>>>>>>>>>>>>>>>>>>                                                 475-8248
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>>>                                                 Hassick*
>>>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>>>                                                 Developer
>>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>>                                                 InterOperability
>>>>>>>>>>>>>>>>>>                                                 Lab
>>>>>>>>>>>>>>>>>>                                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                 iol.unh.edu
>>>>>>>>>>>>>>>>>>                                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>>>                                                 (603)
>>>>>>>>>>>>>>>>>>                                                 475-8248
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>>>                                                 Hassick*
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>>>                                                 Developer
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>>                                                 InterOperability
>>>>>>>>>>>>>>>>>>                                                 Lab
>>>>>>>>>>>>>>>>>>                                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                 iol.unh.edu
>>>>>>>>>>>>>>>>>>                                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>>>                                                 (603)
>>>>>>>>>>>>>>>>>>                                                 475-8248
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>>                                                 Hassick*
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>>                                                 Developer
>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>>                                                 InterOperability
>>>>>>>>>>>>>>>>>                                                 Lab
>>>>>>>>>>>>>>>>>                                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                 iol.unh.edu
>>>>>>>>>>>>>>>>>                                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>                                                 (603)
>>>>>>>>>>>>>>>>>                                                 475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                 -- 
>>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>>                                                 Hassick*
>>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>>                                                 Developer
>>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>>                                                 InterOperability
>>>>>>>>>>>>>>>>                                                 Lab
>>>>>>>>>>>>>>>>                                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>                                                 iol.unh.edu
>>>>>>>>>>>>>>>>                                                 <http://iol.unh.edu>
>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>                                                 +1
>>>>>>>>>>>>>>>>                                                 (603)
>>>>>>>>>>>>>>>>                                                 475-8248
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>>                                             Senior Developer
>>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                             +1 (603) 475-8248
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                         --
>>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>>                                         UNH
>>>>>>>>>>>>                                         InterOperability Lab
>>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     --
>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>                                     ahassick@iol.unh.edu
>>>>>>>>>>                                     iol.unh.edu
>>>>>>>>>>                                     <https://www.iol.unh.edu/>
>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                             --
>>>>>>>>                             *Adam Hassick*
>>>>>>>>                             Senior Developer
>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>                             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                             +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                         --
>>>>>>>                         *Adam Hassick*
>>>>>>>                         Senior Developer
>>>>>>>                         UNH InterOperability Lab
>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                         +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                     --
>>>>>>>                     *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 242680 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1.1: Type: text/plain, Size: 61031 bytes --]

Hi Andrew,

I've attached the public key we'd like to use.

Thanks,
Adam

On Thu, Oct 26, 2023 at 8:19 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> On 10/25/23 23:27, Adam Hassick wrote:
>
> Hi Andrew,
>
> Sorry about the two week radio silence, we're still trying to sort out
> logistics for deployment on our end.
>
> > I've created setup on ts-factory.io which allows to publish logs. Let
> me know if you'd like to try it and I'll provide credentials, script and
> short instruction.
>
> We're interested in publishing some test logs to the ts-factory Bublik
> instance in the meantime.
>
>
> Please, send me SSH public key which you'd like to use to upload logs.
> I'll provide helper script and instructions to do it.
>
> Andrew.
>
>
> Thanks,
> Adam
>
> On Mon, Oct 23, 2023 at 7:11 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> > Now that our test results are in line with yours, we can begin looking
>> into setting up the production environment.
>>
>> Please, let me know if you need any help with it or waiting for an input
>> from me.
>>
>> Regards,
>> Andrew.
>>
>> On 10/10/23 17:09, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> Thank you for taking a look at our log. Netplan was attempting to run
>> DHCP on our test links, and additionally I discovered that the NIC firmware
>> was transmitting LLDP packets, causing tests to fail in the same way. Now
>> that these problems have been fixed, our pass rate on the XL710 is
>> approximately 91%. Now that our test results are in line with yours, we can
>> begin looking into setting up the production environment.
>>
>> First, is it possible to run the test agent on ARM hosts? Our ARM
>> testbeds have the best topology for running this test suite, with separate
>> tester and DUT servers.
>>
>> We are testing this test suite on two x86 development servers using the
>> test suite's recommended server topology. In contrast, our existing x86
>> production testbeds which run DTS have a single server topology. This
>> single server has both the tester NIC and the device under test NIC
>> installed, with NUMA node separation between TRex and DPDK. We're going to
>> test running the two test agent processes on the single-server testbeds if
>> we cannot run this on ARM. Is there any reason you can think of that would
>> prevent this setup from working?
>>
>> Once we figure out where this can live in production, then we will begin
>> setting up log storage, Jenkins integration, and Bublik.
>>
>> Thanks,
>> Adam
>>
>> On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> > Do these default to vfio-pci?
>>>
>>> Yes, vfio-pci is the default.
>>> However, it does not work in the case of Mellanox which uses bifurcated
>>> driver. It should mlx5_core for Mellanox NICs.
>>>
>>> > Here is the text log from a run on our Intel XL710 NICs, with the
>>> expected result profile set to the X710.
>>>
>>> It is hard to analyze all tests using text logs, but I definitely see
>>> one common problem. Tests receive unexpected packets and fail because of it.
>>> Tests are written very strict from this point of view and it brought
>>> fruits in the past when HW had bugs.
>>> Are DUT and tester connected back-to-back on tested interfaces or via
>>> switch?
>>> If via switch, is it possible to isolate it from everything else?
>>> If back-to-back, it could be some embedded SW/FW which regenerates these
>>> packet.
>>> I definitely see unexpected DHCP packets.
>>>
>>> > We haven't set up the Jenkins integration yet, however if this is
>>> required to import the logs then we will prioritize that.
>>>
>>> Unfortunately manual runs do not generate all artifacts required to
>>> import logs. However, we have almost solved it right now. Hopefully we'll
>>> finalize it in a day or two. I'll let you know when these changes are
>>> available.
>>>
>>> Regards,
>>> Andrew.
>>>
>>> On 10/4/23 16:48, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> Ok, that makes sense. I don't see TE_ENV_H1/H2_DPDK_DRIVER set anywhere
>>> in the default configurations for the Intel X710. Do these default to
>>> vfio-pci?
>>> We have IOMMU enabled on our development testbed, and should be able to
>>> bind vfio-pci.
>>> Here is the text log from a run on our Intel XL710 NICs, with the
>>> expected result profile set to the X710. We haven't set up the Jenkins
>>> integration yet, however if this is required to import the logs then we
>>> will prioritize that.
>>>  log.txt.tar.gz
>>> <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> On 9/18/23 17:44, Adam Hassick wrote:
>>>>
>>>> Hi Andrew and Konstantin,
>>>>
>>>> Thank you for adding the tester-dial feature, this opens up the
>>>> possibility for us to do CI integrated testing in the future.
>>>>
>>>> Our Mellanox pass rate is similar to yours (about ~2400 passing, ~4400
>>>> failing), however our Intel pass rates are far worse.
>>>> I will try running tests on the XL710 with the trc-tags argument set
>>>> and see if it improves the pass rate.
>>>> Another thing I noticed in the results you uploaded is that the results
>>>> are tagged with vfio-pci and not i40e.
>>>> Though in the environment dump, the driver on the test machine and the
>>>> DUT are set to use the i40e driver. Is this important at all?
>>>>
>>>>
>>>> I think it is a misunderstanding here. There are two kinds of driver in
>>>> configuration: net driver and so-called DPDK driver.
>>>> Net driver is a Linux kernel network device driver used on Tester side.
>>>> DPDK driver is a Linux kernel driver to bind device to to use it with
>>>> DPDK. So, it is NOT a driver inside DPDK (drivers/net/*).
>>>> In the case of bifurcated driver (like mlx5_core) it is the same in
>>>> both cases.
>>>> In non-bifurcated case DPDK driver is some UIO driver(vfio-pci,
>>>> uio-pci-generic or igb_uio).
>>>> Some expectations depend on used UIO. For example, uio-pci-generic do
>>>> not support many interrupts (used by usecases/rx_intr test cases).
>>>> That's why we care corresponding TRC tag.
>>>>
>>>> TE_ENV_*_DPDK_DRIVER variables should be vfio-pc  in 710's Intel case.
>>>> Or uio-pci-generic if IOMMU is turned off on corresponding machines and
>>>> Linux distro does not support VFIO no IOMMU mode.
>>>>
>>>> Andrew.
>>>>
>>>> There isn't anything preventing us from pushing our results up to the
>>>> existing Bublik instance running at ts-factory.io that I can think of
>>>> at the moment.
>>>> We will have to work out how to submit our results to your Bublik
>>>> instance in a controlled and secure manner in that case.
>>>> As far as I know we won't need access controls for the results
>>>> themselves. I'll discuss this with Patrick and will let you know once we
>>>> confirm that it's fine.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> should we always auto-assign the tags or you don’t do it since it
>>>>> slows down (by some seconds) the TE startup?
>>>>>
>>>>>
>>>>> Tags are auto-assigned, but I guess it differs in Adam's case since
>>>>> NIC is a bit different. Below test will help to understand if it is the
>>>>> root cause of very different expectations. If pass rate will be close to
>>>>> mine, I'll simply update TRC database to share expectations for mine NIC
>>>>> and NIC used by Adam.
>>>>>
>>>>> Hi Adam,
>>>>>
>>>>> I think I second the question from Andrew - happy to help you with the
>>>>> triage so that we get to the same baseline. Do you have a good way for us
>>>>> to share the logs? I.e. say upload to ts-factory if we add strict
>>>>> permissions system so it’s not publishing or any other way.
>>>>>
>>>>> Thanks,
>>>>> Konstantin
>>>>>
>>>>> On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>>>
>>>>> Hi Adam,
>>>>>
>>>>> I've uploaded fresh testing results to ts-factory.io [1] to be on the
>>>>> same page.
>>>>>
>>>>> I think I know why your and mine results on Intel 710 series NICs
>>>>> differ so much. Testing results expectations database
>>>>> (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags.  I.e.
>>>>> expectations depends on TRC tags discovered by helper scripts when testing
>>>>> is started. These tags identify various aspects of what is tested. Ideally
>>>>> expectations should be written in terms of root cause of the expected
>>>>> behaviour. If it is a driver expectations, driver tag should be used. If it
>>>>> is HW limitation, tags with PCI IDs should be used. However, it is not
>>>>> always easy to classify it correctly if you're not involved in driver
>>>>> development. So, in order case expectations for 710's Intel are filled in
>>>>> in terms of PCI IDs. I guess PCI ID differ in your case and that's why
>>>>> expectations filled in for my NIC do not apply to your runs.
>>>>>
>>>>> Just try to add the following option when you run on your 710's Intel
>>>>> in order to mimic mine and see if it helps to achieve better pass rate.
>>>>> --trc-tag=pci-8086-1572
>>>>>
>>>>> BTW, fresh TE tag v1.21.0 has improved algorithm to choose tests for
>>>>> --tester-dial option. It should have better coverage now.
>>>>>
>>>>> Andrew.
>>>>>
>>>>> [1]
>>>>> https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>>>
>>>>> On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>>
>>>>> Hi Adam,
>>>>>
>>>>> I've pushed new TE tag v1.20.0 which supported a new command-line
>>>>> option --tester-dial=NUM where NUM is from 0 to 100. it allows to choose
>>>>> percentage of tests to run. If you want stable set, you should pass
>>>>> --tester-random-seed=0 (or other integer). It is the first sketch and we
>>>>> have plans to improve it, but feedback would be welcome.
>>>>>
>>>>> > Is it needed on the tester?
>>>>>
>>>>> It is hard to say if it is strictly required for simple tests.
>>>>> However, it is better to update Tester as well, since performance tests run
>>>>> DPDK on Tester as well.
>>>>>
>>>>> > Are there any other manual setup steps for these devices that I
>>>>> might be missing?
>>>>>
>>>>> I don't remember anything else.
>>>>>
>>>>> I think it is better to get down to details and take a look at logs.
>>>>> I'm ready to help with it and explain what's happening there. May be it
>>>>> will help to understand if it is a problem with setup/configuration.
>>>>>
>>>>> Text logs are not very convenient. Ideally logs should be imported to
>>>>> bublik, however, manual runs do not provide all required artifacts right
>>>>> now (Jenkins jobs generate all required artifacts).
>>>>> Other option is 'tmp_raw_log' file (should be packed to make it
>>>>> smaller) which could be converted to various log formats.
>>>>> Would it be OK for you if I import your logs to bublik at
>>>>> ts-factory.io? Or is it a problem that it is publicly available?
>>>>> Would it help if we add authentication and access control there?
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 9/8/23 17:57, Adam Hassick wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> I have a couple questions about needed setup of the NICs for the
>>>>> ethdev test suite.
>>>>>
>>>>> Our MCX5s and XL710s are failing the checkup tests. The pass rate
>>>>> appears to be much worse on the XL710s (40 of 73 tests failed, 3 passed
>>>>> unexpectedly).
>>>>>
>>>>> For the XL710s, I've updated the driver and NVM versions to match the
>>>>> minimum supported versions in the compatibility matrix found on the DPDK
>>>>> documentation. This did not change the failure rate much.
>>>>> For the MCX5s, I've installed the latest LTS version of the OFED
>>>>> bifurcated driver on the DUT. Is it needed on the tester?
>>>>>
>>>>> Are there any other manual setup steps for these devices that I might
>>>>> be missing?
>>>>>
>>>>> Thanks,
>>>>> Adam
>>>>>
>>>>> On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <ahassick@iol.unh.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> Yes, I copied the X710 configs to set up XL710 configs. I changed the
>>>>>> environment variable names from the X710 suffix to XL710 suffix in the
>>>>>> script, and forgot to change them in the corresponding environment file.
>>>>>> That fixed the issue.
>>>>>>
>>>>>> I got the checkup tests working on the XL710 now. Most of them are
>>>>>> failing, which leads me to believe this is an issue with our testbed. Based
>>>>>> on the DPDK documentation for i40e, the firmware and driver versions are
>>>>>> much older than what DPDK 22.11 LTS and main prefer, so I'll try updating
>>>>>> those.
>>>>>>
>>>>>> For now I'm working on getting the XL710 checkup tests passing, and
>>>>>> will pick up getting the E810 configured properly next. I'll let you know
>>>>>> if I run into any more issues in relation to the test engine.
>>>>>>
>>>>>> Thanks,
>>>>>> Adam
>>>>>>
>>>>>> On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <
>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>> Hi Adam,
>>>>>>>
>>>>>>> On 9/5/23 18:01, Adam Hassick wrote:
>>>>>>>
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> The compilation warning issue is now resolved. Again, thank you guys
>>>>>>> for fixing this for us. I can run the tests on the Mellanox CX5s again,
>>>>>>> however I'm running into a couple new issues with running the prologues on
>>>>>>> the Intel cards.
>>>>>>>
>>>>>>> When running testing on the Intel XL710s, I see this error appear in
>>>>>>> the log:
>>>>>>>
>>>>>>> ERROR  prologue  Environment LIB  14:16:13.650
>>>>>>>> Too few networks in available configuration (0) in comparison with
>>>>>>>> required (1)
>>>>>>>>
>>>>>>>
>>>>>>> This seems like a trivial configuration error, perhaps this is
>>>>>>> something I need to set up in ts-rigs. I briefly searched through the
>>>>>>> examples there and didn't see any mention of how to set up a network.
>>>>>>> I will attach this log just in case you need more information.
>>>>>>>
>>>>>>>
>>>>>>> Unfortunately logs are insufficient to understand it. I've pushed
>>>>>>> new tag to TE v1.19.0 which add log message with TE_* environment variables.
>>>>>>> Most likely something is wrong with variables which are used as
>>>>>>> conditions when available networks are defined in
>>>>>>> ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>>> TE_PCI_INSTANCE_IUT_TST1
>>>>>>> TE_PCI_INSTANCE_IUT_TST1a
>>>>>>> TE_PCI_INSTANCE_TST1a_IUT
>>>>>>> TE_PCI_INSTANCE_TST1_IUT
>>>>>>> My guess it that you change naming a bit, but script like
>>>>>>> ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.
>>>>>>>
>>>>>>> There is a different error when running on the Intel E810s. It
>>>>>>> appears to me like it starts DPDK, does some configuration inside DPDK and
>>>>>>> on the device, and then fails to bring the device back up. Since this error
>>>>>>> seems very non-trivial, I will also attach this log.
>>>>>>>
>>>>>>>
>>>>>>> This one is a bit simpler. Few lines after the first ERROR in log I
>>>>>>> see the following:
>>>>>>> WARN  RCF  DPDK  13:06:00.144
>>>>>>> ice_program_hw_rx_queue(): currently package doesn't support RXDID
>>>>>>> (22)
>>>>>>> ice_rx_queue_start(): fail to program RX queue 0
>>>>>>> ice_dev_start(): fail to start Rx queue 0
>>>>>>> Device with port_id=0 already stopped
>>>>>>>
>>>>>>> It is stdout/stderr from test agent which runs DPDK. Same logs in
>>>>>>> plain format are available in ta.DPDK file.
>>>>>>> I'm not an expert here, but I vaguely remember that E810 requires
>>>>>>> correct firmware and DDP to be loaded.
>>>>>>> There is some information in dpdk/doc/guides/nics/ice.rst.
>>>>>>>
>>>>>>> You can try to add --dev-args=safe-mode-support=1 command-line
>>>>>>> option described there.
>>>>>>>
>>>>>>> Hope it helps,
>>>>>>> Andrew.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Adam
>>>>>>>
>>>>>>> On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>
>>>>>>>> Hi Adam,
>>>>>>>>
>>>>>>>> On 8/31/23 22:38, Adam Hassick wrote:
>>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> I have one additional question as well: Does the test engine
>>>>>>>> support running tests on two ARMv8 test agents?
>>>>>>>>
>>>>>>>> 1. We'll sort out warnings this week. Thanks for heads up.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Great. Let me know when that's fixed.
>>>>>>>>
>>>>>>>>
>>>>>>>> Done. We also fixed a number of warnings in TE.
>>>>>>>> Also we fixed root test package name to be consistent with the
>>>>>>>> repository name.
>>>>>>>>
>>>>>>>> Support for old LTS branches was dropped some time ago, but in the
>>>>>>>>> future it is definitely possible to keep it for new LTS branches. I think
>>>>>>>>> 22.11 is supported, but I'm not sure about older LTS releases.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Good to know.
>>>>>>>>
>>>>>>>>
>>>>>>>>> 2. You can add command-line option --sanity to run tests marked
>>>>>>>>> with TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and
>>>>>>>>> grep TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked).
>>>>>>>>> Yes, there is a space for terminology improvement here. We'll do it.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Done. Now it is called --checkup.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also it takes a lot of time because of failures and tests which
>>>>>>>>> wait for some timeout.
>>>>>>>>>
>>>>>>>>
>>>>>>>> That makes sense to me. We'll use the time to complete tests on
>>>>>>>> virtio or the Intel devices as a reference for how long the tests really
>>>>>>>> take to complete.
>>>>>>>> We will explore the possibility of periodically running the sanity
>>>>>>>> tests for patches.
>>>>>>>>
>>>>>>>>
>>>>>>>> I'll double-check and let you know how long entire TS runs on Intel
>>>>>>>> X710, E810, Mellanox CX5 and virtio net. Just to ensure that time observed
>>>>>>>> in your case looks the same.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> The test harness can provide coverage reports based on gcov, but
>>>>>>>>> I'm not sure what you mean by a "dial" to control test coverage. Provided
>>>>>>>>> reports are rather for human to analyze.
>>>>>>>>>
>>>>>>>>
>>>>>>>> The general idea is to have some kind of parameter on the test
>>>>>>>> suite, which could be an integer ranging from zero to ten, that controls
>>>>>>>> how many tests are run based on how important the test is.
>>>>>>>>
>>>>>>>> Similar to how some command line interfaces provide a verbosity
>>>>>>>> level parameter (some number of "-v" arguments) to control the importance
>>>>>>>> of the information in the log.
>>>>>>>> The verbosity level zero only prints very important log messages,
>>>>>>>> while ten prints everything.
>>>>>>>>
>>>>>>>> In much the same manner as above, this "dial" parameter controls
>>>>>>>> what tests are run and with what parameters based on how important those
>>>>>>>> tests and test parameter combinations are.
>>>>>>>> Coverage Level zero tells the suite to run a very basic set of
>>>>>>>> important tests, with minimal parameterization. This mode would take only
>>>>>>>> ~5-10 minutes to run.
>>>>>>>> In contrast, Coverage Level ten includes all the edge cases, every
>>>>>>>> combination of test parameters, everything the test suite can do, which
>>>>>>>> takes the normal several hours to run.
>>>>>>>> The values 1 - 9 are between those two extremes, allowing the user
>>>>>>>> to get a gradient of test coverage in the results and to limit the running
>>>>>>>> time.
>>>>>>>>
>>>>>>>> Then we could, for example, run the "run.sh" with a level of 2 or 3
>>>>>>>> for incoming patches that need quick results, and with a level of 10 for
>>>>>>>> the less often run periodic tests performed on main or LTS branches.
>>>>>>>>
>>>>>>>>
>>>>>>>> Understood now. Thanks a lot for the idea. We'll discuss it and
>>>>>>>> come back.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>>>>>>>>> testing results. Unfortunately it is time consuming to fill in expectations
>>>>>>>>> database since it is necessary to analyze testing results and classify if
>>>>>>>>> it is a bug or just acceptable behaviour aspect.
>>>>>>>>>
>>>>>>>>> Bublik allows to compare results of two runs. It is useful for
>>>>>>>>> human, but still not good for automation.
>>>>>>>>>
>>>>>>>>> I have local patch for mlx5 driver which reports Tx ring size
>>>>>>>>> maximum. It makes pass rate higher. It is a problem for test harness that
>>>>>>>>> mlx5 does not report limits right now.
>>>>>>>>>
>>>>>>>>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>>>>>>>>> virtio net is 99% right now and could be done 100% easily (just one thing
>>>>>>>>> to fix in expectations).
>>>>>>>>>
>>>>>>>>> I think logs storage setup is essential for logs analysis. Of
>>>>>>>>> course, you can request HTML logs when you run tests (--log-html=html) or
>>>>>>>>> generate after run using dpdk-ethdev-ts/scripts/html-log.sh and open
>>>>>>>>> index.html in a browser, but logs storage makes it more convenient.
>>>>>>>>>
>>>>>>>>
>>>>>>>> We are interested in setting up Bublik, potentially as an
>>>>>>>> externally-facing component, once we have our process of running the test
>>>>>>>> suite stabilized.
>>>>>>>> Once we are able to run the test suite again, I'll see what the
>>>>>>>> pass rate is on our other hardware.
>>>>>>>> Good to know that it isn't an issue with our dev testbed causing
>>>>>>>> the high fail rate.
>>>>>>>>
>>>>>>>> For Intel hardware, we have an XL710 and an Intel E810-C in our
>>>>>>>> development testbed. Although they are slightly different devices, ideally
>>>>>>>> the pass rate will be identical or similar. I have yet to set up a VM pair
>>>>>>>> for virtio, but we will soon.
>>>>>>>>
>>>>>>>> Latest version of test-environment has examples of our CGI scripts
>>>>>>>>> which we use for log storage (see tools/log_server/README.md).
>>>>>>>>>
>>>>>>>>> Also all bits for Jenkins setup are available. See
>>>>>>>>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>>>>>>>>> ts-rigs-sample.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Jenkins integration, setting up production rig configurations, and
>>>>>>>> permanent log storage will be our next steps once I am able to run the
>>>>>>>> tests again.
>>>>>>>> Unless there is an easy way to have meson not pass "-Werror" into
>>>>>>>> GCC. Then I would be able to run the test suite.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hopefully it is resolved now.
>>>>>>>>
>>>>>>>> I thought a bit more about your usecase for Jenkins. I'm not 100%
>>>>>>>> sure that existing pipelines are convenient for your usecase.
>>>>>>>> Fill free to ask questions when you are on it.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Adam
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 8/29/23 17:02, Adam Hassick wrote:
>>>>>>>>>
>>>>>>>>> Hi Andrew,
>>>>>>>>>
>>>>>>>>> That fix seems to have resolved the issue, thanks for the quick
>>>>>>>>> turnaround time on that patch.
>>>>>>>>> Now that we have the RCF timeout issue resolved, there are a few
>>>>>>>>> other questions and issues that we have about the tests themselves.
>>>>>>>>>
>>>>>>>>> 1. The test suite fails to build with a couple warnings.
>>>>>>>>>
>>>>>>>>> Below is the stderr log from compilation:
>>>>>>>>>
>>>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>>>>>>>>> literal and no format arguments [-Werror=format-security]
>>>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>>>> |     ^~
>>>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>>>> ninja: build stopped: subcommand failed.
>>>>>>>>>> ninja: Entering directory `.'
>>>>>>>>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>>>>>>>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>>>>>>>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>>>>>>>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c: In function
>>>>>>>>>> ‘test_create_traffic_generator_params’:
>>>>>>>>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>>>>>>>>> literal and no format arguments [-Werror=format-security]
>>>>>>>>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>>>>>>>>> |     ^~
>>>>>>>>>> cc1: all warnings being treated as errors
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This error wasn't occurring last week, which was the last time I
>>>>>>>>> ran the tests.
>>>>>>>>> The TE host and the DUT have GCC v9.4.0 installed, and the tester
>>>>>>>>> has GCC v11.4.0 installed, if this information is helpful.
>>>>>>>>>
>>>>>>>>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>>>>>>>> collectively take around 9 hours. Is it possible, and would it make sense,
>>>>>>>>> to lower the test coverage and have the test suite run faster?
>>>>>>>>>
>>>>>>>>> For some context, we run immediate testing on incoming patches for
>>>>>>>>> DPDK main and development branches, as well as periodic test runs on the
>>>>>>>>> main, stable, and LTS branches.
>>>>>>>>> For us to consider including this test suite as part of our
>>>>>>>>> immediate testing on patches, we would have to reduce the test coverage to
>>>>>>>>> the most important tests.
>>>>>>>>> This is primarily to reduce the testing time to, for example, less
>>>>>>>>> than 30 minutes. Testing on patches can't take too long because the lab can
>>>>>>>>> receive numerous patches each day, which each require individual testing
>>>>>>>>> runs.
>>>>>>>>>
>>>>>>>>> At what frequency we run these tests, and on what, still needs to
>>>>>>>>> be discussed with the DPDK community, but it would be nice to know if the
>>>>>>>>> test suite had a "dial" to control the testing coverage.
>>>>>>>>>
>>>>>>>>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>>>>>>>> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose these
>>>>>>>>> test failures?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Adam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Adam,
>>>>>>>>>>
>>>>>>>>>> I've pushed the fix in main branch and a new tag v1.18.1. It
>>>>>>>>>> should solve the problem with IPv6 address from DNS.
>>>>>>>>>>
>>>>>>>>>> Andrew.
>>>>>>>>>>
>>>>>>>>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Adam,
>>>>>>>>>>
>>>>>>>>>> > Does the test engine prefer to use IPv6 over IPv4 for
>>>>>>>>>> initiating the RCF connection to the test bed hosts? And if so, is there a
>>>>>>>>>> way to force it to use IPv4?
>>>>>>>>>>
>>>>>>>>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in
>>>>>>>>>> your case, I guess it is the root cause of the problem.
>>>>>>>>>> Of course, it is TE problem since I see really weird code in
>>>>>>>>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>>>>>>>>
>>>>>>>>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>>>>>>>> ts-factory/test-environment repository. Please, try.
>>>>>>>>>>
>>>>>>>>>> It is late night fix with minimal testing and no review. I'll
>>>>>>>>>> pass it through review process tomorrow and
>>>>>>>>>> hopefully it will be released in one-two days.
>>>>>>>>>>
>>>>>>>>>> Andrew.
>>>>>>>>>>
>>>>>>>>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Andrew,
>>>>>>>>>>
>>>>>>>>>> We have yet to notice a distinct pattern with the failures.
>>>>>>>>>> Sometimes, the RCF will start and connect without issue a few times in a
>>>>>>>>>> row before failing to connect again. Once the issue begins to occur,
>>>>>>>>>> neither rebooting all of the hosts (test engine VM, tester, IUT) or
>>>>>>>>>> deleting all of the build directories (suites, agents, inst) and rebooting
>>>>>>>>>> the hosts afterward resolves the issue. When it begins working again seems
>>>>>>>>>> very arbitrary to us.
>>>>>>>>>>
>>>>>>>>>> I do usually try to terminate the test engine with Ctrl+C, but
>>>>>>>>>> when it hangs while trying to start RCF, that does not work.
>>>>>>>>>>
>>>>>>>>>> Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>>>>>>>> the RCF connection to the test bed hosts? And if so, is there a way to
>>>>>>>>>> force it to use IPv4?
>>>>>>>>>>
>>>>>>>>>>  - Adam
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>>
>>>>>>>>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>>>
>>>>>>>>>>> Done. It works fine for me without any issues.
>>>>>>>>>>>
>>>>>>>>>>> Have you noticed any pattern when it works or does not work?
>>>>>>>>>>> May be it is a problem of not clean state after termination?
>>>>>>>>>>> Does it work fine the first time after DUTs reboot?
>>>>>>>>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>>>>>>>>> terminal where you execute run.sh command.
>>>>>>>>>>>  In this case it should shutdown gracefully and close all test
>>>>>>>>>>> agents and engine applications.
>>>>>>>>>>>
>>>>>>>>>>> (I'm trying to understand why you've seen many test agent
>>>>>>>>>>> processes. It should not happen.)
>>>>>>>>>>>
>>>>>>>>>>> Andrew.
>>>>>>>>>>>
>>>>>>>>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>>>>>>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>>>>>>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>>>>>>>>> This could be the source of the problem. I encountered a
>>>>>>>>>>> dependency issue trying to run the Test Engine on 22.04 LTS, so I
>>>>>>>>>>> downgraded the system. Since the tester is also the host having connection
>>>>>>>>>>> issues, I will try downgrading that system to 20.04, and see if that
>>>>>>>>>>> changes anything.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>>>>>>>>>> Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every
>>>>>>>>>>> night.
>>>>>>>>>>> Right now Debian 11 is used for test engine in nightly
>>>>>>>>>>> regressions.
>>>>>>>>>>>
>>>>>>>>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>>>>>>>>
>>>>>>>>>>> I did try passing in the "--vg-rcf" argument to the run.sh
>>>>>>>>>>> script of the test suite after installing valgrind, but there was no
>>>>>>>>>>> additional output that I saw.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sorry, I should valgrind output should be in valgrind.te_rcf
>>>>>>>>>>> (direction where you run test engine).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I will try pulling in the changes you've pushed up, and will see
>>>>>>>>>>> if that fixes anything.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Adam
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>>>>>>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello Adam,
>>>>>>>>>>>>
>>>>>>>>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to try to repeat the problem locally. Which Linux
>>>>>>>>>>>> distro is running on test engine and agents?
>>>>>>>>>>>>
>>>>>>>>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we
>>>>>>>>>>>> have
>>>>>>>>>>>> patch in review to fix it, however, the behaviour is different
>>>>>>>>>>>> in
>>>>>>>>>>>> this case, so it is unlike the same problem.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I've just published a new tag which fixes known test engine
>>>>>>>>>>>> side problems on Debian 12 and Fedora 38.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> One more idea is to install valgrind on the test engine host and
>>>>>>>>>>>> run with option --vg-rcf to check if something weird is
>>>>>>>>>>>> happening.
>>>>>>>>>>>>
>>>>>>>>>>>> What I don't understand right now is why I see just one failed
>>>>>>>>>>>> attempt
>>>>>>>>>>>> to connect in your log.txt and then Logger shutdown after 9
>>>>>>>>>>>> minutes.
>>>>>>>>>>>>
>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>  > Is there any firewall in the network or on test hosts which
>>>>>>>>>>>> could block incoming TCP connection to the port 23571
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the
>>>>>>>>>>>> host where you run test engine?
>>>>>>>>>>>>
>>>>>>>>>>>> Our test engine host and the testbed are on the same subnet.
>>>>>>>>>>>> The connection does work sometimes.
>>>>>>>>>>>>
>>>>>>>>>>>>  > If behaviour the same on the next try and you see that test
>>>>>>>>>>>> agent is kept running, could you check using
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > # netstat -tnlp
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > that Test Agent is listening on the port and try to
>>>>>>>>>>>> establish TCP connection from test agent using
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > and check if TCP connection could be established.
>>>>>>>>>>>>
>>>>>>>>>>>> I was able to replicate the same behavior again, where it hangs
>>>>>>>>>>>> while RCF is trying to start.
>>>>>>>>>>>> Running this command, I see this in the output:
>>>>>>>>>>>>
>>>>>>>>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>>>>>>> <http://0.0.0.0:23571>           0.0.0.0:*
>>>>>>>>>>>> LISTEN      18599/ta
>>>>>>>>>>>>
>>>>>>>>>>>> So it seems like it is listening on the correct port.
>>>>>>>>>>>> Additionally, I was able to connect to the Tester machine from
>>>>>>>>>>>> our Test Engine host using telnet. It printed the PID of the process once
>>>>>>>>>>>> the connection was opened.
>>>>>>>>>>>>
>>>>>>>>>>>> I tried running the "ta" application manually on the command
>>>>>>>>>>>> line, and it didn't print anything at all.
>>>>>>>>>>>> Maybe the issue is something on the Test Engine side.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>>>>>>>>> andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>     Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>      > On the tester host (which appears to be the Peer agent),
>>>>>>>>>>>> there
>>>>>>>>>>>>     are four processes that I see running, which look like the
>>>>>>>>>>>> test
>>>>>>>>>>>>     agent processes.
>>>>>>>>>>>>
>>>>>>>>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>>>>>>>>
>>>>>>>>>>>>     Is there any firewall in the network or on test hosts which
>>>>>>>>>>>> could
>>>>>>>>>>>>     block incoming TCP connection to the port 23571
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>>>>>>>>     where you run test engine?
>>>>>>>>>>>>
>>>>>>>>>>>>     If behaviour the same on the next try and you see that test
>>>>>>>>>>>> agent is
>>>>>>>>>>>>     kept running, could you check using
>>>>>>>>>>>>
>>>>>>>>>>>>     # netstat -tnlp
>>>>>>>>>>>>
>>>>>>>>>>>>     that Test Agent is listening on the port and try to
>>>>>>>>>>>> establish TCP
>>>>>>>>>>>>     connection from test agent using
>>>>>>>>>>>>
>>>>>>>>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>     and check if TCP connection could be established.
>>>>>>>>>>>>
>>>>>>>>>>>>     Another idea is to login Tester under root as testing does,
>>>>>>>>>>>> get
>>>>>>>>>>>>     start TA command from the log and try it by hands without
>>>>>>>>>>>> -n and
>>>>>>>>>>>>     remove extra escaping.
>>>>>>>>>>>>
>>>>>>>>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>
>>>>>>>>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>>>>>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>
>>>>>>>>>>>>     Hopefully in this case test agent directory remains in the
>>>>>>>>>>>> /tmp and
>>>>>>>>>>>>     you don't need to copy it as testing does.
>>>>>>>>>>>>     May be output could shed some light on what's going on.
>>>>>>>>>>>>
>>>>>>>>>>>>     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>     Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>     This is the output that I see in the terminal when this
>>>>>>>>>>>> failure
>>>>>>>>>>>>     occurs, after the test agent binaries build and the test
>>>>>>>>>>>> engine
>>>>>>>>>>>>     starts:
>>>>>>>>>>>>
>>>>>>>>>>>>     Platform default build - pass
>>>>>>>>>>>>     Simple RCF consistency check succeeded
>>>>>>>>>>>>     --->>> Starting Logger...done
>>>>>>>>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>>>>>>>>> timed
>>>>>>>>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>
>>>>>>>>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>>>>>>>>     processes. I let it hang for around 9 minutes.
>>>>>>>>>>>>
>>>>>>>>>>>>     On the tester host (which appears to be the Peer agent),
>>>>>>>>>>>> there are
>>>>>>>>>>>>     four processes that I see running, which look like the test
>>>>>>>>>>>> agent
>>>>>>>>>>>>     processes.
>>>>>>>>>>>>
>>>>>>>>>>>>     ta.Peer is an empty file. I've attached the log.txt from
>>>>>>>>>>>> this run.
>>>>>>>>>>>>
>>>>>>>>>>>>      - Adam
>>>>>>>>>>>>
>>>>>>>>>>>>     On Thu, Aug 24, 2023 at 4:22��AM Andrew Rybchenko
>>>>>>>>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>         Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>>>>>>> double-checked
>>>>>>>>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>>>>>>         Description in in
>>>>>>>>>>>> doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>         says that copy_timeout is in seconds and implementation
>>>>>>>>>>>> in
>>>>>>>>>>>>         lib/rcfunix/rcfunix.c passes the value to select()
>>>>>>>>>>>> tv_sec.
>>>>>>>>>>>>         Theoretically select() could be interrupted by signal,
>>>>>>>>>>>> but I
>>>>>>>>>>>>         think it is unlikely here.
>>>>>>>>>>>>
>>>>>>>>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>>>>>>>>         connection timeout. Does it happen on TE startup when
>>>>>>>>>>>> RCF
>>>>>>>>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could
>>>>>>>>>>>> help. Or
>>>>>>>>>>>>         does it happen when tests are in progress, e.g. in the
>>>>>>>>>>>> middle
>>>>>>>>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and
>>>>>>>>>>>> most
>>>>>>>>>>>>         likely either host with test agent dies or test agent
>>>>>>>>>>>> itself
>>>>>>>>>>>>         crashes. It would be easier for me if classify it if
>>>>>>>>>>>> you share
>>>>>>>>>>>>         text log (log.txt, full or just corresponding fragment
>>>>>>>>>>>> with
>>>>>>>>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>>>>>>>>         depending on which agent has problems could shed some
>>>>>>>>>>>> light.
>>>>>>>>>>>>         Corresponding files contain stdout/stderr of test
>>>>>>>>>>>> agents.
>>>>>>>>>>>>
>>>>>>>>>>>>         Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>         Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>         I've set up a test rig repository here, and have created
>>>>>>>>>>>>         configurations for our development testbed based off of
>>>>>>>>>>>> the
>>>>>>>>>>>>         examples.
>>>>>>>>>>>>         We've been able to get the test suite to run manually on
>>>>>>>>>>>>         Mellanox CX5 devices once.
>>>>>>>>>>>>         However, we are running into an issue where, when RCF
>>>>>>>>>>>> starts,
>>>>>>>>>>>>         the RCF connection times out very frequently. We aren't
>>>>>>>>>>>> sure
>>>>>>>>>>>>         why this is the case.
>>>>>>>>>>>>         It works sometimes, but most of the time when we try to
>>>>>>>>>>>> run
>>>>>>>>>>>>         the test engine, it encounters this issue.
>>>>>>>>>>>>         I've tried changing the RCF port by setting
>>>>>>>>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the
>>>>>>>>>>>> testbed
>>>>>>>>>>>>         machines. Neither seems to fix the issue.
>>>>>>>>>>>>
>>>>>>>>>>>>         It also seems like the timeout takes far longer than 60
>>>>>>>>>>>>         seconds, even when running "export
>>>>>>>>>>>> TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>         before I try to run the test suite.
>>>>>>>>>>>>         I assume the unit for this variable is seconds?
>>>>>>>>>>>>
>>>>>>>>>>>>         Thanks,
>>>>>>>>>>>>         Adam
>>>>>>>>>>>>
>>>>>>>>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>             Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>             Thanks, I've cloned the example repository and will
>>>>>>>>>>>> start
>>>>>>>>>>>>             setting up a configuration for our development
>>>>>>>>>>>> testbed
>>>>>>>>>>>>             today. I'll let you know if I run into any
>>>>>>>>>>>> difficulties
>>>>>>>>>>>>             or have any questions.
>>>>>>>>>>>>
>>>>>>>>>>>>              - Adam
>>>>>>>>>>>>
>>>>>>>>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                 I've published
>>>>>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>                 Hopefully it will help to define your test rigs
>>>>>>>>>>>> and
>>>>>>>>>>>>                 successfully run some tests manually. Feel free
>>>>>>>>>>>> to
>>>>>>>>>>>>                 ask any questions and I'll answer here and try
>>>>>>>>>>>> to
>>>>>>>>>>>>                 update documentation.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Meanwhile I'll prepare missing bits for steps
>>>>>>>>>>>> (2) and
>>>>>>>>>>>>                 (3).
>>>>>>>>>>>>                 Hopefully everything is in place for step (4),
>>>>>>>>>>>> but we
>>>>>>>>>>>>                 need to make steps (2) and (3) first.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                 > I've conferred with the rest of the team, and
>>>>>>>>>>>> we
>>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>> mainly
>>>>>>>>>>>>                 option B.
>>>>>>>>>>>>
>>>>>>>>>>>>                 OK, I'll provide the sample on Monday for you.
>>>>>>>>>>>> It is
>>>>>>>>>>>>                 almost ready right now, but I need to
>>>>>>>>>>>> double-check
>>>>>>>>>>>>                 it before publishing.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Regards,
>>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>>>>>                 conversation. Others in the community might find
>>>>>>>>>>>>                 this conversation valuable.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We do want to run testing on a regular basis.
>>>>>>>>>>>> The
>>>>>>>>>>>>                 Jenkins integration will be very useful for us,
>>>>>>>>>>>> as
>>>>>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>> mainly
>>>>>>>>>>>>                 option B.
>>>>>>>>>>>>                 If you would like to know anything about our
>>>>>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>>>>>>>                 questions you have.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>>>>>                 hardware and CPU architecture. I figured this
>>>>>>>>>>>> might
>>>>>>>>>>>>                 be an important detail.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Thanks,
>>>>>>>>>>>>                 Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>>>>>> Rybchenko
>>>>>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                     Greatings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                     I'm happy to hear that you're trying to
>>>>>>>>>>>> bring
>>>>>>>>>>>>                     it up.
>>>>>>>>>>>>
>>>>>>>>>>>>                     As I understand the final goal is to run it
>>>>>>>>>>>> on
>>>>>>>>>>>>                     regular basis. So, we need to make it
>>>>>>>>>>>> properly
>>>>>>>>>>>>                     from the very beginning.
>>>>>>>>>>>>                     Bring up of all features consists of 4
>>>>>>>>>>>> steps:
>>>>>>>>>>>>
>>>>>>>>>>>>                     1. Create site-specific repository (we call
>>>>>>>>>>>> it
>>>>>>>>>>>>                     ts-rigs) which contains information about
>>>>>>>>>>>> test
>>>>>>>>>>>>                     rigs and other site-specific information
>>>>>>>>>>>> like
>>>>>>>>>>>>                     where to send mails, where to store logs
>>>>>>>>>>>> etc.
>>>>>>>>>>>>                     It is required for manual execution as well,
>>>>>>>>>>>>                     since test rigs description is essential.
>>>>>>>>>>>> I'll
>>>>>>>>>>>>                     return to the topic below.
>>>>>>>>>>>>
>>>>>>>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>>>>>>>                     Basically it is a disk space plus apache2
>>>>>>>>>>>> web
>>>>>>>>>>>>                     server with few CGI scripts which help a
>>>>>>>>>>>> lot to
>>>>>>>>>>>>                     save disk space.
>>>>>>>>>>>>
>>>>>>>>>>>>                     3. Setup Bublik web application which
>>>>>>>>>>>> provides
>>>>>>>>>>>>                     web interface to view testing results. Same
>>>>>>>>>>>> as
>>>>>>>>>>>>                     https://ts-factory.io/bublik
>>>>>>>>>>>>                     <https://ts-factory.io/bublik>
>>>>>>>>>>>> <https://ts-factory.io/bublik>
>>>>>>>>>>>>
>>>>>>>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>>>>>>>                     save logs in log storage (2) and import it
>>>>>>>>>>>> to
>>>>>>>>>>>>                     bublik (3).
>>>>>>>>>>>>
>>>>>>>>>>>>                     Last few month we spent on our homework to
>>>>>>>>>>>> make
>>>>>>>>>>>>                     it simpler to bring up automated execution
>>>>>>>>>>>>                     using Jenkins -
>>>>>>>>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>>>>>                     available tomorrow.
>>>>>>>>>>>>
>>>>>>>>>>>>                     Let's return to the step (1).
>>>>>>>>>>>>
>>>>>>>>>>>>                     Unfortunately there is no publicly available
>>>>>>>>>>>>                     example of the ts-rigs repository since
>>>>>>>>>>>>                     sensitive site-specific information is
>>>>>>>>>>>> located
>>>>>>>>>>>>                     there. But I'm ready to help you to create
>>>>>>>>>>>> it
>>>>>>>>>>>>                     for UNH. I see two options here:
>>>>>>>>>>>>
>>>>>>>>>>>>                     (A) I'll ask questions and based on your
>>>>>>>>>>>>                     answers will create the first draft with my
>>>>>>>>>>>>                     comments.
>>>>>>>>>>>>
>>>>>>>>>>>>                     (B) I'll make a template/example ts-rigs
>>>>>>>>>>>> repo,
>>>>>>>>>>>>                     publish it and you'll create UNH ts-rigs
>>>>>>>>>>>> based
>>>>>>>>>>>>                     on it.
>>>>>>>>>>>>
>>>>>>>>>>>>                     Of course, I'll help to debug and finally
>>>>>>>>>>>> bring
>>>>>>>>>>>>                     it up in any case.
>>>>>>>>>>>>
>>>>>>>>>>>>                     (A) is a bit simpler for me and you, but
>>>>>>>>>>>> (B) is
>>>>>>>>>>>>                     a bit more generic and will help other
>>>>>>>>>>>>                     potential users to bring it up.
>>>>>>>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>>>>>>>                     What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>>                     Thanks,
>>>>>>>>>>>>                     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                     Greetings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>>>>>>                     would be happy to help
>>>>>>>>>>>>
>>>>>>>>>>>>                     Thanks,
>>>>>>>>>>>>                     Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>                     <ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>>                     Greetings Konstantin,
>>>>>>>>>>>>
>>>>>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>>>>>                     Poll Mode Driver test suite as an addition
>>>>>>>>>>>> to
>>>>>>>>>>>>                     our testing coverage for DPDK at the UNH
>>>>>>>>>>>> lab.
>>>>>>>>>>>>
>>>>>>>>>>>>                     I have some questions about how to set the
>>>>>>>>>>>>                     test suite arguments.
>>>>>>>>>>>>
>>>>>>>>>>>>                     I have been able to configure the Test
>>>>>>>>>>>> Engine
>>>>>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>>>>>                     fails to run.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>
>>>>>>>>>>>>                     The documentation mentions that there are
>>>>>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>>>>>                     seem like they would need to be set
>>>>>>>>>>>> somewhere
>>>>>>>>>>>>                     to run many of the tests.
>>>>>>>>>>>>
>>>>>>>>>>>>                     I see in the Test Engine documentation,
>>>>>>>>>>>> there
>>>>>>>>>>>>                     are instructions on how to create new
>>>>>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>>>>>                     some other way.
>>>>>>>>>>>>
>>>>>>>>>>>>                     How should these values be set?
>>>>>>>>>>>>
>>>>>>>>>>>>                     I'm also not sure what environment
>>>>>>>>>>>>                     variables/arguments are strictly necessary
>>>>>>>>>>>> or
>>>>>>>>>>>>                     which are optional.
>>>>>>>>>>>>
>>>>>>>>>>>>                     Regards,
>>>>>>>>>>>>                     Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>>>>>                     Senior Developer
>>>>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 --                 *Adam Hassick*
>>>>>>>>>>>>                 Senior Developer
>>>>>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>>> <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>>>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>             --             *Adam Hassick*
>>>>>>>>>>>>             Senior Developer
>>>>>>>>>>>>             UNH InterOperability Lab
>>>>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>             +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>         --         *Adam Hassick*
>>>>>>>>>>>>         Senior Developer
>>>>>>>>>>>>         UNH InterOperability Lab
>>>>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>         +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>     --     *Adam Hassick*
>>>>>>>>>>>>     Senior Developer
>>>>>>>>>>>>     UNH InterOperability Lab
>>>>>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>>     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>>> Senior Developer
>>>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>>> <https://www.iol.unh.edu/>
>>>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Adam Hassick*
>>>>>>>>>>> Senior Developer
>>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Adam Hassick*
>>>>>>>>>> Senior Developer
>>>>>>>>>> UNH InterOperability Lab
>>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Adam Hassick*
>>>>>>>>> Senior Developer
>>>>>>>>> UNH InterOperability Lab
>>>>>>>>> ahassick@iol.unh.edu
>>>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>> +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Adam Hassick*
>>>>>>> Senior Developer
>>>>>>> UNH InterOperability Lab
>>>>>>> ahassick@iol.unh.edu
>>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>> +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Adam Hassick*
>>>>>> Senior Developer
>>>>>> UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu
>>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>>> +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 239754 bytes --]

[-- Attachment #2: id_ed25519.pub --]
[-- Type: application/octet-stream, Size: 104 bytes --]

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKKuH2LWlFDdPqgnoNM9FFRIMZGV6/a1O5C0JV0JqemZ root@ts-factory-runner

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 275194 bytes --]

Hi Adam,

fresh ts-rigs-sample has all required bindings to publish logs to 
ts-factory.io. See the top-most commit.
So, you just need tune SSH to use correct user, port and key for 
ts-factory.io. You  can add below settings
to corresponding user .ssh/config or add corresponding options to sftp 
command in scripts/publish_logs/prj/ts-factory/publish:

Host ts-factory.io
         User unh-iol
         Port 56777
         IdentitiesOnly yes
         IdentityFile <path-to-private-key>

The publishing is asynchronous in ts-factory case. i.e. user puts logs 
and cron job checks incoming directory every 10 minutes and publish 
found logs.

You can request publishing when you start testing using --publish option 
or publish previous run testing results using ./scripts/publish_logs 
from dpdk-ethdev-ts sources. PWD should be the directory you run tests from.

Please, let me know how it goes.

Regards,
Andrew.

On 10/26/23 20:44, Adam Hassick wrote:
> Hi Andrew,
>
> I've attached the public key we'd like to use.
>
> Thanks,
> Adam
>
> On Thu, Oct 26, 2023 at 8:19 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     On 10/25/23 23:27, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     Sorry about the two week radio silence, we're still trying to
>>     sort out logistics for deployment on our end.
>>
>>     > I've created setup on ts-factory.io <http://ts-factory.io>
>>     which allows to publish logs. Let me know if you'd like to try it
>>     and I'll provide credentials, script and short instruction.
>>
>>     We're interested in publishing some test logs to the ts-factory
>>     Bublik instance in the meantime.
>
>     Please, send me SSH public key which you'd like to use to upload logs.
>     I'll provide helper script and instructions to do it.
>
>     Andrew.
>>
>>     Thanks,
>>     Adam
>>
>>     On Mon, Oct 23, 2023 at 7:11 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         Hi Adam,
>>
>>         > Now that our test results are in line with yours, we can
>>         begin looking into setting up the production environment.
>>
>>         Please, let me know if you need any help with it or waiting
>>         for an input from me.
>>
>>         Regards,
>>         Andrew.
>>
>>         On 10/10/23 17:09, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         Thank you for taking a look at our log. Netplan was
>>>         attempting to run DHCP on our test links, and additionally I
>>>         discovered that the NIC firmware was transmitting LLDP
>>>         packets, causing tests to fail in the same way. Now that
>>>         these problems have been fixed, our pass rate on the XL710
>>>         is approximately 91%. Now that our test results are in line
>>>         with yours, we can begin looking into setting up the
>>>         production environment.
>>>
>>>         First, is it possible to run the test agent on ARM hosts?
>>>         Our ARM testbeds have the best topology for running this
>>>         test suite, with separate tester and DUT servers.
>>>
>>>         We are testing this test suite on two x86 development
>>>         servers using the test suite's recommended server topology.
>>>         In contrast, our existing x86 production testbeds which run
>>>         DTS have a single server topology. This single server has
>>>         both the tester NIC and the device under test NIC installed,
>>>         with NUMA node separation between TRex and DPDK. We're going
>>>         to test running the two test agent processes on the
>>>         single-server testbeds if we cannot run this on ARM. Is
>>>         there any reason you can think of that would prevent this
>>>         setup from working?
>>>
>>>         Once we figure out where this can live in production, then
>>>         we will begin setting up log storage, Jenkins integration,
>>>         and Bublik.
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Thu, Oct 5, 2023 at 6:25 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             Hi Adam,
>>>
>>>             > Do these default to vfio-pci?
>>>
>>>             Yes, vfio-pci is the default.
>>>             However, it does not work in the case of Mellanox which
>>>             uses bifurcated driver. It should mlx5_core for Mellanox
>>>             NICs.
>>>
>>>             > Here is the text log from a run on our Intel XL710
>>>             NICs, with the expected result profile set to the X710.
>>>
>>>             It is hard to analyze all tests using text logs, but I
>>>             definitely see one common problem. Tests receive
>>>             unexpected packets and fail because of it.
>>>             Tests are written very strict from this point of view
>>>             and it brought fruits in the past when HW had bugs.
>>>             Are DUT and tester connected back-to-back on tested
>>>             interfaces or via switch?
>>>             If via switch, is it possible to isolate it from
>>>             everything else?
>>>             If back-to-back, it could be some embedded SW/FW which
>>>             regenerates these packet.
>>>             I definitely see unexpected DHCP packets.
>>>
>>>             > We haven't set up the Jenkins integration yet, however
>>>             if this is required to import the logs then we will
>>>             prioritize that.
>>>
>>>             Unfortunately manual runs do not generate all artifacts
>>>             required to import logs. However, we have almost solved
>>>             it right now. Hopefully we'll finalize it in a day or
>>>             two. I'll let you know when these changes are available.
>>>
>>>             Regards,
>>>             Andrew.
>>>
>>>             On 10/4/23 16:48, Adam Hassick wrote:
>>>>             Hi Andrew,
>>>>
>>>>             Ok, that makes sense. I don't see
>>>>             TE_ENV_H1/H2_DPDK_DRIVER set anywhere in the default
>>>>             configurations for the Intel X710. Do these default to
>>>>             vfio-pci?
>>>>             We have IOMMU enabled on our development testbed, and
>>>>             should be able to bind vfio-pci.
>>>>             Here is the text log from a run on our Intel XL710
>>>>             NICs, with the expected result profile set to the X710.
>>>>             We haven't set up the Jenkins integration yet, however
>>>>             if this is required to import the logs then we will
>>>>             prioritize that.
>>>>             log.txt.tar.gz
>>>>             <https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web>
>>>>
>>>>             Thanks,
>>>>             Adam
>>>>
>>>>             On Mon, Sep 18, 2023 at 11:04 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>                 On 9/18/23 17:44, Adam Hassick wrote:
>>>>>                 Hi Andrew and Konstantin,
>>>>>
>>>>>                 Thank you for adding the tester-dial feature, this
>>>>>                 opens up the possibility for us to do CI
>>>>>                 integrated testing in the future.
>>>>>
>>>>>                 Our Mellanox pass rate is similar to yours (about
>>>>>                 ~2400 passing, ~4400 failing), however our Intel
>>>>>                 pass rates are far worse.
>>>>>                 I will try running tests on the XL710 with the
>>>>>                 trc-tags argument set and see if it improves the
>>>>>                 pass rate.
>>>>>                 Another thing I noticed in the results you
>>>>>                 uploaded is that the results are tagged with
>>>>>                 vfio-pci and not i40e.
>>>>>                 Though in the environment dump, the driver on the
>>>>>                 test machine and the DUT are set to use the i40e
>>>>>                 driver. Is this important at all?
>>>>
>>>>                 I think it is a misunderstanding here. There are
>>>>                 two kinds of driver in configuration: net driver
>>>>                 and so-called DPDK driver.
>>>>                 Net driver is a Linux kernel network device driver
>>>>                 used on Tester side.
>>>>                 DPDK driver is a Linux kernel driver to bind device
>>>>                 to to use it with DPDK. So, it is NOT a driver
>>>>                 inside DPDK (drivers/net/*).
>>>>                 In the case of bifurcated driver (like mlx5_core)
>>>>                 it is the same in both cases.
>>>>                 In non-bifurcated case DPDK driver is some UIO
>>>>                 driver(vfio-pci, uio-pci-generic or igb_uio).
>>>>                 Some expectations depend on used UIO. For example,
>>>>                 uio-pci-generic do not support many interrupts
>>>>                 (used by usecases/rx_intr test cases).
>>>>                 That's why we care corresponding TRC tag.
>>>>
>>>>                 TE_ENV_*_DPDK_DRIVER variables should be vfio-pc 
>>>>                 in 710's Intel case. Or uio-pci-generic if IOMMU is
>>>>                 turned off on corresponding machines and Linux
>>>>                 distro does not support VFIO no IOMMU mode.
>>>>
>>>>                 Andrew.
>>>>
>>>>>                 There isn't anything preventing us from pushing
>>>>>                 our results up to the existing Bublik instance
>>>>>                 running at ts-factory.io <http://ts-factory.io>
>>>>>                 that I can think of at the moment.
>>>>>                 We will have to work out how to submit our results
>>>>>                 to your Bublik instance in a controlled and secure
>>>>>                 manner in that case.
>>>>>                 As far as I know we won't need access controls for
>>>>>                 the results themselves. I'll discuss this with
>>>>>                 Patrick and will let you know once we confirm that
>>>>>                 it's fine.
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>                 On Mon, Sep 18, 2023 at 2:26 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>                     On 9/18/23 09:23, Konstantin Ushakov wrote:
>>>>>>
>>>>>>                     Hi Andrew,
>>>>>>
>>>>>>                     should we always auto-assign the tags or you
>>>>>>                     don’t do it since it slows down (by some
>>>>>>                     seconds) the TE startup?
>>>>>>
>>>>>
>>>>>                     Tags are auto-assigned, but I guess it differs
>>>>>                     in Adam's case since NIC is a bit different.
>>>>>                     Below test will help to understand if it is
>>>>>                     the root cause of very different expectations.
>>>>>                     If pass rate will be close to mine, I'll
>>>>>                     simply update TRC database to share
>>>>>                     expectations for mine NIC and NIC used by Adam.
>>>>>
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     I think I second the question from Andrew -
>>>>>>                     happy to help you with the triage so that we
>>>>>>                     get to the same baseline. Do you have a good
>>>>>>                     way for us to share the logs? I.e. say upload
>>>>>>                     to ts-factory if we add strict permissions
>>>>>>                     system so it’s not publishing or any other way.
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Konstantin
>>>>>>
>>>>>>
>>>>>>                     On 18 Sep 2023, at 9:15, Andrew Rybchenko wrote:
>>>>>>
>>>>>>                         Hi Adam,
>>>>>>
>>>>>>                         I've uploaded fresh testing results to
>>>>>>                         ts-factory.io <http://ts-factory.io> [1]
>>>>>>                         to be on the same page.
>>>>>>
>>>>>>                         I think I know why your and mine results
>>>>>>                         on Intel 710 series NICs differ so much.
>>>>>>                         Testing results expectations database
>>>>>>                         (dpdk-ethdev-ts/trc/*) is filled in in
>>>>>>                         terms of TRC tags.  I.e. expectations
>>>>>>                         depends on TRC tags discovered by helper
>>>>>>                         scripts when testing is started. These
>>>>>>                         tags identify various aspects of what is
>>>>>>                         tested. Ideally expectations should be
>>>>>>                         written in terms of root cause of the
>>>>>>                         expected behaviour. If it is a driver
>>>>>>                         expectations, driver tag should be used.
>>>>>>                         If it is HW limitation, tags with PCI IDs
>>>>>>                         should be used. However, it is not always
>>>>>>                         easy to classify it correctly if you're
>>>>>>                         not involved in driver development. So,
>>>>>>                         in order case expectations for 710's
>>>>>>                         Intel are filled in in terms of PCI IDs.
>>>>>>                         I guess PCI ID differ in your case and
>>>>>>                         that's why expectations filled in for my
>>>>>>                         NIC do not apply to your runs.
>>>>>>
>>>>>>                         Just try to add the following option when
>>>>>>                         you run on your 710's Intel in order to
>>>>>>                         mimic mine and see if it helps to achieve
>>>>>>                         better pass rate.
>>>>>>                         --trc-tag=pci-8086-1572
>>>>>>
>>>>>>                         BTW, fresh TE tag v1.21.0 has improved
>>>>>>                         algorithm to choose tests for
>>>>>>                         --tester-dial option. It should have
>>>>>>                         better coverage now.
>>>>>>
>>>>>>                         Andrew.
>>>>>>
>>>>>>                         [1]
>>>>>>                         https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1
>>>>>>                         <https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1>
>>>>>>
>>>>>>                         On 9/13/23 18:45, Andrew Rybchenko wrote:
>>>>>>>                         Hi Adam,
>>>>>>>
>>>>>>>                         I've pushed new TE tag v1.20.0 which
>>>>>>>                         supported a new command-line option
>>>>>>>                         --tester-dial=NUM where NUM is from 0 to
>>>>>>>                         100. it allows to choose percentage of
>>>>>>>                         tests to run. If you want stable set,
>>>>>>>                         you should pass --tester-random-seed=0
>>>>>>>                         (or other integer). It is the first
>>>>>>>                         sketch and we have plans to improve it,
>>>>>>>                         but feedback would be welcome.
>>>>>>>
>>>>>>>                         > Is it needed on the tester?
>>>>>>>
>>>>>>>                         It is hard to say if it is strictly
>>>>>>>                         required for simple tests. However, it
>>>>>>>                         is better to update Tester as well,
>>>>>>>                         since performance tests run DPDK on
>>>>>>>                         Tester as well.
>>>>>>>
>>>>>>>                         > Are there any other manual setup steps
>>>>>>>                         for these devices that I might be missing?
>>>>>>>
>>>>>>>                         I don't remember anything else.
>>>>>>>
>>>>>>>                         I think it is better to get down to
>>>>>>>                         details and take a look at logs. I'm
>>>>>>>                         ready to help with it and explain what's
>>>>>>>                         happening there. May be it will help to
>>>>>>>                         understand if it is a problem with
>>>>>>>                         setup/configuration.
>>>>>>>
>>>>>>>                         Text logs are not very convenient.
>>>>>>>                         Ideally logs should be imported to
>>>>>>>                         bublik, however, manual runs do not
>>>>>>>                         provide all required artifacts right now
>>>>>>>                         (Jenkins jobs generate all required
>>>>>>>                         artifacts).
>>>>>>>                         Other option is 'tmp_raw_log' file
>>>>>>>                         (should be packed to make it smaller)
>>>>>>>                         which could be converted to various log
>>>>>>>                         formats.
>>>>>>>                         Would it be OK for you if I import your
>>>>>>>                         logs to bublik at ts-factory.io
>>>>>>>                         <http://ts-factory.io>? Or is it a
>>>>>>>                         problem that it is publicly available?
>>>>>>>                         Would it help if we add authentication
>>>>>>>                         and access control there?
>>>>>>>
>>>>>>>                         Andrew.
>>>>>>>
>>>>>>>                         On 9/8/23 17:57, Adam Hassick wrote:
>>>>>>>>                         Hi Andrew,
>>>>>>>>
>>>>>>>>                         I have a couple questions about needed
>>>>>>>>                         setup of the NICs for the ethdev test
>>>>>>>>                         suite.
>>>>>>>>
>>>>>>>>                         Our MCX5s and XL710s are failing the
>>>>>>>>                         checkup tests. The pass rate appears to
>>>>>>>>                         be much worse on the XL710s (40 of 73
>>>>>>>>                         tests failed, 3 passed unexpectedly).
>>>>>>>>
>>>>>>>>                         For the XL710s, I've updated the driver
>>>>>>>>                         and NVM versions to match the minimum
>>>>>>>>                         supported versions in the compatibility
>>>>>>>>                         matrix found on the DPDK documentation.
>>>>>>>>                         This did not change the failure rate much.
>>>>>>>>                         For the MCX5s, I've installed the
>>>>>>>>                         latest LTS version of the OFED
>>>>>>>>                         bifurcated driver on the DUT. Is it
>>>>>>>>                         needed on the tester?
>>>>>>>>
>>>>>>>>                         Are there any other manual setup steps
>>>>>>>>                         for these devices that I might be missing?
>>>>>>>>
>>>>>>>>                         Thanks,
>>>>>>>>                         Adam
>>>>>>>>
>>>>>>>>                         On Wed, Sep 6, 2023 at 11:00 AM Adam
>>>>>>>>                         Hassick <ahassick@iol.unh.edu> wrote:
>>>>>>>>
>>>>>>>>                             Hi Andrew,
>>>>>>>>
>>>>>>>>                             Yes, I copied the X710 configs to
>>>>>>>>                             set up XL710 configs. I changed the
>>>>>>>>                             environment variable names from the
>>>>>>>>                             X710 suffix to XL710 suffix in the
>>>>>>>>                             script, and forgot to change them
>>>>>>>>                             in the corresponding environment file.
>>>>>>>>                             That fixed the issue.
>>>>>>>>
>>>>>>>>                             I got the checkup tests working on
>>>>>>>>                             the XL710 now. Most of them are
>>>>>>>>                             failing, which leads me to believe
>>>>>>>>                             this is an issue with our testbed.
>>>>>>>>                             Based on the DPDK documentation for
>>>>>>>>                             i40e, the firmware and driver
>>>>>>>>                             versions are much older than what
>>>>>>>>                             DPDK 22.11 LTS and main prefer, so
>>>>>>>>                             I'll try updating those.
>>>>>>>>
>>>>>>>>                             For now I'm working on getting the
>>>>>>>>                             XL710 checkup tests passing, and
>>>>>>>>                             will pick up getting the E810
>>>>>>>>                             configured properly next. I'll let
>>>>>>>>                             you know if I run into any more
>>>>>>>>                             issues in relation to the test engine.
>>>>>>>>
>>>>>>>>                             Thanks,
>>>>>>>>                             Adam
>>>>>>>>
>>>>>>>>                             On Wed, Sep 6, 2023 at 7:36 AM
>>>>>>>>                             Andrew Rybchenko
>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>>>
>>>>>>>>                                 Hi Adam,
>>>>>>>>
>>>>>>>>                                 On 9/5/23 18:01, Adam Hassick
>>>>>>>>                                 wrote:
>>>>>>>>>                                 Hi Andrew,
>>>>>>>>>
>>>>>>>>>                                 The compilation warning issue
>>>>>>>>>                                 is now resolved. Again, thank
>>>>>>>>>                                 you guys for fixing this for
>>>>>>>>>                                 us. I can run the tests on the
>>>>>>>>>                                 Mellanox CX5s again, however
>>>>>>>>>                                 I'm running into a couple new
>>>>>>>>>                                 issues with running the
>>>>>>>>>                                 prologues on the Intel cards.
>>>>>>>>>
>>>>>>>>>                                 When running testing on the
>>>>>>>>>                                 Intel XL710s, I see this error
>>>>>>>>>                                 appear in the log:
>>>>>>>>>
>>>>>>>>>                                     ERROR  prologue
>>>>>>>>>                                      Environment LIB  14:16:13.650
>>>>>>>>>                                     Too few networks in
>>>>>>>>>                                     available configuration
>>>>>>>>>                                     (0) in comparison with
>>>>>>>>>                                     required (1)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 This seems like a trivial
>>>>>>>>>                                 configuration error, perhaps
>>>>>>>>>                                 this is something I need to
>>>>>>>>>                                 set up in ts-rigs. I briefly
>>>>>>>>>                                 searched through the examples
>>>>>>>>>                                 there and didn't see any
>>>>>>>>>                                 mention of how to set up a
>>>>>>>>>                                 network.
>>>>>>>>>                                 I will attach this log just in
>>>>>>>>>                                 case you need more information.
>>>>>>>>
>>>>>>>>                                 Unfortunately logs are
>>>>>>>>                                 insufficient to understand it.
>>>>>>>>                                 I've pushed new tag to TE
>>>>>>>>                                 v1.19.0 which add log message
>>>>>>>>                                 with TE_* environment variables.
>>>>>>>>                                 Most likely something is wrong
>>>>>>>>                                 with variables which are used
>>>>>>>>                                 as conditions when available
>>>>>>>>                                 networks are defined in
>>>>>>>>                                 ts-conf/cs/inc.net_cfg_pci_fns.yml:
>>>>>>>>                                 TE_PCI_INSTANCE_IUT_TST1
>>>>>>>>                                 TE_PCI_INSTANCE_IUT_TST1a
>>>>>>>>                                 TE_PCI_INSTANCE_TST1a_IUT
>>>>>>>>                                 TE_PCI_INSTANCE_TST1_IUT
>>>>>>>>                                 My guess it that you change
>>>>>>>>                                 naming a bit, but script like
>>>>>>>>                                 ts-rigs-sample/scripts/iut.h1-x710
>>>>>>>>                                 is not included or not updated.
>>>>>>>>
>>>>>>>>>                                 There is a different error
>>>>>>>>>                                 when running on the Intel
>>>>>>>>>                                 E810s. It appears to me like
>>>>>>>>>                                 it starts DPDK, does some
>>>>>>>>>                                 configuration inside DPDK and
>>>>>>>>>                                 on the device, and then fails
>>>>>>>>>                                 to bring the device back up.
>>>>>>>>>                                 Since this error seems very
>>>>>>>>>                                 non-trivial, I will also
>>>>>>>>>                                 attach this log.
>>>>>>>>
>>>>>>>>                                 This one is a bit simpler. Few
>>>>>>>>                                 lines after the first ERROR in
>>>>>>>>                                 log I see the following:
>>>>>>>>                                 WARN  RCF DPDK 13:06:00.144
>>>>>>>>                                 ice_program_hw_rx_queue():
>>>>>>>>                                 currently package doesn't
>>>>>>>>                                 support RXDID (22)
>>>>>>>>                                 ice_rx_queue_start(): fail to
>>>>>>>>                                 program RX queue 0
>>>>>>>>                                 ice_dev_start(): fail to start
>>>>>>>>                                 Rx queue 0
>>>>>>>>                                 Device with port_id=0 already
>>>>>>>>                                 stopped
>>>>>>>>
>>>>>>>>                                 It is stdout/stderr from test
>>>>>>>>                                 agent which runs DPDK. Same
>>>>>>>>                                 logs in plain format are
>>>>>>>>                                 available in ta.DPDK file.
>>>>>>>>                                 I'm not an expert here, but I
>>>>>>>>                                 vaguely remember that E810
>>>>>>>>                                 requires correct firmware and
>>>>>>>>                                 DDP to be loaded.
>>>>>>>>                                 There is some information in
>>>>>>>>                                 dpdk/doc/guides/nics/ice.rst.
>>>>>>>>
>>>>>>>>                                 You can try to add
>>>>>>>>                                 --dev-args=safe-mode-support=1
>>>>>>>>                                 command-line option described
>>>>>>>>                                 there.
>>>>>>>>
>>>>>>>>                                 Hope it helps,
>>>>>>>>                                 Andrew.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 Thanks,
>>>>>>>>>                                 Adam
>>>>>>>>>
>>>>>>>>>                                 On Fri, Sep 1, 2023 at 3:59 AM
>>>>>>>>>                                 Andrew Rybchenko
>>>>>>>>>                                 <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>                                 wrote:
>>>>>>>>>
>>>>>>>>>                                     Hi Adam,
>>>>>>>>>
>>>>>>>>>                                     On 8/31/23 22:38, Adam
>>>>>>>>>                                     Hassick wrote:
>>>>>>>>>>                                     Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                                     I have one additional
>>>>>>>>>>                                     question as well: Does
>>>>>>>>>>                                     the test engine support
>>>>>>>>>>                                     running tests on two
>>>>>>>>>>                                     ARMv8 test agents?
>>>>>>>>>>
>>>>>>>>>>                                         1. We'll sort out
>>>>>>>>>>                                         warnings this week.
>>>>>>>>>>                                         Thanks for heads up.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     Great. Let me know when
>>>>>>>>>>                                     that's fixed.
>>>>>>>>>
>>>>>>>>>                                     Done. We also fixed a
>>>>>>>>>                                     number of warnings in TE.
>>>>>>>>>                                     Also we fixed root test
>>>>>>>>>                                     package name to be
>>>>>>>>>                                     consistent with the
>>>>>>>>>                                     repository name.
>>>>>>>>>
>>>>>>>>>>                                         Support for old LTS
>>>>>>>>>>                                         branches was dropped
>>>>>>>>>>                                         some time ago, but in
>>>>>>>>>>                                         the future it is
>>>>>>>>>>                                         definitely possible
>>>>>>>>>>                                         to keep it for new
>>>>>>>>>>                                         LTS branches. I think
>>>>>>>>>>                                         22.11 is supported,
>>>>>>>>>>                                         but I'm not sure
>>>>>>>>>>                                         about older LTS releases.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     Good to know.
>>>>>>>>>>
>>>>>>>>>>                                         2. You can add
>>>>>>>>>>                                         command-line option
>>>>>>>>>>                                         --sanity to run tests
>>>>>>>>>>                                         marked with
>>>>>>>>>>                                         TEST_HARNESS_SANITY
>>>>>>>>>>                                         requirement (see
>>>>>>>>>>                                         dpdk-ethdev-ts/scripts/run.sh
>>>>>>>>>>                                         and grep
>>>>>>>>>>                                         TEST_HARNESS_SANITY
>>>>>>>>>>                                         dpdk-ethdev-ts to see
>>>>>>>>>>                                         which tests are
>>>>>>>>>>                                         marked). Yes, there
>>>>>>>>>>                                         is a space for
>>>>>>>>>>                                         terminology
>>>>>>>>>>                                         improvement here.
>>>>>>>>>>                                         We'll do it.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                     Done. Now it is called
>>>>>>>>>                                     --checkup.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                         Also it takes a lot
>>>>>>>>>>                                         of time because of
>>>>>>>>>>                                         failures and tests
>>>>>>>>>>                                         which wait for some
>>>>>>>>>>                                         timeout.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     That makes sense to me.
>>>>>>>>>>                                     We'll use the time to
>>>>>>>>>>                                     complete tests on virtio
>>>>>>>>>>                                     or the Intel devices as a
>>>>>>>>>>                                     reference for how long
>>>>>>>>>>                                     the tests really take to
>>>>>>>>>>                                     complete.
>>>>>>>>>>                                     We will explore the
>>>>>>>>>>                                     possibility of
>>>>>>>>>>                                     periodically running the
>>>>>>>>>>                                     sanity tests for patches.
>>>>>>>>>
>>>>>>>>>                                     I'll double-check and let
>>>>>>>>>                                     you know how long entire
>>>>>>>>>                                     TS runs on Intel X710,
>>>>>>>>>                                     E810, Mellanox CX5 and
>>>>>>>>>                                     virtio net. Just to ensure
>>>>>>>>>                                     that time observed in your
>>>>>>>>>                                     case looks the same.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                         The test harness can
>>>>>>>>>>                                         provide coverage
>>>>>>>>>>                                         reports based on
>>>>>>>>>>                                         gcov, but I'm not
>>>>>>>>>>                                         sure what you mean by
>>>>>>>>>>                                         a "dial" to control
>>>>>>>>>>                                         test coverage.
>>>>>>>>>>                                         Provided reports are
>>>>>>>>>>                                         rather for human to
>>>>>>>>>>                                         analyze.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     The general idea is to
>>>>>>>>>>                                     have some kind of
>>>>>>>>>>                                     parameter on the test
>>>>>>>>>>                                     suite, which could be an
>>>>>>>>>>                                     integer ranging from zero
>>>>>>>>>>                                     to ten, that controls how
>>>>>>>>>>                                     many tests are run based
>>>>>>>>>>                                     on how important the test is.
>>>>>>>>>>
>>>>>>>>>>                                     Similar to how some
>>>>>>>>>>                                     command line interfaces
>>>>>>>>>>                                     provide a verbosity level
>>>>>>>>>>                                     parameter (some number of
>>>>>>>>>>                                     "-v" arguments) to
>>>>>>>>>>                                     control the importance of
>>>>>>>>>>                                     the information in the log.
>>>>>>>>>>                                     The verbosity level zero
>>>>>>>>>>                                     only prints very
>>>>>>>>>>                                     important log messages,
>>>>>>>>>>                                     while ten prints everything.
>>>>>>>>>>
>>>>>>>>>>                                     In much the same manner
>>>>>>>>>>                                     as above, this "dial"
>>>>>>>>>>                                     parameter controls what
>>>>>>>>>>                                     tests are run and with
>>>>>>>>>>                                     what parameters based on
>>>>>>>>>>                                     how important those tests
>>>>>>>>>>                                     and test parameter
>>>>>>>>>>                                     combinations are.
>>>>>>>>>>                                     Coverage Level zero tells
>>>>>>>>>>                                     the suite to run a very
>>>>>>>>>>                                     basic set of important
>>>>>>>>>>                                     tests, with minimal
>>>>>>>>>>                                     parameterization. This
>>>>>>>>>>                                     mode would take only
>>>>>>>>>>                                     ~5-10 minutes to run.
>>>>>>>>>>                                     In contrast, Coverage
>>>>>>>>>>                                     Level ten includes all
>>>>>>>>>>                                     the edge cases, every
>>>>>>>>>>                                     combination of test
>>>>>>>>>>                                     parameters, everything
>>>>>>>>>>                                     the test suite can do,
>>>>>>>>>>                                     which takes the normal
>>>>>>>>>>                                     several hours to run.
>>>>>>>>>>                                     The values 1 - 9 are
>>>>>>>>>>                                     between those two
>>>>>>>>>>                                     extremes, allowing the
>>>>>>>>>>                                     user to get a gradient of
>>>>>>>>>>                                     test coverage in the
>>>>>>>>>>                                     results and to limit the
>>>>>>>>>>                                     running time.
>>>>>>>>>>
>>>>>>>>>>                                     Then we could, for
>>>>>>>>>>                                     example, run the "run.sh"
>>>>>>>>>>                                     with a level of 2 or 3
>>>>>>>>>>                                     for incoming patches that
>>>>>>>>>>                                     need quick results, and
>>>>>>>>>>                                     with a level of 10 for
>>>>>>>>>>                                     the less often run
>>>>>>>>>>                                     periodic tests performed
>>>>>>>>>>                                     on main or LTS branches.
>>>>>>>>>
>>>>>>>>>                                     Understood now. Thanks a
>>>>>>>>>                                     lot for the idea. We'll
>>>>>>>>>                                     discuss it and come back.
>>>>>>>>>
>>>>>>>>>>                                         3. Yes, really many
>>>>>>>>>>                                         tests on Mellanox CX5
>>>>>>>>>>                                         NICs report
>>>>>>>>>>                                         unexpected testing
>>>>>>>>>>                                         results.
>>>>>>>>>>                                         Unfortunately it is
>>>>>>>>>>                                         time consuming to
>>>>>>>>>>                                         fill in expectations
>>>>>>>>>>                                         database since it is
>>>>>>>>>>                                         necessary to analyze
>>>>>>>>>>                                         testing results and
>>>>>>>>>>                                         classify if it is a
>>>>>>>>>>                                         bug or just
>>>>>>>>>>                                         acceptable behaviour
>>>>>>>>>>                                         aspect.
>>>>>>>>>>
>>>>>>>>>>                                         Bublik allows to
>>>>>>>>>>                                         compare results of
>>>>>>>>>>                                         two runs. It is
>>>>>>>>>>                                         useful for human, but
>>>>>>>>>>                                         still not good for
>>>>>>>>>>                                         automation.
>>>>>>>>>>
>>>>>>>>>>                                         I have local patch
>>>>>>>>>>                                         for mlx5 driver which
>>>>>>>>>>                                         reports Tx ring size
>>>>>>>>>>                                         maximum. It makes
>>>>>>>>>>                                         pass rate higher. It
>>>>>>>>>>                                         is a problem for test
>>>>>>>>>>                                         harness that mlx5
>>>>>>>>>>                                         does not report
>>>>>>>>>>                                         limits right now.
>>>>>>>>>>
>>>>>>>>>>                                         Pass rate on Intel
>>>>>>>>>>                                         X710 is about 92% on
>>>>>>>>>>                                         my test rig. Pass
>>>>>>>>>>                                         rate on virtio net is
>>>>>>>>>>                                         99% right now and
>>>>>>>>>>                                         could be done 100%
>>>>>>>>>>                                         easily (just one
>>>>>>>>>>                                         thing to fix in
>>>>>>>>>>                                         expectations).
>>>>>>>>>>
>>>>>>>>>>                                         I think logs storage
>>>>>>>>>>                                         setup is essential
>>>>>>>>>>                                         for logs analysis. Of
>>>>>>>>>>                                         course, you can
>>>>>>>>>>                                         request HTML logs
>>>>>>>>>>                                         when you run tests
>>>>>>>>>>                                         (--log-html=html) or
>>>>>>>>>>                                         generate after run
>>>>>>>>>>                                         using
>>>>>>>>>>                                         dpdk-ethdev-ts/scripts/html-log.sh
>>>>>>>>>>                                         and open index.html
>>>>>>>>>>                                         in a browser, but
>>>>>>>>>>                                         logs storage makes it
>>>>>>>>>>                                         more convenient.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     We are interested in
>>>>>>>>>>                                     setting up Bublik,
>>>>>>>>>>                                     potentially as an
>>>>>>>>>>                                     externally-facing
>>>>>>>>>>                                     component, once we have
>>>>>>>>>>                                     our process of running
>>>>>>>>>>                                     the test suite stabilized.
>>>>>>>>>>                                     Once we are able to run
>>>>>>>>>>                                     the test suite again,
>>>>>>>>>>                                     I'll see what the pass
>>>>>>>>>>                                     rate is on our other
>>>>>>>>>>                                     hardware.
>>>>>>>>>>                                     Good to know that it
>>>>>>>>>>                                     isn't an issue with our
>>>>>>>>>>                                     dev testbed causing the
>>>>>>>>>>                                     high fail rate.
>>>>>>>>>>
>>>>>>>>>>                                     For Intel hardware, we
>>>>>>>>>>                                     have an XL710 and an
>>>>>>>>>>                                     Intel E810-C in our
>>>>>>>>>>                                     development testbed.
>>>>>>>>>>                                     Although they are
>>>>>>>>>>                                     slightly different
>>>>>>>>>>                                     devices, ideally the pass
>>>>>>>>>>                                     rate will be identical or
>>>>>>>>>>                                     similar. I have yet to
>>>>>>>>>>                                     set up a VM pair for
>>>>>>>>>>                                     virtio, but we will soon.
>>>>>>>>>>
>>>>>>>>>>                                         Latest version of
>>>>>>>>>>                                         test-environment has
>>>>>>>>>>                                         examples of our CGI
>>>>>>>>>>                                         scripts which we use
>>>>>>>>>>                                         for log storage (see
>>>>>>>>>>                                         tools/log_server/README.md).
>>>>>>>>>>
>>>>>>>>>>                                         Also all bits for
>>>>>>>>>>                                         Jenkins setup are
>>>>>>>>>>                                         available. See
>>>>>>>>>>                                         dpdk-ethdev-ts/jenkins/README.md
>>>>>>>>>>                                         and examples of
>>>>>>>>>>                                         jenkins files in
>>>>>>>>>>                                         ts-rigs-sample.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     Jenkins integration,
>>>>>>>>>>                                     setting up production rig
>>>>>>>>>>                                     configurations, and
>>>>>>>>>>                                     permanent log storage
>>>>>>>>>>                                     will be our next steps
>>>>>>>>>>                                     once I am able to run the
>>>>>>>>>>                                     tests again.
>>>>>>>>>>                                     Unless there is an easy
>>>>>>>>>>                                     way to have meson not
>>>>>>>>>>                                     pass "-Werror" into GCC.
>>>>>>>>>>                                     Then I would be able to
>>>>>>>>>>                                     run the test suite.
>>>>>>>>>
>>>>>>>>>                                     Hopefully it is resolved now.
>>>>>>>>>
>>>>>>>>>                                     I thought a bit more about
>>>>>>>>>                                     your usecase for Jenkins.
>>>>>>>>>                                     I'm not 100% sure that
>>>>>>>>>                                     existing pipelines are
>>>>>>>>>                                     convenient for your usecase.
>>>>>>>>>                                     Fill free to ask questions
>>>>>>>>>                                     when you are on it.
>>>>>>>>>
>>>>>>>>>                                     Thanks,
>>>>>>>>>                                     Andrew.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                     Thanks,
>>>>>>>>>>                                     Adam
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                         On 8/29/23 17:02,
>>>>>>>>>>                                         Adam Hassick wrote:
>>>>>>>>>>>                                         Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                                         That fix seems to
>>>>>>>>>>>                                         have resolved the
>>>>>>>>>>>                                         issue, thanks for
>>>>>>>>>>>                                         the quick turnaround
>>>>>>>>>>>                                         time on that patch.
>>>>>>>>>>>                                         Now that we have the
>>>>>>>>>>>                                         RCF timeout issue
>>>>>>>>>>>                                         resolved, there are
>>>>>>>>>>>                                         a few other
>>>>>>>>>>>                                         questions and issues
>>>>>>>>>>>                                         that we have about
>>>>>>>>>>>                                         the tests themselves.
>>>>>>>>>>>
>>>>>>>>>>>                                         1. The test suite
>>>>>>>>>>>                                         fails to build with
>>>>>>>>>>>                                         a couple warnings.
>>>>>>>>>>>
>>>>>>>>>>>                                         Below is the stderr
>>>>>>>>>>>                                         log from compilation:
>>>>>>>>>>>
>>>>>>>>>>>                                             FAILED:
>>>>>>>>>>>                                             lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>>>                                             cc
>>>>>>>>>>>                                             -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>>>                                             -Ilib
>>>>>>>>>>>                                             -I../../lib
>>>>>>>>>>>                                             -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>>>                                             -fdiagnostics-color=always
>>>>>>>>>>>                                             -pipe
>>>>>>>>>>>                                             -D_FILE_OFFSET_BITS=64
>>>>>>>>>>>                                             -Wall
>>>>>>>>>>>                                             -Winvalid-pch
>>>>>>>>>>>                                             -Werror -g
>>>>>>>>>>>                                             -D_GNU_SOURCE
>>>>>>>>>>>                                             -O0 -ggdb -Wall
>>>>>>>>>>>                                             -W -fPIC -MD -MQ
>>>>>>>>>>>                                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>>                                             -MF
>>>>>>>>>>>                                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>>>                                             -o
>>>>>>>>>>>                                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>>                                             -c
>>>>>>>>>>>                                             ../../lib/dpdk_pmd_ts.c
>>>>>>>>>>>                                             ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>>>                                             In function
>>>>>>>>>>>                                             ‘test_create_traffic_generator_params’:
>>>>>>>>>>>                                             ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>>>                                             error: format
>>>>>>>>>>>                                             not a string
>>>>>>>>>>>                                             literal and no
>>>>>>>>>>>                                             format arguments
>>>>>>>>>>>                                             [-Werror=format-security]
>>>>>>>>>>>                                             5577 |     rc =
>>>>>>>>>>>                                             te_kvpair_add(result,
>>>>>>>>>>>                                             buf, mode);
>>>>>>>>>>>                                             |     ^~
>>>>>>>>>>>                                             cc1: all
>>>>>>>>>>>                                             warnings being
>>>>>>>>>>>                                             treated as errors
>>>>>>>>>>>                                             ninja: build
>>>>>>>>>>>                                             stopped:
>>>>>>>>>>>                                             subcommand failed.
>>>>>>>>>>>                                             ninja: Entering
>>>>>>>>>>>                                             directory `.'
>>>>>>>>>>>                                             FAILED:
>>>>>>>>>>>                                             lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>>>>>>>>>>                                             cc
>>>>>>>>>>>                                             -Ilib/76b5a35@@ts_dpdk_pmd@sta
>>>>>>>>>>>                                             -Ilib
>>>>>>>>>>>                                             -I../../lib
>>>>>>>>>>>                                             -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>>>>>>>>>>                                             -fdiagnostics-color=always
>>>>>>>>>>>                                             -pipe
>>>>>>>>>>>                                             -D_FILE_OFFSET_BITS=64
>>>>>>>>>>>                                             -Wall
>>>>>>>>>>>                                             -Winvalid-pch
>>>>>>>>>>>                                             -Werror -g
>>>>>>>>>>>                                             -D_GNU_SOURCE
>>>>>>>>>>>                                             -O0 -ggdb -Wall
>>>>>>>>>>>                                             -W -fPIC -MD -MQ
>>>>>>>>>>>                                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>>                                             -MF
>>>>>>>>>>>                                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
>>>>>>>>>>>                                             -o
>>>>>>>>>>>                                             'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>>>>>>>>>>                                             -c
>>>>>>>>>>>                                             ../../lib/dpdk_pmd_ts.c
>>>>>>>>>>>                                             ../../lib/dpdk_pmd_ts.c:
>>>>>>>>>>>                                             In function
>>>>>>>>>>>                                             ‘test_create_traffic_generator_params’:
>>>>>>>>>>>                                             ../../lib/dpdk_pmd_ts.c:5577:5:
>>>>>>>>>>>                                             error: format
>>>>>>>>>>>                                             not a string
>>>>>>>>>>>                                             literal and no
>>>>>>>>>>>                                             format arguments
>>>>>>>>>>>                                             [-Werror=format-security]
>>>>>>>>>>>                                             5577 |     rc =
>>>>>>>>>>>                                             te_kvpair_add(result,
>>>>>>>>>>>                                             buf, mode);
>>>>>>>>>>>                                             |     ^~
>>>>>>>>>>>                                             cc1: all
>>>>>>>>>>>                                             warnings being
>>>>>>>>>>>                                             treated as errors
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                         This error wasn't
>>>>>>>>>>>                                         occurring last week,
>>>>>>>>>>>                                         which was the last
>>>>>>>>>>>                                         time I ran the tests.
>>>>>>>>>>>                                         The TE host and the
>>>>>>>>>>>                                         DUT have GCC v9.4.0
>>>>>>>>>>>                                         installed, and the
>>>>>>>>>>>                                         tester has GCC
>>>>>>>>>>>                                         v11.4.0 installed,
>>>>>>>>>>>                                         if this information
>>>>>>>>>>>                                         is helpful.
>>>>>>>>>>>
>>>>>>>>>>>                                         2. On the Mellanox
>>>>>>>>>>>                                         CX5s, there are over
>>>>>>>>>>>                                         6,000 tests run,
>>>>>>>>>>>                                         which collectively
>>>>>>>>>>>                                         take around 9 hours.
>>>>>>>>>>>                                         Is it possible, and
>>>>>>>>>>>                                         would it make sense,
>>>>>>>>>>>                                         to lower the test
>>>>>>>>>>>                                         coverage and have
>>>>>>>>>>>                                         the test suite run
>>>>>>>>>>>                                         faster?
>>>>>>>>>>>
>>>>>>>>>>>                                         For some context, we
>>>>>>>>>>>                                         run immediate
>>>>>>>>>>>                                         testing on incoming
>>>>>>>>>>>                                         patches for DPDK
>>>>>>>>>>>                                         main and development
>>>>>>>>>>>                                         branches, as well as
>>>>>>>>>>>                                         periodic test runs
>>>>>>>>>>>                                         on the main, stable,
>>>>>>>>>>>                                         and LTS branches.
>>>>>>>>>>>                                         For us to consider
>>>>>>>>>>>                                         including this test
>>>>>>>>>>>                                         suite as part of our
>>>>>>>>>>>                                         immediate testing on
>>>>>>>>>>>                                         patches, we would
>>>>>>>>>>>                                         have to reduce the
>>>>>>>>>>>                                         test coverage to the
>>>>>>>>>>>                                         most important tests.
>>>>>>>>>>>                                         This is primarily to
>>>>>>>>>>>                                         reduce the testing
>>>>>>>>>>>                                         time to, for
>>>>>>>>>>>                                         example, less than
>>>>>>>>>>>                                         30 minutes. Testing
>>>>>>>>>>>                                         on patches can't
>>>>>>>>>>>                                         take too long
>>>>>>>>>>>                                         because the lab can
>>>>>>>>>>>                                         receive numerous
>>>>>>>>>>>                                         patches each day,
>>>>>>>>>>>                                         which each require
>>>>>>>>>>>                                         individual testing runs.
>>>>>>>>>>>
>>>>>>>>>>>                                         At what frequency we
>>>>>>>>>>>                                         run these tests, and
>>>>>>>>>>>                                         on what, still needs
>>>>>>>>>>>                                         to be discussed with
>>>>>>>>>>>                                         the DPDK community,
>>>>>>>>>>>                                         but it would be nice
>>>>>>>>>>>                                         to know if the test
>>>>>>>>>>>                                         suite had a "dial"
>>>>>>>>>>>                                         to control the
>>>>>>>>>>>                                         testing coverage.
>>>>>>>>>>>
>>>>>>>>>>>                                         3. We see a lot of
>>>>>>>>>>>                                         test failures on our
>>>>>>>>>>>                                         Mellanox CX5 NICs.
>>>>>>>>>>>                                         Around 2,300 of
>>>>>>>>>>>                                         ~6,600 tests passed.
>>>>>>>>>>>                                         Is there anything we
>>>>>>>>>>>                                         can do to diagnose
>>>>>>>>>>>                                         these test failures?
>>>>>>>>>>>
>>>>>>>>>>>                                         Thanks,
>>>>>>>>>>>                                         Adam
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                         On Tue, Aug 29, 2023
>>>>>>>>>>>                                         at 8:07 AM Andrew
>>>>>>>>>>>                                         Rybchenko
>>>>>>>>>>>                                         <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>
>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                             I've pushed the
>>>>>>>>>>>                                             fix in main
>>>>>>>>>>>                                             branch and a new
>>>>>>>>>>>                                             tag v1.18.1. It
>>>>>>>>>>>                                             should solve the
>>>>>>>>>>>                                             problem with
>>>>>>>>>>>                                             IPv6 address
>>>>>>>>>>>                                             from DNS.
>>>>>>>>>>>
>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                             On 8/29/23
>>>>>>>>>>>                                             00:05, Andrew
>>>>>>>>>>>                                             Rybchenko wrote:
>>>>>>>>>>>>                                             Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                             > Does the test
>>>>>>>>>>>>                                             engine prefer
>>>>>>>>>>>>                                             to use IPv6
>>>>>>>>>>>>                                             over IPv4 for
>>>>>>>>>>>>                                             initiating the
>>>>>>>>>>>>                                             RCF connection
>>>>>>>>>>>>                                             to the test bed
>>>>>>>>>>>>                                             hosts? And if
>>>>>>>>>>>>                                             so, is there a
>>>>>>>>>>>>                                             way to force it
>>>>>>>>>>>>                                             to use IPv4?
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Brilliant idea.
>>>>>>>>>>>>                                             If DNS returns
>>>>>>>>>>>>                                             both IPv4 and
>>>>>>>>>>>>                                             IPv6 addresses
>>>>>>>>>>>>                                             in your case, I
>>>>>>>>>>>>                                             guess it is the
>>>>>>>>>>>>                                             root cause of
>>>>>>>>>>>>                                             the problem.
>>>>>>>>>>>>                                             Of course, it
>>>>>>>>>>>>                                             is TE problem
>>>>>>>>>>>>                                             since I see
>>>>>>>>>>>>                                             really weird
>>>>>>>>>>>>                                             code in
>>>>>>>>>>>>                                             lib/comm_net_engine/comm_net_engine.c
>>>>>>>>>>>>                                             line 135.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             I've pushed fix
>>>>>>>>>>>>                                             to the branch
>>>>>>>>>>>>                                             user/arybchik/fix_ipv4_only
>>>>>>>>>>>>                                             in
>>>>>>>>>>>>                                             ts-factory/test-environment
>>>>>>>>>>>>                                             repository.
>>>>>>>>>>>>                                             Please, try.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             It is late
>>>>>>>>>>>>                                             night fix with
>>>>>>>>>>>>                                             minimal testing
>>>>>>>>>>>>                                             and no review.
>>>>>>>>>>>>                                             I'll pass it
>>>>>>>>>>>>                                             through review
>>>>>>>>>>>>                                             process
>>>>>>>>>>>>                                             tomorrow and
>>>>>>>>>>>>                                             hopefully it
>>>>>>>>>>>>                                             will be
>>>>>>>>>>>>                                             released in
>>>>>>>>>>>>                                             one-two days.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                             On 8/28/23
>>>>>>>>>>>>                                             18:02, Adam
>>>>>>>>>>>>                                             Hassick wrote:
>>>>>>>>>>>>>                                             Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             We have yet to
>>>>>>>>>>>>>                                             notice a
>>>>>>>>>>>>>                                             distinct
>>>>>>>>>>>>>                                             pattern with
>>>>>>>>>>>>>                                             the failures.
>>>>>>>>>>>>>                                             Sometimes, the
>>>>>>>>>>>>>                                             RCF will start
>>>>>>>>>>>>>                                             and connect
>>>>>>>>>>>>>                                             without issue
>>>>>>>>>>>>>                                             a few times in
>>>>>>>>>>>>>                                             a row before
>>>>>>>>>>>>>                                             failing to
>>>>>>>>>>>>>                                             connect again.
>>>>>>>>>>>>>                                             Once the issue
>>>>>>>>>>>>>                                             begins to
>>>>>>>>>>>>>                                             occur, neither
>>>>>>>>>>>>>                                             rebooting all
>>>>>>>>>>>>>                                             of the hosts
>>>>>>>>>>>>>                                             (test engine
>>>>>>>>>>>>>                                             VM, tester,
>>>>>>>>>>>>>                                             IUT) or
>>>>>>>>>>>>>                                             deleting all
>>>>>>>>>>>>>                                             of the build
>>>>>>>>>>>>>                                             directories
>>>>>>>>>>>>>                                             (suites,
>>>>>>>>>>>>>                                             agents, inst)
>>>>>>>>>>>>>                                             and rebooting
>>>>>>>>>>>>>                                             the hosts
>>>>>>>>>>>>>                                             afterward
>>>>>>>>>>>>>                                             resolves the
>>>>>>>>>>>>>                                             issue. When it
>>>>>>>>>>>>>                                             begins working
>>>>>>>>>>>>>                                             again seems
>>>>>>>>>>>>>                                             very arbitrary
>>>>>>>>>>>>>                                             to us.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             I do usually
>>>>>>>>>>>>>                                             try to
>>>>>>>>>>>>>                                             terminate the
>>>>>>>>>>>>>                                             test engine
>>>>>>>>>>>>>                                             with Ctrl+C,
>>>>>>>>>>>>>                                             but when it
>>>>>>>>>>>>>                                             hangs while
>>>>>>>>>>>>>                                             trying to
>>>>>>>>>>>>>                                             start RCF,
>>>>>>>>>>>>>                                             that does not
>>>>>>>>>>>>>                                             work.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             Does the test
>>>>>>>>>>>>>                                             engine prefer
>>>>>>>>>>>>>                                             to use IPv6
>>>>>>>>>>>>>                                             over IPv4 for
>>>>>>>>>>>>>                                             initiating the
>>>>>>>>>>>>>                                             RCF connection
>>>>>>>>>>>>>                                             to the test
>>>>>>>>>>>>>                                             bed hosts? And
>>>>>>>>>>>>>                                             if so, is
>>>>>>>>>>>>>                                             there a way to
>>>>>>>>>>>>>                                             force it to
>>>>>>>>>>>>>                                             use IPv4?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                              - Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             On Fri, Aug
>>>>>>>>>>>>>                                             25, 2023 at
>>>>>>>>>>>>>                                             1:35 PM Andrew
>>>>>>>>>>>>>                                             Rybchenko
>>>>>>>>>>>>>                                             <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>                                             wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                                 > I'll
>>>>>>>>>>>>>                                                 double-check
>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>                                                 engine on
>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>                                                 20.04 and
>>>>>>>>>>>>>                                                 Ubuntu 22.04.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                                 Done. It
>>>>>>>>>>>>>                                                 works fine
>>>>>>>>>>>>>                                                 for me
>>>>>>>>>>>>>                                                 without
>>>>>>>>>>>>>                                                 any issues.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                                 Have you
>>>>>>>>>>>>>                                                 noticed
>>>>>>>>>>>>>                                                 any
>>>>>>>>>>>>>                                                 pattern
>>>>>>>>>>>>>                                                 when it
>>>>>>>>>>>>>                                                 works or
>>>>>>>>>>>>>                                                 does not work?
>>>>>>>>>>>>>                                                 May be it
>>>>>>>>>>>>>                                                 is a
>>>>>>>>>>>>>                                                 problem of
>>>>>>>>>>>>>                                                 not clean
>>>>>>>>>>>>>                                                 state
>>>>>>>>>>>>>                                                 after
>>>>>>>>>>>>>                                                 termination?
>>>>>>>>>>>>>                                                 Does it
>>>>>>>>>>>>>                                                 work fine
>>>>>>>>>>>>>                                                 the first
>>>>>>>>>>>>>                                                 time after
>>>>>>>>>>>>>                                                 DUTs reboot?
>>>>>>>>>>>>>                                                 How do you
>>>>>>>>>>>>>                                                 terminate
>>>>>>>>>>>>>                                                 testing?
>>>>>>>>>>>>>                                                 It should
>>>>>>>>>>>>>                                                 be done
>>>>>>>>>>>>>                                                 using
>>>>>>>>>>>>>                                                 Ctrl+C in
>>>>>>>>>>>>>                                                 terminal
>>>>>>>>>>>>>                                                 where you
>>>>>>>>>>>>>                                                 execute
>>>>>>>>>>>>>                                                 run.sh
>>>>>>>>>>>>>                                                 command.
>>>>>>>>>>>>>                                                  In this
>>>>>>>>>>>>>                                                 case it
>>>>>>>>>>>>>                                                 should
>>>>>>>>>>>>>                                                 shutdown
>>>>>>>>>>>>>                                                 gracefully
>>>>>>>>>>>>>                                                 and close
>>>>>>>>>>>>>                                                 all test
>>>>>>>>>>>>>                                                 agents and
>>>>>>>>>>>>>                                                 engine
>>>>>>>>>>>>>                                                 applications.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                                 (I'm
>>>>>>>>>>>>>                                                 trying to
>>>>>>>>>>>>>                                                 understand
>>>>>>>>>>>>>                                                 why you've
>>>>>>>>>>>>>                                                 seen many
>>>>>>>>>>>>>                                                 test agent
>>>>>>>>>>>>>                                                 processes.
>>>>>>>>>>>>>                                                 It should
>>>>>>>>>>>>>                                                 not happen.)
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                                 Andrew.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                                 On 8/25/23
>>>>>>>>>>>>>                                                 17:41,
>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>                                                 On
>>>>>>>>>>>>>>                                                 8/25/23
>>>>>>>>>>>>>>                                                 17:06,
>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>                                                 Hassick
>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>                                                 Hi Andrew,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Two of
>>>>>>>>>>>>>>>                                                 our
>>>>>>>>>>>>>>>                                                 systems
>>>>>>>>>>>>>>>                                                 (the
>>>>>>>>>>>>>>>                                                 Test
>>>>>>>>>>>>>>>                                                 Engine
>>>>>>>>>>>>>>>                                                 runner
>>>>>>>>>>>>>>>                                                 and the
>>>>>>>>>>>>>>>                                                 DUT
>>>>>>>>>>>>>>>                                                 host)
>>>>>>>>>>>>>>>                                                 are
>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>>>                                                 20.04
>>>>>>>>>>>>>>>                                                 LTS,
>>>>>>>>>>>>>>>                                                 however
>>>>>>>>>>>>>>>                                                 this
>>>>>>>>>>>>>>>                                                 morning
>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>                                                 noticed
>>>>>>>>>>>>>>>                                                 that the
>>>>>>>>>>>>>>>                                                 tester
>>>>>>>>>>>>>>>                                                 system
>>>>>>>>>>>>>>>                                                 (the one
>>>>>>>>>>>>>>>                                                 having
>>>>>>>>>>>>>>>                                                 issues)
>>>>>>>>>>>>>>>                                                 is
>>>>>>>>>>>>>>>                                                 running
>>>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>>>                                                 22.04 LTS.
>>>>>>>>>>>>>>>                                                 This
>>>>>>>>>>>>>>>                                                 could be
>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>                                                 source
>>>>>>>>>>>>>>>                                                 of the
>>>>>>>>>>>>>>>                                                 problem.
>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>                                                 encountered
>>>>>>>>>>>>>>>                                                 a
>>>>>>>>>>>>>>>                                                 dependency
>>>>>>>>>>>>>>>                                                 issue
>>>>>>>>>>>>>>>                                                 trying
>>>>>>>>>>>>>>>                                                 to run
>>>>>>>>>>>>>>>                                                 the Test
>>>>>>>>>>>>>>>                                                 Engine
>>>>>>>>>>>>>>>                                                 on 22.04
>>>>>>>>>>>>>>>                                                 LTS, so
>>>>>>>>>>>>>>>                                                 I
>>>>>>>>>>>>>>>                                                 downgraded
>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>                                                 system.
>>>>>>>>>>>>>>>                                                 Since
>>>>>>>>>>>>>>>                                                 the
>>>>>>>>>>>>>>>                                                 tester
>>>>>>>>>>>>>>>                                                 is also
>>>>>>>>>>>>>>>                                                 the host
>>>>>>>>>>>>>>>                                                 having
>>>>>>>>>>>>>>>                                                 connection
>>>>>>>>>>>>>>>                                                 issues,
>>>>>>>>>>>>>>>                                                 I will
>>>>>>>>>>>>>>>                                                 try
>>>>>>>>>>>>>>>                                                 downgrading
>>>>>>>>>>>>>>>                                                 that
>>>>>>>>>>>>>>>                                                 system
>>>>>>>>>>>>>>>                                                 to
>>>>>>>>>>>>>>>                                                 20.04,
>>>>>>>>>>>>>>>                                                 and see
>>>>>>>>>>>>>>>                                                 if that
>>>>>>>>>>>>>>>                                                 changes
>>>>>>>>>>>>>>>                                                 anything.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Unlikely,
>>>>>>>>>>>>>>                                                 but who
>>>>>>>>>>>>>>                                                 knows. We
>>>>>>>>>>>>>>                                                 run tests
>>>>>>>>>>>>>>                                                 (DUTs) on
>>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>>                                                 20.04,
>>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>>                                                 22.04,
>>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>>                                                 22.10,
>>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>>                                                 23.04,
>>>>>>>>>>>>>>                                                 Debian 11
>>>>>>>>>>>>>>                                                 and
>>>>>>>>>>>>>>                                                 Fedora 38
>>>>>>>>>>>>>>                                                 every night.
>>>>>>>>>>>>>>                                                 Right now
>>>>>>>>>>>>>>                                                 Debian 11
>>>>>>>>>>>>>>                                                 is used
>>>>>>>>>>>>>>                                                 for test
>>>>>>>>>>>>>>                                                 engine in
>>>>>>>>>>>>>>                                                 nightly
>>>>>>>>>>>>>>                                                 regressions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 I'll
>>>>>>>>>>>>>>                                                 double-check
>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>                                                 engine on
>>>>>>>>>>>>>>                                                 Ubuntu
>>>>>>>>>>>>>>                                                 20.04 and
>>>>>>>>>>>>>>                                                 Ubuntu 22.04.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 I did
>>>>>>>>>>>>>>>                                                 try
>>>>>>>>>>>>>>>                                                 passing
>>>>>>>>>>>>>>>                                                 in the
>>>>>>>>>>>>>>>                                                 "--vg-rcf"
>>>>>>>>>>>>>>>                                                 argument
>>>>>>>>>>>>>>>                                                 to the
>>>>>>>>>>>>>>>                                                 run.sh
>>>>>>>>>>>>>>>                                                 script
>>>>>>>>>>>>>>>                                                 of the
>>>>>>>>>>>>>>>                                                 test
>>>>>>>>>>>>>>>                                                 suite
>>>>>>>>>>>>>>>                                                 after
>>>>>>>>>>>>>>>                                                 installing
>>>>>>>>>>>>>>>                                                 valgrind,
>>>>>>>>>>>>>>>                                                 but
>>>>>>>>>>>>>>>                                                 there
>>>>>>>>>>>>>>>                                                 was no
>>>>>>>>>>>>>>>                                                 additional
>>>>>>>>>>>>>>>                                                 output
>>>>>>>>>>>>>>>                                                 that I saw.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                                 Sorry, I
>>>>>>>>>>>>>>                                                 should
>>>>>>>>>>>>>>                                                 valgrind
>>>>>>>>>>>>>>                                                 output
>>>>>>>>>>>>>>                                                 should be
>>>>>>>>>>>>>>                                                 in
>>>>>>>>>>>>>>                                                 valgrind.te_rcf
>>>>>>>>>>>>>>                                                 (direction
>>>>>>>>>>>>>>                                                 where you
>>>>>>>>>>>>>>                                                 run test
>>>>>>>>>>>>>>                                                 engine).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 I will
>>>>>>>>>>>>>>>                                                 try
>>>>>>>>>>>>>>>                                                 pulling
>>>>>>>>>>>>>>>                                                 in the
>>>>>>>>>>>>>>>                                                 changes
>>>>>>>>>>>>>>>                                                 you've
>>>>>>>>>>>>>>>                                                 pushed
>>>>>>>>>>>>>>>                                                 up, and
>>>>>>>>>>>>>>>                                                 will see
>>>>>>>>>>>>>>>                                                 if that
>>>>>>>>>>>>>>>                                                 fixes
>>>>>>>>>>>>>>>                                                 anything.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 Thanks,
>>>>>>>>>>>>>>>                                                 Adam
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 On Fri,
>>>>>>>>>>>>>>>                                                 Aug 25,
>>>>>>>>>>>>>>>                                                 2023 at
>>>>>>>>>>>>>>>                                                 9:57 AM
>>>>>>>>>>>>>>>                                                 Andrew
>>>>>>>>>>>>>>>                                                 Rybchenko
>>>>>>>>>>>>>>>                                                 <andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>                                                 wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                     Hello
>>>>>>>>>>>>>>>                                                     Adam,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>                                                     8/24/23
>>>>>>>>>>>>>>>                                                     23:54,
>>>>>>>>>>>>>>>                                                     Andrew
>>>>>>>>>>>>>>>                                                     Rybchenko
>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>                                                     I'd
>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>                                                     repeat
>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>                                                     problem
>>>>>>>>>>>>>>>>                                                     locally.
>>>>>>>>>>>>>>>>                                                     Which
>>>>>>>>>>>>>>>>                                                     Linux
>>>>>>>>>>>>>>>>                                                     distro
>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>                                                     running
>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>                                                     engine
>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>                                                     agents?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     In
>>>>>>>>>>>>>>>>                                                     fact
>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>                                                     know
>>>>>>>>>>>>>>>>                                                     one
>>>>>>>>>>>>>>>>                                                     problem
>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>                                                     Debian
>>>>>>>>>>>>>>>>                                                     12
>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>                                                     Fedora
>>>>>>>>>>>>>>>>                                                     38
>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>                                                     we have
>>>>>>>>>>>>>>>>                                                     patch
>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>                                                     review
>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>                                                     fix
>>>>>>>>>>>>>>>>                                                     it,
>>>>>>>>>>>>>>>>                                                     however,
>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>                                                     behaviour
>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>                                                     different
>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>                                                     case,
>>>>>>>>>>>>>>>>                                                     so
>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>                                                     unlike
>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>                                                     same
>>>>>>>>>>>>>>>>                                                     problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>                                                     just
>>>>>>>>>>>>>>>                                                     published
>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>                                                     new
>>>>>>>>>>>>>>>                                                     tag
>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>                                                     fixes
>>>>>>>>>>>>>>>                                                     known
>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>                                                     engine
>>>>>>>>>>>>>>>                                                     side
>>>>>>>>>>>>>>>                                                     problems
>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>                                                     Debian
>>>>>>>>>>>>>>>                                                     12
>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>                                                     Fedora
>>>>>>>>>>>>>>>                                                     38.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     One
>>>>>>>>>>>>>>>>                                                     more
>>>>>>>>>>>>>>>>                                                     idea
>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>                                                     install
>>>>>>>>>>>>>>>>                                                     valgrind
>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>                                                     engine
>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>                                                     option
>>>>>>>>>>>>>>>>                                                     --vg-rcf
>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>                                                     check
>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>                                                     something
>>>>>>>>>>>>>>>>                                                     weird
>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>                                                     happening.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     What
>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>                                                     don't
>>>>>>>>>>>>>>>>                                                     understand
>>>>>>>>>>>>>>>>                                                     right
>>>>>>>>>>>>>>>>                                                     now
>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>                                                     why
>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>                                                     just
>>>>>>>>>>>>>>>>                                                     one
>>>>>>>>>>>>>>>>                                                     failed
>>>>>>>>>>>>>>>>                                                     attempt
>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>                                                     connect
>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>                                                     your
>>>>>>>>>>>>>>>>                                                     log.txt
>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>                                                     then
>>>>>>>>>>>>>>>>                                                     Logger
>>>>>>>>>>>>>>>>                                                     shutdown
>>>>>>>>>>>>>>>>                                                     after
>>>>>>>>>>>>>>>>                                                     9
>>>>>>>>>>>>>>>>                                                     minutes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>                                                     8/24/23
>>>>>>>>>>>>>>>>                                                     23:29,
>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>                                                     Hassick
>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                     Is
>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>                                                     any
>>>>>>>>>>>>>>>>>                                                     firewall
>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     network
>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     hosts
>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>                                                     block
>>>>>>>>>>>>>>>>>                                                     incoming
>>>>>>>>>>>>>>>>>                                                     TCP
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     port
>>>>>>>>>>>>>>>>>                                                     23571
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>>                                                     where
>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     engine?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     Our
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     engine
>>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     testbed
>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     same
>>>>>>>>>>>>>>>>>                                                     subnet.
>>>>>>>>>>>>>>>>>                                                     The
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     does
>>>>>>>>>>>>>>>>>                                                     work
>>>>>>>>>>>>>>>>>                                                     sometimes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                     If
>>>>>>>>>>>>>>>>>                                                     behaviour
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     same
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     next
>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                     kept
>>>>>>>>>>>>>>>>>                                                     running,
>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>                                                     check
>>>>>>>>>>>>>>>>>                                                     using
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                     #
>>>>>>>>>>>>>>>>>                                                     netstat
>>>>>>>>>>>>>>>>>                                                     -tnlp
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>                                                     Test
>>>>>>>>>>>>>>>>>                                                     Agent
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                     listening
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     port
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     establish
>>>>>>>>>>>>>>>>>                                                     TCP
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>                                                     using
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                     $
>>>>>>>>>>>>>>>>>                                                     telnet
>>>>>>>>>>>>>>>>>                                                     iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     23571
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                      >
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     check
>>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>>                                                     TCP
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>                                                     established.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>                                                     was
>>>>>>>>>>>>>>>>>                                                     able
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     replicate
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     same
>>>>>>>>>>>>>>>>>                                                     behavior
>>>>>>>>>>>>>>>>>                                                     again,
>>>>>>>>>>>>>>>>>                                                     where
>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>                                                     hangs
>>>>>>>>>>>>>>>>>                                                     while
>>>>>>>>>>>>>>>>>                                                     RCF
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                     trying
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     start.
>>>>>>>>>>>>>>>>>                                                     Running
>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>                                                     command,
>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     output:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     tcp
>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>                                                      0
>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>                                                      0
>>>>>>>>>>>>>>>>>                                                     0.0.0.0:23571
>>>>>>>>>>>>>>>>>                                                     <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>>>                                                     <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>>>                                                     <http://0.0.0.0:23571>
>>>>>>>>>>>>>>>>>                                                     0.0.0.0:*
>>>>>>>>>>>>>>>>>                                                     LISTEN
>>>>>>>>>>>>>>>>>                                                      18599/ta
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     So
>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>                                                     seems
>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                     listening
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     correct
>>>>>>>>>>>>>>>>>                                                     port.
>>>>>>>>>>>>>>>>>                                                     Additionally,
>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>                                                     was
>>>>>>>>>>>>>>>>>                                                     able
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     connect
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     Tester
>>>>>>>>>>>>>>>>>                                                     machine
>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>                                                     Test
>>>>>>>>>>>>>>>>>                                                     Engine
>>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>>                                                     using
>>>>>>>>>>>>>>>>>                                                     telnet.
>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>                                                     printed
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     PID
>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     process
>>>>>>>>>>>>>>>>>                                                     once
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     was
>>>>>>>>>>>>>>>>>                                                     opened.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>                                                     tried
>>>>>>>>>>>>>>>>>                                                     running
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     "ta"
>>>>>>>>>>>>>>>>>                                                     application
>>>>>>>>>>>>>>>>>                                                     manually
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     command
>>>>>>>>>>>>>>>>>                                                     line,
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>                                                     didn't
>>>>>>>>>>>>>>>>>                                                     print
>>>>>>>>>>>>>>>>>                                                     anything
>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>                                                     all.
>>>>>>>>>>>>>>>>>                                                     Maybe
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     issue
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                     something
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     Test
>>>>>>>>>>>>>>>>>                                                     Engine
>>>>>>>>>>>>>>>>>                                                     side.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                                     Thu,
>>>>>>>>>>>>>>>>>                                                     Aug
>>>>>>>>>>>>>>>>>                                                     24,
>>>>>>>>>>>>>>>>>                                                     2023
>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>                                                     2:35 PM
>>>>>>>>>>>>>>>>>                                                     Andrew
>>>>>>>>>>>>>>>>>                                                     Rybchenko
>>>>>>>>>>>>>>>>>                                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>                                                     Adam,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                         
>>>>>>>>>>>>>>>>>                                                     >
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     tester
>>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>>                                                     (which
>>>>>>>>>>>>>>>>>                                                     appears
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     Peer
>>>>>>>>>>>>>>>>>                                                     agent),
>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>                                                     four
>>>>>>>>>>>>>>>>>                                                     processes
>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>                                                     running,
>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>                                                     look
>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>                                                     processes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     Before
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     next
>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>                                                     I'd
>>>>>>>>>>>>>>>>>                                                     recommend
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     kill
>>>>>>>>>>>>>>>>>                                                     these
>>>>>>>>>>>>>>>>>                                                     processes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     Is
>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>                                                     any
>>>>>>>>>>>>>>>>>                                                     firewall
>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     network
>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     hosts
>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     block
>>>>>>>>>>>>>>>>>                                                     incoming
>>>>>>>>>>>>>>>>>                                                     TCP
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     port
>>>>>>>>>>>>>>>>>                                                     23571
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     where
>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     engine?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     If
>>>>>>>>>>>>>>>>>                                                     behaviour
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     same
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     next
>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     kept
>>>>>>>>>>>>>>>>>                                                     running,
>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>                                                     check
>>>>>>>>>>>>>>>>>                                                     using
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     #
>>>>>>>>>>>>>>>>>                                                     netstat
>>>>>>>>>>>>>>>>>                                                     -tnlp
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>                                                     Test
>>>>>>>>>>>>>>>>>                                                     Agent
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                     listening
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     port
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     establish
>>>>>>>>>>>>>>>>>                                                     TCP
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>                                                     using
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     $
>>>>>>>>>>>>>>>>>                                                     telnet
>>>>>>>>>>>>>>>>>                                                     iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     23571
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     check
>>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>>                                                     TCP
>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>                                                     established.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     Another
>>>>>>>>>>>>>>>>>                                                     idea
>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     login
>>>>>>>>>>>>>>>>>                                                     Tester
>>>>>>>>>>>>>>>>>                                                     under
>>>>>>>>>>>>>>>>>                                                     root
>>>>>>>>>>>>>>>>>                                                     as
>>>>>>>>>>>>>>>>>                                                     testing
>>>>>>>>>>>>>>>>>                                                     does,
>>>>>>>>>>>>>>>>>                                                     get
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     start
>>>>>>>>>>>>>>>>>                                                     TA
>>>>>>>>>>>>>>>>>                                                     command
>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     log
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>                                                     by
>>>>>>>>>>>>>>>>>                                                     hands
>>>>>>>>>>>>>>>>>                                                     without
>>>>>>>>>>>>>>>>>                                                     -n and
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     remove
>>>>>>>>>>>>>>>>>                                                     extra
>>>>>>>>>>>>>>>>>                                                     escaping.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     #
>>>>>>>>>>>>>>>>>                                                     sudo
>>>>>>>>>>>>>>>>>                                                     PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>>>                                                     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>>>>>>>>>>                                                     /tmp/linux_x86_root_76872_1692885663_1/ta
>>>>>>>>>>>>>>>>>                                                     Peer
>>>>>>>>>>>>>>>>>                                                     23571
>>>>>>>>>>>>>>>>>                                                     host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     Hopefully
>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>                                                     case
>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>                                                     directory
>>>>>>>>>>>>>>>>>                                                     remains
>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>                                                     /tmp
>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>                                                     don't
>>>>>>>>>>>>>>>>>                                                     need
>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>                                                     copy
>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>                                                     as
>>>>>>>>>>>>>>>>>                                                     testing
>>>>>>>>>>>>>>>>>                                                     does.
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     May
>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>                                                     output
>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>                                                     shed
>>>>>>>>>>>>>>>>>                                                     some
>>>>>>>>>>>>>>>>>                                                     light
>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>                                                     what's
>>>>>>>>>>>>>>>>>                                                     going
>>>>>>>>>>>>>>>>>                                                     on.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>                                                     8/24/23
>>>>>>>>>>>>>>>>>                                                     17:30,
>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>                                                     Hassick
>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>>                                                     Andrew,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     This
>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     output
>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     terminal
>>>>>>>>>>>>>>>>>>                                                     when
>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>                                                     failure
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     occurs,
>>>>>>>>>>>>>>>>>>                                                     after
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>>                                                     binaries
>>>>>>>>>>>>>>>>>>                                                     build
>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>                                                     engine
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     starts:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     Platform
>>>>>>>>>>>>>>>>>>                                                     default
>>>>>>>>>>>>>>>>>>                                                     build
>>>>>>>>>>>>>>>>>>                                                     -
>>>>>>>>>>>>>>>>>>                                                     pass
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     Simple
>>>>>>>>>>>>>>>>>>                                                     RCF
>>>>>>>>>>>>>>>>>>                                                     consistency
>>>>>>>>>>>>>>>>>>                                                     check
>>>>>>>>>>>>>>>>>>                                                     succeeded
>>>>>>>>>>>>>>>>>>                                                     --->>>
>>>>>>>>>>>>>>>>>>                                                     Starting
>>>>>>>>>>>>>>>>>>                                                     Logger...done
>>>>>>>>>>>>>>>>>>                                                     --->>>
>>>>>>>>>>>>>>>>>>                                                     Starting
>>>>>>>>>>>>>>>>>>                                                     RCF...rcf_net_engine_connect():
>>>>>>>>>>>>>>>>>>                                                     Connection
>>>>>>>>>>>>>>>>>>                                                     timed
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     out
>>>>>>>>>>>>>>>>>>                                                     iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>>                                                     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     Then,
>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>                                                     hangs
>>>>>>>>>>>>>>>>>>                                                     here
>>>>>>>>>>>>>>>>>>                                                     until
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                                     kill
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     "te_rcf"
>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>                                                     "te_tee"
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     processes.
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                                     let
>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>                                                     hang
>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>                                                     around
>>>>>>>>>>>>>>>>>>                                                     9
>>>>>>>>>>>>>>>>>>                                                     minutes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     tester
>>>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>>>                                                     (which
>>>>>>>>>>>>>>>>>>                                                     appears
>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     Peer
>>>>>>>>>>>>>>>>>>                                                     agent),
>>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     four
>>>>>>>>>>>>>>>>>>                                                     processes
>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>>                                                     running,
>>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>>                                                     look
>>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     processes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     ta.Peer
>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>                                                     an
>>>>>>>>>>>>>>>>>>                                                     empty
>>>>>>>>>>>>>>>>>>                                                     file.
>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>                                                     attached
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     log.txt
>>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>                                                     run.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                      -
>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>                                                     Thu,
>>>>>>>>>>>>>>>>>>                                                     Aug
>>>>>>>>>>>>>>>>>>                                                     24,
>>>>>>>>>>>>>>>>>>                                                     2023
>>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>>                                                     4:22��AM
>>>>>>>>>>>>>>>>>>                                                     Andrew
>>>>>>>>>>>>>>>>>>                                                     Rybchenko
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>>                                                     Adam,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     Yes,
>>>>>>>>>>>>>>>>>>                                                     TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     seconds.
>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>                                                     double-checked
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>                                                     goes
>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>                                                     'copy_timeout'
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     ts-conf/rcf.conf.
>>>>>>>>>>>>>>>>>>                                                     Description
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     says
>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>                                                     copy_timeout
>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     seconds
>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>                                                     implementation
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     lib/rcfunix/rcfunix.c
>>>>>>>>>>>>>>>>>>                                                     passes
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     value
>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>                                                     select()
>>>>>>>>>>>>>>>>>>                                                     tv_sec.
>>>>>>>>>>>>>>>>>>                                                     Theoretically
>>>>>>>>>>>>>>>>>>                                                     select()
>>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>                                                     interrupted
>>>>>>>>>>>>>>>>>>                                                     by
>>>>>>>>>>>>>>>>>>                                                     signal,
>>>>>>>>>>>>>>>>>>                                                     but I
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     think
>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>                                                     unlikely
>>>>>>>>>>>>>>>>>>                                                     here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>>>                                                     not
>>>>>>>>>>>>>>>>>>                                                     sure
>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>                                                     understand
>>>>>>>>>>>>>>>>>>                                                     what
>>>>>>>>>>>>>>>>>>                                                     do
>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>                                                     mean
>>>>>>>>>>>>>>>>>>                                                     by
>>>>>>>>>>>>>>>>>>                                                     RCF
>>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>>                                                     timeout.
>>>>>>>>>>>>>>>>>>                                                     Does
>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>                                                     happen
>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>                                                     TE
>>>>>>>>>>>>>>>>>>                                                     startup
>>>>>>>>>>>>>>>>>>                                                     when
>>>>>>>>>>>>>>>>>>                                                     RCF
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     starts
>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>                                                     agents.
>>>>>>>>>>>>>>>>>>                                                     If
>>>>>>>>>>>>>>>>>>                                                     so,
>>>>>>>>>>>>>>>>>>                                                     TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>>                                                     help.
>>>>>>>>>>>>>>>>>>                                                     Or
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     does
>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>                                                     happen
>>>>>>>>>>>>>>>>>>                                                     when
>>>>>>>>>>>>>>>>>>                                                     tests
>>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     progress,
>>>>>>>>>>>>>>>>>>                                                     e.g.
>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>                                                     middle
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>                                                     test.
>>>>>>>>>>>>>>>>>>                                                     If
>>>>>>>>>>>>>>>>>>                                                     so,
>>>>>>>>>>>>>>>>>>                                                     TE_RCFUNIX_TIMEOUT
>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>                                                     unrelated
>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>                                                     most
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     likely
>>>>>>>>>>>>>>>>>>                                                     either
>>>>>>>>>>>>>>>>>>                                                     host
>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>>                                                     dies
>>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>>                                                     itself
>>>>>>>>>>>>>>>>>>                                                     crashes.
>>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>>                                                     would
>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>                                                     easier
>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>                                                     me
>>>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>>>                                                     classify
>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>                                                     share
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     text
>>>>>>>>>>>>>>>>>>                                                     log
>>>>>>>>>>>>>>>>>>                                                     (log.txt,
>>>>>>>>>>>>>>>>>>                                                     full
>>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>>                                                     just
>>>>>>>>>>>>>>>>>>                                                     corresponding
>>>>>>>>>>>>>>>>>>                                                     fragment
>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     some
>>>>>>>>>>>>>>>>>>                                                     context).
>>>>>>>>>>>>>>>>>>                                                     Also
>>>>>>>>>>>>>>>>>>                                                     content
>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>                                                     ta.DPDK
>>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>>                                                     ta.Peer
>>>>>>>>>>>>>>>>>>                                                     file
>>>>>>>>>>>>>>>>>>                                                     depending
>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>>                                                     agent
>>>>>>>>>>>>>>>>>>                                                     has
>>>>>>>>>>>>>>>>>>                                                     problems
>>>>>>>>>>>>>>>>>>                                                     could
>>>>>>>>>>>>>>>>>>                                                     shed
>>>>>>>>>>>>>>>>>>                                                     some
>>>>>>>>>>>>>>>>>>                                                     light.
>>>>>>>>>>>>>>>>>>                                                     Corresponding
>>>>>>>>>>>>>>>>>>                                                     files
>>>>>>>>>>>>>>>>>>                                                     contain
>>>>>>>>>>>>>>>>>>                                                     stdout/stderr
>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>                                                     agents.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>                                                     8/23/23
>>>>>>>>>>>>>>>>>>                                                     17:45,
>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>                                                     Hassick
>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>>>                                                     Andrew,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>>                                                     set
>>>>>>>>>>>>>>>>>>>                                                     up
>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>                                                     rig
>>>>>>>>>>>>>>>>>>>                                                     repository
>>>>>>>>>>>>>>>>>>>                                                     here,
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                     have
>>>>>>>>>>>>>>>>>>>                                                     created
>>>>>>>>>>>>>>>>>>>                                                     configurations
>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>>>                                                     development
>>>>>>>>>>>>>>>>>>>                                                     testbed
>>>>>>>>>>>>>>>>>>>                                                     based
>>>>>>>>>>>>>>>>>>>                                                     off
>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     examples.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     We've
>>>>>>>>>>>>>>>>>>>                                                     been
>>>>>>>>>>>>>>>>>>>                                                     able
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                     get
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>                                                     suite
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>                                                     manually
>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>                                                     Mellanox
>>>>>>>>>>>>>>>>>>>                                                     CX5
>>>>>>>>>>>>>>>>>>>                                                     devices
>>>>>>>>>>>>>>>>>>>                                                     once.
>>>>>>>>>>>>>>>>>>>                                                     However,
>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>>>                                                     running
>>>>>>>>>>>>>>>>>>>                                                     into
>>>>>>>>>>>>>>>>>>>                                                     an
>>>>>>>>>>>>>>>>>>>                                                     issue
>>>>>>>>>>>>>>>>>>>                                                     where,
>>>>>>>>>>>>>>>>>>>                                                     when
>>>>>>>>>>>>>>>>>>>                                                     RCF
>>>>>>>>>>>>>>>>>>>                                                     starts,
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     RCF
>>>>>>>>>>>>>>>>>>>                                                     connection
>>>>>>>>>>>>>>>>>>>                                                     times
>>>>>>>>>>>>>>>>>>>                                                     out
>>>>>>>>>>>>>>>>>>>                                                     very
>>>>>>>>>>>>>>>>>>>                                                     frequently.
>>>>>>>>>>>>>>>>>>>                                                     We
>>>>>>>>>>>>>>>>>>>                                                     aren't
>>>>>>>>>>>>>>>>>>>                                                     sure
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     why
>>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     case.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>>>                                                     works
>>>>>>>>>>>>>>>>>>>                                                     sometimes,
>>>>>>>>>>>>>>>>>>>                                                     but
>>>>>>>>>>>>>>>>>>>                                                     most
>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     time
>>>>>>>>>>>>>>>>>>>                                                     when
>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>                                                     engine,
>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>                                                     encounters
>>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>>                                                     issue.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>>                                                     tried
>>>>>>>>>>>>>>>>>>>                                                     changing
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     RCF
>>>>>>>>>>>>>>>>>>>                                                     port
>>>>>>>>>>>>>>>>>>>                                                     by
>>>>>>>>>>>>>>>>>>>                                                     setting
>>>>>>>>>>>>>>>>>>>                                                     "TE_RCF_PORT=<some
>>>>>>>>>>>>>>>>>>>                                                     port
>>>>>>>>>>>>>>>>>>>                                                     number>"
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                     rebooting
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     testbed
>>>>>>>>>>>>>>>>>>>                                                     machines.
>>>>>>>>>>>>>>>>>>>                                                     Neither
>>>>>>>>>>>>>>>>>>>                                                     seems
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                     fix
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     issue.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>>>                                                     also
>>>>>>>>>>>>>>>>>>>                                                     seems
>>>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     timeout
>>>>>>>>>>>>>>>>>>>                                                     takes
>>>>>>>>>>>>>>>>>>>                                                     far
>>>>>>>>>>>>>>>>>>>                                                     longer
>>>>>>>>>>>>>>>>>>>                                                     than
>>>>>>>>>>>>>>>>>>>                                                     60
>>>>>>>>>>>>>>>>>>>                                                     seconds,
>>>>>>>>>>>>>>>>>>>                                                     even
>>>>>>>>>>>>>>>>>>>                                                     when
>>>>>>>>>>>>>>>>>>>                                                     running
>>>>>>>>>>>>>>>>>>>                                                     "export
>>>>>>>>>>>>>>>>>>>                                                     TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     before
>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>                                                     suite.
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>                                                     assume
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     unit
>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>>                                                     variable
>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>                                                     seconds?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                     Thanks,
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>>                                                     Mon,
>>>>>>>>>>>>>>>>>>>                                                     Aug
>>>>>>>>>>>>>>>>>>>                                                     21,
>>>>>>>>>>>>>>>>>>>                                                     2023
>>>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>>>                                                     10:19 AM
>>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>>                                                     Hassick
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     <ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>>
>>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>>>                                                     Andrew,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                     Thanks,
>>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>>                                                     cloned
>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>                                                     example
>>>>>>>>>>>>>>>>>>>                                                     repository
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                     will
>>>>>>>>>>>>>>>>>>>                                                     start
>>>>>>>>>>>>>>>>>>>                                                     setting
>>>>>>>>>>>>>>>>>>>                                                     up
>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>                                                     configuration
>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>>>                                                     development
>>>>>>>>>>>>>>>>>>>                                                     testbed
>>>>>>>>>>>>>>>>>>>                                                     today.
>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>                                                     let
>>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>>                                                     know
>>>>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>                                                     into
>>>>>>>>>>>>>>>>>>>                                                     any
>>>>>>>>>>>>>>>>>>>                                                     difficulties
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>>>                                                     have
>>>>>>>>>>>>>>>>>>>                                                     any
>>>>>>>>>>>>>>>>>>>                                                     questions.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                      -
>>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>>                                                     Sun,
>>>>>>>>>>>>>>>>>>>                                                     Aug
>>>>>>>>>>>>>>>>>>>                                                     20,
>>>>>>>>>>>>>>>>>>>                                                     2023
>>>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>>>                                                     4:40 AM
>>>>>>>>>>>>>>>>>>>                                                     Andrew
>>>>>>>>>>>>>>>>>>>                                                     Rybchenko
>>>>>>>>>>>>>>>>>>>                                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>>>                                                     Adam,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>>                                                     published
>>>>>>>>>>>>>>>>>>>                                                     https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>>>>>>>>>>                                                     <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>>>>>>>>>>                                                     <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     Hopefully
>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>                                                     will
>>>>>>>>>>>>>>>>>>>                                                     help
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                     define
>>>>>>>>>>>>>>>>>>>                                                     your
>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>                                                     rigs
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     successfully
>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>                                                     some
>>>>>>>>>>>>>>>>>>>                                                     tests
>>>>>>>>>>>>>>>>>>>                                                     manually.
>>>>>>>>>>>>>>>>>>>                                                     Feel
>>>>>>>>>>>>>>>>>>>                                                     free
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     ask
>>>>>>>>>>>>>>>>>>>                                                     any
>>>>>>>>>>>>>>>>>>>                                                     questions
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>                                                     answer
>>>>>>>>>>>>>>>>>>>                                                     here
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                     try
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     update
>>>>>>>>>>>>>>>>>>>                                                     documentation.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     Meanwhile
>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>                                                     prepare
>>>>>>>>>>>>>>>>>>>                                                     missing
>>>>>>>>>>>>>>>>>>>                                                     bits
>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>                                                     steps
>>>>>>>>>>>>>>>>>>>                                                     (2)
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     (3).
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     Hopefully
>>>>>>>>>>>>>>>>>>>                                                     everything
>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>                                                     place
>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>                                                     step
>>>>>>>>>>>>>>>>>>>                                                     (4),
>>>>>>>>>>>>>>>>>>>                                                     but
>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     need
>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>                                                     make
>>>>>>>>>>>>>>>>>>>                                                     steps
>>>>>>>>>>>>>>>>>>>                                                     (2)
>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>                                                     (3)
>>>>>>>>>>>>>>>>>>>                                                     first.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>>                                                     8/18/23
>>>>>>>>>>>>>>>>>>>                                                     21:40,
>>>>>>>>>>>>>>>>>>>                                                     Andrew
>>>>>>>>>>>>>>>>>>>                                                     Rybchenko
>>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>>>>                                                     Adam,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     >
>>>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>>>                                                     conferred
>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>                                                     rest
>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>                                                     team,
>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     think
>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>                                                     would
>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>                                                     best
>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>                                                     move
>>>>>>>>>>>>>>>>>>>>                                                     forward
>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>                                                     mainly
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     option
>>>>>>>>>>>>>>>>>>>>                                                     B.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     OK,
>>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>>                                                     provide
>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>                                                     sample
>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>                                                     Monday
>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>                                                     you.
>>>>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     almost
>>>>>>>>>>>>>>>>>>>>                                                     ready
>>>>>>>>>>>>>>>>>>>>                                                     right
>>>>>>>>>>>>>>>>>>>>                                                     now,
>>>>>>>>>>>>>>>>>>>>                                                     but
>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>                                                     need
>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>                                                     double-check
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>                                                     before
>>>>>>>>>>>>>>>>>>>>                                                     publishing.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     Regards,
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>>>                                                     8/17/23
>>>>>>>>>>>>>>>>>>>>                                                     20:03,
>>>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>>>                                                     Hassick
>>>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>>>                                                     Hi
>>>>>>>>>>>>>>>>>>>>>                                                     Andrew,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>>>>>>                                                     adding
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     CI
>>>>>>>>>>>>>>>>>>>>>                                                     mailing
>>>>>>>>>>>>>>>>>>>>>                                                     list
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     conversation.
>>>>>>>>>>>>>>>>>>>>>                                                     Others
>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     community
>>>>>>>>>>>>>>>>>>>>>                                                     might
>>>>>>>>>>>>>>>>>>>>>                                                     find
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>>>>                                                     conversation
>>>>>>>>>>>>>>>>>>>>>                                                     valuable.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     We
>>>>>>>>>>>>>>>>>>>>>                                                     do
>>>>>>>>>>>>>>>>>>>>>                                                     want
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>>>                                                     testing
>>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>>>                                                     regular
>>>>>>>>>>>>>>>>>>>>>                                                     basis.
>>>>>>>>>>>>>>>>>>>>>                                                     The
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     Jenkins
>>>>>>>>>>>>>>>>>>>>>                                                     integration
>>>>>>>>>>>>>>>>>>>>>                                                     will
>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>                                                     very
>>>>>>>>>>>>>>>>>>>>>                                                     useful
>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>                                                     us,
>>>>>>>>>>>>>>>>>>>>>                                                     as
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     most
>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>>>>>                                                     CI
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     orchestrated
>>>>>>>>>>>>>>>>>>>>>                                                     by
>>>>>>>>>>>>>>>>>>>>>                                                     Jenkins.
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     I've
>>>>>>>>>>>>>>>>>>>>>                                                     conferred
>>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     rest
>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     team,
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     think
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     would
>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>                                                     best
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     move
>>>>>>>>>>>>>>>>>>>>>                                                     forward
>>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>>                                                     mainly
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     option
>>>>>>>>>>>>>>>>>>>>>                                                     B.
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     If
>>>>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>>>>                                                     would
>>>>>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     know
>>>>>>>>>>>>>>>>>>>>>                                                     anything
>>>>>>>>>>>>>>>>>>>>>                                                     about
>>>>>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     testbeds
>>>>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>>>>                                                     would
>>>>>>>>>>>>>>>>>>>>>                                                     help
>>>>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>>                                                     creating
>>>>>>>>>>>>>>>>>>>>>                                                     an
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     example
>>>>>>>>>>>>>>>>>>>>>                                                     ts-rigs
>>>>>>>>>>>>>>>>>>>>>                                                     repo,
>>>>>>>>>>>>>>>>>>>>>                                                     I'd
>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>                                                     happy
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     answer
>>>>>>>>>>>>>>>>>>>>>                                                     any
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     questions
>>>>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>>>>                                                     have.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     We
>>>>>>>>>>>>>>>>>>>>>                                                     have
>>>>>>>>>>>>>>>>>>>>>                                                     multiple
>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>                                                     rigs
>>>>>>>>>>>>>>>>>>>>>                                                     (we
>>>>>>>>>>>>>>>>>>>>>                                                     call
>>>>>>>>>>>>>>>>>>>>>                                                     these
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     "DUT-tester
>>>>>>>>>>>>>>>>>>>>>                                                     pairs")
>>>>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>>>>>                                                     existing
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     hardware
>>>>>>>>>>>>>>>>>>>>>                                                     testing
>>>>>>>>>>>>>>>>>>>>>                                                     on,
>>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>>                                                     differing
>>>>>>>>>>>>>>>>>>>>>                                                     network
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     hardware
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     CPU
>>>>>>>>>>>>>>>>>>>>>                                                     architecture.
>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>                                                     figured
>>>>>>>>>>>>>>>>>>>>>                                                     this
>>>>>>>>>>>>>>>>>>>>>                                                     might
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>                                                     an
>>>>>>>>>>>>>>>>>>>>>                                                     important
>>>>>>>>>>>>>>>>>>>>>                                                     detail.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     Thanks,
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>>>>                                                     Thu,
>>>>>>>>>>>>>>>>>>>>>                                                     Aug
>>>>>>>>>>>>>>>>>>>>>                                                     17,
>>>>>>>>>>>>>>>>>>>>>                                                     2023
>>>>>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>>>>>                                                     11:44 AM
>>>>>>>>>>>>>>>>>>>>>                                                     Andrew
>>>>>>>>>>>>>>>>>>>>>                                                     Rybchenko
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:andrew.rybchenko@oktetlabs.ru>>
>>>>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Greatings
>>>>>>>>>>>>>>>>>>>>>                                                     Adam,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>>>>>>                                                     happy
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     hear
>>>>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>>>>                                                     you're
>>>>>>>>>>>>>>>>>>>>>                                                     trying
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     bring
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     up.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     As
>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>                                                     understand
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     final
>>>>>>>>>>>>>>>>>>>>>                                                     goal
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     regular
>>>>>>>>>>>>>>>>>>>>>                                                     basis.
>>>>>>>>>>>>>>>>>>>>>                                                     So,
>>>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>>>                                                     need
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     make
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     properly
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     very
>>>>>>>>>>>>>>>>>>>>>                                                     beginning.
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Bring
>>>>>>>>>>>>>>>>>>>>>                                                     up
>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>                                                     all
>>>>>>>>>>>>>>>>>>>>>                                                     features
>>>>>>>>>>>>>>>>>>>>>                                                     consists
>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>                                                     4
>>>>>>>>>>>>>>>>>>>>>                                                     steps:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     1.
>>>>>>>>>>>>>>>>>>>>>                                                     Create
>>>>>>>>>>>>>>>>>>>>>                                                     site-specific
>>>>>>>>>>>>>>>>>>>>>                                                     repository
>>>>>>>>>>>>>>>>>>>>>                                                     (we
>>>>>>>>>>>>>>>>>>>>>                                                     call
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     ts-rigs)
>>>>>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>>>>>                                                     contains
>>>>>>>>>>>>>>>>>>>>>                                                     information
>>>>>>>>>>>>>>>>>>>>>                                                     about
>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     rigs
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     other
>>>>>>>>>>>>>>>>>>>>>                                                     site-specific
>>>>>>>>>>>>>>>>>>>>>                                                     information
>>>>>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     where
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     send
>>>>>>>>>>>>>>>>>>>>>                                                     mails,
>>>>>>>>>>>>>>>>>>>>>                                                     where
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     store
>>>>>>>>>>>>>>>>>>>>>                                                     logs
>>>>>>>>>>>>>>>>>>>>>                                                     etc.
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     It
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     required
>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>                                                     manual
>>>>>>>>>>>>>>>>>>>>>                                                     execution
>>>>>>>>>>>>>>>>>>>>>                                                     as
>>>>>>>>>>>>>>>>>>>>>                                                     well,
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     since
>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>                                                     rigs
>>>>>>>>>>>>>>>>>>>>>                                                     description
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     essential.
>>>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     return
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     topic
>>>>>>>>>>>>>>>>>>>>>                                                     below.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     2.
>>>>>>>>>>>>>>>>>>>>>                                                     Setup
>>>>>>>>>>>>>>>>>>>>>                                                     logs
>>>>>>>>>>>>>>>>>>>>>                                                     storage
>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>                                                     automated
>>>>>>>>>>>>>>>>>>>>>                                                     runs.
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Basically
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>>>                                                     disk
>>>>>>>>>>>>>>>>>>>>>                                                     space
>>>>>>>>>>>>>>>>>>>>>                                                     plus
>>>>>>>>>>>>>>>>>>>>>                                                     apache2
>>>>>>>>>>>>>>>>>>>>>                                                     web
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     server
>>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>>                                                     few
>>>>>>>>>>>>>>>>>>>>>                                                     CGI
>>>>>>>>>>>>>>>>>>>>>                                                     scripts
>>>>>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>>>>>                                                     help
>>>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>>>                                                     lot
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     save
>>>>>>>>>>>>>>>>>>>>>                                                     disk
>>>>>>>>>>>>>>>>>>>>>                                                     space.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     3.
>>>>>>>>>>>>>>>>>>>>>                                                     Setup
>>>>>>>>>>>>>>>>>>>>>                                                     Bublik
>>>>>>>>>>>>>>>>>>>>>                                                     web
>>>>>>>>>>>>>>>>>>>>>                                                     application
>>>>>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>>>>>                                                     provides
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     web
>>>>>>>>>>>>>>>>>>>>>                                                     interface
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     view
>>>>>>>>>>>>>>>>>>>>>                                                     testing
>>>>>>>>>>>>>>>>>>>>>                                                     results.
>>>>>>>>>>>>>>>>>>>>>                                                     Same
>>>>>>>>>>>>>>>>>>>>>                                                     as
>>>>>>>>>>>>>>>>>>>>>                                                     https://ts-factory.io/bublik
>>>>>>>>>>>>>>>>>>>>>                                                     <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>>>                                                     <https://ts-factory.io/bublik>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     4.
>>>>>>>>>>>>>>>>>>>>>                                                     Setup
>>>>>>>>>>>>>>>>>>>>>                                                     Jenkins
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>>>                                                     tests
>>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>>                                                     regularly,
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     save
>>>>>>>>>>>>>>>>>>>>>                                                     logs
>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>                                                     log
>>>>>>>>>>>>>>>>>>>>>                                                     storage
>>>>>>>>>>>>>>>>>>>>>                                                     (2)
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     import
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     bublik
>>>>>>>>>>>>>>>>>>>>>                                                     (3).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Last
>>>>>>>>>>>>>>>>>>>>>                                                     few
>>>>>>>>>>>>>>>>>>>>>                                                     month
>>>>>>>>>>>>>>>>>>>>>                                                     we
>>>>>>>>>>>>>>>>>>>>>                                                     spent
>>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>>>>>                                                     homework
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     make
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     simpler
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     bring
>>>>>>>>>>>>>>>>>>>>>                                                     up
>>>>>>>>>>>>>>>>>>>>>                                                     automated
>>>>>>>>>>>>>>>>>>>>>                                                     execution
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     using
>>>>>>>>>>>>>>>>>>>>>                                                     Jenkins
>>>>>>>>>>>>>>>>>>>>>                                                     -
>>>>>>>>>>>>>>>>>>>>>                                                     https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>>>>>>>>>>                                                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>>>                                                     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Corresponding
>>>>>>>>>>>>>>>>>>>>>                                                     bits
>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>                                                     dpdk-ethdev-ts
>>>>>>>>>>>>>>>>>>>>>                                                     will
>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     available
>>>>>>>>>>>>>>>>>>>>>                                                     tomorrow.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Let's
>>>>>>>>>>>>>>>>>>>>>                                                     return
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     step
>>>>>>>>>>>>>>>>>>>>>                                                     (1).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Unfortunately
>>>>>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     no
>>>>>>>>>>>>>>>>>>>>>                                                     publicly
>>>>>>>>>>>>>>>>>>>>>                                                     available
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     example
>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     ts-rigs
>>>>>>>>>>>>>>>>>>>>>                                                     repository
>>>>>>>>>>>>>>>>>>>>>                                                     since
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     sensitive
>>>>>>>>>>>>>>>>>>>>>                                                     site-specific
>>>>>>>>>>>>>>>>>>>>>                                                     information
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     located
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     there.
>>>>>>>>>>>>>>>>>>>>>                                                     But
>>>>>>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>>>>>>                                                     ready
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     help
>>>>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     create
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>                                                     UNH.
>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>>>>>                                                     two
>>>>>>>>>>>>>>>>>>>>>                                                     options
>>>>>>>>>>>>>>>>>>>>>                                                     here:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     (A)
>>>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>>>                                                     ask
>>>>>>>>>>>>>>>>>>>>>                                                     questions
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     based
>>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>>                                                     your
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     answers
>>>>>>>>>>>>>>>>>>>>>                                                     will
>>>>>>>>>>>>>>>>>>>>>                                                     create
>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>                                                     first
>>>>>>>>>>>>>>>>>>>>>                                                     draft
>>>>>>>>>>>>>>>>>>>>>                                                     with
>>>>>>>>>>>>>>>>>>>>>                                                     my
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     comments.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     (B)
>>>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>>>                                                     make
>>>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>>>                                                     template/example
>>>>>>>>>>>>>>>>>>>>>                                                     ts-rigs
>>>>>>>>>>>>>>>>>>>>>                                                     repo,
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     publish
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     you'll
>>>>>>>>>>>>>>>>>>>>>                                                     create
>>>>>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>>>>>                                                     ts-rigs
>>>>>>>>>>>>>>>>>>>>>                                                     based
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>>                                                     it.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Of
>>>>>>>>>>>>>>>>>>>>>                                                     course,
>>>>>>>>>>>>>>>>>>>>>                                                     I'll
>>>>>>>>>>>>>>>>>>>>>                                                     help
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     debug
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     finally
>>>>>>>>>>>>>>>>>>>>>                                                     bring
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     up
>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>                                                     any
>>>>>>>>>>>>>>>>>>>>>                                                     case.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     (A)
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>>>                                                     bit
>>>>>>>>>>>>>>>>>>>>>                                                     simpler
>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>                                                     me
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     you,
>>>>>>>>>>>>>>>>>>>>>                                                     but
>>>>>>>>>>>>>>>>>>>>>                                                     (B)
>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     a
>>>>>>>>>>>>>>>>>>>>>                                                     bit
>>>>>>>>>>>>>>>>>>>>>                                                     more
>>>>>>>>>>>>>>>>>>>>>                                                     generic
>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>                                                     will
>>>>>>>>>>>>>>>>>>>>>                                                     help
>>>>>>>>>>>>>>>>>>>>>                                                     other
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     potential
>>>>>>>>>>>>>>>>>>>>>                                                     users
>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>                                                     bring
>>>>>>>>>>>>>>>>>>>>>                                                     it
>>>>>>>>>>>>>>>>>>>>>                                                     up.
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     We
>>>>>>>>>>>>>>>>>>>>>                                                     can
>>>>>>>>>>>>>>>>>>>>>                                                     combine
>>>>>>>>>>>>>>>>>>>>>                                                     (A)+(B).
>>>>>>>>>>>>>>>>>>>>>                                                     I.e.
>>>>>>>>>>>>>>>>>>>>>                                                     start
>>>>>>>>>>>>>>>>>>>>>                                                     from
>>>>>>>>>>>>>>>>>>>>>                                                     (A).
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     What
>>>>>>>>>>>>>>>>>>>>>                                                     do
>>>>>>>>>>>>>>>>>>>>>                                                     you
>>>>>>>>>>>>>>>>>>>>>                                                     think?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Thanks,
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     Andrew.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>>>>                                                     8/17/23
>>>>>>>>>>>>>>>>>>>>>                                                     15:18,
>>>>>>>>>>>>>>>>>>>>>                                                     Konstantin
>>>>>>>>>>>>>>>>>>>>>                                                     Ushakov
>>>>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>>>>                                                     Greetings
>>>>>>>>>>>>>>>>>>>>>>                                                     Adam,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>                                                     Thanks
>>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>>                                                     contacting
>>>>>>>>>>>>>>>>>>>>>>                                                     us.
>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>                                                     copy
>>>>>>>>>>>>>>>>>>>>>>                                                     Andrew
>>>>>>>>>>>>>>>>>>>>>>                                                     who
>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>                                                     would
>>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>>                                                     happy
>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>                                                     help
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>                                                     Thanks,
>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>                                                     Konstantin
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                     On
>>>>>>>>>>>>>>>>>>>>>>>                                                     16
>>>>>>>>>>>>>>>>>>>>>>>                                                     Aug
>>>>>>>>>>>>>>>>>>>>>>>                                                     2023,
>>>>>>>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>>>>>>>                                                     21:50,
>>>>>>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>>>>>>                                                     Hassick
>>>>>>>>>>>>>>>>>>>>>>>                                                     <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>>                                                     wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     Greetings
>>>>>>>>>>>>>>>>>>>>>>>                                                     Konstantin,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>>                                                     am
>>>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     process
>>>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>>>                                                     setting
>>>>>>>>>>>>>>>>>>>>>>>                                                     up
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     DPDK
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     Poll
>>>>>>>>>>>>>>>>>>>>>>>                                                     Mode
>>>>>>>>>>>>>>>>>>>>>>>                                                     Driver
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     suite
>>>>>>>>>>>>>>>>>>>>>>>                                                     as
>>>>>>>>>>>>>>>>>>>>>>>                                                     an
>>>>>>>>>>>>>>>>>>>>>>>                                                     addition
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     our
>>>>>>>>>>>>>>>>>>>>>>>                                                     testing
>>>>>>>>>>>>>>>>>>>>>>>                                                     coverage
>>>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>>>                                                     DPDK
>>>>>>>>>>>>>>>>>>>>>>>                                                     at
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>>>>>>>                                                     lab.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>>                                                     have
>>>>>>>>>>>>>>>>>>>>>>>                                                     some
>>>>>>>>>>>>>>>>>>>>>>>                                                     questions
>>>>>>>>>>>>>>>>>>>>>>>                                                     about
>>>>>>>>>>>>>>>>>>>>>>>                                                     how
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     set
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     suite
>>>>>>>>>>>>>>>>>>>>>>>                                                     arguments.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>>                                                     have
>>>>>>>>>>>>>>>>>>>>>>>                                                     been
>>>>>>>>>>>>>>>>>>>>>>>                                                     able
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     configure
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     Test
>>>>>>>>>>>>>>>>>>>>>>>                                                     Engine
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     connect
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     hosts
>>>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     testbed.
>>>>>>>>>>>>>>>>>>>>>>>                                                     The
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     RCF,
>>>>>>>>>>>>>>>>>>>>>>>                                                     Configurator,
>>>>>>>>>>>>>>>>>>>>>>>                                                     and
>>>>>>>>>>>>>>>>>>>>>>>                                                     Tester
>>>>>>>>>>>>>>>>>>>>>>>                                                     all
>>>>>>>>>>>>>>>>>>>>>>>                                                     begin
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     run,
>>>>>>>>>>>>>>>>>>>>>>>                                                     however
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     prelude
>>>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     suite
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     fails
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     run.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>>>>>>>>>>                                                     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>>>                                                     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     The
>>>>>>>>>>>>>>>>>>>>>>>                                                     documentation
>>>>>>>>>>>>>>>>>>>>>>>                                                     mentions
>>>>>>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     several
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     parameters
>>>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     suite,
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     IUT
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     link
>>>>>>>>>>>>>>>>>>>>>>>                                                     MAC,
>>>>>>>>>>>>>>>>>>>>>>>                                                     etc.
>>>>>>>>>>>>>>>>>>>>>>>                                                     These
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     seem
>>>>>>>>>>>>>>>>>>>>>>>                                                     like
>>>>>>>>>>>>>>>>>>>>>>>                                                     they
>>>>>>>>>>>>>>>>>>>>>>>                                                     would
>>>>>>>>>>>>>>>>>>>>>>>                                                     need
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>>>                                                     set
>>>>>>>>>>>>>>>>>>>>>>>                                                     somewhere
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     run
>>>>>>>>>>>>>>>>>>>>>>>                                                     many
>>>>>>>>>>>>>>>>>>>>>>>                                                     of
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     tests.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>>                                                     see
>>>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     Test
>>>>>>>>>>>>>>>>>>>>>>>                                                     Engine
>>>>>>>>>>>>>>>>>>>>>>>                                                     documentation,
>>>>>>>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>>>>>>>                                                     instructions
>>>>>>>>>>>>>>>>>>>>>>>                                                     on
>>>>>>>>>>>>>>>>>>>>>>>                                                     how
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     create
>>>>>>>>>>>>>>>>>>>>>>>                                                     new
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     parameters
>>>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     suites
>>>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     Tester
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     configuration,
>>>>>>>>>>>>>>>>>>>>>>>                                                     but
>>>>>>>>>>>>>>>>>>>>>>>                                                     there
>>>>>>>>>>>>>>>>>>>>>>>                                                     is
>>>>>>>>>>>>>>>>>>>>>>>                                                     nothing
>>>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     user
>>>>>>>>>>>>>>>>>>>>>>>                                                     guide
>>>>>>>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     Tester
>>>>>>>>>>>>>>>>>>>>>>>                                                     guide
>>>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>>>                                                     how
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     set
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     arguments
>>>>>>>>>>>>>>>>>>>>>>>                                                     for
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     parameters
>>>>>>>>>>>>>>>>>>>>>>>                                                     when
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     running
>>>>>>>>>>>>>>>>>>>>>>>                                                     the
>>>>>>>>>>>>>>>>>>>>>>>                                                     test
>>>>>>>>>>>>>>>>>>>>>>>                                                     suite
>>>>>>>>>>>>>>>>>>>>>>>                                                     that
>>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>>                                                     can
>>>>>>>>>>>>>>>>>>>>>>>                                                     find.
>>>>>>>>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     not
>>>>>>>>>>>>>>>>>>>>>>>                                                     sure
>>>>>>>>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>>                                                     need
>>>>>>>>>>>>>>>>>>>>>>>                                                     to
>>>>>>>>>>>>>>>>>>>>>>>                                                     write
>>>>>>>>>>>>>>>>>>>>>>>                                                     my
>>>>>>>>>>>>>>>>>>>>>>>                                                     own
>>>>>>>>>>>>>>>>>>>>>>>                                                     Tester
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     config,
>>>>>>>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>>>>>>>                                                     if
>>>>>>>>>>>>>>>>>>>>>>>                                                     I
>>>>>>>>>>>>>>>>>>>>>>>                                                     should
>>>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>>>                                                     setting
>>>>>>>>>>>>>>>>>>>>>>>                                                     these
>>>>>>>>>>>>>>>>>>>>>>>                                                     in
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     some
>>>>>>>>>>>>>>>>>>>>>>>                                                     other
>>>>>>>>>>>>>>>>>>>>>>>                                                     way.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     How
>>>>>>>>>>>>>>>>>>>>>>>                                                     should
>>>>>>>>>>>>>>>>>>>>>>>                                                     these
>>>>>>>>>>>>>>>>>>>>>>>                                                     values
>>>>>>>>>>>>>>>>>>>>>>>                                                     be
>>>>>>>>>>>>>>>>>>>>>>>                                                     set?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     I'm
>>>>>>>>>>>>>>>>>>>>>>>                                                     also
>>>>>>>>>>>>>>>>>>>>>>>                                                     not
>>>>>>>>>>>>>>>>>>>>>>>                                                     sure
>>>>>>>>>>>>>>>>>>>>>>>                                                     what
>>>>>>>>>>>>>>>>>>>>>>>                                                     environment
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     variables/arguments
>>>>>>>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>>>>>>>                                                     strictly
>>>>>>>>>>>>>>>>>>>>>>>                                                     necessary
>>>>>>>>>>>>>>>>>>>>>>>                                                     or
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     which
>>>>>>>>>>>>>>>>>>>>>>>                                                     are
>>>>>>>>>>>>>>>>>>>>>>>                                                     optional.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     Regards,
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     Adam
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     *Adam
>>>>>>>>>>>>>>>>>>>>>>>                                                     Hassick*
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     Senior
>>>>>>>>>>>>>>>>>>>>>>>                                                     Developer
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>>>>>>>                                                     InterOperability
>>>>>>>>>>>>>>>>>>>>>>>                                                     Lab
>>>>>>>>>>>>>>>>>>>>>>>                                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>>                                                     iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>>>                                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>>>                                                                        
>>>>>>>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>>>>>>>                                                     (603)
>>>>>>>>>>>>>>>>>>>>>>>                                                     475-8248
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     *Adam
>>>>>>>>>>>>>>>>>>>>>                                                     Hassick*
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     Senior
>>>>>>>>>>>>>>>>>>>>>                                                     Developer
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>>>>>                                                     InterOperability
>>>>>>>>>>>>>>>>>>>>>                                                     Lab
>>>>>>>>>>>>>>>>>>>>>                                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                                     iol.unh.edu
>>>>>>>>>>>>>>>>>>>>>                                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>>>                                                                    
>>>>>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>>>>>                                                     (603)
>>>>>>>>>>>>>>>>>>>>>                                                     475-8248
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>>>                                                     *Adam
>>>>>>>>>>>>>>>>>>>                                                     Hassick*
>>>>>>>>>>>>>>>>>>>                                                     Senior
>>>>>>>>>>>>>>>>>>>                                                     Developer
>>>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>>>                                                     InterOperability
>>>>>>>>>>>>>>>>>>>                                                     Lab
>>>>>>>>>>>>>>>>>>>                                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                                     iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                                                
>>>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>>>                                                     (603)
>>>>>>>>>>>>>>>>>>>                                                     475-8248
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     *Adam
>>>>>>>>>>>>>>>>>>>                                                     Hassick*
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     Senior
>>>>>>>>>>>>>>>>>>>                                                     Developer
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>>>                                                     InterOperability
>>>>>>>>>>>>>>>>>>>                                                     Lab
>>>>>>>>>>>>>>>>>>>                                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                                     iol.unh.edu
>>>>>>>>>>>>>>>>>>>                                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>>                                                            
>>>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>>>                                                     (603)
>>>>>>>>>>>>>>>>>>>                                                     475-8248
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     --
>>>>>>>>>>>>>>>>>>                                                     *Adam
>>>>>>>>>>>>>>>>>>                                                     Hassick*
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     Senior
>>>>>>>>>>>>>>>>>>                                                     Developer
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>>                                                     InterOperability
>>>>>>>>>>>>>>>>>>                                                     Lab
>>>>>>>>>>>>>>>>>>                                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                     iol.unh.edu
>>>>>>>>>>>>>>>>>>                                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>>                                                        
>>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>>                                                     (603)
>>>>>>>>>>>>>>>>>>                                                     475-8248
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                                     -- 
>>>>>>>>>>>>>>>>>                                                     *Adam
>>>>>>>>>>>>>>>>>                                                     Hassick*
>>>>>>>>>>>>>>>>>                                                     Senior
>>>>>>>>>>>>>>>>>                                                     Developer
>>>>>>>>>>>>>>>>>                                                     UNH
>>>>>>>>>>>>>>>>>                                                     InterOperability
>>>>>>>>>>>>>>>>>                                                     Lab
>>>>>>>>>>>>>>>>>                                                     ahassick@iol.unh.edu
>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                     iol.unh.edu
>>>>>>>>>>>>>>>>>                                                     <http://iol.unh.edu>
>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                     <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>>>                                                     +1
>>>>>>>>>>>>>>>>>                                                     (603)
>>>>>>>>>>>>>>>>>                                                     475-8248
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                                                 --
>>>>>>>>>>>>>>>                                                 *Adam
>>>>>>>>>>>>>>>                                                 Hassick*
>>>>>>>>>>>>>>>                                                 Senior
>>>>>>>>>>>>>>>                                                 Developer
>>>>>>>>>>>>>>>                                                 UNH
>>>>>>>>>>>>>>>                                                 InterOperability
>>>>>>>>>>>>>>>                                                 Lab
>>>>>>>>>>>>>>>                                                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>>                                                 iol.unh.edu
>>>>>>>>>>>>>>>                                                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>>                                                 +1 (603)
>>>>>>>>>>>>>>>                                                 475-8248
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                             --
>>>>>>>>>>>>>                                             *Adam Hassick*
>>>>>>>>>>>>>                                             Senior Developer
>>>>>>>>>>>>>                                             UNH
>>>>>>>>>>>>>                                             InterOperability
>>>>>>>>>>>>>                                             Lab
>>>>>>>>>>>>>                                             ahassick@iol.unh.edu
>>>>>>>>>>>>>                                             iol.unh.edu
>>>>>>>>>>>>>                                             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                             +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                         --
>>>>>>>>>>>                                         *Adam Hassick*
>>>>>>>>>>>                                         Senior Developer
>>>>>>>>>>>                                         UNH InterOperability Lab
>>>>>>>>>>>                                         ahassick@iol.unh.edu
>>>>>>>>>>>                                         iol.unh.edu
>>>>>>>>>>>                                         <https://www.iol.unh.edu/>
>>>>>>>>>>>                                         +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                                 --
>>>>>>>>>                                 *Adam Hassick*
>>>>>>>>>                                 Senior Developer
>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>                                 ahassick@iol.unh.edu
>>>>>>>>>                                 iol.unh.edu
>>>>>>>>>                                 <https://www.iol.unh.edu/>
>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                             --
>>>>>>>>                             *Adam Hassick*
>>>>>>>>                             Senior Developer
>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>                             ahassick@iol.unh.edu
>>>>>>>>                             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                         --
>>>>>>>>                         *Adam Hassick*
>>>>>>>>                         Senior Developer
>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>                         ahassick@iol.unh.edu
>>>>>>>>                         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                         +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 -- 
>>>>>                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>

[-- Attachment #2: Type: text/html, Size: 256336 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

Hi Adam,

fresh ts-rigs-sample has all required bindings to publish logs to 
ts-factory.io. See the top-most commit.
So, you just need tune SSH to use correct user, port and key for 
ts-factory.io. You  can add below settings
to corresponding user .ssh/config or add corresponding options to sftp 
command in scripts/publish_logs/prj/ts-factory/publish:

Host ts-factory.io
         User unh-iol
         Port 56777
         IdentitiesOnly yes
         IdentityFile <path-to-private-key>

The publishing is asynchronous in ts-factory case. i.e. user puts logs 
and cron job checks incoming directory every 10 minutes and publish 
found logs.

You can request publishing when you start testing using --publish option 
or publish previous run testing results using ./scripts/publish_logs 
from dpdk-ethdev-ts sources. PWD should be the directory you run tests from.

Please, let me know how it goes.

Regards,
Andrew.

(dropped history, since it became too big and require moderation)

[-- Attachment #2: Type: text/html, Size: 1454 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 1358 bytes --]

Hi Andrew,

I've started a test run, it should finish at some point overnight. I tested
the authentication and it seems to work, fingers crossed that we see some
results appear tomorrow.

Thanks,
Adam

On Fri, Oct 27, 2023 at 3:13 PM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> fresh ts-rigs-sample has all required bindings to publish logs to
> ts-factory.io. See the top-most commit.
> So, you just need tune SSH to use correct user, port and key for
> ts-factory.io. You  can add below settings
> to corresponding user .ssh/config or add corresponding options to sftp
> command in scripts/publish_logs/prj/ts-factory/publish:
>
> Host ts-factory.io
>         User unh-iol
>         Port 56777
>         IdentitiesOnly yes
>         IdentityFile <path-to-private-key>
>
> The publishing is asynchronous in ts-factory case. i.e. user puts logs and
> cron job checks incoming directory every 10 minutes and publish found logs.
>
> You can request publishing when you start testing using --publish option
> or publish previous run testing results using ./scripts/publish_logs from
> dpdk-ethdev-ts sources. PWD should be the directory you run tests from.
>
> Please, let me know how it goes.
>
> Regards,
> Andrew.
>
> (dropped history, since it became too big and require moderation)
>

[-- Attachment #2: Type: text/html, Size: 2157 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 1813 bytes --]

Hi Adam,

I see no attempt to publish logs, so I guess something went wrong. 
Please, let me know if you need any help from me. Hopefully there is no 
need to rerun tests and publish could be debugged using 
./scripts/publish_logs.

Andrew.

On 11/7/23 02:16, Adam Hassick wrote:
> Hi Andrew,
>
> I've started a test run, it should finish at some point overnight. I 
> tested the authentication and it seems to work, fingers crossed that 
> we see some results appear tomorrow.
>
> Thanks,
> Adam
>
> On Fri, Oct 27, 2023 at 3:13 PM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     fresh ts-rigs-sample has all required bindings to publish logs to
>     ts-factory.io <http://ts-factory.io>. See the top-most commit.
>     So, you just need tune SSH to use correct user, port and key for
>     ts-factory.io <http://ts-factory.io>. You  can add below settings
>     to corresponding user .ssh/config or add corresponding options to
>     sftp command in scripts/publish_logs/prj/ts-factory/publish:
>
>     Host ts-factory.io <http://ts-factory.io>
>             User unh-iol
>             Port 56777
>             IdentitiesOnly yes
>             IdentityFile <path-to-private-key>
>
>     The publishing is asynchronous in ts-factory case. i.e. user puts
>     logs and cron job checks incoming directory every 10 minutes and
>     publish found logs.
>
>     You can request publishing when you start testing using --publish
>     option or publish previous run testing results using
>     ./scripts/publish_logs from dpdk-ethdev-ts sources. PWD should be
>     the directory you run tests from.
>
>     Please, let me know how it goes.
>
>     Regards,
>     Andrew.
>
>     (dropped history, since it became too big and require moderation)
>

[-- Attachment #2: Type: text/html, Size: 3599 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 2649 bytes --]

Hi Andrew,

The runner machine was missing a dependency for one of the scripts, "pixz".
After installing that, it appears to have worked. I can see the results
listed on the ts-factory Bublik instance.
In the latest revision of ts-rigs, there appears to be a syntax error at
line 42 within the script located at
"ts-rigs/scripts/publish_logs/prj/ts-factory/publish", within the if
condition. I fixed it locally to get it to run.

Taking a quick look at a comparison against your most recent X710 run, it
looks like we're NOK on around ~400 more test cases. By percentage of
tests, we're 1% off, however, it looks like whole subsets of the test suite
that contain low numbers of tests are failing. I wonder if this is due to
differences between the Intel X710 and XL710 or issues in our dev testbed.

Thanks,
Adam

On Tue, Nov 7, 2023 at 11:58 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> I see no attempt to publish logs, so I guess something went wrong. Please,
> let me know if you need any help from me. Hopefully there is no need to
> rerun tests and publish could be debugged using ./scripts/publish_logs.
>
> Andrew.
>
> On 11/7/23 02:16, Adam Hassick wrote:
>
> Hi Andrew,
>
> I've started a test run, it should finish at some point overnight. I
> tested the authentication and it seems to work, fingers crossed that we see
> some results appear tomorrow.
>
> Thanks,
> Adam
>
> On Fri, Oct 27, 2023 at 3:13 PM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> fresh ts-rigs-sample has all required bindings to publish logs to
>> ts-factory.io. See the top-most commit.
>> So, you just need tune SSH to use correct user, port and key for
>> ts-factory.io. You  can add below settings
>> to corresponding user .ssh/config or add corresponding options to sftp
>> command in scripts/publish_logs/prj/ts-factory/publish:
>>
>> Host ts-factory.io
>>         User unh-iol
>>         Port 56777
>>         IdentitiesOnly yes
>>         IdentityFile <path-to-private-key>
>>
>> The publishing is asynchronous in ts-factory case. i.e. user puts logs
>> and cron job checks incoming directory every 10 minutes and publish found
>> logs.
>>
>> You can request publishing when you start testing using --publish option
>> or publish previous run testing results using ./scripts/publish_logs from
>> dpdk-ethdev-ts sources. PWD should be the directory you run tests from.
>>
>> Please, let me know how it goes.
>>
>> Regards,
>> Andrew.
>>
>> (dropped history, since it became too big and require moderation)
>>
>
>

[-- Attachment #2: Type: text/html, Size: 4553 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 3090 bytes --]

Hi Adam,

On 11/7/23 23:30, Adam Hassick wrote:
> Hi Andrew,
>
> The runner machine was missing a dependency for one of the scripts, 
> "pixz". After installing that, it appears to have worked. I can see 
> the results listed on the ts-factory Bublik instance.

If you use copy of dpdk-ethdev-ts has 
398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass 
--trc-tag=pci-8086-1572 any more since corresponding changeset updates 
expectations to have the same for pci-8086-1583.

> In the latest revision of ts-rigs, there appears to be a syntax error 
> at line 42 within the script located at 
> "ts-rigs/scripts/publish_logs/prj/ts-factory/publish", within the if 
> condition. I fixed it locally to get it to run.

Sorry, but I've failed to find what's wrong there.

> Taking a quick look at a comparison against your most recent X710 run, 
> it looks like we're NOK on around ~400 more test cases. By percentage 
> of tests, we're 1% off, however, it looks like whole subsets of the 
> test suite that contain low numbers of tests are failing. I wonder if 
> this is due to differences between the Intel X710 and XL710 or issues 
> in our dev testbed.

As far as I can see LLDP packets spoil testing results:
https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63

As far as I can see main prologue disables FW LLDP on Tester
https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
but I guess it could be still enabled on DUT side and DPDK do not 
provide means to disable it as far as I know. I vaguely remember that 
Intel provides FW configuration tools which can do it.
It is interesting since DPDK gets unexpected LLDP packets but may be 
packets sent by FW go via loopback and visible to PF as well.
Other possible source of LLDP packet is a switch if NICs are connected 
via switch. If so, LLDP should be disabled on corresponding switch ports.

As far as I can see fixing the problem should make results much closer. 
However, I already see some differences in behaviour which should be 
simply fixed in TRC. For example, X710 gets 9 packets less than 
configuration number of Rx descriptors, but XL710 gets 10 packets less.

Also I see that performance tests are not run because of failed prologue:
https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
I'll investigate it, but I guess the source of difference is that we 
always run tests on single interface. Just add -p0 
(--cfg=iol-dts-xl710-p0) to your configuration name. You don't need to 
change ts-rigs for it since the suffix is handled by generic code. It 
simply comments the second instance and forces take the first interface 
only into account. Initially it was introduced to run independent tests 
on different ports to be able to share configuration, but I guess right 
now it has limitations for some packages like representors which require 
entire NIC.

Regards,
Andrew.

> Thanks,
> Adam

(dropped history, to keep mail size small)

[-- Attachment #2: Type: text/html, Size: 5616 bytes --]

Re: Setting up DPDK PMD Test Suite

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 6247 bytes --]

Hi Andrew,

If you use copy of dpdk-ethdev-ts has
> 398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass --trc-tag=pci-8086-1572
> any more since corresponding changeset updates expectations to have the
> same for pci-8086-1583.


I'll try this for the next run.

Sorry, but I've failed to find what's wrong there.


That if statement works if using the traditional single-bracket
conditional, or it needs to be rewritten as "[[ -z "${test_log}" ]] || [[ !
-r "${test_log}" ]]". The latter is the change I made, but both work.

As far as I can see LLDP packets spoil testing results:
>
> https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63
>
> As far as I can see main prologue disables FW LLDP on Tester
>
> https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
> but I guess it could be still enabled on DUT side and DPDK do not provide
> means to disable it as far as I know. I vaguely remember that Intel
> provides FW configuration tools which can do it.
> It is interesting since DPDK gets unexpected LLDP packets but may be
> packets sent by FW go via loopback and visible to PF as well.
> Other possible source of LLDP packet is a switch if NICs are connected via
> switch. If so, LLDP should be disabled on corresponding switch ports.
>
> As far as I can see fixing the problem should make results much closer.
> However, I already see some differences in behaviour which should be simply
> fixed in TRC. For example, X710 gets 9 packets less than configuration
> number of Rx descriptors, but XL710 gets 10 packets less.


I have the "disable-fw-lldp" private flag set on both of the XL710 ports on
the DUT machine. Very strange how there are still LLDP packets appearing in
there.
These systems are not connected to any switch, so maybe a service on the
DUT itself is sending them. I'm not sure how that could be happening
though, because I don't have the LLDP daemon installed on either system.

Also I see that performance tests are not run because of failed prologue:
>
> https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
> I'll investigate it, but I guess the source of difference is that we
> always run tests on single interface. Just add -p0 (--cfg=iol-dts-xl710-p0)
> to your configuration name. You don't need to change ts-rigs for it since
> the suffix is handled by generic code. It simply comments the second
> instance and forces take the first interface only into account. Initially
> it was introduced to run independent tests on different ports to be able to
> share configuration, but I guess right now it has limitations for some
> packages like representors which require entire NIC.


I can try that and will see if it works.

Thanks,
Adam

On Wed, Nov 8, 2023 at 2:20 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> On 11/7/23 23:30, Adam Hassick wrote:
>
> Hi Andrew,
>
> The runner machine was missing a dependency for one of the scripts,
> "pixz". After installing that, it appears to have worked. I can see the
> results listed on the ts-factory Bublik instance.
>
>
> If you use copy of dpdk-ethdev-ts has
> 398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass --trc-tag=pci-8086-1572
> any more since corresponding changeset updates expectations to have the
> same for pci-8086-1583.
>
> In the latest revision of ts-rigs, there appears to be a syntax error at
> line 42 within the script located at
> "ts-rigs/scripts/publish_logs/prj/ts-factory/publish", within the if
> condition. I fixed it locally to get it to run.
>
>
> Sorry, but I've failed to find what's wrong there.
>
> Taking a quick look at a comparison against your most recent X710 run, it
> looks like we're NOK on around ~400 more test cases. By percentage of
> tests, we're 1% off, however, it looks like whole subsets of the test suite
> that contain low numbers of tests are failing. I wonder if this is due to
> differences between the Intel X710 and XL710 or issues in our dev testbed.
>
>
> As far as I can see LLDP packets spoil testing results:
>
> https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63
>
> As far as I can see main prologue disables FW LLDP on Tester
>
> https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
> but I guess it could be still enabled on DUT side and DPDK do not provide
> means to disable it as far as I know. I vaguely remember that Intel
> provides FW configuration tools which can do it.
> It is interesting since DPDK gets unexpected LLDP packets but may be
> packets sent by FW go via loopback and visible to PF as well.
> Other possible source of LLDP packet is a switch if NICs are connected via
> switch. If so, LLDP should be disabled on corresponding switch ports.
>
> As far as I can see fixing the problem should make results much closer.
> However, I already see some differences in behaviour which should be simply
> fixed in TRC. For example, X710 gets 9 packets less than configuration
> number of Rx descriptors, but XL710 gets 10 packets less.
>
> Also I see that performance tests are not run because of failed prologue:
>
> https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
> I'll investigate it, but I guess the source of difference is that we
> always run tests on single interface. Just add -p0 (--cfg=iol-dts-xl710-p0)
> to your configuration name. You don't need to change ts-rigs for it since
> the suffix is handled by generic code. It simply comments the second
> instance and forces take the first interface only into account. Initially
> it was introduced to run independent tests on different ports to be able to
> share configuration, but I guess right now it has limitations for some
> packages like representors which require entire NIC.
>
> Regards,
> Andrew.
>
> Thanks,
> Adam
>
>
> (dropped history, to keep mail size small)
>

[-- Attachment #2: Type: text/html, Size: 9750 bytes --]

DPDK Coverity test run

[Mcnamara, John]

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

Hi Folks,

AFAIK the DPDK Coverity tests run on Monday, Wednesday and Friday. I didn’t see a run on Wednesday. Is that because Monday’s run had already captured RC3 and there weren’t any additional changes to run on Wednesday? Or did it fail to run on Wednesday for some reason?

John




[-- Attachment #2: Type: text/html, Size: 2740 bytes --]

Re: DPDK Coverity test run

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

Hi John,

I am seeing a Nov 15 run here:
https://scan.coverity.com/projects/dpdk-data-plane-development-kit

Let me know if that isn't right. Thanks.

Patrick

On Thu, Nov 16, 2023 at 3:38 PM Mcnamara, John <john.mcnamara@intel.com>
wrote:

> Hi Folks,
>
>
>
> AFAIK the DPDK Coverity tests run on Monday, Wednesday and Friday. I
> didn’t see a run on Wednesday. Is that because Monday’s run had already
> captured RC3 and there weren’t any additional changes to run on Wednesday?
> Or did it fail to run on Wednesday for some reason?
>
>
>
> John
>
>
>
>
>
>
>


-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 3878 bytes --]

RE: DPDK Coverity test run

[Mcnamara, John]

[-- Attachment #1.1: Type: text/plain, Size: 1061 bytes --]

My bad Patrick. You are right. Thanks.

From: Patrick Robb <probb@iol.unh.edu>
Sent: Thursday, November 16, 2023 8:44 PM
To: Mcnamara, John <john.mcnamara@intel.com>
Cc: ci@dpdk.org; Adam Hassick <ahassick@iol.unh.edu>
Subject: Re: DPDK Coverity test run

Hi John,

I am seeing a Nov 15 run here: https://scan.coverity.com/projects/dpdk-data-plane-development-kit

Let me know if that isn't right. Thanks.

Patrick

On Thu, Nov 16, 2023 at 3:38 PM Mcnamara, John <john.mcnamara@intel.com<mailto:john.mcnamara@intel.com>> wrote:
Hi Folks,

AFAIK the DPDK Coverity tests run on Monday, Wednesday and Friday. I didn’t see a run on Wednesday. Is that because Monday’s run had already captured RC3 and there weren’t any additional changes to run on Wednesday? Or did it fail to run on Wednesday for some reason?

John





--

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu<http://www.iol.unh.edu/>



[Image removed by sender.]

[-- Attachment #1.2: Type: text/html, Size: 6962 bytes --]

[-- Attachment #2: ~WRD0000.jpg --]
[-- Type: image/jpeg, Size: 823 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 7644 bytes --]

Hi Adam,

On 11/16/23 23:03, Adam Hassick wrote:
> Hi Andrew,
>
>     If you use copy of dpdk-ethdev-ts has
>     398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass
>     --trc-tag=pci-8086-1572 any more since corresponding changeset
>     updates expectations to have the same for pci-8086-1583.
>
>
> I'll try this for the next run.
>
>     Sorry, but I've failed to find what's wrong there.
>
>
> That if statement works if using the traditional single-bracket 
> conditional, or it needs to be rewritten as "[[ -z "${test_log}" ]] || 
> [[ ! -r "${test_log}" ]]". The latter is the change I made, but both work.

Thanks a lot. Hopefully fixed.

>
>     As far as I can see LLDP packets spoil testing results:
>     https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63
>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63>
>
>     As far as I can see main prologue disables FW LLDP on Tester
>     https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80>
>     but I guess it could be still enabled on DUT side and DPDK do not
>     provide means to disable it as far as I know. I vaguely remember
>     that Intel provides FW configuration tools which can do it.
>     It is interesting since DPDK gets unexpected LLDP packets but may
>     be packets sent by FW go via loopback and visible to PF as well.
>     Other possible source of LLDP packet is a switch if NICs are
>     connected via switch. If so, LLDP should be disabled on
>     corresponding switch ports.
>
>     As far as I can see fixing the problem should make results much
>     closer. However, I already see some differences in behaviour which
>     should be simply fixed in TRC. For example, X710 gets 9 packets
>     less than configuration number of Rx descriptors, but XL710 gets
>     10 packets less.
>
>
> I have the "disable-fw-lldp" private flag set on both of the XL710 
> ports on the DUT machine. Very strange how there are still LLDP 
> packets appearing in there.

Me too. Corresponding packet has source MAC from Peer/Tester machine NIC.
It is really strange since prologue disabled LLDP there as well. I'll 
try to play with it locally more, but have no good ideas in fact.

> These systems are not connected to any switch, so maybe a service on 
> the DUT itself is sending them. I'm not sure how that could be 
> happening though, because I don't have the LLDP daemon installed on 
> either system.
>
>     Also I see that performance tests are not run because of failed
>     prologue:
>     https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
>     <https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true>
>     I'll investigate it, but I guess the source of difference is that
>     we always run tests on single interface. Just add -p0
>     (--cfg=iol-dts-xl710-p0) to your configuration name. You don't
>     need to change ts-rigs for it since the suffix is handled by
>     generic code. It simply comments the second instance and forces
>     take the first interface only into account. Initially it was
>     introduced to run independent tests on different ports to be able
>     to share configuration, but I guess right now it has limitations
>     for some packages like representors which require entire NIC.
>
>
> I can try that and will see if it works.

This problem is fixed in fresh TE and dpdk-ethdev-ts published on GitHub.

Regards,
Andrew.

>
> Thanks,
> Adam
>
> On Wed, Nov 8, 2023 at 2:20 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     On 11/7/23 23:30, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     The runner machine was missing a dependency for one of the
>>     scripts, "pixz". After installing that, it appears to have
>>     worked. I can see the results listed on the ts-factory Bublik
>>     instance.
>
>     If you use copy of dpdk-ethdev-ts has
>     398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass
>     --trc-tag=pci-8086-1572 any more since corresponding changeset
>     updates expectations to have the same for pci-8086-1583.
>
>>     In the latest revision of ts-rigs, there appears to be a syntax
>>     error at line 42 within the script located at
>>     "ts-rigs/scripts/publish_logs/prj/ts-factory/publish", within the
>>     if condition. I fixed it locally to get it to run.
>
>     Sorry, but I've failed to find what's wrong there.
>
>>     Taking a quick look at a comparison against your most recent X710
>>     run, it looks like we're NOK on around ~400 more test cases. By
>>     percentage of tests, we're 1% off, however, it looks like whole
>>     subsets of the test suite that contain low numbers of tests are
>>     failing. I wonder if this is due to differences between the Intel
>>     X710 and XL710 or issues in our dev testbed.
>
>     As far as I can see LLDP packets spoil testing results:
>     https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63
>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63>
>
>     As far as I can see main prologue disables FW LLDP on Tester
>     https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80>
>     but I guess it could be still enabled on DUT side and DPDK do not
>     provide means to disable it as far as I know. I vaguely remember
>     that Intel provides FW configuration tools which can do it.
>     It is interesting since DPDK gets unexpected LLDP packets but may
>     be packets sent by FW go via loopback and visible to PF as well.
>     Other possible source of LLDP packet is a switch if NICs are
>     connected via switch. If so, LLDP should be disabled on
>     corresponding switch ports.
>
>     As far as I can see fixing the problem should make results much
>     closer. However, I already see some differences in behaviour which
>     should be simply fixed in TRC. For example, X710 gets 9 packets
>     less than configuration number of Rx descriptors, but XL710 gets
>     10 packets less.
>
>     Also I see that performance tests are not run because of failed
>     prologue:
>     https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
>     <https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true>
>     I'll investigate it, but I guess the source of difference is that
>     we always run tests on single interface. Just add -p0
>     (--cfg=iol-dts-xl710-p0) to your configuration name. You don't
>     need to change ts-rigs for it since the suffix is handled by
>     generic code. It simply comments the second instance and forces
>     take the first interface only into account. Initially it was
>     introduced to run independent tests on different ports to be able
>     to share configuration, but I guess right now it has limitations
>     for some packages like representors which require entire NIC.
>
>     Regards,
>     Andrew.
>
>>     Thanks,
>>     Adam
>
>     (dropped history, to keep mail size small)
>

[-- Attachment #2: Type: text/html, Size: 13454 bytes --]

Re: Setting up DPDK PMD Test Suite

[Andrew Rybchenko]

[-- Attachment #1: Type: text/plain, Size: 8617 bytes --]

Hi Adam,

I have no good ideas on the problem with LLDP packets. I've tried 
various things in attempt to repeat the problem without any luck. Since 
these packet come from Peer/Tester (based on source MAC information), I 
think it would make sense to check ethtool priv flags while tests run 
running to see FW LLDP status. May be it is enabled and testing do not 
notice it (theoretically configuration is synced and checked to match 
after each test, so it should not be a problem). I think it would be 
useful to double-check on all interfaces of the NIC.

Do you have any progress with run on ARM DUTs? Does it work?

Please, let me know if you need any help and if something is blocking it 
to be moved forward and used on regular basis.

Andrew.

On 11/20/23 20:18, Andrew Rybchenko wrote:
> Hi Adam,
>
> On 11/16/23 23:03, Adam Hassick wrote:
>> Hi Andrew,
>>
>>     If you use copy of dpdk-ethdev-ts has
>>     398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass
>>     --trc-tag=pci-8086-1572 any more since corresponding changeset
>>     updates expectations to have the same for pci-8086-1583.
>>
>>
>> I'll try this for the next run.
>>
>>     Sorry, but I've failed to find what's wrong there.
>>
>>
>> That if statement works if using the traditional single-bracket 
>> conditional, or it needs to be rewritten as "[[ -z "${test_log}" ]] 
>> || [[ ! -r "${test_log}" ]]". The latter is the change I made, but 
>> both work.
>
> Thanks a lot. Hopefully fixed.
>
>>
>>     As far as I can see LLDP packets spoil testing results:
>>     https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63
>>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63>
>>
>>     As far as I can see main prologue disables FW LLDP on Tester
>>     https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
>>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80>
>>     but I guess it could be still enabled on DUT side and DPDK do not
>>     provide means to disable it as far as I know. I vaguely remember
>>     that Intel provides FW configuration tools which can do it.
>>     It is interesting since DPDK gets unexpected LLDP packets but may
>>     be packets sent by FW go via loopback and visible to PF as well.
>>     Other possible source of LLDP packet is a switch if NICs are
>>     connected via switch. If so, LLDP should be disabled on
>>     corresponding switch ports.
>>
>>     As far as I can see fixing the problem should make results much
>>     closer. However, I already see some differences in behaviour
>>     which should be simply fixed in TRC. For example, X710 gets 9
>>     packets less than configuration number of Rx descriptors, but
>>     XL710 gets 10 packets less.
>>
>>
>> I have the "disable-fw-lldp" private flag set on both of the XL710 
>> ports on the DUT machine. Very strange how there are still LLDP 
>> packets appearing in there.
>
> Me too. Corresponding packet has source MAC from Peer/Tester machine NIC.
> It is really strange since prologue disabled LLDP there as well. I'll 
> try to play with it locally more, but have no good ideas in fact.
>
>> These systems are not connected to any switch, so maybe a service on 
>> the DUT itself is sending them. I'm not sure how that could be 
>> happening though, because I don't have the LLDP daemon installed on 
>> either system.
>>
>>     Also I see that performance tests are not run because of failed
>>     prologue:
>>     https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
>>     <https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true>
>>     I'll investigate it, but I guess the source of difference is that
>>     we always run tests on single interface. Just add -p0
>>     (--cfg=iol-dts-xl710-p0) to your configuration name. You don't
>>     need to change ts-rigs for it since the suffix is handled by
>>     generic code. It simply comments the second instance and forces
>>     take the first interface only into account. Initially it was
>>     introduced to run independent tests on different ports to be able
>>     to share configuration, but I guess right now it has limitations
>>     for some packages like representors which require entire NIC.
>>
>>
>> I can try that and will see if it works.
>
> This problem is fixed in fresh TE and dpdk-ethdev-ts published on GitHub.
>
> Regards,
> Andrew.
>
>>
>> Thanks,
>> Adam
>>
>> On Wed, Nov 8, 2023 at 2:20 AM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>     Hi Adam,
>>
>>     On 11/7/23 23:30, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     The runner machine was missing a dependency for one of the
>>>     scripts, "pixz". After installing that, it appears to have
>>>     worked. I can see the results listed on the ts-factory Bublik
>>>     instance.
>>
>>     If you use copy of dpdk-ethdev-ts has
>>     398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass
>>     --trc-tag=pci-8086-1572 any more since corresponding changeset
>>     updates expectations to have the same for pci-8086-1583.
>>
>>>     In the latest revision of ts-rigs, there appears to be a syntax
>>>     error at line 42 within the script located at
>>>     "ts-rigs/scripts/publish_logs/prj/ts-factory/publish", within
>>>     the if condition. I fixed it locally to get it to run.
>>
>>     Sorry, but I've failed to find what's wrong there.
>>
>>>     Taking a quick look at a comparison against your most recent
>>>     X710 run, it looks like we're NOK on around ~400 more test
>>>     cases. By percentage of tests, we're 1% off, however, it looks
>>>     like whole subsets of the test suite that contain low numbers of
>>>     tests are failing. I wonder if this is due to differences
>>>     between the Intel X710 and XL710 or issues in our dev testbed.
>>
>>     As far as I can see LLDP packets spoil testing results:
>>     https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63
>>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63>
>>
>>     As far as I can see main prologue disables FW LLDP on Tester
>>     https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
>>     <https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80>
>>     but I guess it could be still enabled on DUT side and DPDK do not
>>     provide means to disable it as far as I know. I vaguely remember
>>     that Intel provides FW configuration tools which can do it.
>>     It is interesting since DPDK gets unexpected LLDP packets but may
>>     be packets sent by FW go via loopback and visible to PF as well.
>>     Other possible source of LLDP packet is a switch if NICs are
>>     connected via switch. If so, LLDP should be disabled on
>>     corresponding switch ports.
>>
>>     As far as I can see fixing the problem should make results much
>>     closer. However, I already see some differences in behaviour
>>     which should be simply fixed in TRC. For example, X710 gets 9
>>     packets less than configuration number of Rx descriptors, but
>>     XL710 gets 10 packets less.
>>
>>     Also I see that performance tests are not run because of failed
>>     prologue:
>>     https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
>>     <https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true>
>>     I'll investigate it, but I guess the source of difference is that
>>     we always run tests on single interface. Just add -p0
>>     (--cfg=iol-dts-xl710-p0) to your configuration name. You don't
>>     need to change ts-rigs for it since the suffix is handled by
>>     generic code. It simply comments the second instance and forces
>>     take the first interface only into account. Initially it was
>>     introduced to run independent tests on different ports to be able
>>     to share configuration, but I guess right now it has limitations
>>     for some packages like representors which require entire NIC.
>>
>>     Regards,
>>     Andrew.
>>
>>>     Thanks,
>>>     Adam
>>
>>     (dropped history, to keep mail size small)
>>
>

[-- Attachment #2: Type: text/html, Size: 15245 bytes --]