DPDK CI discussions
 help / color / mirror / Atom feed
* Re: Setting up DPDK PMD Test Suite
       [not found]   ` <35340484-1d7e-7e5f-cad4-c965ba541397@oktetlabs.ru>
@ 2023-08-17 17:03     ` Adam Hassick
  2023-08-18 18:40       ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-17 17:03 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 4789 bytes --]

Hi Andrew,

I'm adding the CI mailing list to this conversation. Others in the
community might find this conversation valuable.

We do want to run testing on a regular basis. The Jenkins integration will
be very useful for us, as most of our CI is orchestrated by Jenkins.
I've conferred with the rest of the team, and we think it would be best to
move forward with mainly option B.
If you would like to know anything about our testbeds that would help you
with creating an example ts-rigs repo, I'd be happy to answer any questions
you have.

We have multiple test rigs (we call these "DUT-tester pairs") that we run
our existing hardware testing on, with differing network hardware and CPU
architecture. I figured this might be an important detail.

Thanks,
Adam

On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Greatings Adam,
>
> I'm happy to hear that you're trying to bring it up.
>
> As I understand the final goal is to run it on regular basis. So, we need
> to make it properly from the very beginning.
> Bring up of all features consists of 4 steps:
>
> 1. Create site-specific repository (we call it ts-rigs) which contains
> information about test rigs and other site-specific information like where
> to send mails, where to store logs etc. It is required for manual execution
> as well, since test rigs description is essential. I'll return to the topic
> below.
>
> 2. Setup logs storage for automated runs. Basically it is a disk space
> plus apache2 web server with few CGI scripts which help a lot to save disk
> space.
>
> 3. Setup Bublik web application which provides web interface to view
> testing results. Same as https://ts-factory.io/bublik
>
> 4. Setup Jenkins to run tests on regularly, save logs in log storage (2)
> and import it to bublik (3).
>
> Last few month we spent on our homework to make it simpler to bring up
> automated execution using Jenkins -
> https://github.com/ts-factory/te-jenkins
> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>
> Let's return to the step (1).
>
> Unfortunately there is no publicly available example of the ts-rigs
> repository since sensitive site-specific information is located there. But
> I'm ready to help you to create it for UNH. I see two options here:
>
> (A) I'll ask questions and based on your answers will create the first
> draft with my comments.
>
> (B) I'll make a template/example ts-rigs repo, publish it and you'll
> create UNH ts-rigs based on it.
>
> Of course, I'll help to debug and finally bring it up in any case.
>
> (A) is a bit simpler for me and you, but (B) is a bit more generic and
> will help other potential users to bring it up.
> We can combine (A)+(B). I.e. start from (A). What do you think?
>
> Thanks,
> Andrew.
>
> On 8/17/23 15:18, Konstantin Ushakov wrote:
>
> Greetings Adam,
>
>
> Thanks for contacting us. I copy Andrew who would be happy to help
>
> Thanks,
> Konstantin
>
> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu> wrote:
>
> 
> Greetings Konstantin,
>
> I am in the process of setting up the DPDK Poll Mode Driver test suite as
> an addition to our testing coverage for DPDK at the UNH lab.
>
> I have some questions about how to set the test suite arguments.
>
> I have been able to configure the Test Engine to connect to the hosts in
> the testbed. The RCF, Configurator, and Tester all begin to run, however
> the prelude of the test suite fails to run.
>
> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>
> The documentation mentions that there are several test parameters for the
> test suite, like for the IUT test link MAC, etc. These seem like they would
> need to be set somewhere to run many of the tests.
>
> I see in the Test Engine documentation, there are instructions on how to
> create new parameters for test suites in the Tester configuration, but
> there is nothing in the user guide or in the Tester guide for how to set
> the arguments for the parameters when running the test suite that I can
> find. I'm not sure if I need to write my own Tester config, or if I should
> be setting these in some other way.
>
> How should these values be set?
>
> I'm also not sure what environment variables/arguments are strictly
> necessary or which are optional.
>
> Regards,
> Adam
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 9885 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-17 17:03     ` Setting up DPDK PMD Test Suite Adam Hassick
@ 2023-08-18 18:40       ` Andrew Rybchenko
  2023-08-20  8:40         ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-18 18:40 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 5439 bytes --]

Hi Adam,

 > I've conferred with the rest of the team, and we think it would be 
best to move forward with mainly option B.

OK, I'll provide the sample on Monday for you. It is almost ready right 
now, but I need to double-check it before publishing.

Regards,
Andrew.

On 8/17/23 20:03, Adam Hassick wrote:
> Hi Andrew,
>
> I'm adding the CI mailing list to this conversation. Others in the 
> community might find this conversation valuable.
>
> We do want to run testing on a regular basis. The Jenkins integration 
> will be very useful for us, as most of our CI is orchestrated by Jenkins.
> I've conferred with the rest of the team, and we think it would be 
> best to move forward with mainly option B.
> If you would like to know anything about our testbeds that would help 
> you with creating an example ts-rigs repo, I'd be happy to answer any 
> questions you have.
>
> We have multiple test rigs (we call these "DUT-tester pairs") that we 
> run our existing hardware testing on, with differing network hardware 
> and CPU architecture. I figured this might be an important detail.
>
> Thanks,
> Adam
>
> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Greatings Adam,
>
>     I'm happy to hear that you're trying to bring it up.
>
>     As I understand the final goal is to run it on regular basis. So,
>     we need to make it properly from the very beginning.
>     Bring up of all features consists of 4 steps:
>
>     1. Create site-specific repository (we call it ts-rigs) which
>     contains information about test rigs and other site-specific
>     information like where to send mails, where to store logs etc. It
>     is required for manual execution as well, since test rigs
>     description is essential. I'll return to the topic below.
>
>     2. Setup logs storage for automated runs. Basically it is a disk
>     space plus apache2 web server with few CGI scripts which help a
>     lot to save disk space.
>
>     3. Setup Bublik web application which provides web interface to
>     view testing results. Same as https://ts-factory.io/bublik
>
>     4. Setup Jenkins to run tests on regularly, save logs in log
>     storage (2) and import it to bublik (3).
>
>     Last few month we spent on our homework to make it simpler to
>     bring up automated execution using Jenkins -
>     https://github.com/ts-factory/te-jenkins
>     Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>
>     Let's return to the step (1).
>
>     Unfortunately there is no publicly available example of the
>     ts-rigs repository since sensitive site-specific information is
>     located there. But I'm ready to help you to create it for UNH. I
>     see two options here:
>
>     (A) I'll ask questions and based on your answers will create the
>     first draft with my comments.
>
>     (B) I'll make a template/example ts-rigs repo, publish it and
>     you'll create UNH ts-rigs based on it.
>
>     Of course, I'll help to debug and finally bring it up in any case.
>
>     (A) is a bit simpler for me and you, but (B) is a bit more generic
>     and will help other potential users to bring it up.
>     We can combine (A)+(B). I.e. start from (A). What do you think?
>
>     Thanks,
>     Andrew.
>
>     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>     Greetings Adam,
>>
>>
>>     Thanks for contacting us. I copy Andrew who would be happy to help
>>
>>     Thanks,
>>     Konstantin
>>
>>>     On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>
>>>     
>>>     Greetings Konstantin,
>>>
>>>     I am in the process of setting up the DPDK Poll Mode Driver test
>>>     suite as an addition to our testing coverage for DPDK at the UNH
>>>     lab.
>>>
>>>     I have some questions about how to set the test suite arguments.
>>>
>>>     I have been able to configure the Test Engine to connect to the
>>>     hosts in the testbed. The RCF, Configurator, and Tester all
>>>     begin to run, however the prelude of the test suite fails to run.
>>>
>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>
>>>     The documentation mentions that there are several test
>>>     parameters for the test suite, like for the IUT test link MAC,
>>>     etc. These seem like they would need to be set somewhere to run
>>>     many of the tests.
>>>
>>>     I see in the Test Engine documentation, there are instructions
>>>     on how to create new parameters for test suites in the Tester
>>>     configuration, but there is nothing in the user guide or in the
>>>     Tester guide for how to set the arguments for the parameters
>>>     when running the test suite that I can find. I'm not sure if I
>>>     need to write my own Tester config, or if I should be setting
>>>     these in some other way.
>>>
>>>     How should these values be set?
>>>
>>>     I'm also not sure what environment variables/arguments are
>>>     strictly necessary or which are optional.
>>>
>>>     Regards,
>>>     Adam
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 13621 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-18 18:40       ` Andrew Rybchenko
@ 2023-08-20  8:40         ` Andrew Rybchenko
  2023-08-21 14:19           ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-20  8:40 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 6053 bytes --]

Hi Adam,

I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully 
it will help to define your test rigs and successfully run some tests 
manually. Feel free to ask any questions and I'll answer here and try to 
update documentation.

Meanwhile I'll prepare missing bits for steps (2) and (3).
Hopefully everything is in place for step (4), but we need to make steps 
(2) and (3) first.

Andrew.

On 8/18/23 21:40, Andrew Rybchenko wrote:
> Hi Adam,
>
> > I've conferred with the rest of the team, and we think it would be 
> best to move forward with mainly option B.
>
> OK, I'll provide the sample on Monday for you. It is almost ready 
> right now, but I need to double-check it before publishing.
>
> Regards,
> Andrew.
>
> On 8/17/23 20:03, Adam Hassick wrote:
>> Hi Andrew,
>>
>> I'm adding the CI mailing list to this conversation. Others in the 
>> community might find this conversation valuable.
>>
>> We do want to run testing on a regular basis. The Jenkins integration 
>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>> I've conferred with the rest of the team, and we think it would be 
>> best to move forward with mainly option B.
>> If you would like to know anything about our testbeds that would help 
>> you with creating an example ts-rigs repo, I'd be happy to answer any 
>> questions you have.
>>
>> We have multiple test rigs (we call these "DUT-tester pairs") that we 
>> run our existing hardware testing on, with differing network hardware 
>> and CPU architecture. I figured this might be an important detail.
>>
>> Thanks,
>> Adam
>>
>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>     Greatings Adam,
>>
>>     I'm happy to hear that you're trying to bring it up.
>>
>>     As I understand the final goal is to run it on regular basis. So,
>>     we need to make it properly from the very beginning.
>>     Bring up of all features consists of 4 steps:
>>
>>     1. Create site-specific repository (we call it ts-rigs) which
>>     contains information about test rigs and other site-specific
>>     information like where to send mails, where to store logs etc. It
>>     is required for manual execution as well, since test rigs
>>     description is essential. I'll return to the topic below.
>>
>>     2. Setup logs storage for automated runs. Basically it is a disk
>>     space plus apache2 web server with few CGI scripts which help a
>>     lot to save disk space.
>>
>>     3. Setup Bublik web application which provides web interface to
>>     view testing results. Same as https://ts-factory.io/bublik
>>
>>     4. Setup Jenkins to run tests on regularly, save logs in log
>>     storage (2) and import it to bublik (3).
>>
>>     Last few month we spent on our homework to make it simpler to
>>     bring up automated execution using Jenkins -
>>     https://github.com/ts-factory/te-jenkins
>>     Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>
>>     Let's return to the step (1).
>>
>>     Unfortunately there is no publicly available example of the
>>     ts-rigs repository since sensitive site-specific information is
>>     located there. But I'm ready to help you to create it for UNH. I
>>     see two options here:
>>
>>     (A) I'll ask questions and based on your answers will create the
>>     first draft with my comments.
>>
>>     (B) I'll make a template/example ts-rigs repo, publish it and
>>     you'll create UNH ts-rigs based on it.
>>
>>     Of course, I'll help to debug and finally bring it up in any case.
>>
>>     (A) is a bit simpler for me and you, but (B) is a bit more
>>     generic and will help other potential users to bring it up.
>>     We can combine (A)+(B). I.e. start from (A). What do you think?
>>
>>     Thanks,
>>     Andrew.
>>
>>     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>     Greetings Adam,
>>>
>>>
>>>     Thanks for contacting us. I copy Andrew who would be happy to help
>>>
>>>     Thanks,
>>>     Konstantin
>>>
>>>>     On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>>
>>>>     
>>>>     Greetings Konstantin,
>>>>
>>>>     I am in the process of setting up the DPDK Poll Mode Driver
>>>>     test suite as an addition to our testing coverage for DPDK at
>>>>     the UNH lab.
>>>>
>>>>     I have some questions about how to set the test suite arguments.
>>>>
>>>>     I have been able to configure the Test Engine to connect to the
>>>>     hosts in the testbed. The RCF, Configurator, and Tester all
>>>>     begin to run, however the prelude of the test suite fails to run.
>>>>
>>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>
>>>>     The documentation mentions that there are several test
>>>>     parameters for the test suite, like for the IUT test link MAC,
>>>>     etc. These seem like they would need to be set somewhere to run
>>>>     many of the tests.
>>>>
>>>>     I see in the Test Engine documentation, there are instructions
>>>>     on how to create new parameters for test suites in the Tester
>>>>     configuration, but there is nothing in the user guide or in the
>>>>     Tester guide for how to set the arguments for the parameters
>>>>     when running the test suite that I can find. I'm not sure if I
>>>>     need to write my own Tester config, or if I should be setting
>>>>     these in some other way.
>>>>
>>>>     How should these values be set?
>>>>
>>>>     I'm also not sure what environment variables/arguments are
>>>>     strictly necessary or which are optional.
>>>>
>>>>     Regards,
>>>>     Adam
>>>>
>>>>     -- 
>>>>     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu
>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 15205 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-20  8:40         ` Andrew Rybchenko
@ 2023-08-21 14:19           ` Adam Hassick
  2023-08-23 14:45             ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-21 14:19 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 6258 bytes --]

Hi Andrew,

Thanks, I've cloned the example repository and will start setting up a
configuration for our development testbed today. I'll let you know if I run
into any difficulties or have any questions.

 - Adam

On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully it
> will help to define your test rigs and successfully run some tests
> manually. Feel free to ask any questions and I'll answer here and try to
> update documentation.
>
> Meanwhile I'll prepare missing bits for steps (2) and (3).
> Hopefully everything is in place for step (4), but we need to make steps
> (2) and (3) first.
>
> Andrew.
>
> On 8/18/23 21:40, Andrew Rybchenko wrote:
>
> Hi Adam,
>
> > I've conferred with the rest of the team, and we think it would be best
> to move forward with mainly option B.
>
> OK, I'll provide the sample on Monday for you. It is almost ready right
> now, but I need to double-check it before publishing.
>
> Regards,
> Andrew.
>
> On 8/17/23 20:03, Adam Hassick wrote:
>
> Hi Andrew,
>
> I'm adding the CI mailing list to this conversation. Others in the
> community might find this conversation valuable.
>
> We do want to run testing on a regular basis. The Jenkins integration will
> be very useful for us, as most of our CI is orchestrated by Jenkins.
> I've conferred with the rest of the team, and we think it would be best to
> move forward with mainly option B.
> If you would like to know anything about our testbeds that would help you
> with creating an example ts-rigs repo, I'd be happy to answer any questions
> you have.
>
> We have multiple test rigs (we call these "DUT-tester pairs") that we run
> our existing hardware testing on, with differing network hardware and CPU
> architecture. I figured this might be an important detail.
>
> Thanks,
> Adam
>
> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Greatings Adam,
>>
>> I'm happy to hear that you're trying to bring it up.
>>
>> As I understand the final goal is to run it on regular basis. So, we need
>> to make it properly from the very beginning.
>> Bring up of all features consists of 4 steps:
>>
>> 1. Create site-specific repository (we call it ts-rigs) which contains
>> information about test rigs and other site-specific information like where
>> to send mails, where to store logs etc. It is required for manual execution
>> as well, since test rigs description is essential. I'll return to the topic
>> below.
>>
>> 2. Setup logs storage for automated runs. Basically it is a disk space
>> plus apache2 web server with few CGI scripts which help a lot to save disk
>> space.
>>
>> 3. Setup Bublik web application which provides web interface to view
>> testing results. Same as https://ts-factory.io/bublik
>>
>> 4. Setup Jenkins to run tests on regularly, save logs in log storage (2)
>> and import it to bublik (3).
>>
>> Last few month we spent on our homework to make it simpler to bring up
>> automated execution using Jenkins -
>> https://github.com/ts-factory/te-jenkins
>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>
>> Let's return to the step (1).
>>
>> Unfortunately there is no publicly available example of the ts-rigs
>> repository since sensitive site-specific information is located there. But
>> I'm ready to help you to create it for UNH. I see two options here:
>>
>> (A) I'll ask questions and based on your answers will create the first
>> draft with my comments.
>>
>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>> create UNH ts-rigs based on it.
>>
>> Of course, I'll help to debug and finally bring it up in any case.
>>
>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>> will help other potential users to bring it up.
>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>
>> Thanks,
>> Andrew.
>>
>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>
>> Greetings Adam,
>>
>>
>> Thanks for contacting us. I copy Andrew who would be happy to help
>>
>> Thanks,
>> Konstantin
>>
>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu> wrote:
>>
>> 
>> Greetings Konstantin,
>>
>> I am in the process of setting up the DPDK Poll Mode Driver test suite as
>> an addition to our testing coverage for DPDK at the UNH lab.
>>
>> I have some questions about how to set the test suite arguments.
>>
>> I have been able to configure the Test Engine to connect to the hosts in
>> the testbed. The RCF, Configurator, and Tester all begin to run, however
>> the prelude of the test suite fails to run.
>>
>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>
>> The documentation mentions that there are several test parameters for the
>> test suite, like for the IUT test link MAC, etc. These seem like they would
>> need to be set somewhere to run many of the tests.
>>
>> I see in the Test Engine documentation, there are instructions on how to
>> create new parameters for test suites in the Tester configuration, but
>> there is nothing in the user guide or in the Tester guide for how to set
>> the arguments for the parameters when running the test suite that I can
>> find. I'm not sure if I need to write my own Tester config, or if I should
>> be setting these in some other way.
>>
>> How should these values be set?
>>
>> I'm also not sure what environment variables/arguments are strictly
>> necessary or which are optional.
>>
>> Regards,
>> Adam
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 15492 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-21 14:19           ` Adam Hassick
@ 2023-08-23 14:45             ` Adam Hassick
  2023-08-24  8:22               ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-23 14:45 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 7543 bytes --]

Hi Andrew,

I've set up a test rig repository here, and have created configurations for
our development testbed based off of the examples.
We've been able to get the test suite to run manually on Mellanox CX5
devices once.
However, we are running into an issue where, when RCF starts, the RCF
connection times out very frequently. We aren't sure why this is the case.
It works sometimes, but most of the time when we try to run the test
engine, it encounters this issue.
I've tried changing the RCF port by setting "TE_RCF_PORT=<some port
number>" and rebooting the testbed machines. Neither seems to fix the issue.

It also seems like the timeout takes far longer than 60 seconds, even when
running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the test suite.
I assume the unit for this variable is seconds?

Thanks,
Adam

On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu> wrote:

> Hi Andrew,
>
> Thanks, I've cloned the example repository and will start setting up a
> configuration for our development testbed today. I'll let you know if I run
> into any difficulties or have any questions.
>
>  - Adam
>
> On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully
>> it will help to define your test rigs and successfully run some tests
>> manually. Feel free to ask any questions and I'll answer here and try to
>> update documentation.
>>
>> Meanwhile I'll prepare missing bits for steps (2) and (3).
>> Hopefully everything is in place for step (4), but we need to make steps
>> (2) and (3) first.
>>
>> Andrew.
>>
>> On 8/18/23 21:40, Andrew Rybchenko wrote:
>>
>> Hi Adam,
>>
>> > I've conferred with the rest of the team, and we think it would be best
>> to move forward with mainly option B.
>>
>> OK, I'll provide the sample on Monday for you. It is almost ready right
>> now, but I need to double-check it before publishing.
>>
>> Regards,
>> Andrew.
>>
>> On 8/17/23 20:03, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> I'm adding the CI mailing list to this conversation. Others in the
>> community might find this conversation valuable.
>>
>> We do want to run testing on a regular basis. The Jenkins integration
>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>> I've conferred with the rest of the team, and we think it would be best
>> to move forward with mainly option B.
>> If you would like to know anything about our testbeds that would help you
>> with creating an example ts-rigs repo, I'd be happy to answer any questions
>> you have.
>>
>> We have multiple test rigs (we call these "DUT-tester pairs") that we run
>> our existing hardware testing on, with differing network hardware and CPU
>> architecture. I figured this might be an important detail.
>>
>> Thanks,
>> Adam
>>
>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Greatings Adam,
>>>
>>> I'm happy to hear that you're trying to bring it up.
>>>
>>> As I understand the final goal is to run it on regular basis. So, we
>>> need to make it properly from the very beginning.
>>> Bring up of all features consists of 4 steps:
>>>
>>> 1. Create site-specific repository (we call it ts-rigs) which contains
>>> information about test rigs and other site-specific information like where
>>> to send mails, where to store logs etc. It is required for manual execution
>>> as well, since test rigs description is essential. I'll return to the topic
>>> below.
>>>
>>> 2. Setup logs storage for automated runs. Basically it is a disk space
>>> plus apache2 web server with few CGI scripts which help a lot to save disk
>>> space.
>>>
>>> 3. Setup Bublik web application which provides web interface to view
>>> testing results. Same as https://ts-factory.io/bublik
>>>
>>> 4. Setup Jenkins to run tests on regularly, save logs in log storage (2)
>>> and import it to bublik (3).
>>>
>>> Last few month we spent on our homework to make it simpler to bring up
>>> automated execution using Jenkins -
>>> https://github.com/ts-factory/te-jenkins
>>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>>
>>> Let's return to the step (1).
>>>
>>> Unfortunately there is no publicly available example of the ts-rigs
>>> repository since sensitive site-specific information is located there. But
>>> I'm ready to help you to create it for UNH. I see two options here:
>>>
>>> (A) I'll ask questions and based on your answers will create the first
>>> draft with my comments.
>>>
>>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>>> create UNH ts-rigs based on it.
>>>
>>> Of course, I'll help to debug and finally bring it up in any case.
>>>
>>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>>> will help other potential users to bring it up.
>>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>>
>>> Thanks,
>>> Andrew.
>>>
>>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>
>>> Greetings Adam,
>>>
>>>
>>> Thanks for contacting us. I copy Andrew who would be happy to help
>>>
>>> Thanks,
>>> Konstantin
>>>
>>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu> wrote:
>>>
>>> 
>>> Greetings Konstantin,
>>>
>>> I am in the process of setting up the DPDK Poll Mode Driver test suite
>>> as an addition to our testing coverage for DPDK at the UNH lab.
>>>
>>> I have some questions about how to set the test suite arguments.
>>>
>>> I have been able to configure the Test Engine to connect to the hosts in
>>> the testbed. The RCF, Configurator, and Tester all begin to run, however
>>> the prelude of the test suite fails to run.
>>>
>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>
>>> The documentation mentions that there are several test parameters for
>>> the test suite, like for the IUT test link MAC, etc. These seem like they
>>> would need to be set somewhere to run many of the tests.
>>>
>>> I see in the Test Engine documentation, there are instructions on how to
>>> create new parameters for test suites in the Tester configuration, but
>>> there is nothing in the user guide or in the Tester guide for how to set
>>> the arguments for the parameters when running the test suite that I can
>>> find. I'm not sure if I need to write my own Tester config, or if I should
>>> be setting these in some other way.
>>>
>>> How should these values be set?
>>>
>>> I'm also not sure what environment variables/arguments are strictly
>>> necessary or which are optional.
>>>
>>> Regards,
>>> Adam
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>


-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 17766 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-23 14:45             ` Adam Hassick
@ 2023-08-24  8:22               ` Andrew Rybchenko
  2023-08-24 14:30                 ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-24  8:22 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 10146 bytes --]

Hi Adam,

Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it goes 
to 'copy_timeout' in ts-conf/rcf.conf.
Description in in doc/sphinx/pages/group_te_engine_rcf.rst says that 
copy_timeout is in seconds and implementation in lib/rcfunix/rcfunix.c 
passes the value to select() tv_sec. Theoretically select() could be 
interrupted by signal, but I think it is unlikely here.

I'm not sure that I understand what do you mean by RCF connection 
timeout. Does it happen on TE startup when RCF starts test agents. If 
so, TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests are in 
progress, e.g. in the middle of a test. If so, TE_RCFUNIX_TIMEOUT is 
unrelated and most likely either host with test agent dies or test agent 
itself crashes. It would be easier for me if classify it if you share 
text log (log.txt, full or just corresponding fragment with some 
context). Also content of ta.DPDK or ta.Peer file depending on which 
agent has problems could shed some light. Corresponding files contain 
stdout/stderr of test agents.

Andrew.

On 8/23/23 17:45, Adam Hassick wrote:
> Hi Andrew,
>
> I've set up a test rig repository here, and have created 
> configurations for our development testbed based off of the examples.
> We've been able to get the test suite to run manually on Mellanox CX5 
> devices once.
> However, we are running into an issue where, when RCF starts, the RCF 
> connection times out very frequently. We aren't sure why this is the case.
> It works sometimes, but most of the time when we try to run the test 
> engine, it encounters this issue.
> I've tried changing the RCF port by setting "TE_RCF_PORT=<some port 
> number>" and rebooting the testbed machines. Neither seems to fix the 
> issue.
>
> It also seems like the timeout takes far longer than 60 seconds, even 
> when running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the 
> test suite.
> I assume the unit for this variable is seconds?
>
> Thanks,
> Adam
>
> On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu> 
> wrote:
>
>     Hi Andrew,
>
>     Thanks, I've cloned the example repository and will start setting
>     up a configuration for our development testbed today. I'll let you
>     know if I run into any difficulties or have any questions.
>
>      - Adam
>
>     On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>     <andrew.rybchenko@oktetlabs.ru> wrote:
>
>         Hi Adam,
>
>         I've published https://github.com/ts-factory/ts-rigs-sample.
>         Hopefully it will help to define your test rigs and
>         successfully run some tests manually. Feel free to ask any
>         questions and I'll answer here and try to update documentation.
>
>         Meanwhile I'll prepare missing bits for steps (2) and (3).
>         Hopefully everything is in place for step (4), but we need to
>         make steps (2) and (3) first.
>
>         Andrew.
>
>         On 8/18/23 21:40, Andrew Rybchenko wrote:
>>         Hi Adam,
>>
>>         > I've conferred with the rest of the team, and we think it
>>         would be best to move forward with mainly option B.
>>
>>         OK, I'll provide the sample on Monday for you. It is almost
>>         ready right now, but I need to double-check it before publishing.
>>
>>         Regards,
>>         Andrew.
>>
>>         On 8/17/23 20:03, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         I'm adding the CI mailing list to this conversation. Others
>>>         in the community might find this conversation valuable.
>>>
>>>         We do want to run testing on a regular basis. The Jenkins
>>>         integration will be very useful for us, as most of our CI is
>>>         orchestrated by Jenkins.
>>>         I've conferred with the rest of the team, and we think it
>>>         would be best to move forward with mainly option B.
>>>         If you would like to know anything about our testbeds that
>>>         would help you with creating an example ts-rigs repo, I'd be
>>>         happy to answer any questions you have.
>>>
>>>         We have multiple test rigs (we call these "DUT-tester
>>>         pairs") that we run our existing hardware testing on, with
>>>         differing network hardware and CPU architecture. I figured
>>>         this might be an important detail.
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>             Greatings Adam,
>>>
>>>             I'm happy to hear that you're trying to bring it up.
>>>
>>>             As I understand the final goal is to run it on regular
>>>             basis. So, we need to make it properly from the very
>>>             beginning.
>>>             Bring up of all features consists of 4 steps:
>>>
>>>             1. Create site-specific repository (we call it ts-rigs)
>>>             which contains information about test rigs and other
>>>             site-specific information like where to send mails,
>>>             where to store logs etc. It is required for manual
>>>             execution as well, since test rigs description is
>>>             essential. I'll return to the topic below.
>>>
>>>             2. Setup logs storage for automated runs. Basically it
>>>             is a disk space plus apache2 web server with few CGI
>>>             scripts which help a lot to save disk space.
>>>
>>>             3. Setup Bublik web application which provides web
>>>             interface to view testing results. Same as
>>>             https://ts-factory.io/bublik
>>>
>>>             4. Setup Jenkins to run tests on regularly, save logs in
>>>             log storage (2) and import it to bublik (3).
>>>
>>>             Last few month we spent on our homework to make it
>>>             simpler to bring up automated execution using Jenkins -
>>>             https://github.com/ts-factory/te-jenkins
>>>             Corresponding bits in dpdk-ethdev-ts will be available
>>>             tomorrow.
>>>
>>>             Let's return to the step (1).
>>>
>>>             Unfortunately there is no publicly available example of
>>>             the ts-rigs repository since sensitive site-specific
>>>             information is located there. But I'm ready to help you
>>>             to create it for UNH. I see two options here:
>>>
>>>             (A) I'll ask questions and based on your answers will
>>>             create the first draft with my comments.
>>>
>>>             (B) I'll make a template/example ts-rigs repo, publish
>>>             it and you'll create UNH ts-rigs based on it.
>>>
>>>             Of course, I'll help to debug and finally bring it up in
>>>             any case.
>>>
>>>             (A) is a bit simpler for me and you, but (B) is a bit
>>>             more generic and will help other potential users to
>>>             bring it up.
>>>             We can combine (A)+(B). I.e. start from (A). What do you
>>>             think?
>>>
>>>             Thanks,
>>>             Andrew.
>>>
>>>             On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>             Greetings Adam,
>>>>
>>>>
>>>>             Thanks for contacting us. I copy Andrew who would be
>>>>             happy to help
>>>>
>>>>             Thanks,
>>>>             Konstantin
>>>>
>>>>>             On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>             <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>             wrote:
>>>>>
>>>>>             
>>>>>             Greetings Konstantin,
>>>>>
>>>>>             I am in the process of setting up the DPDK Poll Mode
>>>>>             Driver test suite as an addition to our testing
>>>>>             coverage for DPDK at the UNH lab.
>>>>>
>>>>>             I have some questions about how to set the test suite
>>>>>             arguments.
>>>>>
>>>>>             I have been able to configure the Test Engine to
>>>>>             connect to the hosts in the testbed. The RCF,
>>>>>             Configurator, and Tester all begin to run, however the
>>>>>             prelude of the test suite fails to run.
>>>>>
>>>>>             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>
>>>>>             The documentation mentions that there are several test
>>>>>             parameters for the test suite, like for the IUT test
>>>>>             link MAC, etc. These seem like they would need to be
>>>>>             set somewhere to run many of the tests.
>>>>>
>>>>>             I see in the Test Engine documentation, there are
>>>>>             instructions on how to create new parameters for test
>>>>>             suites in the Tester configuration, but there is
>>>>>             nothing in the user guide or in the Tester guide for
>>>>>             how to set the arguments for the parameters when
>>>>>             running the test suite that I can find. I'm not sure
>>>>>             if I need to write my own Tester config, or if I
>>>>>             should be setting these in some other way.
>>>>>
>>>>>             How should these values be set?
>>>>>
>>>>>             I'm also not sure what environment variables/arguments
>>>>>             are strictly necessary or which are optional.
>>>>>
>>>>>             Regards,
>>>>>             Adam
>>>>>
>>>>>             -- 
>>>>>             *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>             ahassick@iol.unh.edu
>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>
>
>
>     -- 
>     *Adam Hassick*
>     Senior Developer
>     UNH InterOperability Lab
>     ahassick@iol.unh.edu
>     iol.unh.edu <https://www.iol.unh.edu/>
>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 28191 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-24  8:22               ` Andrew Rybchenko
@ 2023-08-24 14:30                 ` Adam Hassick
  2023-08-24 18:34                   ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-24 14:30 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci


[-- Attachment #1.1: Type: text/plain, Size: 9906 bytes --]

Hi Andrew,

This is the output that I see in the terminal when this failure occurs,
after the test agent binaries build and the test engine starts:

Platform default build - pass
Simple RCF consistency check succeeded
--->>> Starting Logger...done
--->>> Starting RCF...rcf_net_engine_connect(): Connection timed out
iol-dts-tester.dpdklab.iol.unh.edu:23571

Then, it hangs here until I kill the "te_rcf" and "te_tee" processes. I let
it hang for around 9 minutes.

On the tester host (which appears to be the Peer agent), there are four
processes that I see running, which look like the test agent processes.

ta.Peer is an empty file. I've attached the log.txt from this run.

 - Adam

On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it goes to
> 'copy_timeout' in ts-conf/rcf.conf.
> Description in in doc/sphinx/pages/group_te_engine_rcf.rst says that
> copy_timeout is in seconds and implementation in lib/rcfunix/rcfunix.c
> passes the value to select() tv_sec. Theoretically select() could be
> interrupted by signal, but I think it is unlikely here.
>
> I'm not sure that I understand what do you mean by RCF connection timeout.
> Does it happen on TE startup when RCF starts test agents. If so,
> TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests are in
> progress, e.g. in the middle of a test. If so, TE_RCFUNIX_TIMEOUT is
> unrelated and most likely either host with test agent dies or test agent
> itself crashes. It would be easier for me if classify it if you share text
> log (log.txt, full or just corresponding fragment with some context). Also
> content of ta.DPDK or ta.Peer file depending on which agent has problems
> could shed some light. Corresponding files contain stdout/stderr of test
> agents.
>
> Andrew.
>
> On 8/23/23 17:45, Adam Hassick wrote:
>
> Hi Andrew,
>
> I've set up a test rig repository here, and have created configurations
> for our development testbed based off of the examples.
> We've been able to get the test suite to run manually on Mellanox CX5
> devices once.
> However, we are running into an issue where, when RCF starts, the RCF
> connection times out very frequently. We aren't sure why this is the case.
> It works sometimes, but most of the time when we try to run the test
> engine, it encounters this issue.
> I've tried changing the RCF port by setting "TE_RCF_PORT=<some port
> number>" and rebooting the testbed machines. Neither seems to fix the issue.
>
> It also seems like the timeout takes far longer than 60 seconds, even when
> running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the test suite.
> I assume the unit for this variable is seconds?
>
> Thanks,
> Adam
>
> On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu>
> wrote:
>
>> Hi Andrew,
>>
>> Thanks, I've cloned the example repository and will start setting up a
>> configuration for our development testbed today. I'll let you know if I run
>> into any difficulties or have any questions.
>>
>>  - Adam
>>
>> On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully
>>> it will help to define your test rigs and successfully run some tests
>>> manually. Feel free to ask any questions and I'll answer here and try to
>>> update documentation.
>>>
>>> Meanwhile I'll prepare missing bits for steps (2) and (3).
>>> Hopefully everything is in place for step (4), but we need to make steps
>>> (2) and (3) first.
>>>
>>> Andrew.
>>>
>>> On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>
>>> Hi Adam,
>>>
>>> > I've conferred with the rest of the team, and we think it would be
>>> best to move forward with mainly option B.
>>>
>>> OK, I'll provide the sample on Monday for you. It is almost ready right
>>> now, but I need to double-check it before publishing.
>>>
>>> Regards,
>>> Andrew.
>>>
>>> On 8/17/23 20:03, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> I'm adding the CI mailing list to this conversation. Others in the
>>> community might find this conversation valuable.
>>>
>>> We do want to run testing on a regular basis. The Jenkins integration
>>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>>> I've conferred with the rest of the team, and we think it would be best
>>> to move forward with mainly option B.
>>> If you would like to know anything about our testbeds that would help
>>> you with creating an example ts-rigs repo, I'd be happy to answer any
>>> questions you have.
>>>
>>> We have multiple test rigs (we call these "DUT-tester pairs") that we
>>> run our existing hardware testing on, with differing network hardware and
>>> CPU architecture. I figured this might be an important detail.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Greatings Adam,
>>>>
>>>> I'm happy to hear that you're trying to bring it up.
>>>>
>>>> As I understand the final goal is to run it on regular basis. So, we
>>>> need to make it properly from the very beginning.
>>>> Bring up of all features consists of 4 steps:
>>>>
>>>> 1. Create site-specific repository (we call it ts-rigs) which contains
>>>> information about test rigs and other site-specific information like where
>>>> to send mails, where to store logs etc. It is required for manual execution
>>>> as well, since test rigs description is essential. I'll return to the topic
>>>> below.
>>>>
>>>> 2. Setup logs storage for automated runs. Basically it is a disk space
>>>> plus apache2 web server with few CGI scripts which help a lot to save disk
>>>> space.
>>>>
>>>> 3. Setup Bublik web application which provides web interface to view
>>>> testing results. Same as https://ts-factory.io/bublik
>>>>
>>>> 4. Setup Jenkins to run tests on regularly, save logs in log storage
>>>> (2) and import it to bublik (3).
>>>>
>>>> Last few month we spent on our homework to make it simpler to bring up
>>>> automated execution using Jenkins -
>>>> https://github.com/ts-factory/te-jenkins
>>>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>>>
>>>> Let's return to the step (1).
>>>>
>>>> Unfortunately there is no publicly available example of the ts-rigs
>>>> repository since sensitive site-specific information is located there. But
>>>> I'm ready to help you to create it for UNH. I see two options here:
>>>>
>>>> (A) I'll ask questions and based on your answers will create the first
>>>> draft with my comments.
>>>>
>>>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>>>> create UNH ts-rigs based on it.
>>>>
>>>> Of course, I'll help to debug and finally bring it up in any case.
>>>>
>>>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>>>> will help other potential users to bring it up.
>>>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>>>
>>>> Thanks,
>>>> Andrew.
>>>>
>>>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>
>>>> Greetings Adam,
>>>>
>>>>
>>>> Thanks for contacting us. I copy Andrew who would be happy to help
>>>>
>>>> Thanks,
>>>> Konstantin
>>>>
>>>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu> wrote:
>>>>
>>>> 
>>>> Greetings Konstantin,
>>>>
>>>> I am in the process of setting up the DPDK Poll Mode Driver test suite
>>>> as an addition to our testing coverage for DPDK at the UNH lab.
>>>>
>>>> I have some questions about how to set the test suite arguments.
>>>>
>>>> I have been able to configure the Test Engine to connect to the hosts
>>>> in the testbed. The RCF, Configurator, and Tester all begin to run, however
>>>> the prelude of the test suite fails to run.
>>>>
>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>
>>>> The documentation mentions that there are several test parameters for
>>>> the test suite, like for the IUT test link MAC, etc. These seem like they
>>>> would need to be set somewhere to run many of the tests.
>>>>
>>>> I see in the Test Engine documentation, there are instructions on how
>>>> to create new parameters for test suites in the Tester configuration, but
>>>> there is nothing in the user guide or in the Tester guide for how to set
>>>> the arguments for the parameters when running the test suite that I can
>>>> find. I'm not sure if I need to write my own Tester config, or if I should
>>>> be setting these in some other way.
>>>>
>>>> How should these values be set?
>>>>
>>>> I'm also not sure what environment variables/arguments are strictly
>>>> necessary or which are optional.
>>>>
>>>> Regards,
>>>> Adam
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #1.2: Type: text/html, Size: 27971 bytes --]

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 3752 bytes --]

Log report
~~~~~~~~~~

RING  Dispatcher  Command-line options  13:55:53.387
--conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --opts=run/iol-dts-mcx5 --opts=opts.ts

RING  Dispatcher  Expanded command-line options  13:55:53.402
 --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --script=env/iol-dts --script=env/mlx-cx5 --script=scripts/iut.h1 --script=scripts/iut.h1-mcx5 --conf-cs=cs/dpdk-pmd-ts.yml --script=scripts/ta-def --script=scripts/defaults --tester-script=scripts/dpdk-trc-tags --tester-script=scripts/os-trc-tags --script=scripts/net-modules --script=scripts/iut-net-driver-loaded --script=scripts/disable_unused_agts

RING  Dispatcher  Start  14:01:03.417
Starting TEN applications

RING  Dispatcher  Start  14:01:03.435
Start Logger:  /opt/tsf/ts-rigs/logger.conf

RING  Logger  Cfg file  14:01:03.461
Opening config file: /opt/tsf/ts-rigs/logger.conf

RING  Logger  Log streaming  14:01:03.462
Current listeners configuration:
Listeners:
Filters:

RING  Dispatcher  Start  14:01:03.481
Start RCF:  /opt/tsf/ts-conf/rcf.conf

RING  RCF  RCF Unix  14:01:03.487
Starting TA 'Peer' type 'linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell='

RING  RCF  RCF Unix  14:01:03.487
CMD to copy: ssh -qxTn -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_76872_1692885663_1" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_76872_1692885663_1 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  14:01:05.063
Command to detect shell name: ssh -qxTn -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  14:01:05.321
Shell is: /bin/bash

RING  RCF  RCF Unix  14:01:05.321
Command to start TA: ssh -qxTn -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_76872_1692885663_1 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=" 2>&1 | te_tee RCF Peer 10 >ta.Peer 

WARN  RCF  RCF Unix  14:11:16.984
Connecting to TA Peer iol-dts-tester.dpdklab.iol.unh.edu:23571 failed (COMM-ETIMEDOUT) - connect again after delay

RING  RCF  RCF Unix  14:11:16.984
Sleeping 1 seconds

RING  Dispatcher  Start  14:20:25.622
Shutdown Logger

RING  Logger  Self  14:20:25.624
Logger shutdown ...

WARN  Logger  Self  14:20:25.624
Logger is shut down without polling of TAs

WARN  Logger  Log streaming  14:20:25.624
Not all messages in listener queue have been processed

RING  Logger  Self  14:20:25.624
Shutdown is completed

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-24 14:30                 ` Adam Hassick
@ 2023-08-24 18:34                   ` Andrew Rybchenko
  2023-08-24 20:29                     ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-24 18:34 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 13978 bytes --]

Hi Adam,

 > On the tester host (which appears to be the Peer agent), there are 
four processes that I see running, which look like the test agent processes.

Before the next try I'd recommend to kill these processes.

Is there any firewall in the network or on test hosts which could block 
incoming TCP connection to the port 23571 
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where 
you run test engine?

If behaviour the same on the next try and you see that test agent is 
kept running, could you check using

# netstat -tnlp

that Test Agent is listening on the port and try to establish TCP 
connection from test agent using

$ telnet iol-dts-tester.dpdklab.iol.unh.edu 
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571 
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571>

and check if TCP connection could be established.

Another idea is to login Tester under root as testing does, get start TA 
command from the log and try it by hands without -n and remove extra 
escaping.

# sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1 
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=

Hopefully in this case test agent directory remains in the /tmp and you 
don't need to copy it as testing does.
May be output could shed some light on what's going on.

Andrew.

On 8/24/23 17:30, Adam Hassick wrote:
> Hi Andrew,
>
> This is the output that I see in the terminal when this failure 
> occurs, after the test agent binaries build and the test engine starts:
>
> Platform default build - pass
> Simple RCF consistency check succeeded
> --->>> Starting Logger...done
> --->>> Starting RCF...rcf_net_engine_connect(): Connection timed out 
> iol-dts-tester.dpdklab.iol.unh.edu:23571 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
> Then, it hangs here until I kill the "te_rcf" and "te_tee" processes. 
> I let it hang for around 9 minutes.
>
> On the tester host (which appears to be the Peer agent), there are 
> four processes that I see running, which look like the test agent 
> processes.
>
> ta.Peer is an empty file. I've attached the log.txt from this run.
>
>  - Adam
>
> On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it
>     goes to 'copy_timeout' in ts-conf/rcf.conf.
>     Description in in doc/sphinx/pages/group_te_engine_rcf.rst says
>     that copy_timeout is in seconds and implementation in
>     lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>     Theoretically select() could be interrupted by signal, but I think
>     it is unlikely here.
>
>     I'm not sure that I understand what do you mean by RCF connection
>     timeout. Does it happen on TE startup when RCF starts test agents.
>     If so, TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests
>     are in progress, e.g. in the middle of a test. If so,
>     TE_RCFUNIX_TIMEOUT is unrelated and most likely either host with
>     test agent dies or test agent itself crashes. It would be easier
>     for me if classify it if you share text log (log.txt, full or just
>     corresponding fragment with some context). Also content of ta.DPDK
>     or ta.Peer file depending on which agent has problems could shed
>     some light. Corresponding files contain stdout/stderr of test agents.
>
>     Andrew.
>
>     On 8/23/23 17:45, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     I've set up a test rig repository here, and have created
>>     configurations for our development testbed based off of the examples.
>>     We've been able to get the test suite to run manually on Mellanox
>>     CX5 devices once.
>>     However, we are running into an issue where, when RCF starts, the
>>     RCF connection times out very frequently. We aren't sure why this
>>     is the case.
>>     It works sometimes, but most of the time when we try to run the
>>     test engine, it encounters this issue.
>>     I've tried changing the RCF port by setting "TE_RCF_PORT=<some
>>     port number>" and rebooting the testbed machines. Neither seems
>>     to fix the issue.
>>
>>     It also seems like the timeout takes far longer than 60 seconds,
>>     even when running "export TE_RCFUNIX_TIMEOUT=60" before I try to
>>     run the test suite.
>>     I assume the unit for this variable is seconds?
>>
>>     Thanks,
>>     Adam
>>
>>     On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>     <ahassick@iol.unh.edu> wrote:
>>
>>         Hi Andrew,
>>
>>         Thanks, I've cloned the example repository and will start
>>         setting up a configuration for our development testbed today.
>>         I'll let you know if I run into any difficulties or have any
>>         questions.
>>
>>          - Adam
>>
>>         On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>             Hi Adam,
>>
>>             I've published
>>             https://github.com/ts-factory/ts-rigs-sample. Hopefully
>>             it will help to define your test rigs and successfully
>>             run some tests manually. Feel free to ask any questions
>>             and I'll answer here and try to update documentation.
>>
>>             Meanwhile I'll prepare missing bits for steps (2) and (3).
>>             Hopefully everything is in place for step (4), but we
>>             need to make steps (2) and (3) first.
>>
>>             Andrew.
>>
>>             On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>             Hi Adam,
>>>
>>>             > I've conferred with the rest of the team, and we think
>>>             it would be best to move forward with mainly option B.
>>>
>>>             OK, I'll provide the sample on Monday for you. It is
>>>             almost ready right now, but I need to double-check it
>>>             before publishing.
>>>
>>>             Regards,
>>>             Andrew.
>>>
>>>             On 8/17/23 20:03, Adam Hassick wrote:
>>>>             Hi Andrew,
>>>>
>>>>             I'm adding the CI mailing list to this conversation.
>>>>             Others in the community might find this conversation
>>>>             valuable.
>>>>
>>>>             We do want to run testing on a regular basis. The
>>>>             Jenkins integration will be very useful for us, as most
>>>>             of our CI is orchestrated by Jenkins.
>>>>             I've conferred with the rest of the team, and we think
>>>>             it would be best to move forward with mainly option B.
>>>>             If you would like to know anything about our testbeds
>>>>             that would help you with creating an example ts-rigs
>>>>             repo, I'd be happy to answer any questions you have.
>>>>
>>>>             We have multiple test rigs (we call these "DUT-tester
>>>>             pairs") that we run our existing hardware testing on,
>>>>             with differing network hardware and CPU architecture. I
>>>>             figured this might be an important detail.
>>>>
>>>>             Thanks,
>>>>             Adam
>>>>
>>>>             On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>                 Greatings Adam,
>>>>
>>>>                 I'm happy to hear that you're trying to bring it up.
>>>>
>>>>                 As I understand the final goal is to run it on
>>>>                 regular basis. So, we need to make it properly from
>>>>                 the very beginning.
>>>>                 Bring up of all features consists of 4 steps:
>>>>
>>>>                 1. Create site-specific repository (we call it
>>>>                 ts-rigs) which contains information about test rigs
>>>>                 and other site-specific information like where to
>>>>                 send mails, where to store logs etc. It is required
>>>>                 for manual execution as well, since test rigs
>>>>                 description is essential. I'll return to the topic
>>>>                 below.
>>>>
>>>>                 2. Setup logs storage for automated runs. Basically
>>>>                 it is a disk space plus apache2 web server with few
>>>>                 CGI scripts which help a lot to save disk space.
>>>>
>>>>                 3. Setup Bublik web application which provides web
>>>>                 interface to view testing results. Same as
>>>>                 https://ts-factory.io/bublik
>>>>
>>>>                 4. Setup Jenkins to run tests on regularly, save
>>>>                 logs in log storage (2) and import it to bublik (3).
>>>>
>>>>                 Last few month we spent on our homework to make it
>>>>                 simpler to bring up automated execution using
>>>>                 Jenkins - https://github.com/ts-factory/te-jenkins
>>>>                 Corresponding bits in dpdk-ethdev-ts will be
>>>>                 available tomorrow.
>>>>
>>>>                 Let's return to the step (1).
>>>>
>>>>                 Unfortunately there is no publicly available
>>>>                 example of the ts-rigs repository since sensitive
>>>>                 site-specific information is located there. But I'm
>>>>                 ready to help you to create it for UNH. I see two
>>>>                 options here:
>>>>
>>>>                 (A) I'll ask questions and based on your answers
>>>>                 will create the first draft with my comments.
>>>>
>>>>                 (B) I'll make a template/example ts-rigs repo,
>>>>                 publish it and you'll create UNH ts-rigs based on it.
>>>>
>>>>                 Of course, I'll help to debug and finally bring it
>>>>                 up in any case.
>>>>
>>>>                 (A) is a bit simpler for me and you, but (B) is a
>>>>                 bit more generic and will help other potential
>>>>                 users to bring it up.
>>>>                 We can combine (A)+(B). I.e. start from (A). What
>>>>                 do you think?
>>>>
>>>>                 Thanks,
>>>>                 Andrew.
>>>>
>>>>                 On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>                 Greetings Adam,
>>>>>
>>>>>
>>>>>                 Thanks for contacting us. I copy Andrew who would
>>>>>                 be happy to help
>>>>>
>>>>>                 Thanks,
>>>>>                 Konstantin
>>>>>
>>>>>>                 On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>                 <ahassick@iol.unh.edu>
>>>>>>                 <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>
>>>>>>                 
>>>>>>                 Greetings Konstantin,
>>>>>>
>>>>>>                 I am in the process of setting up the DPDK Poll
>>>>>>                 Mode Driver test suite as an addition to our
>>>>>>                 testing coverage for DPDK at the UNH lab.
>>>>>>
>>>>>>                 I have some questions about how to set the test
>>>>>>                 suite arguments.
>>>>>>
>>>>>>                 I have been able to configure the Test Engine to
>>>>>>                 connect to the hosts in the testbed. The RCF,
>>>>>>                 Configurator, and Tester all begin to run,
>>>>>>                 however the prelude of the test suite fails to run.
>>>>>>
>>>>>>                 https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>
>>>>>>                 The documentation mentions that there are several
>>>>>>                 test parameters for the test suite, like for the
>>>>>>                 IUT test link MAC, etc. These seem like they
>>>>>>                 would need to be set somewhere to run many of the
>>>>>>                 tests.
>>>>>>
>>>>>>                 I see in the Test Engine documentation, there are
>>>>>>                 instructions on how to create new parameters for
>>>>>>                 test suites in the Tester configuration, but
>>>>>>                 there is nothing in the user guide or in the
>>>>>>                 Tester guide for how to set the arguments for the
>>>>>>                 parameters when running the test suite that I can
>>>>>>                 find. I'm not sure if I need to write my own
>>>>>>                 Tester config, or if I should be setting these in
>>>>>>                 some other way.
>>>>>>
>>>>>>                 How should these values be set?
>>>>>>
>>>>>>                 I'm also not sure what environment
>>>>>>                 variables/arguments are strictly necessary or
>>>>>>                 which are optional.
>>>>>>
>>>>>>                 Regards,
>>>>>>                 Adam
>>>>>>
>>>>>>                 -- 
>>>>>>                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>                 ahassick@iol.unh.edu
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>
>>
>>
>>
>>         -- 
>>         *Adam Hassick*
>>         Senior Developer
>>         UNH InterOperability Lab
>>         ahassick@iol.unh.edu
>>         iol.unh.edu <https://www.iol.unh.edu/>
>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 39041 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-24 18:34                   ` Andrew Rybchenko
@ 2023-08-24 20:29                     ` Adam Hassick
  2023-08-24 20:54                       ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-24 20:29 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 13548 bytes --]

> Is there any firewall in the network or on test hosts which could block
incoming TCP connection to the port 23571
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where you
run test engine?

Our test engine host and the testbed are on the same subnet. The connection
does work sometimes.

> If behaviour the same on the next try and you see that test agent is kept
running, could you check using
>
> # netstat -tnlp
>
> that Test Agent is listening on the port and try to establish TCP
connection from test agent using
>
> $ telnet iol-dts-tester.dpdklab.iol.unh.edu
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
<http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
> and check if TCP connection could be established.

I was able to replicate the same behavior again, where it hangs while RCF
is trying to start.
Running this command, I see this in the output:

tcp        0      0 0.0.0.0:23571           0.0.0.0:*               LISTEN
     18599/ta

So it seems like it is listening on the correct port.
Additionally, I was able to connect to the Tester machine from our Test
Engine host using telnet. It printed the PID of the process once the
connection was opened.

I tried running the "ta" application manually on the command line, and it
didn't print anything at all.
Maybe the issue is something on the Test Engine side.

On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> > On the tester host (which appears to be the Peer agent), there are four
> processes that I see running, which look like the test agent processes.
>
> Before the next try I'd recommend to kill these processes.
>
> Is there any firewall in the network or on test hosts which could block
> incoming TCP connection to the port 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where you
> run test engine?
>
> If behaviour the same on the next try and you see that test agent is kept
> running, could you check using
>
> # netstat -tnlp
>
> that Test Agent is listening on the port and try to establish TCP
> connection from test agent using
>
> $ telnet iol-dts-tester.dpdklab.iol.unh.edu
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
> and check if TCP connection could be established.
>
> Another idea is to login Tester under root as testing does, get start TA
> command from the log and try it by hands without -n and remove extra
> escaping.
>
> # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
> host=iol-dts-tester.dpdklab.iol.unh.edu:
> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>
> Hopefully in this case test agent directory remains in the /tmp and you
> don't need to copy it as testing does.
> May be output could shed some light on what's going on.
>
> Andrew.
>
> On 8/24/23 17:30, Adam Hassick wrote:
>
> Hi Andrew,
>
> This is the output that I see in the terminal when this failure occurs,
> after the test agent binaries build and the test engine starts:
>
> Platform default build - pass
> Simple RCF consistency check succeeded
> --->>> Starting Logger...done
> --->>> Starting RCF...rcf_net_engine_connect(): Connection timed out
> iol-dts-tester.dpdklab.iol.unh.edu:23571
>
> Then, it hangs here until I kill the "te_rcf" and "te_tee" processes. I
> let it hang for around 9 minutes.
>
> On the tester host (which appears to be the Peer agent), there are four
> processes that I see running, which look like the test agent processes.
>
> ta.Peer is an empty file. I've attached the log.txt from this run.
>
>  - Adam
>
> On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked that it goes
>> to 'copy_timeout' in ts-conf/rcf.conf.
>> Description in in doc/sphinx/pages/group_te_engine_rcf.rst says that
>> copy_timeout is in seconds and implementation in lib/rcfunix/rcfunix.c
>> passes the value to select() tv_sec. Theoretically select() could be
>> interrupted by signal, but I think it is unlikely here.
>>
>> I'm not sure that I understand what do you mean by RCF connection
>> timeout. Does it happen on TE startup when RCF starts test agents. If so,
>> TE_RCFUNIX_TIMEOUT could help. Or does it happen when tests are in
>> progress, e.g. in the middle of a test. If so, TE_RCFUNIX_TIMEOUT is
>> unrelated and most likely either host with test agent dies or test agent
>> itself crashes. It would be easier for me if classify it if you share text
>> log (log.txt, full or just corresponding fragment with some context). Also
>> content of ta.DPDK or ta.Peer file depending on which agent has problems
>> could shed some light. Corresponding files contain stdout/stderr of test
>> agents.
>>
>> Andrew.
>>
>> On 8/23/23 17:45, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> I've set up a test rig repository here, and have created configurations
>> for our development testbed based off of the examples.
>> We've been able to get the test suite to run manually on Mellanox CX5
>> devices once.
>> However, we are running into an issue where, when RCF starts, the RCF
>> connection times out very frequently. We aren't sure why this is the case.
>> It works sometimes, but most of the time when we try to run the test
>> engine, it encounters this issue.
>> I've tried changing the RCF port by setting "TE_RCF_PORT=<some port
>> number>" and rebooting the testbed machines. Neither seems to fix the issue.
>>
>> It also seems like the timeout takes far longer than 60 seconds, even
>> when running "export TE_RCFUNIX_TIMEOUT=60" before I try to run the test
>> suite.
>> I assume the unit for this variable is seconds?
>>
>> Thanks,
>> Adam
>>
>> On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick <ahassick@iol.unh.edu>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> Thanks, I've cloned the example repository and will start setting up a
>>> configuration for our development testbed today. I'll let you know if I run
>>> into any difficulties or have any questions.
>>>
>>>  - Adam
>>>
>>> On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hi Adam,
>>>>
>>>> I've published https://github.com/ts-factory/ts-rigs-sample. Hopefully
>>>> it will help to define your test rigs and successfully run some tests
>>>> manually. Feel free to ask any questions and I'll answer here and try to
>>>> update documentation.
>>>>
>>>> Meanwhile I'll prepare missing bits for steps (2) and (3).
>>>> Hopefully everything is in place for step (4), but we need to make
>>>> steps (2) and (3) first.
>>>>
>>>> Andrew.
>>>>
>>>> On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>
>>>> Hi Adam,
>>>>
>>>> > I've conferred with the rest of the team, and we think it would be
>>>> best to move forward with mainly option B.
>>>>
>>>> OK, I'll provide the sample on Monday for you. It is almost ready right
>>>> now, but I need to double-check it before publishing.
>>>>
>>>> Regards,
>>>> Andrew.
>>>>
>>>> On 8/17/23 20:03, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> I'm adding the CI mailing list to this conversation. Others in the
>>>> community might find this conversation valuable.
>>>>
>>>> We do want to run testing on a regular basis. The Jenkins integration
>>>> will be very useful for us, as most of our CI is orchestrated by Jenkins.
>>>> I've conferred with the rest of the team, and we think it would be best
>>>> to move forward with mainly option B.
>>>> If you would like to know anything about our testbeds that would help
>>>> you with creating an example ts-rigs repo, I'd be happy to answer any
>>>> questions you have.
>>>>
>>>> We have multiple test rigs (we call these "DUT-tester pairs") that we
>>>> run our existing hardware testing on, with differing network hardware and
>>>> CPU architecture. I figured this might be an important detail.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> Greatings Adam,
>>>>>
>>>>> I'm happy to hear that you're trying to bring it up.
>>>>>
>>>>> As I understand the final goal is to run it on regular basis. So, we
>>>>> need to make it properly from the very beginning.
>>>>> Bring up of all features consists of 4 steps:
>>>>>
>>>>> 1. Create site-specific repository (we call it ts-rigs) which contains
>>>>> information about test rigs and other site-specific information like where
>>>>> to send mails, where to store logs etc. It is required for manual execution
>>>>> as well, since test rigs description is essential. I'll return to the topic
>>>>> below.
>>>>>
>>>>> 2. Setup logs storage for automated runs. Basically it is a disk space
>>>>> plus apache2 web server with few CGI scripts which help a lot to save disk
>>>>> space.
>>>>>
>>>>> 3. Setup Bublik web application which provides web interface to view
>>>>> testing results. Same as https://ts-factory.io/bublik
>>>>>
>>>>> 4. Setup Jenkins to run tests on regularly, save logs in log storage
>>>>> (2) and import it to bublik (3).
>>>>>
>>>>> Last few month we spent on our homework to make it simpler to bring up
>>>>> automated execution using Jenkins -
>>>>> https://github.com/ts-factory/te-jenkins
>>>>> Corresponding bits in dpdk-ethdev-ts will be available tomorrow.
>>>>>
>>>>> Let's return to the step (1).
>>>>>
>>>>> Unfortunately there is no publicly available example of the ts-rigs
>>>>> repository since sensitive site-specific information is located there. But
>>>>> I'm ready to help you to create it for UNH. I see two options here:
>>>>>
>>>>> (A) I'll ask questions and based on your answers will create the first
>>>>> draft with my comments.
>>>>>
>>>>> (B) I'll make a template/example ts-rigs repo, publish it and you'll
>>>>> create UNH ts-rigs based on it.
>>>>>
>>>>> Of course, I'll help to debug and finally bring it up in any case.
>>>>>
>>>>> (A) is a bit simpler for me and you, but (B) is a bit more generic and
>>>>> will help other potential users to bring it up.
>>>>> We can combine (A)+(B). I.e. start from (A). What do you think?
>>>>>
>>>>> Thanks,
>>>>> Andrew.
>>>>>
>>>>> On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>
>>>>> Greetings Adam,
>>>>>
>>>>>
>>>>> Thanks for contacting us. I copy Andrew who would be happy to help
>>>>>
>>>>> Thanks,
>>>>> Konstantin
>>>>>
>>>>> On 16 Aug 2023, at 21:50, Adam Hassick <ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>
>>>>> 
>>>>> Greetings Konstantin,
>>>>>
>>>>> I am in the process of setting up the DPDK Poll Mode Driver test suite
>>>>> as an addition to our testing coverage for DPDK at the UNH lab.
>>>>>
>>>>> I have some questions about how to set the test suite arguments.
>>>>>
>>>>> I have been able to configure the Test Engine to connect to the hosts
>>>>> in the testbed. The RCF, Configurator, and Tester all begin to run, however
>>>>> the prelude of the test suite fails to run.
>>>>>
>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>
>>>>> The documentation mentions that there are several test parameters for
>>>>> the test suite, like for the IUT test link MAC, etc. These seem like they
>>>>> would need to be set somewhere to run many of the tests.
>>>>>
>>>>> I see in the Test Engine documentation, there are instructions on how
>>>>> to create new parameters for test suites in the Tester configuration, but
>>>>> there is nothing in the user guide or in the Tester guide for how to set
>>>>> the arguments for the parameters when running the test suite that I can
>>>>> find. I'm not sure if I need to write my own Tester config, or if I should
>>>>> be setting these in some other way.
>>>>>
>>>>> How should these values be set?
>>>>>
>>>>> I'm also not sure what environment variables/arguments are strictly
>>>>> necessary or which are optional.
>>>>>
>>>>> Regards,
>>>>> Adam
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu
>>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 39065 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-24 20:29                     ` Adam Hassick
@ 2023-08-24 20:54                       ` Andrew Rybchenko
  2023-08-25 13:57                         ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-24 20:54 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

I'd like to try to repeat the problem locally. Which Linux distro is 
running on test engine and agents?

In fact I know one problem with Debian 12 and Fedora 38 and we have
patch in review to fix it, however, the behaviour is different in
this case, so it is unlike the same problem.

One more idea is to install valgrind on the test engine host and
run with option --vg-rcf to check if something weird is happening.

What I don't understand right now is why I see just one failed attempt
to connect in your log.txt and then Logger shutdown after 9 minutes.

Andrew.

On 8/24/23 23:29, Adam Hassick wrote:
>  > Is there any firewall in the network or on test hosts which could 
> block incoming TCP connection to the port 23571 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where 
> you run test engine?
> 
> Our test engine host and the testbed are on the same subnet. The 
> connection does work sometimes.
> 
>  > If behaviour the same on the next try and you see that test agent is 
> kept running, could you check using
>  >
>  > # netstat -tnlp
>  >
>  > that Test Agent is listening on the port and try to establish TCP 
> connection from test agent using
>  >
>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571 
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>  >
>  > and check if TCP connection could be established.
> 
> I was able to replicate the same behavior again, where it hangs while 
> RCF is trying to start.
> Running this command, I see this in the output:
> 
> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>           
> 0.0.0.0:*               LISTEN      18599/ta
> 
> So it seems like it is listening on the correct port.
> Additionally, I was able to connect to the Tester machine from our Test 
> Engine host using telnet. It printed the PID of the process once the 
> connection was opened.
> 
> I tried running the "ta" application manually on the command line, and 
> it didn't print anything at all.
> Maybe the issue is something on the Test Engine side.
> 
> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>> 
> wrote:
> 
>     Hi Adam,
> 
>      > On the tester host (which appears to be the Peer agent), there
>     are four processes that I see running, which look like the test
>     agent processes.
> 
>     Before the next try I'd recommend to kill these processes.
> 
>     Is there any firewall in the network or on test hosts which could
>     block incoming TCP connection to the port 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>     where you run test engine?
> 
>     If behaviour the same on the next try and you see that test agent is
>     kept running, could you check using
> 
>     # netstat -tnlp
> 
>     that Test Agent is listening on the port and try to establish TCP
>     connection from test agent using
> 
>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> 
>     and check if TCP connection could be established.
> 
>     Another idea is to login Tester under root as testing does, get
>     start TA command from the log and try it by hands without -n and
>     remove extra escaping.
> 
>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
> 
>     Hopefully in this case test agent directory remains in the /tmp and
>     you don't need to copy it as testing does.
>     May be output could shed some light on what's going on.
> 
>     Andrew.
> 
>     On 8/24/23 17:30, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     This is the output that I see in the terminal when this failure
>>     occurs, after the test agent binaries build and the test engine
>>     starts:
>>
>>     Platform default build - pass
>>     Simple RCF consistency check succeeded
>>     --->>> Starting Logger...done
>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>     processes. I let it hang for around 9 minutes.
>>
>>     On the tester host (which appears to be the Peer agent), there are
>>     four processes that I see running, which look like the test agent
>>     processes.
>>
>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>
>>      - Adam
>>
>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru
>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>         Hi Adam,
>>
>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>         says that copy_timeout is in seconds and implementation in
>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>         Theoretically select() could be interrupted by signal, but I
>>         think it is unlikely here.
>>
>>         I'm not sure that I understand what do you mean by RCF
>>         connection timeout. Does it happen on TE startup when RCF
>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>         does it happen when tests are in progress, e.g. in the middle
>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>         likely either host with test agent dies or test agent itself
>>         crashes. It would be easier for me if classify it if you share
>>         text log (log.txt, full or just corresponding fragment with
>>         some context). Also content of ta.DPDK or ta.Peer file
>>         depending on which agent has problems could shed some light.
>>         Corresponding files contain stdout/stderr of test agents.
>>
>>         Andrew.
>>
>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>         Hi Andrew,
>>>
>>>         I've set up a test rig repository here, and have created
>>>         configurations for our development testbed based off of the
>>>         examples.
>>>         We've been able to get the test suite to run manually on
>>>         Mellanox CX5 devices once.
>>>         However, we are running into an issue where, when RCF starts,
>>>         the RCF connection times out very frequently. We aren't sure
>>>         why this is the case.
>>>         It works sometimes, but most of the time when we try to run
>>>         the test engine, it encounters this issue.
>>>         I've tried changing the RCF port by setting
>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>         machines. Neither seems to fix the issue.
>>>
>>>         It also seems like the timeout takes far longer than 60
>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>         before I try to run the test suite.
>>>         I assume the unit for this variable is seconds?
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>> wrote:
>>>
>>>             Hi Andrew,
>>>
>>>             Thanks, I've cloned the example repository and will start
>>>             setting up a configuration for our development testbed
>>>             today. I'll let you know if I run into any difficulties
>>>             or have any questions.
>>>
>>>              - Adam
>>>
>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>             <andrew.rybchenko@oktetlabs.ru
>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>                 Hi Adam,
>>>
>>>                 I've published
>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>                 <https://github.com/ts-factory/ts-rigs-sample>.
>>>                 Hopefully it will help to define your test rigs and
>>>                 successfully run some tests manually. Feel free to
>>>                 ask any questions and I'll answer here and try to
>>>                 update documentation.
>>>
>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>                 (3).
>>>                 Hopefully everything is in place for step (4), but we
>>>                 need to make steps (2) and (3) first.
>>>
>>>                 Andrew.
>>>
>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>                 Hi Adam,
>>>>
>>>>                 > I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>
>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>                 almost ready right now, but I need to double-check
>>>>                 it before publishing.
>>>>
>>>>                 Regards,
>>>>                 Andrew.
>>>>
>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 I'm adding the CI mailing list to this
>>>>>                 conversation. Others in the community might find
>>>>>                 this conversation valuable.
>>>>>
>>>>>                 We do want to run testing on a regular basis. The
>>>>>                 Jenkins integration will be very useful for us, as
>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>                 I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>                 If you would like to know anything about our
>>>>>                 testbeds that would help you with creating an
>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>                 questions you have.
>>>>>
>>>>>                 We have multiple test rigs (we call these
>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>                 hardware testing on, with differing network
>>>>>                 hardware and CPU architecture. I figured this might
>>>>>                 be an important detail.
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                     Greatings Adam,
>>>>>
>>>>>                     I'm happy to hear that you're trying to bring
>>>>>                     it up.
>>>>>
>>>>>                     As I understand the final goal is to run it on
>>>>>                     regular basis. So, we need to make it properly
>>>>>                     from the very beginning.
>>>>>                     Bring up of all features consists of 4 steps:
>>>>>
>>>>>                     1. Create site-specific repository (we call it
>>>>>                     ts-rigs) which contains information about test
>>>>>                     rigs and other site-specific information like
>>>>>                     where to send mails, where to store logs etc.
>>>>>                     It is required for manual execution as well,
>>>>>                     since test rigs description is essential. I'll
>>>>>                     return to the topic below.
>>>>>
>>>>>                     2. Setup logs storage for automated runs.
>>>>>                     Basically it is a disk space plus apache2 web
>>>>>                     server with few CGI scripts which help a lot to
>>>>>                     save disk space.
>>>>>
>>>>>                     3. Setup Bublik web application which provides
>>>>>                     web interface to view testing results. Same as
>>>>>                     https://ts-factory.io/bublik
>>>>>                     <https://ts-factory.io/bublik>
>>>>>
>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>                     save logs in log storage (2) and import it to
>>>>>                     bublik (3).
>>>>>
>>>>>                     Last few month we spent on our homework to make
>>>>>                     it simpler to bring up automated execution
>>>>>                     using Jenkins -
>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>                     available tomorrow.
>>>>>
>>>>>                     Let's return to the step (1).
>>>>>
>>>>>                     Unfortunately there is no publicly available
>>>>>                     example of the ts-rigs repository since
>>>>>                     sensitive site-specific information is located
>>>>>                     there. But I'm ready to help you to create it
>>>>>                     for UNH. I see two options here:
>>>>>
>>>>>                     (A) I'll ask questions and based on your
>>>>>                     answers will create the first draft with my
>>>>>                     comments.
>>>>>
>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>                     on it.
>>>>>
>>>>>                     Of course, I'll help to debug and finally bring
>>>>>                     it up in any case.
>>>>>
>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>                     a bit more generic and will help other
>>>>>                     potential users to bring it up.
>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>                     What do you think?
>>>>>
>>>>>                     Thanks,
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>                     Greetings Adam,
>>>>>>
>>>>>>
>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>                     would be happy to help
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Konstantin
>>>>>>
>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>                     <ahassick@iol.unh.edu>
>>>>>>>                     <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>
>>>>>>>                     
>>>>>>>                     Greetings Konstantin,
>>>>>>>
>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>
>>>>>>>                     I have some questions about how to set the
>>>>>>>                     test suite arguments.
>>>>>>>
>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>                     run, however the prelude of the test suite
>>>>>>>                     fails to run.
>>>>>>>
>>>>>>>                     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>
>>>>>>>                     The documentation mentions that there are
>>>>>>>                     several test parameters for the test suite,
>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>                     to run many of the tests.
>>>>>>>
>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>                     are instructions on how to create new
>>>>>>>                     parameters for test suites in the Tester
>>>>>>>                     configuration, but there is nothing in the
>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>                     set the arguments for the parameters when
>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>                     config, or if I should be setting these in
>>>>>>>                     some other way.
>>>>>>>
>>>>>>>                     How should these values be set?
>>>>>>>
>>>>>>>                     I'm also not sure what environment
>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>                     which are optional.
>>>>>>>
>>>>>>>                     Regards,
>>>>>>>                     Adam
>>>>>>>
>>>>>>>                     -- 
>>>>>>>                     *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>                     ahassick@iol.unh.edu
>>>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>                 -- 
>>>>>                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>             -- 
>>>             *Adam Hassick*
>>>             Senior Developer
>>>             UNH InterOperability Lab
>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         -- 
>>>         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
> 
> 
> 
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-24 20:54                       ` Andrew Rybchenko
@ 2023-08-25 13:57                         ` Andrew Rybchenko
  2023-08-25 14:06                           ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-25 13:57 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 22293 bytes --]

Hello Adam,

On 8/24/23 23:54, Andrew Rybchenko wrote:
> I'd like to try to repeat the problem locally. Which Linux distro is 
> running on test engine and agents?
>
> In fact I know one problem with Debian 12 and Fedora 38 and we have
> patch in review to fix it, however, the behaviour is different in
> this case, so it is unlike the same problem.

I've just published a new tag which fixes known test engine side 
problems on Debian 12 and Fedora 38.

>
> One more idea is to install valgrind on the test engine host and
> run with option --vg-rcf to check if something weird is happening.
>
> What I don't understand right now is why I see just one failed attempt
> to connect in your log.txt and then Logger shutdown after 9 minutes.
>
> Andrew.
>
> On 8/24/23 23:29, Adam Hassick wrote:
>>  > Is there any firewall in the network or on test hosts which could 
>> block incoming TCP connection to the port 23571 
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where 
>> you run test engine?
>>
>> Our test engine host and the testbed are on the same subnet. The 
>> connection does work sometimes.
>>
>>  > If behaviour the same on the next try and you see that test agent 
>> is kept running, could you check using
>>  >
>>  > # netstat -tnlp
>>  >
>>  > that Test Agent is listening on the port and try to establish TCP 
>> connection from test agent using
>>  >
>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu 
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571 
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>  >
>>  > and check if TCP connection could be established.
>>
>> I was able to replicate the same behavior again, where it hangs while 
>> RCF is trying to start.
>> Running this command, I see this in the output:
>>
>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>           
>> 0.0.0.0:* LISTEN      18599/ta
>>
>> So it seems like it is listening on the correct port.
>> Additionally, I was able to connect to the Tester machine from our 
>> Test Engine host using telnet. It printed the PID of the process once 
>> the connection was opened.
>>
>> I tried running the "ta" application manually on the command line, 
>> and it didn't print anything at all.
>> Maybe the issue is something on the Test Engine side.
>>
>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru 
>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>     Hi Adam,
>>
>>      > On the tester host (which appears to be the Peer agent), there
>>     are four processes that I see running, which look like the test
>>     agent processes.
>>
>>     Before the next try I'd recommend to kill these processes.
>>
>>     Is there any firewall in the network or on test hosts which could
>>     block incoming TCP connection to the port 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>     where you run test engine?
>>
>>     If behaviour the same on the next try and you see that test agent is
>>     kept running, could you check using
>>
>>     # netstat -tnlp
>>
>>     that Test Agent is listening on the port and try to establish TCP
>>     connection from test agent using
>>
>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     and check if TCP connection could be established.
>>
>>     Another idea is to login Tester under root as testing does, get
>>     start TA command from the log and try it by hands without -n and
>>     remove extra escaping.
>>
>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1 
>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 
>> host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>
>>     Hopefully in this case test agent directory remains in the /tmp and
>>     you don't need to copy it as testing does.
>>     May be output could shed some light on what's going on.
>>
>>     Andrew.
>>
>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     This is the output that I see in the terminal when this failure
>>>     occurs, after the test agent binaries build and the test engine
>>>     starts:
>>>
>>>     Platform default build - pass
>>>     Simple RCF consistency check succeeded
>>>     --->>> Starting Logger...done
>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>     processes. I let it hang for around 9 minutes.
>>>
>>>     On the tester host (which appears to be the Peer agent), there are
>>>     four processes that I see running, which look like the test agent
>>>     processes.
>>>
>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>
>>>      - Adam
>>>
>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru
>>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>         Hi Adam,
>>>
>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>         says that copy_timeout is in seconds and implementation in
>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>         Theoretically select() could be interrupted by signal, but I
>>>         think it is unlikely here.
>>>
>>>         I'm not sure that I understand what do you mean by RCF
>>>         connection timeout. Does it happen on TE startup when RCF
>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>         does it happen when tests are in progress, e.g. in the middle
>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>         likely either host with test agent dies or test agent itself
>>>         crashes. It would be easier for me if classify it if you share
>>>         text log (log.txt, full or just corresponding fragment with
>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>         depending on which agent has problems could shed some light.
>>>         Corresponding files contain stdout/stderr of test agents.
>>>
>>>         Andrew.
>>>
>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         I've set up a test rig repository here, and have created
>>>>         configurations for our development testbed based off of the
>>>>         examples.
>>>>         We've been able to get the test suite to run manually on
>>>>         Mellanox CX5 devices once.
>>>>         However, we are running into an issue where, when RCF starts,
>>>>         the RCF connection times out very frequently. We aren't sure
>>>>         why this is the case.
>>>>         It works sometimes, but most of the time when we try to run
>>>>         the test engine, it encounters this issue.
>>>>         I've tried changing the RCF port by setting
>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>         machines. Neither seems to fix the issue.
>>>>
>>>>         It also seems like the timeout takes far longer than 60
>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>         before I try to run the test suite.
>>>>         I assume the unit for this variable is seconds?
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>> wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             Thanks, I've cloned the example repository and will start
>>>>             setting up a configuration for our development testbed
>>>>             today. I'll let you know if I run into any difficulties
>>>>             or have any questions.
>>>>
>>>>              - Adam
>>>>
>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru
>>>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 I've published
>>>> https://github.com/ts-factory/ts-rigs-sample
>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>                 Hopefully it will help to define your test rigs and
>>>>                 successfully run some tests manually. Feel free to
>>>>                 ask any questions and I'll answer here and try to
>>>>                 update documentation.
>>>>
>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>                 (3).
>>>>                 Hopefully everything is in place for step (4), but we
>>>>                 need to make steps (2) and (3) first.
>>>>
>>>>                 Andrew.
>>>>
>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>                 Hi Adam,
>>>>>
>>>>>                 > I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>
>>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>>                 almost ready right now, but I need to double-check
>>>>>                 it before publishing.
>>>>>
>>>>>                 Regards,
>>>>>                 Andrew.
>>>>>
>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 I'm adding the CI mailing list to this
>>>>>>                 conversation. Others in the community might find
>>>>>>                 this conversation valuable.
>>>>>>
>>>>>>                 We do want to run testing on a regular basis. The
>>>>>>                 Jenkins integration will be very useful for us, as
>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>                 I've conferred with the rest of the team, and we
>>>>>>                 think it would be best to move forward with mainly
>>>>>>                 option B.
>>>>>>                 If you would like to know anything about our
>>>>>>                 testbeds that would help you with creating an
>>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>>                 questions you have.
>>>>>>
>>>>>>                 We have multiple test rigs (we call these
>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>                 hardware testing on, with differing network
>>>>>>                 hardware and CPU architecture. I figured this might
>>>>>>                 be an important detail.
>>>>>>
>>>>>>                 Thanks,
>>>>>>                 Adam
>>>>>>
>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>> <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                     Greatings Adam,
>>>>>>
>>>>>>                     I'm happy to hear that you're trying to bring
>>>>>>                     it up.
>>>>>>
>>>>>>                     As I understand the final goal is to run it on
>>>>>>                     regular basis. So, we need to make it properly
>>>>>>                     from the very beginning.
>>>>>>                     Bring up of all features consists of 4 steps:
>>>>>>
>>>>>>                     1. Create site-specific repository (we call it
>>>>>>                     ts-rigs) which contains information about test
>>>>>>                     rigs and other site-specific information like
>>>>>>                     where to send mails, where to store logs etc.
>>>>>>                     It is required for manual execution as well,
>>>>>>                     since test rigs description is essential. I'll
>>>>>>                     return to the topic below.
>>>>>>
>>>>>>                     2. Setup logs storage for automated runs.
>>>>>>                     Basically it is a disk space plus apache2 web
>>>>>>                     server with few CGI scripts which help a lot to
>>>>>>                     save disk space.
>>>>>>
>>>>>>                     3. Setup Bublik web application which provides
>>>>>>                     web interface to view testing results. Same as
>>>>>> https://ts-factory.io/bublik
>>>>>> <https://ts-factory.io/bublik>
>>>>>>
>>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>>                     save logs in log storage (2) and import it to
>>>>>>                     bublik (3).
>>>>>>
>>>>>>                     Last few month we spent on our homework to make
>>>>>>                     it simpler to bring up automated execution
>>>>>>                     using Jenkins -
>>>>>> https://github.com/ts-factory/te-jenkins
>>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>>                     available tomorrow.
>>>>>>
>>>>>>                     Let's return to the step (1).
>>>>>>
>>>>>>                     Unfortunately there is no publicly available
>>>>>>                     example of the ts-rigs repository since
>>>>>>                     sensitive site-specific information is located
>>>>>>                     there. But I'm ready to help you to create it
>>>>>>                     for UNH. I see two options here:
>>>>>>
>>>>>>                     (A) I'll ask questions and based on your
>>>>>>                     answers will create the first draft with my
>>>>>>                     comments.
>>>>>>
>>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>>                     on it.
>>>>>>
>>>>>>                     Of course, I'll help to debug and finally bring
>>>>>>                     it up in any case.
>>>>>>
>>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>>                     a bit more generic and will help other
>>>>>>                     potential users to bring it up.
>>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>>                     What do you think?
>>>>>>
>>>>>>                     Thanks,
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>                     Greetings Adam,
>>>>>>>
>>>>>>>
>>>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>>>                     would be happy to help
>>>>>>>
>>>>>>>                     Thanks,
>>>>>>>                     Konstantin
>>>>>>>
>>>>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>> <ahassick@iol.unh.edu>
>>>>>>>> <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>
>>>>>>>>                     
>>>>>>>>                     Greetings Konstantin,
>>>>>>>>
>>>>>>>>                     I am in the process of setting up the DPDK
>>>>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>>>>
>>>>>>>>                     I have some questions about how to set the
>>>>>>>>                     test suite arguments.
>>>>>>>>
>>>>>>>>                     I have been able to configure the Test Engine
>>>>>>>>                     to connect to the hosts in the testbed. The
>>>>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>>>>                     run, however the prelude of the test suite
>>>>>>>>                     fails to run.
>>>>>>>>
>>>>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters 
>>>>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters> 
>>>>>>>>
>>>>>>>>
>>>>>>>>                     The documentation mentions that there are
>>>>>>>>                     several test parameters for the test suite,
>>>>>>>>                     like for the IUT test link MAC, etc. These
>>>>>>>>                     seem like they would need to be set somewhere
>>>>>>>>                     to run many of the tests.
>>>>>>>>
>>>>>>>>                     I see in the Test Engine documentation, there
>>>>>>>>                     are instructions on how to create new
>>>>>>>>                     parameters for test suites in the Tester
>>>>>>>>                     configuration, but there is nothing in the
>>>>>>>>                     user guide or in the Tester guide for how to
>>>>>>>>                     set the arguments for the parameters when
>>>>>>>>                     running the test suite that I can find. I'm
>>>>>>>>                     not sure if I need to write my own Tester
>>>>>>>>                     config, or if I should be setting these in
>>>>>>>>                     some other way.
>>>>>>>>
>>>>>>>>                     How should these values be set?
>>>>>>>>
>>>>>>>>                     I'm also not sure what environment
>>>>>>>>                     variables/arguments are strictly necessary or
>>>>>>>>                     which are optional.
>>>>>>>>
>>>>>>>>                     Regards,
>>>>>>>>                     Adam
>>>>>>>>
>>>>>>>>                     --                     *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>> ahassick@iol.unh.edu
>>>>>>>> <mailto:ahassick@iol.unh.edu>
>>>>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --                 *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>             --             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>     --     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 39090 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-25 13:57                         ` Andrew Rybchenko
@ 2023-08-25 14:06                           ` Adam Hassick
  2023-08-25 14:41                             ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-25 14:06 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 20482 bytes --]

Hi Andrew,

Two of our systems (the Test Engine runner and the DUT host) are running
Ubuntu 20.04 LTS, however this morning I noticed that the tester system
(the one having issues) is running Ubuntu 22.04 LTS.
This could be the source of the problem. I encountered a dependency issue
trying to run the Test Engine on 22.04 LTS, so I downgraded the system.
Since the tester is also the host having connection issues, I will try
downgrading that system to 20.04, and see if that changes anything.

I did try passing in the "--vg-rcf" argument to the run.sh script of the
test suite after installing valgrind, but there was no additional output
that I saw.

I will try pulling in the changes you've pushed up, and will see if that
fixes anything.

Thanks,
Adam

On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hello Adam,
>
> On 8/24/23 23:54, Andrew Rybchenko wrote:
>
> I'd like to try to repeat the problem locally. Which Linux distro is
> running on test engine and agents?
>
> In fact I know one problem with Debian 12 and Fedora 38 and we have
> patch in review to fix it, however, the behaviour is different in
> this case, so it is unlike the same problem.
>
>
> I've just published a new tag which fixes known test engine side problems
> on Debian 12 and Fedora 38.
>
>
> One more idea is to install valgrind on the test engine host and
> run with option --vg-rcf to check if something weird is happening.
>
> What I don't understand right now is why I see just one failed attempt
> to connect in your log.txt and then Logger shutdown after 9 minutes.
>
> Andrew.
>
> On 8/24/23 23:29, Adam Hassick wrote:
>
>  > Is there any firewall in the network or on test hosts which could block
> incoming TCP connection to the port 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where you
> run test engine?
>
> Our test engine host and the testbed are on the same subnet. The
> connection does work sometimes.
>
>  > If behaviour the same on the next try and you see that test agent is
> kept running, could you check using
>  >
>  > # netstat -tnlp
>  >
>  > that Test Agent is listening on the port and try to establish TCP
> connection from test agent using
>  >
>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>  >
>  > and check if TCP connection could be established.
>
> I was able to replicate the same behavior again, where it hangs while RCF
> is trying to start.
> Running this command, I see this in the output:
>
> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>  18599/ta
>
> So it seems like it is listening on the correct port.
> Additionally, I was able to connect to the Tester machine from our Test
> Engine host using telnet. It printed the PID of the process once the
> connection was opened.
>
> I tried running the "ta" application manually on the command line, and it
> didn't print anything at all.
> Maybe the issue is something on the Test Engine side.
>
> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>     Hi Adam,
>
>      > On the tester host (which appears to be the Peer agent), there
>     are four processes that I see running, which look like the test
>     agent processes.
>
>     Before the next try I'd recommend to kill these processes.
>
>     Is there any firewall in the network or on test hosts which could
>     block incoming TCP connection to the port 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>     where you run test engine?
>
>     If behaviour the same on the next try and you see that test agent is
>     kept running, could you check using
>
>     # netstat -tnlp
>
>     that Test Agent is listening on the port and try to establish TCP
>     connection from test agent using
>
>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
>     and check if TCP connection could be established.
>
>     Another idea is to login Tester under root as testing does, get
>     start TA command from the log and try it by hands without -n and
>     remove extra escaping.
>
>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>
> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
> host=iol-dts-tester.dpdklab.iol.unh.edu:
> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>
>     Hopefully in this case test agent directory remains in the /tmp and
>     you don't need to copy it as testing does.
>     May be output could shed some light on what's going on.
>
>     Andrew.
>
>     On 8/24/23 17:30, Adam Hassick wrote:
>
>     Hi Andrew,
>
>     This is the output that I see in the terminal when this failure
>     occurs, after the test agent binaries build and the test engine
>     starts:
>
>     Platform default build - pass
>     Simple RCF consistency check succeeded
>     --->>> Starting Logger...done
>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>     processes. I let it hang for around 9 minutes.
>
>     On the tester host (which appears to be the Peer agent), there are
>     four processes that I see running, which look like the test agent
>     processes.
>
>     ta.Peer is an empty file. I've attached the log.txt from this run.
>
>      - Adam
>
>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>     <andrew.rybchenko@oktetlabs.ru
>     <mailto:andrew.rybchenko@oktetlabs.ru> <andrew.rybchenko@oktetlabs.ru>>
> wrote:
>
>         Hi Adam,
>
>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>         says that copy_timeout is in seconds and implementation in
>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>         Theoretically select() could be interrupted by signal, but I
>         think it is unlikely here.
>
>         I'm not sure that I understand what do you mean by RCF
>         connection timeout. Does it happen on TE startup when RCF
>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>         does it happen when tests are in progress, e.g. in the middle
>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>         likely either host with test agent dies or test agent itself
>         crashes. It would be easier for me if classify it if you share
>         text log (log.txt, full or just corresponding fragment with
>         some context). Also content of ta.DPDK or ta.Peer file
>         depending on which agent has problems could shed some light.
>         Corresponding files contain stdout/stderr of test agents.
>
>         Andrew.
>
>         On 8/23/23 17:45, Adam Hassick wrote:
>
>         Hi Andrew,
>
>         I've set up a test rig repository here, and have created
>         configurations for our development testbed based off of the
>         examples.
>         We've been able to get the test suite to run manually on
>         Mellanox CX5 devices once.
>         However, we are running into an issue where, when RCF starts,
>         the RCF connection times out very frequently. We aren't sure
>         why this is the case.
>         It works sometimes, but most of the time when we try to run
>         the test engine, it encounters this issue.
>         I've tried changing the RCF port by setting
>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>         machines. Neither seems to fix the issue.
>
>         It also seems like the timeout takes far longer than 60
>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>         before I try to run the test suite.
>         I assume the unit for this variable is seconds?
>
>         Thanks,
>         Adam
>
>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>> wrote:
>
>             Hi Andrew,
>
>             Thanks, I've cloned the example repository and will start
>             setting up a configuration for our development testbed
>             today. I'll let you know if I run into any difficulties
>             or have any questions.
>
>              - Adam
>
>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>             <andrew.rybchenko@oktetlabs.ru
>             <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>                 Hi Adam,
>
>                 I've published
>                 https://github.com/ts-factory/ts-rigs-sample
>                 <https://github.com/ts-factory/ts-rigs-sample>
> <https://github.com/ts-factory/ts-rigs-sample>.
>                 Hopefully it will help to define your test rigs and
>                 successfully run some tests manually. Feel free to
>                 ask any questions and I'll answer here and try to
>                 update documentation.
>
>                 Meanwhile I'll prepare missing bits for steps (2) and
>                 (3).
>                 Hopefully everything is in place for step (4), but we
>                 need to make steps (2) and (3) first.
>
>                 Andrew.
>
>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>
>                 Hi Adam,
>
>                 > I've conferred with the rest of the team, and we
>                 think it would be best to move forward with mainly
>                 option B.
>
>                 OK, I'll provide the sample on Monday for you. It is
>                 almost ready right now, but I need to double-check
>                 it before publishing.
>
>                 Regards,
>                 Andrew.
>
>                 On 8/17/23 20:03, Adam Hassick wrote:
>
>                 Hi Andrew,
>
>                 I'm adding the CI mailing list to this
>                 conversation. Others in the community might find
>                 this conversation valuable.
>
>                 We do want to run testing on a regular basis. The
>                 Jenkins integration will be very useful for us, as
>                 most of our CI is orchestrated by Jenkins.
>                 I've conferred with the rest of the team, and we
>                 think it would be best to move forward with mainly
>                 option B.
>                 If you would like to know anything about our
>                 testbeds that would help you with creating an
>                 example ts-rigs repo, I'd be happy to answer any
>                 questions you have.
>
>                 We have multiple test rigs (we call these
>                 "DUT-tester pairs") that we run our existing
>                 hardware testing on, with differing network
>                 hardware and CPU architecture. I figured this might
>                 be an important detail.
>
>                 Thanks,
>                 Adam
>
>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>                 <andrew.rybchenko@oktetlabs.ru
>                 <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>                     Greatings Adam,
>
>                     I'm happy to hear that you're trying to bring
>                     it up.
>
>                     As I understand the final goal is to run it on
>                     regular basis. So, we need to make it properly
>                     from the very beginning.
>                     Bring up of all features consists of 4 steps:
>
>                     1. Create site-specific repository (we call it
>                     ts-rigs) which contains information about test
>                     rigs and other site-specific information like
>                     where to send mails, where to store logs etc.
>                     It is required for manual execution as well,
>                     since test rigs description is essential. I'll
>                     return to the topic below.
>
>                     2. Setup logs storage for automated runs.
>                     Basically it is a disk space plus apache2 web
>                     server with few CGI scripts which help a lot to
>                     save disk space.
>
>                     3. Setup Bublik web application which provides
>                     web interface to view testing results. Same as
>                     https://ts-factory.io/bublik
>                     <https://ts-factory.io/bublik>
> <https://ts-factory.io/bublik>
>
>                     4. Setup Jenkins to run tests on regularly,
>                     save logs in log storage (2) and import it to
>                     bublik (3).
>
>                     Last few month we spent on our homework to make
>                     it simpler to bring up automated execution
>                     using Jenkins -
>                     https://github.com/ts-factory/te-jenkins
>                     <https://github.com/ts-factory/te-jenkins>
> <https://github.com/ts-factory/te-jenkins>
>                     Corresponding bits in dpdk-ethdev-ts will be
>                     available tomorrow.
>
>                     Let's return to the step (1).
>
>                     Unfortunately there is no publicly available
>                     example of the ts-rigs repository since
>                     sensitive site-specific information is located
>                     there. But I'm ready to help you to create it
>                     for UNH. I see two options here:
>
>                     (A) I'll ask questions and based on your
>                     answers will create the first draft with my
>                     comments.
>
>                     (B) I'll make a template/example ts-rigs repo,
>                     publish it and you'll create UNH ts-rigs based
>                     on it.
>
>                     Of course, I'll help to debug and finally bring
>                     it up in any case.
>
>                     (A) is a bit simpler for me and you, but (B) is
>                     a bit more generic and will help other
>                     potential users to bring it up.
>                     We can combine (A)+(B). I.e. start from (A).
>                     What do you think?
>
>                     Thanks,
>                     Andrew.
>
>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>
>                     Greetings Adam,
>
>
>                     Thanks for contacting us. I copy Andrew who
>                     would be happy to help
>
>                     Thanks,
>                     Konstantin
>
>                     On 16 Aug 2023, at 21:50, Adam Hassick
>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
> wrote:
>
>                     
>                     Greetings Konstantin,
>
>                     I am in the process of setting up the DPDK
>                     Poll Mode Driver test suite as an addition to
>                     our testing coverage for DPDK at the UNH lab.
>
>                     I have some questions about how to set the
>                     test suite arguments.
>
>                     I have been able to configure the Test Engine
>                     to connect to the hosts in the testbed. The
>                     RCF, Configurator, and Tester all begin to
>                     run, however the prelude of the test suite
>                     fails to run.
>
>
> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>
>                     The documentation mentions that there are
>                     several test parameters for the test suite,
>                     like for the IUT test link MAC, etc. These
>                     seem like they would need to be set somewhere
>                     to run many of the tests.
>
>                     I see in the Test Engine documentation, there
>                     are instructions on how to create new
>                     parameters for test suites in the Tester
>                     configuration, but there is nothing in the
>                     user guide or in the Tester guide for how to
>                     set the arguments for the parameters when
>                     running the test suite that I can find. I'm
>                     not sure if I need to write my own Tester
>                     config, or if I should be setting these in
>                     some other way.
>
>                     How should these values be set?
>
>                     I'm also not sure what environment
>                     variables/arguments are strictly necessary or
>                     which are optional.
>
>                     Regards,
>                     Adam
>
>                     --                     *Adam Hassick*
>                     Senior Developer
>                     UNH InterOperability Lab
>                     ahassick@iol.unh.edu
>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>                     iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>                     +1 (603) 475-8248
>
>
>
>
>                 --                 *Adam Hassick*
>                 Senior Developer
>                 UNH InterOperability Lab
>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>                 iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>                 +1 (603) 475-8248
>
>
>
>
>
>             --             *Adam Hassick*
>             Senior Developer
>             UNH InterOperability Lab
>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>             iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>             +1 (603) 475-8248
>
>
>
>         --         *Adam Hassick*
>         Senior Developer
>         UNH InterOperability Lab
>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>         iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>         +1 (603) 475-8248
>
>
>
>
>     --     *Adam Hassick*
>     Senior Developer
>     UNH InterOperability Lab
>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>     +1 (603) 475-8248
>
>
>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 42032 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-25 14:06                           ` Adam Hassick
@ 2023-08-25 14:41                             ` Andrew Rybchenko
  2023-08-25 17:35                               ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-25 14:41 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 28066 bytes --]

On 8/25/23 17:06, Adam Hassick wrote:
> Hi Andrew,
>
> Two of our systems (the Test Engine runner and the DUT host) are 
> running Ubuntu 20.04 LTS, however this morning I noticed that the 
> tester system (the one having issues) is running Ubuntu 22.04 LTS.
> This could be the source of the problem. I encountered a dependency 
> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the 
> system. Since the tester is also the host having connection issues, I 
> will try downgrading that system to 20.04, and see if that changes 
> anything.

Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu 
22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
Right now Debian 11 is used for test engine in nightly regressions.

I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.

> I did try passing in the "--vg-rcf" argument to the run.sh script of 
> the test suite after installing valgrind, but there was no additional 
> output that I saw.

Sorry, I should valgrind output should be in valgrind.te_rcf (direction 
where you run test engine).

>
> I will try pulling in the changes you've pushed up, and will see if 
> that fixes anything.
>
> Thanks,
> Adam
>
> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hello Adam,
>
>     On 8/24/23 23:54, Andrew Rybchenko wrote:
>>     I'd like to try to repeat the problem locally. Which Linux distro
>>     is running on test engine and agents?
>>
>>     In fact I know one problem with Debian 12 and Fedora 38 and we have
>>     patch in review to fix it, however, the behaviour is different in
>>     this case, so it is unlike the same problem.
>
>     I've just published a new tag which fixes known test engine side
>     problems on Debian 12 and Fedora 38.
>
>>
>>     One more idea is to install valgrind on the test engine host and
>>     run with option --vg-rcf to check if something weird is happening.
>>
>>     What I don't understand right now is why I see just one failed
>>     attempt
>>     to connect in your log.txt and then Logger shutdown after 9 minutes.
>>
>>     Andrew.
>>
>>     On 8/24/23 23:29, Adam Hassick wrote:
>>>      > Is there any firewall in the network or on test hosts which
>>>     could block incoming TCP connection to the port 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>     where you run test engine?
>>>
>>>     Our test engine host and the testbed are on the same subnet. The
>>>     connection does work sometimes.
>>>
>>>      > If behaviour the same on the next try and you see that test
>>>     agent is kept running, could you check using
>>>      >
>>>      > # netstat -tnlp
>>>      >
>>>      > that Test Agent is listening on the port and try to establish
>>>     TCP connection from test agent using
>>>      >
>>>      > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>      >
>>>      > and check if TCP connection could be established.
>>>
>>>     I was able to replicate the same behavior again, where it hangs
>>>     while RCF is trying to start.
>>>     Running this command, I see this in the output:
>>>
>>>     tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>     <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*        
>>>           LISTEN      18599/ta
>>>
>>>     So it seems like it is listening on the correct port.
>>>     Additionally, I was able to connect to the Tester machine from
>>>     our Test Engine host using telnet. It printed the PID of the
>>>     process once the connection was opened.
>>>
>>>     I tried running the "ta" application manually on the command
>>>     line, and it didn't print anything at all.
>>>     Maybe the issue is something on the Test Engine side.
>>>
>>>     On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru
>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>         Hi Adam,
>>>
>>>          > On the tester host (which appears to be the Peer agent),
>>>     there
>>>         are four processes that I see running, which look like the test
>>>         agent processes.
>>>
>>>         Before the next try I'd recommend to kill these processes.
>>>
>>>         Is there any firewall in the network or on test hosts which
>>>     could
>>>         block incoming TCP connection to the port 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>         where you run test engine?
>>>
>>>         If behaviour the same on the next try and you see that test
>>>     agent is
>>>         kept running, could you check using
>>>
>>>         # netstat -tnlp
>>>
>>>         that Test Agent is listening on the port and try to
>>>     establish TCP
>>>         connection from test agent using
>>>
>>>         $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>         and check if TCP connection could be established.
>>>
>>>         Another idea is to login Tester under root as testing does, get
>>>         start TA command from the log and try it by hands without -n
>>>     and
>>>         remove extra escaping.
>>>
>>>         # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>     /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>     host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>
>>>         Hopefully in this case test agent directory remains in the
>>>     /tmp and
>>>         you don't need to copy it as testing does.
>>>         May be output could shed some light on what's going on.
>>>
>>>         Andrew.
>>>
>>>         On 8/24/23 17:30, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         This is the output that I see in the terminal when this
>>>>     failure
>>>>         occurs, after the test agent binaries build and the test
>>>>     engine
>>>>         starts:
>>>>
>>>>         Platform default build - pass
>>>>         Simple RCF consistency check succeeded
>>>>         --->>> Starting Logger...done
>>>>         --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>     timed
>>>>         out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>         Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>         processes. I let it hang for around 9 minutes.
>>>>
>>>>         On the tester host (which appears to be the Peer agent),
>>>>     there are
>>>>         four processes that I see running, which look like the test
>>>>     agent
>>>>         processes.
>>>>
>>>>         ta.Peer is an empty file. I've attached the log.txt from
>>>>     this run.
>>>>
>>>>          - Adam
>>>>
>>>>         On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>             Hi Adam,
>>>>
>>>>             Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>             that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>             Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>             says that copy_timeout is in seconds and implementation in
>>>>             lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>             Theoretically select() could be interrupted by signal,
>>>>     but I
>>>>             think it is unlikely here.
>>>>
>>>>             I'm not sure that I understand what do you mean by RCF
>>>>             connection timeout. Does it happen on TE startup when RCF
>>>>             starts test agents. If so, TE_RCFUNIX_TIMEOUT could
>>>>     help. Or
>>>>             does it happen when tests are in progress, e.g. in the
>>>>     middle
>>>>             of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>             likely either host with test agent dies or test agent
>>>>     itself
>>>>             crashes. It would be easier for me if classify it if
>>>>     you share
>>>>             text log (log.txt, full or just corresponding fragment
>>>>     with
>>>>             some context). Also content of ta.DPDK or ta.Peer file
>>>>             depending on which agent has problems could shed some
>>>>     light.
>>>>             Corresponding files contain stdout/stderr of test agents.
>>>>
>>>>             Andrew.
>>>>
>>>>             On 8/23/23 17:45, Adam Hassick wrote:
>>>>>             Hi Andrew,
>>>>>
>>>>>             I've set up a test rig repository here, and have created
>>>>>             configurations for our development testbed based off
>>>>>     of the
>>>>>             examples.
>>>>>             We've been able to get the test suite to run manually on
>>>>>             Mellanox CX5 devices once.
>>>>>             However, we are running into an issue where, when RCF
>>>>>     starts,
>>>>>             the RCF connection times out very frequently. We
>>>>>     aren't sure
>>>>>             why this is the case.
>>>>>             It works sometimes, but most of the time when we try
>>>>>     to run
>>>>>             the test engine, it encounters this issue.
>>>>>             I've tried changing the RCF port by setting
>>>>>             "TE_RCF_PORT=<some port number>" and rebooting the
>>>>>     testbed
>>>>>             machines. Neither seems to fix the issue.
>>>>>
>>>>>             It also seems like the timeout takes far longer than 60
>>>>>             seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>             before I try to run the test suite.
>>>>>             I assume the unit for this variable is seconds?
>>>>>
>>>>>             Thanks,
>>>>>             Adam
>>>>>
>>>>>             On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>             <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 Thanks, I've cloned the example repository and
>>>>>     will start
>>>>>                 setting up a configuration for our development
>>>>>     testbed
>>>>>                 today. I'll let you know if I run into any
>>>>>     difficulties
>>>>>                 or have any questions.
>>>>>
>>>>>                  - Adam
>>>>>
>>>>>                 On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                     Hi Adam,
>>>>>
>>>>>                     I've published
>>>>>     https://github.com/ts-factory/ts-rigs-sample
>>>>>     <https://github.com/ts-factory/ts-rigs-sample>
>>>>>     <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>                     Hopefully it will help to define your test
>>>>>     rigs and
>>>>>                     successfully run some tests manually. Feel
>>>>>     free to
>>>>>                     ask any questions and I'll answer here and try to
>>>>>                     update documentation.
>>>>>
>>>>>                     Meanwhile I'll prepare missing bits for steps
>>>>>     (2) and
>>>>>                     (3).
>>>>>                     Hopefully everything is in place for step (4),
>>>>>     but we
>>>>>                     need to make steps (2) and (3) first.
>>>>>
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     > I've conferred with the rest of the team,
>>>>>>     and we
>>>>>>                     think it would be best to move forward with
>>>>>>     mainly
>>>>>>                     option B.
>>>>>>
>>>>>>                     OK, I'll provide the sample on Monday for
>>>>>>     you. It is
>>>>>>                     almost ready right now, but I need to
>>>>>>     double-check
>>>>>>                     it before publishing.
>>>>>>
>>>>>>                     Regards,
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>                     Hi Andrew,
>>>>>>>
>>>>>>>                     I'm adding the CI mailing list to this
>>>>>>>                     conversation. Others in the community might
>>>>>>>     find
>>>>>>>                     this conversation valuable.
>>>>>>>
>>>>>>>                     We do want to run testing on a regular
>>>>>>>     basis. The
>>>>>>>                     Jenkins integration will be very useful for
>>>>>>>     us, as
>>>>>>>                     most of our CI is orchestrated by Jenkins.
>>>>>>>                     I've conferred with the rest of the team,
>>>>>>>     and we
>>>>>>>                     think it would be best to move forward with
>>>>>>>     mainly
>>>>>>>                     option B.
>>>>>>>                     If you would like to know anything about our
>>>>>>>                     testbeds that would help you with creating an
>>>>>>>                     example ts-rigs repo, I'd be happy to answer
>>>>>>>     any
>>>>>>>                     questions you have.
>>>>>>>
>>>>>>>                     We have multiple test rigs (we call these
>>>>>>>                     "DUT-tester pairs") that we run our existing
>>>>>>>                     hardware testing on, with differing network
>>>>>>>                     hardware and CPU architecture. I figured
>>>>>>>     this might
>>>>>>>                     be an important detail.
>>>>>>>
>>>>>>>                     Thanks,
>>>>>>>                     Adam
>>>>>>>
>>>>>>>                     On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>     Rybchenko
>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                         Greatings Adam,
>>>>>>>
>>>>>>>                         I'm happy to hear that you're trying to
>>>>>>>     bring
>>>>>>>                         it up.
>>>>>>>
>>>>>>>                         As I understand the final goal is to run
>>>>>>>     it on
>>>>>>>                         regular basis. So, we need to make it
>>>>>>>     properly
>>>>>>>                         from the very beginning.
>>>>>>>                         Bring up of all features consists of 4
>>>>>>>     steps:
>>>>>>>
>>>>>>>                         1. Create site-specific repository (we
>>>>>>>     call it
>>>>>>>                         ts-rigs) which contains information
>>>>>>>     about test
>>>>>>>                         rigs and other site-specific information
>>>>>>>     like
>>>>>>>                         where to send mails, where to store logs
>>>>>>>     etc.
>>>>>>>                         It is required for manual execution as
>>>>>>>     well,
>>>>>>>                         since test rigs description is
>>>>>>>     essential. I'll
>>>>>>>                         return to the topic below.
>>>>>>>
>>>>>>>                         2. Setup logs storage for automated runs.
>>>>>>>                         Basically it is a disk space plus
>>>>>>>     apache2 web
>>>>>>>                         server with few CGI scripts which help a
>>>>>>>     lot to
>>>>>>>                         save disk space.
>>>>>>>
>>>>>>>                         3. Setup Bublik web application which
>>>>>>>     provides
>>>>>>>                         web interface to view testing results.
>>>>>>>     Same as
>>>>>>>     https://ts-factory.io/bublik
>>>>>>>     <https://ts-factory.io/bublik> <https://ts-factory.io/bublik>
>>>>>>>
>>>>>>>                         4. Setup Jenkins to run tests on regularly,
>>>>>>>                         save logs in log storage (2) and import
>>>>>>>     it to
>>>>>>>                         bublik (3).
>>>>>>>
>>>>>>>                         Last few month we spent on our homework
>>>>>>>     to make
>>>>>>>                         it simpler to bring up automated execution
>>>>>>>                         using Jenkins -
>>>>>>>     https://github.com/ts-factory/te-jenkins
>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>                         Corresponding bits in dpdk-ethdev-ts
>>>>>>>     will be
>>>>>>>                         available tomorrow.
>>>>>>>
>>>>>>>                         Let's return to the step (1).
>>>>>>>
>>>>>>>                         Unfortunately there is no publicly
>>>>>>>     available
>>>>>>>                         example of the ts-rigs repository since
>>>>>>>                         sensitive site-specific information is
>>>>>>>     located
>>>>>>>                         there. But I'm ready to help you to
>>>>>>>     create it
>>>>>>>                         for UNH. I see two options here:
>>>>>>>
>>>>>>>                         (A) I'll ask questions and based on your
>>>>>>>                         answers will create the first draft with my
>>>>>>>                         comments.
>>>>>>>
>>>>>>>                         (B) I'll make a template/example ts-rigs
>>>>>>>     repo,
>>>>>>>                         publish it and you'll create UNH ts-rigs
>>>>>>>     based
>>>>>>>                         on it.
>>>>>>>
>>>>>>>                         Of course, I'll help to debug and
>>>>>>>     finally bring
>>>>>>>                         it up in any case.
>>>>>>>
>>>>>>>                         (A) is a bit simpler for me and you, but
>>>>>>>     (B) is
>>>>>>>                         a bit more generic and will help other
>>>>>>>                         potential users to bring it up.
>>>>>>>                         We can combine (A)+(B). I.e. start from
>>>>>>>     (A).
>>>>>>>                         What do you think?
>>>>>>>
>>>>>>>                         Thanks,
>>>>>>>                         Andrew.
>>>>>>>
>>>>>>>                         On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>>>>     Greetings Adam,
>>>>>>>>
>>>>>>>>
>>>>>>>>                         Thanks for contacting us. I copy Andrew
>>>>>>>>     who
>>>>>>>>                         would be happy to help
>>>>>>>>
>>>>>>>>                         Thanks,
>>>>>>>>                         Konstantin
>>>>>>>>
>>>>>>>>>                         On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>     <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>
>>>>>>>>>                         
>>>>>>>>>                         Greetings Konstantin,
>>>>>>>>>
>>>>>>>>>                         I am in the process of setting up the
>>>>>>>>>     DPDK
>>>>>>>>>                         Poll Mode Driver test suite as an
>>>>>>>>>     addition to
>>>>>>>>>                         our testing coverage for DPDK at the
>>>>>>>>>     UNH lab.
>>>>>>>>>
>>>>>>>>>                         I have some questions about how to set
>>>>>>>>>     the
>>>>>>>>>                         test suite arguments.
>>>>>>>>>
>>>>>>>>>                         I have been able to configure the Test
>>>>>>>>>     Engine
>>>>>>>>>                         to connect to the hosts in the
>>>>>>>>>     testbed. The
>>>>>>>>>                         RCF, Configurator, and Tester all
>>>>>>>>>     begin to
>>>>>>>>>                         run, however the prelude of the test
>>>>>>>>>     suite
>>>>>>>>>                         fails to run.
>>>>>>>>>
>>>>>>>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         The documentation mentions that there are
>>>>>>>>>                         several test parameters for the test
>>>>>>>>>     suite,
>>>>>>>>>                         like for the IUT test link MAC, etc.
>>>>>>>>>     These
>>>>>>>>>                         seem like they would need to be set
>>>>>>>>>     somewhere
>>>>>>>>>                         to run many of the tests.
>>>>>>>>>
>>>>>>>>>                         I see in the Test Engine
>>>>>>>>>     documentation, there
>>>>>>>>>                         are instructions on how to create new
>>>>>>>>>                         parameters for test suites in the Tester
>>>>>>>>>                         configuration, but there is nothing in
>>>>>>>>>     the
>>>>>>>>>                         user guide or in the Tester guide for
>>>>>>>>>     how to
>>>>>>>>>                         set the arguments for the parameters when
>>>>>>>>>                         running the test suite that I can
>>>>>>>>>     find. I'm
>>>>>>>>>                         not sure if I need to write my own Tester
>>>>>>>>>                         config, or if I should be setting
>>>>>>>>>     these in
>>>>>>>>>                         some other way.
>>>>>>>>>
>>>>>>>>>                         How should these values be set?
>>>>>>>>>
>>>>>>>>>                         I'm also not sure what environment
>>>>>>>>>                         variables/arguments are strictly
>>>>>>>>>     necessary or
>>>>>>>>>                         which are optional.
>>>>>>>>>
>>>>>>>>>                         Regards,
>>>>>>>>>                         Adam
>>>>>>>>>
>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>                         Senior Developer
>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>     ahassick@iol.unh.edu
>>>>>>>>>     <mailto:ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>     <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                     --                 *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>>     <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 --             *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>     <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>             --         *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>     <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --     *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>     <mailto:ahassick@iol.unh.edu>
>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>     <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>     <mailto:ahassick@iol.unh.edu>
>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>     <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 48828 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-25 14:41                             ` Andrew Rybchenko
@ 2023-08-25 17:35                               ` Andrew Rybchenko
  2023-08-28 15:02                                 ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-25 17:35 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 29469 bytes --]

 > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.

Done. It works fine for me without any issues.

Have you noticed any pattern when it works or does not work?
May be it is a problem of not clean state after termination?
Does it work fine the first time after DUTs reboot?
How do you terminate testing? It should be done using Ctrl+C in terminal 
where you execute run.sh command.
  In this case it should shutdown gracefully and close all test agents 
and engine applications.

(I'm trying to understand why you've seen many test agent processes. It 
should not happen.)

Andrew.

On 8/25/23 17:41, Andrew Rybchenko wrote:
> On 8/25/23 17:06, Adam Hassick wrote:
>> Hi Andrew,
>>
>> Two of our systems (the Test Engine runner and the DUT host) are 
>> running Ubuntu 20.04 LTS, however this morning I noticed that the 
>> tester system (the one having issues) is running Ubuntu 22.04 LTS.
>> This could be the source of the problem. I encountered a dependency 
>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the 
>> system. Since the tester is also the host having connection issues, I 
>> will try downgrading that system to 20.04, and see if that changes 
>> anything.
>
> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu 
> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
> Right now Debian 11 is used for test engine in nightly regressions.
>
> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
>> I did try passing in the "--vg-rcf" argument to the run.sh script of 
>> the test suite after installing valgrind, but there was no additional 
>> output that I saw.
>
> Sorry, I should valgrind output should be in valgrind.te_rcf 
> (direction where you run test engine).
>
>>
>> I will try pulling in the changes you've pushed up, and will see if 
>> that fixes anything.
>>
>> Thanks,
>> Adam
>>
>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>     Hello Adam,
>>
>>     On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>     I'd like to try to repeat the problem locally. Which Linux
>>>     distro is running on test engine and agents?
>>>
>>>     In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>     patch in review to fix it, however, the behaviour is different in
>>>     this case, so it is unlike the same problem.
>>
>>     I've just published a new tag which fixes known test engine side
>>     problems on Debian 12 and Fedora 38.
>>
>>>
>>>     One more idea is to install valgrind on the test engine host and
>>>     run with option --vg-rcf to check if something weird is happening.
>>>
>>>     What I don't understand right now is why I see just one failed
>>>     attempt
>>>     to connect in your log.txt and then Logger shutdown after 9
>>>     minutes.
>>>
>>>     Andrew.
>>>
>>>     On 8/24/23 23:29, Adam Hassick wrote:
>>>>      > Is there any firewall in the network or on test hosts which
>>>>     could block incoming TCP connection to the port 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>     where you run test engine?
>>>>
>>>>     Our test engine host and the testbed are on the same subnet.
>>>>     The connection does work sometimes.
>>>>
>>>>      > If behaviour the same on the next try and you see that test
>>>>     agent is kept running, could you check using
>>>>      >
>>>>      > # netstat -tnlp
>>>>      >
>>>>      > that Test Agent is listening on the port and try to
>>>>     establish TCP connection from test agent using
>>>>      >
>>>>      > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>      >
>>>>      > and check if TCP connection could be established.
>>>>
>>>>     I was able to replicate the same behavior again, where it hangs
>>>>     while RCF is trying to start.
>>>>     Running this command, I see this in the output:
>>>>
>>>>     tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>     <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*        
>>>>           LISTEN      18599/ta
>>>>
>>>>     So it seems like it is listening on the correct port.
>>>>     Additionally, I was able to connect to the Tester machine from
>>>>     our Test Engine host using telnet. It printed the PID of the
>>>>     process once the connection was opened.
>>>>
>>>>     I tried running the "ta" application manually on the command
>>>>     line, and it didn't print anything at all.
>>>>     Maybe the issue is something on the Test Engine side.
>>>>
>>>>     On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>         Hi Adam,
>>>>
>>>>          > On the tester host (which appears to be the Peer agent),
>>>>     there
>>>>         are four processes that I see running, which look like the
>>>>     test
>>>>         agent processes.
>>>>
>>>>         Before the next try I'd recommend to kill these processes.
>>>>
>>>>         Is there any firewall in the network or on test hosts which
>>>>     could
>>>>         block incoming TCP connection to the port 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>         where you run test engine?
>>>>
>>>>         If behaviour the same on the next try and you see that test
>>>>     agent is
>>>>         kept running, could you check using
>>>>
>>>>         # netstat -tnlp
>>>>
>>>>         that Test Agent is listening on the port and try to
>>>>     establish TCP
>>>>         connection from test agent using
>>>>
>>>>         $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>         and check if TCP connection could be established.
>>>>
>>>>         Another idea is to login Tester under root as testing does,
>>>>     get
>>>>         start TA command from the log and try it by hands without
>>>>     -n and
>>>>         remove extra escaping.
>>>>
>>>>         # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>     LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>     /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>     host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>
>>>>         Hopefully in this case test agent directory remains in the
>>>>     /tmp and
>>>>         you don't need to copy it as testing does.
>>>>         May be output could shed some light on what's going on.
>>>>
>>>>         Andrew.
>>>>
>>>>         On 8/24/23 17:30, Adam Hassick wrote:
>>>>>         Hi Andrew,
>>>>>
>>>>>         This is the output that I see in the terminal when this
>>>>>     failure
>>>>>         occurs, after the test agent binaries build and the test
>>>>>     engine
>>>>>         starts:
>>>>>
>>>>>         Platform default build - pass
>>>>>         Simple RCF consistency check succeeded
>>>>>         --->>> Starting Logger...done
>>>>>         --->>> Starting RCF...rcf_net_engine_connect(): Connection
>>>>>     timed
>>>>>         out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>         Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>         processes. I let it hang for around 9 minutes.
>>>>>
>>>>>         On the tester host (which appears to be the Peer agent),
>>>>>     there are
>>>>>         four processes that I see running, which look like the
>>>>>     test agent
>>>>>         processes.
>>>>>
>>>>>         ta.Peer is an empty file. I've attached the log.txt from
>>>>>     this run.
>>>>>
>>>>>          - Adam
>>>>>
>>>>>         On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>             Hi Adam,
>>>>>
>>>>>             Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>     double-checked
>>>>>             that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>             Description in in
>>>>>     doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>             says that copy_timeout is in seconds and
>>>>>     implementation in
>>>>>             lib/rcfunix/rcfunix.c passes the value to select()
>>>>>     tv_sec.
>>>>>             Theoretically select() could be interrupted by signal,
>>>>>     but I
>>>>>             think it is unlikely here.
>>>>>
>>>>>             I'm not sure that I understand what do you mean by RCF
>>>>>             connection timeout. Does it happen on TE startup when RCF
>>>>>             starts test agents. If so, TE_RCFUNIX_TIMEOUT could
>>>>>     help. Or
>>>>>             does it happen when tests are in progress, e.g. in the
>>>>>     middle
>>>>>             of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and
>>>>>     most
>>>>>             likely either host with test agent dies or test agent
>>>>>     itself
>>>>>             crashes. It would be easier for me if classify it if
>>>>>     you share
>>>>>             text log (log.txt, full or just corresponding fragment
>>>>>     with
>>>>>             some context). Also content of ta.DPDK or ta.Peer file
>>>>>             depending on which agent has problems could shed some
>>>>>     light.
>>>>>             Corresponding files contain stdout/stderr of test agents.
>>>>>
>>>>>             Andrew.
>>>>>
>>>>>             On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>             Hi Andrew,
>>>>>>
>>>>>>             I've set up a test rig repository here, and have created
>>>>>>             configurations for our development testbed based off
>>>>>>     of the
>>>>>>             examples.
>>>>>>             We've been able to get the test suite to run manually on
>>>>>>             Mellanox CX5 devices once.
>>>>>>             However, we are running into an issue where, when RCF
>>>>>>     starts,
>>>>>>             the RCF connection times out very frequently. We
>>>>>>     aren't sure
>>>>>>             why this is the case.
>>>>>>             It works sometimes, but most of the time when we try
>>>>>>     to run
>>>>>>             the test engine, it encounters this issue.
>>>>>>             I've tried changing the RCF port by setting
>>>>>>             "TE_RCF_PORT=<some port number>" and rebooting the
>>>>>>     testbed
>>>>>>             machines. Neither seems to fix the issue.
>>>>>>
>>>>>>             It also seems like the timeout takes far longer than 60
>>>>>>             seconds, even when running "export
>>>>>>     TE_RCFUNIX_TIMEOUT=60"
>>>>>>             before I try to run the test suite.
>>>>>>             I assume the unit for this variable is seconds?
>>>>>>
>>>>>>             Thanks,
>>>>>>             Adam
>>>>>>
>>>>>>             On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>             <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>     <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>
>>>>>>                 Hi Andrew,
>>>>>>
>>>>>>                 Thanks, I've cloned the example repository and
>>>>>>     will start
>>>>>>                 setting up a configuration for our development
>>>>>>     testbed
>>>>>>                 today. I'll let you know if I run into any
>>>>>>     difficulties
>>>>>>                 or have any questions.
>>>>>>
>>>>>>                  - Adam
>>>>>>
>>>>>>                 On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                     Hi Adam,
>>>>>>
>>>>>>                     I've published
>>>>>>     https://github.com/ts-factory/ts-rigs-sample
>>>>>>     <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>     <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>                     Hopefully it will help to define your test
>>>>>>     rigs and
>>>>>>                     successfully run some tests manually. Feel
>>>>>>     free to
>>>>>>                     ask any questions and I'll answer here and
>>>>>>     try to
>>>>>>                     update documentation.
>>>>>>
>>>>>>                     Meanwhile I'll prepare missing bits for steps
>>>>>>     (2) and
>>>>>>                     (3).
>>>>>>                     Hopefully everything is in place for step
>>>>>>     (4), but we
>>>>>>                     need to make steps (2) and (3) first.
>>>>>>
>>>>>>                     Andrew.
>>>>>>
>>>>>>                     On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>                     Hi Adam,
>>>>>>>
>>>>>>>                     > I've conferred with the rest of the team,
>>>>>>>     and we
>>>>>>>                     think it would be best to move forward with
>>>>>>>     mainly
>>>>>>>                     option B.
>>>>>>>
>>>>>>>                     OK, I'll provide the sample on Monday for
>>>>>>>     you. It is
>>>>>>>                     almost ready right now, but I need to
>>>>>>>     double-check
>>>>>>>                     it before publishing.
>>>>>>>
>>>>>>>                     Regards,
>>>>>>>                     Andrew.
>>>>>>>
>>>>>>>                     On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>                     Hi Andrew,
>>>>>>>>
>>>>>>>>                     I'm adding the CI mailing list to this
>>>>>>>>                     conversation. Others in the community might
>>>>>>>>     find
>>>>>>>>                     this conversation valuable.
>>>>>>>>
>>>>>>>>                     We do want to run testing on a regular
>>>>>>>>     basis. The
>>>>>>>>                     Jenkins integration will be very useful for
>>>>>>>>     us, as
>>>>>>>>                     most of our CI is orchestrated by Jenkins.
>>>>>>>>                     I've conferred with the rest of the team,
>>>>>>>>     and we
>>>>>>>>                     think it would be best to move forward with
>>>>>>>>     mainly
>>>>>>>>                     option B.
>>>>>>>>                     If you would like to know anything about our
>>>>>>>>                     testbeds that would help you with creating an
>>>>>>>>                     example ts-rigs repo, I'd be happy to
>>>>>>>>     answer any
>>>>>>>>                     questions you have.
>>>>>>>>
>>>>>>>>                     We have multiple test rigs (we call these
>>>>>>>>                     "DUT-tester pairs") that we run our existing
>>>>>>>>                     hardware testing on, with differing network
>>>>>>>>                     hardware and CPU architecture. I figured
>>>>>>>>     this might
>>>>>>>>                     be an important detail.
>>>>>>>>
>>>>>>>>                     Thanks,
>>>>>>>>                     Adam
>>>>>>>>
>>>>>>>>                     On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>>     Rybchenko
>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                         Greatings Adam,
>>>>>>>>
>>>>>>>>                         I'm happy to hear that you're trying to
>>>>>>>>     bring
>>>>>>>>                         it up.
>>>>>>>>
>>>>>>>>                         As I understand the final goal is to
>>>>>>>>     run it on
>>>>>>>>                         regular basis. So, we need to make it
>>>>>>>>     properly
>>>>>>>>                         from the very beginning.
>>>>>>>>                         Bring up of all features consists of 4
>>>>>>>>     steps:
>>>>>>>>
>>>>>>>>                         1. Create site-specific repository (we
>>>>>>>>     call it
>>>>>>>>                         ts-rigs) which contains information
>>>>>>>>     about test
>>>>>>>>                         rigs and other site-specific
>>>>>>>>     information like
>>>>>>>>                         where to send mails, where to store
>>>>>>>>     logs etc.
>>>>>>>>                         It is required for manual execution as
>>>>>>>>     well,
>>>>>>>>                         since test rigs description is
>>>>>>>>     essential. I'll
>>>>>>>>                         return to the topic below.
>>>>>>>>
>>>>>>>>                         2. Setup logs storage for automated runs.
>>>>>>>>                         Basically it is a disk space plus
>>>>>>>>     apache2 web
>>>>>>>>                         server with few CGI scripts which help
>>>>>>>>     a lot to
>>>>>>>>                         save disk space.
>>>>>>>>
>>>>>>>>                         3. Setup Bublik web application which
>>>>>>>>     provides
>>>>>>>>                         web interface to view testing results.
>>>>>>>>     Same as
>>>>>>>>     https://ts-factory.io/bublik
>>>>>>>>     <https://ts-factory.io/bublik> <https://ts-factory.io/bublik>
>>>>>>>>
>>>>>>>>                         4. Setup Jenkins to run tests on
>>>>>>>>     regularly,
>>>>>>>>                         save logs in log storage (2) and import
>>>>>>>>     it to
>>>>>>>>                         bublik (3).
>>>>>>>>
>>>>>>>>                         Last few month we spent on our homework
>>>>>>>>     to make
>>>>>>>>                         it simpler to bring up automated execution
>>>>>>>>                         using Jenkins -
>>>>>>>>     https://github.com/ts-factory/te-jenkins
>>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>     <https://github.com/ts-factory/te-jenkins>
>>>>>>>>                         Corresponding bits in dpdk-ethdev-ts
>>>>>>>>     will be
>>>>>>>>                         available tomorrow.
>>>>>>>>
>>>>>>>>                         Let's return to the step (1).
>>>>>>>>
>>>>>>>>                         Unfortunately there is no publicly
>>>>>>>>     available
>>>>>>>>                         example of the ts-rigs repository since
>>>>>>>>                         sensitive site-specific information is
>>>>>>>>     located
>>>>>>>>                         there. But I'm ready to help you to
>>>>>>>>     create it
>>>>>>>>                         for UNH. I see two options here:
>>>>>>>>
>>>>>>>>                         (A) I'll ask questions and based on your
>>>>>>>>                         answers will create the first draft
>>>>>>>>     with my
>>>>>>>>                         comments.
>>>>>>>>
>>>>>>>>                         (B) I'll make a template/example
>>>>>>>>     ts-rigs repo,
>>>>>>>>                         publish it and you'll create UNH
>>>>>>>>     ts-rigs based
>>>>>>>>                         on it.
>>>>>>>>
>>>>>>>>                         Of course, I'll help to debug and
>>>>>>>>     finally bring
>>>>>>>>                         it up in any case.
>>>>>>>>
>>>>>>>>                         (A) is a bit simpler for me and you,
>>>>>>>>     but (B) is
>>>>>>>>                         a bit more generic and will help other
>>>>>>>>                         potential users to bring it up.
>>>>>>>>                         We can combine (A)+(B). I.e. start from
>>>>>>>>     (A).
>>>>>>>>                         What do you think?
>>>>>>>>
>>>>>>>>                         Thanks,
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/17/23 15:18, Konstantin Ushakov
>>>>>>>>     wrote:
>>>>>>>>>     Greetings Adam,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         Thanks for contacting us. I copy
>>>>>>>>>     Andrew who
>>>>>>>>>                         would be happy to help
>>>>>>>>>
>>>>>>>>>                         Thanks,
>>>>>>>>>                         Konstantin
>>>>>>>>>
>>>>>>>>>>     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>     <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>     <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>                         
>>>>>>>>>>                         Greetings Konstantin,
>>>>>>>>>>
>>>>>>>>>>                         I am in the process of setting up the
>>>>>>>>>>     DPDK
>>>>>>>>>>                         Poll Mode Driver test suite as an
>>>>>>>>>>     addition to
>>>>>>>>>>                         our testing coverage for DPDK at the
>>>>>>>>>>     UNH lab.
>>>>>>>>>>
>>>>>>>>>>                         I have some questions about how to
>>>>>>>>>>     set the
>>>>>>>>>>                         test suite arguments.
>>>>>>>>>>
>>>>>>>>>>                         I have been able to configure the
>>>>>>>>>>     Test Engine
>>>>>>>>>>                         to connect to the hosts in the
>>>>>>>>>>     testbed. The
>>>>>>>>>>                         RCF, Configurator, and Tester all
>>>>>>>>>>     begin to
>>>>>>>>>>                         run, however the prelude of the test
>>>>>>>>>>     suite
>>>>>>>>>>                         fails to run.
>>>>>>>>>>
>>>>>>>>>>     https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>     <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         The documentation mentions that there
>>>>>>>>>>     are
>>>>>>>>>>                         several test parameters for the test
>>>>>>>>>>     suite,
>>>>>>>>>>                         like for the IUT test link MAC, etc.
>>>>>>>>>>     These
>>>>>>>>>>                         seem like they would need to be set
>>>>>>>>>>     somewhere
>>>>>>>>>>                         to run many of the tests.
>>>>>>>>>>
>>>>>>>>>>                         I see in the Test Engine
>>>>>>>>>>     documentation, there
>>>>>>>>>>                         are instructions on how to create new
>>>>>>>>>>                         parameters for test suites in the Tester
>>>>>>>>>>                         configuration, but there is nothing
>>>>>>>>>>     in the
>>>>>>>>>>                         user guide or in the Tester guide for
>>>>>>>>>>     how to
>>>>>>>>>>                         set the arguments for the parameters
>>>>>>>>>>     when
>>>>>>>>>>                         running the test suite that I can
>>>>>>>>>>     find. I'm
>>>>>>>>>>                         not sure if I need to write my own
>>>>>>>>>>     Tester
>>>>>>>>>>                         config, or if I should be setting
>>>>>>>>>>     these in
>>>>>>>>>>                         some other way.
>>>>>>>>>>
>>>>>>>>>>                         How should these values be set?
>>>>>>>>>>
>>>>>>>>>>                         I'm also not sure what environment
>>>>>>>>>>                         variables/arguments are strictly
>>>>>>>>>>     necessary or
>>>>>>>>>>                         which are optional.
>>>>>>>>>>
>>>>>>>>>>                         Regards,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>     ahassick@iol.unh.edu
>>>>>>>>>>     <mailto:ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>     iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>     <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                     --                 *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>>>     <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                 --             *Adam Hassick*
>>>>>>                 Senior Developer
>>>>>>                 UNH InterOperability Lab
>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>     <https://www.iol.unh.edu/>
>>>>>>                 +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>             --         *Adam Hassick*
>>>>>>             Senior Developer
>>>>>>             UNH InterOperability Lab
>>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>>     <https://www.iol.unh.edu/>
>>>>>>             +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>         --     *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>     <mailto:ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>>     <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>     -- 
>>>>     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>     <mailto:ahassick@iol.unh.edu>
>>>>     iol.unh.edu <http://iol.unh.edu> <https://www.iol.unh.edu/>
>>>>     <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>>
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 51855 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-25 17:35                               ` Andrew Rybchenko
@ 2023-08-28 15:02                                 ` Adam Hassick
  2023-08-28 21:05                                   ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-28 15:02 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 23185 bytes --]

Hi Andrew,

We have yet to notice a distinct pattern with the failures. Sometimes, the
RCF will start and connect without issue a few times in a row before
failing to connect again. Once the issue begins to occur, neither rebooting
all of the hosts (test engine VM, tester, IUT) or deleting all of the build
directories (suites, agents, inst) and rebooting the hosts afterward
resolves the issue. When it begins working again seems very arbitrary to us.

I do usually try to terminate the test engine with Ctrl+C, but when it
hangs while trying to start RCF, that does not work.

Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
connection to the test bed hosts? And if so, is there a way to force it to
use IPv4?

 - Adam

On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
> Done. It works fine for me without any issues.
>
> Have you noticed any pattern when it works or does not work?
> May be it is a problem of not clean state after termination?
> Does it work fine the first time after DUTs reboot?
> How do you terminate testing? It should be done using Ctrl+C in terminal
> where you execute run.sh command.
>  In this case it should shutdown gracefully and close all test agents and
> engine applications.
>
> (I'm trying to understand why you've seen many test agent processes. It
> should not happen.)
>
> Andrew.
>
> On 8/25/23 17:41, Andrew Rybchenko wrote:
>
> On 8/25/23 17:06, Adam Hassick wrote:
>
> Hi Andrew,
>
> Two of our systems (the Test Engine runner and the DUT host) are running
> Ubuntu 20.04 LTS, however this morning I noticed that the tester system
> (the one having issues) is running Ubuntu 22.04 LTS.
> This could be the source of the problem. I encountered a dependency issue
> trying to run the Test Engine on 22.04 LTS, so I downgraded the system.
> Since the tester is also the host having connection issues, I will try
> downgrading that system to 20.04, and see if that changes anything.
>
>
> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
> Right now Debian 11 is used for test engine in nightly regressions.
>
> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
> I did try passing in the "--vg-rcf" argument to the run.sh script of the
> test suite after installing valgrind, but there was no additional output
> that I saw.
>
>
> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
> where you run test engine).
>
>
> I will try pulling in the changes you've pushed up, and will see if that
> fixes anything.
>
> Thanks,
> Adam
>
> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hello Adam,
>>
>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>
>> I'd like to try to repeat the problem locally. Which Linux distro is
>> running on test engine and agents?
>>
>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>> patch in review to fix it, however, the behaviour is different in
>> this case, so it is unlike the same problem.
>>
>>
>> I've just published a new tag which fixes known test engine side problems
>> on Debian 12 and Fedora 38.
>>
>>
>> One more idea is to install valgrind on the test engine host and
>> run with option --vg-rcf to check if something weird is happening.
>>
>> What I don't understand right now is why I see just one failed attempt
>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>
>> Andrew.
>>
>> On 8/24/23 23:29, Adam Hassick wrote:
>>
>>  > Is there any firewall in the network or on test hosts which could
>> block incoming TCP connection to the port 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>> you run test engine?
>>
>> Our test engine host and the testbed are on the same subnet. The
>> connection does work sometimes.
>>
>>  > If behaviour the same on the next try and you see that test agent is
>> kept running, could you check using
>>  >
>>  > # netstat -tnlp
>>  >
>>  > that Test Agent is listening on the port and try to establish TCP
>> connection from test agent using
>>  >
>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>  >
>>  > and check if TCP connection could be established.
>>
>> I was able to replicate the same behavior again, where it hangs while RCF
>> is trying to start.
>> Running this command, I see this in the output:
>>
>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>  18599/ta
>>
>> So it seems like it is listening on the correct port.
>> Additionally, I was able to connect to the Tester machine from our Test
>> Engine host using telnet. It printed the PID of the process once the
>> connection was opened.
>>
>> I tried running the "ta" application manually on the command line, and it
>> didn't print anything at all.
>> Maybe the issue is something on the Test Engine side.
>>
>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>     Hi Adam,
>>
>>      > On the tester host (which appears to be the Peer agent), there
>>     are four processes that I see running, which look like the test
>>     agent processes.
>>
>>     Before the next try I'd recommend to kill these processes.
>>
>>     Is there any firewall in the network or on test hosts which could
>>     block incoming TCP connection to the port 23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>     where you run test engine?
>>
>>     If behaviour the same on the next try and you see that test agent is
>>     kept running, could you check using
>>
>>     # netstat -tnlp
>>
>>     that Test Agent is listening on the port and try to establish TCP
>>     connection from test agent using
>>
>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     and check if TCP connection could be established.
>>
>>     Another idea is to login Tester under root as testing does, get
>>     start TA command from the log and try it by hands without -n and
>>     remove extra escaping.
>>
>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>
>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>
>>     Hopefully in this case test agent directory remains in the /tmp and
>>     you don't need to copy it as testing does.
>>     May be output could shed some light on what's going on.
>>
>>     Andrew.
>>
>>     On 8/24/23 17:30, Adam Hassick wrote:
>>
>>     Hi Andrew,
>>
>>     This is the output that I see in the terminal when this failure
>>     occurs, after the test agent binaries build and the test engine
>>     starts:
>>
>>     Platform default build - pass
>>     Simple RCF consistency check succeeded
>>     --->>> Starting Logger...done
>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>
>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>     processes. I let it hang for around 9 minutes.
>>
>>     On the tester host (which appears to be the Peer agent), there are
>>     four processes that I see running, which look like the test agent
>>     processes.
>>
>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>
>>      - Adam
>>
>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru
>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>         Hi Adam,
>>
>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>         says that copy_timeout is in seconds and implementation in
>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>         Theoretically select() could be interrupted by signal, but I
>>         think it is unlikely here.
>>
>>         I'm not sure that I understand what do you mean by RCF
>>         connection timeout. Does it happen on TE startup when RCF
>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>         does it happen when tests are in progress, e.g. in the middle
>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>         likely either host with test agent dies or test agent itself
>>         crashes. It would be easier for me if classify it if you share
>>         text log (log.txt, full or just corresponding fragment with
>>         some context). Also content of ta.DPDK or ta.Peer file
>>         depending on which agent has problems could shed some light.
>>         Corresponding files contain stdout/stderr of test agents.
>>
>>         Andrew.
>>
>>         On 8/23/23 17:45, Adam Hassick wrote:
>>
>>         Hi Andrew,
>>
>>         I've set up a test rig repository here, and have created
>>         configurations for our development testbed based off of the
>>         examples.
>>         We've been able to get the test suite to run manually on
>>         Mellanox CX5 devices once.
>>         However, we are running into an issue where, when RCF starts,
>>         the RCF connection times out very frequently. We aren't sure
>>         why this is the case.
>>         It works sometimes, but most of the time when we try to run
>>         the test engine, it encounters this issue.
>>         I've tried changing the RCF port by setting
>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>         machines. Neither seems to fix the issue.
>>
>>         It also seems like the timeout takes far longer than 60
>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>         before I try to run the test suite.
>>         I assume the unit for this variable is seconds?
>>
>>         Thanks,
>>         Adam
>>
>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>> wrote:
>>
>>             Hi Andrew,
>>
>>             Thanks, I've cloned the example repository and will start
>>             setting up a configuration for our development testbed
>>             today. I'll let you know if I run into any difficulties
>>             or have any questions.
>>
>>              - Adam
>>
>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>             <andrew.rybchenko@oktetlabs.ru
>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>                 Hi Adam,
>>
>>                 I've published
>>                 https://github.com/ts-factory/ts-rigs-sample
>>                 <https://github.com/ts-factory/ts-rigs-sample>
>> <https://github.com/ts-factory/ts-rigs-sample>.
>>                 Hopefully it will help to define your test rigs and
>>                 successfully run some tests manually. Feel free to
>>                 ask any questions and I'll answer here and try to
>>                 update documentation.
>>
>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>                 (3).
>>                 Hopefully everything is in place for step (4), but we
>>                 need to make steps (2) and (3) first.
>>
>>                 Andrew.
>>
>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>
>>                 Hi Adam,
>>
>>                 > I've conferred with the rest of the team, and we
>>                 think it would be best to move forward with mainly
>>                 option B.
>>
>>                 OK, I'll provide the sample on Monday for you. It is
>>                 almost ready right now, but I need to double-check
>>                 it before publishing.
>>
>>                 Regards,
>>                 Andrew.
>>
>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>
>>                 Hi Andrew,
>>
>>                 I'm adding the CI mailing list to this
>>                 conversation. Others in the community might find
>>                 this conversation valuable.
>>
>>                 We do want to run testing on a regular basis. The
>>                 Jenkins integration will be very useful for us, as
>>                 most of our CI is orchestrated by Jenkins.
>>                 I've conferred with the rest of the team, and we
>>                 think it would be best to move forward with mainly
>>                 option B.
>>                 If you would like to know anything about our
>>                 testbeds that would help you with creating an
>>                 example ts-rigs repo, I'd be happy to answer any
>>                 questions you have.
>>
>>                 We have multiple test rigs (we call these
>>                 "DUT-tester pairs") that we run our existing
>>                 hardware testing on, with differing network
>>                 hardware and CPU architecture. I figured this might
>>                 be an important detail.
>>
>>                 Thanks,
>>                 Adam
>>
>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>                 <andrew.rybchenko@oktetlabs.ru
>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>
>>                     Greatings Adam,
>>
>>                     I'm happy to hear that you're trying to bring
>>                     it up.
>>
>>                     As I understand the final goal is to run it on
>>                     regular basis. So, we need to make it properly
>>                     from the very beginning.
>>                     Bring up of all features consists of 4 steps:
>>
>>                     1. Create site-specific repository (we call it
>>                     ts-rigs) which contains information about test
>>                     rigs and other site-specific information like
>>                     where to send mails, where to store logs etc.
>>                     It is required for manual execution as well,
>>                     since test rigs description is essential. I'll
>>                     return to the topic below.
>>
>>                     2. Setup logs storage for automated runs.
>>                     Basically it is a disk space plus apache2 web
>>                     server with few CGI scripts which help a lot to
>>                     save disk space.
>>
>>                     3. Setup Bublik web application which provides
>>                     web interface to view testing results. Same as
>>                     https://ts-factory.io/bublik
>>                     <https://ts-factory.io/bublik>
>> <https://ts-factory.io/bublik>
>>
>>                     4. Setup Jenkins to run tests on regularly,
>>                     save logs in log storage (2) and import it to
>>                     bublik (3).
>>
>>                     Last few month we spent on our homework to make
>>                     it simpler to bring up automated execution
>>                     using Jenkins -
>>                     https://github.com/ts-factory/te-jenkins
>>                     <https://github.com/ts-factory/te-jenkins>
>> <https://github.com/ts-factory/te-jenkins>
>>                     Corresponding bits in dpdk-ethdev-ts will be
>>                     available tomorrow.
>>
>>                     Let's return to the step (1).
>>
>>                     Unfortunately there is no publicly available
>>                     example of the ts-rigs repository since
>>                     sensitive site-specific information is located
>>                     there. But I'm ready to help you to create it
>>                     for UNH. I see two options here:
>>
>>                     (A) I'll ask questions and based on your
>>                     answers will create the first draft with my
>>                     comments.
>>
>>                     (B) I'll make a template/example ts-rigs repo,
>>                     publish it and you'll create UNH ts-rigs based
>>                     on it.
>>
>>                     Of course, I'll help to debug and finally bring
>>                     it up in any case.
>>
>>                     (A) is a bit simpler for me and you, but (B) is
>>                     a bit more generic and will help other
>>                     potential users to bring it up.
>>                     We can combine (A)+(B). I.e. start from (A).
>>                     What do you think?
>>
>>                     Thanks,
>>                     Andrew.
>>
>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>
>>                     Greetings Adam,
>>
>>
>>                     Thanks for contacting us. I copy Andrew who
>>                     would be happy to help
>>
>>                     Thanks,
>>                     Konstantin
>>
>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>> wrote:
>>
>>                     
>>                     Greetings Konstantin,
>>
>>                     I am in the process of setting up the DPDK
>>                     Poll Mode Driver test suite as an addition to
>>                     our testing coverage for DPDK at the UNH lab.
>>
>>                     I have some questions about how to set the
>>                     test suite arguments.
>>
>>                     I have been able to configure the Test Engine
>>                     to connect to the hosts in the testbed. The
>>                     RCF, Configurator, and Tester all begin to
>>                     run, however the prelude of the test suite
>>                     fails to run.
>>
>>
>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>
>>                     The documentation mentions that there are
>>                     several test parameters for the test suite,
>>                     like for the IUT test link MAC, etc. These
>>                     seem like they would need to be set somewhere
>>                     to run many of the tests.
>>
>>                     I see in the Test Engine documentation, there
>>                     are instructions on how to create new
>>                     parameters for test suites in the Tester
>>                     configuration, but there is nothing in the
>>                     user guide or in the Tester guide for how to
>>                     set the arguments for the parameters when
>>                     running the test suite that I can find. I'm
>>                     not sure if I need to write my own Tester
>>                     config, or if I should be setting these in
>>                     some other way.
>>
>>                     How should these values be set?
>>
>>                     I'm also not sure what environment
>>                     variables/arguments are strictly necessary or
>>                     which are optional.
>>
>>                     Regards,
>>                     Adam
>>
>>                     --                     *Adam Hassick*
>>                     Senior Developer
>>                     UNH InterOperability Lab
>>                     ahassick@iol.unh.edu
>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>                     iol.unh.edu <https://www.iol.unh.edu/>
>> <https://www.iol.unh.edu/>
>>                     +1 (603) 475-8248
>>
>>
>>
>>
>>                 --                 *Adam Hassick*
>>                 Senior Developer
>>                 UNH InterOperability Lab
>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>                 iol.unh.edu <https://www.iol.unh.edu/>
>> <https://www.iol.unh.edu/>
>>                 +1 (603) 475-8248
>>
>>
>>
>>
>>
>>             --             *Adam Hassick*
>>             Senior Developer
>>             UNH InterOperability Lab
>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>             iol.unh.edu <https://www.iol.unh.edu/>
>> <https://www.iol.unh.edu/>
>>             +1 (603) 475-8248
>>
>>
>>
>>         --         *Adam Hassick*
>>         Senior Developer
>>         UNH InterOperability Lab
>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>         iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>         +1 (603) 475-8248
>>
>>
>>
>>
>>     --     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>> <ahassick@iol.unh.edu>
>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>>
>>
>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 49907 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-28 15:02                                 ` Adam Hassick
@ 2023-08-28 21:05                                   ` Andrew Rybchenko
  2023-08-29 12:07                                     ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-28 21:05 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 34391 bytes --]

Hi Adam,

 > Does the test engine prefer to use IPv6 over IPv4 for initiating the 
RCF connection to the test bed hosts? And if so, is there a way to force 
it to use IPv4?

Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your 
case, I guess it is the root cause of the problem.
Of course, it is TE problem since I see really weird code in 
lib/comm_net_engine/comm_net_engine.c line 135.

I've pushed fix to the branch user/arybchik/fix_ipv4_only in 
ts-factory/test-environment repository. Please, try.

It is late night fix with minimal testing and no review. I'll pass it 
through review process tomorrow and
hopefully it will be released in one-two days.

Andrew.

On 8/28/23 18:02, Adam Hassick wrote:
> Hi Andrew,
>
> We have yet to notice a distinct pattern with the failures. Sometimes, 
> the RCF will start and connect without issue a few times in a row 
> before failing to connect again. Once the issue begins to occur, 
> neither rebooting all of the hosts (test engine VM, tester, IUT) or 
> deleting all of the build directories (suites, agents, inst) and 
> rebooting the hosts afterward resolves the issue. When it begins 
> working again seems very arbitrary to us.
>
> I do usually try to terminate the test engine with Ctrl+C, but when it 
> hangs while trying to start RCF, that does not work.
>
> Does the test engine prefer to use IPv6 over IPv4 for initiating the 
> RCF connection to the test bed hosts? And if so, is there a way to 
> force it to use IPv4?
>
>  - Adam
>
> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>
>     Done. It works fine for me without any issues.
>
>     Have you noticed any pattern when it works or does not work?
>     May be it is a problem of not clean state after termination?
>     Does it work fine the first time after DUTs reboot?
>     How do you terminate testing? It should be done using Ctrl+C in
>     terminal where you execute run.sh command.
>      In this case it should shutdown gracefully and close all test
>     agents and engine applications.
>
>     (I'm trying to understand why you've seen many test agent
>     processes. It should not happen.)
>
>     Andrew.
>
>     On 8/25/23 17:41, Andrew Rybchenko wrote:
>>     On 8/25/23 17:06, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     Two of our systems (the Test Engine runner and the DUT host) are
>>>     running Ubuntu 20.04 LTS, however this morning I noticed that
>>>     the tester system (the one having issues) is running Ubuntu
>>>     22.04 LTS.
>>>     This could be the source of the problem. I encountered a
>>>     dependency issue trying to run the Test Engine on 22.04 LTS, so
>>>     I downgraded the system. Since the tester is also the host
>>>     having connection issues, I will try downgrading that system to
>>>     20.04, and see if that changes anything.
>>
>>     Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>     Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38
>>     every night.
>>     Right now Debian 11 is used for test engine in nightly regressions.
>>
>>     I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>>>     I did try passing in the "--vg-rcf" argument to the run.sh
>>>     script of the test suite after installing valgrind, but there
>>>     was no additional output that I saw.
>>
>>     Sorry, I should valgrind output should be in valgrind.te_rcf
>>     (direction where you run test engine).
>>
>>>
>>>     I will try pulling in the changes you've pushed up, and will see
>>>     if that fixes anything.
>>>
>>>     Thanks,
>>>     Adam
>>>
>>>     On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>         Hello Adam,
>>>
>>>         On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>         I'd like to try to repeat the problem locally. Which Linux
>>>>         distro is running on test engine and agents?
>>>>
>>>>         In fact I know one problem with Debian 12 and Fedora 38 and
>>>>         we have
>>>>         patch in review to fix it, however, the behaviour is
>>>>         different in
>>>>         this case, so it is unlike the same problem.
>>>
>>>         I've just published a new tag which fixes known test engine
>>>         side problems on Debian 12 and Fedora 38.
>>>
>>>>
>>>>         One more idea is to install valgrind on the test engine
>>>>         host and
>>>>         run with option --vg-rcf to check if something weird is
>>>>         happening.
>>>>
>>>>         What I don't understand right now is why I see just one
>>>>         failed attempt
>>>>         to connect in your log.txt and then Logger shutdown after 9
>>>>         minutes.
>>>>
>>>>         Andrew.
>>>>
>>>>         On 8/24/23 23:29, Adam Hassick wrote:
>>>>>          > Is there any firewall in the network or on test hosts
>>>>>         which could block incoming TCP connection to the port
>>>>>         23571 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the
>>>>>         host where you run test engine?
>>>>>
>>>>>         Our test engine host and the testbed are on the same
>>>>>         subnet. The connection does work sometimes.
>>>>>
>>>>>          > If behaviour the same on the next try and you see that
>>>>>         test agent is kept running, could you check using
>>>>>          >
>>>>>          > # netstat -tnlp
>>>>>          >
>>>>>          > that Test Agent is listening on the port and try to
>>>>>         establish TCP connection from test agent using
>>>>>          >
>>>>>          > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>          >
>>>>>          > and check if TCP connection could be established.
>>>>>
>>>>>         I was able to replicate the same behavior again, where it
>>>>>         hangs while RCF is trying to start.
>>>>>         Running this command, I see this in the output:
>>>>>
>>>>>         tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>         <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*  
>>>>>                     LISTEN      18599/ta
>>>>>
>>>>>         So it seems like it is listening on the correct port.
>>>>>         Additionally, I was able to connect to the Tester machine
>>>>>         from our Test Engine host using telnet. It printed the PID
>>>>>         of the process once the connection was opened.
>>>>>
>>>>>         I tried running the "ta" application manually on the
>>>>>         command line, and it didn't print anything at all.
>>>>>         Maybe the issue is something on the Test Engine side.
>>>>>
>>>>>         On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>             Hi Adam,
>>>>>
>>>>>              > On the tester host (which appears to be the Peer
>>>>>         agent), there
>>>>>             are four processes that I see running, which look like
>>>>>         the test
>>>>>             agent processes.
>>>>>
>>>>>             Before the next try I'd recommend to kill these
>>>>>         processes.
>>>>>
>>>>>             Is there any firewall in the network or on test hosts
>>>>>         which could
>>>>>             block incoming TCP connection to the port 23571
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the
>>>>>         host
>>>>>             where you run test engine?
>>>>>
>>>>>             If behaviour the same on the next try and you see that
>>>>>         test agent is
>>>>>             kept running, could you check using
>>>>>
>>>>>             # netstat -tnlp
>>>>>
>>>>>             that Test Agent is listening on the port and try to
>>>>>         establish TCP
>>>>>             connection from test agent using
>>>>>
>>>>>             $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>             and check if TCP connection could be established.
>>>>>
>>>>>             Another idea is to login Tester under root as testing
>>>>>         does, get
>>>>>             start TA command from the log and try it by hands
>>>>>         without -n and
>>>>>             remove extra escaping.
>>>>>
>>>>>             # sudo
>>>>>         PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>         LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>         /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>         host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>
>>>>>             Hopefully in this case test agent directory remains in
>>>>>         the /tmp and
>>>>>             you don't need to copy it as testing does.
>>>>>             May be output could shed some light on what's going on.
>>>>>
>>>>>             Andrew.
>>>>>
>>>>>             On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>             Hi Andrew,
>>>>>>
>>>>>>             This is the output that I see in the terminal when
>>>>>>         this failure
>>>>>>             occurs, after the test agent binaries build and the
>>>>>>         test engine
>>>>>>             starts:
>>>>>>
>>>>>>             Platform default build - pass
>>>>>>             Simple RCF consistency check succeeded
>>>>>>             --->>> Starting Logger...done
>>>>>>             --->>> Starting RCF...rcf_net_engine_connect():
>>>>>>         Connection timed
>>>>>>             out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>
>>>>>>             Then, it hangs here until I kill the "te_rcf" and
>>>>>>         "te_tee"
>>>>>>             processes. I let it hang for around 9 minutes.
>>>>>>
>>>>>>             On the tester host (which appears to be the Peer
>>>>>>         agent), there are
>>>>>>             four processes that I see running, which look like
>>>>>>         the test agent
>>>>>>             processes.
>>>>>>
>>>>>>             ta.Peer is an empty file. I've attached the log.txt
>>>>>>         from this run.
>>>>>>
>>>>>>              - Adam
>>>>>>
>>>>>>             On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>                 Hi Adam,
>>>>>>
>>>>>>                 Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>         double-checked
>>>>>>                 that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>                 Description in in
>>>>>>         doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>                 says that copy_timeout is in seconds and
>>>>>>         implementation in
>>>>>>                 lib/rcfunix/rcfunix.c passes the value to
>>>>>>         select() tv_sec.
>>>>>>                 Theoretically select() could be interrupted by
>>>>>>         signal, but I
>>>>>>                 think it is unlikely here.
>>>>>>
>>>>>>                 I'm not sure that I understand what do you mean
>>>>>>         by RCF
>>>>>>                 connection timeout. Does it happen on TE startup
>>>>>>         when RCF
>>>>>>                 starts test agents. If so, TE_RCFUNIX_TIMEOUT
>>>>>>         could help. Or
>>>>>>                 does it happen when tests are in progress, e.g.
>>>>>>         in the middle
>>>>>>                 of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated
>>>>>>         and most
>>>>>>                 likely either host with test agent dies or test
>>>>>>         agent itself
>>>>>>                 crashes. It would be easier for me if classify it
>>>>>>         if you share
>>>>>>                 text log (log.txt, full or just corresponding
>>>>>>         fragment with
>>>>>>                 some context). Also content of ta.DPDK or ta.Peer
>>>>>>         file
>>>>>>                 depending on which agent has problems could shed
>>>>>>         some light.
>>>>>>                 Corresponding files contain stdout/stderr of test
>>>>>>         agents.
>>>>>>
>>>>>>                 Andrew.
>>>>>>
>>>>>>                 On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>                 Hi Andrew,
>>>>>>>
>>>>>>>                 I've set up a test rig repository here, and have
>>>>>>>         created
>>>>>>>                 configurations for our development testbed based
>>>>>>>         off of the
>>>>>>>                 examples.
>>>>>>>                 We've been able to get the test suite to run
>>>>>>>         manually on
>>>>>>>                 Mellanox CX5 devices once.
>>>>>>>                 However, we are running into an issue where,
>>>>>>>         when RCF starts,
>>>>>>>                 the RCF connection times out very frequently. We
>>>>>>>         aren't sure
>>>>>>>                 why this is the case.
>>>>>>>                 It works sometimes, but most of the time when we
>>>>>>>         try to run
>>>>>>>                 the test engine, it encounters this issue.
>>>>>>>                 I've tried changing the RCF port by setting
>>>>>>>                 "TE_RCF_PORT=<some port number>" and rebooting
>>>>>>>         the testbed
>>>>>>>                 machines. Neither seems to fix the issue.
>>>>>>>
>>>>>>>                 It also seems like the timeout takes far longer
>>>>>>>         than 60
>>>>>>>                 seconds, even when running "export
>>>>>>>         TE_RCFUNIX_TIMEOUT=60"
>>>>>>>                 before I try to run the test suite.
>>>>>>>                 I assume the unit for this variable is seconds?
>>>>>>>
>>>>>>>                 Thanks,
>>>>>>>                 Adam
>>>>>>>
>>>>>>>                 On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>                 <ahassick@iol.unh.edu
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>
>>>>>>>                     Hi Andrew,
>>>>>>>
>>>>>>>                     Thanks, I've cloned the example repository
>>>>>>>         and will start
>>>>>>>                     setting up a configuration for our
>>>>>>>         development testbed
>>>>>>>                     today. I'll let you know if I run into any
>>>>>>>         difficulties
>>>>>>>                     or have any questions.
>>>>>>>
>>>>>>>                      - Adam
>>>>>>>
>>>>>>>                     On Sun, Aug 20, 2023 at 4:40 AM Andrew
>>>>>>>         Rybchenko
>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                         Hi Adam,
>>>>>>>
>>>>>>>                         I've published
>>>>>>>         https://github.com/ts-factory/ts-rigs-sample
>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>                         Hopefully it will help to define your
>>>>>>>         test rigs and
>>>>>>>                         successfully run some tests manually.
>>>>>>>         Feel free to
>>>>>>>                         ask any questions and I'll answer here
>>>>>>>         and try to
>>>>>>>                         update documentation.
>>>>>>>
>>>>>>>                         Meanwhile I'll prepare missing bits for
>>>>>>>         steps (2) and
>>>>>>>                         (3).
>>>>>>>                         Hopefully everything is in place for
>>>>>>>         step (4), but we
>>>>>>>                         need to make steps (2) and (3) first.
>>>>>>>
>>>>>>>                         Andrew.
>>>>>>>
>>>>>>>                         On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>                         Hi Adam,
>>>>>>>>
>>>>>>>>                         > I've conferred with the rest of the
>>>>>>>>         team, and we
>>>>>>>>                         think it would be best to move forward
>>>>>>>>         with mainly
>>>>>>>>                         option B.
>>>>>>>>
>>>>>>>>                         OK, I'll provide the sample on Monday
>>>>>>>>         for you. It is
>>>>>>>>                         almost ready right now, but I need to
>>>>>>>>         double-check
>>>>>>>>                         it before publishing.
>>>>>>>>
>>>>>>>>                         Regards,
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>         Hi Andrew,
>>>>>>>>>
>>>>>>>>>                         I'm adding the CI mailing list to this
>>>>>>>>>                         conversation. Others in the community
>>>>>>>>>         might find
>>>>>>>>>                         this conversation valuable.
>>>>>>>>>
>>>>>>>>>                         We do want to run testing on a regular
>>>>>>>>>         basis. The
>>>>>>>>>                         Jenkins integration will be very
>>>>>>>>>         useful for us, as
>>>>>>>>>                         most of our CI is orchestrated by
>>>>>>>>>         Jenkins.
>>>>>>>>>                         I've conferred with the rest of the
>>>>>>>>>         team, and we
>>>>>>>>>                         think it would be best to move forward
>>>>>>>>>         with mainly
>>>>>>>>>                         option B.
>>>>>>>>>                         If you would like to know anything
>>>>>>>>>         about our
>>>>>>>>>                         testbeds that would help you with
>>>>>>>>>         creating an
>>>>>>>>>                         example ts-rigs repo, I'd be happy to
>>>>>>>>>         answer any
>>>>>>>>>                         questions you have.
>>>>>>>>>
>>>>>>>>>                         We have multiple test rigs (we call these
>>>>>>>>>                         "DUT-tester pairs") that we run our
>>>>>>>>>         existing
>>>>>>>>>                         hardware testing on, with differing
>>>>>>>>>         network
>>>>>>>>>                         hardware and CPU architecture. I
>>>>>>>>>         figured this might
>>>>>>>>>                         be an important detail.
>>>>>>>>>
>>>>>>>>>                         Thanks,
>>>>>>>>>                         Adam
>>>>>>>>>
>>>>>>>>>                         On Thu, Aug 17, 2023 at 11:44 AM
>>>>>>>>>         Andrew Rybchenko
>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                             Greatings Adam,
>>>>>>>>>
>>>>>>>>>                             I'm happy to hear that you're
>>>>>>>>>         trying to bring
>>>>>>>>>                             it up.
>>>>>>>>>
>>>>>>>>>                             As I understand the final goal is
>>>>>>>>>         to run it on
>>>>>>>>>                             regular basis. So, we need to make
>>>>>>>>>         it properly
>>>>>>>>>                             from the very beginning.
>>>>>>>>>                             Bring up of all features consists
>>>>>>>>>         of 4 steps:
>>>>>>>>>
>>>>>>>>>                             1. Create site-specific repository
>>>>>>>>>         (we call it
>>>>>>>>>                             ts-rigs) which contains
>>>>>>>>>         information about test
>>>>>>>>>                             rigs and other site-specific
>>>>>>>>>         information like
>>>>>>>>>                             where to send mails, where to
>>>>>>>>>         store logs etc.
>>>>>>>>>                             It is required for manual
>>>>>>>>>         execution as well,
>>>>>>>>>                             since test rigs description is
>>>>>>>>>         essential. I'll
>>>>>>>>>                             return to the topic below.
>>>>>>>>>
>>>>>>>>>                             2. Setup logs storage for
>>>>>>>>>         automated runs.
>>>>>>>>>                             Basically it is a disk space plus
>>>>>>>>>         apache2 web
>>>>>>>>>                             server with few CGI scripts which
>>>>>>>>>         help a lot to
>>>>>>>>>                             save disk space.
>>>>>>>>>
>>>>>>>>>                             3. Setup Bublik web application
>>>>>>>>>         which provides
>>>>>>>>>                             web interface to view testing
>>>>>>>>>         results. Same as
>>>>>>>>>         https://ts-factory.io/bublik
>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>
>>>>>>>>>                             4. Setup Jenkins to run tests on
>>>>>>>>>         regularly,
>>>>>>>>>                             save logs in log storage (2) and
>>>>>>>>>         import it to
>>>>>>>>>                             bublik (3).
>>>>>>>>>
>>>>>>>>>                             Last few month we spent on our
>>>>>>>>>         homework to make
>>>>>>>>>                             it simpler to bring up automated
>>>>>>>>>         execution
>>>>>>>>>                             using Jenkins -
>>>>>>>>>         https://github.com/ts-factory/te-jenkins
>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>                             Corresponding bits in
>>>>>>>>>         dpdk-ethdev-ts will be
>>>>>>>>>                             available tomorrow.
>>>>>>>>>
>>>>>>>>>                             Let's return to the step (1).
>>>>>>>>>
>>>>>>>>>                             Unfortunately there is no publicly
>>>>>>>>>         available
>>>>>>>>>                             example of the ts-rigs repository
>>>>>>>>>         since
>>>>>>>>>                             sensitive site-specific
>>>>>>>>>         information is located
>>>>>>>>>                             there. But I'm ready to help you
>>>>>>>>>         to create it
>>>>>>>>>                             for UNH. I see two options here:
>>>>>>>>>
>>>>>>>>>                             (A) I'll ask questions and based
>>>>>>>>>         on your
>>>>>>>>>                             answers will create the first
>>>>>>>>>         draft with my
>>>>>>>>>                             comments.
>>>>>>>>>
>>>>>>>>>                             (B) I'll make a template/example
>>>>>>>>>         ts-rigs repo,
>>>>>>>>>                             publish it and you'll create UNH
>>>>>>>>>         ts-rigs based
>>>>>>>>>                             on it.
>>>>>>>>>
>>>>>>>>>                             Of course, I'll help to debug and
>>>>>>>>>         finally bring
>>>>>>>>>                             it up in any case.
>>>>>>>>>
>>>>>>>>>                             (A) is a bit simpler for me and
>>>>>>>>>         you, but (B) is
>>>>>>>>>                             a bit more generic and will help
>>>>>>>>>         other
>>>>>>>>>                             potential users to bring it up.
>>>>>>>>>                             We can combine (A)+(B). I.e. start
>>>>>>>>>         from (A).
>>>>>>>>>                             What do you think?
>>>>>>>>>
>>>>>>>>>                             Thanks,
>>>>>>>>>                             Andrew.
>>>>>>>>>
>>>>>>>>>                             On 8/17/23 15:18, Konstantin
>>>>>>>>>         Ushakov wrote:
>>>>>>>>>>         Greetings Adam,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             Thanks for contacting us. I copy
>>>>>>>>>>         Andrew who
>>>>>>>>>>                             would be happy to help
>>>>>>>>>>
>>>>>>>>>>                             Thanks,
>>>>>>>>>>                             Konstantin
>>>>>>>>>>
>>>>>>>>>>>         On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>         <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>                             
>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>
>>>>>>>>>>>                             I am in the process of setting
>>>>>>>>>>>         up the DPDK
>>>>>>>>>>>                             Poll Mode Driver test suite as
>>>>>>>>>>>         an addition to
>>>>>>>>>>>                             our testing coverage for DPDK at
>>>>>>>>>>>         the UNH lab.
>>>>>>>>>>>
>>>>>>>>>>>                             I have some questions about how
>>>>>>>>>>>         to set the
>>>>>>>>>>>                             test suite arguments.
>>>>>>>>>>>
>>>>>>>>>>>                             I have been able to configure
>>>>>>>>>>>         the Test Engine
>>>>>>>>>>>                             to connect to the hosts in the
>>>>>>>>>>>         testbed. The
>>>>>>>>>>>                             RCF, Configurator, and Tester
>>>>>>>>>>>         all begin to
>>>>>>>>>>>                             run, however the prelude of the
>>>>>>>>>>>         test suite
>>>>>>>>>>>                             fails to run.
>>>>>>>>>>>
>>>>>>>>>>>         https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             The documentation mentions that
>>>>>>>>>>>         there are
>>>>>>>>>>>                             several test parameters for the
>>>>>>>>>>>         test suite,
>>>>>>>>>>>                             like for the IUT test link MAC,
>>>>>>>>>>>         etc. These
>>>>>>>>>>>                             seem like they would need to be
>>>>>>>>>>>         set somewhere
>>>>>>>>>>>                             to run many of the tests.
>>>>>>>>>>>
>>>>>>>>>>>                             I see in the Test Engine
>>>>>>>>>>>         documentation, there
>>>>>>>>>>>                             are instructions on how to
>>>>>>>>>>>         create new
>>>>>>>>>>>                             parameters for test suites in
>>>>>>>>>>>         the Tester
>>>>>>>>>>>                             configuration, but there is
>>>>>>>>>>>         nothing in the
>>>>>>>>>>>                             user guide or in the Tester
>>>>>>>>>>>         guide for how to
>>>>>>>>>>>                             set the arguments for the
>>>>>>>>>>>         parameters when
>>>>>>>>>>>                             running the test suite that I
>>>>>>>>>>>         can find. I'm
>>>>>>>>>>>                             not sure if I need to write my
>>>>>>>>>>>         own Tester
>>>>>>>>>>>                             config, or if I should be
>>>>>>>>>>>         setting these in
>>>>>>>>>>>                             some other way.
>>>>>>>>>>>
>>>>>>>>>>>                             How should these values be set?
>>>>>>>>>>>
>>>>>>>>>>>                             I'm also not sure what environment
>>>>>>>>>>>         variables/arguments are strictly necessary or
>>>>>>>>>>>                             which are optional.
>>>>>>>>>>>
>>>>>>>>>>>                             Regards,
>>>>>>>>>>>                             Adam
>>>>>>>>>>>
>>>>>>>>>>>                             --                     *Adam
>>>>>>>>>>>         Hassick*
>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>         ahassick@iol.unh.edu
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>                         Senior Developer
>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                     --             *Adam Hassick*
>>>>>>>                     Senior Developer
>>>>>>>                     UNH InterOperability Lab
>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>                     +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                 --         *Adam Hassick*
>>>>>>>                 Senior Developer
>>>>>>>                 UNH InterOperability Lab
>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>                 +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>             --     *Adam Hassick*
>>>>>>             Senior Developer
>>>>>>             UNH InterOperability Lab
>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>             +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>         -- 
>>>>>         *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 65155 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-28 21:05                                   ` Andrew Rybchenko
@ 2023-08-29 12:07                                     ` Andrew Rybchenko
  2023-08-29 14:02                                       ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-29 12:07 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 35242 bytes --]

Hi Adam,

I've pushed the fix in main branch and a new tag v1.18.1. It should 
solve the problem with IPv6 address from DNS.

Andrew.

On 8/29/23 00:05, Andrew Rybchenko wrote:
> Hi Adam,
>
> > Does the test engine prefer to use IPv6 over IPv4 for initiating the 
> RCF connection to the test bed hosts? And if so, is there a way to 
> force it to use IPv4?
>
> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your 
> case, I guess it is the root cause of the problem.
> Of course, it is TE problem since I see really weird code in 
> lib/comm_net_engine/comm_net_engine.c line 135.
>
> I've pushed fix to the branch user/arybchik/fix_ipv4_only in 
> ts-factory/test-environment repository. Please, try.
>
> It is late night fix with minimal testing and no review. I'll pass it 
> through review process tomorrow and
> hopefully it will be released in one-two days.
>
> Andrew.
>
> On 8/28/23 18:02, Adam Hassick wrote:
>> Hi Andrew,
>>
>> We have yet to notice a distinct pattern with the failures. 
>> Sometimes, the RCF will start and connect without issue a few times 
>> in a row before failing to connect again. Once the issue begins to 
>> occur, neither rebooting all of the hosts (test engine VM, tester, 
>> IUT) or deleting all of the build directories (suites, agents, inst) 
>> and rebooting the hosts afterward resolves the issue. When it begins 
>> working again seems very arbitrary to us.
>>
>> I do usually try to terminate the test engine with Ctrl+C, but when 
>> it hangs while trying to start RCF, that does not work.
>>
>> Does the test engine prefer to use IPv6 over IPv4 for initiating the 
>> RCF connection to the test bed hosts? And if so, is there a way to 
>> force it to use IPv4?
>>
>>  - Adam
>>
>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko 
>> <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>     > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>>     Done. It works fine for me without any issues.
>>
>>     Have you noticed any pattern when it works or does not work?
>>     May be it is a problem of not clean state after termination?
>>     Does it work fine the first time after DUTs reboot?
>>     How do you terminate testing? It should be done using Ctrl+C in
>>     terminal where you execute run.sh command.
>>      In this case it should shutdown gracefully and close all test
>>     agents and engine applications.
>>
>>     (I'm trying to understand why you've seen many test agent
>>     processes. It should not happen.)
>>
>>     Andrew.
>>
>>     On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>     On 8/25/23 17:06, Adam Hassick wrote:
>>>>     Hi Andrew,
>>>>
>>>>     Two of our systems (the Test Engine runner and the DUT host)
>>>>     are running Ubuntu 20.04 LTS, however this morning I noticed
>>>>     that the tester system (the one having issues) is running
>>>>     Ubuntu 22.04 LTS.
>>>>     This could be the source of the problem. I encountered a
>>>>     dependency issue trying to run the Test Engine on 22.04 LTS, so
>>>>     I downgraded the system. Since the tester is also the host
>>>>     having connection issues, I will try downgrading that system to
>>>>     20.04, and see if that changes anything.
>>>
>>>     Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
>>>     Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora
>>>     38 every night.
>>>     Right now Debian 11 is used for test engine in nightly regressions.
>>>
>>>     I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>>>     I did try passing in the "--vg-rcf" argument to the run.sh
>>>>     script of the test suite after installing valgrind, but there
>>>>     was no additional output that I saw.
>>>
>>>     Sorry, I should valgrind output should be in valgrind.te_rcf
>>>     (direction where you run test engine).
>>>
>>>>
>>>>     I will try pulling in the changes you've pushed up, and will
>>>>     see if that fixes anything.
>>>>
>>>>     Thanks,
>>>>     Adam
>>>>
>>>>     On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>         Hello Adam,
>>>>
>>>>         On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>         I'd like to try to repeat the problem locally. Which Linux
>>>>>         distro is running on test engine and agents?
>>>>>
>>>>>         In fact I know one problem with Debian 12 and Fedora 38
>>>>>         and we have
>>>>>         patch in review to fix it, however, the behaviour is
>>>>>         different in
>>>>>         this case, so it is unlike the same problem.
>>>>
>>>>         I've just published a new tag which fixes known test engine
>>>>         side problems on Debian 12 and Fedora 38.
>>>>
>>>>>
>>>>>         One more idea is to install valgrind on the test engine
>>>>>         host and
>>>>>         run with option --vg-rcf to check if something weird is
>>>>>         happening.
>>>>>
>>>>>         What I don't understand right now is why I see just one
>>>>>         failed attempt
>>>>>         to connect in your log.txt and then Logger shutdown after
>>>>>         9 minutes.
>>>>>
>>>>>         Andrew.
>>>>>
>>>>>         On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>          > Is there any firewall in the network or on test hosts
>>>>>>         which could block incoming TCP connection to the port
>>>>>>         23571 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from
>>>>>>         the host where you run test engine?
>>>>>>
>>>>>>         Our test engine host and the testbed are on the same
>>>>>>         subnet. The connection does work sometimes.
>>>>>>
>>>>>>          > If behaviour the same on the next try and you see that
>>>>>>         test agent is kept running, could you check using
>>>>>>          >
>>>>>>          > # netstat -tnlp
>>>>>>          >
>>>>>>          > that Test Agent is listening on the port and try to
>>>>>>         establish TCP connection from test agent using
>>>>>>          >
>>>>>>          > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>          >
>>>>>>          > and check if TCP connection could be established.
>>>>>>
>>>>>>         I was able to replicate the same behavior again, where it
>>>>>>         hangs while RCF is trying to start.
>>>>>>         Running this command, I see this in the output:
>>>>>>
>>>>>>         tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>>>         <http://0.0.0.0:23571> <http://0.0.0.0:23571> 0.0.0.0:*  
>>>>>>                     LISTEN      18599/ta
>>>>>>
>>>>>>         So it seems like it is listening on the correct port.
>>>>>>         Additionally, I was able to connect to the Tester machine
>>>>>>         from our Test Engine host using telnet. It printed the
>>>>>>         PID of the process once the connection was opened.
>>>>>>
>>>>>>         I tried running the "ta" application manually on the
>>>>>>         command line, and it didn't print anything at all.
>>>>>>         Maybe the issue is something on the Test Engine side.
>>>>>>
>>>>>>         On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>>>         <andrew.rybchenko@oktetlabs.ru
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>
>>>>>>             Hi Adam,
>>>>>>
>>>>>>              > On the tester host (which appears to be the Peer
>>>>>>         agent), there
>>>>>>             are four processes that I see running, which look
>>>>>>         like the test
>>>>>>             agent processes.
>>>>>>
>>>>>>             Before the next try I'd recommend to kill these
>>>>>>         processes.
>>>>>>
>>>>>>             Is there any firewall in the network or on test hosts
>>>>>>         which could
>>>>>>             block incoming TCP connection to the port 23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from
>>>>>>         the host
>>>>>>             where you run test engine?
>>>>>>
>>>>>>             If behaviour the same on the next try and you see
>>>>>>         that test agent is
>>>>>>             kept running, could you check using
>>>>>>
>>>>>>             # netstat -tnlp
>>>>>>
>>>>>>             that Test Agent is listening on the port and try to
>>>>>>         establish TCP
>>>>>>             connection from test agent using
>>>>>>
>>>>>>             $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>
>>>>>>             and check if TCP connection could be established.
>>>>>>
>>>>>>             Another idea is to login Tester under root as testing
>>>>>>         does, get
>>>>>>             start TA command from the log and try it by hands
>>>>>>         without -n and
>>>>>>             remove extra escaping.
>>>>>>
>>>>>>             # sudo
>>>>>>         PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>         LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>         /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>         host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>
>>>>>>             Hopefully in this case test agent directory remains
>>>>>>         in the /tmp and
>>>>>>             you don't need to copy it as testing does.
>>>>>>             May be output could shed some light on what's going on.
>>>>>>
>>>>>>             Andrew.
>>>>>>
>>>>>>             On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>             Hi Andrew,
>>>>>>>
>>>>>>>             This is the output that I see in the terminal when
>>>>>>>         this failure
>>>>>>>             occurs, after the test agent binaries build and the
>>>>>>>         test engine
>>>>>>>             starts:
>>>>>>>
>>>>>>>             Platform default build - pass
>>>>>>>             Simple RCF consistency check succeeded
>>>>>>>             --->>> Starting Logger...done
>>>>>>>             --->>> Starting RCF...rcf_net_engine_connect():
>>>>>>>         Connection timed
>>>>>>>             out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>         <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>
>>>>>>>             Then, it hangs here until I kill the "te_rcf" and
>>>>>>>         "te_tee"
>>>>>>>             processes. I let it hang for around 9 minutes.
>>>>>>>
>>>>>>>             On the tester host (which appears to be the Peer
>>>>>>>         agent), there are
>>>>>>>             four processes that I see running, which look like
>>>>>>>         the test agent
>>>>>>>             processes.
>>>>>>>
>>>>>>>             ta.Peer is an empty file. I've attached the log.txt
>>>>>>>         from this run.
>>>>>>>
>>>>>>>              - Adam
>>>>>>>
>>>>>>>             On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                 Hi Adam,
>>>>>>>
>>>>>>>                 Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>>         double-checked
>>>>>>>                 that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>>>                 Description in in
>>>>>>>         doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>                 says that copy_timeout is in seconds and
>>>>>>>         implementation in
>>>>>>>                 lib/rcfunix/rcfunix.c passes the value to
>>>>>>>         select() tv_sec.
>>>>>>>                 Theoretically select() could be interrupted by
>>>>>>>         signal, but I
>>>>>>>                 think it is unlikely here.
>>>>>>>
>>>>>>>                 I'm not sure that I understand what do you mean
>>>>>>>         by RCF
>>>>>>>                 connection timeout. Does it happen on TE startup
>>>>>>>         when RCF
>>>>>>>                 starts test agents. If so, TE_RCFUNIX_TIMEOUT
>>>>>>>         could help. Or
>>>>>>>                 does it happen when tests are in progress, e.g.
>>>>>>>         in the middle
>>>>>>>                 of a test. If so, TE_RCFUNIX_TIMEOUT is
>>>>>>>         unrelated and most
>>>>>>>                 likely either host with test agent dies or test
>>>>>>>         agent itself
>>>>>>>                 crashes. It would be easier for me if classify
>>>>>>>         it if you share
>>>>>>>                 text log (log.txt, full or just corresponding
>>>>>>>         fragment with
>>>>>>>                 some context). Also content of ta.DPDK or
>>>>>>>         ta.Peer file
>>>>>>>                 depending on which agent has problems could shed
>>>>>>>         some light.
>>>>>>>                 Corresponding files contain stdout/stderr of
>>>>>>>         test agents.
>>>>>>>
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>                 Hi Andrew,
>>>>>>>>
>>>>>>>>                 I've set up a test rig repository here, and
>>>>>>>>         have created
>>>>>>>>                 configurations for our development testbed
>>>>>>>>         based off of the
>>>>>>>>                 examples.
>>>>>>>>                 We've been able to get the test suite to run
>>>>>>>>         manually on
>>>>>>>>                 Mellanox CX5 devices once.
>>>>>>>>                 However, we are running into an issue where,
>>>>>>>>         when RCF starts,
>>>>>>>>                 the RCF connection times out very frequently.
>>>>>>>>         We aren't sure
>>>>>>>>                 why this is the case.
>>>>>>>>                 It works sometimes, but most of the time when
>>>>>>>>         we try to run
>>>>>>>>                 the test engine, it encounters this issue.
>>>>>>>>                 I've tried changing the RCF port by setting
>>>>>>>>                 "TE_RCF_PORT=<some port number>" and rebooting
>>>>>>>>         the testbed
>>>>>>>>                 machines. Neither seems to fix the issue.
>>>>>>>>
>>>>>>>>                 It also seems like the timeout takes far longer
>>>>>>>>         than 60
>>>>>>>>                 seconds, even when running "export
>>>>>>>>         TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>                 before I try to run the test suite.
>>>>>>>>                 I assume the unit for this variable is seconds?
>>>>>>>>
>>>>>>>>                 Thanks,
>>>>>>>>                 Adam
>>>>>>>>
>>>>>>>>                 On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>                 <ahassick@iol.unh.edu
>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>         <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>>
>>>>>>>>                     Hi Andrew,
>>>>>>>>
>>>>>>>>                     Thanks, I've cloned the example repository
>>>>>>>>         and will start
>>>>>>>>                     setting up a configuration for our
>>>>>>>>         development testbed
>>>>>>>>                     today. I'll let you know if I run into any
>>>>>>>>         difficulties
>>>>>>>>                     or have any questions.
>>>>>>>>
>>>>>>>>                      - Adam
>>>>>>>>
>>>>>>>>                     On Sun, Aug 20, 2023 at 4:40 AM Andrew
>>>>>>>>         Rybchenko
>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                         Hi Adam,
>>>>>>>>
>>>>>>>>                         I've published
>>>>>>>>         https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>         <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>                         Hopefully it will help to define your
>>>>>>>>         test rigs and
>>>>>>>>                         successfully run some tests manually.
>>>>>>>>         Feel free to
>>>>>>>>                         ask any questions and I'll answer here
>>>>>>>>         and try to
>>>>>>>>                         update documentation.
>>>>>>>>
>>>>>>>>                         Meanwhile I'll prepare missing bits for
>>>>>>>>         steps (2) and
>>>>>>>>                         (3).
>>>>>>>>                         Hopefully everything is in place for
>>>>>>>>         step (4), but we
>>>>>>>>                         need to make steps (2) and (3) first.
>>>>>>>>
>>>>>>>>                         Andrew.
>>>>>>>>
>>>>>>>>                         On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>>>>>         Hi Adam,
>>>>>>>>>
>>>>>>>>>                         > I've conferred with the rest of the
>>>>>>>>>         team, and we
>>>>>>>>>                         think it would be best to move forward
>>>>>>>>>         with mainly
>>>>>>>>>                         option B.
>>>>>>>>>
>>>>>>>>>                         OK, I'll provide the sample on Monday
>>>>>>>>>         for you. It is
>>>>>>>>>                         almost ready right now, but I need to
>>>>>>>>>         double-check
>>>>>>>>>                         it before publishing.
>>>>>>>>>
>>>>>>>>>                         Regards,
>>>>>>>>>                         Andrew.
>>>>>>>>>
>>>>>>>>>                         On 8/17/23 20:03, Adam Hassick wrote:
>>>>>>>>>>         Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                         I'm adding the CI mailing list to this
>>>>>>>>>>                         conversation. Others in the community
>>>>>>>>>>         might find
>>>>>>>>>>                         this conversation valuable.
>>>>>>>>>>
>>>>>>>>>>                         We do want to run testing on a
>>>>>>>>>>         regular basis. The
>>>>>>>>>>                         Jenkins integration will be very
>>>>>>>>>>         useful for us, as
>>>>>>>>>>                         most of our CI is orchestrated by
>>>>>>>>>>         Jenkins.
>>>>>>>>>>                         I've conferred with the rest of the
>>>>>>>>>>         team, and we
>>>>>>>>>>                         think it would be best to move
>>>>>>>>>>         forward with mainly
>>>>>>>>>>                         option B.
>>>>>>>>>>                         If you would like to know anything
>>>>>>>>>>         about our
>>>>>>>>>>                         testbeds that would help you with
>>>>>>>>>>         creating an
>>>>>>>>>>                         example ts-rigs repo, I'd be happy to
>>>>>>>>>>         answer any
>>>>>>>>>>                         questions you have.
>>>>>>>>>>
>>>>>>>>>>                         We have multiple test rigs (we call
>>>>>>>>>>         these
>>>>>>>>>>                         "DUT-tester pairs") that we run our
>>>>>>>>>>         existing
>>>>>>>>>>                         hardware testing on, with differing
>>>>>>>>>>         network
>>>>>>>>>>                         hardware and CPU architecture. I
>>>>>>>>>>         figured this might
>>>>>>>>>>                         be an important detail.
>>>>>>>>>>
>>>>>>>>>>                         Thanks,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         On Thu, Aug 17, 2023 at 11:44 AM
>>>>>>>>>>         Andrew Rybchenko
>>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>         <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>                             Greatings Adam,
>>>>>>>>>>
>>>>>>>>>>                             I'm happy to hear that you're
>>>>>>>>>>         trying to bring
>>>>>>>>>>                             it up.
>>>>>>>>>>
>>>>>>>>>>                             As I understand the final goal is
>>>>>>>>>>         to run it on
>>>>>>>>>>                             regular basis. So, we need to
>>>>>>>>>>         make it properly
>>>>>>>>>>                             from the very beginning.
>>>>>>>>>>                             Bring up of all features consists
>>>>>>>>>>         of 4 steps:
>>>>>>>>>>
>>>>>>>>>>                             1. Create site-specific
>>>>>>>>>>         repository (we call it
>>>>>>>>>>                             ts-rigs) which contains
>>>>>>>>>>         information about test
>>>>>>>>>>                             rigs and other site-specific
>>>>>>>>>>         information like
>>>>>>>>>>                             where to send mails, where to
>>>>>>>>>>         store logs etc.
>>>>>>>>>>                             It is required for manual
>>>>>>>>>>         execution as well,
>>>>>>>>>>                             since test rigs description is
>>>>>>>>>>         essential. I'll
>>>>>>>>>>                             return to the topic below.
>>>>>>>>>>
>>>>>>>>>>                             2. Setup logs storage for
>>>>>>>>>>         automated runs.
>>>>>>>>>>                             Basically it is a disk space plus
>>>>>>>>>>         apache2 web
>>>>>>>>>>                             server with few CGI scripts which
>>>>>>>>>>         help a lot to
>>>>>>>>>>                             save disk space.
>>>>>>>>>>
>>>>>>>>>>                             3. Setup Bublik web application
>>>>>>>>>>         which provides
>>>>>>>>>>                             web interface to view testing
>>>>>>>>>>         results. Same as
>>>>>>>>>>         https://ts-factory.io/bublik
>>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>>         <https://ts-factory.io/bublik>
>>>>>>>>>>
>>>>>>>>>>                             4. Setup Jenkins to run tests on
>>>>>>>>>>         regularly,
>>>>>>>>>>                             save logs in log storage (2) and
>>>>>>>>>>         import it to
>>>>>>>>>>                             bublik (3).
>>>>>>>>>>
>>>>>>>>>>                             Last few month we spent on our
>>>>>>>>>>         homework to make
>>>>>>>>>>                             it simpler to bring up automated
>>>>>>>>>>         execution
>>>>>>>>>>                             using Jenkins -
>>>>>>>>>>         https://github.com/ts-factory/te-jenkins
>>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>         <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>                             Corresponding bits in
>>>>>>>>>>         dpdk-ethdev-ts will be
>>>>>>>>>>                             available tomorrow.
>>>>>>>>>>
>>>>>>>>>>                             Let's return to the step (1).
>>>>>>>>>>
>>>>>>>>>>                             Unfortunately there is no
>>>>>>>>>>         publicly available
>>>>>>>>>>                             example of the ts-rigs repository
>>>>>>>>>>         since
>>>>>>>>>>                             sensitive site-specific
>>>>>>>>>>         information is located
>>>>>>>>>>                             there. But I'm ready to help you
>>>>>>>>>>         to create it
>>>>>>>>>>                             for UNH. I see two options here:
>>>>>>>>>>
>>>>>>>>>>                             (A) I'll ask questions and based
>>>>>>>>>>         on your
>>>>>>>>>>                             answers will create the first
>>>>>>>>>>         draft with my
>>>>>>>>>>                             comments.
>>>>>>>>>>
>>>>>>>>>>                             (B) I'll make a template/example
>>>>>>>>>>         ts-rigs repo,
>>>>>>>>>>                             publish it and you'll create UNH
>>>>>>>>>>         ts-rigs based
>>>>>>>>>>                             on it.
>>>>>>>>>>
>>>>>>>>>>                             Of course, I'll help to debug and
>>>>>>>>>>         finally bring
>>>>>>>>>>                             it up in any case.
>>>>>>>>>>
>>>>>>>>>>                             (A) is a bit simpler for me and
>>>>>>>>>>         you, but (B) is
>>>>>>>>>>                             a bit more generic and will help
>>>>>>>>>>         other
>>>>>>>>>>                             potential users to bring it up.
>>>>>>>>>>                             We can combine (A)+(B). I.e.
>>>>>>>>>>         start from (A).
>>>>>>>>>>                             What do you think?
>>>>>>>>>>
>>>>>>>>>>                             Thanks,
>>>>>>>>>>                             Andrew.
>>>>>>>>>>
>>>>>>>>>>                             On 8/17/23 15:18, Konstantin
>>>>>>>>>>         Ushakov wrote:
>>>>>>>>>>>         Greetings Adam,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             Thanks for contacting us. I copy
>>>>>>>>>>>         Andrew who
>>>>>>>>>>>                             would be happy to help
>>>>>>>>>>>
>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>                             Konstantin
>>>>>>>>>>>
>>>>>>>>>>>>         On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>         <ahassick@iol.unh.edu> <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                             
>>>>>>>>>>>>                             Greetings Konstantin,
>>>>>>>>>>>>
>>>>>>>>>>>>                             I am in the process of setting
>>>>>>>>>>>>         up the DPDK
>>>>>>>>>>>>                             Poll Mode Driver test suite as
>>>>>>>>>>>>         an addition to
>>>>>>>>>>>>                             our testing coverage for DPDK
>>>>>>>>>>>>         at the UNH lab.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I have some questions about how
>>>>>>>>>>>>         to set the
>>>>>>>>>>>>                             test suite arguments.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I have been able to configure
>>>>>>>>>>>>         the Test Engine
>>>>>>>>>>>>                             to connect to the hosts in the
>>>>>>>>>>>>         testbed. The
>>>>>>>>>>>>                             RCF, Configurator, and Tester
>>>>>>>>>>>>         all begin to
>>>>>>>>>>>>                             run, however the prelude of the
>>>>>>>>>>>>         test suite
>>>>>>>>>>>>                             fails to run.
>>>>>>>>>>>>
>>>>>>>>>>>>         https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>         <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                             The documentation mentions that
>>>>>>>>>>>>         there are
>>>>>>>>>>>>                             several test parameters for the
>>>>>>>>>>>>         test suite,
>>>>>>>>>>>>                             like for the IUT test link MAC,
>>>>>>>>>>>>         etc. These
>>>>>>>>>>>>                             seem like they would need to be
>>>>>>>>>>>>         set somewhere
>>>>>>>>>>>>                             to run many of the tests.
>>>>>>>>>>>>
>>>>>>>>>>>>                             I see in the Test Engine
>>>>>>>>>>>>         documentation, there
>>>>>>>>>>>>                             are instructions on how to
>>>>>>>>>>>>         create new
>>>>>>>>>>>>                             parameters for test suites in
>>>>>>>>>>>>         the Tester
>>>>>>>>>>>>         configuration, but there is nothing in the
>>>>>>>>>>>>                             user guide or in the Tester
>>>>>>>>>>>>         guide for how to
>>>>>>>>>>>>                             set the arguments for the
>>>>>>>>>>>>         parameters when
>>>>>>>>>>>>                             running the test suite that I
>>>>>>>>>>>>         can find. I'm
>>>>>>>>>>>>                             not sure if I need to write my
>>>>>>>>>>>>         own Tester
>>>>>>>>>>>>                             config, or if I should be
>>>>>>>>>>>>         setting these in
>>>>>>>>>>>>                             some other way.
>>>>>>>>>>>>
>>>>>>>>>>>>                             How should these values be set?
>>>>>>>>>>>>
>>>>>>>>>>>>                             I'm also not sure what environment
>>>>>>>>>>>>         variables/arguments are strictly necessary or
>>>>>>>>>>>>                             which are optional.
>>>>>>>>>>>>
>>>>>>>>>>>>                             Regards,
>>>>>>>>>>>>                             Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                             --                     *Adam
>>>>>>>>>>>>         Hassick*
>>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>>         ahassick@iol.unh.edu
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                     --             *Adam Hassick*
>>>>>>>>                     Senior Developer
>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 --         *Adam Hassick*
>>>>>>>>                 Senior Developer
>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>                 +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             --     *Adam Hassick*
>>>>>>>             Senior Developer
>>>>>>>             UNH InterOperability Lab
>>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>             +1 (603) 475-8248
>>>>>>
>>>>>>
>>>>>>
>>>>>>         -- 
>>>>>>         *Adam Hassick*
>>>>>>         Senior Developer
>>>>>>         UNH InterOperability Lab
>>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>         <mailto:ahassick@iol.unh.edu>
>>>>>>         iol.unh.edu <http://iol.unh.edu>
>>>>>>         <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>         +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>     -- 
>>>>     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu
>>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>>
>>
>>
>>
>> -- 
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 68596 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-29 12:07                                     ` Andrew Rybchenko
@ 2023-08-29 14:02                                       ` Adam Hassick
  2023-08-29 20:43                                         ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-29 14:02 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 28452 bytes --]

Hi Andrew,

That fix seems to have resolved the issue, thanks for the quick turnaround
time on that patch.
Now that we have the RCF timeout issue resolved, there are a few other
questions and issues that we have about the tests themselves.

1. The test suite fails to build with a couple warnings.

Below is the stderr log from compilation:

FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ 'lib/76b5a35@
> @ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
> -o 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
> ../../lib/dpdk_pmd_ts.c
> ../../lib/dpdk_pmd_ts.c: In function
> ‘test_create_traffic_generator_params’:
> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
> format arguments [-Werror=format-security]
> 5577 |     rc = te_kvpair_add(result, buf, mode);
> |     ^~
> cc1: all warnings being treated as errors
> ninja: build stopped: subcommand failed.
> ninja: Entering directory `.'
> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ 'lib/76b5a35@
> @ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d'
> -o 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
> ../../lib/dpdk_pmd_ts.c
> ../../lib/dpdk_pmd_ts.c: In function
> ‘test_create_traffic_generator_params’:
> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
> format arguments [-Werror=format-security]
> 5577 |     rc = te_kvpair_add(result, buf, mode);
> |     ^~
> cc1: all warnings being treated as errors
>

This error wasn't occurring last week, which was the last time I ran the
tests.
The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC
v11.4.0 installed, if this information is helpful.

2. On the Mellanox CX5s, there are over 6,000 tests run, which collectively
take around 9 hours. Is it possible, and would it make sense, to lower the
test coverage and have the test suite run faster?

For some context, we run immediate testing on incoming patches for DPDK
main and development branches, as well as periodic test runs on the main,
stable, and LTS branches.
For us to consider including this test suite as part of our immediate
testing on patches, we would have to reduce the test coverage to the most
important tests.
This is primarily to reduce the testing time to, for example, less than 30
minutes. Testing on patches can't take too long because the lab can receive
numerous patches each day, which each require individual testing runs.

At what frequency we run these tests, and on what, still needs to be
discussed with the DPDK community, but it would be nice to know if the test
suite had a "dial" to control the testing coverage.

3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300 of
~6,600 tests passed. Is there anything we can do to diagnose these test
failures?

Thanks,
Adam


On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> I've pushed the fix in main branch and a new tag v1.18.1. It should solve
> the problem with IPv6 address from DNS.
>
> Andrew.
>
> On 8/29/23 00:05, Andrew Rybchenko wrote:
>
> Hi Adam,
>
> > Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
> connection to the test bed hosts? And if so, is there a way to force it to
> use IPv4?
>
> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your case,
> I guess it is the root cause of the problem.
> Of course, it is TE problem since I see really weird code in
> lib/comm_net_engine/comm_net_engine.c line 135.
>
> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
> ts-factory/test-environment repository. Please, try.
>
> It is late night fix with minimal testing and no review. I'll pass it
> through review process tomorrow and
> hopefully it will be released in one-two days.
>
> Andrew.
>
> On 8/28/23 18:02, Adam Hassick wrote:
>
> Hi Andrew,
>
> We have yet to notice a distinct pattern with the failures. Sometimes, the
> RCF will start and connect without issue a few times in a row before
> failing to connect again. Once the issue begins to occur, neither rebooting
> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
> directories (suites, agents, inst) and rebooting the hosts afterward
> resolves the issue. When it begins working again seems very arbitrary to us.
>
> I do usually try to terminate the test engine with Ctrl+C, but when it
> hangs while trying to start RCF, that does not work.
>
> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
> connection to the test bed hosts? And if so, is there a way to force it to
> use IPv4?
>
>  - Adam
>
> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>> Done. It works fine for me without any issues.
>>
>> Have you noticed any pattern when it works or does not work?
>> May be it is a problem of not clean state after termination?
>> Does it work fine the first time after DUTs reboot?
>> How do you terminate testing? It should be done using Ctrl+C in terminal
>> where you execute run.sh command.
>>  In this case it should shutdown gracefully and close all test agents and
>> engine applications.
>>
>> (I'm trying to understand why you've seen many test agent processes. It
>> should not happen.)
>>
>> Andrew.
>>
>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>
>> On 8/25/23 17:06, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> Two of our systems (the Test Engine runner and the DUT host) are running
>> Ubuntu 20.04 LTS, however this morning I noticed that the tester system
>> (the one having issues) is running Ubuntu 22.04 LTS.
>> This could be the source of the problem. I encountered a dependency issue
>> trying to run the Test Engine on 22.04 LTS, so I downgraded the system.
>> Since the tester is also the host having connection issues, I will try
>> downgrading that system to 20.04, and see if that changes anything.
>>
>>
>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>> Right now Debian 11 is used for test engine in nightly regressions.
>>
>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>
>> I did try passing in the "--vg-rcf" argument to the run.sh script of the
>> test suite after installing valgrind, but there was no additional output
>> that I saw.
>>
>>
>> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
>> where you run test engine).
>>
>>
>> I will try pulling in the changes you've pushed up, and will see if that
>> fixes anything.
>>
>> Thanks,
>> Adam
>>
>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hello Adam,
>>>
>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>
>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>> running on test engine and agents?
>>>
>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>> patch in review to fix it, however, the behaviour is different in
>>> this case, so it is unlike the same problem.
>>>
>>>
>>> I've just published a new tag which fixes known test engine side
>>> problems on Debian 12 and Fedora 38.
>>>
>>>
>>> One more idea is to install valgrind on the test engine host and
>>> run with option --vg-rcf to check if something weird is happening.
>>>
>>> What I don't understand right now is why I see just one failed attempt
>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>
>>> Andrew.
>>>
>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>
>>>  > Is there any firewall in the network or on test hosts which could
>>> block incoming TCP connection to the port 23571
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>>> you run test engine?
>>>
>>> Our test engine host and the testbed are on the same subnet. The
>>> connection does work sometimes.
>>>
>>>  > If behaviour the same on the next try and you see that test agent is
>>> kept running, could you check using
>>>  >
>>>  > # netstat -tnlp
>>>  >
>>>  > that Test Agent is listening on the port and try to establish TCP
>>> connection from test agent using
>>>  >
>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>  >
>>>  > and check if TCP connection could be established.
>>>
>>> I was able to replicate the same behavior again, where it hangs while
>>> RCF is trying to start.
>>> Running this command, I see this in the output:
>>>
>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>  18599/ta
>>>
>>> So it seems like it is listening on the correct port.
>>> Additionally, I was able to connect to the Tester machine from our Test
>>> Engine host using telnet. It printed the PID of the process once the
>>> connection was opened.
>>>
>>> I tried running the "ta" application manually on the command line, and
>>> it didn't print anything at all.
>>> Maybe the issue is something on the Test Engine side.
>>>
>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>     Hi Adam,
>>>
>>>      > On the tester host (which appears to be the Peer agent), there
>>>     are four processes that I see running, which look like the test
>>>     agent processes.
>>>
>>>     Before the next try I'd recommend to kill these processes.
>>>
>>>     Is there any firewall in the network or on test hosts which could
>>>     block incoming TCP connection to the port 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>     where you run test engine?
>>>
>>>     If behaviour the same on the next try and you see that test agent is
>>>     kept running, could you check using
>>>
>>>     # netstat -tnlp
>>>
>>>     that Test Agent is listening on the port and try to establish TCP
>>>     connection from test agent using
>>>
>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>     and check if TCP connection could be established.
>>>
>>>     Another idea is to login Tester under root as testing does, get
>>>     start TA command from the log and try it by hands without -n and
>>>     remove extra escaping.
>>>
>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>
>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>
>>>     Hopefully in this case test agent directory remains in the /tmp and
>>>     you don't need to copy it as testing does.
>>>     May be output could shed some light on what's going on.
>>>
>>>     Andrew.
>>>
>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>
>>>     Hi Andrew,
>>>
>>>     This is the output that I see in the terminal when this failure
>>>     occurs, after the test agent binaries build and the test engine
>>>     starts:
>>>
>>>     Platform default build - pass
>>>     Simple RCF consistency check succeeded
>>>     --->>> Starting Logger...done
>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>
>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>     processes. I let it hang for around 9 minutes.
>>>
>>>     On the tester host (which appears to be the Peer agent), there are
>>>     four processes that I see running, which look like the test agent
>>>     processes.
>>>
>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>
>>>      - Adam
>>>
>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru
>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>         Hi Adam,
>>>
>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>         says that copy_timeout is in seconds and implementation in
>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>         Theoretically select() could be interrupted by signal, but I
>>>         think it is unlikely here.
>>>
>>>         I'm not sure that I understand what do you mean by RCF
>>>         connection timeout. Does it happen on TE startup when RCF
>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>         does it happen when tests are in progress, e.g. in the middle
>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>         likely either host with test agent dies or test agent itself
>>>         crashes. It would be easier for me if classify it if you share
>>>         text log (log.txt, full or just corresponding fragment with
>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>         depending on which agent has problems could shed some light.
>>>         Corresponding files contain stdout/stderr of test agents.
>>>
>>>         Andrew.
>>>
>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>
>>>         Hi Andrew,
>>>
>>>         I've set up a test rig repository here, and have created
>>>         configurations for our development testbed based off of the
>>>         examples.
>>>         We've been able to get the test suite to run manually on
>>>         Mellanox CX5 devices once.
>>>         However, we are running into an issue where, when RCF starts,
>>>         the RCF connection times out very frequently. We aren't sure
>>>         why this is the case.
>>>         It works sometimes, but most of the time when we try to run
>>>         the test engine, it encounters this issue.
>>>         I've tried changing the RCF port by setting
>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>         machines. Neither seems to fix the issue.
>>>
>>>         It also seems like the timeout takes far longer than 60
>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>         before I try to run the test suite.
>>>         I assume the unit for this variable is seconds?
>>>
>>>         Thanks,
>>>         Adam
>>>
>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>> wrote:
>>>
>>>             Hi Andrew,
>>>
>>>             Thanks, I've cloned the example repository and will start
>>>             setting up a configuration for our development testbed
>>>             today. I'll let you know if I run into any difficulties
>>>             or have any questions.
>>>
>>>              - Adam
>>>
>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>             <andrew.rybchenko@oktetlabs.ru
>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>                 Hi Adam,
>>>
>>>                 I've published
>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>                 Hopefully it will help to define your test rigs and
>>>                 successfully run some tests manually. Feel free to
>>>                 ask any questions and I'll answer here and try to
>>>                 update documentation.
>>>
>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>                 (3).
>>>                 Hopefully everything is in place for step (4), but we
>>>                 need to make steps (2) and (3) first.
>>>
>>>                 Andrew.
>>>
>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>
>>>                 Hi Adam,
>>>
>>>                 > I've conferred with the rest of the team, and we
>>>                 think it would be best to move forward with mainly
>>>                 option B.
>>>
>>>                 OK, I'll provide the sample on Monday for you. It is
>>>                 almost ready right now, but I need to double-check
>>>                 it before publishing.
>>>
>>>                 Regards,
>>>                 Andrew.
>>>
>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>
>>>                 Hi Andrew,
>>>
>>>                 I'm adding the CI mailing list to this
>>>                 conversation. Others in the community might find
>>>                 this conversation valuable.
>>>
>>>                 We do want to run testing on a regular basis. The
>>>                 Jenkins integration will be very useful for us, as
>>>                 most of our CI is orchestrated by Jenkins.
>>>                 I've conferred with the rest of the team, and we
>>>                 think it would be best to move forward with mainly
>>>                 option B.
>>>                 If you would like to know anything about our
>>>                 testbeds that would help you with creating an
>>>                 example ts-rigs repo, I'd be happy to answer any
>>>                 questions you have.
>>>
>>>                 We have multiple test rigs (we call these
>>>                 "DUT-tester pairs") that we run our existing
>>>                 hardware testing on, with differing network
>>>                 hardware and CPU architecture. I figured this might
>>>                 be an important detail.
>>>
>>>                 Thanks,
>>>                 Adam
>>>
>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>                 <andrew.rybchenko@oktetlabs.ru
>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>
>>>                     Greatings Adam,
>>>
>>>                     I'm happy to hear that you're trying to bring
>>>                     it up.
>>>
>>>                     As I understand the final goal is to run it on
>>>                     regular basis. So, we need to make it properly
>>>                     from the very beginning.
>>>                     Bring up of all features consists of 4 steps:
>>>
>>>                     1. Create site-specific repository (we call it
>>>                     ts-rigs) which contains information about test
>>>                     rigs and other site-specific information like
>>>                     where to send mails, where to store logs etc.
>>>                     It is required for manual execution as well,
>>>                     since test rigs description is essential. I'll
>>>                     return to the topic below.
>>>
>>>                     2. Setup logs storage for automated runs.
>>>                     Basically it is a disk space plus apache2 web
>>>                     server with few CGI scripts which help a lot to
>>>                     save disk space.
>>>
>>>                     3. Setup Bublik web application which provides
>>>                     web interface to view testing results. Same as
>>>                     https://ts-factory.io/bublik
>>>                     <https://ts-factory.io/bublik>
>>> <https://ts-factory.io/bublik>
>>>
>>>                     4. Setup Jenkins to run tests on regularly,
>>>                     save logs in log storage (2) and import it to
>>>                     bublik (3).
>>>
>>>                     Last few month we spent on our homework to make
>>>                     it simpler to bring up automated execution
>>>                     using Jenkins -
>>>                     https://github.com/ts-factory/te-jenkins
>>>                     <https://github.com/ts-factory/te-jenkins>
>>> <https://github.com/ts-factory/te-jenkins>
>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>                     available tomorrow.
>>>
>>>                     Let's return to the step (1).
>>>
>>>                     Unfortunately there is no publicly available
>>>                     example of the ts-rigs repository since
>>>                     sensitive site-specific information is located
>>>                     there. But I'm ready to help you to create it
>>>                     for UNH. I see two options here:
>>>
>>>                     (A) I'll ask questions and based on your
>>>                     answers will create the first draft with my
>>>                     comments.
>>>
>>>                     (B) I'll make a template/example ts-rigs repo,
>>>                     publish it and you'll create UNH ts-rigs based
>>>                     on it.
>>>
>>>                     Of course, I'll help to debug and finally bring
>>>                     it up in any case.
>>>
>>>                     (A) is a bit simpler for me and you, but (B) is
>>>                     a bit more generic and will help other
>>>                     potential users to bring it up.
>>>                     We can combine (A)+(B). I.e. start from (A).
>>>                     What do you think?
>>>
>>>                     Thanks,
>>>                     Andrew.
>>>
>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>
>>>                     Greetings Adam,
>>>
>>>
>>>                     Thanks for contacting us. I copy Andrew who
>>>                     would be happy to help
>>>
>>>                     Thanks,
>>>                     Konstantin
>>>
>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>> wrote:
>>>
>>>                     
>>>                     Greetings Konstantin,
>>>
>>>                     I am in the process of setting up the DPDK
>>>                     Poll Mode Driver test suite as an addition to
>>>                     our testing coverage for DPDK at the UNH lab.
>>>
>>>                     I have some questions about how to set the
>>>                     test suite arguments.
>>>
>>>                     I have been able to configure the Test Engine
>>>                     to connect to the hosts in the testbed. The
>>>                     RCF, Configurator, and Tester all begin to
>>>                     run, however the prelude of the test suite
>>>                     fails to run.
>>>
>>>
>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>
>>>                     The documentation mentions that there are
>>>                     several test parameters for the test suite,
>>>                     like for the IUT test link MAC, etc. These
>>>                     seem like they would need to be set somewhere
>>>                     to run many of the tests.
>>>
>>>                     I see in the Test Engine documentation, there
>>>                     are instructions on how to create new
>>>                     parameters for test suites in the Tester
>>>                     configuration, but there is nothing in the
>>>                     user guide or in the Tester guide for how to
>>>                     set the arguments for the parameters when
>>>                     running the test suite that I can find. I'm
>>>                     not sure if I need to write my own Tester
>>>                     config, or if I should be setting these in
>>>                     some other way.
>>>
>>>                     How should these values be set?
>>>
>>>                     I'm also not sure what environment
>>>                     variables/arguments are strictly necessary or
>>>                     which are optional.
>>>
>>>                     Regards,
>>>                     Adam
>>>
>>>                     --                     *Adam Hassick*
>>>                     Senior Developer
>>>                     UNH InterOperability Lab
>>>                     ahassick@iol.unh.edu
>>>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>                     +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>>                 --                 *Adam Hassick*
>>>                 Senior Developer
>>>                 UNH InterOperability Lab
>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>                 +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>>
>>>             --             *Adam Hassick*
>>>             Senior Developer
>>>             UNH InterOperability Lab
>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>             +1 (603) 475-8248
>>>
>>>
>>>
>>>         --         *Adam Hassick*
>>>         Senior Developer
>>>         UNH InterOperability Lab
>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>> <https://www.iol.unh.edu/>
>>>         +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>>     --     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>> <ahassick@iol.unh.edu>
>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 67404 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-29 14:02                                       ` Adam Hassick
@ 2023-08-29 20:43                                         ` Andrew Rybchenko
  2023-08-31 19:38                                           ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-08-29 20:43 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 43784 bytes --]

1. We'll sort out warnings this week. Thanks for heads up.

2. You can add command-line option --sanity to run tests marked with 
TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and 
grep TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). 
Yes, there is a space for terminology improvement here. We'll do it.

Also it takes a lot of time because of failures and tests which wait for 
some timeout.

Support for old LTS branches was dropped some time ago, but in the 
future it is definitely possible to keep it for new LTS branches. I 
think 22.11 is supported, but I'm not sure about older LTS releases.

The test harness can provide coverage reports based on gcov, but I'm not 
sure what you mean by a "dial" to control test coverage. Provided 
reports are rather for human to analyze.

3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing 
results. Unfortunately it is time consuming to fill in expectations 
database since it is necessary to analyze testing results and classify 
if it is a bug or just acceptable behaviour aspect.

Bublik allows to compare results of two runs. It is useful for human, 
but still not good for automation.

I have local patch for mlx5 driver which reports Tx ring size maximum. 
It makes pass rate higher. It is a problem for test harness that mlx5 
does not report limits right now.

Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio 
net is 99% right now and could be done 100% easily (just one thing to 
fix in expectations).

I think logs storage setup is essential for logs analysis. Of course, 
you can request HTML logs when you run tests (--log-html=html) or 
generate after run using dpdk-ethdev-ts/scripts/html-log.sh and open 
index.html in a browser, but logs storage makes it more convenient.

Latest version of test-environment has examples of our CGI scripts which 
we use for log storage (see tools/log_server/README.md).

Also all bits for Jenkins setup are available. See 
dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in 
ts-rigs-sample.

Of course, bublik makes logs analysis even more convenient, but it uses 
logs storage anyway.

Andrew.

On 8/29/23 17:02, Adam Hassick wrote:
> Hi Andrew,
>
> That fix seems to have resolved the issue, thanks for the quick 
> turnaround time on that patch.
> Now that we have the RCF timeout issue resolved, there are a few other 
> questions and issues that we have about the tests themselves.
>
> 1. The test suite fails to build with a couple warnings.
>
> Below is the stderr log from compilation:
>
>     FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>     -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>     -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC
>     -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>     ../../lib/dpdk_pmd_ts.c
>     ../../lib/dpdk_pmd_ts.c: In function
>     ‘test_create_traffic_generator_params’:
>     ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>     and no format arguments [-Werror=format-security]
>     5577 |     rc = te_kvpair_add(result, buf, mode);
>     |     ^~
>     cc1: all warnings being treated as errors
>     ninja: build stopped: subcommand failed.
>     ninja: Entering directory `.'
>     FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>     cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>     -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>     -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>     -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC
>     -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>     'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>     ../../lib/dpdk_pmd_ts.c
>     ../../lib/dpdk_pmd_ts.c: In function
>     ‘test_create_traffic_generator_params’:
>     ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal
>     and no format arguments [-Werror=format-security]
>     5577 |     rc = te_kvpair_add(result, buf, mode);
>     |     ^~
>     cc1: all warnings being treated as errors
>
>
> This error wasn't occurring last week, which was the last time I ran 
> the tests.
> The TE host and the DUT have GCC v9.4.0 installed, and the tester has 
> GCC v11.4.0 installed, if this information is helpful.
>
> 2. On the Mellanox CX5s, there are over 6,000 tests run, which 
> collectively take around 9 hours. Is it possible, and would it make 
> sense, to lower the test coverage and have the test suite run faster?
>
> For some context, we run immediate testing on incoming patches for 
> DPDK main and development branches, as well as periodic test runs on 
> the main, stable, and LTS branches.
> For us to consider including this test suite as part of our immediate 
> testing on patches, we would have to reduce the test coverage to the 
> most important tests.
> This is primarily to reduce the testing time to, for example, less 
> than 30 minutes. Testing on patches can't take too long because the 
> lab can receive numerous patches each day, which each require 
> individual testing runs.
>
> At what frequency we run these tests, and on what, still needs to be 
> discussed with the DPDK community, but it would be nice to know if the 
> test suite had a "dial" to control the testing coverage.
>
> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 
> 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose 
> these test failures?
>
> Thanks,
> Adam
>
>
> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko 
> <andrew.rybchenko@oktetlabs.ru> wrote:
>
>     Hi Adam,
>
>     I've pushed the fix in main branch and a new tag v1.18.1. It
>     should solve the problem with IPv6 address from DNS.
>
>     Andrew.
>
>     On 8/29/23 00:05, Andrew Rybchenko wrote:
>>     Hi Adam,
>>
>>     > Does the test engine prefer to use IPv6 over IPv4 for
>>     initiating the RCF connection to the test bed hosts? And if so,
>>     is there a way to force it to use IPv4?
>>
>>     Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in
>>     your case, I guess it is the root cause of the problem.
>>     Of course, it is TE problem since I see really weird code in
>>     lib/comm_net_engine/comm_net_engine.c line 135.
>>
>>     I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>     ts-factory/test-environment repository. Please, try.
>>
>>     It is late night fix with minimal testing and no review. I'll
>>     pass it through review process tomorrow and
>>     hopefully it will be released in one-two days.
>>
>>     Andrew.
>>
>>     On 8/28/23 18:02, Adam Hassick wrote:
>>>     Hi Andrew,
>>>
>>>     We have yet to notice a distinct pattern with the failures.
>>>     Sometimes, the RCF will start and connect without issue a few
>>>     times in a row before failing to connect again. Once the issue
>>>     begins to occur, neither rebooting all of the hosts (test engine
>>>     VM, tester, IUT) or deleting all of the build directories
>>>     (suites, agents, inst) and rebooting the hosts afterward
>>>     resolves the issue. When it begins working again seems very
>>>     arbitrary to us.
>>>
>>>     I do usually try to terminate the test engine with Ctrl+C, but
>>>     when it hangs while trying to start RCF, that does not work.
>>>
>>>     Does the test engine prefer to use IPv6 over IPv4 for initiating
>>>     the RCF connection to the test bed hosts? And if so, is there a
>>>     way to force it to use IPv4?
>>>
>>>      - Adam
>>>
>>>     On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko
>>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>         > I'll double-check test engine on Ubuntu 20.04 and Ubuntu
>>>         22.04.
>>>
>>>         Done. It works fine for me without any issues.
>>>
>>>         Have you noticed any pattern when it works or does not work?
>>>         May be it is a problem of not clean state after termination?
>>>         Does it work fine the first time after DUTs reboot?
>>>         How do you terminate testing? It should be done using Ctrl+C
>>>         in terminal where you execute run.sh command.
>>>          In this case it should shutdown gracefully and close all
>>>         test agents and engine applications.
>>>
>>>         (I'm trying to understand why you've seen many test agent
>>>         processes. It should not happen.)
>>>
>>>         Andrew.
>>>
>>>         On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>         On 8/25/23 17:06, Adam Hassick wrote:
>>>>>         Hi Andrew,
>>>>>
>>>>>         Two of our systems (the Test Engine runner and the DUT
>>>>>         host) are running Ubuntu 20.04 LTS, however this morning I
>>>>>         noticed that the tester system (the one having issues) is
>>>>>         running Ubuntu 22.04 LTS.
>>>>>         This could be the source of the problem. I encountered a
>>>>>         dependency issue trying to run the Test Engine on 22.04
>>>>>         LTS, so I downgraded the system. Since the tester is also
>>>>>         the host having connection issues, I will try downgrading
>>>>>         that system to 20.04, and see if that changes anything.
>>>>
>>>>         Unlikely, but who knows. We run tests (DUTs) on Ubuntu
>>>>         20.04, Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11
>>>>         and Fedora 38 every night.
>>>>         Right now Debian 11 is used for test engine in nightly
>>>>         regressions.
>>>>
>>>>         I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>
>>>>>         I did try passing in the "--vg-rcf" argument to the run.sh
>>>>>         script of the test suite after installing valgrind, but
>>>>>         there was no additional output that I saw.
>>>>
>>>>         Sorry, I should valgrind output should be in
>>>>         valgrind.te_rcf (direction where you run test engine).
>>>>
>>>>>
>>>>>         I will try pulling in the changes you've pushed up, and
>>>>>         will see if that fixes anything.
>>>>>
>>>>>         Thanks,
>>>>>         Adam
>>>>>
>>>>>         On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>
>>>>>             Hello Adam,
>>>>>
>>>>>             On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>             I'd like to try to repeat the problem locally. Which
>>>>>>             Linux distro is running on test engine and agents?
>>>>>>
>>>>>>             In fact I know one problem with Debian 12 and Fedora
>>>>>>             38 and we have
>>>>>>             patch in review to fix it, however, the behaviour is
>>>>>>             different in
>>>>>>             this case, so it is unlike the same problem.
>>>>>
>>>>>             I've just published a new tag which fixes known test
>>>>>             engine side problems on Debian 12 and Fedora 38.
>>>>>
>>>>>>
>>>>>>             One more idea is to install valgrind on the test
>>>>>>             engine host and
>>>>>>             run with option --vg-rcf to check if something weird
>>>>>>             is happening.
>>>>>>
>>>>>>             What I don't understand right now is why I see just
>>>>>>             one failed attempt
>>>>>>             to connect in your log.txt and then Logger shutdown
>>>>>>             after 9 minutes.
>>>>>>
>>>>>>             Andrew.
>>>>>>
>>>>>>             On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>              > Is there any firewall in the network or on test
>>>>>>>             hosts which could block incoming TCP connection to
>>>>>>>             the port 23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             from the host where you run test engine?
>>>>>>>
>>>>>>>             Our test engine host and the testbed are on the same
>>>>>>>             subnet. The connection does work sometimes.
>>>>>>>
>>>>>>>              > If behaviour the same on the next try and you see
>>>>>>>             that test agent is kept running, could you check using
>>>>>>>              >
>>>>>>>              > # netstat -tnlp
>>>>>>>              >
>>>>>>>              > that Test Agent is listening on the port and try
>>>>>>>             to establish TCP connection from test agent using
>>>>>>>              >
>>>>>>>              > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>              >
>>>>>>>              > and check if TCP connection could be established.
>>>>>>>
>>>>>>>             I was able to replicate the same behavior again,
>>>>>>>             where it hangs while RCF is trying to start.
>>>>>>>             Running this command, I see this in the output:
>>>>>>>
>>>>>>>             tcp        0      0 0.0.0.0:23571
>>>>>>>             <http://0.0.0.0:23571> <http://0.0.0.0:23571>
>>>>>>>             <http://0.0.0.0:23571> 0.0.0.0:*              
>>>>>>>             LISTEN  18599/ta
>>>>>>>
>>>>>>>             So it seems like it is listening on the correct port.
>>>>>>>             Additionally, I was able to connect to the Tester
>>>>>>>             machine from our Test Engine host using telnet. It
>>>>>>>             printed the PID of the process once the connection
>>>>>>>             was opened.
>>>>>>>
>>>>>>>             I tried running the "ta" application manually on the
>>>>>>>             command line, and it didn't print anything at all.
>>>>>>>             Maybe the issue is something on the Test Engine side.
>>>>>>>
>>>>>>>             On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko
>>>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>
>>>>>>>                 Hi Adam,
>>>>>>>
>>>>>>>                  > On the tester host (which appears to be the
>>>>>>>             Peer agent), there
>>>>>>>                 are four processes that I see running, which
>>>>>>>             look like the test
>>>>>>>                 agent processes.
>>>>>>>
>>>>>>>                 Before the next try I'd recommend to kill these
>>>>>>>             processes.
>>>>>>>
>>>>>>>                 Is there any firewall in the network or on test
>>>>>>>             hosts which could
>>>>>>>                 block incoming TCP connection to the port 23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             from the host
>>>>>>>                 where you run test engine?
>>>>>>>
>>>>>>>                 If behaviour the same on the next try and you
>>>>>>>             see that test agent is
>>>>>>>                 kept running, could you check using
>>>>>>>
>>>>>>>                 # netstat -tnlp
>>>>>>>
>>>>>>>                 that Test Agent is listening on the port and try
>>>>>>>             to establish TCP
>>>>>>>                 connection from test agent using
>>>>>>>
>>>>>>>                 $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>
>>>>>>>                 and check if TCP connection could be established.
>>>>>>>
>>>>>>>                 Another idea is to login Tester under root as
>>>>>>>             testing does, get
>>>>>>>                 start TA command from the log and try it by
>>>>>>>             hands without -n and
>>>>>>>                 remove extra escaping.
>>>>>>>
>>>>>>>                 # sudo
>>>>>>>             PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>             LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>             /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>>>>             host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>
>>>>>>>                 Hopefully in this case test agent directory
>>>>>>>             remains in the /tmp and
>>>>>>>                 you don't need to copy it as testing does.
>>>>>>>                 May be output could shed some light on what's
>>>>>>>             going on.
>>>>>>>
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>                 Hi Andrew,
>>>>>>>>
>>>>>>>>                 This is the output that I see in the terminal
>>>>>>>>             when this failure
>>>>>>>>                 occurs, after the test agent binaries build and
>>>>>>>>             the test engine
>>>>>>>>                 starts:
>>>>>>>>
>>>>>>>>                 Platform default build - pass
>>>>>>>>                 Simple RCF consistency check succeeded
>>>>>>>>                 --->>> Starting Logger...done
>>>>>>>>                 --->>> Starting RCF...rcf_net_engine_connect():
>>>>>>>>             Connection timed
>>>>>>>>                 out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>             <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>
>>>>>>>>                 Then, it hangs here until I kill the "te_rcf"
>>>>>>>>             and "te_tee"
>>>>>>>>                 processes. I let it hang for around 9 minutes.
>>>>>>>>
>>>>>>>>                 On the tester host (which appears to be the
>>>>>>>>             Peer agent), there are
>>>>>>>>                 four processes that I see running, which look
>>>>>>>>             like the test agent
>>>>>>>>                 processes.
>>>>>>>>
>>>>>>>>                 ta.Peer is an empty file. I've attached the
>>>>>>>>             log.txt from this run.
>>>>>>>>
>>>>>>>>                  - Adam
>>>>>>>>
>>>>>>>>                 On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                     Hi Adam,
>>>>>>>>
>>>>>>>>                     Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
>>>>>>>>             double-checked
>>>>>>>>                     that it goes to 'copy_timeout' in
>>>>>>>>             ts-conf/rcf.conf.
>>>>>>>>                     Description in in
>>>>>>>>             doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>                     says that copy_timeout is in seconds and
>>>>>>>>             implementation in
>>>>>>>>                     lib/rcfunix/rcfunix.c passes the value to
>>>>>>>>             select() tv_sec.
>>>>>>>>                     Theoretically select() could be interrupted
>>>>>>>>             by signal, but I
>>>>>>>>                     think it is unlikely here.
>>>>>>>>
>>>>>>>>                     I'm not sure that I understand what do you
>>>>>>>>             mean by RCF
>>>>>>>>                     connection timeout. Does it happen on TE
>>>>>>>>             startup when RCF
>>>>>>>>                     starts test agents. If so,
>>>>>>>>             TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>>>                     does it happen when tests are in progress,
>>>>>>>>             e.g. in the middle
>>>>>>>>                     of a test. If so, TE_RCFUNIX_TIMEOUT is
>>>>>>>>             unrelated and most
>>>>>>>>                     likely either host with test agent dies or
>>>>>>>>             test agent itself
>>>>>>>>                     crashes. It would be easier for me if
>>>>>>>>             classify it if you share
>>>>>>>>                     text log (log.txt, full or just
>>>>>>>>             corresponding fragment with
>>>>>>>>                     some context). Also content of ta.DPDK or
>>>>>>>>             ta.Peer file
>>>>>>>>                     depending on which agent has problems could
>>>>>>>>             shed some light.
>>>>>>>>                     Corresponding files contain stdout/stderr
>>>>>>>>             of test agents.
>>>>>>>>
>>>>>>>>                     Andrew.
>>>>>>>>
>>>>>>>>                     On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>                     Hi Andrew,
>>>>>>>>>
>>>>>>>>>                     I've set up a test rig repository here,
>>>>>>>>>             and have created
>>>>>>>>>                     configurations for our development testbed
>>>>>>>>>             based off of the
>>>>>>>>>                     examples.
>>>>>>>>>                     We've been able to get the test suite to
>>>>>>>>>             run manually on
>>>>>>>>>                     Mellanox CX5 devices once.
>>>>>>>>>                     However, we are running into an issue
>>>>>>>>>             where, when RCF starts,
>>>>>>>>>                     the RCF connection times out very
>>>>>>>>>             frequently. We aren't sure
>>>>>>>>>                     why this is the case.
>>>>>>>>>                     It works sometimes, but most of the time
>>>>>>>>>             when we try to run
>>>>>>>>>                     the test engine, it encounters this issue.
>>>>>>>>>                     I've tried changing the RCF port by setting
>>>>>>>>>                     "TE_RCF_PORT=<some port number>" and
>>>>>>>>>             rebooting the testbed
>>>>>>>>>                     machines. Neither seems to fix the issue.
>>>>>>>>>
>>>>>>>>>                     It also seems like the timeout takes far
>>>>>>>>>             longer than 60
>>>>>>>>>                     seconds, even when running "export
>>>>>>>>>             TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>                     before I try to run the test suite.
>>>>>>>>>                     I assume the unit for this variable is
>>>>>>>>>             seconds?
>>>>>>>>>
>>>>>>>>>                     Thanks,
>>>>>>>>>                     Adam
>>>>>>>>>
>>>>>>>>>                     On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>>>>>                     <ahassick@iol.unh.edu
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>>>
>>>>>>>>>                         Hi Andrew,
>>>>>>>>>
>>>>>>>>>                         Thanks, I've cloned the example
>>>>>>>>>             repository and will start
>>>>>>>>>                         setting up a configuration for our
>>>>>>>>>             development testbed
>>>>>>>>>                         today. I'll let you know if I run into
>>>>>>>>>             any difficulties
>>>>>>>>>                         or have any questions.
>>>>>>>>>
>>>>>>>>>                          - Adam
>>>>>>>>>
>>>>>>>>>                         On Sun, Aug 20, 2023 at 4:40 AM Andrew
>>>>>>>>>             Rybchenko
>>>>>>>>>                         <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                             Hi Adam,
>>>>>>>>>
>>>>>>>>>                             I've published
>>>>>>>>>             https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>             <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>             <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>                             Hopefully it will help to define
>>>>>>>>>             your test rigs and
>>>>>>>>>                             successfully run some tests
>>>>>>>>>             manually. Feel free to
>>>>>>>>>                             ask any questions and I'll answer
>>>>>>>>>             here and try to
>>>>>>>>>                             update documentation.
>>>>>>>>>
>>>>>>>>>                             Meanwhile I'll prepare missing
>>>>>>>>>             bits for steps (2) and
>>>>>>>>>                             (3).
>>>>>>>>>                             Hopefully everything is in place
>>>>>>>>>             for step (4), but we
>>>>>>>>>                             need to make steps (2) and (3) first.
>>>>>>>>>
>>>>>>>>>                             Andrew.
>>>>>>>>>
>>>>>>>>>                             On 8/18/23 21:40, Andrew Rybchenko
>>>>>>>>>             wrote:
>>>>>>>>>>             Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                             > I've conferred with the rest of
>>>>>>>>>>             the team, and we
>>>>>>>>>>                             think it would be best to move
>>>>>>>>>>             forward with mainly
>>>>>>>>>>                             option B.
>>>>>>>>>>
>>>>>>>>>>                             OK, I'll provide the sample on
>>>>>>>>>>             Monday for you. It is
>>>>>>>>>>                             almost ready right now, but I
>>>>>>>>>>             need to double-check
>>>>>>>>>>                             it before publishing.
>>>>>>>>>>
>>>>>>>>>>                             Regards,
>>>>>>>>>>                             Andrew.
>>>>>>>>>>
>>>>>>>>>>                             On 8/17/23 20:03, Adam Hassick
>>>>>>>>>>             wrote:
>>>>>>>>>>>             Hi Andrew,
>>>>>>>>>>>
>>>>>>>>>>>                             I'm adding the CI mailing list
>>>>>>>>>>>             to this
>>>>>>>>>>>             conversation. Others in the community might find
>>>>>>>>>>>                             this conversation valuable.
>>>>>>>>>>>
>>>>>>>>>>>                             We do want to run testing on a
>>>>>>>>>>>             regular basis. The
>>>>>>>>>>>                             Jenkins integration will be very
>>>>>>>>>>>             useful for us, as
>>>>>>>>>>>                             most of our CI is orchestrated
>>>>>>>>>>>             by Jenkins.
>>>>>>>>>>>                             I've conferred with the rest of
>>>>>>>>>>>             the team, and we
>>>>>>>>>>>                             think it would be best to move
>>>>>>>>>>>             forward with mainly
>>>>>>>>>>>                             option B.
>>>>>>>>>>>                             If you would like to know
>>>>>>>>>>>             anything about our
>>>>>>>>>>>                             testbeds that would help you
>>>>>>>>>>>             with creating an
>>>>>>>>>>>                             example ts-rigs repo, I'd be
>>>>>>>>>>>             happy to answer any
>>>>>>>>>>>                             questions you have.
>>>>>>>>>>>
>>>>>>>>>>>                             We have multiple test rigs (we
>>>>>>>>>>>             call these
>>>>>>>>>>>                             "DUT-tester pairs") that we run
>>>>>>>>>>>             our existing
>>>>>>>>>>>                             hardware testing on, with
>>>>>>>>>>>             differing network
>>>>>>>>>>>                             hardware and CPU architecture. I
>>>>>>>>>>>             figured this might
>>>>>>>>>>>                             be an important detail.
>>>>>>>>>>>
>>>>>>>>>>>                             Thanks,
>>>>>>>>>>>                             Adam
>>>>>>>>>>>
>>>>>>>>>>>                             On Thu, Aug 17, 2023 at 11:44 AM
>>>>>>>>>>>             Andrew Rybchenko
>>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>             Greatings Adam,
>>>>>>>>>>>
>>>>>>>>>>>                                 I'm happy to hear that
>>>>>>>>>>>             you're trying to bring
>>>>>>>>>>>                                 it up.
>>>>>>>>>>>
>>>>>>>>>>>                                 As I understand the final
>>>>>>>>>>>             goal is to run it on
>>>>>>>>>>>                                 regular basis. So, we need
>>>>>>>>>>>             to make it properly
>>>>>>>>>>>                                 from the very beginning.
>>>>>>>>>>>                                 Bring up of all features
>>>>>>>>>>>             consists of 4 steps:
>>>>>>>>>>>
>>>>>>>>>>>                                 1. Create site-specific
>>>>>>>>>>>             repository (we call it
>>>>>>>>>>>                                 ts-rigs) which contains
>>>>>>>>>>>             information about test
>>>>>>>>>>>                                 rigs and other site-specific
>>>>>>>>>>>             information like
>>>>>>>>>>>                                 where to send mails, where
>>>>>>>>>>>             to store logs etc.
>>>>>>>>>>>                                 It is required for manual
>>>>>>>>>>>             execution as well,
>>>>>>>>>>>                                 since test rigs description
>>>>>>>>>>>             is essential. I'll
>>>>>>>>>>>                                 return to the topic below.
>>>>>>>>>>>
>>>>>>>>>>>                                 2. Setup logs storage for
>>>>>>>>>>>             automated runs.
>>>>>>>>>>>             Basically it is a disk space plus apache2 web
>>>>>>>>>>>                                 server with few CGI scripts
>>>>>>>>>>>             which help a lot to
>>>>>>>>>>>                                 save disk space.
>>>>>>>>>>>
>>>>>>>>>>>                                 3. Setup Bublik web
>>>>>>>>>>>             application which provides
>>>>>>>>>>>                                 web interface to view
>>>>>>>>>>>             testing results. Same as
>>>>>>>>>>>             https://ts-factory.io/bublik
>>>>>>>>>>>             <https://ts-factory.io/bublik>
>>>>>>>>>>>             <https://ts-factory.io/bublik>
>>>>>>>>>>>
>>>>>>>>>>>                                 4. Setup Jenkins to run
>>>>>>>>>>>             tests on regularly,
>>>>>>>>>>>                                 save logs in log storage (2)
>>>>>>>>>>>             and import it to
>>>>>>>>>>>                                 bublik (3).
>>>>>>>>>>>
>>>>>>>>>>>                                 Last few month we spent on
>>>>>>>>>>>             our homework to make
>>>>>>>>>>>                                 it simpler to bring up
>>>>>>>>>>>             automated execution
>>>>>>>>>>>                                 using Jenkins -
>>>>>>>>>>>             https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>             <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>             Corresponding bits in dpdk-ethdev-ts will be
>>>>>>>>>>>             available tomorrow.
>>>>>>>>>>>
>>>>>>>>>>>                                 Let's return to the step (1).
>>>>>>>>>>>
>>>>>>>>>>>             Unfortunately there is no publicly available
>>>>>>>>>>>                                 example of the ts-rigs
>>>>>>>>>>>             repository since
>>>>>>>>>>>             sensitive site-specific information is located
>>>>>>>>>>>                                 there. But I'm ready to help
>>>>>>>>>>>             you to create it
>>>>>>>>>>>                                 for UNH. I see two options
>>>>>>>>>>>             here:
>>>>>>>>>>>
>>>>>>>>>>>                                 (A) I'll ask questions and
>>>>>>>>>>>             based on your
>>>>>>>>>>>                                 answers will create the
>>>>>>>>>>>             first draft with my
>>>>>>>>>>>             comments.
>>>>>>>>>>>
>>>>>>>>>>>                                 (B) I'll make a
>>>>>>>>>>>             template/example ts-rigs repo,
>>>>>>>>>>>                                 publish it and you'll create
>>>>>>>>>>>             UNH ts-rigs based
>>>>>>>>>>>                                 on it.
>>>>>>>>>>>
>>>>>>>>>>>                                 Of course, I'll help to
>>>>>>>>>>>             debug and finally bring
>>>>>>>>>>>                                 it up in any case.
>>>>>>>>>>>
>>>>>>>>>>>                                 (A) is a bit simpler for me
>>>>>>>>>>>             and you, but (B) is
>>>>>>>>>>>                                 a bit more generic and will
>>>>>>>>>>>             help other
>>>>>>>>>>>             potential users to bring it up.
>>>>>>>>>>>                                 We can combine (A)+(B). I.e.
>>>>>>>>>>>             start from (A).
>>>>>>>>>>>                                 What do you think?
>>>>>>>>>>>
>>>>>>>>>>>                                 Thanks,
>>>>>>>>>>>                                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                 On 8/17/23 15:18, Konstantin
>>>>>>>>>>>             Ushakov wrote:
>>>>>>>>>>>>             Greetings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                                 Thanks for contacting us. I
>>>>>>>>>>>>             copy Andrew who
>>>>>>>>>>>>                                 would be happy to help
>>>>>>>>>>>>
>>>>>>>>>>>>             Thanks,
>>>>>>>>>>>>             Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>>>             On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>>             <ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 
>>>>>>>>>>>>>             Greetings Konstantin,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I am in the process of
>>>>>>>>>>>>>             setting up the DPDK
>>>>>>>>>>>>>                                 Poll Mode Driver test
>>>>>>>>>>>>>             suite as an addition to
>>>>>>>>>>>>>                                 our testing coverage for
>>>>>>>>>>>>>             DPDK at the UNH lab.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I have some questions
>>>>>>>>>>>>>             about how to set the
>>>>>>>>>>>>>                                 test suite arguments.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I have been able to
>>>>>>>>>>>>>             configure the Test Engine
>>>>>>>>>>>>>                                 to connect to the hosts in
>>>>>>>>>>>>>             the testbed. The
>>>>>>>>>>>>>                                 RCF, Configurator, and
>>>>>>>>>>>>>             Tester all begin to
>>>>>>>>>>>>>                                 run, however the prelude
>>>>>>>>>>>>>             of the test suite
>>>>>>>>>>>>>             fails to run.
>>>>>>>>>>>>>
>>>>>>>>>>>>>             https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>             <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 The documentation mentions
>>>>>>>>>>>>>             that there are
>>>>>>>>>>>>>             several test parameters for the test suite,
>>>>>>>>>>>>>                                 like for the IUT test link
>>>>>>>>>>>>>             MAC, etc. These
>>>>>>>>>>>>>                                 seem like they would need
>>>>>>>>>>>>>             to be set somewhere
>>>>>>>>>>>>>                                 to run many of the tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I see in the Test Engine
>>>>>>>>>>>>>             documentation, there
>>>>>>>>>>>>>                                 are instructions on how to
>>>>>>>>>>>>>             create new
>>>>>>>>>>>>>             parameters for test suites in the Tester
>>>>>>>>>>>>>             configuration, but there is nothing in the
>>>>>>>>>>>>>                                 user guide or in the
>>>>>>>>>>>>>             Tester guide for how to
>>>>>>>>>>>>>                                 set the arguments for the
>>>>>>>>>>>>>             parameters when
>>>>>>>>>>>>>             running the test suite that I can find. I'm
>>>>>>>>>>>>>                                 not sure if I need to
>>>>>>>>>>>>>             write my own Tester
>>>>>>>>>>>>>             config, or if I should be setting these in
>>>>>>>>>>>>>                                 some other way.
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 How should these values be
>>>>>>>>>>>>>             set?
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 I'm also not sure what
>>>>>>>>>>>>>             environment
>>>>>>>>>>>>>             variables/arguments are strictly necessary or
>>>>>>>>>>>>>             which are optional.
>>>>>>>>>>>>>
>>>>>>>>>>>>>             Regards,
>>>>>>>>>>>>>                                 Adam
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                 -- *Adam Hassick*
>>>>>>>>>>>>>             Senior Developer
>>>>>>>>>>>>>                                 UNH InterOperability Lab
>>>>>>>>>>>>>             ahassick@iol.unh.edu
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>>>                                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                             --                 *Adam Hassick*
>>>>>>>>>>>                             Senior Developer
>>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>>             ahassick@iol.unh.edu
>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>             <https://www.iol.unh.edu/>
>>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                         --             *Adam Hassick*
>>>>>>>>>                         Senior Developer
>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                     --         *Adam Hassick*
>>>>>>>>>                     Senior Developer
>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 --     *Adam Hassick*
>>>>>>>>                 Senior Developer
>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>>                 +1 (603) 475-8248
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             -- 
>>>>>>>             *Adam Hassick*
>>>>>>>             Senior Developer
>>>>>>>             UNH InterOperability Lab
>>>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>>>>             <mailto:ahassick@iol.unh.edu>
>>>>>>>             iol.unh.edu <http://iol.unh.edu>
>>>>>>>             <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>>>             +1 (603) 475-8248
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>         -- 
>>>>>         *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>         ahassick@iol.unh.edu
>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>
>>>
>>>
>>>
>>>     -- 
>>>     *Adam Hassick*
>>>     Senior Developer
>>>     UNH InterOperability Lab
>>>     ahassick@iol.unh.edu
>>>     iol.unh.edu <https://www.iol.unh.edu/>
>>>     +1 (603) 475-8248
>>
>
>
>
> -- 
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248

[-- Attachment #2: Type: text/html, Size: 94983 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-29 20:43                                         ` Andrew Rybchenko
@ 2023-08-31 19:38                                           ` Adam Hassick
  2023-09-01  7:59                                             ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-08-31 19:38 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 34163 bytes --]

Hi Andrew,

I have one additional question as well: Does the test engine support
running tests on two ARMv8 test agents?

1. We'll sort out warnings this week. Thanks for heads up.
>

Great. Let me know when that's fixed.

Support for old LTS branches was dropped some time ago, but in the future
> it is definitely possible to keep it for new LTS branches. I think 22.11 is
> supported, but I'm not sure about older LTS releases.
>

Good to know.


> 2. You can add command-line option --sanity to run tests marked with
> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
> there is a space for terminology improvement here. We'll do it.
>
> Also it takes a lot of time because of failures and tests which wait for
> some timeout.
>

That makes sense to me. We'll use the time to complete tests on virtio or
the Intel devices as a reference for how long the tests really take to
complete.
We will explore the possibility of periodically running the sanity tests
for patches.


> The test harness can provide coverage reports based on gcov, but I'm not
> sure what you mean by a "dial" to control test coverage. Provided reports
> are rather for human to analyze.
>

The general idea is to have some kind of parameter on the test suite, which
could be an integer ranging from zero to ten, that controls how many tests
are run based on how important the test is.

Similar to how some command line interfaces provide a verbosity level
parameter (some number of "-v" arguments) to control the importance of the
information in the log.
The verbosity level zero only prints very important log messages, while ten
prints everything.

In much the same manner as above, this "dial" parameter controls what tests
are run and with what parameters based on how important those tests and
test parameter combinations are.
Coverage Level zero tells the suite to run a very basic set of important
tests, with minimal parameterization. This mode would take only ~5-10
minutes to run.
In contrast, Coverage Level ten includes all the edge cases, every
combination of test parameters, everything the test suite can do, which
takes the normal several hours to run.
The values 1 - 9 are between those two extremes, allowing the user to get a
gradient of test coverage in the results and to limit the running time.

Then we could, for example, run the "run.sh" with a level of 2 or 3 for
incoming patches that need quick results, and with a level of 10 for the
less often run periodic tests performed on main or LTS branches.


> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing
> results. Unfortunately it is time consuming to fill in expectations
> database since it is necessary to analyze testing results and classify if
> it is a bug or just acceptable behaviour aspect.
>
> Bublik allows to compare results of two runs. It is useful for human, but
> still not good for automation.
>
> I have local patch for mlx5 driver which reports Tx ring size maximum. It
> makes pass rate higher. It is a problem for test harness that mlx5 does not
> report limits right now.
>
> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio
> net is 99% right now and could be done 100% easily (just one thing to fix
> in expectations).
>
> I think logs storage setup is essential for logs analysis. Of course, you
> can request HTML logs when you run tests (--log-html=html) or generate
> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
> browser, but logs storage makes it more convenient.
>

We are interested in setting up Bublik, potentially as an externally-facing
component, once we have our process of running the test suite stabilized.
Once we are able to run the test suite again, I'll see what the pass rate
is on our other hardware.
Good to know that it isn't an issue with our dev testbed causing the high
fail rate.

For Intel hardware, we have an XL710 and an Intel E810-C in our development
testbed. Although they are slightly different devices, ideally the pass
rate will be identical or similar. I have yet to set up a VM pair for
virtio, but we will soon.

Latest version of test-environment has examples of our CGI scripts which we
> use for log storage (see tools/log_server/README.md).
>
> Also all bits for Jenkins setup are available. See
> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
> ts-rigs-sample.
>

Jenkins integration, setting up production rig configurations, and
permanent log storage will be our next steps once I am able to run the
tests again.
Unless there is an easy way to have meson not pass "-Werror" into GCC. Then
I would be able to run the test suite.

Thanks,
Adam



>
> On 8/29/23 17:02, Adam Hassick wrote:
>
> Hi Andrew,
>
> That fix seems to have resolved the issue, thanks for the quick turnaround
> time on that patch.
> Now that we have the RCF timeout issue resolved, there are a few other
> questions and issues that we have about the tests themselves.
>
> 1. The test suite fails to build with a couple warnings.
>
> Below is the stderr log from compilation:
>
> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>> ../../lib/dpdk_pmd_ts.c: In function
>> ‘test_create_traffic_generator_params’:
>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
>> format arguments [-Werror=format-security]
>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>> |     ^~
>> cc1: all warnings being treated as errors
>> ninja: build stopped: subcommand failed.
>> ninja: Entering directory `.'
>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>> ../../lib/dpdk_pmd_ts.c: In function
>> ‘test_create_traffic_generator_params’:
>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no
>> format arguments [-Werror=format-security]
>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>> |     ^~
>> cc1: all warnings being treated as errors
>>
>
> This error wasn't occurring last week, which was the last time I ran the
> tests.
> The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC
> v11.4.0 installed, if this information is helpful.
>
> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
> collectively take around 9 hours. Is it possible, and would it make sense,
> to lower the test coverage and have the test suite run faster?
>
> For some context, we run immediate testing on incoming patches for DPDK
> main and development branches, as well as periodic test runs on the main,
> stable, and LTS branches.
> For us to consider including this test suite as part of our immediate
> testing on patches, we would have to reduce the test coverage to the most
> important tests.
> This is primarily to reduce the testing time to, for example, less than 30
> minutes. Testing on patches can't take too long because the lab can receive
> numerous patches each day, which each require individual testing runs.
>
> At what frequency we run these tests, and on what, still needs to be
> discussed with the DPDK community, but it would be nice to know if the test
> suite had a "dial" to control the testing coverage.
>
> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300 of
> ~6,600 tests passed. Is there anything we can do to diagnose these test
> failures?
>
> Thanks,
> Adam
>
>
> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> I've pushed the fix in main branch and a new tag v1.18.1. It should solve
>> the problem with IPv6 address from DNS.
>>
>> Andrew.
>>
>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>
>> Hi Adam,
>>
>> > Does the test engine prefer to use IPv6 over IPv4 for initiating the
>> RCF connection to the test bed hosts? And if so, is there a way to force it
>> to use IPv4?
>>
>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your case,
>> I guess it is the root cause of the problem.
>> Of course, it is TE problem since I see really weird code in
>> lib/comm_net_engine/comm_net_engine.c line 135.
>>
>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>> ts-factory/test-environment repository. Please, try.
>>
>> It is late night fix with minimal testing and no review. I'll pass it
>> through review process tomorrow and
>> hopefully it will be released in one-two days.
>>
>> Andrew.
>>
>> On 8/28/23 18:02, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> We have yet to notice a distinct pattern with the failures. Sometimes,
>> the RCF will start and connect without issue a few times in a row before
>> failing to connect again. Once the issue begins to occur, neither rebooting
>> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
>> directories (suites, agents, inst) and rebooting the hosts afterward
>> resolves the issue. When it begins working again seems very arbitrary to us.
>>
>> I do usually try to terminate the test engine with Ctrl+C, but when it
>> hangs while trying to start RCF, that does not work.
>>
>> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
>> connection to the test bed hosts? And if so, is there a way to force it to
>> use IPv4?
>>
>>  - Adam
>>
>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>> Done. It works fine for me without any issues.
>>>
>>> Have you noticed any pattern when it works or does not work?
>>> May be it is a problem of not clean state after termination?
>>> Does it work fine the first time after DUTs reboot?
>>> How do you terminate testing? It should be done using Ctrl+C in terminal
>>> where you execute run.sh command.
>>>  In this case it should shutdown gracefully and close all test agents
>>> and engine applications.
>>>
>>> (I'm trying to understand why you've seen many test agent processes. It
>>> should not happen.)
>>>
>>> Andrew.
>>>
>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>
>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> Two of our systems (the Test Engine runner and the DUT host) are running
>>> Ubuntu 20.04 LTS, however this morning I noticed that the tester system
>>> (the one having issues) is running Ubuntu 22.04 LTS.
>>> This could be the source of the problem. I encountered a dependency
>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>> system. Since the tester is also the host having connection issues, I will
>>> try downgrading that system to 20.04, and see if that changes anything.
>>>
>>>
>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>
>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>> I did try passing in the "--vg-rcf" argument to the run.sh script of the
>>> test suite after installing valgrind, but there was no additional output
>>> that I saw.
>>>
>>>
>>> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
>>> where you run test engine).
>>>
>>>
>>> I will try pulling in the changes you've pushed up, and will see if that
>>> fixes anything.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hello Adam,
>>>>
>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>
>>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>>> running on test engine and agents?
>>>>
>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>> patch in review to fix it, however, the behaviour is different in
>>>> this case, so it is unlike the same problem.
>>>>
>>>>
>>>> I've just published a new tag which fixes known test engine side
>>>> problems on Debian 12 and Fedora 38.
>>>>
>>>>
>>>> One more idea is to install valgrind on the test engine host and
>>>> run with option --vg-rcf to check if something weird is happening.
>>>>
>>>> What I don't understand right now is why I see just one failed attempt
>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>
>>>> Andrew.
>>>>
>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>
>>>>  > Is there any firewall in the network or on test hosts which could
>>>> block incoming TCP connection to the port 23571
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>>>> you run test engine?
>>>>
>>>> Our test engine host and the testbed are on the same subnet. The
>>>> connection does work sometimes.
>>>>
>>>>  > If behaviour the same on the next try and you see that test agent is
>>>> kept running, could you check using
>>>>  >
>>>>  > # netstat -tnlp
>>>>  >
>>>>  > that Test Agent is listening on the port and try to establish TCP
>>>> connection from test agent using
>>>>  >
>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>  >
>>>>  > and check if TCP connection could be established.
>>>>
>>>> I was able to replicate the same behavior again, where it hangs while
>>>> RCF is trying to start.
>>>> Running this command, I see this in the output:
>>>>
>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>  18599/ta
>>>>
>>>> So it seems like it is listening on the correct port.
>>>> Additionally, I was able to connect to the Tester machine from our Test
>>>> Engine host using telnet. It printed the PID of the process once the
>>>> connection was opened.
>>>>
>>>> I tried running the "ta" application manually on the command line, and
>>>> it didn't print anything at all.
>>>> Maybe the issue is something on the Test Engine side.
>>>>
>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>     Hi Adam,
>>>>
>>>>      > On the tester host (which appears to be the Peer agent), there
>>>>     are four processes that I see running, which look like the test
>>>>     agent processes.
>>>>
>>>>     Before the next try I'd recommend to kill these processes.
>>>>
>>>>     Is there any firewall in the network or on test hosts which could
>>>>     block incoming TCP connection to the port 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>     where you run test engine?
>>>>
>>>>     If behaviour the same on the next try and you see that test agent
>>>> is
>>>>     kept running, could you check using
>>>>
>>>>     # netstat -tnlp
>>>>
>>>>     that Test Agent is listening on the port and try to establish TCP
>>>>     connection from test agent using
>>>>
>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>     and check if TCP connection could be established.
>>>>
>>>>     Another idea is to login Tester under root as testing does, get
>>>>     start TA command from the log and try it by hands without -n and
>>>>     remove extra escaping.
>>>>
>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>
>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>
>>>>     Hopefully in this case test agent directory remains in the /tmp and
>>>>     you don't need to copy it as testing does.
>>>>     May be output could shed some light on what's going on.
>>>>
>>>>     Andrew.
>>>>
>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>
>>>>     Hi Andrew,
>>>>
>>>>     This is the output that I see in the terminal when this failure
>>>>     occurs, after the test agent binaries build and the test engine
>>>>     starts:
>>>>
>>>>     Platform default build - pass
>>>>     Simple RCF consistency check succeeded
>>>>     --->>> Starting Logger...done
>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>     processes. I let it hang for around 9 minutes.
>>>>
>>>>     On the tester host (which appears to be the Peer agent), there are
>>>>     four processes that I see running, which look like the test agent
>>>>     processes.
>>>>
>>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>>
>>>>      - Adam
>>>>
>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>         Hi Adam,
>>>>
>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>         says that copy_timeout is in seconds and implementation in
>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>         Theoretically select() could be interrupted by signal, but I
>>>>         think it is unlikely here.
>>>>
>>>>         I'm not sure that I understand what do you mean by RCF
>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>         does it happen when tests are in progress, e.g. in the middle
>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>         likely either host with test agent dies or test agent itself
>>>>         crashes. It would be easier for me if classify it if you share
>>>>         text log (log.txt, full or just corresponding fragment with
>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>         depending on which agent has problems could shed some light.
>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>
>>>>         Andrew.
>>>>
>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>
>>>>         Hi Andrew,
>>>>
>>>>         I've set up a test rig repository here, and have created
>>>>         configurations for our development testbed based off of the
>>>>         examples.
>>>>         We've been able to get the test suite to run manually on
>>>>         Mellanox CX5 devices once.
>>>>         However, we are running into an issue where, when RCF starts,
>>>>         the RCF connection times out very frequently. We aren't sure
>>>>         why this is the case.
>>>>         It works sometimes, but most of the time when we try to run
>>>>         the test engine, it encounters this issue.
>>>>         I've tried changing the RCF port by setting
>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>         machines. Neither seems to fix the issue.
>>>>
>>>>         It also seems like the timeout takes far longer than 60
>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>         before I try to run the test suite.
>>>>         I assume the unit for this variable is seconds?
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>> wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             Thanks, I've cloned the example repository and will start
>>>>             setting up a configuration for our development testbed
>>>>             today. I'll let you know if I run into any difficulties
>>>>             or have any questions.
>>>>
>>>>              - Adam
>>>>
>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 I've published
>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>                 Hopefully it will help to define your test rigs and
>>>>                 successfully run some tests manually. Feel free to
>>>>                 ask any questions and I'll answer here and try to
>>>>                 update documentation.
>>>>
>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>                 (3).
>>>>                 Hopefully everything is in place for step (4), but we
>>>>                 need to make steps (2) and (3) first.
>>>>
>>>>                 Andrew.
>>>>
>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 > I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>
>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>                 almost ready right now, but I need to double-check
>>>>                 it before publishing.
>>>>
>>>>                 Regards,
>>>>                 Andrew.
>>>>
>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>
>>>>                 Hi Andrew,
>>>>
>>>>                 I'm adding the CI mailing list to this
>>>>                 conversation. Others in the community might find
>>>>                 this conversation valuable.
>>>>
>>>>                 We do want to run testing on a regular basis. The
>>>>                 Jenkins integration will be very useful for us, as
>>>>                 most of our CI is orchestrated by Jenkins.
>>>>                 I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>                 If you would like to know anything about our
>>>>                 testbeds that would help you with creating an
>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>                 questions you have.
>>>>
>>>>                 We have multiple test rigs (we call these
>>>>                 "DUT-tester pairs") that we run our existing
>>>>                 hardware testing on, with differing network
>>>>                 hardware and CPU architecture. I figured this might
>>>>                 be an important detail.
>>>>
>>>>                 Thanks,
>>>>                 Adam
>>>>
>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                     Greatings Adam,
>>>>
>>>>                     I'm happy to hear that you're trying to bring
>>>>                     it up.
>>>>
>>>>                     As I understand the final goal is to run it on
>>>>                     regular basis. So, we need to make it properly
>>>>                     from the very beginning.
>>>>                     Bring up of all features consists of 4 steps:
>>>>
>>>>                     1. Create site-specific repository (we call it
>>>>                     ts-rigs) which contains information about test
>>>>                     rigs and other site-specific information like
>>>>                     where to send mails, where to store logs etc.
>>>>                     It is required for manual execution as well,
>>>>                     since test rigs description is essential. I'll
>>>>                     return to the topic below.
>>>>
>>>>                     2. Setup logs storage for automated runs.
>>>>                     Basically it is a disk space plus apache2 web
>>>>                     server with few CGI scripts which help a lot to
>>>>                     save disk space.
>>>>
>>>>                     3. Setup Bublik web application which provides
>>>>                     web interface to view testing results. Same as
>>>>                     https://ts-factory.io/bublik
>>>>                     <https://ts-factory.io/bublik>
>>>> <https://ts-factory.io/bublik>
>>>>
>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>                     save logs in log storage (2) and import it to
>>>>                     bublik (3).
>>>>
>>>>                     Last few month we spent on our homework to make
>>>>                     it simpler to bring up automated execution
>>>>                     using Jenkins -
>>>>                     https://github.com/ts-factory/te-jenkins
>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>> <https://github.com/ts-factory/te-jenkins>
>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>                     available tomorrow.
>>>>
>>>>                     Let's return to the step (1).
>>>>
>>>>                     Unfortunately there is no publicly available
>>>>                     example of the ts-rigs repository since
>>>>                     sensitive site-specific information is located
>>>>                     there. But I'm ready to help you to create it
>>>>                     for UNH. I see two options here:
>>>>
>>>>                     (A) I'll ask questions and based on your
>>>>                     answers will create the first draft with my
>>>>                     comments.
>>>>
>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>                     publish it and you'll create UNH ts-rigs based
>>>>                     on it.
>>>>
>>>>                     Of course, I'll help to debug and finally bring
>>>>                     it up in any case.
>>>>
>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>                     a bit more generic and will help other
>>>>                     potential users to bring it up.
>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>                     What do you think?
>>>>
>>>>                     Thanks,
>>>>                     Andrew.
>>>>
>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>
>>>>                     Greetings Adam,
>>>>
>>>>
>>>>                     Thanks for contacting us. I copy Andrew who
>>>>                     would be happy to help
>>>>
>>>>                     Thanks,
>>>>                     Konstantin
>>>>
>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>                     <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu> wrote:
>>>>
>>>>                     
>>>>                     Greetings Konstantin,
>>>>
>>>>                     I am in the process of setting up the DPDK
>>>>                     Poll Mode Driver test suite as an addition to
>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>
>>>>                     I have some questions about how to set the
>>>>                     test suite arguments.
>>>>
>>>>                     I have been able to configure the Test Engine
>>>>                     to connect to the hosts in the testbed. The
>>>>                     RCF, Configurator, and Tester all begin to
>>>>                     run, however the prelude of the test suite
>>>>                     fails to run.
>>>>
>>>>
>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>
>>>>                     The documentation mentions that there are
>>>>                     several test parameters for the test suite,
>>>>                     like for the IUT test link MAC, etc. These
>>>>                     seem like they would need to be set somewhere
>>>>                     to run many of the tests.
>>>>
>>>>                     I see in the Test Engine documentation, there
>>>>                     are instructions on how to create new
>>>>                     parameters for test suites in the Tester
>>>>                     configuration, but there is nothing in the
>>>>                     user guide or in the Tester guide for how to
>>>>                     set the arguments for the parameters when
>>>>                     running the test suite that I can find. I'm
>>>>                     not sure if I need to write my own Tester
>>>>                     config, or if I should be setting these in
>>>>                     some other way.
>>>>
>>>>                     How should these values be set?
>>>>
>>>>                     I'm also not sure what environment
>>>>                     variables/arguments are strictly necessary or
>>>>                     which are optional.
>>>>
>>>>                     Regards,
>>>>                     Adam
>>>>
>>>>                     --                     *Adam Hassick*
>>>>                     Senior Developer
>>>>                     UNH InterOperability Lab
>>>>                     ahassick@iol.unh.edu
>>>>                     <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>                     +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>                 --                 *Adam Hassick*
>>>>                 Senior Developer
>>>>                 UNH InterOperability Lab
>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>                 +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>             --             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>     --     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

[-- Attachment #2: Type: text/html, Size: 90038 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-08-31 19:38                                           ` Adam Hassick
@ 2023-09-01  7:59                                             ` Andrew Rybchenko
  2023-09-05 15:01                                               ` Adam Hassick
  0 siblings, 1 reply; 51+ messages in thread
From: Andrew Rybchenko @ 2023-09-01  7:59 UTC (permalink / raw)
  To: Adam Hassick; +Cc: Patrick Robb, Konstantin Ushakov, ci

[-- Attachment #1: Type: text/plain, Size: 51510 bytes --]

Hi Adam,

On 8/31/23 22:38, Adam Hassick wrote:
> Hi Andrew,
>
> I have one additional question as well: Does the test engine support 
> running tests on two ARMv8 test agents?
>
>     1. We'll sort out warnings this week. Thanks for heads up.
>
>
> Great. Let me know when that's fixed.

Done. We also fixed a number of warnings in TE.
Also we fixed root test package name to be consistent with the 
repository name.

>     Support for old LTS branches was dropped some time ago, but in the
>     future it is definitely possible to keep it for new LTS branches.
>     I think 22.11 is supported, but I'm not sure about older LTS releases.
>
>
> Good to know.
>
>     2. You can add command-line option --sanity to run tests marked
>     with TEST_HARNESS_SANITY requirement (see
>     dpdk-ethdev-ts/scripts/run.sh and grep TEST_HARNESS_SANITY
>     dpdk-ethdev-ts to see which tests are marked). Yes, there is a
>     space for terminology improvement here. We'll do it.
>

Done. Now it is called --checkup.

>
>     Also it takes a lot of time because of failures and tests which
>     wait for some timeout.
>
>
> That makes sense to me. We'll use the time to complete tests on virtio 
> or the Intel devices as a reference for how long the tests really take 
> to complete.
> We will explore the possibility of periodically running the sanity 
> tests for patches.

I'll double-check and let you know how long entire TS runs on Intel 
X710, E810, Mellanox CX5 and virtio net. Just to ensure that time 
observed in your case looks the same.

>
>     The test harness can provide coverage reports based on gcov, but
>     I'm not sure what you mean by a "dial" to control test coverage.
>     Provided reports are rather for human to analyze.
>
>
> The general idea is to have some kind of parameter on the test suite, 
> which could be an integer ranging from zero to ten, that controls how 
> many tests are run based on how important the test is.
>
> Similar to how some command line interfaces provide a verbosity level 
> parameter (some number of "-v" arguments) to control the importance of 
> the information in the log.
> The verbosity level zero only prints very important log messages, 
> while ten prints everything.
>
> In much the same manner as above, this "dial" parameter controls what 
> tests are run and with what parameters based on how important those 
> tests and test parameter combinations are.
> Coverage Level zero tells the suite to run a very basic set of 
> important tests, with minimal parameterization. This mode would take 
> only ~5-10 minutes to run.
> In contrast, Coverage Level ten includes all the edge cases, every 
> combination of test parameters, everything the test suite can do, 
> which takes the normal several hours to run.
> The values 1 - 9 are between those two extremes, allowing the user to 
> get a gradient of test coverage in the results and to limit the 
> running time.
>
> Then we could, for example, run the "run.sh" with a level of 2 or 3 
> for incoming patches that need quick results, and with a level of 10 
> for the less often run periodic tests performed on main or LTS branches.

Understood now. Thanks a lot for the idea. We'll discuss it and come back.

>     3. Yes, really many tests on Mellanox CX5 NICs report unexpected
>     testing results. Unfortunately it is time consuming to fill in
>     expectations database since it is necessary to analyze testing
>     results and classify if it is a bug or just acceptable behaviour
>     aspect.
>
>     Bublik allows to compare results of two runs. It is useful for
>     human, but still not good for automation.
>
>     I have local patch for mlx5 driver which reports Tx ring size
>     maximum. It makes pass rate higher. It is a problem for test
>     harness that mlx5 does not report limits right now.
>
>     Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
>     virtio net is 99% right now and could be done 100% easily (just
>     one thing to fix in expectations).
>
>     I think logs storage setup is essential for logs analysis. Of
>     course, you can request HTML logs when you run tests
>     (--log-html=html) or generate after run using
>     dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>     browser, but logs storage makes it more convenient.
>
>
> We are interested in setting up Bublik, potentially as an 
> externally-facing component, once we have our process of running the 
> test suite stabilized.
> Once we are able to run the test suite again, I'll see what the pass 
> rate is on our other hardware.
> Good to know that it isn't an issue with our dev testbed causing the 
> high fail rate.
>
> For Intel hardware, we have an XL710 and an Intel E810-C in our 
> development testbed. Although they are slightly different devices, 
> ideally the pass rate will be identical or similar. I have yet to set 
> up a VM pair for virtio, but we will soon.
>
>     Latest version of test-environment has examples of our CGI scripts
>     which we use for log storage (see tools/log_server/README.md).
>
>     Also all bits for Jenkins setup are available. See
>     dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>     ts-rigs-sample.
>
>
> Jenkins integration, setting up production rig configurations, and 
> permanent log storage will be our next steps once I am able to run the 
> tests again.
> Unless there is an easy way to have meson not pass "-Werror" into GCC. 
> Then I would be able to run the test suite.

Hopefully it is resolved now.

I thought a bit more about your usecase for Jenkins. I'm not 100% sure 
that existing pipelines are convenient for your usecase.
Fill free to ask questions when you are on it.

Thanks,
Andrew.

>
> Thanks,
> Adam
>
>
>     On 8/29/23 17:02, Adam Hassick wrote:
>>     Hi Andrew,
>>
>>     That fix seems to have resolved the issue, thanks for the quick
>>     turnaround time on that patch.
>>     Now that we have the RCF timeout issue resolved, there are a few
>>     other questions and issues that we have about the tests themselves.
>>
>>     1. The test suite fails to build with a couple warnings.
>>
>>     Below is the stderr log from compilation:
>>
>>         FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>         cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>         -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>>         -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>         -fPIC -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>         -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>>         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>         ../../lib/dpdk_pmd_ts.c
>>         ../../lib/dpdk_pmd_ts.c: In function
>>         ‘test_create_traffic_generator_params’:
>>         ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>         literal and no format arguments [-Werror=format-security]
>>         5577 |     rc = te_kvpair_add(result, buf, mode);
>>         |     ^~
>>         cc1: all warnings being treated as errors
>>         ninja: build stopped: subcommand failed.
>>         ninja: Entering directory `.'
>>         FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>         cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>         -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>         -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
>>         -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
>>         -fPIC -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o'
>>         -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o
>>         'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c
>>         ../../lib/dpdk_pmd_ts.c
>>         ../../lib/dpdk_pmd_ts.c: In function
>>         ‘test_create_traffic_generator_params’:
>>         ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
>>         literal and no format arguments [-Werror=format-security]
>>         5577 |     rc = te_kvpair_add(result, buf, mode);
>>         |     ^~
>>         cc1: all warnings being treated as errors
>>
>>
>>     This error wasn't occurring last week, which was the last time I
>>     ran the tests.
>>     The TE host and the DUT have GCC v9.4.0 installed, and the tester
>>     has GCC v11.4.0 installed, if this information is helpful.
>>
>>     2. On the Mellanox CX5s, there are over 6,000 tests run, which
>>     collectively take around 9 hours. Is it possible, and would it
>>     make sense, to lower the test coverage and have the test suite
>>     run faster?
>>
>>     For some context, we run immediate testing on incoming patches
>>     for DPDK main and development branches, as well as periodic test
>>     runs on the main, stable, and LTS branches.
>>     For us to consider including this test suite as part of our
>>     immediate testing on patches, we would have to reduce the test
>>     coverage to the most important tests.
>>     This is primarily to reduce the testing time to, for example,
>>     less than 30 minutes. Testing on patches can't take too long
>>     because the lab can receive numerous patches each day, which each
>>     require individual testing runs.
>>
>>     At what frequency we run these tests, and on what, still needs to
>>     be discussed with the DPDK community, but it would be nice to
>>     know if the test suite had a "dial" to control the testing coverage.
>>
>>     3. We see a lot of test failures on our Mellanox CX5 NICs. Around
>>     2,300 of ~6,600 tests passed. Is there anything we can do to
>>     diagnose these test failures?
>>
>>     Thanks,
>>     Adam
>>
>>
>>     On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko
>>     <andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>         Hi Adam,
>>
>>         I've pushed the fix in main branch and a new tag v1.18.1. It
>>         should solve the problem with IPv6 address from DNS.
>>
>>         Andrew.
>>
>>         On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>         Hi Adam,
>>>
>>>         > Does the test engine prefer to use IPv6 over IPv4 for
>>>         initiating the RCF connection to the test bed hosts? And if
>>>         so, is there a way to force it to use IPv4?
>>>
>>>         Brilliant idea. If DNS returns both IPv4 and IPv6 addresses
>>>         in your case, I guess it is the root cause of the problem.
>>>         Of course, it is TE problem since I see really weird code in
>>>         lib/comm_net_engine/comm_net_engine.c line 135.
>>>
>>>         I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>>         ts-factory/test-environment repository. Please, try.
>>>
>>>         It is late night fix with minimal testing and no review.
>>>         I'll pass it through review process tomorrow and
>>>         hopefully it will be released in one-two days.
>>>
>>>         Andrew.
>>>
>>>         On 8/28/23 18:02, Adam Hassick wrote:
>>>>         Hi Andrew,
>>>>
>>>>         We have yet to notice a distinct pattern with the failures.
>>>>         Sometimes, the RCF will start and connect without issue a
>>>>         few times in a row before failing to connect again. Once
>>>>         the issue begins to occur, neither rebooting all of the
>>>>         hosts (test engine VM, tester, IUT) or deleting all of the
>>>>         build directories (suites, agents, inst) and rebooting the
>>>>         hosts afterward resolves the issue. When it begins working
>>>>         again seems very arbitrary to us.
>>>>
>>>>         I do usually try to terminate the test engine with Ctrl+C,
>>>>         but when it hangs while trying to start RCF, that does not
>>>>         work.
>>>>
>>>>         Does the test engine prefer to use IPv6 over IPv4 for
>>>>         initiating the RCF connection to the test bed hosts? And if
>>>>         so, is there a way to force it to use IPv4?
>>>>
>>>>          - Adam
>>>>
>>>>         On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko
>>>>         <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>             > I'll double-check test engine on Ubuntu 20.04 and
>>>>             Ubuntu 22.04.
>>>>
>>>>             Done. It works fine for me without any issues.
>>>>
>>>>             Have you noticed any pattern when it works or does not
>>>>             work?
>>>>             May be it is a problem of not clean state after
>>>>             termination?
>>>>             Does it work fine the first time after DUTs reboot?
>>>>             How do you terminate testing? It should be done using
>>>>             Ctrl+C in terminal where you execute run.sh command.
>>>>              In this case it should shutdown gracefully and close
>>>>             all test agents and engine applications.
>>>>
>>>>             (I'm trying to understand why you've seen many test
>>>>             agent processes. It should not happen.)
>>>>
>>>>             Andrew.
>>>>
>>>>             On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>>             On 8/25/23 17:06, Adam Hassick wrote:
>>>>>>             Hi Andrew,
>>>>>>
>>>>>>             Two of our systems (the Test Engine runner and the
>>>>>>             DUT host) are running Ubuntu 20.04 LTS, however this
>>>>>>             morning I noticed that the tester system (the one
>>>>>>             having issues) is running Ubuntu 22.04 LTS.
>>>>>>             This could be the source of the problem. I
>>>>>>             encountered a dependency issue trying to run the Test
>>>>>>             Engine on 22.04 LTS, so I downgraded the system.
>>>>>>             Since the tester is also the host having connection
>>>>>>             issues, I will try downgrading that system to 20.04,
>>>>>>             and see if that changes anything.
>>>>>
>>>>>             Unlikely, but who knows. We run tests (DUTs) on Ubuntu
>>>>>             20.04, Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04,
>>>>>             Debian 11 and Fedora 38 every night.
>>>>>             Right now Debian 11 is used for test engine in nightly
>>>>>             regressions.
>>>>>
>>>>>             I'll double-check test engine on Ubuntu 20.04 and
>>>>>             Ubuntu 22.04.
>>>>>
>>>>>>             I did try passing in the "--vg-rcf" argument to the
>>>>>>             run.sh script of the test suite after installing
>>>>>>             valgrind, but there was no additional output that I saw.
>>>>>
>>>>>             Sorry, I should valgrind output should be in
>>>>>             valgrind.te_rcf (direction where you run test engine).
>>>>>
>>>>>>
>>>>>>             I will try pulling in the changes you've pushed up,
>>>>>>             and will see if that fixes anything.
>>>>>>
>>>>>>             Thanks,
>>>>>>             Adam
>>>>>>
>>>>>>             On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko
>>>>>>             <andrew.rybchenko@oktetlabs.ru> wrote:
>>>>>>
>>>>>>                 Hello Adam,
>>>>>>
>>>>>>                 On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>>>                 I'd like to try to repeat the problem locally.
>>>>>>>                 Which Linux distro is running on test engine and
>>>>>>>                 agents?
>>>>>>>
>>>>>>>                 In fact I know one problem with Debian 12 and
>>>>>>>                 Fedora 38 and we have
>>>>>>>                 patch in review to fix it, however, the
>>>>>>>                 behaviour is different in
>>>>>>>                 this case, so it is unlike the same problem.
>>>>>>
>>>>>>                 I've just published a new tag which fixes known
>>>>>>                 test engine side problems on Debian 12 and Fedora 38.
>>>>>>
>>>>>>>
>>>>>>>                 One more idea is to install valgrind on the test
>>>>>>>                 engine host and
>>>>>>>                 run with option --vg-rcf to check if something
>>>>>>>                 weird is happening.
>>>>>>>
>>>>>>>                 What I don't understand right now is why I see
>>>>>>>                 just one failed attempt
>>>>>>>                 to connect in your log.txt and then Logger
>>>>>>>                 shutdown after 9 minutes.
>>>>>>>
>>>>>>>                 Andrew.
>>>>>>>
>>>>>>>                 On 8/24/23 23:29, Adam Hassick wrote:
>>>>>>>>                  > Is there any firewall in the network or on
>>>>>>>>                 test hosts which could block incoming TCP
>>>>>>>>                 connection to the port 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 from the host where you run test engine?
>>>>>>>>
>>>>>>>>                 Our test engine host and the testbed are on the
>>>>>>>>                 same subnet. The connection does work sometimes.
>>>>>>>>
>>>>>>>>                  > If behaviour the same on the next try and
>>>>>>>>                 you see that test agent is kept running, could
>>>>>>>>                 you check using
>>>>>>>>                  >
>>>>>>>>                  > # netstat -tnlp
>>>>>>>>                  >
>>>>>>>>                  > that Test Agent is listening on the port and
>>>>>>>>                 try to establish TCP connection from test agent
>>>>>>>>                 using
>>>>>>>>                  >
>>>>>>>>                  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                  >
>>>>>>>>                  > and check if TCP connection could be
>>>>>>>>                 established.
>>>>>>>>
>>>>>>>>                 I was able to replicate the same behavior
>>>>>>>>                 again, where it hangs while RCF is trying to
>>>>>>>>                 start.
>>>>>>>>                 Running this command, I see this in the output:
>>>>>>>>
>>>>>>>>                 tcp        0      0 0.0.0.0:23571
>>>>>>>>                 <http://0.0.0.0:23571> <http://0.0.0.0:23571>
>>>>>>>>                 <http://0.0.0.0:23571> 0.0.0.0:* LISTEN    
>>>>>>>>                  18599/ta
>>>>>>>>
>>>>>>>>                 So it seems like it is listening on the correct
>>>>>>>>                 port.
>>>>>>>>                 Additionally, I was able to connect to the
>>>>>>>>                 Tester machine from our Test Engine host using
>>>>>>>>                 telnet. It printed the PID of the process once
>>>>>>>>                 the connection was opened.
>>>>>>>>
>>>>>>>>                 I tried running the "ta" application manually
>>>>>>>>                 on the command line, and it didn't print
>>>>>>>>                 anything at all.
>>>>>>>>                 Maybe the issue is something on the Test Engine
>>>>>>>>                 side.
>>>>>>>>
>>>>>>>>                 On Thu, Aug 24, 2023 at 2:35 PM Andrew
>>>>>>>>                 Rybchenko <andrew.rybchenko@oktetlabs.ru
>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>
>>>>>>>>                     Hi Adam,
>>>>>>>>
>>>>>>>>                      > On the tester host (which appears to be
>>>>>>>>                 the Peer agent), there
>>>>>>>>                     are four processes that I see running,
>>>>>>>>                 which look like the test
>>>>>>>>                     agent processes.
>>>>>>>>
>>>>>>>>                     Before the next try I'd recommend to kill
>>>>>>>>                 these processes.
>>>>>>>>
>>>>>>>>                     Is there any firewall in the network or on
>>>>>>>>                 test hosts which could
>>>>>>>>                     block incoming TCP connection to the port
>>>>>>>>                 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 from the host
>>>>>>>>                     where you run test engine?
>>>>>>>>
>>>>>>>>                     If behaviour the same on the next try and
>>>>>>>>                 you see that test agent is
>>>>>>>>                     kept running, could you check using
>>>>>>>>
>>>>>>>>                     # netstat -tnlp
>>>>>>>>
>>>>>>>>                     that Test Agent is listening on the port
>>>>>>>>                 and try to establish TCP
>>>>>>>>                     connection from test agent using
>>>>>>>>
>>>>>>>>                     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 23571
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>
>>>>>>>>                     and check if TCP connection could be
>>>>>>>>                 established.
>>>>>>>>
>>>>>>>>                     Another idea is to login Tester under root
>>>>>>>>                 as testing does, get
>>>>>>>>                     start TA command from the log and try it by
>>>>>>>>                 hands without -n and
>>>>>>>>                     remove extra escaping.
>>>>>>>>
>>>>>>>>                     # sudo
>>>>>>>>                 PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>
>>>>>>>>                 LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>>>>>                 /tmp/linux_x86_root_76872_1692885663_1/ta Peer
>>>>>>>>                 23571
>>>>>>>>                 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>>>>
>>>>>>>>                     Hopefully in this case test agent directory
>>>>>>>>                 remains in the /tmp and
>>>>>>>>                     you don't need to copy it as testing does.
>>>>>>>>                     May be output could shed some light on
>>>>>>>>                 what's going on.
>>>>>>>>
>>>>>>>>                     Andrew.
>>>>>>>>
>>>>>>>>                     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>>>>>                 Hi Andrew,
>>>>>>>>>
>>>>>>>>>                     This is the output that I see in the
>>>>>>>>>                 terminal when this failure
>>>>>>>>>                     occurs, after the test agent binaries
>>>>>>>>>                 build and the test engine
>>>>>>>>>                     starts:
>>>>>>>>>
>>>>>>>>>                     Platform default build - pass
>>>>>>>>>                     Simple RCF consistency check succeeded
>>>>>>>>>                     --->>> Starting Logger...done
>>>>>>>>>                     --->>> Starting
>>>>>>>>>                 RCF...rcf_net_engine_connect(): Connection timed
>>>>>>>>>                     out
>>>>>>>>>                 iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>                 <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>>>>>
>>>>>>>>>                     Then, it hangs here until I kill the
>>>>>>>>>                 "te_rcf" and "te_tee"
>>>>>>>>>                     processes. I let it hang for around 9
>>>>>>>>>                 minutes.
>>>>>>>>>
>>>>>>>>>                     On the tester host (which appears to be
>>>>>>>>>                 the Peer agent), there are
>>>>>>>>>                     four processes that I see running, which
>>>>>>>>>                 look like the test agent
>>>>>>>>>                     processes.
>>>>>>>>>
>>>>>>>>>                     ta.Peer is an empty file. I've attached
>>>>>>>>>                 the log.txt from this run.
>>>>>>>>>
>>>>>>>>>                      - Adam
>>>>>>>>>
>>>>>>>>>                     On Thu, Aug 24, 2023 at 4:22 AM Andrew
>>>>>>>>>                 Rybchenko
>>>>>>>>>                     <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>
>>>>>>>>>                         Hi Adam,
>>>>>>>>>
>>>>>>>>>                         Yes, TE_RCFUNIX_TIMEOUT is in seconds.
>>>>>>>>>                 I've double-checked
>>>>>>>>>                         that it goes to 'copy_timeout' in
>>>>>>>>>                 ts-conf/rcf.conf.
>>>>>>>>>                         Description in in
>>>>>>>>>                 doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>>>>>                         says that copy_timeout is in seconds
>>>>>>>>>                 and implementation in
>>>>>>>>>                 lib/rcfunix/rcfunix.c passes the value to
>>>>>>>>>                 select() tv_sec.
>>>>>>>>>                         Theoretically select() could be
>>>>>>>>>                 interrupted by signal, but I
>>>>>>>>>                         think it is unlikely here.
>>>>>>>>>
>>>>>>>>>                         I'm not sure that I understand what do
>>>>>>>>>                 you mean by RCF
>>>>>>>>>                         connection timeout. Does it happen on
>>>>>>>>>                 TE startup when RCF
>>>>>>>>>                         starts test agents. If so,
>>>>>>>>>                 TE_RCFUNIX_TIMEOUT could help. Or
>>>>>>>>>                         does it happen when tests are in
>>>>>>>>>                 progress, e.g. in the middle
>>>>>>>>>                         of a test. If so, TE_RCFUNIX_TIMEOUT
>>>>>>>>>                 is unrelated and most
>>>>>>>>>                         likely either host with test agent
>>>>>>>>>                 dies or test agent itself
>>>>>>>>>                         crashes. It would be easier for me if
>>>>>>>>>                 classify it if you share
>>>>>>>>>                         text log (log.txt, full or just
>>>>>>>>>                 corresponding fragment with
>>>>>>>>>                         some context). Also content of ta.DPDK
>>>>>>>>>                 or ta.Peer file
>>>>>>>>>                         depending on which agent has problems
>>>>>>>>>                 could shed some light.
>>>>>>>>>                         Corresponding files contain
>>>>>>>>>                 stdout/stderr of test agents.
>>>>>>>>>
>>>>>>>>>                         Andrew.
>>>>>>>>>
>>>>>>>>>                         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                         I've set up a test rig repository
>>>>>>>>>>                 here, and have created
>>>>>>>>>>                 configurations for our development testbed
>>>>>>>>>>                 based off of the
>>>>>>>>>>                         examples.
>>>>>>>>>>                         We've been able to get the test suite
>>>>>>>>>>                 to run manually on
>>>>>>>>>>                         Mellanox CX5 devices once.
>>>>>>>>>>                         However, we are running into an issue
>>>>>>>>>>                 where, when RCF starts,
>>>>>>>>>>                         the RCF connection times out very
>>>>>>>>>>                 frequently. We aren't sure
>>>>>>>>>>                         why this is the case.
>>>>>>>>>>                         It works sometimes, but most of the
>>>>>>>>>>                 time when we try to run
>>>>>>>>>>                         the test engine, it encounters this
>>>>>>>>>>                 issue.
>>>>>>>>>>                         I've tried changing the RCF port by
>>>>>>>>>>                 setting
>>>>>>>>>>                 "TE_RCF_PORT=<some port number>" and
>>>>>>>>>>                 rebooting the testbed
>>>>>>>>>>                         machines. Neither seems to fix the
>>>>>>>>>>                 issue.
>>>>>>>>>>
>>>>>>>>>>                         It also seems like the timeout takes
>>>>>>>>>>                 far longer than 60
>>>>>>>>>>                         seconds, even when running "export
>>>>>>>>>>                 TE_RCFUNIX_TIMEOUT=60"
>>>>>>>>>>                         before I try to run the test suite.
>>>>>>>>>>                         I assume the unit for this variable
>>>>>>>>>>                 is seconds?
>>>>>>>>>>
>>>>>>>>>>                         Thanks,
>>>>>>>>>>                         Adam
>>>>>>>>>>
>>>>>>>>>>                         On Mon, Aug 21, 2023 at 10:19 AM Adam
>>>>>>>>>>                 Hassick
>>>>>>>>>>                         <ahassick@iol.unh.edu
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>> wrote:
>>>>>>>>>>
>>>>>>>>>>                             Hi Andrew,
>>>>>>>>>>
>>>>>>>>>>                             Thanks, I've cloned the example
>>>>>>>>>>                 repository and will start
>>>>>>>>>>                             setting up a configuration for
>>>>>>>>>>                 our development testbed
>>>>>>>>>>                             today. I'll let you know if I run
>>>>>>>>>>                 into any difficulties
>>>>>>>>>>                             or have any questions.
>>>>>>>>>>
>>>>>>>>>>                              - Adam
>>>>>>>>>>
>>>>>>>>>>                             On Sun, Aug 20, 2023 at 4:40 AM
>>>>>>>>>>                 Andrew Rybchenko
>>>>>>>>>>                             <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>
>>>>>>>>>>                                 Hi Adam,
>>>>>>>>>>
>>>>>>>>>>                                 I've published
>>>>>>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>>>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>>>>>>                 Hopefully it will help to define your test
>>>>>>>>>>                 rigs and
>>>>>>>>>>                 successfully run some tests manually. Feel
>>>>>>>>>>                 free to
>>>>>>>>>>                                 ask any questions and I'll
>>>>>>>>>>                 answer here and try to
>>>>>>>>>>                 update documentation.
>>>>>>>>>>
>>>>>>>>>>                 Meanwhile I'll prepare missing bits for steps
>>>>>>>>>>                 (2) and
>>>>>>>>>>                                 (3).
>>>>>>>>>>                 Hopefully everything is in place for step
>>>>>>>>>>                 (4), but we
>>>>>>>>>>                                 need to make steps (2) and
>>>>>>>>>>                 (3) first.
>>>>>>>>>>
>>>>>>>>>>                 Andrew.
>>>>>>>>>>
>>>>>>>>>>                                 On 8/18/23 21:40, Andrew
>>>>>>>>>>                 Rybchenko wrote:
>>>>>>>>>>>                 Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>>                 > I've conferred with the rest of the team,
>>>>>>>>>>>                 and we
>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>                 mainly
>>>>>>>>>>>                 option B.
>>>>>>>>>>>
>>>>>>>>>>>                 OK, I'll provide the sample on Monday for
>>>>>>>>>>>                 you. It is
>>>>>>>>>>>                 almost ready right now, but I need to
>>>>>>>>>>>                 double-check
>>>>>>>>>>>                                 it before publishing.
>>>>>>>>>>>
>>>>>>>>>>>                 Regards,
>>>>>>>>>>>                 Andrew.
>>>>>>>>>>>
>>>>>>>>>>>                                 On 8/17/23 20:03, Adam
>>>>>>>>>>>                 Hassick wrote:
>>>>>>>>>>>>                 Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>>                 I'm adding the CI mailing list to this
>>>>>>>>>>>>                 conversation. Others in the community might
>>>>>>>>>>>>                 find
>>>>>>>>>>>>                 this conversation valuable.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We do want to run testing on a regular
>>>>>>>>>>>>                 basis. The
>>>>>>>>>>>>                 Jenkins integration will be very useful for
>>>>>>>>>>>>                 us, as
>>>>>>>>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>>>>>>>>                 I've conferred with the rest of the team,
>>>>>>>>>>>>                 and we
>>>>>>>>>>>>                 think it would be best to move forward with
>>>>>>>>>>>>                 mainly
>>>>>>>>>>>>                 option B.
>>>>>>>>>>>>                 If you would like to know anything about our
>>>>>>>>>>>>                 testbeds that would help you with creating an
>>>>>>>>>>>>                 example ts-rigs repo, I'd be happy to
>>>>>>>>>>>>                 answer any
>>>>>>>>>>>>                 questions you have.
>>>>>>>>>>>>
>>>>>>>>>>>>                 We have multiple test rigs (we call these
>>>>>>>>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>>>>>>>>                 hardware testing on, with differing network
>>>>>>>>>>>>                 hardware and CPU architecture. I figured
>>>>>>>>>>>>                 this might
>>>>>>>>>>>>                 be an important detail.
>>>>>>>>>>>>
>>>>>>>>>>>>                 Thanks,
>>>>>>>>>>>>                 Adam
>>>>>>>>>>>>
>>>>>>>>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew
>>>>>>>>>>>>                 Rybchenko
>>>>>>>>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>>>>>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Greatings Adam,
>>>>>>>>>>>>
>>>>>>>>>>>>                                     I'm happy to hear that
>>>>>>>>>>>>                 you're trying to bring
>>>>>>>>>>>>                                     it up.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     As I understand the
>>>>>>>>>>>>                 final goal is to run it on
>>>>>>>>>>>>                                     regular basis. So, we
>>>>>>>>>>>>                 need to make it properly
>>>>>>>>>>>>                                     from the very beginning.
>>>>>>>>>>>>                                     Bring up of all
>>>>>>>>>>>>                 features consists of 4 steps:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     1. Create site-specific
>>>>>>>>>>>>                 repository (we call it
>>>>>>>>>>>>                                     ts-rigs) which contains
>>>>>>>>>>>>                 information about test
>>>>>>>>>>>>                                     rigs and other
>>>>>>>>>>>>                 site-specific information like
>>>>>>>>>>>>                                     where to send mails,
>>>>>>>>>>>>                 where to store logs etc.
>>>>>>>>>>>>                                     It is required for
>>>>>>>>>>>>                 manual execution as well,
>>>>>>>>>>>>                                     since test rigs
>>>>>>>>>>>>                 description is essential. I'll
>>>>>>>>>>>>                                     return to the topic below.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     2. Setup logs storage
>>>>>>>>>>>>                 for automated runs.
>>>>>>>>>>>>                                     Basically it is a disk
>>>>>>>>>>>>                 space plus apache2 web
>>>>>>>>>>>>                                     server with few CGI
>>>>>>>>>>>>                 scripts which help a lot to
>>>>>>>>>>>>                                     save disk space.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     3. Setup Bublik web
>>>>>>>>>>>>                 application which provides
>>>>>>>>>>>>                                     web interface to view
>>>>>>>>>>>>                 testing results. Same as
>>>>>>>>>>>>                 https://ts-factory.io/bublik
>>>>>>>>>>>>                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>                 <https://ts-factory.io/bublik>
>>>>>>>>>>>>
>>>>>>>>>>>>                                     4. Setup Jenkins to run
>>>>>>>>>>>>                 tests on regularly,
>>>>>>>>>>>>                                     save logs in log
>>>>>>>>>>>>                 storage (2) and import it to
>>>>>>>>>>>>                                     bublik (3).
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Last few month we spent
>>>>>>>>>>>>                 on our homework to make
>>>>>>>>>>>>                                     it simpler to bring up
>>>>>>>>>>>>                 automated execution
>>>>>>>>>>>>                                     using Jenkins -
>>>>>>>>>>>>                 https://github.com/ts-factory/te-jenkins
>>>>>>>>>>>>                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>                 <https://github.com/ts-factory/te-jenkins>
>>>>>>>>>>>>                                     Corresponding bits in
>>>>>>>>>>>>                 dpdk-ethdev-ts will be
>>>>>>>>>>>>                                     available tomorrow.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Let's return to the
>>>>>>>>>>>>                 step (1).
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Unfortunately there is
>>>>>>>>>>>>                 no publicly available
>>>>>>>>>>>>                                     example of the ts-rigs
>>>>>>>>>>>>                 repository since
>>>>>>>>>>>>                                     sensitive site-specific
>>>>>>>>>>>>                 information is located
>>>>>>>>>>>>                                     there. But I'm ready to
>>>>>>>>>>>>                 help you to create it
>>>>>>>>>>>>                                     for UNH. I see two
>>>>>>>>>>>>                 options here:
>>>>>>>>>>>>
>>>>>>>>>>>>                                     (A) I'll ask questions
>>>>>>>>>>>>                 and based on your
>>>>>>>>>>>>                                     answers will create the
>>>>>>>>>>>>                 first draft with my
>>>>>>>>>>>>                                     comments.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     (B) I'll make a
>>>>>>>>>>>>                 template/example ts-rigs repo,
>>>>>>>>>>>>                                     publish it and you'll
>>>>>>>>>>>>                 create UNH ts-rigs based
>>>>>>>>>>>>                                     on it.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Of course, I'll help to
>>>>>>>>>>>>                 debug and finally bring
>>>>>>>>>>>>                                     it up in any case.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     (A) is a bit simpler
>>>>>>>>>>>>                 for me and you, but (B) is
>>>>>>>>>>>>                                     a bit more generic and
>>>>>>>>>>>>                 will help other
>>>>>>>>>>>>                                     potential users to
>>>>>>>>>>>>                 bring it up.
>>>>>>>>>>>>                                     We can combine (A)+(B).
>>>>>>>>>>>>                 I.e. start from (A).
>>>>>>>>>>>>                                     What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>                                     Andrew.
>>>>>>>>>>>>
>>>>>>>>>>>>                                     On 8/17/23 15:18,
>>>>>>>>>>>>                 Konstantin Ushakov wrote:
>>>>>>>>>>>>>                 Greetings Adam,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Thanks for contacting
>>>>>>>>>>>>>                 us. I copy Andrew who
>>>>>>>>>>>>>                                     would be happy to help
>>>>>>>>>>>>>
>>>>>>>>>>>>>                                     Thanks,
>>>>>>>>>>>>>                                     Konstantin
>>>>>>>>>>>>>
>>>>>>>>>>>>>>                 On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>>>>>>>>>>                 <ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     
>>>>>>>>>>>>>>                                     Greetings Konstantin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I am in the process
>>>>>>>>>>>>>>                 of setting up the DPDK
>>>>>>>>>>>>>>                                     Poll Mode Driver test
>>>>>>>>>>>>>>                 suite as an addition to
>>>>>>>>>>>>>>                                     our testing coverage
>>>>>>>>>>>>>>                 for DPDK at the UNH lab.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I have some questions
>>>>>>>>>>>>>>                 about how to set the
>>>>>>>>>>>>>>                                     test suite arguments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I have been able to
>>>>>>>>>>>>>>                 configure the Test Engine
>>>>>>>>>>>>>>                                     to connect to the
>>>>>>>>>>>>>>                 hosts in the testbed. The
>>>>>>>>>>>>>>                                     RCF, Configurator,
>>>>>>>>>>>>>>                 and Tester all begin to
>>>>>>>>>>>>>>                                     run, however the
>>>>>>>>>>>>>>                 prelude of the test suite
>>>>>>>>>>>>>>                                     fails to run.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                 https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>>>>>>>>>>>                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>                 <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     The documentation
>>>>>>>>>>>>>>                 mentions that there are
>>>>>>>>>>>>>>                                     several test
>>>>>>>>>>>>>>                 parameters for the test suite,
>>>>>>>>>>>>>>                                     like for the IUT test
>>>>>>>>>>>>>>                 link MAC, etc. These
>>>>>>>>>>>>>>                                     seem like they would
>>>>>>>>>>>>>>                 need to be set somewhere
>>>>>>>>>>>>>>                                     to run many of the
>>>>>>>>>>>>>>                 tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I see in the Test
>>>>>>>>>>>>>>                 Engine documentation, there
>>>>>>>>>>>>>>                                     are instructions on
>>>>>>>>>>>>>>                 how to create new
>>>>>>>>>>>>>>                                     parameters for test
>>>>>>>>>>>>>>                 suites in the Tester
>>>>>>>>>>>>>>                                     configuration, but
>>>>>>>>>>>>>>                 there is nothing in the
>>>>>>>>>>>>>>                                     user guide or in the
>>>>>>>>>>>>>>                 Tester guide for how to
>>>>>>>>>>>>>>                                     set the arguments for
>>>>>>>>>>>>>>                 the parameters when
>>>>>>>>>>>>>>                                     running the test
>>>>>>>>>>>>>>                 suite that I can find. I'm
>>>>>>>>>>>>>>                                     not sure if I need to
>>>>>>>>>>>>>>                 write my own Tester
>>>>>>>>>>>>>>                                     config, or if I
>>>>>>>>>>>>>>                 should be setting these in
>>>>>>>>>>>>>>                                     some other way.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     How should these
>>>>>>>>>>>>>>                 values be set?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     I'm also not sure
>>>>>>>>>>>>>>                 what environment
>>>>>>>>>>>>>>                                     variables/arguments
>>>>>>>>>>>>>>                 are strictly necessary or
>>>>>>>>>>>>>>                                     which are optional.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     Regards,
>>>>>>>>>>>>>>                                     Adam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                     --
>>>>>>>>>>>>>>                                     *Adam Hassick*
>>>>>>>>>>>>>>                                     Senior Developer
>>>>>>>>>>>>>>                                     UNH InterOperability Lab
>>>>>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>>>                                     +1 (603) 475-8248
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 -- *Adam Hassick*
>>>>>>>>>>>>                 Senior Developer
>>>>>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>>>                 +1 (603) 475-8248
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                             --             *Adam Hassick*
>>>>>>>>>>                             Senior Developer
>>>>>>>>>>                             UNH InterOperability Lab
>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                             +1 (603) 475-8248
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                         -- *Adam Hassick*
>>>>>>>>>>                         Senior Developer
>>>>>>>>>>                         UNH InterOperability Lab
>>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>>                         +1 (603) 475-8248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                     --     *Adam Hassick*
>>>>>>>>>                     Senior Developer
>>>>>>>>>                     UNH InterOperability Lab
>>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>>                     +1 (603) 475-8248
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 -- 
>>>>>>>>                 *Adam Hassick*
>>>>>>>>                 Senior Developer
>>>>>>>>                 UNH InterOperability Lab
>>>>>>>>                 ahassick@iol.unh.edu
>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>                 <mailto:ahassick@iol.unh.edu>
>>>>>>>>                 iol.unh.edu <http://iol.unh.edu>
>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>                 <https://www.iol.unh.edu/>
>>>>>>>>                 +1 (603) 475-8248
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             -- 
>>>>>>             *Adam Hassick*
>>>>>>             Senior Developer
>>>>>>             UNH InterOperability Lab
>>>>>>             ahassick@iol.unh.edu
>>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>>>             +1 (603) 475-8248
>>>>>
>>>>
>>>>
>>>>
>>>>         -- 
>>>>         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>
>>
>>
>>
>>     -- 
>>     *Adam Hassick*
>>     Senior Developer
>>     UNH InterOperability Lab
>>     ahassick@iol.unh.edu
>>     iol.unh.edu <https://www.iol.unh.edu/>
>>     +1 (603) 475-8248
>

[-- Attachment #2: Type: text/html, Size: 115388 bytes --]

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: Setting up DPDK PMD Test Suite
  2023-09-01  7:59                                             ` Andrew Rybchenko
@ 2023-09-05 15:01                                               ` Adam Hassick
  2023-09-06 11:36                                                 ` Andrew Rybchenko
  0 siblings, 1 reply; 51+ messages in thread
From: Adam Hassick @ 2023-09-05 15:01 UTC (permalink / raw)
  To: Andrew Rybchenko; +Cc: Patrick Robb, Konstantin Ushakov, ci


[-- Attachment #1.1: Type: text/plain, Size: 37124 bytes --]

Hi Andrew,

The compilation warning issue is now resolved. Again, thank you guys for
fixing this for us. I can run the tests on the Mellanox CX5s again, however
I'm running into a couple new issues with running the prologues on the
Intel cards.

When running testing on the Intel XL710s, I see this error appear in the
log:

ERROR  prologue  Environment LIB  14:16:13.650
> Too few networks in available configuration (0) in comparison with
> required (1)
>

This seems like a trivial configuration error, perhaps this is something I
need to set up in ts-rigs. I briefly searched through the examples there
and didn't see any mention of how to set up a network.
I will attach this log just in case you need more information.

There is a different error when running on the Intel E810s. It appears to
me like it starts DPDK, does some configuration inside DPDK and on the
device, and then fails to bring the device back up. Since this error seems
very non-trivial, I will also attach this log.

Thanks,
Adam

On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hi Adam,
>
> On 8/31/23 22:38, Adam Hassick wrote:
>
> Hi Andrew,
>
> I have one additional question as well: Does the test engine support
> running tests on two ARMv8 test agents?
>
> 1. We'll sort out warnings this week. Thanks for heads up.
>>
>
> Great. Let me know when that's fixed.
>
>
> Done. We also fixed a number of warnings in TE.
> Also we fixed root test package name to be consistent with the repository
> name.
>
> Support for old LTS branches was dropped some time ago, but in the future
>> it is definitely possible to keep it for new LTS branches. I think 22.11 is
>> supported, but I'm not sure about older LTS releases.
>>
>
> Good to know.
>
>
>> 2. You can add command-line option --sanity to run tests marked with
>> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and grep
>> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
>> there is a space for terminology improvement here. We'll do it.
>>
>
> Done. Now it is called --checkup.
>
>
>> Also it takes a lot of time because of failures and tests which wait for
>> some timeout.
>>
>
> That makes sense to me. We'll use the time to complete tests on virtio or
> the Intel devices as a reference for how long the tests really take to
> complete.
> We will explore the possibility of periodically running the sanity tests
> for patches.
>
>
> I'll double-check and let you know how long entire TS runs on Intel X710,
> E810, Mellanox CX5 and virtio net. Just to ensure that time observed in
> your case looks the same.
>
>
>
>> The test harness can provide coverage reports based on gcov, but I'm not
>> sure what you mean by a "dial" to control test coverage. Provided reports
>> are rather for human to analyze.
>>
>
> The general idea is to have some kind of parameter on the test suite,
> which could be an integer ranging from zero to ten, that controls how many
> tests are run based on how important the test is.
>
> Similar to how some command line interfaces provide a verbosity level
> parameter (some number of "-v" arguments) to control the importance of the
> information in the log.
> The verbosity level zero only prints very important log messages, while
> ten prints everything.
>
> In much the same manner as above, this "dial" parameter controls what
> tests are run and with what parameters based on how important those tests
> and test parameter combinations are.
> Coverage Level zero tells the suite to run a very basic set of important
> tests, with minimal parameterization. This mode would take only ~5-10
> minutes to run.
> In contrast, Coverage Level ten includes all the edge cases, every
> combination of test parameters, everything the test suite can do, which
> takes the normal several hours to run.
> The values 1 - 9 are between those two extremes, allowing the user to get
> a gradient of test coverage in the results and to limit the running time.
>
> Then we could, for example, run the "run.sh" with a level of 2 or 3 for
> incoming patches that need quick results, and with a level of 10 for the
> less often run periodic tests performed on main or LTS branches.
>
>
> Understood now. Thanks a lot for the idea. We'll discuss it and come back.
>
>
>
>> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing
>> results. Unfortunately it is time consuming to fill in expectations
>> database since it is necessary to analyze testing results and classify if
>> it is a bug or just acceptable behaviour aspect.
>>
>> Bublik allows to compare results of two runs. It is useful for human, but
>> still not good for automation.
>>
>> I have local patch for mlx5 driver which reports Tx ring size maximum. It
>> makes pass rate higher. It is a problem for test harness that mlx5 does not
>> report limits right now.
>>
>> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio
>> net is 99% right now and could be done 100% easily (just one thing to fix
>> in expectations).
>>
>> I think logs storage setup is essential for logs analysis. Of course, you
>> can request HTML logs when you run tests (--log-html=html) or generate
>> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
>> browser, but logs storage makes it more convenient.
>>
>
> We are interested in setting up Bublik, potentially as an
> externally-facing component, once we have our process of running the test
> suite stabilized.
> Once we are able to run the test suite again, I'll see what the pass rate
> is on our other hardware.
> Good to know that it isn't an issue with our dev testbed causing the high
> fail rate.
>
> For Intel hardware, we have an XL710 and an Intel E810-C in our
> development testbed. Although they are slightly different devices, ideally
> the pass rate will be identical or similar. I have yet to set up a VM pair
> for virtio, but we will soon.
>
> Latest version of test-environment has examples of our CGI scripts which
>> we use for log storage (see tools/log_server/README.md).
>>
>> Also all bits for Jenkins setup are available. See
>> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
>> ts-rigs-sample.
>>
>
> Jenkins integration, setting up production rig configurations, and
> permanent log storage will be our next steps once I am able to run the
> tests again.
> Unless there is an easy way to have meson not pass "-Werror" into GCC.
> Then I would be able to run the test suite.
>
>
> Hopefully it is resolved now.
>
> I thought a bit more about your usecase for Jenkins. I'm not 100% sure
> that existing pipelines are convenient for your usecase.
> Fill free to ask questions when you are on it.
>
> Thanks,
> Andrew.
>
>
> Thanks,
> Adam
>
>
>
>>
>> On 8/29/23 17:02, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> That fix seems to have resolved the issue, thanks for the quick
>> turnaround time on that patch.
>> Now that we have the RCF timeout issue resolved, there are a few other
>> questions and issues that we have about the tests themselves.
>>
>> 1. The test suite fails to build with a couple warnings.
>>
>> Below is the stderr log from compilation:
>>
>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>>> ../../lib/dpdk_pmd_ts.c: In function
>>> ‘test_create_traffic_generator_params’:
>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>> no format arguments [-Werror=format-security]
>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>> |     ^~
>>> cc1: all warnings being treated as errors
>>> ninja: build stopped: subcommand failed.
>>> ninja: Entering directory `.'
>>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>>> -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
>>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>>> ../../lib/dpdk_pmd_ts.c: In function
>>> ‘test_create_traffic_generator_params’:
>>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
>>> no format arguments [-Werror=format-security]
>>> 5577 |     rc = te_kvpair_add(result, buf, mode);
>>> |     ^~
>>> cc1: all warnings being treated as errors
>>>
>>
>> This error wasn't occurring last week, which was the last time I ran the
>> tests.
>> The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC
>> v11.4.0 installed, if this information is helpful.
>>
>> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
>> collectively take around 9 hours. Is it possible, and would it make sense,
>> to lower the test coverage and have the test suite run faster?
>>
>> For some context, we run immediate testing on incoming patches for DPDK
>> main and development branches, as well as periodic test runs on the main,
>> stable, and LTS branches.
>> For us to consider including this test suite as part of our immediate
>> testing on patches, we would have to reduce the test coverage to the most
>> important tests.
>> This is primarily to reduce the testing time to, for example, less than
>> 30 minutes. Testing on patches can't take too long because the lab can
>> receive numerous patches each day, which each require individual testing
>> runs.
>>
>> At what frequency we run these tests, and on what, still needs to be
>> discussed with the DPDK community, but it would be nice to know if the test
>> suite had a "dial" to control the testing coverage.
>>
>> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300
>> of ~6,600 tests passed. Is there anything we can do to diagnose these test
>> failures?
>>
>> Thanks,
>> Adam
>>
>>
>> On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> Hi Adam,
>>>
>>> I've pushed the fix in main branch and a new tag v1.18.1. It should
>>> solve the problem with IPv6 address from DNS.
>>>
>>> Andrew.
>>>
>>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>>
>>> Hi Adam,
>>>
>>> > Does the test engine prefer to use IPv6 over IPv4 for initiating the
>>> RCF connection to the test bed hosts? And if so, is there a way to force it
>>> to use IPv4?
>>>
>>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your
>>> case, I guess it is the root cause of the problem.
>>> Of course, it is TE problem since I see really weird code in
>>> lib/comm_net_engine/comm_net_engine.c line 135.
>>>
>>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>>> ts-factory/test-environment repository. Please, try.
>>>
>>> It is late night fix with minimal testing and no review. I'll pass it
>>> through review process tomorrow and
>>> hopefully it will be released in one-two days.
>>>
>>> Andrew.
>>>
>>> On 8/28/23 18:02, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> We have yet to notice a distinct pattern with the failures. Sometimes,
>>> the RCF will start and connect without issue a few times in a row before
>>> failing to connect again. Once the issue begins to occur, neither rebooting
>>> all of the hosts (test engine VM, tester, IUT) or deleting all of the build
>>> directories (suites, agents, inst) and rebooting the hosts afterward
>>> resolves the issue. When it begins working again seems very arbitrary to us.
>>>
>>> I do usually try to terminate the test engine with Ctrl+C, but when it
>>> hangs while trying to start RCF, that does not work.
>>>
>>> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
>>> connection to the test bed hosts? And if so, is there a way to force it to
>>> use IPv4?
>>>
>>>  - Adam
>>>
>>> On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>
>>>> Done. It works fine for me without any issues.
>>>>
>>>> Have you noticed any pattern when it works or does not work?
>>>> May be it is a problem of not clean state after termination?
>>>> Does it work fine the first time after DUTs reboot?
>>>> How do you terminate testing? It should be done using Ctrl+C in
>>>> terminal where you execute run.sh command.
>>>>  In this case it should shutdown gracefully and close all test agents
>>>> and engine applications.
>>>>
>>>> (I'm trying to understand why you've seen many test agent processes. It
>>>> should not happen.)
>>>>
>>>> Andrew.
>>>>
>>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>>
>>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> Two of our systems (the Test Engine runner and the DUT host) are
>>>> running Ubuntu 20.04 LTS, however this morning I noticed that the tester
>>>> system (the one having issues) is running Ubuntu 22.04 LTS.
>>>> This could be the source of the problem. I encountered a dependency
>>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>>> system. Since the tester is also the host having connection issues, I will
>>>> try downgrading that system to 20.04, and see if that changes anything.
>>>>
>>>>
>>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>>
>>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>>
>>>> I did try passing in the "--vg-rcf" argument to the run.sh script of
>>>> the test suite after installing valgrind, but there was no additional
>>>> output that I saw.
>>>>
>>>>
>>>> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
>>>> where you run test engine).
>>>>
>>>>
>>>> I will try pulling in the changes you've pushed up, and will see if
>>>> that fixes anything.
>>>>
>>>> Thanks,
>>>> Adam
>>>>
>>>> On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>>
>>>>> Hello Adam,
>>>>>
>>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>>
>>>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>>>> running on test engine and agents?
>>>>>
>>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>>> patch in review to fix it, however, the behaviour is different in
>>>>> this case, so it is unlike the same problem.
>>>>>
>>>>>
>>>>> I've just published a new tag which fixes known test engine side
>>>>> problems on Debian 12 and Fedora 38.
>>>>>
>>>>>
>>>>> One more idea is to install valgrind on the test engine host and
>>>>> run with option --vg-rcf to check if something weird is happening.
>>>>>
>>>>> What I don't understand right now is why I see just one failed attempt
>>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>>
>>>>>  > Is there any firewall in the network or on test hosts which could
>>>>> block incoming TCP connection to the port 23571
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>>>>> you run test engine?
>>>>>
>>>>> Our test engine host and the testbed are on the same subnet. The
>>>>> connection does work sometimes.
>>>>>
>>>>>  > If behaviour the same on the next try and you see that test agent
>>>>> is kept running, could you check using
>>>>>  >
>>>>>  > # netstat -tnlp
>>>>>  >
>>>>>  > that Test Agent is listening on the port and try to establish TCP
>>>>> connection from test agent using
>>>>>  >
>>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>  >
>>>>>  > and check if TCP connection could be established.
>>>>>
>>>>> I was able to replicate the same behavior again, where it hangs while
>>>>> RCF is trying to start.
>>>>> Running this command, I see this in the output:
>>>>>
>>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>>  18599/ta
>>>>>
>>>>> So it seems like it is listening on the correct port.
>>>>> Additionally, I was able to connect to the Tester machine from our
>>>>> Test Engine host using telnet. It printed the PID of the process once the
>>>>> connection was opened.
>>>>>
>>>>> I tried running the "ta" application manually on the command line, and
>>>>> it didn't print anything at all.
>>>>> Maybe the issue is something on the Test Engine side.
>>>>>
>>>>> On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <
>>>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>     Hi Adam,
>>>>>
>>>>>      > On the tester host (which appears to be the Peer agent), there
>>>>>     are four processes that I see running, which look like the test
>>>>>     agent processes.
>>>>>
>>>>>     Before the next try I'd recommend to kill these processes.
>>>>>
>>>>>     Is there any firewall in the network or on test hosts which could
>>>>>     block incoming TCP connection to the port 23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>>     where you run test engine?
>>>>>
>>>>>     If behaviour the same on the next try and you see that test agent
>>>>> is
>>>>>     kept running, could you check using
>>>>>
>>>>>     # netstat -tnlp
>>>>>
>>>>>     that Test Agent is listening on the port and try to establish TCP
>>>>>     connection from test agent using
>>>>>
>>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>     and check if TCP connection could be established.
>>>>>
>>>>>     Another idea is to login Tester under root as testing does, get
>>>>>     start TA command from the log and try it by hands without -n and
>>>>>     remove extra escaping.
>>>>>
>>>>>     # sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>>
>>>>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
>>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>>> host=iol-dts-tester.dpdklab.iol.unh.edu:
>>>>> port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=
>>>>>
>>>>>     Hopefully in this case test agent directory remains in the /tmp
>>>>> and
>>>>>     you don't need to copy it as testing does.
>>>>>     May be output could shed some light on what's going on.
>>>>>
>>>>>     Andrew.
>>>>>
>>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>>
>>>>>     Hi Andrew,
>>>>>
>>>>>     This is the output that I see in the terminal when this failure
>>>>>     occurs, after the test agent binaries build and the test engine
>>>>>     starts:
>>>>>
>>>>>     Platform default build - pass
>>>>>     Simple RCF consistency check succeeded
>>>>>     --->>> Starting Logger...done
>>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>>
>>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>>     processes. I let it hang for around 9 minutes.
>>>>>
>>>>>     On the tester host (which appears to be the Peer agent), there are
>>>>>     four processes that I see running, which look like the test agent
>>>>>     processes.
>>>>>
>>>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>>>
>>>>>      - Adam
>>>>>
>>>>>     On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
>>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>         Hi Adam,
>>>>>
>>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>>         says that copy_timeout is in seconds and implementation in
>>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>>         Theoretically select() could be interrupted by signal, but I
>>>>>         think it is unlikely here.
>>>>>
>>>>>         I'm not sure that I understand what do you mean by RCF
>>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>>         does it happen when tests are in progress, e.g. in the middle
>>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>>         likely either host with test agent dies or test agent itself
>>>>>         crashes. It would be easier for me if classify it if you share
>>>>>         text log (log.txt, full or just corresponding fragment with
>>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>>         depending on which agent has problems could shed some light.
>>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>>
>>>>>         Andrew.
>>>>>
>>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>>
>>>>>         Hi Andrew,
>>>>>
>>>>>         I've set up a test rig repository here, and have created
>>>>>         configurations for our development testbed based off of the
>>>>>         examples.
>>>>>         We've been able to get the test suite to run manually on
>>>>>         Mellanox CX5 devices once.
>>>>>         However, we are running into an issue where, when RCF starts,
>>>>>         the RCF connection times out very frequently. We aren't sure
>>>>>         why this is the case.
>>>>>         It works sometimes, but most of the time when we try to run
>>>>>         the test engine, it encounters this issue.
>>>>>         I've tried changing the RCF port by setting
>>>>>         "TE_RCF_PORT=<some port number>" and rebooting the testbed
>>>>>         machines. Neither seems to fix the issue.
>>>>>
>>>>>         It also seems like the timeout takes far longer than 60
>>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=60"
>>>>>         before I try to run the test suite.
>>>>>         I assume the unit for this variable is seconds?
>>>>>
>>>>>         Thanks,
>>>>>         Adam
>>>>>
>>>>>         On Mon, Aug 21, 2023 at 10:19 AM Adam Hassick
>>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>> wrote:
>>>>>
>>>>>             Hi Andrew,
>>>>>
>>>>>             Thanks, I've cloned the example repository and will start
>>>>>             setting up a configuration for our development testbed
>>>>>             today. I'll let you know if I run into any difficulties
>>>>>             or have any questions.
>>>>>
>>>>>              - Adam
>>>>>
>>>>>             On Sun, Aug 20, 2023 at 4:40 AM Andrew Rybchenko
>>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                 Hi Adam,
>>>>>
>>>>>                 I've published
>>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>>                 Hopefully it will help to define your test rigs and
>>>>>                 successfully run some tests manually. Feel free to
>>>>>                 ask any questions and I'll answer here and try to
>>>>>                 update documentation.
>>>>>
>>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>>                 (3).
>>>>>                 Hopefully everything is in place for step (4), but we
>>>>>                 need to make steps (2) and (3) first.
>>>>>
>>>>>                 Andrew.
>>>>>
>>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>>
>>>>>                 Hi Adam,
>>>>>
>>>>>                 > I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>
>>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>>                 almost ready right now, but I need to double-check
>>>>>                 it before publishing.
>>>>>
>>>>>                 Regards,
>>>>>                 Andrew.
>>>>>
>>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>>
>>>>>                 Hi Andrew,
>>>>>
>>>>>                 I'm adding the CI mailing list to this
>>>>>                 conversation. Others in the community might find
>>>>>                 this conversation valuable.
>>>>>
>>>>>                 We do want to run testing on a regular basis. The
>>>>>                 Jenkins integration will be very useful for us, as
>>>>>                 most of our CI is orchestrated by Jenkins.
>>>>>                 I've conferred with the rest of the team, and we
>>>>>                 think it would be best to move forward with mainly
>>>>>                 option B.
>>>>>                 If you would like to know anything about our
>>>>>                 testbeds that would help you with creating an
>>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>>                 questions you have.
>>>>>
>>>>>                 We have multiple test rigs (we call these
>>>>>                 "DUT-tester pairs") that we run our existing
>>>>>                 hardware testing on, with differing network
>>>>>                 hardware and CPU architecture. I figured this might
>>>>>                 be an important detail.
>>>>>
>>>>>                 Thanks,
>>>>>                 Adam
>>>>>
>>>>>                 On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko
>>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>>
>>>>>                     Greatings Adam,
>>>>>
>>>>>                     I'm happy to hear that you're trying to bring
>>>>>                     it up.
>>>>>
>>>>>                     As I understand the final goal is to run it on
>>>>>                     regular basis. So, we need to make it properly
>>>>>                     from the very beginning.
>>>>>                     Bring up of all features consists of 4 steps:
>>>>>
>>>>>                     1. Create site-specific repository (we call it
>>>>>                     ts-rigs) which contains information about test
>>>>>                     rigs and other site-specific information like
>>>>>                     where to send mails, where to store logs etc.
>>>>>                     It is required for manual execution as well,
>>>>>                     since test rigs description is essential. I'll
>>>>>                     return to the topic below.
>>>>>
>>>>>                     2. Setup logs storage for automated runs.
>>>>>                     Basically it is a disk space plus apache2 web
>>>>>                     server with few CGI scripts which help a lot to
>>>>>                     save disk space.
>>>>>
>>>>>                     3. Setup Bublik web application which provides
>>>>>                     web interface to view testing results. Same as
>>>>>                     https://ts-factory.io/bublik
>>>>>                     <https://ts-factory.io/bublik>
>>>>> <https://ts-factory.io/bublik>
>>>>>
>>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>>                     save logs in log storage (2) and import it to
>>>>>                     bublik (3).
>>>>>
>>>>>                     Last few month we spent on our homework to make
>>>>>                     it simpler to bring up automated execution
>>>>>                     using Jenkins -
>>>>>                     https://github.com/ts-factory/te-jenkins
>>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>>> <https://github.com/ts-factory/te-jenkins>
>>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>>                     available tomorrow.
>>>>>
>>>>>                     Let's return to the step (1).
>>>>>
>>>>>                     Unfortunately there is no publicly available
>>>>>                     example of the ts-rigs repository since
>>>>>                     sensitive site-specific information is located
>>>>>                     there. But I'm ready to help you to create it
>>>>>                     for UNH. I see two options here:
>>>>>
>>>>>                     (A) I'll ask questions and based on your
>>>>>                     answers will create the first draft with my
>>>>>                     comments.
>>>>>
>>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>>                     publish it and you'll create UNH ts-rigs based
>>>>>                     on it.
>>>>>
>>>>>                     Of course, I'll help to debug and finally bring
>>>>>                     it up in any case.
>>>>>
>>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>>                     a bit more generic and will help other
>>>>>                     potential users to bring it up.
>>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>>                     What do you think?
>>>>>
>>>>>                     Thanks,
>>>>>                     Andrew.
>>>>>
>>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>>
>>>>>                     Greetings Adam,
>>>>>
>>>>>
>>>>>                     Thanks for contacting us. I copy Andrew who
>>>>>                     would be happy to help
>>>>>
>>>>>                     Thanks,
>>>>>                     Konstantin
>>>>>
>>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu> wrote:
>>>>>
>>>>>                     
>>>>>                     Greetings Konstantin,
>>>>>
>>>>>                     I am in the process of setting up the DPDK
>>>>>                     Poll Mode Driver test suite as an addition to
>>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>>
>>>>>                     I have some questions about how to set the
>>>>>                     test suite arguments.
>>>>>
>>>>>                     I have been able to configure the Test Engine
>>>>>                     to connect to the hosts in the testbed. The
>>>>>                     RCF, Configurator, and Tester all begin to
>>>>>                     run, however the prelude of the test suite
>>>>>                     fails to run.
>>>>>
>>>>>
>>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>>
>>>>>                     The documentation mentions that there are
>>>>>                     several test parameters for the test suite,
>>>>>                     like for the IUT test link MAC, etc. These
>>>>>                     seem like they would need to be set somewhere
>>>>>                     to run many of the tests.
>>>>>
>>>>>                     I see in the Test Engine documentation, there
>>>>>                     are instructions on how to create new
>>>>>                     parameters for test suites in the Tester
>>>>>                     configuration, but there is nothing in the
>>>>>                     user guide or in the Tester guide for how to
>>>>>                     set the arguments for the parameters when
>>>>>                     running the test suite that I can find. I'm
>>>>>                     not sure if I need to write my own Tester
>>>>>                     config, or if I should be setting these in
>>>>>                     some other way.
>>>>>
>>>>>                     How should these values be set?
>>>>>
>>>>>                     I'm also not sure what environment
>>>>>                     variables/arguments are strictly necessary or
>>>>>                     which are optional.
>>>>>
>>>>>                     Regards,
>>>>>                     Adam
>>>>>
>>>>>                     --                     *Adam Hassick*
>>>>>                     Senior Developer
>>>>>                     UNH InterOperability Lab
>>>>>                     ahassick@iol.unh.edu
>>>>>                     <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>                     +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 --                 *Adam Hassick*
>>>>>                 Senior Developer
>>>>>                 UNH InterOperability Lab
>>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>                 +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>             --             *Adam Hassick*
>>>>>             Senior Developer
>>>>>             UNH InterOperability Lab
>>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>             +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>         --         *Adam Hassick*
>>>>>         Senior Developer
>>>>>         UNH InterOperability Lab
>>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>>> <https://www.iol.unh.edu/>
>>>>>         +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>     --     *Adam Hassick*
>>>>>     Senior Developer
>>>>>     UNH InterOperability Lab
>>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>>     +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Adam Hassick*
>>>>> Senior Developer
>>>>> UNH InterOperability Lab
>>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>>> <ahassick@iol.unh.edu>
>>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>> +1 (603) 475-8248
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu
>>>> iol.unh.edu <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>

-- 
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

[-- Attachment #1.2: Type: text/html, Size: 107232 bytes --]

[-- Attachment #2: intel-e810-main-fail.txt --]
[-- Type: text/plain, Size: 369975 bytes --]

Log report
~~~~~~~~~~

RING  Dispatcher  Command-line options  13:01:03.175
--conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --tester-req=TEST_HARNESS_CHECKUP --tester-req=!NO_TEST_HARNESS_CHECKUP --opts=run/iol-dts-e810 --opts=opts.ts

RING  Dispatcher  Expanded command-line options  13:01:03.188
 --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --trc-html=trc-brief.html --trc-no-expected --trc-no-total --trc-no-unspec --trc-keep-artifacts --tester-req=TEST_HARNESS_CHECKUP --tester-req=!NO_TEST_HARNESS_CHECKUP --script=env/iol-dts --script=env/intel-e810 --script=scripts/iut.h1 --script=scripts/iut.h1-e810 --conf-cs=cs/dpdk-pmd-ts.yml --script=scripts/ta-def --script=scripts/defaults --tester-script=scripts/dpdk-trc-tags --tester-script=scripts/os-trc-tags --script=scripts/net-modules --script=scripts/iut-net-driver-loaded --script=scripts/disable_unused_agts

RING  Dispatcher  Start  13:04:57.364
Starting TEN applications

RING  Dispatcher  Start  13:04:57.381
Start Logger:  /opt/tsf/ts-rigs/logger.conf

RING  Logger  Cfg file  13:04:57.393
Opening config file: /opt/tsf/ts-rigs/logger.conf

RING  Logger  Log streaming  13:04:57.393
Current listeners configuration:
Listeners:
Filters:

RING  Dispatcher  Start  13:04:57.410
Start RCF:  /opt/tsf/ts-conf/rcf.conf

RING  RCF  RCF Unix  13:04:57.415
Starting TA 'Peer' type 'linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell='

RING  RCF  RCF Unix  13:04:57.415
CMD to copy: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_353319_1693919097_1" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_35__kernel5_15_0_79__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_353319_1693919097_1 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  13:04:59.168
Command to detect shell name: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  13:04:59.421
Shell is: /bin/bash

RING  RCF  RCF Unix  13:04:59.421
Command to start TA: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-tester.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_353319_1693919097_1 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_353319_1693919097_1 /tmp/linux_x86_root_353319_1693919097_1/ta Peer 23571 host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=" 2>&1 | te_tee RCF Peer 10 >ta.Peer 

RING  @Peer  Main  13:04:59.679
Starting

RING  @Peer  TA unix VM  13:04:59.680
KVM is supported

RING  RCF  RCF Unix  13:05:00.423
Starting TA 'DPDK' type 'linux_x86_64_linux_gnu__glibc2_31__kernel5_4_0_156__cpu_avx512bw__cpu_bmi2' conf_str 'host=iol-dts-dut.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:ssh_proxy=:copy_timeout=15:kill_timeout=15:sudo=:shell=:ld_preload='

RING  RCF  RCF Unix  13:05:00.423
CMD to copy: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "mkdir /tmp/linux_x86_root_353319_1693919100_2" && echo put /opt/tsf/dpdk-ethdev-ts/ts/inst/agents/linux_x86_64_linux_gnu__glibc2_31__kernel5_4_0_156__cpu_avx512bw__cpu_bmi2//. /tmp/linux_x86_root_353319_1693919100_2 | sftp -rpq -P 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu

RING  RCF  RCF Unix  13:05:08.494
Command to detect shell name: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "echo -n \$SHELL"

RING  RCF  RCF Unix  13:05:12.304
Shell is: /bin/bash

RING  RCF  RCF Unix  13:05:12.304
Command to start TA: ssh -qxT -o BatchMode=yes -p 22 -i /opt/tsf/keys/id_ed25519  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@iol-dts-dut.dpdklab.iol.unh.edu "sudo -n PATH=\${PATH}:/tmp/linux_x86_root_353319_1693919100_2 LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_353319_1693919100_2 /tmp/linux_x86_root_353319_1693919100_2/ta DPDK 23571 host=iol-dts-dut.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:ssh_proxy=:copy_timeout=15:kill_timeout=15:sudo=:shell=:ld_preload=" 2>&1 | te_tee RCF DPDK 10 >ta.DPDK 

RING  @DPDK  Main  13:05:16.111
Starting

WARN  @DPDK  TA unix VM  13:05:16.112
KVM is not supported

RING  Dispatcher  Start  13:05:16.328
Start Configurator: --conf-dirs=/opt/tsf/dpdk-ethdev-ts/conf:/opt/tsf/ts-rigs:/opt/tsf/ts-conf:/opt/tsf/dpdk-ethdev-ts/ts/inst/default/share/cm/ /opt/tsf/ts-conf/cs/dpdk-pmd-ts.yml

RING  Configurator  RCF API  13:05:22.938
Set /agent:DPDK/rpcprovider: to dpdkrpc: OK

RING  Configurator  RCF API  13:05:22.938
Set /agent:Peer/rpcprovider: to ta_rpcs: OK

RING  Configurator  RCF API  13:05:22.939
Set /agent:DPDK/sys:/core_pattern: to /var/tmp/core.te.%h-%p-%t: OK

RING  Configurator  RCF API  13:05:22.940
Set /agent:Peer/sys:/core_pattern: to /var/tmp/core.te.%h-%p-%t: OK

RING  Dispatcher  Start  13:05:38.273
Start Tester: --trc-db=/opt/tsf/dpdk-ethdev-ts/trc/top.xml --trc-comparison=normalised --req="TEST_HARNESS_CHECKUP" --req="!NO_TEST_HARNESS_CHECKUP" --trc-tag=dpdk-23.11.0-rc0 --trc-tag=dpdk:23110000 --trc-tag=vfio-pci --trc-tag=pci-8086-1592 --trc-tag=pci-8086 --trc-tag=pci-sub-8086-0002 --trc-tag=pci-sub-8086 --trc-tag=num_vfs:128 --trc-tag=kernel-linux --trc-tag=linux-mm:504 /opt/tsf/dpdk-ethdev-ts/conf/tester.conf

MI  Tester  Process Info  13:05:38.991
{
  "type": "tester_pid",
  "version": 1,
  "pid": 353414
}

RING  Tester  Target Requirements  13:05:38.991
TEST_HARNESS_CHECKUP & !NO_TEST_HARNESS_CHECKUP

MI  Tester  TRC tags  13:05:38.991
TRC tags:
 dpdk-23.11.0-rc0
 dpdk:23110000
 vfio-pci
 pci-8086-1592
 pci-8086
 pci-sub-8086-0002
 pci-sub-8086
 num_vfs:128
 kernel-linux
 linux-mm:504

RING  Tester  Self  13:05:38.991
Random seed is 1693919138

RING  Tester  Build  13:05:38.991
Build Test Suite 'dpdk-ethdev-ts' from '${TE_TS_SRC}'

RING  Tester  Self  13:05:41.745
Starting...

RING  Tester  Globals  13:05:41.745
Tester global variables list:
  env.peer2peer='net':IUT{'iut_host'{{'iut_rpcs':IUT,if:'iut_port'},addr:'iut_addr':inet:unicast,addr:'mcast_addr':ether:multicast,addr:'bcast_addr':ether:broadcast,addr:'iut_alien_mac':ether:alien, addr:'alien_addr':inet:alien},'tst_host'{{'tst_rpcs':tester},addr:'tst_addr':inet:unicast,if:'tst_if',addr:'tst_lladdr':ether:fake, addr:'tst_alien_mac':ether:alien}}
  env.peer2peer_ip6='net':IUT{'iut_host'{{'iut_rpcs':IUT,if:'iut_port'},addr:'iut_addr':inet6:unicast,addr:'mcast_addr':ether:multicast,addr:'bcast_addr':ether:broadcast,addr:'iut_alien_mac':ether:alien},'tst_host'{{'tst_rpcs':tester},addr:'tst_addr':inet6:unicast,if:'tst_if',addr:'tst_lladdr':ether:fake, addr:'tst_alien_mac':ether:alien}}
  tmpl.any.eth={
              pdus {
                eth:{
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2tst.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2mcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.mcast_addr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2bcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "param.iut_mac" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2mcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.mcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2bcast.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  length-type plain:8111
                }
              }
            }
  tmpl.iut2tst.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.icmp4={
                pdus {
                    icmp4:{
                        code plain: 1,
                        type plain: 1
                    },
                    ip4:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" },
                        protocol plain: 1
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.icmp4={
                pdus {
                    icmp4:{
                        code plain: 1,
                        type plain: 1
                    },
                    ip4:{
                        src-addr env:{ name "env.addr.iut_addr" },
                        dst-addr env:{ name "env.addr.tst_addr" },
                        protocol plain: 1
                    },
                    eth:{
                        dst-addr env:{ name "env.addr.tst_lladdr" },
                        src-addr env:{ name "param.iut_mac" }
                    }
                }
            }
  tmpl.tst2iut.udp6={
                pdus {
                    udp:{
                        src-port env:{ name "env.addr.tst_addr.port" },
                        dst-port env:{ name "env.addr.iut_addr.port" }
                    },
                    ip6:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" }
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" },
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.tcp6={
                pdus {
                    tcp:{
                        src-port env:{ name "env.addr.tst_addr.port" },
                        dst-port env:{ name "env.addr.iut_addr.port" },
                        seqn script:"expr:rand()",
                        ackn plain:0
                    },
                    ip6:{
                        src-addr env:{ name "env.addr.tst_addr" },
                        dst-addr env:{ name "env.addr.iut_addr" }
                    },
                    eth:{
                        dst-addr env:{ name "param.iut_mac" },
                        src-addr env:{ name "env.addr.tst_lladdr" }
                    }
                }
            }
  tmpl.iut2tst.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.iut2tst.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.iut_addr.port" },
                  dst-port env:{ name "env.addr.tst_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr env:{ name "env.addr.iut_addr" },
                  dst-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "param.iut_mac" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.bcast_addr" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vlan.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.udp4={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.tcp4={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.eth={
              pdus {
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_first={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_middle={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.ip4_frag_last={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" },
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr env:{ name "env.addr.tst_lladdr" },
                  snd-proto-addr env:{ name "env.addr.tst_addr" },
                  tgt-hw-addr env:{ name "param.iut_mac" },
                  tgt-proto-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.udp6={
              pdus {
                udp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq.tcp6={
              pdus {
                tcp:{
                  src-port env:{ name "env.addr.tst_addr.port" },
                  dst-port env:{ name "env.addr.iut_addr.port" },
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr env:{ name "env.addr.tst_addr" },
                  dst-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }

  tmpl.tst2iut.vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "par
  tmpl.tst2iut.vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
  tmpl.tst2iut.vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                e
  tmpl.tst2iut.vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                
  tmpl.tst2iut.vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }

  tmpl.tst2iut.vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "par
  tmpl.tst2iut.vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
  tmpl.tst2iut.vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                e
  tmpl.tst2iut.vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                
  tmpl.tst2iut.vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.vlan_vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.vlan_vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.vlan_vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
         
  tmpl.tst2iut.vlan_vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.vlan_vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.vlan_vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.vlan_vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
         
  tmpl.tst2iut.vlan_vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.vlan_vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_vxlan4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.qinq_vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.qinq_vxlan4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
               
  tmpl.tst2iut.qinq_vxlan4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
       
  tmpl.tst2iut.qinq_vxlan4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_vxlan4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.qinq_vxlan4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_vxlan6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladd
  tmpl.tst2iut.qinq_vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name
  tmpl.tst2iut.qinq_vxlan6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
               
  tmpl.tst2iut.qinq_vxlan6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
       
  tmpl.tst2iut.qinq_vxlan6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_llad
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_vxlan6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
           
  tmpl.tst2iut.qinq_vxlan6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env
  tmpl.tst2iut.geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" 
  tmpl.tst2iut.geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "p
  tmpl.tst2iut.geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr"
  tmpl.tst2iut.geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
               
  tmpl.tst2iut.geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.ad
  tmpl.tst2iut.geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" 
  tmpl.tst2iut.geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "p
  tmpl.tst2iut.geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr"
  tmpl.tst2iut.geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
               
  tmpl.tst2iut.geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
            
  tmpl.tst2iut.geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
              
  tmpl.tst2iut.geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.ad
  tmpl.tst2iut.vlan_geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.vlan_geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.vlan_geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
       
  tmpl.tst2iut.vlan_geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.vlan_geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.vlan_geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.vlan_geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.vlan_geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.vlan_geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.vlan_geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
       
  tmpl.tst2iut.vlan_geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.vlan_geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.vlan_geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.vlan_geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.vlan_geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.qinq_geneve4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.qinq_geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.qinq_geneve4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
             
  tmpl.tst2iut.qinq_geneve4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
     
  tmpl.tst2iut.qinq_geneve4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.qinq_geneve4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.qinq_geneve4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.qinq_geneve4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_geneve4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.qinq_geneve6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lla
  tmpl.tst2iut.qinq_geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ na
  tmpl.tst2iut.qinq_geneve6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
             
  tmpl.tst2iut.qinq_geneve6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
     
  tmpl.tst2iut.qinq_geneve6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_ll
  tmpl.tst2iut.qinq_geneve6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
          
  tmpl.tst2iut.qinq_geneve6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
       
  tmpl.tst2iut.qinq_geneve6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
         
  tmpl.tst2iut.qinq_geneve6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.tst_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "e
  tmpl.tst2iut.nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },

  tmpl.tst2iut.nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" 
  tmpl.tst2iut.nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
  tmpl.tst2iut.nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
        
  tmpl.tst2iut.nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr
  tmpl.tst2iut.nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" }
                }
              }
            }
  tmpl.tst2iut.nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },

  tmpl.tst2iut.nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" 
  tmpl.tst2iut.nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
  tmpl.tst2iut.nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
        
  tmpl.tst2iut.vlan_nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vl
  tmpl.tst2iut.vlan_nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    v
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.vlan_nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.vlan_nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.vlan_nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vl
  tmpl.tst2iut.vlan_nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.vlan_nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    vlan-id plain:210
                  }
                }
              }
            }
  tmpl.tst2iut.vlan_nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged tagged:{
                    v
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.vlan_nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.vlan_nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.qinq_nvgre4.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
               
  tmpl.tst2iut.qinq_nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_nvgre4.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq_nvgre4.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
     
  tmpl.tst2iut.qinq_nvgre4.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
              
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.qinq_nvgre4.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.qinq_nvgre4.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.tst2iut.qinq_nvgre6.udp4={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
               
  tmpl.tst2iut.qinq_nvgre6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env
  tmpl.tst2iut.qinq_nvgre6.eth={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  }
                }
              }
            }
  tmpl.tst2iut.qinq_nvgre6.vlan={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged tagged:{
                    vlan-id plain:210
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
     
  tmpl.tst2iut.qinq_nvgre6.qinq={
              pdus {
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H,
                  tagged double-tagged:{
                    outer {
                      vid plain:103
                    },
                    inner {
                      vid plain:420
                    }
                  },
                  length-type plain:8111
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac" },
                  src-addr env:{ name "env.addr.tst_lladdr" },
                  tagged double-tagged:{
              
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_first={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:0
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_mac
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_middle={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:1,
                  frag-offset plain:160
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_
  tmpl.tst2iut.qinq_nvgre6.ip4_frag_last={
              pdus {
                udp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()"
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  dont-frag plain:0,
                  more-frags plain:0,
                  frag-offset plain:320
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
                  dst-addr env:{ name "param.iut_ma
  tmpl.tst2iut.qinq_nvgre6.arp4={
              pdus {
                arp:{
                  proto-type plain:2048,
                  hw-type plain:1,
                  hw-size plain:6,
                  proto-size plain:4,
                  opcode plain:1,
                  snd-hw-addr plain:'00 01 02 03 04 05'H,
                  snd-proto-addr plain:'0a 89 03 01'H,
                  tgt-hw-addr plain:'00 48 54 12 6C 92'H,
                  tgt-proto-addr plain:'0a 89 03 02'H
                },
                eth:{
                  dst-addr plain:'FF FF FF FF FF FF'H,
                  src-addr plain:'00 01 02 03 04 05'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.iut_addr" },
                  src-addr env:{ name "env.addr.tst_addr" }
                },
                eth:{
   
  tmpl.iut2tst.vxlan4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env
  tmpl.iut2tst.vxlan4.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{

  tmpl.iut2tst.vxlan6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env
  tmpl.iut2tst.vxlan6.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{

  tmpl.iut2tst.geneve4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "e
  tmpl.iut2tst.geneve4.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:
  tmpl.iut2tst.geneve6.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "e
  tmpl.iut2tst.geneve6.tcp6={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:
  tmpl.iut2tst.vxlan4.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr
  tmpl.iut2tst.vxlan4.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_ad
  tmpl.iut2tst.vxlan6.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr
  tmpl.iut2tst.vxlan6.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                vxlan:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:4789,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_ad
  tmpl.iut2tst.geneve4.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.ad
  tmpl.iut2tst.geneve4.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_
  tmpl.iut2tst.geneve6.tcp4_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()",
                  ip-ident script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.ad
  tmpl.iut2tst.geneve6.tcp6_fin_psh_cwr={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0,
                  flags plain:137
                },
                ip6:{
                  src-addr plain:'20010DB885A3000000008A2E03707334'H,
                  dst-addr plain:'20010DB885A3000000008A2E03707335'H
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                geneve:{
                  vni script:"expr:rand()"
                },
                udp:{
                  dst-port plain:6081,
                  src-port env:{ name "env.addr.iut_addr.port" }
                },
                ip6:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_
  tmpl.iut2tst.nvgre4.tcp4={
              pdus {
                tcp:{
                  dst-port script:"expr:rand()",
                  src-port script:"expr:rand()",
                  seqn script:"expr:rand()",
                  ackn plain:0
                },
                ip4:{
                  dst-addr script:"expr:rand()",
                  src-addr script:"expr:rand()"
                },
                eth:{
                  dst-addr plain:'00 01 02 03 04 05'H,
                  src-addr plain:'00 48 54 12 6C 92'H
                },
                gre:{
                  key-present plain:1,
                  opt-key nvgre:{
                    vsid plain:404
                  }
                },
                ip4:{
                  dst-addr env:{ name "env.addr.tst_addr" },
                  src-addr env:{ name "env.addr.iut_addr" }
                },
                eth:{
                  dst-addr env:{ name "env.addr.tst_lladdr" },
                  src-addr env:{ name "pa
  flow_rule_pattern.ethertype={
              eth:{
                length-type plain:8111,
              }
            }
  flow_rule_pattern.ethertype.arp={
              eth:{
              },
              arp:{
              }
            }
  flow_rule_pattern.ethertype.pppoed={
              eth:{
                length-type plain:34915
              },
              pppoe:{
              }
            }
  flow_rule_pattern.ethertype.pppoes={
              eth:{
                length-type plain:34916
              },
              pppoe:{
              }
            }
  flow_rule_pattern.ethertype.ip4={
              eth:{
              },
              ip4:{
              }
            }
  flow_rule_pattern.ethertype.ip6={
              eth:{
              },
              ip6:{
              }
            }
  flow_rule_pattern.ethertype.outer_vid={
              eth:{
                length-type plain:8111,
                tagged tagged:{
                  vlan-id plain:13,
                }
              }
            }
  flow_rule_pattern.dst_mac.outer_vid={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H,
                tagged tagged:{
                  vlan-id plain:13,
                }
              }
            }
  flow_rule_pattern.ethertype.outer_vid.inner_vid={
              eth:{
                length-type plain:8111,
                tagged double-tagged:{
                  outer {
                    vid plain:13
                  },
                  inner {
                    vid plain:15
                  }
                }
              }
            }
  flow_rule_pattern.dst_mac={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              }
            }
  flow_rule_pattern.unknown_ucast_dst_mac={
              eth:{
                dst-addr range:{
                  first '00 01 02 03 04 05'H,
                  mask '01 00 00 00 00 00'H
                }
              }
            }
  flow_rule_pattern.unknown_mcast_dst_mac={
              eth:{
                dst-addr range:{
                  first '01 02 03 04 05 06'H,
                  mask '01 00 00 00 00 00'H
                }
              }
            }
  flow_rule_pattern.dst_mac.ethertype={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H,
                length-type plain:8111
              }
            }
  flow_rule_pattern.dst_mac.ethertype.ip4={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              }
            }
  flow_rule_pattern.outer_vid.ip_proto={
              eth:{
                tagged tagged:{
                  vlan-id plain:13,
                }
              },
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.ip_proto={
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.ip_proto.icmp4={
              eth:{
              },
              ip4:{
              },
              icmp4:{
              }
            }
  flow_rule_pattern.ip_proto.udp={
              eth:{
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.ip_proto.tcp={
              eth:{
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.dst_mac.ip_proto={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                protocol plain:143
              }
            }
  flow_rule_pattern.dst_mac.ip_proto.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              }
            }
  flow_rule_pattern.3tuple.udp={
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.3tuple.udp6={
              ip6:{
                dst-addr plain:'20010DB885A3000000008A2E03707334'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.5tuple.udp={
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.5tuple.udp6={
              ip6:{
                src-addr plain:'20010DB885A3000000008A2E03707334'H,
                dst-addr plain:'20010DB885A3000000008A2E03707335'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.3tuple.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              udp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.dst_mac.5tuple.udp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.3tuple.tcp={
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              tcp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.5tuple.tcp={
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              tcp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.3tuple.tcp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                dst-addr plain:'c0 a8 01 01'H
              },
              tcp:{
                dst-port plain:10
              }
            }
  flow_rule_pattern.dst_mac.5tuple.tcp={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              tcp:{
                src-port plain:10,
                dst-port plain:11
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.vxlan={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.vxlan6={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip6:{
              },
              udp:{
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.vsid.ifrm_dst_mac.vxlan={
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ip4.udp_dst.vsid.ifrm_dst_mac.vxlan={
              eth:{
              },
              ip4:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ip6.udp_dst.vsid.ifrm_dst_mac.vxlan={
              eth:{
              },
              ip6:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.vsid.vxlan={
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.ip4.udp_dst.vsid.vxlan={
              eth:{
              },
              ip4:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.ip6.udp_dst.vsid.vxlan={
              eth:{
              },
              ip6:{
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
                vni plain:13
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.geneve={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              udp:{
              },
              geneve:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.geneve6={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip6:{
              },
              udp:{
              },
              geneve:{
                vni plain:13
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.dst_mac.vsid.ifrm_dst_mac.nvgre={
              eth:{
                dst-addr plain:'00 01 02 03 04 05'H
              },
              ip4:{
              },
              gre:{
                opt-key nvgre:{
                  vsid plain:13
                }
              },
              eth:{
                dst-addr plain:'00 01 02 03 04 06'H
              }
            }
  flow_rule_pattern.ethertype.4tuple.vxlan={
              eth:{
              },
              ip4:{
                src-addr plain:'c0 a8 01 01'H,
                dst-addr plain:'c0 a8 01 02'H
              },
              udp:{
                dst-port plain:4789
              },
              vxlan:{
              }
            }
  flow_rule_actions.flag={
              {
                type flag
              }
            }
  flow_rule_actions.mark={
              {
                type mark
              }
            }
  flow_rule.rss.3tuple.tcp.default={
              attr {
                ingress 1
              },
              pattern {
                ip4:{
                  dst-addr plain:'0a 89 03 01'H,
                  src-addr script:"expr:rand()"
                },
                tcp:{
                  dst-port plain:9201,
                  src-port script:"expr:rand()"
                }
              },
              actions {
                {
                  type rss,
                  conf rss:{
                    queue {
                      1, 5, 2, 4
                    }
                  }
                }
              }
            }
  flow_rule.rss.3tuple.tcp.custom={
              attr {
                ingress 1
              },
              pattern {
                ip4:{
                  dst-addr plain:'0a 89 03 01'H,
                  src-addr script:"expr:rand()"
                },
                tcp:{
                  dst-port plain:9201,
                  src-port script:"expr:rand()"
                }
              },
              actions {
                {
                  type rss,
                  conf rss:{
                    rss-conf {
                      rss-key '6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a
                               6d 5a 6d 5a 6d 5a 6d 5a'H,
                      rss-hf {
                        ip 1,
                        tcp 1,
                      }
                    },
                    queue {
                      1, 5, 
  env.perf.iut_only='net':IUT{
                'iut_host'{
                    {'iut_jobs_ctrl':tester, if:'iut_port'}
                }
            }
  env.perf.peer2peer='net':IUT{
                'iut_host'{
                    {'iut_jobs_ctrl':tester, if:'iut_port'}
                },
                'tst_host'{
                    {'tst_jobs_ctrl':tester, if:'tst_port'}
                }
            }

MI  Tester  Execution Plan  13:05:41.776
{
  "type": "test_plan",
  "version": 1,
  "plan": {
    "type": "pkg",
    "name": "dpdk-ethdev-ts",
    "objective": "DPDM EthDev Test Suite",
    "authors": [
      {
        "name": null,
        "email": "Andrew.Rybchenko@oktetlabs.ru"
      }
    ],
    "prologue": {
      "type": "test",
      "name": "prologue"
    },
    "children": [
      {
        "type": "pkg",
        "name": "usecases",
        "objective": "Main use cases of the PMD",
        "authors": [
          {
            "name": null,
            "email": "Andrew.Rybchenko@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "test",
            "name": "tx_burst_simple",
            "iterations": 6
          },
          {
            "type": "test",
            "name": "rx_burst_simple",
            "iterations": 6
          },
          {
            "type": "test",
            "name": "rss",
            "iterations": 5
          },
          {
            "type": "test",
            "name": "rx_scatter",
            "iterations": 2
          },
          {
            "type": "test",
            "name": "tx_stats",
            "iterations": 3
          },
          {
            "type": "test",
            "name": "rx_offload_checksum",
            "iterations": 20
          }
        ]
      },
      {
        "type": "pkg",
        "name": "xmit",
        "objective": "Transmit Functionality",
        "authors": [
          {
            "name": null,
            "email": "Ivan.Malov@oktetlabs.ru"
          }
        ],
        "children": [
          {
            "type": "session",
            "name": "one_packet_all",
            "children": [
              {
                "type": "test",
                "name": "one_packet_tunnel",
                "iterations": 12
              },
              {
                "type": "session",
                "name": "one_packet_ip4",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4",
                    "children": [
                      {
                        "type": "test",
                        "name": "tso_packet_ip4_seg"
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip4"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip4",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4",
                    "children": [
                      {
                        "type": "test",
                        "name": "tso_packet_ip4_seg"
                      }
                    ]
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip4"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip4"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip6",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip6"
                  }
                ]
              },
              {
                "type": "session",
                "name": "one_packet_ip6",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip4_encap_ip6"
                  },
                  {
                    "type": "session",
                    "name": "tso_packet_ip6_encap_ip6"
                  }
                ]
              }
            ]
          },
          {
            "type": "session",
            "name": "one_packet_with_dpdk_rx_all",
            "prologue": {
              "type": "test",
              "name": "one_packet_with_dpdk_rx_prologue"
            },
            "children": [
              {
                "type": "session",
                "name": "session",
                "children": [
                  {
                    "type": "session",
                    "name": "one_packet_with_dpdk_rx_cksum_offloads",
                    "children": [
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_plain",
                        "children": [
                          {
                            "type": "test",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_plain_ip4",
                            "iterations": 8
                          }
                        ]
                      },
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_encap",
                        "children": [
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip6"
                          }
                        ]
                      },
                      {
                        "type": "session",
                        "name": "one_packet_with_dpdk_rx_cksum_offloads_encap",
                        "children": [
                          {
                            "type": "session",
                            "name": "one_packet_with_dpdk_rx_cksum_offloads_encap_ip4"
                          },
                          {
                            "type": "session",