DTS testpmd and SCAPY integration

DTS testpmd and SCAPY integration

[Etelson, Gregory]

Hello,

Consider an option to describe DTS test with testpmd and SCAPY plain text 
commands.

For example:

Scenario:
  - Configure UDP packet in SCAPY and a flow in testpmd.
  - Send UDP packet and validate testpmd output triggered by that packet.

```yaml
phase_0:
     name: CONFIGURATION
     tg: |
         udp_pkt = Ether()/IP()/UDP(sport=31)/Raw('== TEST ==')
         print('packet is ready')
     dut: |
         start
         set verbose 1
         flow create 0 ingress pattern eth / ipv4 / udp src is 31 / end actions queue index 1 / end
     result:
         dut: 'Flow rule #0 created'
         tg: 'packet is ready'

phase_1:
     name: SEND and VALIDATE
     tg: sendp(udp_pkt, iface=pf0)
     result:
         dut: '- RSS queue=0x1 -'
```

Test described as a sequence of phases.

Phase definition:
~~~~~~~~~~~~~~~~

```
<phase id>: # unique phase ID
     name: <phase name>                       # phase name
     <application ID (APP1)>: <APP1 commands> # application APP1 commands
     ...
     <application ID (APPy)>: <APPy commands> # application APPy commands
     ...
     <application ID (APPx)>: <APPx commands> # application APPx commands

     result:                # optional phase results verification section
         <application ID APPx>: <expected output>  # APPx expected output
         ...
         <application ID APPy>: <expected output>  # APPy expected output
```

- Application commands in a phase executed sequentially,
   in order of application IDs: <APPy> commands executed
   before <APPx> commands.

- Application results in a phase validated sequentially,
   in order of application IDs: <APPx> result validated
   before APPy result.

- Application result is a regular expression.


Test application definition:
~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
<application ID>: # unique application ID
     agent:        # mandatory application type identifier: {testpmd|scapy}
     cmd:          # optional application command template
```

Example:

```yaml

dut:
     agent: testpmd
     cmd: 'dpdk-testpmd -a pci0 -- -i --rxq=4 --txq=4'

tg:
     agent: scapy
```

Test commands do not bound to a specific setup.
Therefore, testpmd commad line and SCAPY sendp() function use encoding
to describe relative interface position in a tested HBA.

PCI encoding scheme for testpmd:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- PF PCI: `pciX`
   Example: `pci0: '0000:08:00.0'`


- PCI SR-IOV: `pciXvfY`
   Example: `pci0vf0: '0000:08:00.2'`

Network devices encoding scheme for SCAPY:
- PF: `pfX`
   Example: `pf0: enp8s0f0np0`


- PCI SR-IOV: `pfXvfY`
   Example: `pf0vf0: enp5s0f0v0`


- Network device representor: `pfXrfY`

      Example: `pf0rf0: enp5s0f0npf0vf0`


Test execution requires an additional file to describe tested setup.

Setup file format:
~~~~~~~~~~~~~~~~~

```
<application id>: # unique application ID
     host:         # hostname or IPvX address
     path:         # optional application path
     hba:          # optional HBA description
         pmd:      # PMD
         hw:       # HW type
```

Example:

```yaml
dut:
     host: 1.2.3.4
     path: /opt/dpdk.org/build/app
     hba:
         pmd: mlx5
         hw: mt4125
tg:
     host: ::1234
```

```yaml
dut:
     agent: testpmd
     cmd: 'dpdk-testpmd -a pci0 -- -i --rxq=4 --txq=4'
tg:
     agent: scapy

test:
     -
         phases: [ *ref_phase0 ]
         repeat: 1
     -
         phases: [ *ref_phase1 ]
         repeat: 3

phase_0: &ref_phase0
     name: CONFIGURATION
     tg: |
         udp_pkt = Ether()/IP()/UDP(sport=31)/Raw('== TEST ==')
         print('packet is ready')
     dut: |
         start
         set verbose 1
         flow create 0 ingress pattern eth / ipv4 / udp src is 31 / end actions queue index 1 / end
     result:
         dut: 'Flow rule #0 created'
         tg: 'packet is ready'

phase_1: &ref_phase1
     name: SEND and VALIDATE
     tg: sendp(udp_pkt, iface=pf0)
     result:
         dut: '- RSS queue=0x1 -'
```

The plain text format provides minimalistic and intuitive framework for DTS 
tests.
DTS can use plan text testpmd/scapy command format in addition to Python 
framework.

Regards,
Gregory

RE: DTS testpmd and SCAPY integration

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Etelson, Gregory <getelson@nvidia.com>
> Sent: Tuesday, December 26, 2023 1:32 AM
> To: thomas@monjalon.net; Juraj Linkeš <juraj.linkes@pantheon.tech>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Paul Szczepanek
> <Paul.Szczepanek@arm.com>; Luca Vizzarro <Luca.Vizzarro@arm.com>; Yoan
> Picchi <yoan.picchi@foss.arm.com>; Jeremy Spewock
> <jspewock@iol.unh.edu>; Gregory Etelson <getelson@nvidia.com>; Patrick
> Robb <probb@iol.unh.edu>; ci@dpdk.org; dev@dpdk.org
> Subject: DTS testpmd and SCAPY integration
> 
> Hello,
> 
> Consider an option to describe DTS test with testpmd and SCAPY plain text
> commands.
> 
> For example:
> 
> Scenario:
>   - Configure UDP packet in SCAPY and a flow in testpmd.
>   - Send UDP packet and validate testpmd output triggered by that packet.
> 
> ```yaml
> phase_0:
>      name: CONFIGURATION
>      tg: |
>          udp_pkt = Ether()/IP()/UDP(sport=31)/Raw('== TEST ==')
>          print('packet is ready')
>      dut: |
>          start
>          set verbose 1
>          flow create 0 ingress pattern eth / ipv4 / udp src is 31 / end actions queue
> index 1 / end
>      result:
>          dut: 'Flow rule #0 created'
>          tg: 'packet is ready'
> 
> phase_1:
>      name: SEND and VALIDATE
>      tg: sendp(udp_pkt, iface=pf0)
>      result:
>          dut: '- RSS queue=0x1 -'
> ```
> 
> Test described as a sequence of phases.
> 
> Phase definition:
> ~~~~~~~~~~~~~~~~
> 
> ```
> <phase id>: # unique phase ID
>      name: <phase name>                       # phase name
>      <application ID (APP1)>: <APP1 commands> # application APP1 commands
>      ...
>      <application ID (APPy)>: <APPy commands> # application APPy commands
>      ...
>      <application ID (APPx)>: <APPx commands> # application APPx commands
> 
>      result:                # optional phase results verification section
>          <application ID APPx>: <expected output>  # APPx expected output
>          ...
>          <application ID APPy>: <expected output>  # APPy expected output ```
> 
> - Application commands in a phase executed sequentially,
>    in order of application IDs: <APPy> commands executed
>    before <APPx> commands.
> 
> - Application results in a phase validated sequentially,
>    in order of application IDs: <APPx> result validated
>    before APPy result.
> 
> - Application result is a regular expression.
> 
> 
> Test application definition:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> ```
> <application ID>: # unique application ID
>      agent:        # mandatory application type identifier: {testpmd|scapy}
>      cmd:          # optional application command template
> ```
> 
> Example:
> 
> ```yaml
> 
> dut:
>      agent: testpmd
>      cmd: 'dpdk-testpmd -a pci0 -- -i --rxq=4 --txq=4'
> 
> tg:
>      agent: scapy
> ```
> 
> Test commands do not bound to a specific setup.
> Therefore, testpmd commad line and SCAPY sendp() function use encoding to
> describe relative interface position in a tested HBA.
> 
> PCI encoding scheme for testpmd:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> - PF PCI: `pciX`
>    Example: `pci0: '0000:08:00.0'`
> 
> 
> - PCI SR-IOV: `pciXvfY`
>    Example: `pci0vf0: '0000:08:00.2'`
> 
> Network devices encoding scheme for SCAPY:
> - PF: `pfX`
>    Example: `pf0: enp8s0f0np0`
> 
> 
> - PCI SR-IOV: `pfXvfY`
>    Example: `pf0vf0: enp5s0f0v0`
> 
> 
> - Network device representor: `pfXrfY`
> 
>       Example: `pf0rf0: enp5s0f0npf0vf0`
> 
> 
> Test execution requires an additional file to describe tested setup.
> 
> Setup file format:
> ~~~~~~~~~~~~~~~~~
> 
> ```
> <application id>: # unique application ID
>      host:         # hostname or IPvX address
>      path:         # optional application path
>      hba:          # optional HBA description
>          pmd:      # PMD
>          hw:       # HW type
> ```
> 
> Example:
> 
> ```yaml
> dut:
>      host: 1.2.3.4
>      path: /opt/dpdk.org/build/app
>      hba:
>          pmd: mlx5
>          hw: mt4125
> tg:
>      host: ::1234
> ```
> 
> ```yaml
> dut:
>      agent: testpmd
>      cmd: 'dpdk-testpmd -a pci0 -- -i --rxq=4 --txq=4'
> tg:
>      agent: scapy
> 
> test:
>      -
>          phases: [ *ref_phase0 ]
>          repeat: 1
>      -
>          phases: [ *ref_phase1 ]
>          repeat: 3
> 
> phase_0: &ref_phase0
>      name: CONFIGURATION
>      tg: |
>          udp_pkt = Ether()/IP()/UDP(sport=31)/Raw('== TEST ==')
>          print('packet is ready')
>      dut: |
>          start
>          set verbose 1
>          flow create 0 ingress pattern eth / ipv4 / udp src is 31 / end actions queue
> index 1 / end
>      result:
>          dut: 'Flow rule #0 created'
>          tg: 'packet is ready'
> 
> phase_1: &ref_phase1
>      name: SEND and VALIDATE
>      tg: sendp(udp_pkt, iface=pf0)
>      result:
>          dut: '- RSS queue=0x1 -'
> ```
> 
> The plain text format provides minimalistic and intuitive framework for DTS
> tests.
> DTS can use plan text testpmd/scapy command format in addition to Python
> framework.
Hi Gregory,
	I do not fully understand your proposal, it will be helpful to join the DTS meetings to discuss this further.

YAML has wide support built around it. By using our own text format, we will have to build the parsing support etc ourselves.

However, YAML is supposed to be easy to read and understand. Is it just a matter for getting used to it?

Thank you,
Honnappa

> 
> Regards,
> Gregory

RE: DTS testpmd and SCAPY integration

[Etelson, Gregory]

Hello Honnappa,

[snip]

> Hi Gregory,
>        I do not fully understand your proposal, it will be helpful to join the DTS meetings to discuss this further.
>

Agree, let's discuss the proposal details during the DTS meeting.

> YAML has wide support built around it. By using our own text format, we will have to build the parsing support etc ourselves.
>
> However, YAML is supposed to be easy to read and understand. Is it just a matter for getting used to it?
>

I selected YAML for 2 reasons:
   * Plain and intuitive YAML format minimized test meta data.
     By the meta data I refer to control tags and markup characters
     that are not test commands.
   * YAML has Python parser.

Regards,
Gregory

RE: DTS testpmd and SCAPY integration

[Honnappa Nagarahalli]

<snip>

> 
> Hello Honnappa,
> 
> [snip]
> 
> > Hi Gregory,
> >        I do not fully understand your proposal, it will be helpful to join the DTS
> meetings to discuss this further.
> >
> 
> Agree, let's discuss the proposal details during the DTS meeting.
> 
> > YAML has wide support built around it. By using our own text format, we will
> have to build the parsing support etc ourselves.
> >
> > However, YAML is supposed to be easy to read and understand. Is it just a
> matter for getting used to it?
> >
> 
> I selected YAML for 2 reasons:
>    * Plain and intuitive YAML format minimized test meta data.
>      By the meta data I refer to control tags and markup characters
>      that are not test commands.
>    * YAML has Python parser.
I have mis-understood your proposal. I agree with your above comments.
+1 for the proposal.

> 
> Regards,
> Gregory
> 
> 

Re: DTS testpmd and SCAPY integration

[Juraj Linkeš]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

Hi folks,

Let me summarize the yesterday's discussion in a few keys points:

   - Greg's proposal aims at simplicity and is useful mainly for test cases
   which can be written in a few minutes. More complex test cases are not
   suitable for the YAML approach.
   - The above implies that the YAML based test cases would be supported
   alongside the existing approach. This fast way to implement simple test
   cases would likely be a valuable addition.
   - The big picture idea behind the YAML test cases is to take an
   application with interactive input, send commands, collect output and
   compare the output with expected string(s).
   - Greg may be able to make the code available and may assess how to
   integrate it with DTS.

Regards,
Juraj

On Mon, Jan 8, 2024 at 6:36 PM Honnappa Nagarahalli <
Honnappa.Nagarahalli@arm.com> wrote:

> <snip>
>
> >
> > Hello Honnappa,
> >
> > [snip]
> >
> > > Hi Gregory,
> > >        I do not fully understand your proposal, it will be helpful to
> join the DTS
> > meetings to discuss this further.
> > >
> >
> > Agree, let's discuss the proposal details during the DTS meeting.
> >
> > > YAML has wide support built around it. By using our own text format,
> we will
> > have to build the parsing support etc ourselves.
> > >
> > > However, YAML is supposed to be easy to read and understand. Is it
> just a
> > matter for getting used to it?
> > >
> >
> > I selected YAML for 2 reasons:
> >    * Plain and intuitive YAML format minimized test meta data.
> >      By the meta data I refer to control tags and markup characters
> >      that are not test commands.
> >    * YAML has Python parser.
> I have mis-understood your proposal. I agree with your above comments.
> +1 for the proposal.
>
> >
> > Regards,
> > Gregory
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 2385 bytes --]

Re: DTS testpmd and SCAPY integration

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 2645 bytes --]

Thanks for summarizing Juraj.

As we discussed at the CI meeting yesterday, we should get together next
week to workshop some rough testsuites based on the ideas which have been
proposed above, and discussed in the DTS meeting and CI testing meeting.

Just to get something on the calendar I scheduled a zoom for next wednesday
at 2pm UTC, but yall let me know if you prefer something different. All are
welcome of course.

https://unh.zoom.us/j/94747183133



On Thu, Jan 18, 2024 at 7:32 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:

> Hi folks,
>
> Let me summarize the yesterday's discussion in a few keys points:
>
>    - Greg's proposal aims at simplicity and is useful mainly for test
>    cases which can be written in a few minutes. More complex test cases are
>    not suitable for the YAML approach.
>    - The above implies that the YAML based test cases would be supported
>    alongside the existing approach. This fast way to implement simple test
>    cases would likely be a valuable addition.
>    - The big picture idea behind the YAML test cases is to take an
>    application with interactive input, send commands, collect output and
>    compare the output with expected string(s).
>    - Greg may be able to make the code available and may assess how to
>    integrate it with DTS.
>
> Regards,
> Juraj
>
> On Mon, Jan 8, 2024 at 6:36 PM Honnappa Nagarahalli <
> Honnappa.Nagarahalli@arm.com> wrote:
>
>> <snip>
>>
>> >
>> > Hello Honnappa,
>> >
>> > [snip]
>> >
>> > > Hi Gregory,
>> > >        I do not fully understand your proposal, it will be helpful to
>> join the DTS
>> > meetings to discuss this further.
>> > >
>> >
>> > Agree, let's discuss the proposal details during the DTS meeting.
>> >
>> > > YAML has wide support built around it. By using our own text format,
>> we will
>> > have to build the parsing support etc ourselves.
>> > >
>> > > However, YAML is supposed to be easy to read and understand. Is it
>> just a
>> > matter for getting used to it?
>> > >
>> >
>> > I selected YAML for 2 reasons:
>> >    * Plain and intuitive YAML format minimized test meta data.
>> >      By the meta data I refer to control tags and markup characters
>> >      that are not test commands.
>> >    * YAML has Python parser.
>> I have mis-understood your proposal. I agree with your above comments.
>> +1 for the proposal.
>>
>> >
>> > Regards,
>> > Gregory
>> >
>> >
>>
>>

-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 5515 bytes --]

Re: DTS testpmd and SCAPY integration

[Luca Vizzarro]

Hi Honnappa,

> YAML has wide support built around it. By using our own text format, we will have to build the parsing support etc ourselves.
> 
> However, YAML is supposed to be easy to read and understand. Is it just a matter for getting used to it?

I may be wrong, please feel free to correct me, but I believe what 
Gregory meant is to write raw Scapy and testpmd commands directly in 
YAML-written tests. As in, we wouldn't be using our own text format, 
we'd just need to feed the raw commands from YAML straight to the Scapy 
and testpmd shells. Finally, validate their responses to what was set in 
the YAML test. In Gregory's example this is just comparing stdout.

Best,
Luca

RE: DTS testpmd and SCAPY integration

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Luca Vizzarro <Luca.Vizzarro@arm.com>
> Sent: Monday, January 8, 2024 6:18 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Etelson,
> Gregory <getelson@nvidia.com>; thomas@monjalon.net; Juraj Linkeš
> <juraj.linkes@pantheon.tech>; Paul Szczepanek <Paul.Szczepanek@arm.com>;
> Yoan Picchi <yoan.picchi@foss.arm.com>; Jeremy Spewock
> <jspewock@iol.unh.edu>; Patrick Robb <probb@iol.unh.edu>; ci@dpdk.org;
> dev@dpdk.org
> Subject: Re: DTS testpmd and SCAPY integration
> 
> Hi Honnappa,
> 
> > YAML has wide support built around it. By using our own text format, we will
> have to build the parsing support etc ourselves.
> >
> > However, YAML is supposed to be easy to read and understand. Is it just a
> matter for getting used to it?
> 
> I may be wrong, please feel free to correct me, but I believe what Gregory
> meant is to write raw Scapy and testpmd commands directly in YAML-written
> tests. As in, we wouldn't be using our own text format, we'd just need to feed
> the raw commands from YAML straight to the Scapy and testpmd shells.
> Finally, validate their responses to what was set in the YAML test. In Gregory's
> example this is just comparing stdout.
Yes, I have misunderstood Gregory's proposal. I am all for using YAML. Let us discuss more in the DTS meetings and hear what others have to say.

> 
> Best,
> Luca

Re: DTS testpmd and SCAPY integration

[Luca Vizzarro]

Hi Gregory,

Your proposal sounds rather interesting. Certainly enabling DTS to 
accept YAML-written tests sounds more developer-friendly and should 
enable quicker test-writing. As this is an extra feature though – and a 
nice-to-have, it should definitely be discussed in the DTS meetings as 
Honnappa suggested already.

Another thing, I am not sure that the intention is that Scapy will be 
the only traffic generator supported. Could be very wrong here. In the 
case we'd support others too, how would you tackle this problem? We can 
discuss this in the meeting as well if needed.

Best,
Luca

Re: DTS testpmd and SCAPY integration

[Etelson, Gregory]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

Hello Luca,

> Your proposal sounds rather interesting. Certainly enabling DTS to
> accept YAML-written tests sounds more developer-friendly and should
> enable quicker test-writing. As this is an extra feature though – and a
> nice-to-have, it should definitely be discussed in the DTS meetings as
> Honnappa suggested already.
>
> Another thing, I am not sure that the intention is that Scapy will be
> the only traffic generator supported. Could be very wrong here. In the
> case we'd support others too, how would you tackle this problem? We can
> discuss this in the meeting as well if needed.
>

The proposed design works with any DPDK, traffic generator, sniffer or debugger 
application that ether accepts input from STDIN or dumps output to SDTOUT.
I used testpmd / scapy in the example because that was the most obvious combo.

Regards,
Gregory

Re: DTS testpmd and SCAPY integration

[Thomas Monjalon]

08/01/2024 13:10, Luca Vizzarro:
> Your proposal sounds rather interesting. Certainly enabling DTS to 
> accept YAML-written tests sounds more developer-friendly and should 
> enable quicker test-writing. As this is an extra feature though – and a 
> nice-to-have, it should definitely be discussed in the DTS meetings as 
> Honnappa suggested already.

I would not classify this idea as "nice-to-have".
I would split this proposal in 2 parts:
	1/ YAML is an implementation alternative.
	2/ Being able to write a test with a small number of lines,
	reusing some commands from existing tools,
	should be our "must-have" common goal.

Others have mentioned that YAML may not be suitable in complex cases,
and that it would be an additional language for test writing.
I personnaly think we should focus on a single path which is easy to read and maintain.
For the configuration side, YAML is already used in DTS.
For the test suite logic, do you think we can achieve the same simplicity
with some Python code?

We discussed how to progress with this proposal during the CI meeting last week.
We need to check how it could look and what we can improve to reach this goal.
Patrick proposes a meeting this Wednesday at 2pm UTC.

RE: DTS testpmd and SCAPY integration

[Honnappa Nagarahalli]

<snip>

> 
> 08/01/2024 13:10, Luca Vizzarro:
> > Your proposal sounds rather interesting. Certainly enabling DTS to
> > accept YAML-written tests sounds more developer-friendly and should
> > enable quicker test-writing. As this is an extra feature though – and
> > a nice-to-have, it should definitely be discussed in the DTS meetings
> > as Honnappa suggested already.
> 
> I would not classify this idea as "nice-to-have".
> I would split this proposal in 2 parts:
> 	1/ YAML is an implementation alternative.
> 	2/ Being able to write a test with a small number of lines,
> 	reusing some commands from existing tools,
> 	should be our "must-have" common goal.
> 
> Others have mentioned that YAML may not be suitable in complex cases, and
> that it would be an additional language for test writing.
> I personnaly think we should focus on a single path which is easy to read and
> maintain.
I think we are digressing from the plan we had put forward if we have to go down this path. We should understand what it means by going down the YAML format. Also, what would happen if there is another innovation in 3 months?

We already have scatter-gather test suite ported to DPDK repo and has undergone review in the community.

In the last meeting we went through a simple test case. Is it possible to write the scatter-gather test case in YAML and see how they compare?

> For the configuration side, YAML is already used in DTS.
> For the test suite logic, do you think we can achieve the same simplicity with
> some Python code?
> 
> We discussed how to progress with this proposal during the CI meeting last
> week.
> We need to check how it could look and what we can improve to reach this
> goal.
> Patrick proposes a meeting this Wednesday at 2pm UTC.
> 

Re: DTS testpmd and SCAPY integration

[Thomas Monjalon]

23/01/2024 04:42, Honnappa Nagarahalli:
> > 08/01/2024 13:10, Luca Vizzarro:
> > > Your proposal sounds rather interesting. Certainly enabling DTS to
> > > accept YAML-written tests sounds more developer-friendly and should
> > > enable quicker test-writing. As this is an extra feature though – and
> > > a nice-to-have, it should definitely be discussed in the DTS meetings
> > > as Honnappa suggested already.
> > 
> > I would not classify this idea as "nice-to-have".
> > I would split this proposal in 2 parts:
> > 	1/ YAML is an implementation alternative.
> > 	2/ Being able to write a test with a small number of lines,
> > 	reusing some commands from existing tools,
> > 	should be our "must-have" common goal.
> > 
> > Others have mentioned that YAML may not be suitable in complex cases, and
> > that it would be an additional language for test writing.
> > I personnaly think we should focus on a single path which is easy to read and
> > maintain.
> 
> I think we are digressing from the plan we had put forward if we have to go down this path.
> We should understand what it means by going down the YAML format.
> Also, what would happen if there is another innovation in 3 months?

There is a misunderstanding here.
I suggest to take this proposal as an example of the simplicity to reach.
But I agree with you it is more reasonnable to continue with the Python path.

> We already have scatter-gather test suite ported to DPDK repo and has undergone review in the community.
> 
> In the last meeting we went through a simple test case. Is it possible to write the scatter-gather test case in YAML and see how they compare?

After the latest CI meeting we thought about writing a simple text case
in Python with some virtual functions which would abstract all the boilerplate code,
so it would have the same level of simplicity as this YAML proposal.

> > For the configuration side, YAML is already used in DTS.
> > For the test suite logic, do you think we can achieve the same simplicity with
> > some Python code?
> > 
> > We discussed how to progress with this proposal during the CI meeting last
> > week.
> > We need to check how it could look and what we can improve to reach this
> > goal.
> > Patrick proposes a meeting this Wednesday at 2pm UTC.

Re: DTS testpmd and SCAPY integration

[Jeremy Spewock]

[-- Attachment #1: Type: text/plain, Size: 4953 bytes --]

On Tue, Jan 23, 2024 at 3:50 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> 23/01/2024 04:42, Honnappa Nagarahalli:
> > > 08/01/2024 13:10, Luca Vizzarro:
> > > > Your proposal sounds rather interesting. Certainly enabling DTS to
> > > > accept YAML-written tests sounds more developer-friendly and should
> > > > enable quicker test-writing. As this is an extra feature though – and
> > > > a nice-to-have, it should definitely be discussed in the DTS meetings
> > > > as Honnappa suggested already.
> > >
> > > I would not classify this idea as "nice-to-have".
> > > I would split this proposal in 2 parts:
> > >     1/ YAML is an implementation alternative.
> > >     2/ Being able to write a test with a small number of lines,
> > >     reusing some commands from existing tools,
> > >     should be our "must-have" common goal.
> > >
> > > Others have mentioned that YAML may not be suitable in complex cases,
> and
> > > that it would be an additional language for test writing.
> > > I personnaly think we should focus on a single path which is easy to
> read and
> > > maintain.
> >
> > I think we are digressing from the plan we had put forward if we have to
> go down this path.
> > We should understand what it means by going down the YAML format.
> > Also, what would happen if there is another innovation in 3 months?
>
> There is a misunderstanding here.
> I suggest to take this proposal as an example of the simplicity to reach.
> But I agree with you it is more reasonnable to continue with the Python
> path.
>

I agree that it would be easier to develop both the framework and some of
the more complex test cases in python and think sticking with Python is a
good route.

I think taking this proposal and using it as an example of something we
could work towards is also a great outlook because it gives a more
structured goal instead of the idea being "let's just make it as simple as
we can."


>
> > We already have scatter-gather test suite ported to DPDK repo and has
> undergone review in the community.
> >
> > In the last meeting we went through a simple test case. Is it possible
> to write the scatter-gather test case in YAML and see how they compare?
>
> After the latest CI meeting we thought about writing a simple text case
> in Python with some virtual functions which would abstract all the
> boilerplate code,
> so it would have the same level of simplicity as this YAML proposal.
>

Looking at what we have for DTS currently, we have put thought into
achieving this and, in some categories, have made things similar to the
YAML approach. Like, for example, when it comes to things like sending,
receiving, and verifying a packet, all you have to do is create a packet
and call a method. All of the boilerplate regarding how to send and receive
and how it can be different between types of traffic generators, as well as
setting up the ability to forward the packet is completely abstracted away
from you. It's as simple as you send a packet and you get some packets back
from the test suite perspective. The os_udp testing suite that Juraj wrote
is a great example of this and how short the simple case can be.
Additionally we have created centralized APIs between the SUT and TG nodes
so that all of the methods that the developer would have to use can be
accessed either through these nodes, or through the method inherited from
the base class for test suites.

I think that when you try to make things extremely simple there is a common
tradeoff; the more simplistic it is, the more restrictive it can become. We
would have to make more assumptions about use cases and how things would
look in these common cases. This generally isn't a super big problem in our
case because if support for something that is needed is missing, we can add
it to the API and with these test cases there are things that will
generally be similar every time. This tradeoff can also be offset by
creating the functions to make these things simple while still exposing
less restrictive underlying methods so that more niche use cases can still
be fulfilled. Just something to keep in mind.


>
> > > For the configuration side, YAML is already used in DTS.
> > > For the test suite logic, do you think we can achieve the same
> simplicity with
> > > some Python code?
> > >
> > > We discussed how to progress with this proposal during the CI meeting
> last
> > > week.
> > > We need to check how it could look and what we can improve to reach
> this
> > > goal.
> > > Patrick proposes a meeting this Wednesday at 2pm UTC.
>
>
>
> Generally I think this discussion is very useful and having some kind of
ideal mock suite would also be useful to reference. It also can help shine
a light on the importance of thinking about what things could be the common
case and abstracting them away from the developer in the framework.

[-- Attachment #2: Type: text/html, Size: 6765 bytes --]

Re: DTS testpmd and SCAPY integration

[Gregory Etelson]

[-- Attachment #1: Type: text/plain, Size: 5671 bytes --]

Hello,

I've uploaded the code snapshot and functional description here:
https://drive.google.com/drive/folders/1cHrPwx4fUJ6ibUCtHd4kNKsrmmvQvvOj?usp=drive_link

Regards,
Gregory
________________________________
From: Jeremy Spewock <jspewock@iol.unh.edu>
Sent: Tuesday, January 23, 2024 20:26
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gregory Etelson <getelson@nvidia.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>; Paul Szczepanek <Paul.Szczepanek@arm.com>; Yoan Picchi <yoan.picchi@foss.arm.com>; Patrick Robb <probb@iol.unh.edu>; Luca Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; dev@dpdk.org <dev@dpdk.org>; nd <nd@arm.com>
Subject: Re: DTS testpmd and SCAPY integration

External email: Use caution opening links or attachments



On Tue, Jan 23, 2024 at 3:50 AM Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>> wrote:
23/01/2024 04:42, Honnappa Nagarahalli:
> > 08/01/2024 13:10, Luca Vizzarro:
> > > Your proposal sounds rather interesting. Certainly enabling DTS to
> > > accept YAML-written tests sounds more developer-friendly and should
> > > enable quicker test-writing. As this is an extra feature though – and
> > > a nice-to-have, it should definitely be discussed in the DTS meetings
> > > as Honnappa suggested already.
> >
> > I would not classify this idea as "nice-to-have".
> > I would split this proposal in 2 parts:
> >     1/ YAML is an implementation alternative.
> >     2/ Being able to write a test with a small number of lines,
> >     reusing some commands from existing tools,
> >     should be our "must-have" common goal.
> >
> > Others have mentioned that YAML may not be suitable in complex cases, and
> > that it would be an additional language for test writing.
> > I personnaly think we should focus on a single path which is easy to read and
> > maintain.
>
> I think we are digressing from the plan we had put forward if we have to go down this path.
> We should understand what it means by going down the YAML format.
> Also, what would happen if there is another innovation in 3 months?

There is a misunderstanding here.
I suggest to take this proposal as an example of the simplicity to reach.
But I agree with you it is more reasonnable to continue with the Python path.

I agree that it would be easier to develop both the framework and some of the more complex test cases in python and think sticking with Python is a good route.

I think taking this proposal and using it as an example of something we could work towards is also a great outlook because it gives a more structured goal instead of the idea being "let's just make it as simple as we can."


> We already have scatter-gather test suite ported to DPDK repo and has undergone review in the community.
>
> In the last meeting we went through a simple test case. Is it possible to write the scatter-gather test case in YAML and see how they compare?

After the latest CI meeting we thought about writing a simple text case
in Python with some virtual functions which would abstract all the boilerplate code,
so it would have the same level of simplicity as this YAML proposal.

Looking at what we have for DTS currently, we have put thought into achieving this and, in some categories, have made things similar to the YAML approach. Like, for example, when it comes to things like sending, receiving, and verifying a packet, all you have to do is create a packet and call a method. All of the boilerplate regarding how to send and receive and how it can be different between types of traffic generators, as well as setting up the ability to forward the packet is completely abstracted away from you. It's as simple as you send a packet and you get some packets back from the test suite perspective. The os_udp testing suite that Juraj wrote is a great example of this and how short the simple case can be. Additionally we have created centralized APIs between the SUT and TG nodes so that all of the methods that the developer would have to use can be accessed either through these nodes, or through the method inherited from the base class for test suites.

I think that when you try to make things extremely simple there is a common tradeoff; the more simplistic it is, the more restrictive it can become. We would have to make more assumptions about use cases and how things would look in these common cases. This generally isn't a super big problem in our case because if support for something that is needed is missing, we can add it to the API and with these test cases there are things that will generally be similar every time. This tradeoff can also be offset by creating the functions to make these things simple while still exposing less restrictive underlying methods so that more niche use cases can still be fulfilled. Just something to keep in mind.


> > For the configuration side, YAML is already used in DTS.
> > For the test suite logic, do you think we can achieve the same simplicity with
> > some Python code?
> >
> > We discussed how to progress with this proposal during the CI meeting last
> > week.
> > We need to check how it could look and what we can improve to reach this
> > goal.
> > Patrick proposes a meeting this Wednesday at 2pm UTC.



Generally I think this discussion is very useful and having some kind of ideal mock suite would also be useful to reference. It also can help shine a light on the importance of thinking about what things could be the common case and abstracting them away from the developer in the framework.

[-- Attachment #2: Type: text/html, Size: 9420 bytes --]

Re: DTS testpmd and SCAPY integration

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 7717 bytes --]

Thank you for sharing Gregory. I did not get an opportunity to look through
the code today, but I did run through the presentation. A few points I
noted:

1. The presentation shows an example testpmd testcase for creating a flow
rule, and then shows a validation step in which standard out is compared
against the expected string ("flow rule x created") and we can conclude
whether we are able to create flow rules. Are you also sending packets
according to the flow rules and validating that what is
sent/received corresponds to the expected behavior of the flow rules? When
I look at the old DTS framework, and an example flow rules testsuite (
https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want
feature parity with, I think that validation for this testing framework
needs to primarily rely on comparing packets sent and packets received. In
any case, there may be some testsuites which can be written which are small
enough in scope that validating by standard out in this way may be
appropriate. I'm not sure but we should keep our options open.

2. If the implementation overhead is not too significant for the
configuration step in the DTS execution a "--fast" option like you use may
be a good improvement for the framework. In your mind, is the main benefit
A. reduced execution time, B. reduced user setup time (don't have to write
full config file) or C. Something else?

Thanks for making this available to use so we can use it as a reference in
making DTS better. :)


On Sun, Jan 28, 2024 at 8:44 AM Gregory Etelson <getelson@nvidia.com> wrote:

> Hello,
>
> I've uploaded the code snapshot and functional description here:
>
> https://drive.google.com/drive/folders/1cHrPwx4fUJ6ibUCtHd4kNKsrmmvQvvOj?usp=drive_link
>
> Regards,
> Gregory
> ------------------------------
> *From:* Jeremy Spewock <jspewock@iol.unh.edu>
> *Sent:* Tuesday, January 23, 2024 20:26
> *To:* NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
> *Cc:* Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gregory
> Etelson <getelson@nvidia.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>;
> Paul Szczepanek <Paul.Szczepanek@arm.com>; Yoan Picchi <
> yoan.picchi@foss.arm.com>; Patrick Robb <probb@iol.unh.edu>; Luca
> Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; dev@dpdk.org
> <dev@dpdk.org>; nd <nd@arm.com>
> *Subject:* Re: DTS testpmd and SCAPY integration
>
> *External email: Use caution opening links or attachments*
>
>
> On Tue, Jan 23, 2024 at 3:50 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
> 23/01/2024 04:42, Honnappa Nagarahalli:
> > > 08/01/2024 13:10, Luca Vizzarro:
> > > > Your proposal sounds rather interesting. Certainly enabling DTS to
> > > > accept YAML-written tests sounds more developer-friendly and should
> > > > enable quicker test-writing. As this is an extra feature though – and
> > > > a nice-to-have, it should definitely be discussed in the DTS meetings
> > > > as Honnappa suggested already.
> > >
> > > I would not classify this idea as "nice-to-have".
> > > I would split this proposal in 2 parts:
> > >     1/ YAML is an implementation alternative.
> > >     2/ Being able to write a test with a small number of lines,
> > >     reusing some commands from existing tools,
> > >     should be our "must-have" common goal.
> > >
> > > Others have mentioned that YAML may not be suitable in complex cases,
> and
> > > that it would be an additional language for test writing.
> > > I personnaly think we should focus on a single path which is easy to
> read and
> > > maintain.
> >
> > I think we are digressing from the plan we had put forward if we have to
> go down this path.
> > We should understand what it means by going down the YAML format.
> > Also, what would happen if there is another innovation in 3 months?
>
> There is a misunderstanding here.
> I suggest to take this proposal as an example of the simplicity to reach.
> But I agree with you it is more reasonnable to continue with the Python
> path.
>
>
> I agree that it would be easier to develop both the framework and some of
> the more complex test cases in python and think sticking with Python is a
> good route.
>
> I think taking this proposal and using it as an example of something we
> could work towards is also a great outlook because it gives a more
> structured goal instead of the idea being "let's just make it as simple as
> we can."
>
>
>
> > We already have scatter-gather test suite ported to DPDK repo and has
> undergone review in the community.
> >
> > In the last meeting we went through a simple test case. Is it possible
> to write the scatter-gather test case in YAML and see how they compare?
>
> After the latest CI meeting we thought about writing a simple text case
> in Python with some virtual functions which would abstract all the
> boilerplate code,
> so it would have the same level of simplicity as this YAML proposal.
>
>
> Looking at what we have for DTS currently, we have put thought into
> achieving this and, in some categories, have made things similar to the
> YAML approach. Like, for example, when it comes to things like sending,
> receiving, and verifying a packet, all you have to do is create a packet
> and call a method. All of the boilerplate regarding how to send and receive
> and how it can be different between types of traffic generators, as well as
> setting up the ability to forward the packet is completely abstracted away
> from you. It's as simple as you send a packet and you get some packets back
> from the test suite perspective. The os_udp testing suite that Juraj wrote
> is a great example of this and how short the simple case can be.
> Additionally we have created centralized APIs between the SUT and TG nodes
> so that all of the methods that the developer would have to use can be
> accessed either through these nodes, or through the method inherited from
> the base class for test suites.
>
> I think that when you try to make things extremely simple there is a
> common tradeoff; the more simplistic it is, the more restrictive it can
> become. We would have to make more assumptions about use cases and how
> things would look in these common cases. This generally isn't a super big
> problem in our case because if support for something that is needed is
> missing, we can add it to the API and with these test cases there are
> things that will generally be similar every time. This tradeoff can also be
> offset by creating the functions to make these things simple while still
> exposing less restrictive underlying methods so that more niche use cases
> can still be fulfilled. Just something to keep in mind.
>
>
>
> > > For the configuration side, YAML is already used in DTS.
> > > For the test suite logic, do you think we can achieve the same
> simplicity with
> > > some Python code?
> > >
> > > We discussed how to progress with this proposal during the CI meeting
> last
> > > week.
> > > We need to check how it could look and what we can improve to reach
> this
> > > goal.
> > > Patrick proposes a meeting this Wednesday at 2pm UTC.
>
>
>
> Generally I think this discussion is very useful and having some kind of
> ideal mock suite would also be useful to reference. It also can help shine
> a light on the importance of thinking about what things could be the common
> case and abstracting them away from the developer in the framework.
>


-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 13585 bytes --]

Re: DTS testpmd and SCAPY integration

[Etelson, Gregory]

[-- Attachment #1: Type: text/plain, Size: 4059 bytes --]

Hello Patrick,

> External email: Use caution opening links or attachments
> Thank you for sharing Gregory. I did not get an opportunity to look through the code today, but I did run
> through the presentation. A few points I noted:
> 1. The presentation shows an example testpmd testcase for creating a flow rule, and then shows a
> validation step in which standard out is compared against the expected string ("flow rule x created") and
> we can conclude whether we are able to create flow rules. Are you also sending packets according to the
> flow rules and validating that what is sent/received corresponds to the expected behavior of the flow
> rules? When I look at the old DTS framework, and an example flow rules testsuite
> (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want feature parity with, I think
> that validation for this testing framework needs to primarily rely on comparing packets sent and packets
> received.

The unit test infrastructure validates flow rule creation and
a result produced by that flow.
Flow result is triggered by a packet.
However, flow result validation does not always can be done by testing a packet.
Unit test implements 2 flow validation methods.

The first validation method tests testpmd output triggered by a test packet.

Example: use the MODIFY_FIELD action to copy packet VLAN ID to flow TAG item.
Flow tag is internal flow resource. It must be validated in DPDK application.

Test creates 2 flow rules:

Rule 1: use MODIFY_FILED to copy packet VLAN ID to flow TAG item
pattern eth / vlan / end \
actions modify_field op set dst_type tag ... src_type vlan_id ... / end

Rule 2: validate the TAG item:
pattern tag data is 0x31 ... / end actions mark id 0xaaa / rss / end

The test sends a packet with VLAN ID 0x31: / Dot1Q(vlan=0x31) /
The test matches tespmd output triggered by the packet for
`FDIR matched ID=0xaaa`.

The second validation method tests a packet after it was processed by a flow.

Unit test operates in a static environment. It does not compare
source and target packets. The test "knows" valid target packet configuration.

Example: push VLAN header into a packet.

There is a single flow rule in that example:
pattern eth / end \
actions of_push_vlan ethertype 0x8100 / \
         of_set_vlan_vid vlan_vid 3103 .../ port_id id 1 / end


There are 2 SCAPY processes in that test: `tg` runs on peer host and
sends a source packet. `vm` runs on the same host as testpmd. It validates
incoming packet.

Phase 0 prepares test packet on the `tg` and starts AsyncSniffer on the `vm`.
Phase 1 sends the packet.
Phase 2 validates the packet.
The test can repeat phases 1 and 2.


phase0:
   vm: |
     sniff = AsyncSniffer(iface=pf1vf0, filter='udp and src port 1234')

   tg: |
     udp_packet = Ether(src='11:22:33:44:55:66',
                        dst='aa:bb:cc:dd:ee:aa')/
                  IP(src='1.1.1.1', dst='2.2.2.2')/
                  UDP(sport=1234, dport=5678)/Raw('== TEST ==')

phase1: &phase1
   vm: sniff.start()
   tg: sendp(udp_packet, iface=pf1)

phase2: &phase2
   vm: |
     cap = sniff.stop()
     if len(cap[UDP]) > 0: cap[UDP][0][Ether].command()
   result:
 	vm: vlan=3103

>In any case, there may be some testsuites which can be written which are small 
>enough in scope
> that validating by standard out in this way may be appropriate. I'm not sure but we should keep our
> options open. 
> 
> 2. If the implementation overhead is not too significant for the configuration step in the DTS execution a
> "--fast" option like you use may be a good improvement for the framework. In your mind, is the main
> benefit A. reduced execution time, B. reduced user setup time (don't have to write full config file) or C.
> Something else?

A user must always provide test configuration.
However a host can already have prepared setup before the test execution.
In that case a user can skip host setup phase and reduce execution time.
 
> 
> Thanks for making this available to use so we can use it as a reference in making DTS better. :) 
> 
>

Re: DTS testpmd and SCAPY integration

[Gregory Etelson]

[-- Attachment #1: Type: text/plain, Size: 5000 bytes --]

Hello Patrick,

Did you have time to check the Unit Test design ?
Do you think it can be used for short functional DTS tests ?

Regards,
Gregory

________________________________
From: Gregory Etelson
Sent: Wednesday, January 31, 2024 09:43
To: Patrick Robb <probb@iol.unh.edu>
Cc: Gregory Etelson <getelson@nvidia.com>; Jeremy Spewock <jspewock@iol.unh.edu>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>; Paul Szczepanek <Paul.Szczepanek@arm.com>; Yoan Picchi <yoan.picchi@foss.arm.com>; Luca Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; dev@dpdk.org <dev@dpdk.org>; nd <nd@arm.com>; Maayan Kashani <mkashani@nvidia.com>; Asaf Penso <asafp@nvidia.com>
Subject: Re: DTS testpmd and SCAPY integration

Hello Patrick,

> External email: Use caution opening links or attachments
> Thank you for sharing Gregory. I did not get an opportunity to look through the code today, but I did run
> through the presentation. A few points I noted:
> 1. The presentation shows an example testpmd testcase for creating a flow rule, and then shows a
> validation step in which standard out is compared against the expected string ("flow rule x created") and
> we can conclude whether we are able to create flow rules. Are you also sending packets according to the
> flow rules and validating that what is sent/received corresponds to the expected behavior of the flow
> rules? When I look at the old DTS framework, and an example flow rules testsuite
> (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want feature parity with, I think
> that validation for this testing framework needs to primarily rely on comparing packets sent and packets
> received.

The unit test infrastructure validates flow rule creation and
a result produced by that flow.
Flow result is triggered by a packet.
However, flow result validation does not always can be done by testing a packet.
Unit test implements 2 flow validation methods.

The first validation method tests testpmd output triggered by a test packet.

Example: use the MODIFY_FIELD action to copy packet VLAN ID to flow TAG item.
Flow tag is internal flow resource. It must be validated in DPDK application.

Test creates 2 flow rules:

Rule 1: use MODIFY_FILED to copy packet VLAN ID to flow TAG item
pattern eth / vlan / end \
actions modify_field op set dst_type tag ... src_type vlan_id ... / end

Rule 2: validate the TAG item:
pattern tag data is 0x31 ... / end actions mark id 0xaaa / rss / end

The test sends a packet with VLAN ID 0x31: / Dot1Q(vlan=0x31) /
The test matches tespmd output triggered by the packet for
`FDIR matched ID=0xaaa`.

The second validation method tests a packet after it was processed by a flow.

Unit test operates in a static environment. It does not compare
source and target packets. The test "knows" valid target packet configuration.

Example: push VLAN header into a packet.

There is a single flow rule in that example:
pattern eth / end \
actions of_push_vlan ethertype 0x8100 / \
         of_set_vlan_vid vlan_vid 3103 .../ port_id id 1 / end


There are 2 SCAPY processes in that test: `tg` runs on peer host and
sends a source packet. `vm` runs on the same host as testpmd. It validates
incoming packet.

Phase 0 prepares test packet on the `tg` and starts AsyncSniffer on the `vm`.
Phase 1 sends the packet.
Phase 2 validates the packet.
The test can repeat phases 1 and 2.


phase0:
   vm: |
     sniff = AsyncSniffer(iface=pf1vf0, filter='udp and src port 1234')

   tg: |
     udp_packet = Ether(src='11:22:33:44:55:66',
                        dst='aa:bb:cc:dd:ee:aa')/
                  IP(src='1.1.1.1', dst='2.2.2.2')/
                  UDP(sport=1234, dport=5678)/Raw('== TEST ==')

phase1: &phase1
   vm: sniff.start()
   tg: sendp(udp_packet, iface=pf1)

phase2: &phase2
   vm: |
     cap = sniff.stop()
     if len(cap[UDP]) > 0: cap[UDP][0][Ether].command()
   result:
         vm: vlan=3103

>In any case, there may be some testsuites which can be written which are small
>enough in scope
> that validating by standard out in this way may be appropriate. I'm not sure but we should keep our
> options open.
>
> 2. If the implementation overhead is not too significant for the configuration step in the DTS execution a
> "--fast" option like you use may be a good improvement for the framework. In your mind, is the main
> benefit A. reduced execution time, B. reduced user setup time (don't have to write full config file) or C.
> Something else?

A user must always provide test configuration.
However a host can already have prepared setup before the test execution.
In that case a user can skip host setup phase and reduce execution time.

>
> Thanks for making this available to use so we can use it as a reference in making DTS better. :)
>
>

[-- Attachment #2: Type: text/html, Size: 8261 bytes --]

Re: DTS testpmd and SCAPY integration

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 7718 bytes --]

Hi Gregory,

I apologize for my delayed response. I have now taken a look through the
code. I do think a number of the design decisions you made for the unit
test framework can be carried over to DTS - in fact we have some tickets
currently for refactoring DTS to this end. Namely:
    -Breaking out test agent setup config and test config into separate
files
    -Reducing the amount of config values which need to be set by the user,
lowering the barrier for entry for running the testing framework.
    -Add some level of testsuite specific YAML which can be mapped to the
testsuite its written for (just for some values, not explicit commands)

But, I figure what you are asking about here really is the design of
setting phases in YAML and writing the scapy/testpmd commands there.
Although I (and I think the DTS group broadly) agree that this system does
provide a good user experience for quickly writing new testsuites, there
are other reasons (testsuite flexibility, platform/TG agnosticism) we don't
want to provide the 100% YAML option for a testsuite. We discussed the idea
of providing two paths for writing a testsuite (YAML file only, or YAML +
python testsuite file), but honestly I don't think it's a good idea to
split our efforts (particularly early on). If we want to consider building
support for a YAML testsuite approach for simple func testsuites in the
(far) future we can discuss it, but that is an unknown. I think our efforts
now need to remain on the python testsuites.

Still, seeing your project has been a big inspiration. When we can abstract
away some boilerplate, or use YAML to override default behavior, we want to
do it. But we still plan to handle sending commands to TG/SUT from the
python testsuite files. I think you are helping us a lot with your ideas,
even if the end implementation is a little different in approach.

Note: incorporating a function(s) to load different mlnx device firmware as
a part of the testsuite is an interesting touch. Do you mind explaining why
it is made a part of the testing framework, as opposed to a pre-step for
the user to complete ahead of running the framework? Does managing loading
the firmware this way make device firmware development and testing it
against DPDK easier, or is there another reason? Thanks.

On Wed, Feb 14, 2024 at 12:27 PM Gregory Etelson <getelson@nvidia.com>
wrote:

> Hello Patrick,
>
> Did you have time to check the Unit Test design ?
> Do you think it can be used for short functional DTS tests ?
>
> Regards,
> Gregory
>
> ------------------------------
> *From:* Gregory Etelson
> *Sent:* Wednesday, January 31, 2024 09:43
> *To:* Patrick Robb <probb@iol.unh.edu>
> *Cc:* Gregory Etelson <getelson@nvidia.com>; Jeremy Spewock <
> jspewock@iol.unh.edu>; NBU-Contact-Thomas Monjalon (EXTERNAL) <
> thomas@monjalon.net>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> Juraj Linkeš <juraj.linkes@pantheon.tech>; Paul Szczepanek <
> Paul.Szczepanek@arm.com>; Yoan Picchi <yoan.picchi@foss.arm.com>; Luca
> Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; dev@dpdk.org
> <dev@dpdk.org>; nd <nd@arm.com>; Maayan Kashani <mkashani@nvidia.com>;
> Asaf Penso <asafp@nvidia.com>
> *Subject:* Re: DTS testpmd and SCAPY integration
>
> Hello Patrick,
>
> > External email: Use caution opening links or attachments
> > Thank you for sharing Gregory. I did not get an opportunity to look
> through the code today, but I did run
> > through the presentation. A few points I noted:
> > 1. The presentation shows an example testpmd testcase for creating a
> flow rule, and then shows a
> > validation step in which standard out is compared against the expected
> string ("flow rule x created") and
> > we can conclude whether we are able to create flow rules. Are you also
> sending packets according to the
> > flow rules and validating that what is sent/received corresponds to the
> expected behavior of the flow
> > rules? When I look at the old DTS framework, and an example flow rules
> testsuite
> > (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we
> want feature parity with, I think
> > that validation for this testing framework needs to primarily rely on
> comparing packets sent and packets
> > received.
>
> The unit test infrastructure validates flow rule creation and
> a result produced by that flow.
> Flow result is triggered by a packet.
> However, flow result validation does not always can be done by testing a
> packet.
> Unit test implements 2 flow validation methods.
>
> The first validation method tests testpmd output triggered by a test
> packet.
>
> Example: use the MODIFY_FIELD action to copy packet VLAN ID to flow TAG
> item.
> Flow tag is internal flow resource. It must be validated in DPDK
> application.
>
> Test creates 2 flow rules:
>
> Rule 1: use MODIFY_FILED to copy packet VLAN ID to flow TAG item
> pattern eth / vlan / end \
> actions modify_field op set dst_type tag ... src_type vlan_id ... / end
>
> Rule 2: validate the TAG item:
> pattern tag data is 0x31 ... / end actions mark id 0xaaa / rss / end
>
> The test sends a packet with VLAN ID 0x31: / Dot1Q(vlan=0x31) /
> The test matches tespmd output triggered by the packet for
> `FDIR matched ID=0xaaa`.
>
> The second validation method tests a packet after it was processed by a
> flow.
>
> Unit test operates in a static environment. It does not compare
> source and target packets. The test "knows" valid target packet
> configuration.
>
> Example: push VLAN header into a packet.
>
> There is a single flow rule in that example:
> pattern eth / end \
> actions of_push_vlan ethertype 0x8100 / \
>          of_set_vlan_vid vlan_vid 3103 .../ port_id id 1 / end
>
>
> There are 2 SCAPY processes in that test: `tg` runs on peer host and
> sends a source packet. `vm` runs on the same host as testpmd. It validates
> incoming packet.
>
> Phase 0 prepares test packet on the `tg` and starts AsyncSniffer on the
> `vm`.
> Phase 1 sends the packet.
> Phase 2 validates the packet.
> The test can repeat phases 1 and 2.
>
>
> phase0:
>    vm: |
>      sniff = AsyncSniffer(iface=pf1vf0, filter='udp and src port 1234')
>
>    tg: |
>      udp_packet = Ether(src='11:22:33:44:55:66',
>                         dst='aa:bb:cc:dd:ee:aa')/
>                   IP(src='1.1.1.1', dst='2.2.2.2')/
>                   UDP(sport=1234, dport=5678)/Raw('== TEST ==')
>
> phase1: &phase1
>    vm: sniff.start()
>    tg: sendp(udp_packet, iface=pf1)
>
> phase2: &phase2
>    vm: |
>      cap = sniff.stop()
>      if len(cap[UDP]) > 0: cap[UDP][0][Ether].command()
>    result:
>          vm: vlan=3103
>
> >In any case, there may be some testsuites which can be written which are
> small
> >enough in scope
> > that validating by standard out in this way may be appropriate. I'm not
> sure but we should keep our
> > options open.
> >
> > 2. If the implementation overhead is not too significant for the
> configuration step in the DTS execution a
> > "--fast" option like you use may be a good improvement for the
> framework. In your mind, is the main
> > benefit A. reduced execution time, B. reduced user setup time (don't
> have to write full config file) or C.
> > Something else?
>
> A user must always provide test configuration.
> However a host can already have prepared setup before the test execution.
> In that case a user can skip host setup phase and reduce execution time.
>
> >
> > Thanks for making this available to use so we can use it as a reference
> in making DTS better. :)
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 11046 bytes --]

Re: DTS testpmd and SCAPY integration

[Gregory Etelson]

[-- Attachment #1: Type: text/plain, Size: 9301 bytes --]

Hello Patrick,

Note: incorporating a function(s) to load different mlnx device firmware as a part of the testsuite is an interesting touch. Do you mind explaining why it is made a part of the testing framework, as opposed to a pre-step for the user to complete ahead of running the framework? Does managing loading the firmware this way make device firmware development and testing it against DPDK easier, or is there another reason?

The Unit Test was designed as an infrastructure to test DPDK feature functionality during and after the feature development.
Some new DPDK features require specific firmware version.
Unit test configuration scheme allows to specify minimal firmware version required for the tested feature.

Regards,
Gregory
________________________________
From: Patrick Robb <probb@iol.unh.edu>
Sent: Monday, February 19, 2024 07:08
To: Gregory Etelson <getelson@nvidia.com>
Cc: Jeremy Spewock <jspewock@iol.unh.edu>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>; Paul Szczepanek <Paul.Szczepanek@arm.com>; Yoan Picchi <yoan.picchi@foss.arm.com>; Luca Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; dev@dpdk.org <dev@dpdk.org>; nd <nd@arm.com>; Maayan Kashani <mkashani@nvidia.com>; Asaf Penso <asafp@nvidia.com>
Subject: Re: DTS testpmd and SCAPY integration

External email: Use caution opening links or attachments

Hi Gregory,

I apologize for my delayed response. I have now taken a look through the code. I do think a number of the design decisions you made for the unit test framework can be carried over to DTS - in fact we have some tickets currently for refactoring DTS to this end. Namely:
    -Breaking out test agent setup config and test config into separate files
    -Reducing the amount of config values which need to be set by the user, lowering the barrier for entry for running the testing framework.
    -Add some level of testsuite specific YAML which can be mapped to the testsuite its written for (just for some values, not explicit commands)

But, I figure what you are asking about here really is the design of setting phases in YAML and writing the scapy/testpmd commands there. Although I (and I think the DTS group broadly) agree that this system does provide a good user experience for quickly writing new testsuites, there are other reasons (testsuite flexibility, platform/TG agnosticism) we don't want to provide the 100% YAML option for a testsuite. We discussed the idea of providing two paths for writing a testsuite (YAML file only, or YAML + python testsuite file), but honestly I don't think it's a good idea to split our efforts (particularly early on). If we want to consider building support for a YAML testsuite approach for simple func testsuites in the (far) future we can discuss it, but that is an unknown. I think our efforts now need to remain on the python testsuites.

Still, seeing your project has been a big inspiration. When we can abstract away some boilerplate, or use YAML to override default behavior, we want to do it. But we still plan to handle sending commands to TG/SUT from the python testsuite files. I think you are helping us a lot with your ideas, even if the end implementation is a little different in approach.

Note: incorporating a function(s) to load different mlnx device firmware as a part of the testsuite is an interesting touch. Do you mind explaining why it is made a part of the testing framework, as opposed to a pre-step for the user to complete ahead of running the framework? Does managing loading the firmware this way make device firmware development and testing it against DPDK easier, or is there another reason? Thanks.

On Wed, Feb 14, 2024 at 12:27 PM Gregory Etelson <getelson@nvidia.com<mailto:getelson@nvidia.com>> wrote:
Hello Patrick,

Did you have time to check the Unit Test design ?
Do you think it can be used for short functional DTS tests ?

Regards,
Gregory

________________________________
From: Gregory Etelson
Sent: Wednesday, January 31, 2024 09:43
To: Patrick Robb <probb@iol.unh.edu<mailto:probb@iol.unh.edu>>
Cc: Gregory Etelson <getelson@nvidia.com<mailto:getelson@nvidia.com>>; Jeremy Spewock <jspewock@iol.unh.edu<mailto:jspewock@iol.unh.edu>>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net<mailto:thomas@monjalon.net>>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>; Juraj Linkeš <juraj.linkes@pantheon.tech>; Paul Szczepanek <Paul.Szczepanek@arm.com<mailto:Paul.Szczepanek@arm.com>>; Yoan Picchi <yoan.picchi@foss.arm.com<mailto:yoan.picchi@foss.arm.com>>; Luca Vizzarro <Luca.Vizzarro@arm.com<mailto:Luca.Vizzarro@arm.com>>; ci@dpdk.org<mailto:ci@dpdk.org> <ci@dpdk.org<mailto:ci@dpdk.org>>; dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>; nd <nd@arm.com<mailto:nd@arm.com>>; Maayan Kashani <mkashani@nvidia.com<mailto:mkashani@nvidia.com>>; Asaf Penso <asafp@nvidia.com<mailto:asafp@nvidia.com>>
Subject: Re: DTS testpmd and SCAPY integration

Hello Patrick,

> External email: Use caution opening links or attachments
> Thank you for sharing Gregory. I did not get an opportunity to look through the code today, but I did run
> through the presentation. A few points I noted:
> 1. The presentation shows an example testpmd testcase for creating a flow rule, and then shows a
> validation step in which standard out is compared against the expected string ("flow rule x created") and
> we can conclude whether we are able to create flow rules. Are you also sending packets according to the
> flow rules and validating that what is sent/received corresponds to the expected behavior of the flow
> rules? When I look at the old DTS framework, and an example flow rules testsuite
> (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want feature parity with, I think
> that validation for this testing framework needs to primarily rely on comparing packets sent and packets
> received.

The unit test infrastructure validates flow rule creation and
a result produced by that flow.
Flow result is triggered by a packet.
However, flow result validation does not always can be done by testing a packet.
Unit test implements 2 flow validation methods.

The first validation method tests testpmd output triggered by a test packet.

Example: use the MODIFY_FIELD action to copy packet VLAN ID to flow TAG item.
Flow tag is internal flow resource. It must be validated in DPDK application.

Test creates 2 flow rules:

Rule 1: use MODIFY_FILED to copy packet VLAN ID to flow TAG item
pattern eth / vlan / end \
actions modify_field op set dst_type tag ... src_type vlan_id ... / end

Rule 2: validate the TAG item:
pattern tag data is 0x31 ... / end actions mark id 0xaaa / rss / end

The test sends a packet with VLAN ID 0x31: / Dot1Q(vlan=0x31) /
The test matches tespmd output triggered by the packet for
`FDIR matched ID=0xaaa`.

The second validation method tests a packet after it was processed by a flow.

Unit test operates in a static environment. It does not compare
source and target packets. The test "knows" valid target packet configuration.

Example: push VLAN header into a packet.

There is a single flow rule in that example:
pattern eth / end \
actions of_push_vlan ethertype 0x8100 / \
         of_set_vlan_vid vlan_vid 3103 .../ port_id id 1 / end


There are 2 SCAPY processes in that test: `tg` runs on peer host and
sends a source packet. `vm` runs on the same host as testpmd. It validates
incoming packet.

Phase 0 prepares test packet on the `tg` and starts AsyncSniffer on the `vm`.
Phase 1 sends the packet.
Phase 2 validates the packet.
The test can repeat phases 1 and 2.


phase0:
   vm: |
     sniff = AsyncSniffer(iface=pf1vf0, filter='udp and src port 1234')

   tg: |
     udp_packet = Ether(src='11:22:33:44:55:66',
                        dst='aa:bb:cc:dd:ee:aa')/
                  IP(src='1.1.1.1', dst='2.2.2.2')/
                  UDP(sport=1234, dport=5678)/Raw('== TEST ==')

phase1: &phase1
   vm: sniff.start()
   tg: sendp(udp_packet, iface=pf1)

phase2: &phase2
   vm: |
     cap = sniff.stop()
     if len(cap[UDP]) > 0: cap[UDP][0][Ether].command()
   result:
         vm: vlan=3103

>In any case, there may be some testsuites which can be written which are small
>enough in scope
> that validating by standard out in this way may be appropriate. I'm not sure but we should keep our
> options open.
>
> 2. If the implementation overhead is not too significant for the configuration step in the DTS execution a
> "--fast" option like you use may be a good improvement for the framework. In your mind, is the main
> benefit A. reduced execution time, B. reduced user setup time (don't have to write full config file) or C.
> Something else?

A user must always provide test configuration.
However a host can already have prepared setup before the test execution.
In that case a user can skip host setup phase and reduce execution time.

>
> Thanks for making this available to use so we can use it as a reference in making DTS better. :)
>
>


[-- Attachment #2: Type: text/html, Size: 15480 bytes --]