From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3838543B48; Mon, 19 Feb 2024 06:08:44 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2F15340608; Mon, 19 Feb 2024 06:08:44 +0100 (CET) Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) by mails.dpdk.org (Postfix) with ESMTP id F29784026C for ; Mon, 19 Feb 2024 06:08:40 +0100 (CET) Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3bb9b28acb4so2849849b6e.2 for ; Sun, 18 Feb 2024 21:08:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1708319320; x=1708924120; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=E5+mYhAQxHStx4OFci07TG47Dt3FNkyKUYzUBj3wkic=; b=D2o3fD+GQCX56QmIhVskklnEJb5vyqagpjM0WHYY5EbM9Qx8mHyeq3qoZN8YP9/p2y 55PQIesSR/c2RezuY6hi6h3b6WWaSGN1+VopCboHndAZtmV0Q3kfewt/IADMO/orOKNt d6pVwLNX0zTKDMCKJBbUwN75d/TErgZoKev24= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708319320; x=1708924120; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=E5+mYhAQxHStx4OFci07TG47Dt3FNkyKUYzUBj3wkic=; b=bDju0NJFQP/rgteiEiM8g4AzNEKxzUeJL7xaAVEP9xtbdJmgnKJuprz/G0Ch6pjNxB wG5EsupN0Km6hrRYpXtshDIYIIKn8d/+aQKS0C3zurWbvWs3fZ37psu9A134IATWPaW7 n9QY1Qz52zlzfw/NrRtHV7pOoxJm27XfIvEJCJ00BnEifeBJxTgHmCw8u0IFXvOX8rWE 3etGhqaiW4kbovunad02qhFc0yMir4efKiNGPH0MpT/+t7Lv+xeccWYgSdZ1Zz9smY/H 90djzZY+F1oi8RoQbmP68L/AlXQhQAMfjiUqiG/mA9u0N8Ga8Wjn3f7fiI0g2u51H0Jp Yx3g== X-Forwarded-Encrypted: i=1; AJvYcCUM4YXbitX+/0TDpjXIZzP8S5okJBjSETtgGrESh0OEHKq1aK7FY6p1+l9tmSE2wBqzBe4+0hefV9CohA== X-Gm-Message-State: AOJu0YwRefPF03cSAd8DLJt4vF1NvJ4RUvmvjzj8MCOtzHMbzYIvfEEF zfQcUKxP0O4vKNFe+xK5JTfxhkMlZjd7+E9sMD58xVLnyXxO5fTR79wr4amp/em+r+KAnld0Mc9 Vj+HhRQBeyI2gMAgOj0vXYTBdUBPy3zfqAzsjXQ== X-Google-Smtp-Source: AGHT+IF6U5kLQImelRr9DZ2CXzbDOHYeon1sS6TzLW1E7jiXpzLQOEOfvx3csbpmBsyIAxMNPdSFpBGoGa1/zxZh6SY= X-Received: by 2002:a05:6830:2014:b0:6e4:34ba:adcc with SMTP id e20-20020a056830201400b006e434baadccmr7766712otp.7.1708319320013; Sun, 18 Feb 2024 21:08:40 -0800 (PST) MIME-Version: 1.0 References: <2a287ee7-cda4-f2ab-a4e6-a47021f8573f@nvidia.com> <2127794.bB369e8A3T@thomas> <6608561.G0QQBjFxQf@thomas> In-Reply-To: From: Patrick Robb Date: Mon, 19 Feb 2024 00:08:29 -0500 Message-ID: Subject: Re: DTS testpmd and SCAPY integration To: Gregory Etelson Cc: Jeremy Spewock , "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Honnappa Nagarahalli , =?UTF-8?Q?Juraj_Linke=C5=A1?= , Paul Szczepanek , Yoan Picchi , Luca Vizzarro , "ci@dpdk.org" , "dev@dpdk.org" , nd , Maayan Kashani , Asaf Penso Content-Type: multipart/alternative; boundary="0000000000003b90100611b516b6" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --0000000000003b90100611b516b6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Gregory, I apologize for my delayed response. I have now taken a look through the code. I do think a number of the design decisions you made for the unit test framework can be carried over to DTS - in fact we have some tickets currently for refactoring DTS to this end. Namely: -Breaking out test agent setup config and test config into separate files -Reducing the amount of config values which need to be set by the user, lowering the barrier for entry for running the testing framework. -Add some level of testsuite specific YAML which can be mapped to the testsuite its written for (just for some values, not explicit commands) But, I figure what you are asking about here really is the design of setting phases in YAML and writing the scapy/testpmd commands there. Although I (and I think the DTS group broadly) agree that this system does provide a good user experience for quickly writing new testsuites, there are other reasons (testsuite flexibility, platform/TG agnosticism) we don't want to provide the 100% YAML option for a testsuite. We discussed the idea of providing two paths for writing a testsuite (YAML file only, or YAML + python testsuite file), but honestly I don't think it's a good idea to split our efforts (particularly early on). If we want to consider building support for a YAML testsuite approach for simple func testsuites in the (far) future we can discuss it, but that is an unknown. I think our efforts now need to remain on the python testsuites. Still, seeing your project has been a big inspiration. When we can abstract away some boilerplate, or use YAML to override default behavior, we want to do it. But we still plan to handle sending commands to TG/SUT from the python testsuite files. I think you are helping us a lot with your ideas, even if the end implementation is a little different in approach. Note: incorporating a function(s) to load different mlnx device firmware as a part of the testsuite is an interesting touch. Do you mind explaining why it is made a part of the testing framework, as opposed to a pre-step for the user to complete ahead of running the framework? Does managing loading the firmware this way make device firmware development and testing it against DPDK easier, or is there another reason? Thanks. On Wed, Feb 14, 2024 at 12:27=E2=80=AFPM Gregory Etelson wrote: > Hello Patrick, > > Did you have time to check the Unit Test design ? > Do you think it can be used for short functional DTS tests ? > > Regards, > Gregory > > ------------------------------ > *From:* Gregory Etelson > *Sent:* Wednesday, January 31, 2024 09:43 > *To:* Patrick Robb > *Cc:* Gregory Etelson ; Jeremy Spewock < > jspewock@iol.unh.edu>; NBU-Contact-Thomas Monjalon (EXTERNAL) < > thomas@monjalon.net>; Honnappa Nagarahalli = ; > Juraj Linke=C5=A1 ; Paul Szczepanek < > Paul.Szczepanek@arm.com>; Yoan Picchi ; Luca > Vizzarro ; ci@dpdk.org ; dev@dpdk.org > ; nd ; Maayan Kashani ; > Asaf Penso > *Subject:* Re: DTS testpmd and SCAPY integration > > Hello Patrick, > > > External email: Use caution opening links or attachments > > Thank you for sharing Gregory. I did not get an opportunity to look > through the code today, but I did run > > through the presentation. A few points I noted: > > 1. The presentation shows an example testpmd testcase for creating a > flow rule, and then shows a > > validation step in which standard out is compared against the expected > string ("flow rule x created") and > > we can conclude whether we are able to create flow rules. Are you also > sending packets according to the > > flow rules and validating that what is sent/received corresponds to the > expected behavior of the flow > > rules? When I look at the old DTS framework, and an example flow rules > testsuite > > (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we > want feature parity with, I think > > that validation for this testing framework needs to primarily rely on > comparing packets sent and packets > > received. > > The unit test infrastructure validates flow rule creation and > a result produced by that flow. > Flow result is triggered by a packet. > However, flow result validation does not always can be done by testing a > packet. > Unit test implements 2 flow validation methods. > > The first validation method tests testpmd output triggered by a test > packet. > > Example: use the MODIFY_FIELD action to copy packet VLAN ID to flow TAG > item. > Flow tag is internal flow resource. It must be validated in DPDK > application. > > Test creates 2 flow rules: > > Rule 1: use MODIFY_FILED to copy packet VLAN ID to flow TAG item > pattern eth / vlan / end \ > actions modify_field op set dst_type tag ... src_type vlan_id ... / end > > Rule 2: validate the TAG item: > pattern tag data is 0x31 ... / end actions mark id 0xaaa / rss / end > > The test sends a packet with VLAN ID 0x31: / Dot1Q(vlan=3D0x31) / > The test matches tespmd output triggered by the packet for > `FDIR matched ID=3D0xaaa`. > > The second validation method tests a packet after it was processed by a > flow. > > Unit test operates in a static environment. It does not compare > source and target packets. The test "knows" valid target packet > configuration. > > Example: push VLAN header into a packet. > > There is a single flow rule in that example: > pattern eth / end \ > actions of_push_vlan ethertype 0x8100 / \ > of_set_vlan_vid vlan_vid 3103 .../ port_id id 1 / end > > > There are 2 SCAPY processes in that test: `tg` runs on peer host and > sends a source packet. `vm` runs on the same host as testpmd. It validate= s > incoming packet. > > Phase 0 prepares test packet on the `tg` and starts AsyncSniffer on the > `vm`. > Phase 1 sends the packet. > Phase 2 validates the packet. > The test can repeat phases 1 and 2. > > > phase0: > vm: | > sniff =3D AsyncSniffer(iface=3Dpf1vf0, filter=3D'udp and src port 12= 34') > > tg: | > udp_packet =3D Ether(src=3D'11:22:33:44:55:66', > dst=3D'aa:bb:cc:dd:ee:aa')/ > IP(src=3D'1.1.1.1', dst=3D'2.2.2.2')/ > UDP(sport=3D1234, dport=3D5678)/Raw('=3D=3D TEST =3D=3D= ') > > phase1: &phase1 > vm: sniff.start() > tg: sendp(udp_packet, iface=3Dpf1) > > phase2: &phase2 > vm: | > cap =3D sniff.stop() > if len(cap[UDP]) > 0: cap[UDP][0][Ether].command() > result: > vm: vlan=3D3103 > > >In any case, there may be some testsuites which can be written which are > small > >enough in scope > > that validating by standard out in this way may be appropriate. I'm not > sure but we should keep our > > options open. > > > > 2. If the implementation overhead is not too significant for the > configuration step in the DTS execution a > > "--fast" option like you use may be a good improvement for the > framework. In your mind, is the main > > benefit A. reduced execution time, B. reduced user setup time (don't > have to write full config file) or C. > > Something else? > > A user must always provide test configuration. > However a host can already have prepared setup before the test execution. > In that case a user can skip host setup phase and reduce execution time. > > > > > Thanks for making this available to use so we can use it as a reference > in making DTS better. :) > > > > > --0000000000003b90100611b516b6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Gregory,

I apologize for my del= ayed response. I have now taken a look through the code. I do think a numbe= r of the design decisions you made for the unit test framework can be carri= ed over to DTS - in fact we have some tickets currently for refactoring DTS= to this end. Namely:
=C2=A0 =C2=A0 -Breaking out test agent setup conf= ig and test config into separate files
=C2=A0 =C2=A0 -Reducing the amoun= t of config values which need to be set by the user, lowering the barrier f= or entry for running the testing framework.
=C2=A0 =C2=A0 -Add some leve= l of testsuite specific YAML which can be mapped to the testsuite its writt= en for (just for some values, not explicit commands)
=C2=A0 =C2=A0
B= ut, I figure what you are asking about here really is the=C2=A0design of se= tting phases in YAML and writing the scapy/testpmd commands there. Although= I (and I think the DTS group broadly) agree that this system does provide = a good user experience for quickly writing new testsuites, there are other = reasons (testsuite flexibility, platform/TG agnosticism) we don't want = to provide the 100% YAML option for a testsuite. We discussed the idea of p= roviding two paths for writing a testsuite (YAML file only, or YAML + pytho= n testsuite file), but honestly I don't think it's a good idea to s= plit our efforts (particularly early on). If we want to consider building s= upport for a YAML testsuite approach for simple func testsuites in the (far= ) future we can discuss it, but that is an unknown. I think our efforts now= need to remain on the python testsuites.

Still, seeing your project= has been a big inspiration. When we can abstract away some boilerplate, or= use YAML to override default behavior, we want to do it. But we still plan= to handle sending commands to TG/SUT from the python testsuite files. I th= ink you are helping us a lot with your ideas, even if the end implementatio= n is a little different in approach.
=C2=A0 =C2=A0
Note: incorporati= ng a function(s) to load different mlnx device firmware as a part of the te= stsuite is an interesting touch. Do you mind explaining why it is made a pa= rt of the testing framework, as opposed to a pre-step for the user to compl= ete ahead of running the framework? Does managing loading the firmware this= way make device firmware development and testing it against DPDK easier, o= r is there another reason? Thanks.=C2=A0

On Wed, Feb 14, 2024 at 12:27= =E2=80=AFPM Gregory Etelson <gete= lson@nvidia.com> wrote:
Hello Patrick,

Did you have time to check the Unit Test design ?
Do you think it can be used for short functional DTS tests ?

Regards,
Gregory

From:= =C2=A0Gregory Etelson
Sent:=C2=A0Wednesday, January 31, 2024 09:43
To:=C2=A0Patrick Robb <probb@iol.unh.edu>
Cc:=C2=A0Gregory Etelson <getelson@nvidia.com>; Jeremy Spewock <jspewock@iol.unh.edu= >; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; Honnappa Nagara= halli <Honnappa.Nagarahalli@arm.com>; Juraj Linke=C5=A1 <juraj.linkes@p= antheon.tech>; Paul Szczepanek <Paul.Szcz= epanek@arm.com>; Yoan Picchi <yoan.picchi@foss.arm.com>; Luca Vizzarro = <Luca.Vizzarr= o@arm.com>; ci@dpdk= .org <ci@dpdk.org>; dev@dpdk.org = <dev@dpdk.org>;= nd <nd@arm.com>;= Maayan Kashani <mkashani@nvidia.com>; Asaf Penso <asafp@nvidia.com>
Subject:=C2=A0Re: DTS testpmd and SCAPY integration
=C2=A0
Hello Patrick,

> External email: Use caution opening links or attachments
> Thank you for sharing Gregory. I did not get an opportunity to look th= rough the code today, but I did run
> through the presentation. A few points I noted:
> 1. The presentation shows an example testpmd testcase for creating a f= low rule, and then shows a
> validation step in which standard out is compared against the expected= string ("flow rule x created") and
> we can conclude whether we are able to create flow rules. Are you also= sending packets according to the
> flow rules and validating that what is sent/received=C2=A0corresponds = to the expected behavior of the flow
> rules? When I look at the old DTS framework, and an example flow rules= testsuite
> (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want feature parity with, I think
> that validation for this testing framework needs to primarily rely on = comparing packets sent and packets
> received.

The unit test infrastructure validates flow rule creation and
a result produced by that flow.
Flow result is triggered by a packet.
However, flow result validation does not always can be done by testing a pa= cket.
Unit test implements 2 flow validation methods.

The first validation method tests testpmd output triggered by a test packet= .

Example: use the MODIFY_FIELD action to copy packet VLAN ID to flow TAG ite= m.
Flow tag is internal flow resource. It must be validated in DPDK applicatio= n.

Test creates 2 flow rules:

Rule 1: use MODIFY_FILED to copy packet VLAN ID to flow TAG item
pattern eth / vlan / end \
actions modify_field op set dst_type tag ... src_type vlan_id ... / end

Rule 2: validate the TAG item:
pattern tag data is 0x31 ... / end actions mark id 0xaaa / rss / end

The test sends a packet with VLAN ID 0x31: / Dot1Q(vlan=3D0x31) /
The test matches tespmd output triggered by the packet for
`FDIR matched ID=3D0xaaa`.

The second validation method tests a packet after it was processed by a flo= w.

Unit test operates in a static environment. It does not compare
source and target packets. The test "knows" valid target packet c= onfiguration.

Example: push VLAN header into a packet.

There is a single flow rule in that example:
pattern eth / end \
actions of_push_vlan ethertype 0x8100 / \
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 of_set_vlan_vid vlan_vid 3= 103 .../ port_id id 1 / end


There are 2 SCAPY processes in that test: `tg` runs on peer host and
sends a source packet. `vm` runs on the same host as testpmd. It validates<= br> incoming packet.

Phase 0 prepares test packet on the `tg` and starts AsyncSniffer on the `vm= `.
Phase 1 sends the packet.
Phase 2 validates the packet.
The test can repeat phases 1 and 2.


phase0:
=C2=A0=C2=A0 vm: |
=C2=A0=C2=A0=C2=A0=C2=A0 sniff =3D AsyncSniffer(iface=3Dpf1vf0, filter=3D&#= 39;udp and src port 1234')

=C2=A0=C2=A0 tg: |
=C2=A0=C2=A0=C2=A0=C2=A0 udp_packet =3D Ether(src=3D'11:22:33:44:55:66&= #39;,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dst=3D'= aa:bb:cc:dd:ee:aa')/
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 IP(src=3D'1.1.1.1', dst=3D'2.2.2.2&= #39;)/
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 UDP(sport=3D1234, dport=3D5678)/Raw('=3D=3D= TEST =3D=3D')

phase1: &phase1
=C2=A0=C2=A0 vm: sniff.start()
=C2=A0=C2=A0 tg: sendp(udp_packet, iface=3Dpf1)

phase2: &phase2
=C2=A0=C2=A0 vm: |
=C2=A0=C2=A0=C2=A0=C2=A0 cap =3D sniff.stop()
=C2=A0=C2=A0=C2=A0=C2=A0 if len(cap[UDP]) > 0: cap[UDP][0][Ether].comman= d()
=C2=A0=C2=A0 result:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 vm: vlan=3D3103

>In any case, there may be some testsuites which can be written which ar= e small
>enough in scope
> that validating by standard out in this way may be appropriate. I'= m not sure but we should keep our
> options open.=C2=A0
>
> 2. If the implementation overhead is not too significant for the confi= guration step in the DTS execution a
> "--fast" option like you use may be a good improvement for t= he framework. In your mind, is the main
> benefit A. reduced execution time, B. reduced user setup time (don'= ;t have to write full config file) or C.
> Something else?

A user must always provide test configuration.
However a host can already have prepared setup before the test execution. In that case a user can skip host setup phase and reduce execution time. =C2=A0
>
> Thanks for making this available to use so we can use it as a referenc= e in making DTS better. :)=C2=A0
>
>

--0000000000003b90100611b516b6--