From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 16E5B43A15; Tue, 30 Jan 2024 23:03:18 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0F24E40EE2; Tue, 30 Jan 2024 23:03:18 +0100 (CET) Received: from mail-oo1-f48.google.com (mail-oo1-f48.google.com [209.85.161.48]) by mails.dpdk.org (Postfix) with ESMTP id A5FF8402CE for ; Tue, 30 Jan 2024 23:03:15 +0100 (CET) Received: by mail-oo1-f48.google.com with SMTP id 006d021491bc7-595aa5b1fe0so2776528eaf.2 for ; Tue, 30 Jan 2024 14:03:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1706652195; x=1707256995; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=cDl9dTMY38YaK7qSIPhWA3YlQ4qliUBA4AK5LMZuL1s=; b=OGmsjeLuSNCo6i9GmNOtXDWsHX4929ufDZ6s0DquN2Y+GNjgsJRdNolwnMrFuX8jce 58U4X34k5/sLiqPQlV6sDfO6fA0wVyB0A0hHMBUEAuzMnkTcH00Nu8ZTrOxFZk98qzlM JNmdHo9jesGIxlBQL/nr2Q9tIwbgS4mS+ZGkI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706652195; x=1707256995; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cDl9dTMY38YaK7qSIPhWA3YlQ4qliUBA4AK5LMZuL1s=; b=RwGAg7dGalicGzM9k6vzFe6b2iZrfRlopVxvb9NYITm/JbkiNQXMwDnarDBoi6XNN1 JCuixYQJie2ElOSpLXoB5QJQ9ac9Dw28CZEKh5arRCttJ+4nNVUKHpYQpd/RKAxeu5Ks ImDrqlT72/KDMB5tsFfpf/0ENvCZKKcBp96up57rMbhuf/Gn8fv2UOixdDBt+SHO+rpg seDXje5tKY9iwUi83ti6wPQ92tk33tv8+ESMKX/eQThzKOIDZEzBXZx7bm8GcwUqBE2T c3X4jPQMOh5ZCj71U8HIKREsnM7I7r308nqUYSHaQ1mw0r2zgNRTzkmkBIMXsQ9rQxwe DPOQ== X-Gm-Message-State: AOJu0YwJ0NrfAL9AjDxvOopWZf8iTJRm6g7ptAc36HoTwooTByUyDdsN 40N3lM0NMNGggnWMCp5IWkdRdHKbSswula2kLYq5mCnRIK5auDN4aD+5eCR0q/Ue5RB8euWJ30M v1FfJ1s3Iu3gQet3jKQu0C/vD0pEJVYfkftjayw== X-Google-Smtp-Source: AGHT+IHyQPIoI2hp4GQJlBAJmBQtbZnVfl4UZvwZkjW94iPm3prdcyR7DuTX55j5k18VqBF3FFAtOeGmvjkt86fwu5U= X-Received: by 2002:a05:6820:1623:b0:59a:1237:b38e with SMTP id bb35-20020a056820162300b0059a1237b38emr9260637oob.11.1706652194612; Tue, 30 Jan 2024 14:03:14 -0800 (PST) MIME-Version: 1.0 References: <2a287ee7-cda4-f2ab-a4e6-a47021f8573f@nvidia.com> <2127794.bB369e8A3T@thomas> <6608561.G0QQBjFxQf@thomas> In-Reply-To: From: Patrick Robb Date: Tue, 30 Jan 2024 17:03:03 -0500 Message-ID: Subject: Re: DTS testpmd and SCAPY integration To: Gregory Etelson Cc: Jeremy Spewock , "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Honnappa Nagarahalli , =?UTF-8?Q?Juraj_Linke=C5=A1?= , Paul Szczepanek , Yoan Picchi , Luca Vizzarro , "ci@dpdk.org" , "dev@dpdk.org" , nd , Maayan Kashani , Asaf Penso Content-Type: multipart/alternative; boundary="000000000000d0bcff061030ed66" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --000000000000d0bcff061030ed66 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you for sharing Gregory. I did not get an opportunity to look through the code today, but I did run through the presentation. A few points I noted: 1. The presentation shows an example testpmd testcase for creating a flow rule, and then shows a validation step in which standard out is compared against the expected string ("flow rule x created") and we can conclude whether we are able to create flow rules. Are you also sending packets according to the flow rules and validating that what is sent/received corresponds to the expected behavior of the flow rules? When I look at the old DTS framework, and an example flow rules testsuite ( https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want feature parity with, I think that validation for this testing framework needs to primarily rely on comparing packets sent and packets received. In any case, there may be some testsuites which can be written which are small enough in scope that validating by standard out in this way may be appropriate. I'm not sure but we should keep our options open. 2. If the implementation overhead is not too significant for the configuration step in the DTS execution a "--fast" option like you use may be a good improvement for the framework. In your mind, is the main benefit A. reduced execution time, B. reduced user setup time (don't have to write full config file) or C. Something else? Thanks for making this available to use so we can use it as a reference in making DTS better. :) On Sun, Jan 28, 2024 at 8:44=E2=80=AFAM Gregory Etelson wrote: > Hello, > > I've uploaded the code snapshot and functional description here: > > https://drive.google.com/drive/folders/1cHrPwx4fUJ6ibUCtHd4kNKsrmmvQvvOj?= usp=3Ddrive_link > > Regards, > Gregory > ------------------------------ > *From:* Jeremy Spewock > *Sent:* Tuesday, January 23, 2024 20:26 > *To:* NBU-Contact-Thomas Monjalon (EXTERNAL) > *Cc:* Honnappa Nagarahalli ; Gregory > Etelson ; Juraj Linke=C5=A1 ; > Paul Szczepanek ; Yoan Picchi < > yoan.picchi@foss.arm.com>; Patrick Robb ; Luca > Vizzarro ; ci@dpdk.org ; dev@dpdk.org > ; nd > *Subject:* Re: DTS testpmd and SCAPY integration > > *External email: Use caution opening links or attachments* > > > On Tue, Jan 23, 2024 at 3:50=E2=80=AFAM Thomas Monjalon > wrote: > > 23/01/2024 04:42, Honnappa Nagarahalli: > > > 08/01/2024 13:10, Luca Vizzarro: > > > > Your proposal sounds rather interesting. Certainly enabling DTS to > > > > accept YAML-written tests sounds more developer-friendly and should > > > > enable quicker test-writing. As this is an extra feature though =E2= =80=93 and > > > > a nice-to-have, it should definitely be discussed in the DTS meetin= gs > > > > as Honnappa suggested already. > > > > > > I would not classify this idea as "nice-to-have". > > > I would split this proposal in 2 parts: > > > 1/ YAML is an implementation alternative. > > > 2/ Being able to write a test with a small number of lines, > > > reusing some commands from existing tools, > > > should be our "must-have" common goal. > > > > > > Others have mentioned that YAML may not be suitable in complex cases, > and > > > that it would be an additional language for test writing. > > > I personnaly think we should focus on a single path which is easy to > read and > > > maintain. > > > > I think we are digressing from the plan we had put forward if we have t= o > go down this path. > > We should understand what it means by going down the YAML format. > > Also, what would happen if there is another innovation in 3 months? > > There is a misunderstanding here. > I suggest to take this proposal as an example of the simplicity to reach. > But I agree with you it is more reasonnable to continue with the Python > path. > > > I agree that it would be easier to develop both the framework and some of > the more complex test cases in python and think sticking with Python is a > good route. > > I think taking this proposal and using it as an example of something we > could work towards is also a great outlook because it gives a more > structured goal instead of the idea being "let's just make it as simple a= s > we can." > > > > > We already have scatter-gather test suite ported to DPDK repo and has > undergone review in the community. > > > > In the last meeting we went through a simple test case. Is it possible > to write the scatter-gather test case in YAML and see how they compare? > > After the latest CI meeting we thought about writing a simple text case > in Python with some virtual functions which would abstract all the > boilerplate code, > so it would have the same level of simplicity as this YAML proposal. > > > Looking at what we have for DTS currently, we have put thought into > achieving this and, in some categories, have made things similar to the > YAML approach. Like, for example, when it comes to things like sending, > receiving, and verifying a packet, all you have to do is create a packet > and call a method. All of the boilerplate regarding how to send and recei= ve > and how it can be different between types of traffic generators, as well = as > setting up the ability to forward the packet is completely abstracted awa= y > from you. It's as simple as you send a packet and you get some packets ba= ck > from the test suite perspective. The os_udp testing suite that Juraj wrot= e > is a great example of this and how short the simple case can be. > Additionally we have created centralized APIs between the SUT and TG node= s > so that all of the methods that the developer would have to use can be > accessed either through these nodes, or through the method inherited from > the base class for test suites. > > I think that when you try to make things extremely simple there is a > common tradeoff; the more simplistic it is, the more restrictive it can > become. We would have to make more assumptions about use cases and how > things would look in these common cases. This generally isn't a super big > problem in our case because if support for something that is needed is > missing, we can add it to the API and with these test cases there are > things that will generally be similar every time. This tradeoff can also = be > offset by creating the functions to make these things simple while still > exposing less restrictive underlying methods so that more niche use cases > can still be fulfilled. Just something to keep in mind. > > > > > > For the configuration side, YAML is already used in DTS. > > > For the test suite logic, do you think we can achieve the same > simplicity with > > > some Python code? > > > > > > We discussed how to progress with this proposal during the CI meeting > last > > > week. > > > We need to check how it could look and what we can improve to reach > this > > > goal. > > > Patrick proposes a meeting this Wednesday at 2pm UTC. > > > > Generally I think this discussion is very useful and having some kind of > ideal mock suite would also be useful to reference. It also can help shin= e > a light on the importance of thinking about what things could be the comm= on > case and abstracting them away from the developer in the framework. > --=20 Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu --000000000000d0bcff061030ed66 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thank you for sharing Gregory. I did not get an opportunit= y to look through the code today, but I did run through the presentation. A= few points I noted:

1. The presentation shows an exampl= e testpmd testcase for creating a flow rule, and then shows a validation st= ep in which standard out is compared against the expected string ("flo= w rule x created") and we can conclude whether we are able to create f= low rules. Are you also sending packets according to the flow rules and val= idating that what is sent/received=C2=A0corresponds to the expected behavio= r of the flow rules? When I look at the old DTS framework, and an example f= low rules testsuite (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.ht= ml) which we want feature parity with, I think that validation for this= testing framework needs to primarily rely on comparing packets sent and pa= ckets received. In any case, there may be some testsuites which can be writ= ten which are small enough in scope that validating by standard out in this= way may be appropriate. I'm not sure but we should keep our options op= en.=C2=A0

2. If the implementation overhead is not= too significant for the configuration step in the DTS execution a "--= fast" option like you use may be a good improvement for the framework.= In your mind, is the main benefit A. reduced execution time, B. reduced us= er setup time (don't have to write full config file) or C. Something el= se?=C2=A0

Thanks for making this available to use = so we can use it as a reference in making DTS better. :)=C2=A0

On Sun, Jan 28, 2024 at 8:44=E2=80=AFAM Gregory Etelson <getelson@nvidia.com> wrote:
Hello,

I've uploaded the code snapshot and functional description here:
From:= Jeremy Spewock <jspewock@iol.unh.edu>
Sent: Tuesday, January 23, 2024 20:26
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gregory Et= elson <getelson= @nvidia.com>; Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech>; = Paul Szczepanek <Paul.Szczepanek@arm.com>; Yoan Picchi <yoan.picchi@foss.arm.com>;= Patrick Robb <pr= obb@iol.unh.edu>; Luca Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; de= v@dpdk.org <dev@dp= dk.org>; nd <nd@a= rm.com>
Subject: Re: DTS testpmd and SCAPY integration
=C2=A0
External email: Us= e caution opening links or attachments



On Tue, Jan 23, 2024 at 3:50=E2=80=AFAM Thomas Monjalon &l= t;thomas@monjalon.= net> wrote:
23/01/2024 04:42, Honnappa Nagarahalli:
> > 08/01/2024 13:10, Luca Vizzarro:
> > > Your proposal sounds rather interesting. Certainly enabling = DTS to
> > > accept YAML-written tests sounds more developer-friendly and= should
> > > enable quicker test-writing. As this is an extra feature tho= ugh =E2=80=93 and
> > > a nice-to-have, it should definitely be discussed in the DTS= meetings
> > > as Honnappa suggested already.
> >
> > I would not classify this idea as "nice-to-have".
> > I would split this proposal in 2 parts:
> >=C2=A0 =C2=A0 =C2=A01/ YAML is an implementation alternative.
> >=C2=A0 =C2=A0 =C2=A02/ Being able to write a test with a small num= ber of lines,
> >=C2=A0 =C2=A0 =C2=A0reusing some commands from existing tools,
> >=C2=A0 =C2=A0 =C2=A0should be our "must-have" common goa= l.
> >
> > Others have mentioned that YAML may not be suitable in complex ca= ses, and
> > that it would be an additional language for test writing.
> > I personnaly think we should focus on a single path which is easy= to read and
> > maintain.
>
> I think we are digressing from the plan we had put forward if we have = to go down this path.
> We should understand what it means by going down the YAML format.
> Also, what would happen if there is another innovation in 3 months?
There is a misunderstanding here.
I suggest to take this proposal as an example of the simplicity to reach. But I agree with you it is more reasonnable to continue with the Python pat= h.

I agree that it would be easier= to develop both the framework and some of the more complex test cases in p= ython and think sticking with Python is a good route.

I think taking this proposal an= d using it as an example of something we could work towards is also a great= outlook because it gives a more structured goal instead of the idea being = "let's just make it as simple as we can."
=C2=A0

> We already have scatter-gather test suite ported to DPDK repo and has = undergone review in the community.
>
> In the last meeting we went through a simple test case. Is it possible= to write the scatter-gather test case in YAML and see how they compare?
After the latest CI mee= ting we thought about writing a simple text case
in Python with some virtual functions which would abstract all the boilerpl= ate code,
so it would have the same level of simplicity as this YAML proposal.

Looking at what we have for DTS= currently, we have put thought into achieving this and, in some categories= , have made things similar to the YAML approach. Like, for example, when it= comes to things like sending, receiving, and verifying a packet, all you have to do= is create a packet and call a method. All of the boilerplate regarding how= to send and receive and how it can be different between types of traffic g= enerators, as well as setting up the ability to forward the packet is completely abstracted away from you. = It's as simple as you send a packet and you get some packets back from = the test suite perspective. The os_udp testing suite that Juraj wrote is a = great example of this and how short the simple case can be. Additionally we have created centralized APIs betw= een the SUT and TG nodes so that all of the methods that the developer woul= d have to use can be accessed either through these nodes, or through the me= thod inherited from the base class for test suites.

I think that when you try to ma= ke things extremely simple there is a common tradeoff; the more simplistic = it is, the more restrictive it can become. We would have to make more assum= ptions about use cases and how things would look in these common cases. This generally = isn't a super big problem in our case because if support for something = that is needed is missing, we can add it to the API and with these test cas= es there are things that will generally be similar every time. This tradeoff can also be offset by creating the fu= nctions to make these things simple while still exposing less restrictive u= nderlying methods so that more niche use cases can still be fulfilled. Just= something to keep in mind.
=C2=A0

> > For the configuration side, YAML is already used in DTS.
> > For the test suite logic, do you think we can achieve the same si= mplicity with
> > some Python code?
> >
> > We discussed how to progress with this proposal during the CI mee= ting last
> > week.
> > We need to check how it could look and what we can improve to rea= ch this
> > goal.
> > Patrick proposes a meeting this Wednesday at 2pm UTC.



Generally I think this discussi= on is very useful and having some kind of ideal mock suite would also be us= eful to reference. It also can help shine a light on the importance of thin= king about what things could be the common case and abstracting them away from the de= veloper in the framework.


--

Patrick Robb

Technic= al Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu


<= p dir=3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;m= argin-bottom:0pt">

--000000000000d0bcff061030ed66--