From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E8191439AB; Tue, 23 Jan 2024 19:27:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E094E427DC; Tue, 23 Jan 2024 19:27:02 +0100 (CET) Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by mails.dpdk.org (Postfix) with ESMTP id 25F7D40DFB for ; Tue, 23 Jan 2024 19:27:00 +0100 (CET) Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6dd82bfa998so683208b3a.1 for ; Tue, 23 Jan 2024 10:27:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1706034419; x=1706639219; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=wn5t1jF8z8qHMCi1rkppLJLcMzxRxnFyfS3kD6JsAyE=; b=KJWgrzSYkdRtcAhCXOvqJHP1Cji1a9c44vYagUYDPjRSg+0Pk2/H37YSrkjLbEQB2J EyWCUvp3ApMgqBD7+PL4ZIW0XY82OFpi4rFo171qKxGfm9VyUcoJj9O57+707Uwp2nUY PfPMChjA9fiE8g82lFnMVrIP2rpKLMmKNjCuQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706034419; x=1706639219; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wn5t1jF8z8qHMCi1rkppLJLcMzxRxnFyfS3kD6JsAyE=; b=CeebJApWiN5gYtWXtzUYKw+UJU8QvxMopSs1q6JQXM5EJdAQZu0QNT7fXg9Pylcz/4 yzxye/Xou3pTM5iC0ZPPi1vydTO/7/CsZRTlAa+Lg/dKQioGKw/2/HO55WClm9BwVfCE yakurHo7cTihmukPt8TGGHRoCWpCqSS0vJiKxWma7xBJSpdDYa9covnyqe1MMwXz/dfs OfNlzzLUsNW1zFWOcN+ZiCrJ6zEsjAhJXKsMvFjMcgu9JUTjDn2xGm0kazpDgKRUifnG BghqUHRMxiy1YDBhX5NLFhwxadrGucNDFbkGGARtOj43E56Zv5ObQqdL3/8HVrxg0SWY eFEg== X-Gm-Message-State: AOJu0YwQmhCwy1X3cCYR6gbX+f5DLlDvPOjmCd8m3qCeSh3k8U3fSQzC eda8jgPVRby77mdIF918LXpxBQhT8mUMmFr+Spw73vKTkmKL31+tkD8cW+lo9j/5ciBO56TMcrL /Ugccz2umhlIaWLvF2obtNDfKdclspKARhP9PcQ== X-Google-Smtp-Source: AGHT+IGn2wTNLzHOJt3SBPtDnuUp6sx8qRfqdnVlpXYjk51pob2bV6YvIOFwPOjUQ1L/RVIcEjYpk3g49v2WuUtkSrk= X-Received: by 2002:a05:6a00:2301:b0:6db:e463:96d0 with SMTP id h1-20020a056a00230100b006dbe46396d0mr3357778pfh.55.1706034419175; Tue, 23 Jan 2024 10:26:59 -0800 (PST) MIME-Version: 1.0 References: <2a287ee7-cda4-f2ab-a4e6-a47021f8573f@nvidia.com> <2127794.bB369e8A3T@thomas> <6608561.G0QQBjFxQf@thomas> In-Reply-To: <6608561.G0QQBjFxQf@thomas> From: Jeremy Spewock Date: Tue, 23 Jan 2024 13:26:48 -0500 Message-ID: Subject: Re: DTS testpmd and SCAPY integration To: Thomas Monjalon Cc: Honnappa Nagarahalli , "Etelson, Gregory" , =?UTF-8?Q?Juraj_Linke=C5=A1?= , Paul Szczepanek , Yoan Picchi , Patrick Robb , Luca Vizzarro , "ci@dpdk.org" , "dev@dpdk.org" , nd Content-Type: multipart/alternative; boundary="00000000000087aa2f060fa117a2" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --00000000000087aa2f060fa117a2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jan 23, 2024 at 3:50=E2=80=AFAM Thomas Monjalon wrote: > 23/01/2024 04:42, Honnappa Nagarahalli: > > > 08/01/2024 13:10, Luca Vizzarro: > > > > Your proposal sounds rather interesting. Certainly enabling DTS to > > > > accept YAML-written tests sounds more developer-friendly and should > > > > enable quicker test-writing. As this is an extra feature though =E2= =80=93 and > > > > a nice-to-have, it should definitely be discussed in the DTS meetin= gs > > > > as Honnappa suggested already. > > > > > > I would not classify this idea as "nice-to-have". > > > I would split this proposal in 2 parts: > > > 1/ YAML is an implementation alternative. > > > 2/ Being able to write a test with a small number of lines, > > > reusing some commands from existing tools, > > > should be our "must-have" common goal. > > > > > > Others have mentioned that YAML may not be suitable in complex cases, > and > > > that it would be an additional language for test writing. > > > I personnaly think we should focus on a single path which is easy to > read and > > > maintain. > > > > I think we are digressing from the plan we had put forward if we have t= o > go down this path. > > We should understand what it means by going down the YAML format. > > Also, what would happen if there is another innovation in 3 months? > > There is a misunderstanding here. > I suggest to take this proposal as an example of the simplicity to reach. > But I agree with you it is more reasonnable to continue with the Python > path. > I agree that it would be easier to develop both the framework and some of the more complex test cases in python and think sticking with Python is a good route. I think taking this proposal and using it as an example of something we could work towards is also a great outlook because it gives a more structured goal instead of the idea being "let's just make it as simple as we can." > > > We already have scatter-gather test suite ported to DPDK repo and has > undergone review in the community. > > > > In the last meeting we went through a simple test case. Is it possible > to write the scatter-gather test case in YAML and see how they compare? > > After the latest CI meeting we thought about writing a simple text case > in Python with some virtual functions which would abstract all the > boilerplate code, > so it would have the same level of simplicity as this YAML proposal. > Looking at what we have for DTS currently, we have put thought into achieving this and, in some categories, have made things similar to the YAML approach. Like, for example, when it comes to things like sending, receiving, and verifying a packet, all you have to do is create a packet and call a method. All of the boilerplate regarding how to send and receive and how it can be different between types of traffic generators, as well as setting up the ability to forward the packet is completely abstracted away from you. It's as simple as you send a packet and you get some packets back from the test suite perspective. The os_udp testing suite that Juraj wrote is a great example of this and how short the simple case can be. Additionally we have created centralized APIs between the SUT and TG nodes so that all of the methods that the developer would have to use can be accessed either through these nodes, or through the method inherited from the base class for test suites. I think that when you try to make things extremely simple there is a common tradeoff; the more simplistic it is, the more restrictive it can become. We would have to make more assumptions about use cases and how things would look in these common cases. This generally isn't a super big problem in our case because if support for something that is needed is missing, we can add it to the API and with these test cases there are things that will generally be similar every time. This tradeoff can also be offset by creating the functions to make these things simple while still exposing less restrictive underlying methods so that more niche use cases can still be fulfilled. Just something to keep in mind. > > > > For the configuration side, YAML is already used in DTS. > > > For the test suite logic, do you think we can achieve the same > simplicity with > > > some Python code? > > > > > > We discussed how to progress with this proposal during the CI meeting > last > > > week. > > > We need to check how it could look and what we can improve to reach > this > > > goal. > > > Patrick proposes a meeting this Wednesday at 2pm UTC. > > > > Generally I think this discussion is very useful and having some kind of ideal mock suite would also be useful to reference. It also can help shine a light on the importance of thinking about what things could be the common case and abstracting them away from the developer in the framework. --00000000000087aa2f060fa117a2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Jan 23, 2024 at 3:50=E2=80=AFAM = Thomas Monjalon <thomas@monjalon.= net> wrote:
23/01/2024 04:42, Honnappa Nagarahalli:
> > 08/01/2024 13:10, Luca Vizzarro:
> > > Your proposal sounds rather interesting. Certainly enabling = DTS to
> > > accept YAML-written tests sounds more developer-friendly and= should
> > > enable quicker test-writing. As this is an extra feature tho= ugh =E2=80=93 and
> > > a nice-to-have, it should definitely be discussed in the DTS= meetings
> > > as Honnappa suggested already.
> >
> > I would not classify this idea as "nice-to-have".
> > I would split this proposal in 2 parts:
> >=C2=A0 =C2=A0 =C2=A01/ YAML is an implementation alternative.
> >=C2=A0 =C2=A0 =C2=A02/ Being able to write a test with a small num= ber of lines,
> >=C2=A0 =C2=A0 =C2=A0reusing some commands from existing tools,
> >=C2=A0 =C2=A0 =C2=A0should be our "must-have" common goa= l.
> >
> > Others have mentioned that YAML may not be suitable in complex ca= ses, and
> > that it would be an additional language for test writing.
> > I personnaly think we should focus on a single path which is easy= to read and
> > maintain.
>
> I think we are digressing from the plan we had put forward if we have = to go down this path.
> We should understand what it means by going down the YAML format.
> Also, what would happen if there is another innovation in 3 months?
There is a misunderstanding here.
I suggest to take this proposal as an example of the simplicity to reach. But I agree with you it is more reasonnable to continue with the Python pat= h.

I agree that it would be easier to develop = both the framework and some of the more complex test cases in python and th= ink sticking with Python is a good route.

I thi= nk taking this proposal and using it as an example of something we could wo= rk towards is also a great outlook because it gives a more structured goal = instead of the idea being "let's just make it as simple as we can.= "
=C2=A0

> We already have scatter-gather test suite ported to DPDK repo and has = undergone review in the community.
>
> In the last meeting we went through a simple test case. Is it possible= to write the scatter-gather test case in YAML and see how they compare?
After the latest CI meeting we thought about writing a simple text case in Python with some virtual functions which would abstract all the boilerpl= ate code,
so it would have the same level of simplicity as this YAML proposal.

Looking at what we have for DTS currently, we have p= ut thought into achieving this and, in some categories, have made things si= milar to the YAML approach. Like, for example, when it comes to things like= sending, receiving, and verifying a packet, all you have to do is create a= packet and call a method. All of the boilerplate regarding how to send and= receive and how it can be different between types of traffic generators, a= s well as setting up the ability to forward the packet is completely abstra= cted away from you. It's as simple as you send a packet and you get som= e packets back from the test suite perspective. The os_udp testing suite th= at Juraj wrote is a great example of this and how short the simple case can= be. Additionally we have created centralized APIs between the SUT and TG n= odes so that all of the methods that the developer would have to use can be= accessed either through these nodes, or through the method inherited from = the base class for test suites.

I think that when you try to make thi= ngs extremely simple there is a common tradeoff; the more simplistic it is,= the more restrictive it can become. We would have to make more assumptions= about use cases and how things would look in these common cases. This gene= rally isn't a super big problem in our case because if support for some= thing that is needed is missing, we can add it to the API and with these te= st cases there are things that will generally be similar every time. This t= radeoff can also be offset by creating the functions to make these things s= imple while still exposing less restrictive underlying methods so that more= niche use cases can still be fulfilled. Just something to keep in mind.
=C2=A0

> > For the configuration side, YAML is already used in DTS.
> > For the test suite logic, do you think we can achieve the same si= mplicity with
> > some Python code?
> >
> > We discussed how to progress with this proposal during the CI mee= ting last
> > week.
> > We need to check how it could look and what we can improve to rea= ch this
> > goal.
> > Patrick proposes a meeting this Wednesday at 2pm UTC.



Generally I think this discussion is very useful and having some = kind of ideal mock suite would also be useful to reference. It also can hel= p shine a light on the importance of thinking about what things could be th= e common case and abstracting them away from the developer in the framework= .
--00000000000087aa2f060fa117a2--