Re: DTS testpmd and SCAPY integration

[Patrick Robb <probb@iol.unh.edu>]

[-- Attachment #1: Type: text/plain, Size: 7717 bytes --]

Thank you for sharing Gregory. I did not get an opportunity to look through
the code today, but I did run through the presentation. A few points I
noted:

1. The presentation shows an example testpmd testcase for creating a flow
rule, and then shows a validation step in which standard out is compared
against the expected string ("flow rule x created") and we can conclude
whether we are able to create flow rules. Are you also sending packets
according to the flow rules and validating that what is
sent/received corresponds to the expected behavior of the flow rules? When
I look at the old DTS framework, and an example flow rules testsuite (
https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want
feature parity with, I think that validation for this testing framework
needs to primarily rely on comparing packets sent and packets received. In
any case, there may be some testsuites which can be written which are small
enough in scope that validating by standard out in this way may be
appropriate. I'm not sure but we should keep our options open.

2. If the implementation overhead is not too significant for the
configuration step in the DTS execution a "--fast" option like you use may
be a good improvement for the framework. In your mind, is the main benefit
A. reduced execution time, B. reduced user setup time (don't have to write
full config file) or C. Something else?

Thanks for making this available to use so we can use it as a reference in
making DTS better. :)


On Sun, Jan 28, 2024 at 8:44 AM Gregory Etelson <getelson@nvidia.com> wrote:

> Hello,
>
> I've uploaded the code snapshot and functional description here:
>
> https://drive.google.com/drive/folders/1cHrPwx4fUJ6ibUCtHd4kNKsrmmvQvvOj?usp=drive_link
>
> Regards,
> Gregory
> ------------------------------
> *From:* Jeremy Spewock <jspewock@iol.unh.edu>
> *Sent:* Tuesday, January 23, 2024 20:26
> *To:* NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
> *Cc:* Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gregory
> Etelson <getelson@nvidia.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>;
> Paul Szczepanek <Paul.Szczepanek@arm.com>; Yoan Picchi <
> yoan.picchi@foss.arm.com>; Patrick Robb <probb@iol.unh.edu>; Luca
> Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; dev@dpdk.org
> <dev@dpdk.org>; nd <nd@arm.com>
> *Subject:* Re: DTS testpmd and SCAPY integration
>
> *External email: Use caution opening links or attachments*
>
>
> On Tue, Jan 23, 2024 at 3:50 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
> 23/01/2024 04:42, Honnappa Nagarahalli:
> > > 08/01/2024 13:10, Luca Vizzarro:
> > > > Your proposal sounds rather interesting. Certainly enabling DTS to
> > > > accept YAML-written tests sounds more developer-friendly and should
> > > > enable quicker test-writing. As this is an extra feature though – and
> > > > a nice-to-have, it should definitely be discussed in the DTS meetings
> > > > as Honnappa suggested already.
> > >
> > > I would not classify this idea as "nice-to-have".
> > > I would split this proposal in 2 parts:
> > >     1/ YAML is an implementation alternative.
> > >     2/ Being able to write a test with a small number of lines,
> > >     reusing some commands from existing tools,
> > >     should be our "must-have" common goal.
> > >
> > > Others have mentioned that YAML may not be suitable in complex cases,
> and
> > > that it would be an additional language for test writing.
> > > I personnaly think we should focus on a single path which is easy to
> read and
> > > maintain.
> >
> > I think we are digressing from the plan we had put forward if we have to
> go down this path.
> > We should understand what it means by going down the YAML format.
> > Also, what would happen if there is another innovation in 3 months?
>
> There is a misunderstanding here.
> I suggest to take this proposal as an example of the simplicity to reach.
> But I agree with you it is more reasonnable to continue with the Python
> path.
>
>
> I agree that it would be easier to develop both the framework and some of
> the more complex test cases in python and think sticking with Python is a
> good route.
>
> I think taking this proposal and using it as an example of something we
> could work towards is also a great outlook because it gives a more
> structured goal instead of the idea being "let's just make it as simple as
> we can."
>
>
>
> > We already have scatter-gather test suite ported to DPDK repo and has
> undergone review in the community.
> >
> > In the last meeting we went through a simple test case. Is it possible
> to write the scatter-gather test case in YAML and see how they compare?
>
> After the latest CI meeting we thought about writing a simple text case
> in Python with some virtual functions which would abstract all the
> boilerplate code,
> so it would have the same level of simplicity as this YAML proposal.
>
>
> Looking at what we have for DTS currently, we have put thought into
> achieving this and, in some categories, have made things similar to the
> YAML approach. Like, for example, when it comes to things like sending,
> receiving, and verifying a packet, all you have to do is create a packet
> and call a method. All of the boilerplate regarding how to send and receive
> and how it can be different between types of traffic generators, as well as
> setting up the ability to forward the packet is completely abstracted away
> from you. It's as simple as you send a packet and you get some packets back
> from the test suite perspective. The os_udp testing suite that Juraj wrote
> is a great example of this and how short the simple case can be.
> Additionally we have created centralized APIs between the SUT and TG nodes
> so that all of the methods that the developer would have to use can be
> accessed either through these nodes, or through the method inherited from
> the base class for test suites.
>
> I think that when you try to make things extremely simple there is a
> common tradeoff; the more simplistic it is, the more restrictive it can
> become. We would have to make more assumptions about use cases and how
> things would look in these common cases. This generally isn't a super big
> problem in our case because if support for something that is needed is
> missing, we can add it to the API and with these test cases there are
> things that will generally be similar every time. This tradeoff can also be
> offset by creating the functions to make these things simple while still
> exposing less restrictive underlying methods so that more niche use cases
> can still be fulfilled. Just something to keep in mind.
>
>
>
> > > For the configuration side, YAML is already used in DTS.
> > > For the test suite logic, do you think we can achieve the same
> simplicity with
> > > some Python code?
> > >
> > > We discussed how to progress with this proposal during the CI meeting
> last
> > > week.
> > > We need to check how it could look and what we can improve to reach
> this
> > > goal.
> > > Patrick proposes a meeting this Wednesday at 2pm UTC.
>
>
>
> Generally I think this discussion is very useful and having some kind of
> ideal mock suite would also be useful to reference. It also can help shine
> a light on the importance of thinking about what things could be the common
> case and abstracting them away from the developer in the framework.
>


-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 13585 bytes --]