Thank you for sharing Gregory. I did not get an opportunity to look through the code today, but I did run through the presentation. A few points I noted:

1. The presentation shows an example testpmd testcase for creating a flow rule, and then shows a validation step in which standard out is compared against the expected string ("flow rule x created") and we can conclude whether we are able to create flow rules. Are you also sending packets according to the flow rules and validating that what is sent/received corresponds to the expected behavior of the flow rules? When I look at the old DTS framework, and an example flow rules testsuite (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want feature parity with, I think that validation for this testing framework needs to primarily rely on comparing packets sent and packets received. In any case, there may be some testsuites which can be written which are small enough in scope that validating by standard out in this way may be appropriate. I'm not sure but we should keep our options open.

2. If the implementation overhead is not too significant for the configuration step in the DTS execution a "--fast" option like you use may be a good improvement for the framework. In your mind, is the main benefit A. reduced execution time, B. reduced user setup time (don't have to write full config file) or C. Something else?

Thanks for making this available to use so we can use it as a reference in making DTS better. :)

On Sun, Jan 28, 2024 at 8:44 AM Gregory Etelson <getelson@nvidia.com> wrote:

Hello,

I've uploaded the code snapshot and functional description here:

https://drive.google.com/drive/folders/1cHrPwx4fUJ6ibUCtHd4kNKsrmmvQvvOj?usp=drive_link

Regards,
Gregory

From: Jeremy Spewock <jspewock@iol.unh.edu>
Sent: Tuesday, January 23, 2024 20:26
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gregory Etelson <getelson@nvidia.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>; Paul Szczepanek <Paul.Szczepanek@arm.com>; Yoan Picchi <yoan.picchi@foss.arm.com>; Patrick Robb <probb@iol.unh.edu>; Luca Vizzarro <Luca.Vizzarro@arm.com>; ci@dpdk.org <ci@dpdk.org>; dev@dpdk.org <dev@dpdk.org>; nd <nd@arm.com>
Subject: Re: DTS testpmd and SCAPY integration

External email: Use caution opening links or attachments

On Tue, Jan 23, 2024 at 3:50 AM Thomas Monjalon <thomas@monjalon.net> wrote:

23/01/2024 04:42, Honnappa Nagarahalli:
> > 08/01/2024 13:10, Luca Vizzarro:
> > > Your proposal sounds rather interesting. Certainly enabling DTS to
> > > accept YAML-written tests sounds more developer-friendly and should
> > > enable quicker test-writing. As this is an extra feature though – and
> > > a nice-to-have, it should definitely be discussed in the DTS meetings
> > > as Honnappa suggested already.
> >
> > I would not classify this idea as "nice-to-have".
> > I would split this proposal in 2 parts:
> > 1/ YAML is an implementation alternative.
> > 2/ Being able to write a test with a small number of lines,
> > reusing some commands from existing tools,
> > should be our "must-have" common goal.
> >
> > Others have mentioned that YAML may not be suitable in complex cases, and
> > that it would be an additional language for test writing.
> > I personnaly think we should focus on a single path which is easy to read and
> > maintain.
>
> I think we are digressing from the plan we had put forward if we have to go down this path.
> We should understand what it means by going down the YAML format.
> Also, what would happen if there is another innovation in 3 months?

There is a misunderstanding here.
I suggest to take this proposal as an example of the simplicity to reach.
But I agree with you it is more reasonnable to continue with the Python path.

I agree that it would be easier to develop both the framework and some of the more complex test cases in python and think sticking with Python is a good route.

I think taking this proposal and using it as an example of something we could work towards is also a great outlook because it gives a more structured goal instead of the idea being "let's just make it as simple as we can."

> We already have scatter-gather test suite ported to DPDK repo and has undergone review in the community.
>
> In the last meeting we went through a simple test case. Is it possible to write the scatter-gather test case in YAML and see how they compare?

After the latest CI meeting we thought about writing a simple text case
in Python with some virtual functions which would abstract all the boilerplate code,
so it would have the same level of simplicity as this YAML proposal.

Looking at what we have for DTS currently, we have put thought into achieving this and, in some categories, have made things similar to the YAML approach. Like, for example, when it comes to things like sending, receiving, and verifying a packet, all you have to do is create a packet and call a method. All of the boilerplate regarding how to send and receive and how it can be different between types of traffic generators, as well as setting up the ability to forward the packet is completely abstracted away from you. It's as simple as you send a packet and you get some packets back from the test suite perspective. The os_udp testing suite that Juraj wrote is a great example of this and how short the simple case can be. Additionally we have created centralized APIs between the SUT and TG nodes so that all of the methods that the developer would have to use can be accessed either through these nodes, or through the method inherited from the base class for test suites.

I think that when you try to make things extremely simple there is a common tradeoff; the more simplistic it is, the more restrictive it can become. We would have to make more assumptions about use cases and how things would look in these common cases. This generally isn't a super big problem in our case because if support for something that is needed is missing, we can add it to the API and with these test cases there are things that will generally be similar every time. This tradeoff can also be offset by creating the functions to make these things simple while still exposing less restrictive underlying methods so that more niche use cases can still be fulfilled. Just something to keep in mind.

> > For the configuration side, YAML is already used in DTS.
> > For the test suite logic, do you think we can achieve the same simplicity with
> > some Python code?
> >
> > We discussed how to progress with this proposal during the CI meeting last
> > week.
> > We need to check how it could look and what we can improve to reach this
> > goal.
> > Patrick proposes a meeting this Wednesday at 2pm UTC.

Generally I think this discussion is very useful and having some kind of ideal mock suite would also be useful to reference. It also can help shine a light on the importance of thinking about what things could be the common case and abstracting them away from the developer in the framework.