Re: DTS testpmd and SCAPY integration

[Jeremy Spewock <jspewock@iol.unh.edu>]

[-- Attachment #1: Type: text/plain, Size: 4953 bytes --]

On Tue, Jan 23, 2024 at 3:50 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> 23/01/2024 04:42, Honnappa Nagarahalli:
> > > 08/01/2024 13:10, Luca Vizzarro:
> > > > Your proposal sounds rather interesting. Certainly enabling DTS to
> > > > accept YAML-written tests sounds more developer-friendly and should
> > > > enable quicker test-writing. As this is an extra feature though – and
> > > > a nice-to-have, it should definitely be discussed in the DTS meetings
> > > > as Honnappa suggested already.
> > >
> > > I would not classify this idea as "nice-to-have".
> > > I would split this proposal in 2 parts:
> > >     1/ YAML is an implementation alternative.
> > >     2/ Being able to write a test with a small number of lines,
> > >     reusing some commands from existing tools,
> > >     should be our "must-have" common goal.
> > >
> > > Others have mentioned that YAML may not be suitable in complex cases,
> and
> > > that it would be an additional language for test writing.
> > > I personnaly think we should focus on a single path which is easy to
> read and
> > > maintain.
> >
> > I think we are digressing from the plan we had put forward if we have to
> go down this path.
> > We should understand what it means by going down the YAML format.
> > Also, what would happen if there is another innovation in 3 months?
>
> There is a misunderstanding here.
> I suggest to take this proposal as an example of the simplicity to reach.
> But I agree with you it is more reasonnable to continue with the Python
> path.
>

I agree that it would be easier to develop both the framework and some of
the more complex test cases in python and think sticking with Python is a
good route.

I think taking this proposal and using it as an example of something we
could work towards is also a great outlook because it gives a more
structured goal instead of the idea being "let's just make it as simple as
we can."


>
> > We already have scatter-gather test suite ported to DPDK repo and has
> undergone review in the community.
> >
> > In the last meeting we went through a simple test case. Is it possible
> to write the scatter-gather test case in YAML and see how they compare?
>
> After the latest CI meeting we thought about writing a simple text case
> in Python with some virtual functions which would abstract all the
> boilerplate code,
> so it would have the same level of simplicity as this YAML proposal.
>

Looking at what we have for DTS currently, we have put thought into
achieving this and, in some categories, have made things similar to the
YAML approach. Like, for example, when it comes to things like sending,
receiving, and verifying a packet, all you have to do is create a packet
and call a method. All of the boilerplate regarding how to send and receive
and how it can be different between types of traffic generators, as well as
setting up the ability to forward the packet is completely abstracted away
from you. It's as simple as you send a packet and you get some packets back
from the test suite perspective. The os_udp testing suite that Juraj wrote
is a great example of this and how short the simple case can be.
Additionally we have created centralized APIs between the SUT and TG nodes
so that all of the methods that the developer would have to use can be
accessed either through these nodes, or through the method inherited from
the base class for test suites.

I think that when you try to make things extremely simple there is a common
tradeoff; the more simplistic it is, the more restrictive it can become. We
would have to make more assumptions about use cases and how things would
look in these common cases. This generally isn't a super big problem in our
case because if support for something that is needed is missing, we can add
it to the API and with these test cases there are things that will
generally be similar every time. This tradeoff can also be offset by
creating the functions to make these things simple while still exposing
less restrictive underlying methods so that more niche use cases can still
be fulfilled. Just something to keep in mind.


>
> > > For the configuration side, YAML is already used in DTS.
> > > For the test suite logic, do you think we can achieve the same
> simplicity with
> > > some Python code?
> > >
> > > We discussed how to progress with this proposal during the CI meeting
> last
> > > week.
> > > We need to check how it could look and what we can improve to reach
> this
> > > goal.
> > > Patrick proposes a meeting this Wednesday at 2pm UTC.
>
>
>
> Generally I think this discussion is very useful and having some kind of
ideal mock suite would also be useful to reference. It also can help shine
a light on the importance of thinking about what things could be the common
case and abstracting them away from the developer in the framework.

[-- Attachment #2: Type: text/html, Size: 6765 bytes --]