[RFC] Testpmd Maintenance affecting DTS

[RFC] Testpmd Maintenance affecting DTS

[Owen Hilyard]

[-- Attachment #1: Type: text/plain, Size: 2449 bytes --]

Currently, DTS tests DPDK via the example applications (/examples) and the
apps (/app) included with DPDK. The DTS Working group has been trying to
move to primarily using Testpmd, since it is the most complete and only
using one application causes less maintenance work. Since Testpmd provides
a text interface into DPDK, it must be continually updated to expose this
new functionality, sometimes it may lag behind the current feature set of
DPDK. For example, Testpmd currently does not support all of the options
available for rte flow, which limits the testing coverage that can be
provided by DTS for RTE flow. Because DTS is a Python application, it is
not able to directly interface with DPDK. This means that whenever Testpmd
falls behind DPDK, those unexposed parts of DPDK are functionally
untestable as far as DTS is concerned. DTS is the tool used by the CI and
labs to functionally test DPDK in a working configuration, compared to the
unit testing, that may test individual components or parts of the stack.
The DTS improvements WG and the Community Lab have a goal to continue the
expansion of the test functional and performance test coverage of DPDK.
This means that there must be some way for everything that the DPDK
community wants to be tested to be exposed to DTS. I have a few possible
solutions to this problem:


   1.

   Continue to update Testpmd every time new functionality is added to DPDK
   to expose that functionality
   2.

   Use a parser generator or some other method to make adding new options
   to Testpmd much easier, updating this every time new functionality is added
   to testpmd
   3.

   Create a dedicated testing application for DPDK that uses a binding
   generator and Python’s XMLRPC to allow more programmatic access to DPDK on
   the DUT, adding new RPC calls to expose new functionality.


I personally think that option 3 is the best because it would involve a bit
of work up front to make it much easier to expose parts of DPDK to DTS.
This is because after the initial work is done, we would just need to write
a wrapper function in C to expose the functionality we want, and then
expose the wrapper function via RPC.

If you have thoughts or suggestions, we will be discussing this at the DTS
working group meeting.

2:00 PM UTC, Wednesday, February 16
https://armltd.zoom.us/j/97503259680?pwd=VVlmWnlnTXJkVGkwR2JOU3R3b3Vndz09&from=addon

[-- Attachment #2: Type: text/html, Size: 5954 bytes --]

Re: [RFC] Testpmd Maintenance affecting DTS

[Thomas Monjalon]

08/02/2022 22:34, Owen Hilyard:
> Currently, DTS tests DPDK via the example applications (/examples) and the
> apps (/app) included with DPDK. The DTS Working group has been trying to
> move to primarily using Testpmd, since it is the most complete and only
> using one application causes less maintenance work. Since Testpmd provides
> a text interface into DPDK, it must be continually updated to expose this
> new functionality, sometimes it may lag behind the current feature set of
> DPDK. For example, Testpmd currently does not support all of the options
> available for rte flow, which limits the testing coverage that can be
> provided by DTS for RTE flow. Because DTS is a Python application, it is
> not able to directly interface with DPDK. This means that whenever Testpmd
> falls behind DPDK, those unexposed parts of DPDK are functionally
> untestable as far as DTS is concerned. DTS is the tool used by the CI and
> labs to functionally test DPDK in a working configuration, compared to the
> unit testing, that may test individual components or parts of the stack.
> The DTS improvements WG and the Community Lab have a goal to continue the
> expansion of the test functional and performance test coverage of DPDK.
> This means that there must be some way for everything that the DPDK
> community wants to be tested to be exposed to DTS. I have a few possible
> solutions to this problem:
> 
> 
>    1.
> 
>    Continue to update Testpmd every time new functionality is added to DPDK
>    to expose that functionality

Yes, any feature in ethdev should have a testpmd entry.
Which part of rte_flow is not testable with testpmd?

In general, any device API should be testable with the app/ directory.

>    2.
> 
>    Use a parser generator or some other method to make adding new options
>    to Testpmd much easier, updating this every time new functionality is added
>    to testpmd

Please could you elaborate how it would work?

>    3.
> 
>    Create a dedicated testing application for DPDK that uses a binding
>    generator and Python’s XMLRPC to allow more programmatic access to DPDK on
>    the DUT, adding new RPC calls to expose new functionality.

Why do you need RPC when a binding is generated?

> I personally think that option 3 is the best because it would involve a bit
> of work up front to make it much easier to expose parts of DPDK to DTS.
> This is because after the initial work is done, we would just need to write
> a wrapper function in C to expose the functionality we want, and then
> expose the wrapper function via RPC.

I think having a Python binding has 3 drawbacks:
- performance is not good
- it is testing a binding, not the real API
- it is a huge maintenance effort for everybody


> If you have thoughts or suggestions, we will be discussing this at the DTS
> working group meeting.
> 
> 2:00 PM UTC, Wednesday, February 16
> https://armltd.zoom.us/j/97503259680?pwd=VVlmWnlnTXJkVGkwR2JOU3R3b3Vndz09&from=addon

Re: [RFC] Testpmd Maintenance affecting DTS

[Owen Hilyard]

[-- Attachment #1: Type: text/plain, Size: 4268 bytes --]

>
> Yes, any feature in ethdev should have a testpmd entry.
> Which part of rte_flow is not testable with testpmd?


> In general, any device API should be testable with the app/ directory.


 There are currently a lot of fields that appear in the documentation for
rte_flow that are not exposed in testpmd. For example, 5/9 fields
in ARP_ETH_IPV4 are not exposed via testpmd. I could compile a full list if
you want me to.

Please could you elaborate how [a parser generator] would work?


 Instead of the current approach of hand-rolling a parser, something like
Yacc (Yet Another Compiler Compiler) or Bison (GNU Yacc) could be used to
make adding new items much easier. Although they are both designed to aid
in creating compilers, they also work very well as parsers which output a
parsed structure. When I used Bison in the past to write an AWK
implementation, what I did is have it return a struct. In the case of
testpmd, an array of a tagged union might be easier to handle, so you'd get
something like this back:
[
  {
    type: ITEM_ETH,
    inner: {
      eth: {
        dst_is_present: true,
        dst: "00:00:5e:00:53:af",
        src_is_present: true,
        src: "00:00:5e:00:53:af",
        type_is_present: false,
        type: 0
      }
    }
  },
  {
    type: ITEM_IPV4,
    inner: {
      ipv4: {
        dst_is_present: true,
        dst: "127.0.0.1",
        src_is_present: true,
        src: "127.0.0.1",
      }
    }
  }
]

Once you have the tagged union and the inner structs set up, adding a new
field or item should be fairly easy. Also, the <field>_is_present could be
replaced with specified "not present" values (NULL for strings, etc), this
is just emulating the Option/Optional type from higher-level languages more
directly. The process for adding a new field would involve first adding it
to the bison grammar, then adding the field to the output struct.

Why do you need RPC when a binding is generated?


I guess that this is two ideas. First, have C expose an XMLRPC api. How
hard this would be to do would depend greatly on the availability and
quality of libraries, and it would add a fairly large dependency to this
proposed app. The other idea is to generate Python bindings and then use
the RPC server built into Python. This might make adding the RPC layer
easier, and it would mean that we could more easily send Python types to
the DUT. I think that the second one might be easier to work with, if a bit
slower.

performance is not good


All of the performance-sensitive things, like packet generation, would
still be happening entirely in C code. Also, the last time I profiled DTS
it spent most of its time inside of the SSH library, so moving to Python
RPC API may still increase performance, although not as much as a C RPC
API. A good chunk of the rest of the time was spent either in timed
performance tests (nic_single_core_perf) or waiting for testpmd to start up.


> it is testing a binding, not the real API


Python was built to be glue code for C libraries, so it has a very mature
ecosystem in that area. If we use well-tested binding generators, then most
of the underlying behavior would still be DPDK. As is, DTS doesn't really
test DPDK directly, it tests testpmd, and assumes that testpmd interacts
with DPDK correctly. Making a Python RPC API or Python Bindings would mean
moving from a human-oriented API wrapper to a machine-oriented one.


> it is a huge maintenance effort for everybody


I am not advocating hand-writing Python bindings, that would be way too
much work. I am advocating using a binding generator like SWIG to consume
high-level DPDK header files and create python modules from them. Once this
is set up, it should only need occasional maintenance due to build system
changes. It might still require some work to plug new stuff into the RPC
framework, but that would probably be about the same amount of work as
adding the features to testpmd.

I am also open to using languages other than Python, it's just that C
strikes me as a bad choice for building something like this. It's possible
that we could make use of gRPC or something similar that is designed to
perform cross-language RPC easily, in which case directly exposing RPC in C
might be much easier.

Owen Hilyard

[-- Attachment #2: Type: text/html, Size: 5922 bytes --]