Re: [PATCH v1 0/4] [RFC] Testpmd RPC API

[Jerin Jacob <jerinjacobk@gmail.com>]

On Fri, Apr 8, 2022 at 3:17 AM <ohilyard@iol.unh.edu> wrote:
>
> From: Owen Hilyard <ohilyard@iol.unh.edu>
>
>     Currently, DTS uses Testpmd for most of its testing. This has been successful in reducing the need to create more test apps, but it has a few drawbacks. First, if some part of DPDK is not exposed via Testpmd or one of the example applications, for the purposes of DTS it is not testable. This is a situation I’d like to avoid. However, adding new functionality to Testpmd is labor-intensive. Testpmd currently uses a hand-written LL(1) parser (https://en.wikipedia.org/wiki/LL_parser) to parse command line options. This makes adding new functionality difficult since the parser is stored as a series of several thousand line long lookup tables. To look at it another way, 64% of the 52238 lines in Testpmd are related to command line input in some way. The command line interface of testpmd also presents several challenges for the underlying implementation, since it requires that everything a user might want to reference is identified via something that is reasonable to ask a user to type. As of right now, this is handled via either strings or integers. This can be handled by creating a global registry for objects, but it is still extra work that I think can be avoided. In addition, this leads to more places where things can go wrong.
>
> This is what DTS running a single command in testpmd looks like right now:
> https://drive.google.com/file/d/1hvTcjfVdh8-I3CUNoq6bx82EuNQSK6qW/view?usp=sharing
>
>     This approach has a number of disadvantages. First, it requires assembling all commands as strings inside of the test suite and sending them through a full round trip of SSH. This means that any non-trivial command, such as creating an RTE flow, will involve a lot of string templating. This normally wouldn’t be a big issue, except that some of the test suites are designed to hundreds of commands over the course of a test, paying the cost of an SSH round trip for each. Once Testpmd has the commands, it will then call the appropriate functions inside of DPDK, and then print out all of the state to standard out. All of this is sent back to DTS, where the author of the test case then needs to handle all possible outputs of Trex, often by either declaring the presence of a single word or short phrase in the output as meaning success or failure. In my opinion, this is something that is perfectly fine for humans to interact with, but it causes a lot of issues with automation due to its inherent inflexibility and the less-than-ideal methods of information transfer. This is why I am proposing the creation of an automation-oriented pmd, with a focus on exposing as much.
>
> https://drive.google.com/file/d/1wj4-RnFPVERCzM8b68VJswAOEI9cg-X8/view?usp=sharing
>
>         That diagram is a high-level overview of the design, which explicitly excludes implementation details. However, it already has some benefits. First, making DPDK do something is a normal method call, instead of needing to format things into a string. This provides a much better interface for people working in both DTS and DPDK. Second, the ability to return structured data means that there won’t be parsers on both sides of communication anymore. Structured data also allows much more verbosity, since it is no longer an interface designed for humans. If a test case author needs to return the bytes of every received packet back to DTS for comparison with the expected value, they can. If you need to return a pointer for DTS to use later, that becomes reasonable. Simply moving to shuffling structured data around and using RPC already provides a lot of benefits.
>         The next obvious question would be what to use for the implementation. The initial attempt was made using Python on both sides and the standard library xmlrpc module. The RPC aspect of this approach worked very well, with the ability to send arbitrary python objects back and forth between DTS and app. However, having Python interacting with DPDK has a few issues. First, DPDK is generally very multi-threaded and the most common implementation of Python, CPython, does not have concurrency. It has something known as the global interpretr lock, which is a global mutex. This makes it very difficult to interact with blocking, multi-threaded code. The other issue is that I was not able to find a binding generator that I feel would be sufficient for DPDK. Many generators assumed sizeof(int) == 4 or had other portability issues such as assuming GCC or Clang as a C compiler. Others focused on some subset of C, meaning they would throw errors on alignment annotations.
>     Given this, I decided to look for cross-language RPC libraries. Although libraries exist for performing xmlrpc in C, they generally appeared quite difficult to use and required a lot of manual work. The next best option was gRPC. gRPC allows using a simple language, protobuf, with a language extension for rpc. It provides code generation to make it easy to use multiple languages together, since it was developed to make polyglot microservice interaction easier. The only drawback is that it considers C++ good enough for C support. In this case, I was able to easily integrate DPDK with C++, so that isn’t much of a concern. I used C++17 in the attached patches, but the minimum requirements are C++11. If there is concern about modern C++ causing too much mental overhead, a “C with classes” subset of C++ could easily be used. I also added an on-by-default option to use a C++ compiler, allowing anyone who does not have a C++ compiler available to them to turn off everything that uses C++. This disables the application written for this RFC.
>     One of the major benefits of gRPC is the asynchronous API. This allows streaming data on both sides of an RPC call. This allows streaming logs back to DTS, streaming large amounts of data from low-memory systems back to DTS for processing, and would allow DTS to easily feed DPDK data, ending the test quickly on a failure. Currently, due to the overhead of sending data to Testpmd, it is common to just send all of the commands over and run everything since that will be much faster when the test passes, but it can cost a lot of time in the event of a failure. There are also optional security features for requiring authentication before allowing code execution. I think that a discussion on whether using them for DTS is necessary is warranted, although I personally think that it’s not worth the effort given the type of environment this sample application is expected to run in.
>     For this RFC, I ported test-acl because it was mostly self-contained and was something I could run on my laptop. It should be fairly easy to see how you would expand this proof of concept to cover more of DPDK, and I think that most of the functions currently used in testpmd could be ported over to this approach, saving a lot of development time. However, I would like to see some more interest before I take on a task like that. This will require a lot of work on the DTS side to implement, but it will make it much easier to add new features to DTS.


Thanks, Owen for POC.

In my view, May using this scheme is probably over-engineered. The
reason for thinking so is
-Now that, Test code is also part of DPDK, Exposing as services may
not be required.
-Now in DPDK, we have two types of existing test cases to verify the API
-- Noninteractive  - These test cases, can simply run over ssh with
bash invocation and return the test from a remote PC,
-- Interactive - Testpmd one, I believe, Feeding stdin
programmatically would suffice to test all the combinations.
-We need to add all test cases in this model and we need to maintain
two sets of programs.(Traditional tests and gRPC model-based tests).

Just my 2c.


>
> Owen Hilyard (4):
>   app/test-pmd-api: Add C++ Compiler
>   app/test-pmd-api: Add POC with gRPC deps
>   app/test-pmd-api: Add protobuf file
>   app/test-pmd-api: Implementation files for the API
>
>  app/meson.build              |   17 +
>  app/test-pmd-api/api.proto   |   12 +
>  app/test-pmd-api/api_impl.cc | 1160 ++++++++++++++++++++++++++++++++++
>  app/test-pmd-api/api_impl.h  |   10 +
>  app/test-pmd-api/main.c      |   11 +
>  app/test-pmd-api/meson.build |   96 +++
>  meson.build                  |    3 +
>  meson_options.txt            |    2 +
>  8 files changed, 1311 insertions(+)
>  create mode 100644 app/test-pmd-api/api.proto
>  create mode 100644 app/test-pmd-api/api_impl.cc
>  create mode 100644 app/test-pmd-api/api_impl.h
>  create mode 100644 app/test-pmd-api/main.c
>  create mode 100644 app/test-pmd-api/meson.build
>
> --
> 2.30.2
>