> > Though I think we shouldn’t remove existing CLI interface. > I agree, it's a very useful debugging tool for validating environments. I think having two "frontends", the CLI and API, which both consume one "backend" testpmd library would be the easiest way to go about doing that while keeping long-term maintenance low. Conditional compilation (new meson flag or so) is probably good enough for > this case. > One of the changes I made was an on-by-default meson flag to enable C++ compilation. If that flag is on, and all dependencies are present, then the application will be built. Would it be possible to try implement something more realistic with testpmd > itself I would consider it a "phase 2" version of this RFC. The hard part was getting gRPC working inside of Meson, which is why I picked a simple app to port. If this RFC moves forward, I can look at porting the functionality needed for the nic single core performance test ( http://git.dpdk.org/tools/dts/tree/test_plans/nic_single_core_perf_test_plan.rst ). On Thu, Apr 14, 2022 at 8:08 AM Ananyev, Konstantin < konstantin.ananyev@intel.com> wrote: > > Hi everyone, > > First of all thanks Owen for stepping forward with this RFC. > Few thoughts on this subject below. > Konstantin > > > -----Original Message----- > > From: Ananyev, Konstantin > > Sent: Thursday, April 14, 2022 12:59 PM > > To: Ananyev, Konstantin > > Subject: FW: [PATCH v1 0/4] [RFC] Testpmd RPC API > > > > > > > > From: Owen Hilyard > > Sent: Wednesday, April 13, 2022 1:47 PM > > To: Jerin Jacob > > Cc: dpdk-dev ; Honnappa Nagarahalli < > Honnappa.Nagarahalli@arm.com>; Thomas Monjalon > > Subject: Re: [PATCH v1 0/4] [RFC] Testpmd RPC API > > > > If so, I think, gRPC service would be along with existing testpmd > functions, like start_packet_forwarding(). > > > > It was my intention to re-use existing functions. I used the ACL tests > as an example because they are more self-contained then Testpmd, > > which made creating the proof of concept much easier. > > > > Also, We don't need to rewrite the existing testpmd, Instead, RPC > service, we can add in existing app/test-pmd/ > > > > The reason that I split out the services is that there doesn't seem to > be a way to produce multiple binaries without re-writing that section of > > the build system. I wanted to avoid the hard requirement of having a C++ > compiler available in order to be able to use testpmd, since that > > may affect what platforms Testpmd can run on and I want to avoid this > being any kind of breaking change. If we decide to go the route of > > putting it all in a single application, we would need to conditionally > enable the gRPC service at build time. Meson's current lack of support > > for conditionally detecting compilers causes issues here. > > > > I think, DPDK has only one interactive test case which is testpmd, > > > > Could you point me to that test case? Either invocation or source is ok. > I can't see anything that would lead me to assume use of testpmd in > > "meson test --list". To my knowledge, all of the test cases that use > testpmd are in DTS. If there is a test that uses testpmd but is not part of > > DTS, I think it would be a candidate for moving into DTS assuming it's > not a unit test. > > > > How you are planning to do noninteractive test cases? > > > > I'm not planning to make any change to unit testing, you can read more > about how testing is currently conducted > > here: https://www.dpdk.org/blog/2021/07/05/dpdk-testing-approaches/ > > > > If there is a unit test that involves testpmd, there are two > possibilities. > > 1. If we are making a separate application for Testpmd with the gRPC > api, then nothing changes except for possibly changing where some > > of the testpmd source lives in order to enable code reuse between the > two applications. > > 2. If gRPC is being added to Testpmd, then the unit test should still > function as it previously did if I do any necessary refactoring as > correctly. > > > > I think, key should be leveraging existing test cases as much as > possible and make easy for developers to add new test cases. > > > > That is part of the reason why I want to be able to do this. Adding a > new test in DTS is very easy if the functionality needed already exists in > > Testpmd. If the functionality does not exist, then adding the test > becomes difficult, due to the required modifications to the Testpmd lexer > > and parser to accommodate the new command. My plan is to leave unit > testing in C, but help make it easier to expose C functions to Python > > for integration testing. This gives us the best of both worlds in terms > of access to DPDK and the ability to use a high-level language to write > > the tests. > > > > On Tue, Apr 12, 2022 at 2:07 AM Jerin Jacob jerinjacobk@gmail.com> wrote: > > On Mon, Apr 11, 2022 at 11:19 PM Owen Hilyard ohilyard@iol.unh.edu> wrote: > > >> > > >> scheme is probably over-engineered > > > > > > > > > I tried my hardest to keep this as simple as possible. The > requirements imposed by DTS being a distributed system in Python restricted > > what I could do a lot. Needing to be compatible with DPDK's license also > got rid of a lot of options. Binding generators are made for simple > > projects, and DPDK is not a simple project. There were some other > options related to choice in the RPC framework, but very few RPC > > protocols seem to work well with C and be usable from Python, which is > why I ended up using C++ with gRPC. Most of the code in > > api_impl.cc is taken from /app/test-acl/main.c, and the new part is > mostly the C++ class at the bottom. Overall, this proposal comes out to > > ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and 100 > lines of Meson. gRPC may be able to do a lot more than I need it to > > for the proof of concept, but many of the features that are not used, > like bi-directional streaming, become very useful in writing more > > complicated tests. Overall, this solution is probably more capable than > we need it to be, but I think that those extra capabilities don't come > > at a very large cost. > > > > > > Now it is clear, I was carried away with the POC test application and > > I was not knowing existing DTS tests are based on python > > > > Is below a fair summary? > > > > 1) DPDK has interactive test cases and no interactive test cases. > > > > For The interactive test case like testpmd, I agree that we can enable > > RPC service via gRPC server in C++ as and client in Python, and > > something along the lines of exposing the existing test-pmd command > > line function as service > > to avoid command line parsing and reuse the existing python test suite. > > > > If so, I think, gRPC service would be along with existing testpmd > > functions, like start_packet_forwarding(). Also, We don't need to > > rewrite the existing testpmd, > > Instead, RPC service, we can add in existing app/test-pmd/ and hook to > > existing core testpmd functions to bypass the command-line parsing in > > C and control from python client as needed as service. > > > > Also, I agree that pulling in gRPC C++ server boilerplate and hooking > > to C functions is a good idea as it is the best C-based RPC scheme > > available today. > > > > 2)I think, DPDK has only one interactive test case which is testpmd, > > Remaining test cases are non-interactive, non-interactive test cases > > can simply run over ssh with passwordless login. Right? > > Do we need gRPC for that? Will the following scheme suffice? If not, > > How you are planning to do noninteractive test cases? > > i.e > > a)Copy test to target > > b) result=`ssh username@IP /path/to/testapp/in/target` > > > > I think, key should be leveraging existing test cases as much as > > possible and make easy for developers to add new test cases. > > > > > > >> > > >> Now that, Test code is also part of DPDK. > > > > > > > > > DTS is pure python. I tried to use FFI to call directly into DPDK from > Python and then use xmlrpc from the python standard library. As > > mentioned in the writeup, I couldn't find a binding generator that would > properly handle DPDK's allocators, which made it so that anything > > passed to DPDK using python was allocated using the system malloc. I > don't think it is wise to attempt to programmatically re-write the > > generated code to allow for custom allocators. The original reason for > needing to have DTS and DPDK in the same repository was so that > > tests could be committed and run alongside the feature patch. > > > > > >> Interactive - Testpmd one, I believe, Feeding stdin programmatically > would suffice to test all the combinations. > > > > > > > > > One of the issues this is trying to address is that human-readable > strings are a poor way to pass complex information between two > > programs. DTS is a distributed system, and it can have up to 3 physical > servers involved in any given test. This means that it's not stdin via a > > pipe, it's an entire SSH session. This adds a noticeable amount of > overhead when trying to send and verify the result of sending 1,000+ > > packets, since the lack of structured output means each packet must be > checked before the next can be sent. This might be solvable by > > adding a structured output mode to testpmd, but that would involve > committing to writing output twice for every function in testpmd > > forever. > > > > > >> We need to add all test cases in this model and we need to maintain > two sets of programs.(Traditional tests and gRPC model-based > > tests). > > > > > > > > > Assuming by traditional tests you mean the unit tests run by Meson, I > would argue that we are already maintaining 2 kinds of tests. The > > unit tests, and the python-based DTS tests. My intention is to create a > thin wrapper around DPDK that would be exposed via gRPC, like you > > see here, and use that as midware. Then, we would have two front-ends. > Testpmd, which takes text and then calls midware as it does now, > > and the gRPC frontend, which parses messages from the RPC server and > runs the midware. This would enable testpmd to still be used to > > sanity check a DPDK installation, but we would not need to continually > expand Testpmd. The primary issue is that, right now, anything not > > included in Testpmd is not really testable by DTS. This includes > portions of the RTE Flow API, which was part of my reason for proposing > this. > > The RTE Flow API would, in my estimation, if fully implemented into > Testpmd, probably add at least another 10,000 lines of code. As > > mentioned in my proposal, Testpmd already does more parsing and lexing > than it does interaction with DPDK by line count. Also, since I am > > proposing making this a separate application, we would be able to > gradually migrate the tests inside of DTS. This would have no effect on > > anything except for Testpmd, the new application and the addition of a > flag to toggle the use of a C++ compiler. > > > > > > I'm not sure exactly what you mean by gRPC model-based tests. gRPC > uses classes to model services, but for this usecase we are > > essentially using it to transfer function arguments across the internet > and then pass the return value back. Any RPC framework would > > function similarly if I ignored the restrictions of which languages to > use, and the choice is not important to how tests are conducted. Put > > another way, how you write a test for DTS will not change much if you > are using this or testpmd, it's just how you transfer data and get it > > back that I want to change. > > - In general I think it is a good idea to adding gRPC binding to testpmd > to expose/improve testing automation. > Though I think we shouldn’t remove existing CLI interface. > Ideally I’d like to have both – CLI and gRPC for all commands. > Don’t know how realistic is that, but at least for major commands - > port/queue configure, start/stop, etc. > - Conditional compilation (new meson flag or so) is probably good enough > for this case. > - About RFC itself - I understand that you choose testacl for simplicity, > but in fact, it is a standalone application > that has not much common with testpmd itself and the problems that you > mentioned: > interactive commands, parameter and results parsing, etc. > Would it be possible to try implement something more realistic with > testpmd itself, > like simple test-pmd port/queue configure, start, result collection, > etc.? > To get a better idea how it is going to work and how complicated it > would be. > > >