From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 119E0A050B; Wed, 13 Apr 2022 14:47:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AC16B40694; Wed, 13 Apr 2022 14:47:45 +0200 (CEST) Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by mails.dpdk.org (Postfix) with ESMTP id 5F6394068B for ; Wed, 13 Apr 2022 14:47:44 +0200 (CEST) Received: by mail-ot1-f45.google.com with SMTP id 88-20020a9d0ee1000000b005d0ae4e126fso1089413otj.5 for ; Wed, 13 Apr 2022 05:47:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0HsqaCrD/F32Sc6deWwflIlJe/xMqFg5X28DGVAa0rQ=; b=K5CErbtu2EJ3/BLi8//d3NUDjOXc5Y4sjjN8oolrK9Qr0U+KEwa6F7wFyNj7+YqdRl phzcWarfBCSejXpqUWbyjf6BaX00EjbiIYXcNRBMvuLH47Fmb5q+KE8d7UoFISk3w+NH DoAE5mlXlfUeMz3rZwjMtQbbMAJyY4oaemyXo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0HsqaCrD/F32Sc6deWwflIlJe/xMqFg5X28DGVAa0rQ=; b=qJC3pKrK7c4ZFZsQC/eyYiv6zQ/IG1Po5gZa0CgCM1NOvj5qi7LUQDuTVeolKJ+cmx yy5GJp7oKjczbzQeFf4+1zN/x1H4i+eVv0BGGNC/4ueC0agy+ijnBkjq/k8mZkiIFsaQ /U1AVRo2f3PxEe95NYudUcYZ3PAjMd2lHn4pCcpW2YnZYQo5uYQc6ZXj/OBi+gVQkALu dx1YDO7xwg/LgGXbxsopDWUUcrijK1NVL9ce91SdomEhRLfLL510CsyBbspZATnW/xwD 7W7QDjNTI5cx9pZS7oduAIBVuEXtW2fmlnOBO+PntRBOG3vuvdrIOPai5WxpLqLXx61d h3Fw== X-Gm-Message-State: AOAM532g96R8XcuxMztFG6LoM+HVl7aSeBsXiCr6CBtcJ6aZhsMzGN7p jMXzQC0fw1I8UsOsjBx9ZYQgwsaJAUtkIajAvsCYeQ== X-Google-Smtp-Source: ABdhPJyJ8EhFCaNyo4nQBDy0JmUqTIdWLnVkkDS7rqJ48vPq54sWYWcap6SjnMeY0mDTQaGgfvXB87DXRLMJg1XluWc= X-Received: by 2002:a05:6830:2694:b0:5e6:d16a:9486 with SMTP id l20-20020a056830269400b005e6d16a9486mr7647129otu.239.1649854063562; Wed, 13 Apr 2022 05:47:43 -0700 (PDT) MIME-Version: 1.0 References: <20220407214707.29730-1-ohilyard@iol.unh.edu> In-Reply-To: From: Owen Hilyard Date: Wed, 13 Apr 2022 08:47:07 -0400 Message-ID: Subject: Re: [PATCH v1 0/4] [RFC] Testpmd RPC API To: Jerin Jacob Cc: dpdk-dev , Honnappa Nagarahalli , Thomas Monjalon Content-Type: multipart/alternative; boundary="00000000000063e64505dc889569" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --00000000000063e64505dc889569 Content-Type: text/plain; charset="UTF-8" > > If so, I think, gRPC service would be along with existing > testpmd functions, like start_packet_forwarding(). It was my intention to re-use existing functions. I used the ACL tests as an example because they are more self-contained then Testpmd, which made creating the proof of concept much easier. Also, We don't need to rewrite the existing testpmd, Instead, RPC service, > we can add in existing app/test-pmd/ > The reason that I split out the services is that there doesn't seem to be a way to produce multiple binaries without re-writing that section of the build system. I wanted to avoid the hard requirement of having a C++ compiler available in order to be able to use testpmd, since that may affect what platforms Testpmd can run on and I want to avoid this being any kind of breaking change. If we decide to go the route of putting it all in a single application, we would need to conditionally enable the gRPC service at build time. Meson's current lack of support for conditionally detecting compilers causes issues here. I think, DPDK has only one interactive test case which is testpmd, > Could you point me to that test case? Either invocation or source is ok. I can't see anything that would lead me to assume use of testpmd in "meson test --list". To my knowledge, all of the test cases that use testpmd are in DTS. If there is a test that uses testpmd but is not part of DTS, I think it would be a candidate for moving into DTS assuming it's not a unit test. How you are planning to do noninteractive test cases? I'm not planning to make any change to unit testing, you can read more about how testing is currently conducted here: https://www.dpdk.org/blog/2021/07/05/dpdk-testing-approaches/ If there is a unit test that involves testpmd, there are two possibilities. 1. If we are making a separate application for Testpmd with the gRPC api, then nothing changes except for possibly changing where some of the testpmd source lives in order to enable code reuse between the two applications. 2. If gRPC is being added to Testpmd, then the unit test should still function as it previously did if I do any necessary refactoring as correctly. I think, key should be leveraging existing test cases as much as possible > and make easy for developers to add new test cases. That is part of the reason why I want to be able to do this. Adding a new test in DTS is very easy if the functionality needed already exists in Testpmd. If the functionality does not exist, then adding the test becomes difficult, due to the required modifications to the Testpmd lexer and parser to accommodate the new command. My plan is to leave unit testing in C, but help make it easier to expose C functions to Python for integration testing. This gives us the best of both worlds in terms of access to DPDK and the ability to use a high-level language to write the tests. On Tue, Apr 12, 2022 at 2:07 AM Jerin Jacob wrote: > On Mon, Apr 11, 2022 at 11:19 PM Owen Hilyard > wrote: > >> > >> scheme is probably over-engineered > > > > > > I tried my hardest to keep this as simple as possible. The requirements > imposed by DTS being a distributed system in Python restricted what I could > do a lot. Needing to be compatible with DPDK's license also got rid of a > lot of options. Binding generators are made for simple projects, and DPDK > is not a simple project. There were some other options related to choice in > the RPC framework, but very few RPC protocols seem to work well with C and > be usable from Python, which is why I ended up using C++ with gRPC. Most of > the code in api_impl.cc is taken from /app/test-acl/main.c, and the new > part is mostly the C++ class at the bottom. Overall, this proposal comes > out to ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and > 100 lines of Meson. gRPC may be able to do a lot more than I need it to for > the proof of concept, but many of the features that are not used, like > bi-directional streaming, become very useful in writing more complicated > tests. Overall, this solution is probably more capable than we need it to > be, but I think that those extra capabilities don't come at a very large > cost. > > > Now it is clear, I was carried away with the POC test application and > I was not knowing existing DTS tests are based on python > > Is below a fair summary? > > 1) DPDK has interactive test cases and no interactive test cases. > > For The interactive test case like testpmd, I agree that we can enable > RPC service via gRPC server in C++ as and client in Python, and > something along the lines of exposing the existing test-pmd command > line function as service > to avoid command line parsing and reuse the existing python test suite. > > If so, I think, gRPC service would be along with existing testpmd > functions, like start_packet_forwarding(). Also, We don't need to > rewrite the existing testpmd, > Instead, RPC service, we can add in existing app/test-pmd/ and hook to > existing core testpmd functions to bypass the command-line parsing in > C and control from python client as needed as service. > > Also, I agree that pulling in gRPC C++ server boilerplate and hooking > to C functions is a good idea as it is the best C-based RPC scheme > available today. > > 2)I think, DPDK has only one interactive test case which is testpmd, > Remaining test cases are non-interactive, non-interactive test cases > can simply run over ssh with passwordless login. Right? > Do we need gRPC for that? Will the following scheme suffice? If not, > How you are planning to do noninteractive test cases? > i.e > a)Copy test to target > b) result=`ssh username@IP /path/to/testapp/in/target` > > I think, key should be leveraging existing test cases as much as > possible and make easy for developers to add new test cases. > > > >> > >> Now that, Test code is also part of DPDK. > > > > > > DTS is pure python. I tried to use FFI to call directly into DPDK from > Python and then use xmlrpc from the python standard library. As mentioned > in the writeup, I couldn't find a binding generator that would properly > handle DPDK's allocators, which made it so that anything passed to DPDK > using python was allocated using the system malloc. I don't think it is > wise to attempt to programmatically re-write the generated code to allow > for custom allocators. The original reason for needing to have DTS and DPDK > in the same repository was so that tests could be committed and run > alongside the feature patch. > > > >> Interactive - Testpmd one, I believe, Feeding stdin programmatically > would suffice to test all the combinations. > > > > > > One of the issues this is trying to address is that human-readable > strings are a poor way to pass complex information between two programs. > DTS is a distributed system, and it can have up to 3 physical servers > involved in any given test. This means that it's not stdin via a pipe, it's > an entire SSH session. This adds a noticeable amount of overhead when > trying to send and verify the result of sending 1,000+ packets, since the > lack of structured output means each packet must be checked before the next > can be sent. This might be solvable by adding a structured output mode to > testpmd, but that would involve committing to writing output twice for > every function in testpmd forever. > > > >> We need to add all test cases in this model and we need to maintain two > sets of programs.(Traditional tests and gRPC model-based tests). > > > > > > Assuming by traditional tests you mean the unit tests run by Meson, I > would argue that we are already maintaining 2 kinds of tests. The unit > tests, and the python-based DTS tests. My intention is to create a thin > wrapper around DPDK that would be exposed via gRPC, like you see here, and > use that as midware. Then, we would have two front-ends. Testpmd, which > takes text and then calls midware as it does now, and the gRPC frontend, > which parses messages from the RPC server and runs the midware. This would > enable testpmd to still be used to sanity check a DPDK installation, but we > would not need to continually expand Testpmd. The primary issue is that, > right now, anything not included in Testpmd is not really testable by DTS. > This includes portions of the RTE Flow API, which was part of my reason for > proposing this. The RTE Flow API would, in my estimation, if fully > implemented into Testpmd, probably add at least another 10,000 lines of > code. As mentioned in my proposal, Testpmd already does more parsing and > lexing than it does interaction with DPDK by line count. Also, since I am > proposing making this a separate application, we would be able to gradually > migrate the tests inside of DTS. This would have no effect on anything > except for Testpmd, the new application and the addition of a flag to > toggle the use of a C++ compiler. > > > > I'm not sure exactly what you mean by gRPC model-based tests. gRPC uses > classes to model services, but for this usecase we are essentially using it > to transfer function arguments across the internet and then pass the return > value back. Any RPC framework would function similarly if I ignored the > restrictions of which languages to use, and the choice is not important to > how tests are conducted. Put another way, how you write a test for DTS will > not change much if you are using this or testpmd, it's just how you > transfer data and get it back that I want to change. > --00000000000063e64505dc889569 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
If so, I= think, gRPC service would be along with existing testpmd=C2=A0functions, l= ike start_packet_forwarding().=C2=A0

It was= my intention to re-use existing functions. I used the ACL tests as an exam= ple because they are more self-contained then Testpmd, which made creating = the proof of concept much easier.=C2=A0

Also, We don't need to=C2=A0rewrite = the existing testpmd,=C2=A0Instead, RPC service, we can add in existing app= /test-pmd/

The reason that I split out = the services is that there doesn't seem to be a way to produce multiple= binaries without re-writing that section of the build system. I wanted to = avoid the hard requirement of having a C++ compiler available=C2=A0in order= to be able to use testpmd, since that may affect what platforms Testpmd ca= n run on and I want to avoid this being any kind of breaking change. If we = decide to go the route of putting it all in a single application, we would = need to conditionally enable the gRPC service at build time. Meson's cu= rrent lack of support for conditionally detecting compilers causes issues h= ere.=C2=A0

I think, DPDK has only one interactive test case which is testpmd,

Could you point me to that test case? Eit= her invocation or source is ok. I can't see anything that would lead me= to assume use of testpmd in "meson test --list". To my knowledge= , all of the test cases that use testpmd are in DTS. If there is a test tha= t uses testpmd but is not part of DTS, I think it would be a candidate for = moving into DTS assuming it's not a unit test.=C2=A0

How you are planning to= do noninteractive test cases?

I'm not = planning to make any change to unit testing, you can read more about how te= sting is currently conducted here:=C2=A0https://www.dpdk.org/blog/2021/07/05= /dpdk-testing-approaches/

If there is a unit t= est that involves testpmd, there are two possibilities.
1. If we = are making a separate application for Testpmd with the gRPC api, then nothi= ng changes except for possibly changing where some of the testpmd source li= ves in order to enable code reuse between the two applications.=C2=A0
=
2. If gRPC is being added to Testpmd, then the unit test should still = function as it previously did if I do any necessary refactoring as correctl= y.=C2=A0

I think, key should be leveraging existing test cases as much as=C2=A0p= ossible and make easy for developers to add new test cases.

That is part of the reason why I want to be able to do thi= s. Adding a new test in DTS is very easy if the functionality needed alread= y exists in Testpmd. If the functionality does not exist, then adding the t= est becomes difficult, due to the required modifications to the Testpmd lex= er and parser to accommodate the new command. My plan is to leave unit test= ing in C, but help make it easier to expose C functions to Python for integ= ration testing. This gives us the best of both worlds in terms of access to= DPDK and the ability to use a high-level language to write the tests.=C2= =A0

On Tue, Apr 12, 2022 at 2:07 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
On Mon, Apr 11, 2022 at 11:19 P= M Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>>
>> scheme is probably over-engineered
>
>
> I tried my hardest to keep this as simple as possible. The requirement= s imposed by DTS being a distributed system in Python restricted what I cou= ld do a lot. Needing to be compatible with DPDK's license also got rid = of a lot of options. Binding generators are made for simple projects, and D= PDK is not a simple project. There were some other options related to choic= e in the RPC framework, but very few RPC protocols seem to work well with C= and be usable from Python, which is why I ended up using C++ with gRPC. Mo= st of the code in api_impl.cc is taken from /app/test-acl/main.c, and the n= ew part is mostly the C++ class at the bottom. Overall, this proposal comes= out to ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and = 100 lines of Meson. gRPC may be able to do a lot more than I need it to for= the proof of concept, but many of the features that are not used, like bi-= directional streaming, become very useful in writing more complicated tests= . Overall, this solution is probably more capable than we need it to be, bu= t I think that those extra capabilities don't come at a very large cost= .


Now it is clear, I was carried away with the POC test application and
I was not knowing existing DTS tests are based on python

Is below a fair summary?

1) DPDK has interactive test cases and no interactive test cases.

For The interactive test case like testpmd, I agree that we can enable
RPC service via gRPC server in C++ as=C2=A0 and client in Python, and
something along the lines of exposing the existing test-pmd command
line function as service
to avoid command line parsing and reuse the existing python test suite.

If so, I think, gRPC service would be along with existing testpmd
functions, like start_packet_forwarding(). Also, We don't need to
rewrite the existing testpmd,
Instead, RPC service, we can add in existing app/test-pmd/ and hook to
existing core testpmd functions to bypass the command-line parsing in
C and control from python client as needed as service.

Also, I agree that pulling in gRPC C++ server boilerplate and hooking
to C functions is a good idea as it is the best C-based RPC scheme
available today.

2)I think, DPDK has only one interactive test case which is testpmd,
Remaining test cases are non-interactive, non-interactive test cases
can simply run over ssh with passwordless login. Right?
Do we need gRPC for that? Will the following scheme suffice? If not,
How you are planning to do noninteractive test cases?
i.e
a)Copy test to target
b) result=3D`ssh username@IP /path/to/testapp/in/target`

I think, key should be leveraging existing test cases as much as
possible and make easy for developers to add new test cases.


>>
>> Now that, Test code is also part of DPDK.
>
>
> DTS is pure python. I tried to use FFI to call directly into DPDK from= Python and then use xmlrpc from the python standard library. As mentioned = in the writeup, I couldn't find a binding generator that would properly= handle DPDK's allocators, which made it so that anything passed to DPD= K using python was allocated using the system malloc. I don't think it = is wise to attempt to programmatically re-write the generated code to allow= for custom allocators. The original reason for needing to have DTS and DPD= K in the same repository was so that tests could be committed and run along= side the feature patch.
>
>> Interactive - Testpmd one, I believe, Feeding stdin programmatical= ly would suffice to test all the combinations.
>
>
> One of the issues this is trying to address is that human-readable str= ings are a poor way to pass complex information between two programs. DTS i= s a distributed system, and it can have up to 3 physical servers involved i= n any given test. This means that it's not stdin via a pipe, it's a= n entire SSH session. This adds a noticeable amount of overhead when trying= to send and verify the result of sending 1,000+ packets, since the lack of= structured output means each packet must be checked before the next can be= sent. This might be solvable by adding a structured output mode to testpmd= , but that would involve committing to writing output twice for every funct= ion in testpmd forever.
>
>> We need to add all test cases in this model and we need to maintai= n two sets of programs.(Traditional tests and gRPC model-based tests).
>
>
> Assuming by traditional tests you mean the unit tests run by Meson, I = would argue that we are already maintaining 2 kinds of tests. The unit test= s, and the python-based DTS tests. My intention is to create a thin wrapper= around DPDK that would be exposed via gRPC, like you see here, and use tha= t as midware. Then, we would have two front-ends. Testpmd, which takes text= and then calls midware as it does now, and the gRPC frontend, which parses= messages from the RPC server and runs the midware. This would enable testp= md to still be used to sanity check a DPDK installation, but we would not n= eed to continually expand Testpmd. The primary issue is that, right now, an= ything not included in Testpmd is not really testable by DTS. This includes= portions of the RTE Flow API, which was part of my reason for proposing th= is. The RTE Flow API would, in my estimation, if fully implemented into Tes= tpmd, probably add at least another 10,000 lines of code. As mentioned in m= y proposal, Testpmd already does more parsing and lexing than it does inter= action with DPDK by line count. Also, since I am proposing making this a se= parate application, we would be able to gradually migrate the tests inside = of DTS. This would have no effect on anything except for Testpmd, the new a= pplication and the addition of a flag to toggle the use of a C++ compiler.<= br> >
> I'm not sure exactly what you mean by gRPC model-based tests. gRPC= uses classes to model services, but for this usecase we are essentially us= ing it to transfer function arguments across the internet and then pass the= return value back. Any RPC framework would function similarly if I ignored= the restrictions of which languages to use, and the choice is not importan= t to how tests are conducted. Put another way, how you write a test for DTS= will not change much if you are using this or testpmd, it's just how y= ou transfer data and get it back that I want to change.
--00000000000063e64505dc889569--