From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 22390A0501; Mon, 11 Apr 2022 19:49:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C610441611; Mon, 11 Apr 2022 19:49:22 +0200 (CEST) Received: from mail-oa1-f46.google.com (mail-oa1-f46.google.com [209.85.160.46]) by mails.dpdk.org (Postfix) with ESMTP id 01142415D7 for ; Mon, 11 Apr 2022 19:49:21 +0200 (CEST) Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-e2a00f2cc8so8637194fac.4 for ; Mon, 11 Apr 2022 10:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+WPkQb5kk3ZkrETaElleqWbeeOhf798pTXPMgcAdhpA=; b=AUXuclNpXJEnUuG9mKv4cmONl3EQpPkgPXeJzPrpn+tL7pULF7r4+w2gO59df0bgfg KyHTYT37lQJbCV41UDgTkpmc9eovECDl6qYueoTeTU3jFzoCRUlOIx+r3UbVLl3lxbd0 H9O//cvRRWbku022lN3Cc7yrcvJ+ltvJtduRQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+WPkQb5kk3ZkrETaElleqWbeeOhf798pTXPMgcAdhpA=; b=ys0ZscMsc+OK53WdUyUkufHeBA/HGwAI6vJymylDrIRWTjF99KtosmBWMC6F5rqdvX KwCyF3oemgn8ftNWcj37IVdAilma4ACcjcw8RsQFhOqWZnFvw82kvf1roJHtX4tbeXt3 zCLkHxVMJOv1d+wVkrmvU0U3InBSeTBjZ8uODoPyxH7xTLAl8cCXBx2EfCFiHFvZo5Td +UdvCFP0kAxVdTl5xiDJTpDTE9X/RJXyAHSKJTk7LHIQ+4UJC5XKry0Dn8YHYYXNLaro L6+G0llneMh7UDwndAO0MeCHLocMpJs8501H3sdOWmlW6iFpPsl62t7EXwHvHcfTa5za uCcA== X-Gm-Message-State: AOAM532ofeSD/rx1PGukuGVqFie4yO3kVhcOHb0lw06mcgvh/JdOJVp0 XApzIXER1qdVR8/k2/Iq+ktBDEX9r2AZdExHiyCVdg== X-Google-Smtp-Source: ABdhPJyUWZiU4jlOfM1njuNJK6QXM5KdH4PPIRJl6Wtx7FCXLhBBeUkmSggXbOojhrcxHnqyG8olv+2k0Yvv+R7BebE= X-Received: by 2002:a05:6870:61cc:b0:e1:9f6e:675a with SMTP id b12-20020a05687061cc00b000e19f6e675amr180707oah.213.1649699361207; Mon, 11 Apr 2022 10:49:21 -0700 (PDT) MIME-Version: 1.0 References: <20220407214707.29730-1-ohilyard@iol.unh.edu> In-Reply-To: From: Owen Hilyard Date: Mon, 11 Apr 2022 13:48:45 -0400 Message-ID: Subject: Re: [PATCH v1 0/4] [RFC] Testpmd RPC API To: Jerin Jacob Cc: dpdk-dev , Honnappa Nagarahalli , Thomas Monjalon Content-Type: multipart/alternative; boundary="000000000000694c0205dc649011" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000694c0205dc649011 Content-Type: text/plain; charset="UTF-8" > > scheme is probably over-engineered I tried my hardest to keep this as simple as possible. The requirements imposed by DTS being a distributed system in Python restricted what I could do a lot. Needing to be compatible with DPDK's license also got rid of a lot of options. Binding generators are made for simple projects, and DPDK is not a simple project. There were some other options related to choice in the RPC framework, but very few RPC protocols seem to work well with C and be usable from Python, which is why I ended up using C++ with gRPC. Most of the code in api_impl.cc is taken from /app/test-acl/main.c, and the new part is mostly the C++ class at the bottom. Overall, this proposal comes out to ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and 100 lines of Meson. gRPC may be able to do a lot more than I need it to for the proof of concept, but many of the features that are not used, like bi-directional streaming, become very useful in writing more complicated tests. Overall, this solution is probably more capable than we need it to be, but I think that those extra capabilities don't come at a very large cost. > Now that, Test code is also part of DPDK. > DTS is pure python. I tried to use FFI to call directly into DPDK from Python and then use xmlrpc from the python standard library. As mentioned in the writeup, I couldn't find a binding generator that would properly handle DPDK's allocators, which made it so that anything passed to DPDK using python was allocated using the system malloc. I don't think it is wise to attempt to programmatically re-write the generated code to allow for custom allocators. The original reason for needing to have DTS and DPDK in the same repository was so that tests could be committed and run alongside the feature patch. Interactive - Testpmd one, I believe, Feeding stdin programmatically would > suffice to test all the combinations. > One of the issues this is trying to address is that human-readable strings are a poor way to pass complex information between two programs. DTS is a distributed system, and it can have up to 3 physical servers involved in any given test. This means that it's not stdin via a pipe, it's an entire SSH session. This adds a noticeable amount of overhead when trying to send and verify the result of sending 1,000+ packets, since the lack of structured output means each packet must be checked before the next can be sent. This might be solvable by adding a structured output mode to testpmd, but that would involve committing to writing output twice for every function in testpmd forever. We need to add all test cases in this model and we need to maintain two > sets of programs.(Traditional tests and gRPC model-based tests). > Assuming by traditional tests you mean the unit tests run by Meson, I would argue that we are already maintaining 2 kinds of tests. The unit tests, and the python-based DTS tests. My intention is to create a thin wrapper around DPDK that would be exposed via gRPC, like you see here, and use that as midware. Then, we would have two front-ends. Testpmd, which takes text and then calls midware as it does now, and the gRPC frontend, which parses messages from the RPC server and runs the midware. This would enable testpmd to still be used to sanity check a DPDK installation, but we would not need to continually expand Testpmd. The primary issue is that, right now, anything not included in Testpmd is not really testable by DTS. This includes portions of the RTE Flow API, which was part of my reason for proposing this. The RTE Flow API would, in my estimation, if fully implemented into Testpmd, probably add at least another 10,000 lines of code. As mentioned in my proposal, Testpmd already does more parsing and lexing than it does interaction with DPDK by line count. Also, since I am proposing making this a separate application, we would be able to gradually migrate the tests inside of DTS. This would have no effect on anything except for Testpmd, the new application and the addition of a flag to toggle the use of a C++ compiler. I'm not sure exactly what you mean by gRPC model-based tests. gRPC uses classes to model services, but for this usecase we are essentially using it to transfer function arguments across the internet and then pass the return value back. Any RPC framework would function similarly if I ignored the restrictions of which languages to use, and the choice is not important to how tests are conducted. Put another way, how you write a test for DTS will not change much if you are using this or testpmd, it's just how you transfer data and get it back that I want to change. --000000000000694c0205dc649011 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
scheme is probably over-engineered

=
I tried my hardest to keep this as simple as possible. The requi= rements imposed by DTS being a distributed system in Python restricted what= I could do a lot. Needing to be compatible with DPDK's license also go= t rid of a lot of options. Binding generators are made for simple projects,= and DPDK is not a simple project. There were some other options related to= choice in the RPC framework, but very few RPC protocols seem to work well = with C and be usable from Python, which is why I ended up using C++ with gR= PC. Most of the code in api_impl.cc is taken from /app/test-acl/main.c, and= the new part is mostly the C++ class at the bottom. Overall, this proposal= comes out to ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobu= f and 100 lines of Meson. gRPC may be able to do a lot more than I need it = to for the proof of concept, but many of the features that are not used, li= ke bi-directional streaming, become very useful in writing more complicated= tests. Overall, this solution is probably more capable than we need it to = be, but I think that those extra capabilities don't come at a very larg= e cost.=C2=A0
=C2=A0
Now that, Test code is also part of DPDK.
= =C2=A0
DTS is pure python. I tried to use FFI to call directly in= to DPDK from Python and then use xmlrpc from the python standard library. A= s mentioned in the writeup, I couldn't find a binding generator that wo= uld properly handle DPDK's allocators, which made it so that anything p= assed to DPDK using python was allocated using the system malloc. I don'= ;t think it is wise to attempt to programmatically=C2=A0re-write the genera= ted code to allow for custom allocators. The original=C2=A0reason for needi= ng to have DTS and DPDK in the same repository was so that tests could be c= ommitted=C2=A0and run alongside the feature patch.=C2=A0

Interactive - Testpmd o= ne, I believe, Feeding stdin=C2=A0programmatically would suffice to test al= l the combinations.

One of the issues t= his is trying to address is that human-readable strings are a poor way to p= ass complex information between two programs. DTS is a distributed system, = and it can have up to 3 physical servers involved in any given test. This m= eans that it's not stdin=C2=A0via a pipe, it's an entire SSH sessio= n. This adds a noticeable amount of overhead when trying to send and verify= the result of sending 1,000+ packets, since the lack of structured output = means each packet must be checked before the next can be sent. This might b= e solvable by adding a structured output mode to testpmd, but that would in= volve committing to writing output twice for every function in testpmd fore= ver.=C2=A0

We need to add all test cases in this model and we need to maintain= =C2=A0two sets of programs.(Traditional tests and gRPC model-based tests).<= br>

Assuming by traditional tests you mean = the unit tests run by Meson, I would argue that we are already maintaining = 2 kinds of tests. The unit tests, and the python-based DTS tests. My intent= ion is to create a thin wrapper around DPDK that would be exposed via gRPC,= like you see here, and use that as midware. Then, we would have two front-= ends. Testpmd, which takes text and then calls midware as it does now, and = the gRPC frontend, which parses messages from the RPC server and runs the m= idware. This would enable testpmd to still be used to sanity check a DPDK i= nstallation, but we would not need to continually expand Testpmd. The prima= ry issue is that, right now, anything not included in Testpmd is not really= testable by DTS. This includes portions of the RTE Flow API, which was par= t of my reason for proposing this. The RTE Flow API would, in my estimation= , if fully implemented into Testpmd, probably add at least another 10,000 l= ines of code. As mentioned in my proposal, Testpmd already does more parsin= g and lexing than it does interaction with DPDK by line count. Also, since = I am proposing making this a separate application, we would be able to grad= ually migrate the tests inside of DTS. This would have no effect on anythin= g except for Testpmd, the new application and the addition of a flag to tog= gle the use of a C++ compiler.=C2=A0

I'm not s= ure exactly what you mean by gRPC model-based tests. gRPC uses classes to m= odel services, but for this usecase we are essentially using it to transfer= function arguments across the internet and then pass the return value back= . Any RPC framework would function similarly if I ignored the restrictions = of which languages to use, and the choice is not important to how tests are= conducted. Put another way, how you write a test for DTS will not change m= uch if you are using this or testpmd, it's just how you transfer data a= nd get it back that I want to change.=C2=A0
--000000000000694c0205dc649011--