From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 66D77A04A6 for ; Wed, 9 Feb 2022 09:53:51 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6077B41101; Wed, 9 Feb 2022 09:53:51 +0100 (CET) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by mails.dpdk.org (Postfix) with ESMTP id 696FE410FD; Wed, 9 Feb 2022 09:53:50 +0100 (CET) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 98D035C015A; Wed, 9 Feb 2022 03:53:47 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Feb 2022 03:53:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm3; bh=09OqW8Mzc9vnV7 E+jS4dEYm+nhgpyx3I5e16OMfMIuE=; b=VZQxZ9APCDCcxGtwlHJ0UaDJoatx8m DgFnLdixM+3rNxpWbREsIrw2Oi5hLZ/sFOIp2awKvRxjvJ9O5L8VXQdPPj0pIUqm S/QmxHdnmsZfXgZYOWqkbbGavA9OFBofvbF3q+sXwbxQnp/+4oHepS5RfU+v9ebS rg83ZAQyULcdhg/QKSfEKeAZsABHAHzGztLonji/sFEQmxMZqlnI8YzczmGWrky3 /TAwOlfA/APDbvcYCmAgfaXRoqYgXoS92jBlQr7zGhYO0bYEfqVFBppAKriiqJNT JPYhnbMUKYF6Y5A0lBh8O6pmZb1DgLp/FGEkB4E/5voPDOweDN+8W7Sw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=09OqW8Mzc9vnV7E+jS4dEYm+nhgpyx3I5e16OMfMI uE=; b=aCjSuuEQXU7GeBsfb2hGbklrD2VgRMjIn8tFEF6ekmHiqvKe0yZZ99IMT Swyp67QZPIm59tdmq8MLW6w/RqGjuZuaWq0AFX5Xa10C5VIlcFRV2B+6k/OA3in4 bO4ByTIscwAC363W0ankrsxVhqrWG8+G9VscijorGrtFD4+tir7cecn57tgCaGvZ CLTlOQgVu21uM8Ns1PG9m8Kw1ZjncpUbDWM4V28NEw1JxDRXlGSGReyxCYiT0UQN tTBY9z3bMTljR+bFhszJbpPFfp7QLu3RN6BWZq6e+/wQ/bcQULNa/92jT5sL56CI rSEuxWHULsFn++ZzW55+M+IwfKmXg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrheekgdduvddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtqhertddttdejnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecugg ftrfgrthhtvghrnhepheejgeektdegvedufeehhfefgeetkedtleettdfhledvudelveek udduhfekgfdvnecuffhomhgrihhnpeiiohhomhdruhhsnecuvehluhhsthgvrhfuihiivg eptdenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhhonhdr nhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Feb 2022 03:53:46 -0500 (EST) From: Thomas Monjalon To: Owen Hilyard Cc: dts@dpdk.org, dev , ci@dpdk.org, rasland@nvidia.com Subject: Re: [RFC] Testpmd Maintenance affecting DTS Date: Wed, 09 Feb 2022 09:53:44 +0100 Message-ID: <5238984.Sb9uPGUboI@thomas> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org 08/02/2022 22:34, Owen Hilyard: > Currently, DTS tests DPDK via the example applications (/examples) and the > apps (/app) included with DPDK. The DTS Working group has been trying to > move to primarily using Testpmd, since it is the most complete and only > using one application causes less maintenance work. Since Testpmd provides > a text interface into DPDK, it must be continually updated to expose this > new functionality, sometimes it may lag behind the current feature set of > DPDK. For example, Testpmd currently does not support all of the options > available for rte flow, which limits the testing coverage that can be > provided by DTS for RTE flow. Because DTS is a Python application, it is > not able to directly interface with DPDK. This means that whenever Testpmd > falls behind DPDK, those unexposed parts of DPDK are functionally > untestable as far as DTS is concerned. DTS is the tool used by the CI and > labs to functionally test DPDK in a working configuration, compared to the > unit testing, that may test individual components or parts of the stack. > The DTS improvements WG and the Community Lab have a goal to continue the > expansion of the test functional and performance test coverage of DPDK. > This means that there must be some way for everything that the DPDK > community wants to be tested to be exposed to DTS. I have a few possible > solutions to this problem: >=20 >=20 > 1. >=20 > Continue to update Testpmd every time new functionality is added to DP= DK > to expose that functionality Yes, any feature in ethdev should have a testpmd entry. Which part of rte_flow is not testable with testpmd? In general, any device API should be testable with the app/ directory. > 2. >=20 > Use a parser generator or some other method to make adding new options > to Testpmd much easier, updating this every time new functionality is = added > to testpmd Please could you elaborate how it would work? > 3. >=20 > Create a dedicated testing application for DPDK that uses a binding > generator and Python=E2=80=99s XMLRPC to allow more programmatic acces= s to DPDK on > the DUT, adding new RPC calls to expose new functionality. Why do you need RPC when a binding is generated? > I personally think that option 3 is the best because it would involve a b= it > of work up front to make it much easier to expose parts of DPDK to DTS. > This is because after the initial work is done, we would just need to wri= te > a wrapper function in C to expose the functionality we want, and then > expose the wrapper function via RPC. I think having a Python binding has 3 drawbacks: =2D performance is not good =2D it is testing a binding, not the real API =2D it is a huge maintenance effort for everybody > If you have thoughts or suggestions, we will be discussing this at the DTS > working group meeting. >=20 > 2:00 PM UTC, Wednesday, February 16 > https://armltd.zoom.us/j/97503259680?pwd=3DVVlmWnlnTXJkVGkwR2JOU3R3b3Vndz= 09&from=3Daddon