From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4613AA059F; Fri, 10 Apr 2020 16:51:12 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 73F021D537; Fri, 10 Apr 2020 16:51:11 +0200 (CEST) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by dpdk.org (Postfix) with ESMTP id 7736D1C0DC for ; Fri, 10 Apr 2020 16:51:10 +0200 (CEST) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 7E7185C6; Fri, 10 Apr 2020 10:51:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Fri, 10 Apr 2020 10:51:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=91CAt0nAIy69Kg7rNcnjul7MB/QBbytzmkV8vsD5B7s=; b=gGJ1C5d6tock sQbdwa/DocIjdyWeU8WVrIVQhgj3h/PT9PKcZgobzl+sEGS0K97qf/0IsGhr6iNL /48aFHRrj6pgmpen79riA+DZEq5cAmLYjaAsCsamFree9dF1EoF5KgxRbfzm3QZr UF5PhbuNHwdo/8uxhhvPvAkT9UfSM9s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=91CAt0nAIy69Kg7rNcnjul7MB/QBbytzmkV8vsD5B 7s=; b=LkPYG7JUPS/Zl0M8k3vPidICmL10VIYGvptVTTjZicTk2+l8p7apWqEhj lXL5a7/crnv0xBNPQb+sRf0VTpRgDHBt+WF5XkQPT9HyRuur1sZVvWo0QdcjVzab np7dEwESJC+vrXT8eE80tZidgIj1wmyVpK3IZwSYjCDB6pFZsKA08RvRCuztqNBp pdGXBTUSuKIsmAiQABp4EKJQRnL8719PkrHhoTPhaWYcEGyxJkbcW9EjPumQTP0n vDFTCKHclUvM1biedLmnkWvFqmrLXnG/vxDm7Xdw9DtUDnFJzcvFvMB36orhAWWK MXqPQjRdt4dY13JM8s6wztSuHdpYg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrvddvgdekfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffoh hmrghinhepughpughkrdhorhhgnecukfhppeejjedrudefgedrvddtfedrudekgeenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrsh esmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id A3326328005E; Fri, 10 Apr 2020 10:51:05 -0400 (EDT) From: Thomas Monjalon To: "Wiles, Keith" , "Richardson, Bruce" Cc: "Power, Ciara" , dev , "Laatz, Kevin" , "Pattan, Reshma" , "jerinjacobk@gmail.com" , "david.marchand@redhat.com" , "mb@smartsharesystems.com" , Andrzej Ostruszka Date: Fri, 10 Apr 2020 16:51:03 +0200 Message-ID: <12264742.EVyyLHbfrO@thomas> In-Reply-To: <93132D13-8044-43A7-A817-4B526BEEBC29@intel.com> References: <20200319171907.60891-1-ciara.power@intel.com> <8682539.rMLUfLXkoz@thomas> <93132D13-8044-43A7-A817-4B526BEEBC29@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-dev] [PATCH v2 00/16] update and simplify telemetry library. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" 10/04/2020 16:39, Wiles, Keith: > > On Apr 9, 2020, at 4:37 AM, Thomas Monjalon wrote: > > 09/04/2020 11:19, Bruce Richardson: > >> On Wed, Apr 08, 2020 at 08:03:26PM +0200, Thomas Monjalon wrote: > >>> 08/04/2020 18:49, Ciara Power: > >>>> This patchset extensively reworks the telemetry library adding new > >>>> functionality and simplifying much of the existing code, while > >>>> maintaining backward compatibility. > >>>> > >>>> This work is based on the previously sent RFC for a "process info" > >>>> library: https://patchwork.dpdk.org/project/dpdk/list/?series=7741 > >>>> However, rather than creating a new library, this patchset takes > >>>> that work and merges it into the existing telemetry library, as > >>>> mentioned above. > >>>> > >>>> The telemetry library as shipped in 19.11 is based upon the metrics > >>>> library and outputs all statistics based on that as a source. However, > >>>> this limits the telemetry output to only port-level statistics > >>>> information, rather than allowing it to be used as a general scheme for > >>>> telemetry information across all DPDK libraries. > >>>> > >>>> With this patchset applied, rather than the telemetry library being > >>>> responsible for pulling ethdev stats and pushing them into the metrics > >>>> library for retrieval later, each library e.g. ethdev, rawdev, and even > >>>> the metrics library itself (for backwards compatiblity) now handle their > >>>> own stats. Any library or app can register a callback function with > >>>> telemetry, which will be called if requested by the client connected via > >>>> the telemetry socket. The callback function in the library/app then > >>>> formats its stats, or other data, into a JSON string, and returns it to > >>>> telemetry to be sent to the client. > >>> > >>> I think this is a global need in DPDK, and it is usually called RPC, > >>> IPC or control messaging. > >>> We had a similar need for multi-process communication, thus rte_mp IPC. > >>> We also need a control channel for user configuration applications. > >>> We also need to control some features like logging or tracing. > >>> > >>> In my opinion, it is time to introduce a general control channel in DPDK. > >>> The application must be in the loop of the control mechanism. > >>> Making such channel standard will ease application adoption. > >>> > >>> Please read some comments here: > >>> http://inbox.dpdk.org/dev/2580933.jp2sp48Hzj@xps/ > >>> > >> Hi Thomas, > >> > >> I agree that having a single control mechanism or messaging mechanism in > >> DPDK would be nice to have. However, I don't believe the plans for such a > >> scheme should impact this patchset right now as the idea of a common > >> channel was only first mooted about a week ago, and while there has been > >> some email discussion about it, there is as yet no requirements list that > >> I've seen, nobody actually doing coding work on it, no rfc and most > >> importantly no timeline for creating and merging such into DPDK. > > > > Yes, this is a new idea. > > Throwing the idea in this "telemetry" thread and in "IF proxy" thread > > is the first step before starting a dedicated thread to design > > a generic mechanism. > > > >> At present though, DPDK has a telemetry solution that works for the use case > >> of ethdev stats and some power management info, but requires a more general > >> solution to allow monitoring tools like PMDT to introspect DPDK, and also > >> to prove statistics for other parts of DPDK such as cryptodev, eventdev, > >> and other libraries, plus the application itself if the app so desires. > > > > Doing rework on telemetry is similar to a general control mechanism. > > Can we take this opportunity to work on what we believe to be a bigger > > idea? It should be done anyway, so why pushing this temporary solution? > > Sometimes we need a quick answer to an urgent problem. > > But I don't think telemetry is currently in such situation that > > a rework in 20.05 is mandatory. > > Updating telemetry to be more consumable and standardize on a single method to get stats/info out of DPDK is a clean and simple solution. Starting over and creating yet another solution means we are pushing this support out again and many customer are asking for this support now. > > The current telemetry solution in this patch gives us a great starting point and going back to the drawing board is a waste of time IMO and we need something now. To me this is a urgent problem we need to solve now, as I want to push PMDT and if we keep pushing out this type of support then it will never be upstreamed. > > In PMDT I believe I have resolved all of the tech boards concerns and just waiting for this patch and a patch to PCM to push the code back to DPDK again. > > So please let's not redesign this again. I understand your concern. I think we need to go to the drawing board, and consider at least these 5 use cases: 1/ multi-process IPC 2/ telemetry 3/ IF proxy 4/ external user configuration 5/ log/trace start/stop Merging telemetry means we'll rework 1 and 2 later. I am OK with merging telemetry in 20.05 if we can be sure that there will be no resistance and help for reworking it with a more general communication channel if required later. We need a kind of community vote here. Please give +1 / -1. Giving +1 means you will help when needed.