DPDK patches and discussions
 help / color / mirror / Atom feed
From: Thomas Monjalon <thomas@monjalon.net>
To: "Wiles, Keith" <keith.wiles@intel.com>,
	"Richardson, Bruce" <bruce.richardson@intel.com>
Cc: "Power, Ciara" <ciara.power@intel.com>, dev <dev@dpdk.org>,
	"Laatz, Kevin" <kevin.laatz@intel.com>,
	"Pattan, Reshma" <reshma.pattan@intel.com>,
	"jerinjacobk@gmail.com" <jerinjacobk@gmail.com>,
	"david.marchand@redhat.com" <david.marchand@redhat.com>,
	"mb@smartsharesystems.com" <mb@smartsharesystems.com>,
	Andrzej Ostruszka <amo@semihalf.com>
Subject: Re: [dpdk-dev] [PATCH v2 00/16] update and simplify telemetry library.
Date: Fri, 10 Apr 2020 16:51:03 +0200	[thread overview]
Message-ID: <12264742.EVyyLHbfrO@thomas> (raw)
In-Reply-To: <93132D13-8044-43A7-A817-4B526BEEBC29@intel.com>

10/04/2020 16:39, Wiles, Keith:
> > On Apr 9, 2020, at 4:37 AM, Thomas Monjalon <thomas@monjalon.net> wrote:
> > 09/04/2020 11:19, Bruce Richardson:
> >> On Wed, Apr 08, 2020 at 08:03:26PM +0200, Thomas Monjalon wrote:
> >>> 08/04/2020 18:49, Ciara Power:
> >>>> This patchset extensively reworks the telemetry library adding new
> >>>> functionality and simplifying much of the existing code, while
> >>>> maintaining backward compatibility.
> >>>> 
> >>>> This work is based on the previously sent RFC for a "process info"
> >>>> library: https://patchwork.dpdk.org/project/dpdk/list/?series=7741
> >>>> However, rather than creating a new library, this patchset takes
> >>>> that work and merges it into the existing telemetry library, as
> >>>> mentioned above.
> >>>> 
> >>>> The telemetry library as shipped in 19.11 is based upon the metrics
> >>>> library and outputs all statistics based on that as a source. However,
> >>>> this limits the telemetry output to only port-level statistics
> >>>> information, rather than allowing it to be used as a general scheme for
> >>>> telemetry information across all DPDK libraries.
> >>>> 
> >>>> With this patchset applied, rather than the telemetry library being
> >>>> responsible for pulling ethdev stats and pushing them into the metrics
> >>>> library for retrieval later, each library e.g. ethdev, rawdev, and even
> >>>> the metrics library itself (for backwards compatiblity) now handle their
> >>>> own stats.  Any library or app can register a callback function with
> >>>> telemetry, which will be called if requested by the client connected via
> >>>> the telemetry socket. The callback function in the library/app then
> >>>> formats its stats, or other data, into a JSON string, and returns it to
> >>>> telemetry to be sent to the client.
> >>> 
> >>> I think this is a global need in DPDK, and it is usually called RPC,
> >>> IPC or control messaging.
> >>> We had a similar need for multi-process communication, thus rte_mp IPC.
> >>> We also need a control channel for user configuration applications.
> >>> We also need to control some features like logging or tracing.
> >>> 
> >>> In my opinion, it is time to introduce a general control channel in DPDK.
> >>> The application must be in the loop of the control mechanism.
> >>> Making such channel standard will ease application adoption.
> >>> 
> >>> Please read some comments here:
> >>> http://inbox.dpdk.org/dev/2580933.jp2sp48Hzj@xps/
> >>> 
> >> Hi Thomas,
> >> 
> >> I agree that having a single control mechanism or messaging mechanism in
> >> DPDK would be nice to have. However, I don't believe the plans for such a
> >> scheme should impact this patchset right now as the idea of a common
> >> channel was only first mooted about a week ago, and while there has been
> >> some email discussion about it, there is as yet no requirements list that
> >> I've seen, nobody actually doing coding work on it, no rfc and most
> >> importantly no timeline for creating and merging such into DPDK.
> > 
> > Yes, this is a new idea.
> > Throwing the idea in this "telemetry" thread and in "IF proxy" thread
> > is the first step before starting a dedicated thread to design
> > a generic mechanism.
> > 
> >> At present though, DPDK has a telemetry solution that works for the use case
> >> of ethdev stats and some power management info, but requires a more general
> >> solution to allow monitoring tools like PMDT to introspect DPDK, and also
> >> to prove statistics for other parts of DPDK such as cryptodev, eventdev,
> >> and other libraries, plus the application itself if the app so desires.
> > 
> > Doing rework on telemetry is similar to a general control mechanism.
> > Can we take this opportunity to work on what we believe to be a bigger
> > idea? It should be done anyway, so why pushing this temporary solution?
> > Sometimes we need a quick answer to an urgent problem.
> > But I don't think telemetry is currently in such situation that
> > a rework in 20.05 is mandatory.
> 
> Updating telemetry to be more consumable and standardize on a single method to get stats/info out of DPDK is a clean and simple solution. Starting over and creating yet another solution means we are pushing this support out again and many customer are asking for this support now.
> 
> The current telemetry solution in this patch gives us a great starting point and going back to the drawing board is a waste of time IMO and we need something now. To me this is a urgent problem we need to solve now, as I want to push PMDT and if we keep pushing out this type of support then it will never be upstreamed.
> 
> In PMDT I believe I have resolved all of the tech boards concerns and just waiting for this patch and a patch to PCM to push the code back to DPDK again.
> 
> So please let's not redesign this again.

I understand your concern.

I think we need to go to the drawing board,
and consider at least these 5 use cases:
	1/ multi-process IPC
	2/ telemetry
	3/ IF proxy
	4/ external user configuration
	5/ log/trace start/stop

Merging telemetry means we'll rework 1 and 2 later.
I am OK with merging telemetry in 20.05 if we can be sure
that there will be no resistance and help for reworking it
with a more general communication channel if required later.

We need a kind of community vote here. Please give +1 / -1.
Giving +1 means you will help when needed.



  reply	other threads:[~2020-04-10 14:51 UTC|newest]

Thread overview: 130+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-19 17:18 [dpdk-dev] [PATCH 00/12] " Ciara Power
2020-03-19 17:18 ` [dpdk-dev] [PATCH 01/12] telemetry: move code to metrics for later reuse Ciara Power
2020-03-19 17:18 ` [dpdk-dev] [PATCH 02/12] metrics: reduce code taken from telemetry Ciara Power
2020-03-19 17:18 ` [dpdk-dev] [PATCH 03/12] telemetry: invert dependency on metrics Ciara Power
2020-03-19 17:18 ` [dpdk-dev] [PATCH 04/12] telemetry: introduce new telemetry functionality Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 05/12] ethdev: add callback support for telemetry Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 06/12] usertools: add new telemetry python script Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 07/12] rawdev: add callback support for telemetry Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 08/12] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 09/12] telemetry: introduce telemetry backward compatibility Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 10/12] telemetry: remove existing telemetry files Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 11/12] lib: add telemetry as eal dependency Ciara Power
2020-03-20 12:03   ` Jerin Jacob
2020-03-20 13:50     ` Bruce Richardson
2020-03-19 17:19 ` [dpdk-dev] [PATCH 12/12] eal: add eal telemetry callbacks Ciara Power
2020-04-01 15:42 ` [dpdk-dev] [PATCH 00/12] update and simplify telemetry library David Marchand
2020-04-01 16:16   ` Bruce Richardson
2020-04-02  8:30     ` Morten Brørup
2020-04-02  9:38       ` Thomas Monjalon
2020-04-01 16:48 ` Wiles, Keith
2020-04-02 10:09   ` Bruce Richardson
2020-04-08 16:49 ` [dpdk-dev] [PATCH v2 00/16] " Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 01/16] build: add arch-specific header path to global includes Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 02/16] telemetry: move code to metrics for later reuse Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 03/16] metrics: reduce code taken from telemetry Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 04/16] telemetry: invert dependency on metrics Ciara Power
2020-04-10 16:15     ` Pattan, Reshma
2020-04-15  9:50       ` Power, Ciara
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 05/16] telemetry: introduce new telemetry functionality Ciara Power
2020-04-08 17:56     ` Wiles, Keith
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 06/16] telemetry: add utility functions for creating json Ciara Power
2020-04-08 18:12     ` Wiles, Keith
2020-04-09  8:19       ` Bruce Richardson
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 07/16] app/test: add telemetry json tests Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 08/16] ethdev: add callback support for telemetry Ciara Power
2020-04-08 18:16     ` Wiles, Keith
2020-04-09  8:20       ` Bruce Richardson
2020-04-10  9:57         ` [dpdk-dev] [PATCH v2 08/16] ethdev: add callback support fortelemetry Morten Brørup
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 09/16] usertools: add new telemetry python script Ciara Power
2020-04-10  9:43     ` Pattan, Reshma
2020-04-10  9:54       ` Bruce Richardson
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 10/16] rawdev: add callback support for telemetry Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 11/16] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-10  8:42     ` Pattan, Reshma
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 12/16] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-10 17:00     ` Pattan, Reshma
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 13/16] telemetry: remove existing telemetry files Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 14/16] lib: add telemetry as eal dependency Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 15/16] eal: add eal telemetry callbacks Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 16/16] doc: update telemetry documentation Ciara Power
2020-04-08 18:03   ` [dpdk-dev] [PATCH v2 00/16] update and simplify telemetry library Thomas Monjalon
2020-04-09  9:19     ` Bruce Richardson
2020-04-09  9:37       ` Thomas Monjalon
2020-04-10 14:39         ` Wiles, Keith
2020-04-10 14:51           ` Thomas Monjalon [this message]
2020-04-10 14:59             ` Wiles, Keith
2020-04-23 10:30         ` Luca Boccassi
2020-04-23 10:44           ` Thomas Monjalon
2020-04-23 11:46             ` Luca Boccassi
2020-04-10 10:49   ` Morten Brørup
2020-04-10 14:21     ` Wiles, Keith
2020-04-10 18:06       ` [dpdk-dev] [PATCH v2 00/16] update and simplify telemetrylibrary Morten Brørup
2020-04-20 13:18         ` Bruce Richardson
2020-04-20 14:55           ` [dpdk-dev] [PATCH v2 00/16] update and simplifytelemetrylibrary Morten Brørup
2020-04-21 12:39 ` [dpdk-dev] [PATCH v3 00/17] update and simplify telemetry library Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 01/17] build: add arch-specific header path to global includes Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 02/17] telemetry: move code to metrics for later reuse Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 03/17] metrics: reduce code taken from telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 04/17] telemetry: invert dependency on metrics Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 05/17] telemetry: introduce new telemetry functionality Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 06/17] telemetry: add utility functions for creating json Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 07/17] app/test: add telemetry json tests Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 08/17] ethdev: add callback support for telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 09/17] usertools: add new telemetry python script Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 10/17] rawdev: add callback support for telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 11/17] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 12/17] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 13/17] telemetry: remove existing telemetry files Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 14/17] lib: add telemetry as eal dependency Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 15/17] eal: remove rte-option infrastructure Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 16/17] eal: add eal telemetry callbacks Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 17/17] doc: update telemetry documentation Ciara Power
2020-04-24 12:41 ` [dpdk-dev] [PATCH v4 00/18] update and simplify telemetry library Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 01/18] build: add arch-specific header path to global includes Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 02/18] telemetry: move code to metrics for later reuse Ciara Power
2020-04-24 15:29     ` Stephen Hemminger
2020-04-24 15:49       ` Bruce Richardson
2020-04-27  9:53       ` Power, Ciara
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 03/18] metrics: reduce code taken from telemetry Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 04/18] telemetry: invert dependency on metrics Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 05/18] telemetry: add utility functions for creating json Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 06/18] telemetry: introduce new telemetry functionality Ciara Power
2020-04-24 20:50     ` Wiles, Keith
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 07/18] telemetry: add functions for returning callback data Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 08/18] telemetry: add default callback commands Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 09/18] usertools: add new telemetry python script Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 10/18] ethdev: add callback support for telemetry Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 11/18] rawdev: " Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 12/18] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 13/18] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-24 20:59     ` Wiles, Keith
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 14/18] telemetry: remove existing telemetry files Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 15/18] lib: add telemetry as eal dependency Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 16/18] eal: remove rte-option infrastructure Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 17/18] eal: add eal telemetry callbacks Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 18/18] doc: update telemetry documentation Ciara Power
2020-04-24 21:09   ` [dpdk-dev] [PATCH v4 00/18] update and simplify telemetry library Wiles, Keith
2020-04-30 16:01 ` [dpdk-dev] [PATCH v5 " Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 01/18] build: add arch-specific header path to global includes Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 02/18] telemetry: move code to metrics for later reuse Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 03/18] metrics: reduce code taken from telemetry Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 04/18] telemetry: invert dependency on metrics Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 05/18] telemetry: add utility functions for creating json Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 06/18] telemetry: introduce new telemetry functionality Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 07/18] telemetry: add functions for returning callback data Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 08/18] telemetry: add default callback commands Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 09/18] usertools: add new telemetry python script Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 10/18] ethdev: add callback support for telemetry Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 11/18] rawdev: " Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 12/18] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 13/18] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 14/18] telemetry: remove existing telemetry files Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 15/18] lib: add telemetry as eal dependency Ciara Power
2020-05-10 22:29     ` Thomas Monjalon
2020-05-11  8:41       ` Bruce Richardson
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 16/18] eal: remove rte-option infrastructure Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 17/18] eal: add eal telemetry callbacks Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 18/18] doc: update telemetry documentation Ciara Power
2020-05-01 14:41   ` [dpdk-dev] [PATCH v5 00/18] update and simplify telemetry library Wiles, Keith
2020-05-10 22:41   ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=12264742.EVyyLHbfrO@thomas \
    --to=thomas@monjalon.net \
    --cc=amo@semihalf.com \
    --cc=bruce.richardson@intel.com \
    --cc=ciara.power@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=jerinjacobk@gmail.com \
    --cc=keith.wiles@intel.com \
    --cc=kevin.laatz@intel.com \
    --cc=mb@smartsharesystems.com \
    --cc=reshma.pattan@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).