DPDK patches and discussions
 help / color / mirror / Atom feed
From: Ciara Power <ciara.power@intel.com>
To: kevin.laatz@intel.com
Cc: dev@dpdk.org, reshma.pattan@intel.com,
	Ciara Power <ciara.power@intel.com>
Subject: [dpdk-dev] [PATCH 00/12] update and simplify telemetry library.
Date: Thu, 19 Mar 2020 17:18:55 +0000	[thread overview]
Message-ID: <20200319171907.60891-1-ciara.power@intel.com> (raw)

This patchset extensively reworks the telemetry library adding new
functionality and simplifying much of the existing code, while
maintaining backward compatibility.

This work is based on the previously sent RFC for a "process info"
library: https://patchwork.dpdk.org/project/dpdk/list/?series=7741
However, rather than creating a new library, this patchset takes
that work and merges it into the existing telemetry library, as
mentioned above.

The telemetry library as shipped in 19.11 is based upon the metrics
library and outputs all statistics based on that as a source. However,
this limits the telemetry output to only port-level statistics
information, rather than allowing it to be used as a general scheme for
telemetry information across all DPDK libraries.

With this patchset applied, rather than the telemetry library being
responsible for pulling ethdev stats and pushing them into the metrics
library for retrieval later, each library e.g. ethdev, rawdev, and even
the metrics library itself (for backwards compatiblity) now handle their
own stats.  Any library or app can register a callback function with
telemetry, which will be called if requested by the client connected via
the telemetry socket. The callback function in the library/app then
formats its stats, or other data, into a JSON string, and returns it to
telemetry to be sent to the client.

To maintain backward compatibility, e.g. to allow the dpdk telemetry
collectd plugin to continue to work, some of the existing telemetry
code is kept, but is moved into the metrics library, and callbacks are
registered with telemetry for the legacy commands that were supported
previously.

The new version of the library, apart from the legacy interface support
for backward compatibility, does not have an external dependency on the
Jansson library, allowing the library to be enabled by default.

Note: In this version of the patchset, telemetry output is provided by
the ethdev, rawdev and eal libraries, but this may be expanded further
in later versions which are planned ahead of the merge deadline for
20.05

Bruce Richardson (5):
  telemetry: invert dependency on metrics
  telemetry: introduce new telemetry functionality
  ethdev: add callback support for telemetry
  usertools: add new telemetry python script
  eal: add eal telemetry callbacks

Ciara Power (7):
  telemetry: move code to metrics for later reuse
  metrics: reduce code taken from telemetry
  rawdev: add callback support for telemetry
  examples/l3fwd-power: enable use of new telemetry
  telemetry: introduce telemetry backward compatibility
  telemetry: remove existing telemetry files
  lib: add telemetry as eal dependency

 config/common_base                            |    2 +-
 examples/l3fwd-power/Makefile                 |    4 +
 examples/l3fwd-power/main.c                   |   62 +-
 examples/l3fwd-power/meson.build              |    4 +
 lib/Makefile                                  |   10 +-
 lib/librte_eal/common/eal_common_options.c    |   79 +
 lib/librte_eal/common/eal_internal_cfg.h      |    1 +
 lib/librte_eal/common/eal_options.h           |    7 +
 lib/librte_eal/freebsd/eal/Makefile           |    1 +
 lib/librte_eal/freebsd/eal/eal.c              |   14 +
 lib/librte_eal/freebsd/eal/meson.build        |    2 +-
 lib/librte_eal/linux/eal/Makefile             |    1 +
 lib/librte_eal/linux/eal/eal.c                |   14 +
 lib/librte_eal/linux/eal/meson.build          |    2 +-
 lib/librte_eal/meson.build                    |    2 +-
 lib/librte_ethdev/Makefile                    |    4 +
 lib/librte_ethdev/meson.build                 |    4 +
 lib/librte_ethdev/rte_ethdev.c                |   79 +
 lib/librte_metrics/Makefile                   |   14 +
 lib/librte_metrics/meson.build                |   10 +
 lib/librte_metrics/rte_metrics.c              |    6 +-
 lib/librte_metrics/rte_metrics.h              |    5 +-
 lib/librte_metrics/rte_metrics_telemetry.c    |  539 +++++
 lib/librte_metrics/rte_metrics_telemetry.h    |   65 +
 lib/librte_metrics/rte_metrics_version.map    |    7 +
 lib/librte_rawdev/Makefile                    |    5 +
 lib/librte_rawdev/meson.build                 |    5 +
 lib/librte_rawdev/rte_rawdev.c                |   88 +
 lib/librte_telemetry/Makefile                 |   12 +-
 lib/librte_telemetry/meson.build              |   18 +-
 lib/librte_telemetry/rte_telemetry.c          | 1895 -----------------
 lib/librte_telemetry/rte_telemetry.h          |   53 +-
 lib/librte_telemetry/rte_telemetry_internal.h |  112 -
 lib/librte_telemetry/rte_telemetry_legacy.h   |   48 +
 lib/librte_telemetry/rte_telemetry_parser.c   |  682 ------
 lib/librte_telemetry/rte_telemetry_parser.h   |   15 -
 .../rte_telemetry_parser_test.c               |  533 -----
 .../rte_telemetry_socket_tests.h              |   36 -
 .../rte_telemetry_version.map                 |    5 +-
 lib/librte_telemetry/telemetry.c              |  290 +++
 lib/librte_telemetry/telemetry_legacy.c       |  226 ++
 lib/meson.build                               |    3 +-
 mk/rte.app.mk                                 |    9 +-
 mk/rte.vars.mk                                |    2 +
 usertools/test_new_telemetry.py               |   30 +
 45 files changed, 1659 insertions(+), 3346 deletions(-)
 create mode 100644 lib/librte_metrics/rte_metrics_telemetry.c
 create mode 100644 lib/librte_metrics/rte_metrics_telemetry.h
 delete mode 100644 lib/librte_telemetry/rte_telemetry.c
 delete mode 100644 lib/librte_telemetry/rte_telemetry_internal.h
 create mode 100644 lib/librte_telemetry/rte_telemetry_legacy.h
 delete mode 100644 lib/librte_telemetry/rte_telemetry_parser.c
 delete mode 100644 lib/librte_telemetry/rte_telemetry_parser.h
 delete mode 100644 lib/librte_telemetry/rte_telemetry_parser_test.c
 delete mode 100644 lib/librte_telemetry/rte_telemetry_socket_tests.h
 create mode 100644 lib/librte_telemetry/telemetry.c
 create mode 100644 lib/librte_telemetry/telemetry_legacy.c
 create mode 100755 usertools/test_new_telemetry.py

-- 
2.17.1


             reply	other threads:[~2020-03-19 17:34 UTC|newest]

Thread overview: 130+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-19 17:18 Ciara Power [this message]
2020-03-19 17:18 ` [dpdk-dev] [PATCH 01/12] telemetry: move code to metrics for later reuse Ciara Power
2020-03-19 17:18 ` [dpdk-dev] [PATCH 02/12] metrics: reduce code taken from telemetry Ciara Power
2020-03-19 17:18 ` [dpdk-dev] [PATCH 03/12] telemetry: invert dependency on metrics Ciara Power
2020-03-19 17:18 ` [dpdk-dev] [PATCH 04/12] telemetry: introduce new telemetry functionality Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 05/12] ethdev: add callback support for telemetry Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 06/12] usertools: add new telemetry python script Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 07/12] rawdev: add callback support for telemetry Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 08/12] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 09/12] telemetry: introduce telemetry backward compatibility Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 10/12] telemetry: remove existing telemetry files Ciara Power
2020-03-19 17:19 ` [dpdk-dev] [PATCH 11/12] lib: add telemetry as eal dependency Ciara Power
2020-03-20 12:03   ` Jerin Jacob
2020-03-20 13:50     ` Bruce Richardson
2020-03-19 17:19 ` [dpdk-dev] [PATCH 12/12] eal: add eal telemetry callbacks Ciara Power
2020-04-01 15:42 ` [dpdk-dev] [PATCH 00/12] update and simplify telemetry library David Marchand
2020-04-01 16:16   ` Bruce Richardson
2020-04-02  8:30     ` Morten Brørup
2020-04-02  9:38       ` Thomas Monjalon
2020-04-01 16:48 ` Wiles, Keith
2020-04-02 10:09   ` Bruce Richardson
2020-04-08 16:49 ` [dpdk-dev] [PATCH v2 00/16] " Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 01/16] build: add arch-specific header path to global includes Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 02/16] telemetry: move code to metrics for later reuse Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 03/16] metrics: reduce code taken from telemetry Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 04/16] telemetry: invert dependency on metrics Ciara Power
2020-04-10 16:15     ` Pattan, Reshma
2020-04-15  9:50       ` Power, Ciara
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 05/16] telemetry: introduce new telemetry functionality Ciara Power
2020-04-08 17:56     ` Wiles, Keith
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 06/16] telemetry: add utility functions for creating json Ciara Power
2020-04-08 18:12     ` Wiles, Keith
2020-04-09  8:19       ` Bruce Richardson
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 07/16] app/test: add telemetry json tests Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 08/16] ethdev: add callback support for telemetry Ciara Power
2020-04-08 18:16     ` Wiles, Keith
2020-04-09  8:20       ` Bruce Richardson
2020-04-10  9:57         ` [dpdk-dev] [PATCH v2 08/16] ethdev: add callback support fortelemetry Morten Brørup
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 09/16] usertools: add new telemetry python script Ciara Power
2020-04-10  9:43     ` Pattan, Reshma
2020-04-10  9:54       ` Bruce Richardson
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 10/16] rawdev: add callback support for telemetry Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 11/16] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-10  8:42     ` Pattan, Reshma
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 12/16] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-10 17:00     ` Pattan, Reshma
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 13/16] telemetry: remove existing telemetry files Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 14/16] lib: add telemetry as eal dependency Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 15/16] eal: add eal telemetry callbacks Ciara Power
2020-04-08 16:49   ` [dpdk-dev] [PATCH v2 16/16] doc: update telemetry documentation Ciara Power
2020-04-08 18:03   ` [dpdk-dev] [PATCH v2 00/16] update and simplify telemetry library Thomas Monjalon
2020-04-09  9:19     ` Bruce Richardson
2020-04-09  9:37       ` Thomas Monjalon
2020-04-10 14:39         ` Wiles, Keith
2020-04-10 14:51           ` Thomas Monjalon
2020-04-10 14:59             ` Wiles, Keith
2020-04-23 10:30         ` Luca Boccassi
2020-04-23 10:44           ` Thomas Monjalon
2020-04-23 11:46             ` Luca Boccassi
2020-04-10 10:49   ` Morten Brørup
2020-04-10 14:21     ` Wiles, Keith
2020-04-10 18:06       ` [dpdk-dev] [PATCH v2 00/16] update and simplify telemetrylibrary Morten Brørup
2020-04-20 13:18         ` Bruce Richardson
2020-04-20 14:55           ` [dpdk-dev] [PATCH v2 00/16] update and simplifytelemetrylibrary Morten Brørup
2020-04-21 12:39 ` [dpdk-dev] [PATCH v3 00/17] update and simplify telemetry library Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 01/17] build: add arch-specific header path to global includes Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 02/17] telemetry: move code to metrics for later reuse Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 03/17] metrics: reduce code taken from telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 04/17] telemetry: invert dependency on metrics Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 05/17] telemetry: introduce new telemetry functionality Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 06/17] telemetry: add utility functions for creating json Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 07/17] app/test: add telemetry json tests Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 08/17] ethdev: add callback support for telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 09/17] usertools: add new telemetry python script Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 10/17] rawdev: add callback support for telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 11/17] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 12/17] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 13/17] telemetry: remove existing telemetry files Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 14/17] lib: add telemetry as eal dependency Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 15/17] eal: remove rte-option infrastructure Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 16/17] eal: add eal telemetry callbacks Ciara Power
2020-04-21 12:39   ` [dpdk-dev] [PATCH v3 17/17] doc: update telemetry documentation Ciara Power
2020-04-24 12:41 ` [dpdk-dev] [PATCH v4 00/18] update and simplify telemetry library Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 01/18] build: add arch-specific header path to global includes Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 02/18] telemetry: move code to metrics for later reuse Ciara Power
2020-04-24 15:29     ` Stephen Hemminger
2020-04-24 15:49       ` Bruce Richardson
2020-04-27  9:53       ` Power, Ciara
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 03/18] metrics: reduce code taken from telemetry Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 04/18] telemetry: invert dependency on metrics Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 05/18] telemetry: add utility functions for creating json Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 06/18] telemetry: introduce new telemetry functionality Ciara Power
2020-04-24 20:50     ` Wiles, Keith
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 07/18] telemetry: add functions for returning callback data Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 08/18] telemetry: add default callback commands Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 09/18] usertools: add new telemetry python script Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 10/18] ethdev: add callback support for telemetry Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 11/18] rawdev: " Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 12/18] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 13/18] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-24 20:59     ` Wiles, Keith
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 14/18] telemetry: remove existing telemetry files Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 15/18] lib: add telemetry as eal dependency Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 16/18] eal: remove rte-option infrastructure Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 17/18] eal: add eal telemetry callbacks Ciara Power
2020-04-24 12:41   ` [dpdk-dev] [PATCH v4 18/18] doc: update telemetry documentation Ciara Power
2020-04-24 21:09   ` [dpdk-dev] [PATCH v4 00/18] update and simplify telemetry library Wiles, Keith
2020-04-30 16:01 ` [dpdk-dev] [PATCH v5 " Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 01/18] build: add arch-specific header path to global includes Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 02/18] telemetry: move code to metrics for later reuse Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 03/18] metrics: reduce code taken from telemetry Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 04/18] telemetry: invert dependency on metrics Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 05/18] telemetry: add utility functions for creating json Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 06/18] telemetry: introduce new telemetry functionality Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 07/18] telemetry: add functions for returning callback data Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 08/18] telemetry: add default callback commands Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 09/18] usertools: add new telemetry python script Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 10/18] ethdev: add callback support for telemetry Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 11/18] rawdev: " Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 12/18] examples/l3fwd-power: enable use of new telemetry Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 13/18] telemetry: introduce telemetry backward compatibility Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 14/18] telemetry: remove existing telemetry files Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 15/18] lib: add telemetry as eal dependency Ciara Power
2020-05-10 22:29     ` Thomas Monjalon
2020-05-11  8:41       ` Bruce Richardson
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 16/18] eal: remove rte-option infrastructure Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 17/18] eal: add eal telemetry callbacks Ciara Power
2020-04-30 16:01   ` [dpdk-dev] [PATCH v5 18/18] doc: update telemetry documentation Ciara Power
2020-05-01 14:41   ` [dpdk-dev] [PATCH v5 00/18] update and simplify telemetry library Wiles, Keith
2020-05-10 22:41   ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200319171907.60891-1-ciara.power@intel.com \
    --to=ciara.power@intel.com \
    --cc=dev@dpdk.org \
    --cc=kevin.laatz@intel.com \
    --cc=reshma.pattan@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).