From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 10E0BA00C2; Fri, 24 Apr 2020 15:01:46 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AF40B1C217; Fri, 24 Apr 2020 15:01:44 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 9D2121BF3C for ; Fri, 24 Apr 2020 15:01:42 +0200 (CEST) IronPort-SDR: 3f98rL7Ig+4OdniCxokLgrbNZUXfj2qx/QBRGwiVohk/h5mwHAEcu8Xc0dmkkl/ShEYqopL/1I L66+uRljDuZw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Apr 2020 06:01:40 -0700 IronPort-SDR: yuj896Ai5kGT3ajCUF5qqXca9m3PrqW+zc4/4EWOJj1PNfZRe/e1hcUpTuD84arRduqMbADGcs aIvG/4nAtr3w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,311,1583222400"; d="scan'208";a="245228234" Received: from silpixa00399953.ir.intel.com (HELO silpixa00399953.ger.corp.intel.com) ([10.237.222.53]) by orsmga007.jf.intel.com with ESMTP; 24 Apr 2020 06:01:38 -0700 From: Ciara Power To: dev@dpdk.org, kevin.laatz@intel.com Cc: reshma.pattan@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, keith.wiles@intel.com, mb@smartsharesystems.com, thomas@monjalon.net, Ciara Power Date: Fri, 24 Apr 2020 13:41:41 +0100 Message-Id: <20200424124159.45989-1-ciara.power@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200319171907.60891-1-ciara.power@intel.com> References: <20200319171907.60891-1-ciara.power@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v4 00/18] update and simplify telemetry library. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" v4: - Internal representation now uses generic data format instead of JSON. - Help command added to provide help text for a command. v3: - Removed rte_option as it is no longer used. - General cleanup. v2: - Added JSON API, and unit tests, to simplify creation of valid json responses from libraries. - Added printing of basic info, including max output buffer size, app PID and DPDK version on connection. - Added /info command to report that basic info post-connect. This replaces the eal version command from v1. - Renamed stats to xstats in commands to allow a future generic "stats" call. - Added documentation, including updating existing howto and adding programmers guide section and API docs. - Added link status command for ethdev ports. - Fixed windows builds. This patchset extensively reworks the telemetry library adding new functionality and simplifying much of the existing code, while maintaining backward compatibility. This work is based on the previously sent RFC for a "process info" library: https://patchwork.dpdk.org/project/dpdk/list/?series=7741 However, rather than creating a new library, this patchset takes that work and merges it into the existing telemetry library, as mentioned above. The telemetry library as shipped in 19.11 is based upon the metrics library and outputs all statistics based on that as a source. However, this limits the telemetry output to only port-level statistics information, rather than allowing it to be used as a general scheme for telemetry information across all DPDK libraries. With this patchset applied, rather than the telemetry library being responsible for pulling ethdev stats and pushing them into the metrics library for retrieval later, each library e.g. ethdev, rawdev, and even the metrics library itself (for backwards compatiblity) now handle their own stats. Any library or app can register a callback function with telemetry, which will be called if requested by the client connected via the telemetry socket. The callback function in the library/app then formats its stats, or other data, into a generic data structure, and returns it to telemetry to be sent to the client in JSON format. To maintain backward compatibility, e.g. to allow the dpdk telemetry collectd plugin to continue to work, some of the existing telemetry code is kept, but is moved into the metrics library, and callbacks are registered with telemetry for the legacy commands that were supported previously. The new version of the library, apart from the legacy interface support for backward compatibility, does not have an external dependency on the Jansson library, allowing the library to be enabled by default. Note: In this version of the patchset, telemetry output is provided by the ethdev, rawdev and eal libraries, but this may be expanded further in later versions which are planned ahead of the merge deadline for 20.05 Bruce Richardson (8): build: add arch-specific header path to global includes telemetry: invert dependency on metrics telemetry: add utility functions for creating json telemetry: introduce new telemetry functionality telemetry: add functions for returning callback data usertools: add new telemetry python script ethdev: add callback support for telemetry eal: add eal telemetry callbacks Ciara Power (10): telemetry: move code to metrics for later reuse metrics: reduce code taken from telemetry telemetry: add default callback commands rawdev: add callback support for telemetry examples/l3fwd-power: enable use of new telemetry telemetry: introduce telemetry backward compatibility telemetry: remove existing telemetry files lib: add telemetry as eal dependency eal: remove rte-option infrastructure doc: update telemetry documentation app/test/Makefile | 2 + app/test/meson.build | 4 + app/test/test_telemetry_json.c | 136 ++ config/common_base | 2 +- config/meson.build | 7 - doc/guides/howto/telemetry.rst | 108 +- doc/guides/linux_gsg/eal_args.include.rst | 8 + doc/guides/linux_gsg/sys_reqs.rst | 2 - doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/telemetry_lib.rst | 61 + doc/guides/rel_notes/release_20_05.rst | 15 + examples/l3fwd-power/Makefile | 2 +- examples/l3fwd-power/main.c | 52 +- examples/l3fwd-power/meson.build | 2 +- lib/Makefile | 10 +- lib/librte_eal/arm/include/meson.build | 2 - lib/librte_eal/common/eal_common_options.c | 83 +- lib/librte_eal/common/eal_internal_cfg.h | 1 + lib/librte_eal/common/eal_options.h | 9 + lib/librte_eal/common/eal_private.h | 28 - lib/librte_eal/common/meson.build | 2 - lib/librte_eal/common/rte_option.c | 95 - lib/librte_eal/freebsd/Makefile | 2 +- lib/librte_eal/freebsd/eal.c | 26 +- lib/librte_eal/freebsd/meson.build | 2 +- lib/librte_eal/include/meson.build | 1 - lib/librte_eal/include/rte_option.h | 72 - lib/librte_eal/linux/Makefile | 2 +- lib/librte_eal/linux/eal.c | 26 +- lib/librte_eal/linux/meson.build | 2 +- lib/librte_eal/meson.build | 5 +- lib/librte_eal/ppc/include/meson.build | 2 - lib/librte_eal/rte_eal_version.map | 1 - lib/librte_eal/x86/include/meson.build | 2 - lib/librte_ethdev/Makefile | 2 +- lib/librte_ethdev/meson.build | 2 +- lib/librte_ethdev/rte_ethdev.c | 101 + lib/librte_metrics/Makefile | 10 + lib/librte_metrics/meson.build | 9 + lib/librte_metrics/rte_metrics.c | 6 +- lib/librte_metrics/rte_metrics.h | 5 +- lib/librte_metrics/rte_metrics_telemetry.c | 539 +++++ lib/librte_metrics/rte_metrics_telemetry.h | 65 + lib/librte_metrics/rte_metrics_version.map | 7 + lib/librte_rawdev/Makefile | 2 +- lib/librte_rawdev/meson.build | 2 + lib/librte_rawdev/rte_rawdev.c | 73 + lib/librte_telemetry/Makefile | 15 +- lib/librte_telemetry/meson.build | 15 +- lib/librte_telemetry/rte_telemetry.c | 1895 ----------------- lib/librte_telemetry/rte_telemetry.h | 236 +- lib/librte_telemetry/rte_telemetry_internal.h | 112 - lib/librte_telemetry/rte_telemetry_legacy.h | 87 + lib/librte_telemetry/rte_telemetry_parser.c | 682 ------ lib/librte_telemetry/rte_telemetry_parser.h | 15 - .../rte_telemetry_parser_test.c | 533 ----- .../rte_telemetry_socket_tests.h | 36 - .../rte_telemetry_version.map | 14 +- lib/librte_telemetry/telemetry.c | 412 ++++ lib/librte_telemetry/telemetry_data.c | 130 ++ lib/librte_telemetry/telemetry_data.h | 46 + lib/librte_telemetry/telemetry_json.h | 158 ++ lib/librte_telemetry/telemetry_legacy.c | 227 ++ lib/meson.build | 3 +- meson.build | 9 + mk/rte.app.mk | 11 +- usertools/dpdk-telemetry.py | 83 + usertools/meson.build | 2 +- 68 files changed, 2663 insertions(+), 3656 deletions(-) create mode 100644 app/test/test_telemetry_json.c create mode 100644 doc/guides/prog_guide/telemetry_lib.rst delete mode 100644 lib/librte_eal/common/rte_option.c delete mode 100644 lib/librte_eal/include/rte_option.h create mode 100644 lib/librte_metrics/rte_metrics_telemetry.c create mode 100644 lib/librte_metrics/rte_metrics_telemetry.h delete mode 100644 lib/librte_telemetry/rte_telemetry.c delete mode 100644 lib/librte_telemetry/rte_telemetry_internal.h create mode 100644 lib/librte_telemetry/rte_telemetry_legacy.h delete mode 100644 lib/librte_telemetry/rte_telemetry_parser.c delete mode 100644 lib/librte_telemetry/rte_telemetry_parser.h delete mode 100644 lib/librte_telemetry/rte_telemetry_parser_test.c delete mode 100644 lib/librte_telemetry/rte_telemetry_socket_tests.h create mode 100644 lib/librte_telemetry/telemetry.c create mode 100644 lib/librte_telemetry/telemetry_data.c create mode 100644 lib/librte_telemetry/telemetry_data.h create mode 100644 lib/librte_telemetry/telemetry_json.h create mode 100644 lib/librte_telemetry/telemetry_legacy.c create mode 100755 usertools/dpdk-telemetry.py -- 2.17.1