From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id DF60A2C49 for ; Thu, 9 Mar 2017 17:25:38 +0100 (CET) Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP; 09 Mar 2017 08:25:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,136,1486454400"; d="scan'208";a="74743409" Received: from cchotard-mobl2.ger.corp.intel.com (HELO FC23.ger.corp.intel.com) ([10.252.10.254]) by fmsmga006.fm.intel.com with ESMTP; 09 Mar 2017 08:25:37 -0800 From: Remy Horton To: dev@dpdk.org Cc: Thomas Monjalon Date: Thu, 9 Mar 2017 16:25:27 +0000 Message-Id: <1489076734-31474-1-git-send-email-remy.horton@intel.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1486118019-2280-1-git-send-email-remy.horton@intel.com> Subject: [dpdk-dev] [PATCH v11 0/7] Expanded statistics reporting X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 16:25:39 -0000 This patchset consists of three libraries: A Metrics library for distributing device information, a library that calculates bit-rate statistics, and a library that calculates latency statistics. The latter two libraries make use of the first library. Metrics Library --------------- The Metrics library implements a mechanism by which producers can publish numeric information for later querying by consumers. In practice producers will typically be other libraries or primary processes, whereas consumers will typically be applications. Metrics themselves are statistics that are not generated by PMDs. Metric information is populated using a push model, where producers update the values contained within the metric library by calling an update function on the relevant metrics. Consumers receive metric information by querying the central metric data, which is held in shared memory so it can also be accessed by secondary processes. For each metric, a separate value is maintained for each port id, and when publishing metric values the producers need to specify which port is being updated. In addition there is a special id RTE_METRICS_GLOBAL that is intended for global statistics that are not associated with any individual device. Since the metrics library is self-contained, the only restriction on port numbers is that they are less than RTE_MAX_ETHPORTS - there is no requirement for the ports to actually exist. Metrics must first be registered, which is the way producers declare the names of the metrics they will be publishing. Registration can either be done individually, or as a group as a metric set. The intention is for all metrics in a set to be updated in one go, although it is also possible for metrics within a set to be updated individually. It is up to the producers to update metrics as required. Bit-rate statistics library --------------------------- The bit-rate library calculates the mean, the exponentially-weighted moving average, and peak bit-rates for each active port (i.e. network device). These statistics are then reported via the metrics library using the following names: mean_bits_in, mean_bits_out, ewma_bits_in, ewma_bits_out, peak_bits_in, and peak_bits_out. The sampling window used for calculation is decided by the application requiring the statistics. Latency statistics library -------------------------- The latency statistics library calculates the port-to-port latency of packet processing by a DPDK application, reporting the minimum, average, and maximum nano-seconds that packet processing takes, as well as the jitter in processing delay. These statistics are then reported via the metrics library using the following names: min_latency_ns, avg_latency_ns, mac_latency_ns, and jitter_ns. For more details on the metrics library and the Bit-rate and Latency components see the Programmer's Guide updates in the patchset. -- v11 changes: * Rebased * .map references to 17.02 changed to 17.05 * Release ntoes moved to release_17_05.rst * Bit-rate library now also gives unfiltered average v10 changes: * Rebased * Relocated some config-related directives. * Removed incorrect capitalisations in API docs. * Formatting & detail corrections in release notes. * Moved around struct member descriptions. * Rewritten rte_metrics.h file description. * Rewritten description of RTE_METRICS_GLOBAL. * Used 'producers' and 'consumers' as terms. * Removed markup (bold text) in Doxygen tags. * Added programming guide section. v9 changes: * Updated .map files to reflect function changes in v8 * Fixed rte_malloc() of zero bytes in proc_info when no metrics exist * Fixed rte_metrics_init not being called explicitly in testpmd v8 changes: * Release notes correction * Updated copyright years * rte_metric_init() takes socket id & must be explicitly called * rte_metrics_reg_metric renamed to rte_metrics_reg_name() * rte_metrics_update_metric() renamed to rte_metrics_update_value() * Doxygen updates * Changed malloc()/free() to rte_malloc()/rte_free() * Removed redundant memset() * rte_stats_bitrates_s renamed to rte_stats_bitrates_s * Split mbuf change to own patch for visibility * CYCLES_PER_NS now a static inline function * latency: "hidden" pthread creation now has polling API instead. * Struct declarations and variable definitions cleaned up * Double initialization of the latency library now returns -EEXIST * MAINTAINERS entry for layenctstats in correct section v7 changes: * RTE_METRICS_NONPORT renamed to RTE_METRICS_GLOBAL * Multiple changes to rte_metrics.h doxygen documentation * Split apart latency patch into lib, test-pmd, & proc_info parts * Reordered patches by functionality * Insufficent capacity return value changed from -ERANGE to actual size * Cache alignment in bitrate library * Tightened up const usage to avoid STATIC_CONST_CHAR_ARRAY warning * Reshma reinstated as author for (now split) latency patch * Rebase to master v6 changes: * Metrics display now has "Non port specific" rather than "port -1" * Fixed sign issue in EWMA delta calculation * Rebased to master v5 changes: * Updated Shared Library Versions in release notes * Merged in Reshma's latencystats library v4 changes: * References to 16.11 changed to 17.02 * Fetching of non-port values was broken * Added sanity checks to value fetching * rte_stat_value renamed to rte_metric_value * Corrected doxygen descriptions * Added MAINTAINERS entries * Added #ifdef directives to bitrate code in test-pmd v3 changes: * Marked rte_stats_bitrate_s as internal * Minor integer roundoff correction * Coding style corrections * Removed spurious object allocation * Changes to rte_metrics.[ch] moved from Patch 2/3 to 1/3. * Reintroduced non-port values (RTE_METRICS_NONPORT) * Added spinlocks to metric library * Removed spurious test registration/update * Added release notes entries v2 changes: * Uses a new metrics library rather than being part of ethdev Harry van Haaren (3): mbuf: add a timestamp to the mbuf for latencystats lib: added new library for latency stats app/test-pmd: add latency statistics calculation Remy Horton (3): lib: add information metrics library lib: add bitrate statistics library app/test-pmd: add bitrate statistics calculation Reshma Pattan (1): app/proc_info: add metrics displaying MAINTAINERS | 12 + app/proc_info/main.c | 79 ++++- app/test-pmd/parameters.c | 20 +- app/test-pmd/testpmd.c | 78 ++++- app/test-pmd/testpmd.h | 6 +- config/common_base | 15 + doc/api/doxy-api-index.md | 3 + doc/api/doxy-api.conf | 3 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/metrics_lib.rst | 299 +++++++++++++++++ doc/guides/rel_notes/release_17_02.rst | 3 + doc/guides/rel_notes/release_17_05.rst | 18 + lib/Makefile | 3 + lib/librte_bitratestats/Makefile | 53 +++ lib/librte_bitratestats/rte_bitrate.c | 141 ++++++++ lib/librte_bitratestats/rte_bitrate.h | 80 +++++ .../rte_bitratestats_version.map | 9 + lib/librte_latencystats/Makefile | 56 ++++ lib/librte_latencystats/rte_latencystats.c | 366 +++++++++++++++++++++ lib/librte_latencystats/rte_latencystats.h | 154 +++++++++ .../rte_latencystats_version.map | 11 + lib/librte_mbuf/rte_mbuf.h | 3 + lib/librte_metrics/Makefile | 51 +++ lib/librte_metrics/rte_metrics.c | 299 +++++++++++++++++ lib/librte_metrics/rte_metrics.h | 240 ++++++++++++++ lib/librte_metrics/rte_metrics_version.map | 13 + mk/rte.app.mk | 3 + 27 files changed, 2015 insertions(+), 4 deletions(-) create mode 100644 doc/guides/prog_guide/metrics_lib.rst create mode 100644 lib/librte_bitratestats/Makefile create mode 100644 lib/librte_bitratestats/rte_bitrate.c create mode 100644 lib/librte_bitratestats/rte_bitrate.h create mode 100644 lib/librte_bitratestats/rte_bitratestats_version.map create mode 100644 lib/librte_latencystats/Makefile create mode 100644 lib/librte_latencystats/rte_latencystats.c create mode 100644 lib/librte_latencystats/rte_latencystats.h create mode 100644 lib/librte_latencystats/rte_latencystats_version.map create mode 100644 lib/librte_metrics/Makefile create mode 100644 lib/librte_metrics/rte_metrics.c create mode 100644 lib/librte_metrics/rte_metrics.h create mode 100644 lib/librte_metrics/rte_metrics_version.map -- 2.5.5