Re: [PATCH v1 1/2] eal: add lcore busyness telemetry

["Burakov, Anatoly" <anatoly.burakov@intel.com>]

On 15-Jul-22 2:12 PM, Anatoly Burakov wrote:
> Currently, there is no way to measure lcore busyness in a passive way,
> without any modifications to the application. This patch adds a new EAL
> API that will be able to passively track core busyness.
> 
> The busyness is calculated by relying on the fact that most DPDK API's
> will poll for packets. Empty polls can be counted as "idle", while
> non-empty polls can be counted as busy. To measure lcore busyness, we
> simply call the telemetry timestamping function with the number of polls
> a particular code section has processed, and count the number of cycles
> we've spent processing empty bursts. The more empty bursts we encounter,
> the less cycles we spend in "busy" state, and the less core busyness
> will be reported.
> 
> In order for all of the above to work without modifications to the
> application, the library code needs to be instrumented with calls to
> the lcore telemetry busyness timestamping function. The following parts
> of DPDK are instrumented with lcore telemetry calls:
> 
> - All major driver API's:
>    - ethdev
>    - cryptodev
>    - compressdev
>    - regexdev
>    - bbdev
>    - rawdev
>    - eventdev
>    - dmadev
> - Some additional libraries:
>    - ring
>    - distributor
> 
> To avoid performance impact from having lcore telemetry support, a
> global variable is exported by EAL, and a call to timestamping function
> is wrapped into a macro, so that whenever telemetry is disabled, it only
> takes one additional branch and no function calls are performed. It is
> also possible to disable it at compile time by commenting out
> RTE_LCORE_BUSYNESS from build config.
> 
> This patch also adds a telemetry endpoint to report lcore busyness, as
> well as telemetry endpoints to enable/disable lcore telemetry.
> 
> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com>
> Signed-off-by: Conor Walsh <conor.walsh@intel.com>
> Signed-off-by: David Hunt <david.hunt@intel.com>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
> 
> Notes:
>      We did a couple of quick smoke tests to see if this patch causes any performance
>      degradation, and it seemed to have none that we could measure. Telemetry can be
>      disabled at compile time via a config option, while at runtime it can be
>      disabled, seemingly at a cost of one additional branch.
>      
>      That said, our benchmarking efforts were admittedly not very rigorous, so
>      comments welcome!
> 
>   config/rte_config.h                         |   2 +
>   lib/bbdev/rte_bbdev.h                       |  17 +-
>   lib/compressdev/rte_compressdev.c           |   2 +
>   lib/cryptodev/rte_cryptodev.h               |   2 +
>   lib/distributor/rte_distributor.c           |  21 +-
>   lib/distributor/rte_distributor_single.c    |  14 +-
>   lib/dmadev/rte_dmadev.h                     |  15 +-
>   lib/eal/common/eal_common_lcore_telemetry.c | 274 ++++++++++++++++++++
>   lib/eal/common/meson.build                  |   1 +
>   lib/eal/include/rte_lcore.h                 |  80 ++++++
>   lib/eal/meson.build                         |   3 +
>   lib/eal/version.map                         |   7 +
>   lib/ethdev/rte_ethdev.h                     |   2 +
>   lib/eventdev/rte_eventdev.h                 |  10 +-
>   lib/rawdev/rte_rawdev.c                     |   5 +-
>   lib/regexdev/rte_regexdev.h                 |   5 +-
>   lib/ring/rte_ring_elem_pvt.h                |   1 +
>   17 files changed, 437 insertions(+), 24 deletions(-)
>   create mode 100644 lib/eal/common/eal_common_lcore_telemetry.c
> 
> diff --git a/config/rte_config.h b/config/rte_config.h
> index 46549cb062..583cb6f7a5 100644
> --- a/config/rte_config.h
> +++ b/config/rte_config.h
> @@ -39,6 +39,8 @@
>   #define RTE_LOG_DP_LEVEL RTE_LOG_INFO
>   #define RTE_BACKTRACE 1
>   #define RTE_MAX_VFIO_CONTAINERS 64
> +#define RTE_LCORE_BUSYNESS 1
> +#define RTE_LCORE_BUSYNESS_PERIOD 4000000ULL

One possible improvement here would be to specify period in 
microseconds, and use rte_get_tsc_hz() to adjust the telemetry period. 
This would require adding code to EAL init, because we can't use that 
API until EAL has called `rte_eal_timer_init()`, but it would make the 
telemetry period CPU frequency independent.


-- 
Thanks,
Anatoly