RE: [PATCH v7 0/4] Add lcore poll busyness telemetry

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Kevin Laatz [mailto:kevin.laatz@intel.com]
> Sent: Wednesday, 5 October 2022 15.45
> 
> On 14/09/2022 10:29, Kevin Laatz wrote:
> > Currently, there is no way to measure lcore polling busyness in a
> passive
> > way, without any modifications to the application. This patchset adds
> a new
> > EAL API that will be able to passively track core polling busyness.
> As part
> > of the set, new telemetry endpoints are added to read the generate
> metrics.
> >
> > ---
> 
> Based on the feedback in the discussions on this patchset, we have
> decided to revoke the submission of this patchset for the 22.11
> release.
> 
> We will re-evaluate the design with the aim to provide a more
> acceptable
> solution in a future release.

Good call. Thank you!

I suggest having an open discussion about requirements/expectations for such a solution, before you implement any code.

We haven't found the golden solution for our application, but we have discussed it quite a lot internally. Here are some of our thoughts:

The application must feed the library with information about how much work it is doing.

E.g. A pipeline stage that polls the NIC for N ingress packets could feed the busyness library with values such as:
 - "no work": zero packets received,
 - "25 % utilization": less than N packets received (in this example: 8 of max 32 packets = 25 %), or
 - "100% utilization, possibly more work to do": all N packets received (more packets could be ready in the queue, but we don't know).

A pipeline stage that services a QoS scheduler could additionally feed the library with values such as:
 - "100% utilization, definitely more work to do": stopped processing due to some "max work per call" limitation.
 - "waiting, no work until [DELAY] ns": current timeslot has been filled, waiting for the next timeslot to start.

It is important to note that any pipeline stage processing packets (or some other objects!) might process a different maximum number of objects than the ingress pipeline stage. What I mean is: The number N might not be the same for all pipeline stages.


The information should be collected per lcore or thread, also to prevent cache trashing.

Additionally, it could be collected per pipeline stage too, making the collection two-dimensional. This would essentially make it a profiling library, where you - in addition to seeing how much time is spent working - also can see which work the time is spent on.

As mentioned during the previous discussions, APIs should be provided to make the collected information machine readable, so the application can use it for power management and other purposes.

One of the simple things I would like to be able to extract from such a library is CPU Utilization (percentage) per lcore.

And since I want the CPU Utilization to be shown for multiple the time intervals (usually 1, 5 or 15 minutes; but perhaps also 1 second or 1 millisecond) the output data should be exposed as a counter type, so my "loadavg application" can calculate the rate by subtracting the previously obtained value from the current value and divide the difference by the time interval.

-Morten