RE: [PATCH v7 0/4] Add lcore poll busyness telemetry

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> Sent: Monday, 10 October 2022 19.39
> 
> On 2022-10-10 17:22, Morten Brørup wrote:
> >> From: Mattias Rönnblom [mailto:mattias.ronnblom@ericsson.com]
> >> Sent: Thursday, 6 October 2022 17.27
> >>
> >> On 2022-10-06 15:25, Morten Brørup wrote:
> >>>> From: Kevin Laatz [mailto:kevin.laatz@intel.com]
> >>>> Sent: Wednesday, 5 October 2022 15.45
> >>>>
> >>>> On 14/09/2022 10:29, Kevin Laatz wrote:
> >>>>> Currently, there is no way to measure lcore polling busyness in a
> >>>> passive
> >>>>> way, without any modifications to the application. This patchset
> >> adds
> >>>> a new
> >>>>> EAL API that will be able to passively track core polling
> busyness.
> >>>> As part
> >>>>> of the set, new telemetry endpoints are added to read the
> generate
> >>>> metrics.
> >>>>>
> >>>>> ---
> >>>>
> >>>> Based on the feedback in the discussions on this patchset, we have
> >>>> decided to revoke the submission of this patchset for the 22.11
> >>>> release.
> >>>>
> >>>> We will re-evaluate the design with the aim to provide a more
> >>>> acceptable
> >>>> solution in a future release.
> >>>
> >>> Good call. Thank you!
> >>>
> >>> I suggest having an open discussion about requirements/expectations
> >> for such a solution, before you implement any code.
> >>>
> >>> We haven't found the golden solution for our application, but we
> have
> >> discussed it quite a lot internally. Here are some of our thoughts:
> >>>
> >>> The application must feed the library with information about how
> much
> >> work it is doing.
> >>>
> >>> E.g. A pipeline stage that polls the NIC for N ingress packets
> could
> >> feed the busyness library with values such as:
> >>>    - "no work": zero packets received,
> >>>    - "25 % utilization": less than N packets received (in this
> >> example: 8 of max 32 packets = 25 %), or
> >>>    - "100% utilization, possibly more work to do": all N packets
> >> received (more packets could be ready in the queue, but we don't
> know).
> >>>
> >>
> >> If some lcore's NIC RX queue always, for every poll operation,
> produces
> >> 8 packets out of a max burst of 32, I would argue that lcore is 100%
> >> busy. With always something to do, it doesn't have a single cycle to
> >> spare.
> >
> > I would argue that if I have four cores each only processing 25 % of
> the packets, one core would suffice instead. Or, the application could
> schedule the function at 1/4 of the frequency it does now (e.g. call
> the function once every 40 microseconds instead of once every 10
> microseconds).
> >
> 
> Do you mean "only processing packets 25% of the time"? If yes, being
> able to replace four core @ 25% utilization with one core @ 100% might
> be a reasonable first guess. I'm not sure how it relates to what I
> wrote, though.

I meant: "only processing 25 % of the maximum number of packets it could have processed (if 100 % utilized)"

A service is allowed to do some fixed maximum amount of work every time its function is called; in my example receive 32 packets. But if there were only 8 packets to receive in a call, the utilization in that call was only 25 % (although the lcore was 100 % active in duration of the function call). So the function can be called less frequently.

If the utilization per packet is linear, then it makes no difference. However, it is non-linear, so it is more efficient to call the function 1/4 of the times, receiving 32 packets each time, than calling it too frequently and only receive 8 packets each time.

> 
> > However, the business does not scale linearly with the number of
> packets processed - which an intended benefit of bursting.
> >
> 
> Sure, there's usually a non-linear relationship between the system
> capacity used and the resulting CPU utilization. It can be both in the
> manner you describe below, with the per-packet processing latency
> reduced at higher rates, or the other way around. For example, NIC RX
> LLC stashing may cause a lot of LLC evictions, and generally the
> application might have a larger working set size during high load, so
> there may be forces working in the other direction as well.
> 
> It seems to me "busyness" telemetry value should just be lcore thread
> CPU utilization (in total, or with some per-module breakdown).

We probably agree that if we cannot discriminate between useless and useful work, then the utilization will always be 100 % minus some cycles known by the library to be non-productive overhead.

I prefer a scale from 0 to 100 % useful, rather than a Boolean discriminating between useless and useful cycles spent in a function.

> If you
> want to know how much of the system's capacity is used, you need help
> from an application-specific agent, equipped with a model of how CPU
> utilization and capacity relates. Such a heuristic could take other
> factors into account as well, e.g. the average queue sizes, packet
> rates, packet sizes etc.

That is exactly what I'm exploring if this library can approximate in some simple way.

> 
> In my experience, for high touch applications (i.e., those that spends
> thousands of cycles per packet), CPU utilization is a pretty decent
> approximation on how much of the system's capacity is used.

I agree.

We can probably also agree that the number of cycles spent roughly follows a formula like: A + B * x, where B is the number of cycles spent to handle the burst itself, and C is the number of cycles it takes to handle an additional packet (or other unit of work) in the burst.

In high touch services, A might be relatively insignificant; but in other applications, A is very significant.

Going back to my example, and using my profiler numbers from below...

If 679 cycles are spent in every RX function call to receive 8 packets, and the function is called 4 times per time unit, it is 679 * 4 = 2.716 cycles spent per time unit.

If the library can tell us that the function is only doing 25 % useful work, we can instead call the function 1 time per time unit. We will still get the same 32 packets per time unit, but only spend 1.275 cycles per time unit.

This is one of the things I am hoping for the library to help the application achieve. (By feeding the library with information about the work done.)

> 
> > Here are some real life numbers from our in-house profiler library in
> a production environment, which says that polling the NIC for packets
> takes on average:
> >
> > 104 cycles when the NIC returns 0 packets,
> > 529 cycles when the NIC returns 1 packet,
> > 679 cycles when the NIC returns 8 packets, and
> > 1275 cycles when the NIC returns a full burst of 32 packets.
> >
> > (This includes some overhead from our application, so you will see
> other numbers in your application.)
> >
> >>
> >> It seems to me that you basically have two options, if you do
> >> application-level "busyness" reporting.
> >>
> >> Either the application
> >> a) reports when a section of useful work begins, and when it ends,
> as
> >> two separate function calls.
> >> b) after having taken a time stamp, and having completed a section
> of
> >> code which turned out to be something useful, it reports back to the
> >> busyness module with one function call, containing the busy cycles
> >> spent.
> >>
> >> In a), the two calls could be to the same function, with a boolean
> >> argument informing the busyness module if this is the beginning of a
> >> busy or an idle period. In such case, just pass "num_pkts_dequeued >
> 0"
> >> to the call.
> >
> > Our profiler library has a start()and an end() function, and an
> end_and_start() function for when a section directly follows the
> preceding section (to only take one timestamp instead of two).
> >
> 
> I like the idea of a end_and_start() (except for the name, maybe).
> 
> >>
> >> What you would like is a solution which avoid ping-pong between idle
> >> and
> >> busy states (with the resulting time stamping and computations) in
> >> scenarios where a lcore thread mix sources of work which often have
> >> items available, with sources that do not (e.g., packets in a RX
> queue
> >> versus reassembly timeouts in a core-local timer wheel). It would be
> >> better in that situation, to attribute the timer wheel poll cycles
> as
> >> busy cycles.
> >>
> >> Another crucial aspect is that you want the API to be simple, and
> code
> >> changes to be minimal.
> >>
> >> It's unclear to me if you need to account for both idle and busy
> >> cycles,
> >> or only busy cycles, and assume all other cycles are idle. The will
> be
> >> for a traditional 1:1 EAL thread <-> CPU core mapping, but not if
> the
> >> "--lcores" parameter is used to create floating EAL threads, and EAL
> >> threads which share the same core, and thus may not be able to use
> 100%
> >> of the TSC cycles.
> >>
> >>> A pipeline stage that services a QoS scheduler could additionally
> >> feed the library with values such as:
> >>>    - "100% utilization, definitely more work to do": stopped
> >> processing due to some "max work per call" limitation.
> >>>    - "waiting, no work until [DELAY] ns": current timeslot has been
> >> filled, waiting for the next timeslot to start.
> >>>
> >>> It is important to note that any pipeline stage processing packets
> >> (or some other objects!) might process a different maximum number of
> >> objects than the ingress pipeline stage. What I mean is: The number
> N
> >> might not be the same for all pipeline stages.
> >>>
> >>>
> >>> The information should be collected per lcore or thread, also to
> >> prevent cache trashing.
> >>>
> >>> Additionally, it could be collected per pipeline stage too, making
> >> the collection two-dimensional. This would essentially make it a
> >> profiling library, where you - in addition to seeing how much time
> is
> >> spent working - also can see which work the time is spent on.
> >>>
> >>
> >> If you introduce subcategories of "busy", like "busy-with-X", and
> >> "busy-with-Y", the book keeping will be more expensive, since you
> will
> >> transit between states even for 100% busy lcores (which in principle
> >> you
> >> never, or at least very rarely, need to do if you have only busy and
> >> idle as states).
> >>
> >> If your application is organized as DPDK services, you will get this
> >> already today, on a DPDK service level.
> >>
> >> If you have your application organized as a pipeline, and you use an
> >> event device as a scheduler between the stages, that event device
> has a
> >> good opportunity to do this kind of bookkeeping. DSW, for example,
> >> keeps
> >> track of the average processing latency for events, and how many
> events
> >> of various types have been processed.
> >>
> >
> > Lots of good input, Mattias. Let's see what others suggest. :-)
> >
> >>> As mentioned during the previous discussions, APIs should be
> provided
> >> to make the collected information machine readable, so the
> application
> >> can use it for power management and other purposes.
> >>>
> >>> One of the simple things I would like to be able to extract from
> such
> >> a library is CPU Utilization (percentage) per lcore. >
> >>> And since I want the CPU Utilization to be shown for multiple the
> >> time intervals (usually 1, 5 or 15 minutes; but perhaps also 1
> second
> >> or 1 millisecond) the output data should be exposed as a counter
> type,
> >> so my "loadavg application" can calculate the rate by subtracting
> the
> >> previously obtained value from the current value and divide the
> >> difference by the time interval.
> >>>
> >>
> >> I agree. In addition, you also want the "raw data" (lcore busy
> cycles)
> >> so you can do you own sampling, at your own favorite-length
> intervals.
> >>
> >>> -Morten
> >>>
> >