Ethdev tracepoints optimization

Ethdev tracepoints optimization

[Adel Belkhiri]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

Hi DPDK Community,

I am currently working on developing performance analyses for applications
using the ethdev library. These analyses are being implemented in Trace
Compass, an open-source performance analyzer. One of the views I’ve
implemented shows the rate of traffic received or sent by an ethernet port,
measured in packets per second. However, I've encountered an issue with the
lib.ethdev.rx.burst event, which triggers even when no packets are polled,
leading to a significant number of irrelevant events in the trace. This
becomes problematic as these "empty" events can overwhelm the tracer
buffer, potentially causing the loss of more critical events due to their
high frequency.

To address this, I've modified the DPDK code in lib/ethdev/rte_ethdev.h to
add a conditional statement that only triggers the event when nb_rx > 0. My
question to the community is whether there are use cases where an "empty"
lib.ethdev.rx.burst event could be useful. If not, would there be interest
in submitting a patch with this modification?

Moreover, I am looking to develop an analysis that calculates the
throughput (in kb/s, mb/s, etc.) per NIC, utilizing the same events (i.e.,
lib.ethdev.rx.burst and lib.ethdev.tx.burst). These tracepoints do not
provide packet size directly, only a pointer to the packet array. My
attempt to use an eBPF program to iterate through that array to access the
packet sizes was unsuccessful, as I found no method to export the computed
data (e.g., via a custom tracepoint). Does anyone have suggestions or
alternative approaches for achieving a throughput measurement?

I would be grateful for any insights or suggestions you might have.

Thank you!
Adel

[-- Attachment #2: Type: text/html, Size: 1783 bytes --]

Re: Ethdev tracepoints optimization

[Jerin Jacob]

On Fri, Aug 16, 2024 at 4:43 AM Adel Belkhiri <adel.belkhiri@gmail.com> wrote:
>
> Hi DPDK Community,
>
> I am currently working on developing performance analyses for applications using the ethdev library. These analyses are being implemented in Trace Compass, an open-source performance analyzer. One of the views I’ve implemented shows the rate of traffic received or sent by an ethernet port, measured in packets per second. However, I've encountered an issue with the lib.ethdev.rx.burst event, which triggers even when no packets are polled, leading to a significant number of irrelevant events in the trace. This becomes problematic as these "empty" events can overwhelm the tracer buffer, potentially causing the loss of more critical events due to their high frequency.
>
> To address this, I've modified the DPDK code in lib/ethdev/rte_ethdev.h to add a conditional statement that only triggers the event when nb_rx > 0. My question to the community is whether there are use cases where an "empty" lib.ethdev.rx.burst event could be useful. If not, would there be interest in submitting a patch with this modification?
>
> Moreover, I am looking to develop an analysis that calculates the throughput (in kb/s, mb/s, etc.) per NIC, utilizing the same events (i.e., lib.ethdev.rx.burst and lib.ethdev.tx.burst). These tracepoints do not provide packet size directly, only a pointer to the packet array. My attempt to use an eBPF program to iterate through that array to access the packet sizes was unsuccessful, as I found no method to export the computed data (e.g., via a custom tracepoint). Does anyone have suggestions or alternative approaches for achieving a throughput measurement?


Call rte_eth_stats_get() in N second interval and find throughput in slow path.



> I would be grateful for any insights or suggestions you might have.
>
> Thank you!
> Adel

Re: Ethdev tracepoints optimization

[Bruce Richardson]

On Thu, Aug 15, 2024 at 03:32:50PM -0400, Adel Belkhiri wrote:
>    Hi DPDK Community,
>    I am currently working on developing performance analyses for
>    applications using the ethdev library. These analyses are being
>    implemented in Trace Compass, an open-source performance analyzer. One
>    of the views I’ve implemented shows the rate of traffic received or
>    sent by an ethernet port, measured in packets per second. However, I've
>    encountered an issue with the lib.ethdev.rx.burst event, which triggers
>    even when no packets are polled, leading to a significant number of
>    irrelevant events in the trace. This becomes problematic as these
>    "empty" events can overwhelm the tracer buffer, potentially causing the
>    loss of more critical events due to their high frequency.
>    To address this, I've modified the DPDK code in lib/ethdev/rte_ethdev.h
>    to add a conditional statement that only triggers the event when nb_rx
>    > 0. My question to the community is whether there are use cases where
>    an "empty" lib.ethdev.rx.burst event could be useful. If not, would
>    there be interest in submitting a patch with this modification?

Yes, there probably would be, but also likely not in such a simple form. I
think there is value in tracing around empty polls too, but only in such a
way not to overflow the trace buffer.

My suggestion would be to trace the first empty poll only, but suppress any
polls thereafter. That way we can know that there are empty polls rather
than no calls.

An even better solution - though I don't know enough about how the tracing
works to know if it's possible - is to match that initial empty poll trace
with some sort of trace output when the empty polls stop, i.e. we get
traffic, recording the number of empty polls received as a count value.

My 2c.

/Bruce

Re: Ethdev tracepoints optimization

[Ferruh Yigit]

On 8/15/2024 8:32 PM, Adel Belkhiri wrote:
> Hi DPDK Community,
> 
> I am currently working on developing performance analyses for
> applications using the ethdev library. These analyses are being
> implemented in Trace Compass, an open-source performance analyzer. One
> of the views I’ve implemented shows the rate of traffic received or sent
> by an ethernet port, measured in packets per second. However, I've
> encountered an issue with the lib.ethdev.rx.burst event, which triggers
> even when no packets are polled, leading to a significant number of
> irrelevant events in the trace. This becomes problematic as these
> "empty" events can overwhelm the tracer buffer, potentially causing the
> loss of more critical events due to their high frequency.
> 
> To address this, I've modified the DPDK code in lib/ethdev/rte_ethdev.h
> to add a conditional statement that only triggers the event when nb_rx >
> 0. My question to the community is whether there are use cases where an
> "empty" lib.ethdev.rx.burst event could be useful. If not, would there
> be interest in submitting a patch with this modification?
>

Tracepoint is good way to get frequency of the calls, so I believe there
can be value of getting empty burst calls too.

But your usecase also a valid one. I wonder if it works to have separate
trace calls, for empty and non-empty ones, and how this additional
branching impacts the performance, at least branch should be wrapped
with 'RTE_ENABLE_TRACE_FP' macro to not impact non-tracing usage.

Re: Ethdev tracepoints optimization

[Jerin Jacob]

On Mon, Aug 19, 2024 at 4:13 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
>
> On 8/15/2024 8:32 PM, Adel Belkhiri wrote:
> > Hi DPDK Community,
> >
> > I am currently working on developing performance analyses for
> > applications using the ethdev library. These analyses are being
> > implemented in Trace Compass, an open-source performance analyzer. One
> > of the views I’ve implemented shows the rate of traffic received or sent
> > by an ethernet port, measured in packets per second. However, I've
> > encountered an issue with the lib.ethdev.rx.burst event, which triggers
> > even when no packets are polled, leading to a significant number of
> > irrelevant events in the trace. This becomes problematic as these
> > "empty" events can overwhelm the tracer buffer, potentially causing the
> > loss of more critical events due to their high frequency.
> >
> > To address this, I've modified the DPDK code in lib/ethdev/rte_ethdev.h
> > to add a conditional statement that only triggers the event when nb_rx >
> > 0. My question to the community is whether there are use cases where an
> > "empty" lib.ethdev.rx.burst event could be useful. If not, would there
> > be interest in submitting a patch with this modification?
> >
>
> Tracepoint is good way to get frequency of the calls, so I believe there
> can be value of getting empty burst calls too.
>
> But your usecase also a valid one. I wonder if it works to have separate
> trace calls, for empty and non-empty ones, and how this additional
> branching impacts the performance, at least branch should be wrapped
> with 'RTE_ENABLE_TRACE_FP' macro to not impact non-tracing usage.


CTF(Common trace format) post processing tools can check the value for
each field and timestap.
So it will easy to skip the ones with no packet for this case.


>

Re: Ethdev tracepoints optimization

[Bruce Richardson]

On Mon, Aug 19, 2024 at 05:07:18PM +0530, Jerin Jacob wrote:
> On Mon, Aug 19, 2024 at 4:13 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> >
> > On 8/15/2024 8:32 PM, Adel Belkhiri wrote:
> > > Hi DPDK Community,
> > >
> > > I am currently working on developing performance analyses for
> > > applications using the ethdev library. These analyses are being
> > > implemented in Trace Compass, an open-source performance analyzer. One
> > > of the views I’ve implemented shows the rate of traffic received or sent
> > > by an ethernet port, measured in packets per second. However, I've
> > > encountered an issue with the lib.ethdev.rx.burst event, which triggers
> > > even when no packets are polled, leading to a significant number of
> > > irrelevant events in the trace. This becomes problematic as these
> > > "empty" events can overwhelm the tracer buffer, potentially causing the
> > > loss of more critical events due to their high frequency.
> > >
> > > To address this, I've modified the DPDK code in lib/ethdev/rte_ethdev.h
> > > to add a conditional statement that only triggers the event when nb_rx >
> > > 0. My question to the community is whether there are use cases where an
> > > "empty" lib.ethdev.rx.burst event could be useful. If not, would there
> > > be interest in submitting a patch with this modification?
> > >
> >
> > Tracepoint is good way to get frequency of the calls, so I believe there
> > can be value of getting empty burst calls too.
> >
> > But your usecase also a valid one. I wonder if it works to have separate
> > trace calls, for empty and non-empty ones, and how this additional
> > branching impacts the performance, at least branch should be wrapped
> > with 'RTE_ENABLE_TRACE_FP' macro to not impact non-tracing usage.
> 
> 
> CTF(Common trace format) post processing tools can check the value for
> each field and timestap.
> So it will easy to skip the ones with no packet for this case.
>
The trouble with empty polls is that they are very, very fast, so if no
traffic is arriving you can quickly fill your entire trace buffer with
nothing but empty polls. I believe that is the situation the original
poster wanted to avoid.

/Bruce 

RE: Ethdev tracepoints optimization

[Morten Brørup]

> From: Jerin Jacob [mailto:jerinjacobk@gmail.com]
> 
> On Mon, Aug 19, 2024 at 4:13 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> >
> > On 8/15/2024 8:32 PM, Adel Belkhiri wrote:
> > > Hi DPDK Community,
> > >
> > > I am currently working on developing performance analyses for
> > > applications using the ethdev library. These analyses are being
> > > implemented in Trace Compass, an open-source performance analyzer. One
> > > of the views I’ve implemented shows the rate of traffic received or sent
> > > by an ethernet port, measured in packets per second. However, I've
> > > encountered an issue with the lib.ethdev.rx.burst event, which triggers
> > > even when no packets are polled, leading to a significant number of
> > > irrelevant events in the trace. This becomes problematic as these
> > > "empty" events can overwhelm the tracer buffer, potentially causing the
> > > loss of more critical events due to their high frequency.
> > >
> > > To address this, I've modified the DPDK code in lib/ethdev/rte_ethdev.h
> > > to add a conditional statement that only triggers the event when nb_rx >
> > > 0. My question to the community is whether there are use cases where an
> > > "empty" lib.ethdev.rx.burst event could be useful. If not, would there
> > > be interest in submitting a patch with this modification?
> > >
> >
> > Tracepoint is good way to get frequency of the calls, so I believe there
> > can be value of getting empty burst calls too.

Agree. The trace cannot generally omit empty burst calls.

> >
> > But your usecase also a valid one. I wonder if it works to have separate
> > trace calls, for empty and non-empty ones, and how this additional
> > branching impacts the performance, at least branch should be wrapped
> > with 'RTE_ENABLE_TRACE_FP' macro to not impact non-tracing usage.
> 
> 
> CTF(Common trace format) post processing tools can check the value for
> each field and timestap.
> So it will easy to skip the ones with no packet for this case.

Doesn't solve the issue of the tracer itself being overwhelmed.

Although I like Bruce's suggestion, which somewhat resembles Linux Kernel logging, I fear that it might be incompatible with automated tools processing the trace files afterwards.

I think Ferruh's suggestion should work:

Replace the current trace point with two separate trace points for respectively empty and non-empty calls:

RTE_TRACE_POINT_REGISTER(rte_ethdev_trace_rx_burst_empty,
	lib.ethdev.rx.burst.empty)

RTE_TRACE_POINT_REGISTER(rte_ethdev_trace_rx_burst_nonempty,
	lib.ethdev.rx.burst.nonempty)

And update the code in lib/ethdev/rte_ethdev.h to call the appropriate of the two functions depending on nb_rx.


PS: Something similar could be added to the eventdev library's rte_event_dequeue_burst() function. But that's a task for another day. :-)