Re: [dpdk-dev] [RFC] DPDK Trace support

[Aaron Conole <aconole@redhat.com>]

Jerin Jacob Kollanukkaran <jerinj@marvell.com> writes:

> Hi All,
>
> I would like to add tracing support for DPDK.
> I am planning to add this support in v20.05 release.
>
> This RFC attempts to get feedback from the community on
>
> a) Tracing Use cases.
> b) Tracing Requirements.
> b) Implementation choices.
> c) Trace format.
>
> Use-cases
> ---------
> - Most of the cases, The DPDK provider will not have access to the DPDK customer applications.
> To debug/analyze the slow path and fast path DPDK API usage from the field,
> we need to have integrated trace support in DPDK.
>
> - Need a low overhead Fast path multi-core PMD driver debugging/analysis
> infrastructure in DPDK to fix the functional and performance issue(s) of PMD.
>
> - Post trace analysis tools can provide various status across the system such
> as cpu_idle() using the timestamp added in the trace.
>
>
> Requirements:
> -------------
> - Support for Linux, FreeBSD and Windows OS
> - Open trace format
> - Multi-platform Open source trace viewer
> - Absolute low overhead trace API for DPDK fast path tracing/debugging.
> - Dynamic enable/disable of trace events
>
>
> To enable trace support in DPDK, following items need to work out: 
>
> a) Add the DPDK trace points in the DPDK source code.
>
> - This includes updating DPDK functions such as,
> rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() to emit the trace.

I wonder for these if it makes sense to use librte_bpf and a helper
function to actually emit events.  That way rather than static trace
point data, a user can implement some C-code and pull the exact data
that they want.

There could be some downside with the approach (because we might lose
some inlining or variable eliding), but I think it makes the trace point
concept quite a bit more powerful.  Have you given it any thought?

> b) Choosing suitable serialization-format
>
> - Common Trace Format, CTF, is an open format and language to describe trace formats.
> This enables tool reuse, of which line-textual (babeltrace) and 
> graphical (TraceCompass) variants already exist.
>
> CTF should look familiar to C programmers but adds stronger typing. 
> See CTF - A Flexible, High-performance Binary Trace Format.
>
> https://diamon.org/ctf/
>
> c) Writing the on-target serialization code,
>
> See the section below.(Lttng CTF trace emitter vs DPDK specific CTF trace emitter)
>  
> d) Deciding on and writing the I/O transport mechanics,
>
> For performance reasons, it should be backed by a huge-page and write to file IO.
>
> e) Writing the PC-side deserializer/parser,
>
> Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF.
> See: 
> https://lttng.org/viewers/
>
> f) Writing tools for filtering and presentation.
>
> See item (e)
>
>
> Lttng CTF trace emitter vs DPDK specific CTF trace emitter
> ----------------------------------------------------------
>
> I have written a performance evaluation application to measure the overhead
> of Lttng CTF emitter(The fastpath infrastructure used by
> https://lttng.org/ library to emit the trace)
>
> https://github.com/jerinjacobk/lttng-overhead
> https://github.com/jerinjacobk/lttng-overhead/blob/master/README
>
> I could improve the performance by 30% by adding the "DPDK"
> based plugin for get_clock() and get_cpu(),
> Here are the performance numbers after adding the plugin on 
> x86 and various arm64 board that I have access to,
>
> On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the
> last line in the log(ZERO_ARG))
> On arm64, it varies from 312 cycles to 1100 cycles(based on the class of CPU).
> In short, Based on the "IPC capabilities", The cost would be around 100ns to 400ns
> for single void trace(a trace without any argument)
>
>
> [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0
> make[1]: Entering directory '/export/lttng-overhead-x86/calibrate'
> make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate'
> EAL: Detected 56 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: Probing VFIO support...
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> EAL: PCI device 0000:01:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> CPU Timer freq is 2600.000000MHz
> NOP: cycles=0.194834 ns=0.074936
> GET_CLOCK: cyclesG.854658 ns.405638
> GET_CPU: cycles0.995892 ns.921497
> ZERO_ARG: cycles#6.945113 ns‘.132736
>
>
> We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng CTF emitter
> may not fit the DPDK fast path purpose due to the cost associated with generic Lttng features.
>
> One option could be to have, native CTF emitter in EAL/DPDK to emit the
> trace in a hugepage. I think it would be a handful of cycles if we limit the features
> to the requirements above:
>
> The upside of using Lttng CTF emitter:
> a) No need to write a new CTF trace emitter(the item (c))
>
> The downside of Lttng CTF emitter(the item (c))
> a) performance issue(See above)
> b) Lack of Windows OS support. It looks like, it has basic FreeBSD support.
> c) dpdk library dependency to lttng for trace.
>
> So, Probably it good to have native CTF emitter in DPDK and reuse all
> open-source trace viewer(babeltrace and  TraceCompass) and format(CTF) infrastructure.
> I think, it would be best of both world.
>
> Any thoughts on this subject? Based on the community feedback, I can
> work on the patch for v20.05.