From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A09E8A0525; Mon, 20 Jan 2020 08:42:27 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6C55F11A2; Mon, 20 Jan 2020 08:42:27 +0100 (CET) Received: from gateway32.websitewelcome.com (gateway32.websitewelcome.com [192.185.145.170]) by dpdk.org (Postfix) with ESMTP id 44F521BE8A for ; Sat, 18 Jan 2020 16:14:33 +0100 (CET) Received: from cm13.websitewelcome.com (cm13.websitewelcome.com [100.42.49.6]) by gateway32.websitewelcome.com (Postfix) with ESMTP id 48C8070330 for ; Sat, 18 Jan 2020 09:14:32 -0600 (CST) Received: from gator4196.hostgator.com ([108.167.189.21]) by cmsmtp with SMTP id spo0iQLORHiO0spo0iNb7S; Sat, 18 Jan 2020 09:14:32 -0600 X-Authority-Reason: nr=8 Received: from [173.38.117.91] (port=39137 helo=Win10) by gator4196.hostgator.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92) (envelope-from ) id 1ispnz-001pHh-TZ; Sat, 18 Jan 2020 09:14:32 -0600 From: To: "'Ray Kinsella'" , "'Jerin Jacob Kollanukkaran'" , "'dpdk-dev'" References: In-Reply-To: Date: Sat, 18 Jan 2020 10:14:31 -0500 Message-ID: <011001d5ce11$fc804bd0$f580e370$@barachs.net> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQFUldAPp7TuiiF1UjaOjokcgjEhPgDcTyWRqOwTgfA= Content-Language: en-us X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gator4196.hostgator.com X-AntiAbuse: Original Domain - dpdk.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - barachs.net X-BWhitelist: no X-Source-IP: 173.38.117.91 X-Source-L: No X-Exim-ID: 1ispnz-001pHh-TZ X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: (Win10) [173.38.117.91]:39137 X-Source-Auth: dave@barachs.net X-Email-Count: 3 X-Source-Cap: ZGJhcmFjaDtkYmFyYWNoO2dhdG9yNDE5Ni5ob3N0Z2F0b3IuY29t X-Local-Domain: yes X-Mailman-Approved-At: Mon, 20 Jan 2020 08:42:26 +0100 Subject: Re: [dpdk-dev] [RFC] DPDK Trace support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" It would be well worth considering one of the vpp techniques to minimize = trace impact: static inline ring_handler_inline (..., int is_traced) { for (i =3D 0; i < vector_size; i++) { if (is_traced) { do_trace_work; } normal_packet_processing; } } ring_handler (...) { if (PREDICT_FALSE(global_trace_flag !=3D 0)) return ring_handler_inline (..., 1 /* is_traced */); else return ring_handler_inline (..., 0 /* is_traced */); } This reduces the runtime tax to the absolute minimum, but costs space.=20 Please consider it. HTH... Dave -----Original Message----- From: Ray Kinsella =20 Sent: Monday, January 13, 2020 6:00 AM To: Jerin Jacob Kollanukkaran ; dpdk-dev = ; dave@barachs.net Subject: Re: [RFC] [dpdk-dev] DPDK Trace support Hi Jerin, Any idea why lttng performance is so poor? I would have naturally gone there to benefit from the existing = toolchain. Have you looked at the FD.io logging/tracing infrastructure for = inspiration? https://wiki.fd.io/view/VPP/elog Ray K On 13/01/2020 10:40, Jerin Jacob Kollanukkaran wrote: > Hi All, >=20 > I would like to add tracing support for DPDK. > I am planning to add this support in v20.05 release. >=20 > This RFC attempts to get feedback from the community on >=20 > a) Tracing Use cases. > b) Tracing Requirements. > b) Implementation choices. > c) Trace format. >=20 > Use-cases > --------- > - Most of the cases, The DPDK provider will not have access to the = DPDK customer applications. > To debug/analyze the slow path and fast path DPDK API usage from the=20 > field, we need to have integrated trace support in DPDK. >=20 > - Need a low overhead Fast path multi-core PMD driver=20 > debugging/analysis infrastructure in DPDK to fix the functional and = performance issue(s) of PMD. >=20 > - Post trace analysis tools can provide various status across the=20 > system such as cpu_idle() using the timestamp added in the trace. >=20 >=20 > Requirements: > ------------- > - Support for Linux, FreeBSD and Windows OS > - Open trace format > - Multi-platform Open source trace viewer > - Absolute low overhead trace API for DPDK fast path = tracing/debugging. > - Dynamic enable/disable of trace events >=20 >=20 > To enable trace support in DPDK, following items need to work out:=20 >=20 > a) Add the DPDK trace points in the DPDK source code. >=20 > - This includes updating DPDK functions such as,=20 > rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() = to emit the trace. >=20 > b) Choosing suitable serialization-format >=20 > - Common Trace Format, CTF, is an open format and language to describe = trace formats. > This enables tool reuse, of which line-textual (babeltrace) and=20 > graphical (TraceCompass) variants already exist. >=20 > CTF should look familiar to C programmers but adds stronger typing.=20 > See CTF - A Flexible, High-performance Binary Trace Format. >=20 > https://diamon.org/ctf/ >=20 > c) Writing the on-target serialization code, >=20 > See the section below.(Lttng CTF trace emitter vs DPDK specific CTF=20 > trace emitter) > =20 > d) Deciding on and writing the I/O transport mechanics, >=20 > For performance reasons, it should be backed by a huge-page and write = to file IO. >=20 > e) Writing the PC-side deserializer/parser, >=20 > Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF. > See:=20 > https://lttng.org/viewers/ >=20 > f) Writing tools for filtering and presentation. >=20 > See item (e) >=20 >=20 > Lttng CTF trace emitter vs DPDK specific CTF trace emitter > ---------------------------------------------------------- >=20 > I have written a performance evaluation application to measure the=20 > overhead of Lttng CTF emitter(The fastpath infrastructure used by=20 > https://lttng.org/ library to emit the trace) >=20 > https://github.com/jerinjacobk/lttng-overhead > https://github.com/jerinjacobk/lttng-overhead/blob/master/README >=20 > I could improve the performance by 30% by adding the "DPDK" > based plugin for get_clock() and get_cpu(), Here are the performance=20 > numbers after adding the plugin on > x86 and various arm64 board that I have access to, >=20 > On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the=20 > last line in the log(ZERO_ARG)) On arm64, it varies from 312 cycles to = 1100 cycles(based on the class of CPU). > In short, Based on the "IPC capabilities", The cost would be around=20 > 100ns to 400ns for single void trace(a trace without any argument) >=20 >=20 > [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0 > make[1]: Entering directory '/export/lttng-overhead-x86/calibrate' > make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate' > EAL: Detected 56 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'PA' > EAL: Probing VFIO support... > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > EAL: PCI device 0000:01:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > CPU Timer freq is 2600.000000MHz > NOP: cycles=3D0.194834 ns=3D0.074936 > GET_CLOCK: cycles=3D47.854658 ns=3D18.405638 > GET_CPU: cycles=3D30.995892 ns=3D11.921497 > ZERO_ARG: cycles=3D236.945113 ns=3D91.132736 >=20 >=20 > We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng=20 > CTF emitter may not fit the DPDK fast path purpose due to the cost = associated with generic Lttng features. >=20 > One option could be to have, native CTF emitter in EAL/DPDK to emit=20 > the trace in a hugepage. I think it would be a handful of cycles if we = > limit the features to the requirements above: >=20 > The upside of using Lttng CTF emitter: > a) No need to write a new CTF trace emitter(the item (c)) >=20 > The downside of Lttng CTF emitter(the item (c)) > a) performance issue(See above) > b) Lack of Windows OS support. It looks like, it has basic FreeBSD = support. > c) dpdk library dependency to lttng for trace. >=20 > So, Probably it good to have native CTF emitter in DPDK and reuse all=20 > open-source trace viewer(babeltrace and TraceCompass) and format(CTF) = infrastructure. > I think, it would be best of both world. >=20 > Any thoughts on this subject? Based on the community feedback, I can = work on the patch for v20.05. >=20