From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 92319A054E; Sat, 15 Feb 2020 11:22:12 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 61C994C99; Sat, 15 Feb 2020 11:22:12 +0100 (CET) Received: from mail-il1-f195.google.com (mail-il1-f195.google.com [209.85.166.195]) by dpdk.org (Postfix) with ESMTP id 6DA94DE3; Sat, 15 Feb 2020 11:22:10 +0100 (CET) Received: by mail-il1-f195.google.com with SMTP id v13so10241489iln.4; Sat, 15 Feb 2020 02:22:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=B8U3s/wxa88WPyRwic7wQhbTvaKVXzEvAI+L4tW0JZI=; b=qo/Ys05uc1d3r3/OLNg55lnrEd9t8pgK+Pj7Q+y/oRX60OKV2KVxtXj/84kUZY7Rop YX0XO7OaJAMIABSDstzaPuBVPhZOkIFZ7DhMM2VZy6dKy48laJChPVAc++dywSChsEyg dNtQzwXGYfMCFYxiKBVYHw2X3qTALZadj8/9c0MWB97O9Te7NuLE3wQ0ILWGQ0i7e9ZB DoAuIMvVpTj/tb0OLj2t+OIthY2l5ExLxls1jSUCTils9MooM1LioL7oYUKa0rz5DFFH NXwTiwbSQtfJ9q3oGaS/2s2LR1xTtNXf9HAK/2CSjYT/eV93fdlLlF8gzsLcmxxVdQ86 YqOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=B8U3s/wxa88WPyRwic7wQhbTvaKVXzEvAI+L4tW0JZI=; b=J0bIrtuIZZ7MlWpG0KZ6QjYScZ7yYYlvqP6B1CeQTD6BdJRySVq8R3bg3ngZajPxel XLBe0GM1zBbdRUaBv39MsRJ8izU1WikUjrxabHI+RlUOI2I02/r/hfyaGA8mkZDRb+Rt h4D3qnedkWRmsY/SG+nu661OG8Ck8MbrWWQgtwQA5QFooAnxrZiY5OEHJQJajQ9xp2xU xNLpw9uAbPil7DZ7HcAdScWGx+VCZtXQfBps4yEZbpnyKjNVHTreYH7sjTLwp3J3EXPF T0eKY5C+NrIokgBIueJ00z7lzF41ZBvUeDqngegiYMIYjJJgShANo6yt4rFJozCVl0KH rOsQ== X-Gm-Message-State: APjAAAV17a8SRH6rlMmjhqKTCTGOiDLV3AyhsgSUhsiL8wknrRa/HSTo rD5kQzSyfyT3uZ6GC4Re52pNWmTVqk1mjaAM6zc= X-Google-Smtp-Source: APXvYqwpGD+cTvSlndWjd7bikJ1AqYvuZxyECvrNPm2IvDIETIztwW8Il3b2EfBkG/4LHK2SmJnm41R3pPBOzssgLl8= X-Received: by 2002:a05:6e02:f47:: with SMTP id y7mr6877007ilj.162.1581762129605; Sat, 15 Feb 2020 02:22:09 -0800 (PST) MIME-Version: 1.0 References: <20200113130543.GC1645@bricha3-MOBL.ger.corp.intel.com> <20200113145823.GD1645@bricha3-MOBL.ger.corp.intel.com> <20200113161259.GE1645@bricha3-MOBL.ger.corp.intel.com> In-Reply-To: From: Jerin Jacob Date: Sat, 15 Feb 2020 15:51:53 +0530 Message-ID: To: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= Cc: David Marchand , Bruce Richardson , Jerin Jacob Kollanukkaran , "dev@dpdk.org" , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , Ajit Khaparde , Qi Zhang , Xiaolong Ye , Raslan Darawsheh , Maxime Coquelin , Tiwei Bie , Akhil Goyal , Luca Boccassi , Kevin Traynor , "maintainers@dpdk.org" , John McNamara , Marko Kovacevic , Ray Kinsella , Aaron Conole , Michael Santana , Harry van Haaren , Cristian Dumitrescu , Phil Yang , Joyce Kong , Jan Viktorin , Gavin Hu , David Christensen , Konstantin Ananyev , Anatoly Burakov , Harini Ramakrishnan , Omar Cardona , Anand Rawat , Olivier Matz , Gage Eads , Adrien Mazarguil , Nicolas Chautru , Declan Doherty , Fiona Trahe , Ashish Gupta , Erik Gabriel Carrillo , Abhinandan Gujjar , Hemant Agrawal , "Artem V. Andreev" , Nithin Kumar Dabilpuram , Vamsi Krishna Attunuru , Rosen Xu , Sachin Saxena , Stephen Hemminger , Chas Williams , "John W. Linville" , Prasun Kapoor , Marcin Wojtas , Michal Krawczyk , Guy Tzalik , Evgeny Schemeilin , Igor Chauskin , Ravi Kumar , Igor Russkikh , Pavel Belous , Shepard Siegel , Ed Czeck , John Miller , Somnath Kotur , Maciej Czekaj , Shijith Thotton , Srisivasubramanian Srinivasan , Rahul Lakkireddy , John Daley , Hyong Youb Kim , "Wei Hu (Xavier" , "Min Hu (Connor" , Yisen Zhuang , Ziyang Xuan , Xiaoyun Wang , Guoyang Zhou , Beilei Xing , Xiao Wang , Jingjing Wu , Wenzhuo Lu , Qiming Yang , Tomasz Duszynski , Liron Himi , Zyta Szpak , Kiran Kumar Kokkilagadda , Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko , "K. Y. Srinivasan" , Haiyang Zhang , Jan Remes , Heinrich Kuhn , Jan Gutter , Gagandeep Singh , Rasesh Mody , Shahed Shaikh , Yong Wang , Zhihong Wang , Steven Webster , Matt Peters , Keith Wiles , Tetsuya Mukawa , Jasvinder Singh , Jakub Grajciar , Ruifeng Wang , Anoob Joseph , Fan Zhang , Pablo de Lara , John Griffin , Deepak Kumar Jain , Michael Shamis , Nagadheeraj Rottela , Srikanth Jampala , Ankur Dwivedi , Jay Zhou , Lee Daly , Sunila Sahu , Nipun Gupta , Liang Ma , Peter Mccarthy , Tianfei zhang , Satha Koteswara Rao Kottidi , Xiaoyun Li , Bernard Iremonger , Vladimir Medvedkin , David Hunt , Reshma Pattan , Byron Marohn , Sameh Gobriel , Yipeng Wang , Honnappa Nagarahalli , Robert Sanford , Kevin Laatz , Maryam Tahhan , Ori Kam , Radu Nicolau , Tomasz Kantecki , Sunil Kumar Kori , Pavan Nikhilesh Bhagavatula , Kirill Rybalchenko , "Kadam, Pallavi" , dave@barachs.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC] DPDK Trace support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, Jan 17, 2020 at 4:24 PM Jerin Jacob wrote: > > On Fri, Jan 17, 2020 at 4:00 PM Mattias R=C3=B6nnblom > wrote: > > > > > > LTTng kernel tracing only needs kmod support. > > > For the userspace tracing at minium following libraries are required. > > > > > > a) LTTng-UST > > > b) LTTng-tools > > > c) liburcu > > > d) libpopt-dev > > > > This "DPDK CTF trace emitter" would make DPDK interoperate with, but > > without any build-time dependencies to, LTTng. Correct? > > Yes. Native CTF trace emitter without LTTng dependency. > > > > > Do you have any idea of what the performance benefits one would receive > > from having something DPDK native, compared to just depending on LTTng = UST? > > I calibrated LTTng cost and pushed the test code to github[1]. > > I just started working on the DPDK native CTF emitter. > I am sure it will be less than LTTng as we can leverage hugepage, exploit > dpdk worker thread usage to avoid atomics and use per core variables, > avoid a lot function pointers in fast-path etc > I can share the exact overhead after the PoC. I did the almost feature-complete PoC. The code shared in Github[1] The documentation and code cleanup etc is still pending. [1] https://github.com/jerinjacobk/dpdk-trace.git trace overhead data on x86:[2] # 236 cyles with LTTng(>100ns) # 18 cycles(7ns) with Native DPDK CTF emitter. trace overhead data on arm64: # 312 cycles to 1100 cycles with LTTng based on the class of arm64 CPU. # 11 cycles to 13 cycles with Native DPDK CTF emitter based on the class of arm64 CPU. 18 cycles(on x86) vs 11 cycles(on arm64) is due to rdtsc() overhead in x86. It seems rdtsc takes around 15cycles in x86. # The Native DPDK CTF trace support does not have any dependency on third-party library. The generated output file is compatible with LTTng as both are using CTF trace format. The performance gain comes from: 1) exploit dpdk worker thread usage model to avoid atomics and use per core variables 2) use hugepage, 3) avoid a lot function pointers in fast-path etc 4) avoid unaligned store for arm64 etc Features: - APIs and Features are similar to rte_log dynamic framework API(expect log prints on stdout vs it dumps on trace file) - No specific limit on the events. A string-based event like rte_log for pattern matching - Dynamic enable/disable support. - Instructmention overhead is ~1 cycle. i.e cost of adding the code wth out using trace feature. - Timestamp support for all the events using DPDK rte_rtdsc - No dependency on another library. Cleanroom native implementation of CTF. Functional test case: a) echo "trace_autotest" | sudo ./build/app/test/dpdk-test -c 0x3 --trace-level=3D8 The above command emits the following trace events uint8_t i; rte_trace_lib_eal_generic_void(); rte_trace_lib_eal_generic_u64(0x10000000000000); rte_trace_lib_eal_generic_u32(0x10000000); rte_trace_lib_eal_generic_u16(0xffee); rte_trace_lib_eal_generic_u8(0xc); rte_trace_lib_eal_generic_i64(-1234); rte_trace_lib_eal_generic_i32(-1234567); rte_trace_lib_eal_generic_i16(12); rte_trace_lib_eal_generic_i8(-3); rte_trace_lib_eal_generic_string("my string"); rte_trace_lib_eal_generic_function(__func__); for (i =3D 0; i < 128; i++) rte_trace_lib_eal_generic_u8(i); Install babeltrace package in Linux and point the generated trace file to babel trace. By default trace file created under /dpdk-traces/time_stamp/ example: # babeltrace /root/dpdk-traces/rte-2020-02-15-PM-02-56-51 | more [13:27:36.138468807] (+?.?????????) lib.eal.generic.void: { cpu_id =3D 0, name =3D "dpdk-test" }, { } [13:27:36.138468851] (+0.000000044) lib.eal.generic.u64: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 4503599627370496 } [13:27:36.138468860] (+0.000000009) lib.eal.generic.u32: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 268435456 } [13:27:36.138468934] (+0.000000074) lib.eal.generic.u16: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 65518 } [13:27:36.138468949] (+0.000000015) lib.eal.generic.u8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 12 } [13:27:36.138468956] (+0.000000007) lib.eal.generic.i64: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D -1234 } [13:27:36.138468963] (+0.000000007) lib.eal.generic.i32: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D -1234567 } [13:27:36.138469024] (+0.000000061) lib.eal.generic.i16: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 12 } [13:27:36.138469044] (+0.000000020) lib.eal.generic.i8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D -3 } [13:27:36.138469051] (+0.000000007) lib.eal.generic.string: { cpu_id =3D 0, name =3D "dpdk-test" }, { str =3D "my string" } [13:27:36.138469203] (+0.000000152) lib.eal.generic.func: { cpu_id =3D 0, name =3D "dpdk-test" }, { func =3D "test_trace_points" } [13:27:36.138469239] (+0.000000036) lib.eal.generic.u8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 0 } [13:27:36.138469246] (+0.000000007) lib.eal.generic.u8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 1 } [13:27:36.138469252] (+0.000000006) lib.eal.generic.u8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 2 } [13:27:36.138469262] (+0.000000010) lib.eal.generic.u8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 3 } [13:27:36.138469269] (+0.000000007) lib.eal.generic.u8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 4 } [13:27:36.138469276] (+0.000000007) lib.eal.generic.u8: { cpu_id =3D 0, name =3D "dpdk-test" }, { in =3D 5 } # There is a GUI based trace viewer available in Windows, Linux and Mac. It is called as tracecompass.(https://www.eclipse.org/tracecompass/) The example screenshot and Histogram of above DPDK trace using Tracecompass= . https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG [2] Added performance test case to find the Trace overhead. Command to test trace overhead. It is the overhead of writing zero-argument trace. echo "trace_perf" | sudo ./build/app/test/dpdk-test -c 0x3 --trace-level= =3D8 EAL: Detected 56 lcore(s) EAL: Detected 2 NUMA nodes EAL: Trace dir: /root/dpdk-traces/rte-2020-02-15-PM-03-37-33 RTE>>trace_perf Timer running at 2600.00MHz ZERO_ARG: cycles=3D17.901031 ns=3D6.885012 Test OK [3] The above test is ported to LTTng for finding the LTTng trace overhead. It available at https://github.com/jerinjacobk/lttng-overhead https://github.com/jerinjacobk/lttng-overhead/blob/master/README File walkthrough: lib/librte_eal/common/include/rte_trace.h - Public API for Trace provider and Trace control lib/librte_eal/common/eal_common_trace.c - main trace implemention lib/librte_eal/common/eal_common_trace_ctf.c - CTF metadata spec implementa= tion lib/librte_eal/common/eal_common_trace_utils.c - command line utils and filesystem operations. lib/librte_eal/common/eal_common_trace_points.c - trace points for EAL lib= rary lib/librte_eal/common/include/rte_trace_eal.h - EAL tracepoint public API. lib/librte_eal/common/eal_trace.h - Private trace header file. > I think, based on the performance we can decide one or another? The above performance data shows a much higher improvement with LTTng. Let me know if anyone have any comments on this and or any suggestion. If there are no comments, I will submit the v1 with after code clean up before the 20.05 proposal deadline.