From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8135FA0553; Mon, 17 Feb 2020 11:24:11 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 819DF1DA07; Mon, 17 Feb 2020 11:24:10 +0100 (CET) Received: from mail-io1-f66.google.com (mail-io1-f66.google.com [209.85.166.66]) by dpdk.org (Postfix) with ESMTP id 1E5161D561; Mon, 17 Feb 2020 11:24:09 +0100 (CET) Received: by mail-io1-f66.google.com with SMTP id x1so6413885iop.7; Mon, 17 Feb 2020 02:24:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=4ewA9bfmMp1eTL5Z3QvflmXMe49T9OIKYKnjw3OnErQ=; b=VJNNYi5bYE/dh58hX4a24aF7RH7OiznVXzPQQu61qZT7cUXov3+2/ToPNzou1Lfdgc 6LWgzcTzeMVJK5BuH9oYVEiPkWxJr+B3Vx6gXHMJ+0Nn/ntIKdPg+VGCMCWW2Ayn9xGm UC/LAhEyH/+QfeSwk7srnqqcu/jYmhnbb74ZU77EP2R+AvHZsUYsgn4bP/hk32oOuj+m NP+U/wtB3YaZNf6sILkOhBhNC7xxWGfSJZ68athbua7CNQtEZAvhbXej+t/KWLfObjbo xeREXfG/0Jwyug7yrCgPj0EIY36yjwVvg5bKS+PPEpkNndaox8DuALuz8/vZsDExmATE HvCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=4ewA9bfmMp1eTL5Z3QvflmXMe49T9OIKYKnjw3OnErQ=; b=JCEKoPFY+vWdnDbHlPkgNhd3RAKHyYENLDucgfevg34Mj9tO/IqNq4PW14LPeOXfEB n0o1ZV+GakY3LQ4rfFnu5IA/Fg0CQm73NkQ4iMr3f9xagkQK9XOlWzvpmNWCWsUCmiuF itrBTjP0AVWplZX2yTs/C70AIE4/r7DqSum2Gjkq1cCTDV/K0+qT/ecFlZuD/VoUhE0d bpqFqOs399g2JkLjsLf9pmvED5da4gb5AT96uKvn+uyD7xvgE2kY+ee3XeQbN6RkM8Ez 4zDzVQCVO1+EvZ0uEYuD8IWJj43J0JC+vnc+q8lyDcGlmwSaK2Yq5Zg2gSCQBAXkSVtD 8Iww== X-Gm-Message-State: APjAAAXA+h58+V5pxW/iO/IaVv7uZAGFLsZZVmfu+AZmqK3pwBcPiETF BuzEKzXSEjo0naVZRld6y37XzqTz3DMj+Z0AhmI= X-Google-Smtp-Source: APXvYqykLEAtwQ4VPeyfRZ7cLL9Bd50s84fOVM6GBrB3hhJUsEfMfd46SZX1OlHovcl6hVZu/rdgb9oiIMkx9WSHvkQ= X-Received: by 2002:a02:6f1b:: with SMTP id x27mr11924733jab.112.1581935048003; Mon, 17 Feb 2020 02:24:08 -0800 (PST) MIME-Version: 1.0 References: <20200113130543.GC1645@bricha3-MOBL.ger.corp.intel.com> <20200113145823.GD1645@bricha3-MOBL.ger.corp.intel.com> <20200113161259.GE1645@bricha3-MOBL.ger.corp.intel.com> <8d6a08a0-caad-64b1-b618-66c89c40fa62@ericsson.com> In-Reply-To: <8d6a08a0-caad-64b1-b618-66c89c40fa62@ericsson.com> From: Jerin Jacob Date: Mon, 17 Feb 2020 15:53:51 +0530 Message-ID: To: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= Cc: David Marchand , Bruce Richardson , Jerin Jacob Kollanukkaran , "dev@dpdk.org" , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , Ajit Khaparde , Qi Zhang , Xiaolong Ye , Raslan Darawsheh , Maxime Coquelin , Tiwei Bie , Akhil Goyal , Luca Boccassi , Kevin Traynor , "maintainers@dpdk.org" , John McNamara , Marko Kovacevic , Ray Kinsella , Aaron Conole , Michael Santana , Harry van Haaren , Cristian Dumitrescu , Phil Yang , Joyce Kong , Jan Viktorin , Gavin Hu , David Christensen , Konstantin Ananyev , Anatoly Burakov , Harini Ramakrishnan , Omar Cardona , Anand Rawat , Olivier Matz , Gage Eads , Adrien Mazarguil , Nicolas Chautru , Declan Doherty , Fiona Trahe , Ashish Gupta , Erik Gabriel Carrillo , Abhinandan Gujjar , Hemant Agrawal , "Artem V. Andreev" , Nithin Kumar Dabilpuram , Vamsi Krishna Attunuru , Rosen Xu , Sachin Saxena , Stephen Hemminger , Chas Williams , "John W. Linville" , Prasun Kapoor , Marcin Wojtas , Michal Krawczyk , Guy Tzalik , Evgeny Schemeilin , Igor Chauskin , Ravi Kumar , Igor Russkikh , Pavel Belous , Shepard Siegel , Ed Czeck , John Miller , Somnath Kotur , Maciej Czekaj , Shijith Thotton , Srisivasubramanian Srinivasan , Rahul Lakkireddy , John Daley , Hyong Youb Kim , "Wei Hu (Xavier" , "Min Hu (Connor" , Yisen Zhuang , Ziyang Xuan , Xiaoyun Wang , Guoyang Zhou , Beilei Xing , Xiao Wang , Jingjing Wu , Wenzhuo Lu , Qiming Yang , Tomasz Duszynski , Liron Himi , Zyta Szpak , Kiran Kumar Kokkilagadda , Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko , "K. Y. Srinivasan" , Haiyang Zhang , Jan Remes , Heinrich Kuhn , Jan Gutter , Gagandeep Singh , Rasesh Mody , Shahed Shaikh , Yong Wang , Zhihong Wang , Steven Webster , Matt Peters , Keith Wiles , Tetsuya Mukawa , Jasvinder Singh , Jakub Grajciar , Ruifeng Wang , Anoob Joseph , Fan Zhang , Pablo de Lara , John Griffin , Deepak Kumar Jain , Michael Shamis , Nagadheeraj Rottela , Srikanth Jampala , Ankur Dwivedi , Jay Zhou , Lee Daly , Sunila Sahu , Nipun Gupta , Liang Ma , Peter Mccarthy , Tianfei zhang , Satha Koteswara Rao Kottidi , Xiaoyun Li , Bernard Iremonger , Vladimir Medvedkin , David Hunt , Reshma Pattan , Byron Marohn , Sameh Gobriel , Yipeng Wang , Honnappa Nagarahalli , Robert Sanford , Kevin Laatz , Maryam Tahhan , Ori Kam , Radu Nicolau , Tomasz Kantecki , Sunil Kumar Kori , Pavan Nikhilesh Bhagavatula , Kirill Rybalchenko , "Kadam, Pallavi" , "dave@barachs.net" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC] DPDK Trace support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, Feb 17, 2020 at 3:05 PM Mattias R=C3=B6nnblom wrote: > > On 2020-02-15 11:21, Jerin Jacob wrote: > > On Fri, Jan 17, 2020 at 4:24 PM Jerin Jacob wro= te: > >> On Fri, Jan 17, 2020 at 4:00 PM Mattias R=C3=B6nnblom > >> wrote: > >>>> LTTng kernel tracing only needs kmod support. > >>>> For the userspace tracing at minium following libraries are required= . > >>>> > >>>> a) LTTng-UST > >>>> b) LTTng-tools > >>>> c) liburcu > >>>> d) libpopt-dev > >>> This "DPDK CTF trace emitter" would make DPDK interoperate with, but > >>> without any build-time dependencies to, LTTng. Correct? > >> Yes. Native CTF trace emitter without LTTng dependency. > >> > >>> Do you have any idea of what the performance benefits one would recei= ve > >>> from having something DPDK native, compared to just depending on LTTn= g UST? > >> I calibrated LTTng cost and pushed the test code to github[1]. > >> > >> I just started working on the DPDK native CTF emitter. > >> I am sure it will be less than LTTng as we can leverage hugepage, expl= oit > >> dpdk worker thread usage to avoid atomics and use per core variables, > >> avoid a lot function pointers in fast-path etc > >> I can share the exact overhead after the PoC. > > I did the almost feature-complete PoC. The code shared in Github[1] > > The documentation and code cleanup etc is still pending. > > > > [1] > > https://protect2.fireeye.com/v1/url?k=3D74b2fae0-283b556d-74b2ba7b-0cc4= 7ad93de2-2bd7c54f29450187&q=3D1&e=3D2ef0a614-dea6-4d17-ab4e-a79a7c17ac73&u= =3Dhttps%3A%2F%2Fgithub.com%2Fjerinjacobk%2Fdpdk-trace.git > > > > trace overhead data on x86:[2] > > # 236 cyles with LTTng(>100ns) > > # 18 cycles(7ns) with Native DPDK CTF emitter. > > > > trace overhead data on arm64: > > # 312 cycles to 1100 cycles with LTTng based on the class of arm64 C= PU. > > # 11 cycles to 13 cycles with Native DPDK CTF emitter based on the > > class of arm64 CPU. > > > > 18 cycles(on x86) vs 11 cycles(on arm64) is due to rdtsc() overhead in > > x86. It seems rdtsc takes around 15cycles in x86. > > > > # The Native DPDK CTF trace support does not have any dependency on > > third-party library. > > The generated output file is compatible with LTTng as both are using > > CTF trace format. > > > > The performance gain comes from: > > 1) exploit dpdk worker thread usage model to avoid atomics and use per > > core variables > > 2) use hugepage, > > 3) avoid a lot function pointers in fast-path etc > > 4) avoid unaligned store for arm64 etc > > > > Features: > > - APIs and Features are similar to rte_log dynamic framework > > API(expect log prints on stdout vs it dumps on trace file) > > - No specific limit on the events. A string-based event like rte_log > > for pattern matching > > - Dynamic enable/disable support. > > - Instructmention overhead is ~1 cycle. i.e cost of adding the code > > wth out using trace feature. > > - Timestamp support for all the events using DPDK rte_rtdsc > > - No dependency on another library. Cleanroom native implementation of = CTF. > > > > Functional test case: > > a) echo "trace_autotest" | sudo ./build/app/test/dpdk-test -c 0x3 > > --trace-level=3D8 > > > > The above command emits the following trace events > > > > uint8_t i; > > > > rte_trace_lib_eal_generic_void(); > > rte_trace_lib_eal_generic_u64(0x10000000000000); > > rte_trace_lib_eal_generic_u32(0x10000000); > > rte_trace_lib_eal_generic_u16(0xffee); > > rte_trace_lib_eal_generic_u8(0xc); > > rte_trace_lib_eal_generic_i64(-1234); > > rte_trace_lib_eal_generic_i32(-1234567); > > rte_trace_lib_eal_generic_i16(12); > > rte_trace_lib_eal_generic_i8(-3); > > rte_trace_lib_eal_generic_string("my string"); > > rte_trace_lib_eal_generic_function(__func__); > > > > for (i =3D 0; i < 128; i++) > > rte_trace_lib_eal_generic_u8(i); > > > > Is it possible to specify custom types for the events? The equivalent of > the TRACEPOINT_EVENT() macro in LTTng. Yes. It is possble to specify the custom event using array of rte_trace_emit_datatype of rte_trace_emit_string basic blocks. For example, ethdev configure event would be like. RTE_TRACE_POINT_DECLARE(__rte_trace_lib_ethdev_configure); static __rte_always_inline void rte_trace_lib_ethdev_configure(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q, const struct rte_eth_conf *dev_conf) { rte_trace_emit_begin(&__rte_trace_lib_ethdev_configure); rte_trace_emit_datatype(mem, port_id); rte_trace_emit_datatype(mem, nb_rx_q); rte_trace_emit_datatype(mem, nb_tx_q); rte_trace_emit_datatype(mem, dev_conf->link_speeds); .. .. } I tried avoid usage of complex macro as DPDK community prefers non macro solutions. > > > Install babeltrace package in Linux and point the generated trace file > > to babel trace. By default trace file created under > > /dpdk-traces/time_stamp/ > > > > example: > > # babeltrace /root/dpdk-traces/rte-2020-02-15-PM-02-56-51 | more > > > > [13:27:36.138468807] (+?.?????????) lib.eal.generic.void: { cpu_id =3D > > 0, name =3D "dpdk-test" }, { } > > [13:27:36.138468851] (+0.000000044) lib.eal.generic.u64: { cpu_id =3D 0= , > > name =3D "dpdk-test" }, { in =3D 4503599627370496 } > > [13:27:36.138468860] (+0.000000009) lib.eal.generic.u32: { cpu_id =3D 0= , > > name =3D "dpdk-test" }, { in =3D 268435456 } > > [13:27:36.138468934] (+0.000000074) lib.eal.generic.u16: { cpu_id =3D 0= , > > name =3D "dpdk-test" }, { in =3D 65518 } > > [13:27:36.138468949] (+0.000000015) lib.eal.generic.u8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D 12 } > > [13:27:36.138468956] (+0.000000007) lib.eal.generic.i64: { cpu_id =3D 0= , > > name =3D "dpdk-test" }, { in =3D -1234 } > > [13:27:36.138468963] (+0.000000007) lib.eal.generic.i32: { cpu_id =3D 0= , > > name =3D "dpdk-test" }, { in =3D -1234567 } > > [13:27:36.138469024] (+0.000000061) lib.eal.generic.i16: { cpu_id =3D 0= , > > name =3D "dpdk-test" }, { in =3D 12 } > > [13:27:36.138469044] (+0.000000020) lib.eal.generic.i8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D -3 } > > [13:27:36.138469051] (+0.000000007) lib.eal.generic.string: { cpu_id = =3D > > 0, name =3D "dpdk-test" }, { str =3D "my string" } > > [13:27:36.138469203] (+0.000000152) lib.eal.generic.func: { cpu_id =3D > > 0, name =3D "dpdk-test" }, { func =3D "test_trace_points" } > > [13:27:36.138469239] (+0.000000036) lib.eal.generic.u8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D 0 } > > [13:27:36.138469246] (+0.000000007) lib.eal.generic.u8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D 1 } > > [13:27:36.138469252] (+0.000000006) lib.eal.generic.u8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D 2 } > > [13:27:36.138469262] (+0.000000010) lib.eal.generic.u8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D 3 } > > [13:27:36.138469269] (+0.000000007) lib.eal.generic.u8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D 4 } > > [13:27:36.138469276] (+0.000000007) lib.eal.generic.u8: { cpu_id =3D 0, > > name =3D "dpdk-test" }, { in =3D 5 } > > > > # There is a GUI based trace viewer available in Windows, Linux and Ma= c. > > It is called as tracecompass.(https://www.eclipse.org/tracecompass/) > > > > The example screenshot and Histogram of above DPDK trace using Tracecom= pass. > > > > https://protect2.fireeye.com/v1/url?k=3D9cad9dc9-c0243244-9caddd52-0cc4= 7ad93de2-306d4b4409146643&q=3D1&e=3D2ef0a614-dea6-4d17-ab4e-a79a7c17ac73&u= =3Dhttps%3A%2F%2Fgithub.com%2Fjerinjacobk%2Fshare%2Fblob%2Fmaster%2Fdpdk_tr= ace.JPG > > > > > > [2] Added performance test case to find the Trace overhead. > > Command to test trace overhead. It is the overhead of writing > > zero-argument trace. > > > > echo "trace_perf" | sudo ./build/app/test/dpdk-test -c 0x3 --trace-lev= el=3D8 > > EAL: Detected 56 lcore(s) > > EAL: Detected 2 NUMA nodes > > EAL: Trace dir: /root/dpdk-traces/rte-2020-02-15-PM-03-37-33 > > RTE>>trace_perf > > Timer running at 2600.00MHz > > ZERO_ARG: cycles=3D17.901031 ns=3D6.885012 > > Test OK > > > > [3] The above test is ported to LTTng for finding the LTTng trace > > overhead. It available at > > https://protect2.fireeye.com/v1/url?k=3D7eb42ff5-223d8078-7eb46f6e-0cc4= 7ad93de2-e41c22a09211c207&q=3D1&e=3D2ef0a614-dea6-4d17-ab4e-a79a7c17ac73&u= =3Dhttps%3A%2F%2Fgithub.com%2Fjerinjacobk%2Flttng-overhead > > https://protect2.fireeye.com/v1/url?k=3D616a430a-3de3ec87-616a0391-0cc4= 7ad93de2-c75160108b40b11b&q=3D1&e=3D2ef0a614-dea6-4d17-ab4e-a79a7c17ac73&u= =3Dhttps%3A%2F%2Fgithub.com%2Fjerinjacobk%2Flttng-overhead%2Fblob%2Fmaster%= 2FREADME > > > > File walkthrough: > > > > lib/librte_eal/common/include/rte_trace.h - Public API for Trace > > provider and Trace control > > lib/librte_eal/common/eal_common_trace.c - main trace implemention > > lib/librte_eal/common/eal_common_trace_ctf.c - CTF metadata spec implem= entation > > lib/librte_eal/common/eal_common_trace_utils.c - command line utils > > and filesystem operations. > > lib/librte_eal/common/eal_common_trace_points.c - trace points for EAL= library > > lib/librte_eal/common/include/rte_trace_eal.h - EAL tracepoint public A= PI. > > lib/librte_eal/common/eal_trace.h - Private trace header file. > > > > > >> I think, based on the performance we can decide one or another? > > The above performance data shows a much higher improvement with LTTng. > > > > Let me know if anyone have any comments on this and or any suggestion. > > If there are no comments, I will submit the v1 with after code clean > > up before the 20.05 proposal deadline. >