From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 45BDFA0531;
	Mon, 27 Jan 2020 17:13:29 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id CDE051BFB6;
	Mon, 27 Jan 2020 17:13:28 +0100 (CET)
Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com
 [207.211.31.81]) by dpdk.org (Postfix) with ESMTP id 6F7281BEA5
 for <dev@dpdk.org>; Mon, 27 Jan 2020 17:13:26 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
 s=mimecast20190719; t=1580141606;
 h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
 content-transfer-encoding:content-transfer-encoding:
 in-reply-to:in-reply-to:references:references;
 bh=B6H70nvMdKRtpG5WRcDjwtd+sq3IWJtFRIa5WPDfDHY=;
 b=eOLHjFrKiZSJJJKmGzzGtmgZE2hSnEjDVi3LzOmbEuc6fmfKW4YOkhqZioBAf9BxVlOw6E
 1IaemwgylfwANi1DETb4wdgCy16ARvkta74ErTuowMIgdFf2U2+0QW2vxWiq07mPL3gIta
 rlGleWHmtVtBzV8W6D1copIsNaRfbBY=
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com
 [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-275-bgQvnzJ_O8yOkVHir_XrxA-1; Mon, 27 Jan 2020 11:13:21 -0500
Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com
 [10.5.11.12])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0CD66801E67;
 Mon, 27 Jan 2020 16:13:14 +0000 (UTC)
Received: from dhcp-25.97.bos.redhat.com (unknown [10.18.25.126])
 by smtp.corp.redhat.com (Postfix) with ESMTPS id 2418560BFB;
 Mon, 27 Jan 2020 16:12:49 +0000 (UTC)
From: Aaron Conole <aconole@redhat.com>
To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
Cc: "dev\@dpdk.org" <dev@dpdk.org>,  Thomas Monjalon <thomas@monjalon.net>,
 David Marchand <david.marchand@redhat.com>,  Ferruh Yigit
 <ferruh.yigit@intel.com>,  "Andrew Rybchenko" <arybchenko@solarflare.com>,
 Ajit Khaparde <ajit.khaparde@broadcom.com>,  Qi Zhang
 <qi.z.zhang@intel.com>,  Xiaolong Ye <xiaolong.ye@intel.com>,  Raslan
 Darawsheh <rasland@mellanox.com>,  Maxime Coquelin
 <maxime.coquelin@redhat.com>,  Tiwei Bie <tiwei.bie@intel.com>,  Akhil
 Goyal <akhil.goyal@nxp.com>,  "Luca Boccassi" <bluca@debian.org>,  Kevin
 Traynor <ktraynor@redhat.com>,  "maintainers\@dpdk.org"
 <maintainers@dpdk.org>,  John McNamara <john.mcnamara@intel.com>,  Marko
 Kovacevic <marko.kovacevic@intel.com>,  "Ray Kinsella" <mdr@ashroe.eu>,
 Bruce Richardson <bruce.richardson@intel.com>,  Aaron Conole
 <aconole@redhat.com>,  Michael Santana <maicolgabriel@hotmail.com>,  Harry
 van Haaren <harry.van.haaren@intel.com>,  Cristian Dumitrescu
 <cristian.dumitrescu@intel.com>,  Phil Yang <phil.yang@arm.com>,  Joyce
 Kong <joyce.kong@arm.com>,  Mattias =?utf-8?Q?R=C3=B6nnblom?=
 <mattias.ronnblom@ericsson.com>,  "Jan Viktorin"
 <viktorin@rehivetech.com>,  Gavin Hu <gavin.hu@arm.com>,  "David
 Christensen" <drc@linux.vnet.ibm.com>,  Konstantin Ananyev
 <konstantin.ananyev@intel.com>,  Anatoly Burakov
 <anatoly.burakov@intel.com>,  Harini Ramakrishnan
 <harini.ramakrishnan@microsoft.com>,  Omar Cardona
 <ocardona@microsoft.com>,  Anand Rawat <anand.rawat@intel.com>,  Ranjit
 Menon <ranjit.menon@intel.com>,  Olivier Matz <olivier.matz@6wind.com>,
 Gage Eads <gage.eads@intel.com>,  Adrien Mazarguil
 <adrien.mazarguil@6wind.com>,  Nicolas Chautru
 <nicolas.chautru@intel.com>,  Declan Doherty <declan.doherty@intel.com>,
 "Fiona Trahe" <fiona.trahe@intel.com>,  Ashish Gupta
 <ashishg@marvell.com>,  Erik Gabriel Carrillo <erik.g.carrillo@intel.com>,
 Abhinandan Gujjar <abhinandan.gujjar@intel.com>,  Shreyansh Jain
 <shreyansh.jain@nxp.com>,  Hemant Agrawal <hemant.agrawal@nxp.com>,
 "Artem V. Andreev" <artem.andreev@oktetlabs.ru>,  Nithin Kumar Dabilpuram
 <ndabilpuram@marvell.com>,  Vamsi Krishna Attunuru
 <vattunuru@marvell.com>,  Rosen Xu <rosen.xu@intel.com>,  Sachin Saxena
 <sachin.saxena@nxp.com>,  Stephen Hemminger <sthemmin@microsoft.com>,
 Chas Williams <chas3@att.com>,  "John W. Linville"
 <linville@tuxdriver.com>,  "Prasun Kapoor" <pkapoor@marvell.com>,  Marcin
 Wojtas <mw@semihalf.com>,  "Michal Krawczyk" <mk@semihalf.com>,  Guy
 Tzalik <gtzalik@amazon.com>,  "Evgeny Schemeilin" <evgenys@amazon.com>,
 Igor Chauskin <igorch@amazon.com>,  "Ravi Kumar" <ravi1.kumar@amd.com>,
 Igor Russkikh <igor.russkikh@aquantia.com>,  Pavel Belous
 <pavel.belous@aquantia.com>,  Shepard Siegel
 <shepard.siegel@atomicrules.com>,  Ed Czeck <ed.czeck@atomicrules.com>,
 "John Miller" <john.miller@atomicrules.com>,  Somnath Kotur
 <somnath.kotur@broadcom.com>,  Maciej Czekaj <mczekaj@marvell.com>,
 Shijith Thotton <sthotton@marvell.com>,  Srisivasubramanian Srinivasan
 <srinivasan@marvell.com>,  Rahul Lakkireddy
 <rahul.lakkireddy@chelsio.com>,  John Daley <johndale@cisco.com>,  "Hyong
 Youb Kim" <hyonkim@cisco.com>,  "Wei Hu \(Xavier"
 <xavier.huwei@huawei.com>,  "Min Hu \(Connor" <humin29@huawei.com>,  Yisen
 Zhuang <yisen.zhuang@huawei.com>,  "Ziyang Xuan" <xuanziyang2@huawei.com>,
 Xiaoyun Wang <cloud.wangxiaoyun@huawei.com>,  Guoyang Zhou
 <zhouguoyang@huawei.com>,  Beilei Xing <beilei.xing@intel.com>,  "Xiao
 Wang" <xiao.w.wang@intel.com>,  Jingjing Wu <jingjing.wu@intel.com>,
 Wenzhuo Lu <wenzhuo.lu@intel.com>,  Qiming Yang <qiming.yang@intel.com>,
 Tomasz Duszynski <tdu@semihalf.com>,  Liron Himi <lironh@marvell.com>,
 Zyta Szpak <zr@semihalf.com>,  Kiran Kumar Kokkilagadda
 <kirankumark@marvell.com>,  Matan Azrad <matan@mellanox.com>,  Shahaf
 Shuler <shahafs@mellanox.com>,  Viacheslav Ovsiienko
 <viacheslavo@mellanox.com>,  "K. Y. Srinivasan" <kys@microsoft.com>,
 Haiyang Zhang <haiyangz@microsoft.com>,  Jan Remes <remes@netcope.com>,
 Heinrich Kuhn <heinrich.kuhn@netronome.com>,  Jan Gutter
 <jan.gutter@netronome.com>,  Gagandeep Singh <g.singh@nxp.com>,  Rasesh
 Mody <rmody@marvell.com>,  "Shahed Shaikh" <shshaikh@marvell.com>,  "Yong
 Wang" <yongwang@vmware.com>,  Zhihong Wang <zhihong.wang@intel.com>,
 Steven Webster <steven.webster@windriver.com>,  "Matt Peters"
 <matt.peters@windriver.com>,  Keith Wiles <keith.wiles@intel.com>,
 Tetsuya Mukawa <mtetsuyah@gmail.com>,  Gaetan Rivet
 <gaetan.rivet@6wind.com>,  Jasvinder Singh <jasvinder.singh@intel.com>,
 Jakub Grajciar <jgrajcia@cisco.com>,  Ruifeng Wang
 <ruifeng.wang@arm.com>,  "Anoob Joseph" <anoobj@marvell.com>,  Fan Zhang
 <roy.fan.zhang@intel.com>,  Pablo de Lara
 <pablo.de.lara.guarch@intel.com>,  John Griffin <john.griffin@intel.com>,
 "Deepak Kumar Jain" <deepak.k.jain@intel.com>,  Michael Shamis
 <michaelsh@marvell.com>,  Nagadheeraj Rottela <rnagadheeraj@marvell.com>,
 Srikanth Jampala <jsrikanth@marvell.com>,  Ankur Dwivedi
 <adwivedi@marvell.com>,  Jay Zhou <jianjay.zhou@huawei.com>,  Lee Daly
 <lee.daly@intel.com>,  "Sunila Sahu" <ssahu@marvell.com>,  Nipun Gupta
 <nipun.gupta@nxp.com>,  Liang Ma <liang.j.ma@intel.com>,  Peter Mccarthy
 <peter.mccarthy@intel.com>,  Tianfei zhang <tianfei.zhang@intel.com>,
 Satha Koteswara Rao Kottidi <skoteshwar@marvell.com>,  Xiaoyun Li
 <xiaoyun.li@intel.com>,  Bernard Iremonger <bernard.iremonger@intel.com>,
 Vladimir Medvedkin <vladimir.medvedkin@intel.com>,  David Hunt
 <david.hunt@intel.com>,  "Reshma Pattan" <reshma.pattan@intel.com>,  Byron
 Marohn <byron.marohn@intel.com>,  "Sameh Gobriel"
 <sameh.gobriel@intel.com>,  Yipeng Wang <yipeng1.wang@intel.com>,
 Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>,  Robert Sanford
 <rsanford@akamai.com>,  Kevin Laatz <kevin.laatz@intel.com>,  Maryam
 Tahhan <maryam.tahhan@intel.com>,  "Ori Kam" <orika@mellanox.com>,  "Radu
 Nicolau" <radu.nicolau@intel.com>,  Tomasz Kantecki
 <tomasz.kantecki@intel.com>,  Sunil Kumar Kori <skori@marvell.com>,
 "Pavan Nikhilesh Bhagavatula" <pbhagavatula@marvell.com>,  Kirill
 Rybalchenko <kirill.rybalchenko@intel.com>,  "Kadam\, Pallavi"
 <pallavi.kadam@intel.com>
References: <BYAPR18MB2424BF2CAD9AFE5F1C3755F0C8350@BYAPR18MB2424.namprd18.prod.outlook.com>
Date: Mon, 27 Jan 2020 11:12:48 -0500
In-Reply-To: <BYAPR18MB2424BF2CAD9AFE5F1C3755F0C8350@BYAPR18MB2424.namprd18.prod.outlook.com>
 (Jerin Jacob Kollanukkaran's message of "Mon, 13 Jan 2020 10:40:13
 +0000")
Message-ID: <f7ttv4g98u7.fsf@dhcp-25.97.bos.redhat.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12
X-MC-Unique: bgQvnzJ_O8yOkVHir_XrxA-1
X-Mimecast-Spam-Score: 0
X-Mimecast-Originator: redhat.com
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [dpdk-dev] [RFC]  DPDK Trace support
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

Jerin Jacob Kollanukkaran <jerinj@marvell.com> writes:

> Hi All,
>
> I would like to add tracing support for DPDK.
> I am planning to add this support in v20.05 release.
>
> This RFC attempts to get feedback from the community on
>
> a) Tracing Use cases.
> b) Tracing Requirements.
> b) Implementation choices.
> c) Trace format.
>
> Use-cases
> ---------
> - Most of the cases, The DPDK provider will not have access to the DPDK c=
ustomer applications.
> To debug/analyze the slow path and fast path DPDK API usage from the fiel=
d,
> we need to have integrated trace support in DPDK.
>
> - Need a low overhead Fast path multi-core PMD driver debugging/analysis
> infrastructure in DPDK to fix the functional and performance issue(s) of =
PMD.
>
> - Post trace analysis tools can provide various status across the system =
such
> as cpu_idle() using the timestamp added in the trace.
>
>
> Requirements:
> -------------
> - Support for Linux, FreeBSD and Windows OS
> - Open trace format
> - Multi-platform Open source trace viewer
> - Absolute low overhead trace API for DPDK fast path tracing/debugging.
> - Dynamic enable/disable of trace events
>
>
> To enable trace support in DPDK, following items need to work out:=20
>
> a) Add the DPDK trace points in the DPDK source code.
>
> - This includes updating DPDK functions such as,
> rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() to e=
mit the trace.

I wonder for these if it makes sense to use librte_bpf and a helper
function to actually emit events.  That way rather than static trace
point data, a user can implement some C-code and pull the exact data
that they want.

There could be some downside with the approach (because we might lose
some inlining or variable eliding), but I think it makes the trace point
concept quite a bit more powerful.  Have you given it any thought?

> b) Choosing suitable serialization-format
>
> - Common Trace Format, CTF, is an open format and language to describe tr=
ace formats.
> This enables tool reuse, of which line-textual (babeltrace) and=20
> graphical (TraceCompass) variants already exist.
>
> CTF should look familiar to C programmers but adds stronger typing.=20
> See CTF - A Flexible, High-performance Binary Trace Format.
>
> https://diamon.org/ctf/
>
> c) Writing the on-target serialization code,
>
> See the section below.(Lttng CTF trace emitter vs DPDK specific CTF trace=
 emitter)
> =20
> d) Deciding on and writing the I/O transport mechanics,
>
> For performance reasons, it should be backed by a huge-page and write to =
file IO.
>
> e) Writing the PC-side deserializer/parser,
>
> Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF.
> See:=20
> https://lttng.org/viewers/
>
> f) Writing tools for filtering and presentation.
>
> See item (e)
>
>
> Lttng CTF trace emitter vs DPDK specific CTF trace emitter
> ----------------------------------------------------------
>
> I have written a performance evaluation application to measure the overhe=
ad
> of Lttng CTF emitter(The fastpath infrastructure used by
> https://lttng.org/ library to emit the trace)
>
> https://github.com/jerinjacobk/lttng-overhead
> https://github.com/jerinjacobk/lttng-overhead/blob/master/README
>
> I could improve the performance by 30% by adding the "DPDK"
> based plugin for get_clock() and get_cpu(),
> Here are the performance numbers after adding the plugin on=20
> x86 and various arm64 board that I have access to,
>
> On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the
> last line in the log(ZERO_ARG))
> On arm64, it varies from 312 cycles to 1100 cycles(based on the class of =
CPU).
> In short, Based on the "IPC capabilities", The cost would be around 100ns=
 to 400ns
> for single void trace(a trace without any argument)
>
>
> [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0
> make[1]: Entering directory '/export/lttng-overhead-x86/calibrate'
> make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate'
> EAL: Detected 56 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: Probing VFIO support...
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> EAL: PCI device 0000:01:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> CPU Timer freq is 2600.000000MHz
> NOP: cycles=3D0.194834 ns=3D0.074936
> GET_CLOCK: cyclesG.854658 ns.405638
> GET_CPU: cycles0.995892 ns.921497
> ZERO_ARG: cycles#6.945113 ns=E2=80=98.132736
>
>
> We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng CTF=
 emitter
> may not fit the DPDK fast path purpose due to the cost associated with ge=
neric Lttng features.
>
> One option could be to have, native CTF emitter in EAL/DPDK to emit the
> trace in a hugepage. I think it would be a handful of cycles if we limit =
the features
> to the requirements above:
>
> The upside of using Lttng CTF emitter:
> a) No need to write a new CTF trace emitter(the item (c))
>
> The downside of Lttng CTF emitter(the item (c))
> a) performance issue(See above)
> b) Lack of Windows OS support. It looks like, it has basic FreeBSD suppor=
t.
> c) dpdk library dependency to lttng for trace.
>
> So, Probably it good to have native CTF emitter in DPDK and reuse all
> open-source trace viewer(babeltrace and  TraceCompass) and format(CTF) in=
frastructure.
> I think, it would be best of both world.
>
> Any thoughts on this subject? Based on the community feedback, I can
> work on the patch for v20.05.