From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 75271A04F0; Mon, 13 Jan 2020 12:43:23 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9FE471D41F; Mon, 13 Jan 2020 12:43:11 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 6958C1D5C5; Mon, 13 Jan 2020 11:40:47 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 00DAcdF7022682; Mon, 13 Jan 2020 02:40:18 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=AAVZhpdBuUGxKiDcBouPECokFoLZ9d4pGnYog698h2M=; b=olENT2t9i2KKIpgcqKuJvf4sc5a08uaBTXBKnGjIz2XFNSblZ/krhzAR6eXfzzczZ7JE eDAzueXhF+zWQW30PNtOnWI6F8oBywR75GUXqDHavtK0ze9+Yx0+Vg1dROgB9EGHm/p0 mGoN89SLaJQ7dWuZr48tOmB/y2KRF1xn0z+Ur+uXibHg5o7fjO0DvJ5kY1V8Qc7dw7mT DvRTpBDEitiQs8VGypqLRqT7zEdczlKi4dVgtjl/Hqx9mV7FTc8xCsQX17uQ8k7ONTnG LgJlsba5kHuGKnTkSqHj4Y/vHFpD6oeatYHPZd2O9LA0JmDu9+OoyO6/K8cBUdEQyj72 Dw== Received: from sc-exch03.marvell.com ([199.233.58.183]) by mx0b-0016f401.pphosted.com with ESMTP id 2xgng4r979-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 13 Jan 2020 02:40:17 -0800 Received: from SC-EXCH04.marvell.com (10.93.176.84) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 13 Jan 2020 02:40:15 -0800 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.169) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Mon, 13 Jan 2020 02:40:14 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kbqqQ6zBKK6QCSb8riU3Xv1Ztsrb1ocxXCqkLenQqopzVTZX8YMeGgDvSBTtxd7xbYlO1772XkkhHGHf8xv0ASHmqcPhYiXZzRpdU1qi8Xx/+4MAB6DvdR5juToWlzHVbsvTpqXqKnZzpzuH2s9IvduaWn6uJYob5kyHjqSRfWGReUdRWfFBPVs5/iYnzVCWYy3r1yERfSKBKknGNCTjs0oEPdZHCHthSiCET905pb/3EbgVyepTV8e3erjQxhf+I/yqHfrf9J2pLSpU697AqmMkWufIC+YDcNFcr2eKm7LZozTbBur+hmKgXJXQPrtCocPM8YBKFSS83g0XMejc7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AAVZhpdBuUGxKiDcBouPECokFoLZ9d4pGnYog698h2M=; b=DORulVzCMi7Tm54dHB0LlbNh8KjoFooKXpObHNQhnwyt6sJ0ADnNfFRT1eRLZkCMTBeRBgfbark/1IVh0IMt8uftUzfiIJdFU1OKXdcwWNhzLxqlxDO3CWSgzqd5V6tppP4i+TwPiUXMI4fDhtMDvJCXIfCXZtrmBHxpnC48wWOWq9TtGi7nbnt0hBnpOXQIjquw+qNyB3zs8eR6p7sLZuqAKVQLIUp8e8UGE6CXrWJRfvT5rm0q/PioatjG66aMUeyi1yrmalWKk+RFoNZ3C51GhHoNwF7uNvdUJ5I4nqWFI9jee1iAANuBu1Jmfim/XIsSE3HI955ja62MrZEDUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AAVZhpdBuUGxKiDcBouPECokFoLZ9d4pGnYog698h2M=; b=TxfhlWBtD9bVZTP+6OEkn9MCvb07oiL/IqCLGVMTT33f3i6VqsNHIulwDVGaWvT+0OaCYPeDJ+pft5jlG5XqDaUBXR5qGGY/qxBzk103ntvi/FJzjnsyT/Sebx7E85JRuMSxuFHsf3xImqeGuf+44LG5In0Le3MvVopFE5eGexk= Received: from BYAPR18MB2424.namprd18.prod.outlook.com (20.179.91.149) by BYAPR18MB2741.namprd18.prod.outlook.com (20.179.56.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2623.9; Mon, 13 Jan 2020 10:40:13 +0000 Received: from BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::1518:7b9a:3d96:4a2c]) by BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::1518:7b9a:3d96:4a2c%7]) with mapi id 15.20.2623.015; Mon, 13 Jan 2020 10:40:13 +0000 From: Jerin Jacob Kollanukkaran To: "dev@dpdk.org" CC: Thomas Monjalon , David Marchand , Ferruh Yigit , "Andrew Rybchenko" , Ajit Khaparde , Qi Zhang , Xiaolong Ye , Jerin Jacob Kollanukkaran , Raslan Darawsheh , Maxime Coquelin , Tiwei Bie , Akhil Goyal , Jerin Jacob Kollanukkaran , "Luca Boccassi" , Kevin Traynor , "maintainers@dpdk.org" , John McNamara , Marko Kovacevic , "Ray Kinsella" , Bruce Richardson , Aaron Conole , Michael Santana , Harry van Haaren , Cristian Dumitrescu , Phil Yang , Joyce Kong , =?iso-8859-1?Q?Mattias_R=F6nnblom?= , "Jan Viktorin" , Gavin Hu , "Jerin Jacob Kollanukkaran" , Gavin Hu , "David Christensen" , Bruce Richardson , Konstantin Ananyev , Ferruh Yigit , Anatoly Burakov , Harini Ramakrishnan , Omar Cardona , Anand Rawat , Ranjit Menon , Olivier Matz , Andrew Rybchenko , Olivier Matz , Gage Eads , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , Adrien Mazarguil , Nicolas Chautru , Declan Doherty , "Akhil Goyal" , Declan Doherty , "Fiona Trahe" , Ashish Gupta , "Jerin Jacob Kollanukkaran" , Erik Gabriel Carrillo , Abhinandan Gujjar , Shreyansh Jain , Hemant Agrawal , "Artem V. Andreev" , Andrew Rybchenko , Jerin Jacob Kollanukkaran , Nithin Kumar Dabilpuram , Vamsi Krishna Attunuru , Rosen Xu , Hemant Agrawal , Sachin Saxena , Stephen Hemminger , "Ferruh Yigit" , Chas Williams , Ferruh Yigit , "John W. Linville" , Xiaolong Ye , Qi Zhang , "Prasun Kapoor" , Marcin Wojtas , "Michal Krawczyk" , Guy Tzalik , "Evgeny Schemeilin" , Igor Chauskin , "Ravi Kumar" , Igor Russkikh , Pavel Belous , Shepard Siegel , Ed Czeck , "John Miller" , Ajit Khaparde , Somnath Kotur , Jerin Jacob Kollanukkaran , Maciej Czekaj , Shijith Thotton , Srisivasubramanian Srinivasan , "Jerin Jacob Kollanukkaran" , Rahul Lakkireddy , John Daley , "Hyong Youb Kim" , "Wei Hu (Xavier" , "Min Hu (Connor" , Yisen Zhuang , "Ziyang Xuan" , Xiaoyun Wang , Guoyang Zhou , Konstantin Ananyev , Beilei Xing , "Xiao Wang" , Jingjing Wu , Wenzhuo Lu , Xiao Wang , Qiming Yang , Wenzhuo Lu , Rosen Xu , Tomasz Duszynski , Liron Himi , Zyta Szpak , Liron Himi , Jerin Jacob Kollanukkaran , "Nithin Kumar Dabilpuram" , Kiran Kumar Kokkilagadda , Matan Azrad , Shahaf Shuler , Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko , Matan Azrad , Stephen Hemminger , "K. Y. Srinivasan" , Haiyang Zhang , Jan Remes , Jan Remes , Heinrich Kuhn , Jan Gutter , Hemant Agrawal , "Sachin Saxena" , Hemant Agrawal , Sachin Saxena , Gagandeep Singh , Sachin Saxena , Gagandeep Singh , Akhil Goyal , Rasesh Mody , "Shahed Shaikh" , Rasesh Mody , Shahed Shaikh , Andrew Rybchenko , "Yong Wang" , Maxime Coquelin , Tiwei Bie , Zhihong Wang , Maxime Coquelin , Tiwei Bie , Zhihong Wang , Maxime Coquelin , Tiwei Bie , Zhihong Wang , Steven Webster , "Matt Peters" , Ferruh Yigit , Keith Wiles , Ferruh Yigit , Bruce Richardson , Tetsuya Mukawa , Gaetan Rivet , Jasvinder Singh , Cristian Dumitrescu , Jakub Grajciar , "Ravi Kumar" , Ruifeng Wang , "Anoob Joseph" , Fan Zhang , "Declan Doherty" , Pablo de Lara , Declan Doherty , Pablo de Lara , John Griffin , Fiona Trahe , "Deepak Kumar Jain" , Pablo de Lara , Tomasz Duszynski , Michael Shamis , Liron Himi , Nagadheeraj Rottela , Srikanth Jampala , Ankur Dwivedi , Anoob Joseph , Declan Doherty , "Gagandeep Singh" , Hemant Agrawal , Akhil Goyal , Hemant Agrawal , Akhil Goyal , Hemant Agrawal , "Declan Doherty" , Pablo de Lara , Jay Zhou , "Pablo de Lara" , Ashish Gupta , Fiona Trahe , Lee Daly , "Sunila Sahu" , Jerin Jacob Kollanukkaran , Hemant Agrawal , Nipun Gupta , Hemant Agrawal , Nipun Gupta , Harry van Haaren , =?iso-8859-1?Q?Mattias_R=F6nnblom?= , Liang Ma , Peter Mccarthy , Rosen Xu , Tianfei zhang , "Bruce Richardson" , Satha Koteswara Rao Kottidi , Vamsi Krishna Attunuru , Xiaoyun Li , Jingjing Wu , Olivier Matz , Jasvinder Singh , Konstantin Ananyev , Konstantin Ananyev , Bernard Iremonger , Vladimir Medvedkin , Bernard Iremonger , David Hunt , "Reshma Pattan" , Cristian Dumitrescu , Jasvinder Singh , Reshma Pattan , Cristian Dumitrescu , Konstantin Ananyev , Byron Marohn , "Sameh Gobriel" , Vladimir Medvedkin , Yipeng Wang , "Sameh Gobriel" , Vladimir Medvedkin , Honnappa Nagarahalli , Gaetan Rivet , "David Hunt" , Robert Sanford , "Erik Gabriel Carrillo" , Reshma Pattan , Kevin Laatz , "Konstantin Ananyev" , Reshma Pattan , Wenzhuo Lu , Jingjing Wu , Bernard Iremonger , Declan Doherty , Jerin Jacob Kollanukkaran , Maryam Tahhan , Reshma Pattan , Marko Kovacevic , "Ori Kam" , Bruce Richardson , "Radu Nicolau" , Akhil Goyal , "Bruce Richardson" , Tomasz Kantecki , Sunil Kumar Kori , "Pavan Nikhilesh Bhagavatula" , John McNamara , Kirill Rybalchenko , Bruce Richardson , John McNamara , Harry van Haaren , Bruce Richardson , John McNamara , Xiaoyun Li , Prasun Kapoor , Keith Wiles , "Kadam, Pallavi" Thread-Topic: [RFC] [dpdk-dev] DPDK Trace support Thread-Index: AdXJ/aregbMiX+G3RtK8E2U6NaGudg== Date: Mon, 13 Jan 2020 10:40:13 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [14.140.231.66] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 2947cbaa-1fdd-40b9-00c3-08d79814f8b7 x-ms-traffictypediagnostic: BYAPR18MB2741: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4714; x-forefront-prvs: 028166BF91 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(39850400004)(396003)(366004)(136003)(346002)(376002)(478694002)(199004)(189003)(66476007)(9686003)(86362001)(33656002)(7276002)(478600001)(55236004)(6506007)(66556008)(76116006)(6916009)(7696005)(64756008)(966005)(4326008)(7336002)(7406005)(7366002)(7416002)(55016002)(26005)(186003)(66946007)(66446008)(5660300002)(52536014)(8676002)(81166006)(81156014)(71200400001)(54906003)(8936002)(316002)(2906002)(17680700007); DIR:OUT; SFP:1501; SCL:1; SRVR:BYAPR18MB2741; H:BYAPR18MB2424.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: QBSKB2n19shdqWbNmeg2Oi0kVV7by/AsBLNHGsbEP5XeYiHtv40gLdwerjnLQ/vOJxG6nCfbAkmlr8/G9F/95i8bg39Cs806ZzPCJMB15STzP7y7THOo7Hz3N+gHUbcdnKcel2NTiq64u8D9ruznF0jsk/mg9qcA8QZg+ju8Vtt3TeF344F27v6bvDOwmaO9+34QKif1OT5BbzcRi3JgJJwyzu2luAORZAcgTwFcN7QONgE0a/nv+Wee9EHWeeEMQIpn7sYUTWS0tbQyG8PuF0ndAIQy9PC/3WUoQbbX1iGNaZBFHMkZpww7gRso/Sl4GLVM34vGYn3RgLQX+xi15+WyRileWQ+pJrh1lfFQP1AqJ0Zj4T/FfUTh8ZOxc8JDJxMyrY1xQVgMk+2dmBwtBexu6iKjL0LHBu1lsvyZ+JnTmmHQ6b/J9ezb+Wc2Rztfp48whWYdLvtKehrWOsYvFRtidWGRJbmK2E3W5Wxm1RtpyIvBaWYYYbjktNTdIj8Njc8DiH2G6iy2zEMJqcDU73Qj0ygzhqCzAmCaoZaPUhM= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 2947cbaa-1fdd-40b9-00c3-08d79814f8b7 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Jan 2020 10:40:13.2836 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 5UaKuZTEO6ZlFn25/YDDwcBTIO4uLdtdqi02YvjjqulGaJJImclOt07fsCAn1mmDAuVTpQ2SE4G6s6wumKHQ/A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB2741 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-01-13_03:2020-01-13, 2020-01-13 signatures=0 X-Mailman-Approved-At: Mon, 13 Jan 2020 12:43:07 +0100 Subject: [dpdk-dev] [RFC] DPDK Trace support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi All, I would like to add tracing support for DPDK. I am planning to add this support in v20.05 release. This RFC attempts to get feedback from the community on a) Tracing Use cases. b) Tracing Requirements. b) Implementation choices. c) Trace format. Use-cases --------- - Most of the cases, The DPDK provider will not have access to the DPDK cus= tomer applications. To debug/analyze the slow path and fast path DPDK API usage from the field, we need to have integrated trace support in DPDK. - Need a low overhead Fast path multi-core PMD driver debugging/analysis infrastructure in DPDK to fix the functional and performance issue(s) of PM= D. - Post trace analysis tools can provide various status across the system su= ch as cpu_idle() using the timestamp added in the trace. Requirements: ------------- - Support for Linux, FreeBSD and Windows OS - Open trace format - Multi-platform Open source trace viewer - Absolute low overhead trace API for DPDK fast path tracing/debugging. - Dynamic enable/disable of trace events To enable trace support in DPDK, following items need to work out:=20 a) Add the DPDK trace points in the DPDK source code. - This includes updating DPDK functions such as, rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() to emi= t the trace. b) Choosing suitable serialization-format - Common Trace Format, CTF, is an open format and language to describe trac= e formats. This enables tool reuse, of which line-textual (babeltrace) and=20 graphical (TraceCompass) variants already exist. CTF should look familiar to C programmers but adds stronger typing.=20 See CTF - A Flexible, High-performance Binary Trace Format. https://diamon.org/ctf/ c) Writing the on-target serialization code, See the section below.(Lttng CTF trace emitter vs DPDK specific CTF trace e= mitter) =20 d) Deciding on and writing the I/O transport mechanics, For performance reasons, it should be backed by a huge-page and write to fi= le IO. e) Writing the PC-side deserializer/parser, Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF. See:=20 https://lttng.org/viewers/ f) Writing tools for filtering and presentation. See item (e) Lttng CTF trace emitter vs DPDK specific CTF trace emitter ---------------------------------------------------------- I have written a performance evaluation application to measure the overhead of Lttng CTF emitter(The fastpath infrastructure used by https://lttng.org/= library to emit the trace) https://github.com/jerinjacobk/lttng-overhead https://github.com/jerinjacobk/lttng-overhead/blob/master/README I could improve the performance by 30% by adding the "DPDK" based plugin for get_clock() and get_cpu(), Here are the performance numbers after adding the plugin on=20 x86 and various arm64 board that I have access to, On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the last l= ine in the log(ZERO_ARG))=20 On arm64, it varies from 312 cycles to 1100 cycles(based on the class of CP= U). In short, Based on the "IPC capabilities", The cost would be around 100ns t= o 400ns for single void trace(a trace without any argument) [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0 make[1]: Entering directory '/export/lttng-overhead-x86/calibrate' make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate' EAL: Detected 56 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: Probing VFIO support... EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb CPU Timer freq is 2600.000000MHz NOP: cycles=3D0.194834 ns=3D0.074936 GET_CLOCK: cycles=3D47.854658 ns=3D18.405638 GET_CPU: cycles=3D30.995892 ns=3D11.921497 ZERO_ARG: cycles=3D236.945113 ns=3D91.132736 We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng CTF e= mitter may not fit the DPDK fast path purpose due to the cost associated with gene= ric Lttng features. One option could be to have, native CTF emitter in EAL/DPDK to emit the trace in a hugepage. I think it would be a handful of cycles if we limit th= e features to the requirements above: The upside of using Lttng CTF emitter: a) No need to write a new CTF trace emitter(the item (c)) The downside of Lttng CTF emitter(the item (c)) a) performance issue(See above) b) Lack of Windows OS support. It looks like, it has basic FreeBSD support. c) dpdk library dependency to lttng for trace. So, Probably it good to have native CTF emitter in DPDK and reuse all open-source trace viewer(babeltrace and TraceCompass) and format(CTF) infr= astructure. I think, it would be best of both world. Any thoughts on this subject? Based on the community feedback, I can work o= n the patch for v20.05.