From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 814CDA0528; Mon, 20 Jan 2020 13:08:47 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1846611A2; Mon, 20 Jan 2020 13:08:46 +0100 (CET) Received: from qrelay35.mxroute.com (qrelay35.mxroute.com [172.82.139.35]) by dpdk.org (Postfix) with ESMTP id E0787FFA for ; Mon, 20 Jan 2020 13:08:43 +0100 (CET) Received: from filter004.mxroute.com (unknown [94.130.183.33]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by qrelay35.mxroute.com (Postfix) with ESMTPS id 03574A08D4; Mon, 20 Jan 2020 07:08:43 -0500 (EST) Received: from galaxy.mxroute.com (unknown [23.92.70.113]) by filter004.mxroute.com (Postfix) with ESMTPS id 4BCA93E9ED; Mon, 20 Jan 2020 12:08:36 +0000 (UTC) Received: from [192.198.151.43] by galaxy.mxroute.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1itVZ0-00049d-AZ; Mon, 20 Jan 2020 06:49:50 -0500 To: Jerin Jacob Kollanukkaran , "dave@barachs.net" , 'dpdk-dev' References: From: Ray Kinsella Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: Date: Mon, 20 Jan 2020 12:08:29 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [RFC] DPDK Trace support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" +1 - thanks Dave On 20/01/2020 04:48, Jerin Jacob Kollanukkaran wrote: >> -----Original Message----- >> From: dave@barachs.net >> Sent: Saturday, January 18, 2020 8:45 PM >> To: 'Ray Kinsella' ; Jerin Jacob Kollanukkaran >> ; 'dpdk-dev' >> Subject: [EXT] RE: [RFC] [dpdk-dev] DPDK Trace support >> >> It would be well worth considering one of the vpp techniques to minimize trace >> impact: >> >> static inline ring_handler_inline (..., int is_traced) { >> for (i = 0; i < vector_size; i++) >> { >> if (is_traced) >> { >> do_trace_work; >> } >> normal_packet_processing; >> } >> } >> >> ring_handler (...) >> { >> if (PREDICT_FALSE(global_trace_flag != 0)) >> return ring_handler_inline (..., 1 /* is_traced */); >> else >> return ring_handler_inline (..., 0 /* is_traced */); } >> >> This reduces the runtime tax to the absolute minimum, but costs space. >> >> Please consider it. > > Thanks Dave for your thoughts. > >> >> HTH... Dave >> >> -----Original Message----- >> From: Ray Kinsella >> Sent: Monday, January 13, 2020 6:00 AM >> To: Jerin Jacob Kollanukkaran ; dpdk-dev >> ; dave@barachs.net >> Subject: Re: [RFC] [dpdk-dev] DPDK Trace support >> >> Hi Jerin, >> >> Any idea why lttng performance is so poor? >> I would have naturally gone there to benefit from the existing toolchain. >> >> Have you looked at the FD.io logging/tracing infrastructure for inspiration? >> https://urldefense.proofpoint.com/v2/url?u=https- >> 3A__wiki.fd.io_view_VPP_elog&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1 >> DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=b9wJHO_k_ijKT84q47_ >> fO7MrN-LddnfpVSuNh6ce6Ks&s=WNwcIA86Rk2TY_C7O4bNTj3055Ofutab- >> bMPuM9-D4A&e= >> >> Ray K >> >> On 13/01/2020 10:40, Jerin Jacob Kollanukkaran wrote: >>> Hi All, >>> >>> I would like to add tracing support for DPDK. >>> I am planning to add this support in v20.05 release. >>> >>> This RFC attempts to get feedback from the community on >>> >>> a) Tracing Use cases. >>> b) Tracing Requirements. >>> b) Implementation choices. >>> c) Trace format. >>> >>> Use-cases >>> --------- >>> - Most of the cases, The DPDK provider will not have access to the DPDK >> customer applications. >>> To debug/analyze the slow path and fast path DPDK API usage from the >>> field, we need to have integrated trace support in DPDK. >>> >>> - Need a low overhead Fast path multi-core PMD driver >>> debugging/analysis infrastructure in DPDK to fix the functional and >> performance issue(s) of PMD. >>> >>> - Post trace analysis tools can provide various status across the >>> system such as cpu_idle() using the timestamp added in the trace. >>> >>> >>> Requirements: >>> ------------- >>> - Support for Linux, FreeBSD and Windows OS >>> - Open trace format >>> - Multi-platform Open source trace viewer >>> - Absolute low overhead trace API for DPDK fast path tracing/debugging. >>> - Dynamic enable/disable of trace events >>> >>> >>> To enable trace support in DPDK, following items need to work out: >>> >>> a) Add the DPDK trace points in the DPDK source code. >>> >>> - This includes updating DPDK functions such as, >>> rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() to emit >> the trace. >>> >>> b) Choosing suitable serialization-format >>> >>> - Common Trace Format, CTF, is an open format and language to describe >> trace formats. >>> This enables tool reuse, of which line-textual (babeltrace) and >>> graphical (TraceCompass) variants already exist. >>> >>> CTF should look familiar to C programmers but adds stronger typing. >>> See CTF - A Flexible, High-performance Binary Trace Format. >>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__diamon.org_ctf_&d >>> >> =DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4 >> wCNtTa4 >>> UUKvcsvI&m=b9wJHO_k_ijKT84q47_fO7MrN- >> LddnfpVSuNh6ce6Ks&s=QErjHnVHM1me2 >>> 4a6NGGIwiU6O5yot32ZW0vHbPnwZRg&e= >>> >>> c) Writing the on-target serialization code, >>> >>> See the section below.(Lttng CTF trace emitter vs DPDK specific CTF >>> trace emitter) >>> >>> d) Deciding on and writing the I/O transport mechanics, >>> >>> For performance reasons, it should be backed by a huge-page and write to file >> IO. >>> >>> e) Writing the PC-side deserializer/parser, >>> >>> Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF. >>> See: >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lttng.org_viewers >>> >> _&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s >> 5f4wCNt >>> Ta4UUKvcsvI&m=b9wJHO_k_ijKT84q47_fO7MrN- >> LddnfpVSuNh6ce6Ks&s=JCCywchwpf >>> jb7Cta5ykYG-SHkMnNUyqPRHh9QAFIcXg&e= >>> >>> f) Writing tools for filtering and presentation. >>> >>> See item (e) >>> >>> >>> Lttng CTF trace emitter vs DPDK specific CTF trace emitter >>> ---------------------------------------------------------- >>> >>> I have written a performance evaluation application to measure the >>> overhead of Lttng CTF emitter(The fastpath infrastructure used by >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lttng.org_&d=DwIF >>> >> aQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtT >> a4UUKvc >>> svI&m=b9wJHO_k_ijKT84q47_fO7MrN- >> LddnfpVSuNh6ce6Ks&s=dgfSVlEy8_W0IovAga >>> TnUT2ZbwCojfHimNxuyp4w7gI&e= library to emit the trace) >>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jerinj >>> acobk_lttng- >> 2Doverhead&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz >>> 6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=b9wJHO_k_ijKT84q47_fO7MrN- >> LddnfpVSu >>> Nh6ce6Ks&s=uSB4IwIan6cs9NuEUvGezK_jfdJj7Rjp0qrbThjk08M&e= >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jerinj >>> acobk_lttng- >> 2Doverhead_blob_master_README&d=DwIFaQ&c=nKjWec2b6R0mOyPaz >>> >> 7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=b9wJHO_k_i >> jKT84q >>> 47_fO7MrN-LddnfpVSuNh6ce6Ks&s=CudvGIANC2gl_e- >> TIAQt2IfpoczlIJIUee9IF78L >>> GHo&e= >>> >>> I could improve the performance by 30% by adding the "DPDK" >>> based plugin for get_clock() and get_cpu(), Here are the performance >>> numbers after adding the plugin on >>> x86 and various arm64 board that I have access to, >>> >>> On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the >>> last line in the log(ZERO_ARG)) On arm64, it varies from 312 cycles to 1100 >> cycles(based on the class of CPU). >>> In short, Based on the "IPC capabilities", The cost would be around >>> 100ns to 400ns for single void trace(a trace without any argument) >>> >>> >>> [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0 >>> make[1]: Entering directory '/export/lttng-overhead-x86/calibrate' >>> make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate' >>> EAL: Detected 56 lcore(s) >>> EAL: Detected 2 NUMA nodes >>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket >>> EAL: Selected IOVA mode 'PA' >>> EAL: Probing VFIO support... >>> EAL: PCI device 0000:01:00.0 on NUMA socket 0 >>> EAL: probe driver: 8086:1521 net_e1000_igb >>> EAL: PCI device 0000:01:00.1 on NUMA socket 0 >>> EAL: probe driver: 8086:1521 net_e1000_igb >>> CPU Timer freq is 2600.000000MHz >>> NOP: cycles=0.194834 ns=0.074936 >>> GET_CLOCK: cycles=47.854658 ns=18.405638 >>> GET_CPU: cycles=30.995892 ns=11.921497 >>> ZERO_ARG: cycles=236.945113 ns=91.132736 >>> >>> >>> We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng >>> CTF emitter may not fit the DPDK fast path purpose due to the cost >> associated with generic Lttng features. >>> >>> One option could be to have, native CTF emitter in EAL/DPDK to emit >>> the trace in a hugepage. I think it would be a handful of cycles if we >>> limit the features to the requirements above: >>> >>> The upside of using Lttng CTF emitter: >>> a) No need to write a new CTF trace emitter(the item (c)) >>> >>> The downside of Lttng CTF emitter(the item (c)) >>> a) performance issue(See above) >>> b) Lack of Windows OS support. It looks like, it has basic FreeBSD support. >>> c) dpdk library dependency to lttng for trace. >>> >>> So, Probably it good to have native CTF emitter in DPDK and reuse all >>> open-source trace viewer(babeltrace and TraceCompass) and format(CTF) >> infrastructure. >>> I think, it would be best of both world. >>> >>> Any thoughts on this subject? Based on the community feedback, I can work >> on the patch for v20.05. >>> >