From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4E80342D66; Tue, 27 Jun 2023 02:46:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CF17F4113F; Tue, 27 Jun 2023 02:46:28 +0200 (CEST) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by mails.dpdk.org (Postfix) with ESMTP id 3E66840F18 for ; Tue, 27 Jun 2023 02:46:27 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id C5F465C007D; Mon, 26 Jun 2023 20:46:25 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Mon, 26 Jun 2023 20:46:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1687826785; x=1687913185; bh=34HuZM4lCgdNb6u0bJ2iAxGFn2ws5Txyz56 67KXShAE=; b=nvnZXWYWnln01A2+V2vbG00F2vrWdD3AO/JnBvNuNhTYwCvF5DX zaAslAzFPEB4Qyye3WF8MpUSZZEpbF4j+zGVJtebsyLXV9cdxv8ssb4w8d2fyKPr 1aFelLjYr1OJTx2Uqy6dJwrZxR3+1Uxlcs/Fejst3AcTRyqbYgw6FXLopNwaPGy/ /LpN1fEa/exNGP57RC2iWezPZUwEyWql3HdTLDWqrRbd4mDQPHPWwnPDcbpJRyvH YwUkPLaTOHbqtndrEEXIQ2FsqMRSsYdlKXDKJuVcpuBhmjeaa8+5FywJORnl4Z9C lqdYvulEImWV6dj8mzrcOxbHG/T2DlK98eA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1687826785; x=1687913185; bh=34HuZM4lCgdNb6u0bJ2iAxGFn2ws5Txyz56 67KXShAE=; b=IUOSw4+CNfwX0Ba9Nb3KDkLV1+eyu8iRul+iStP1RDi7w7yg/wf wZyO0txj+ZL8cmXmOccCOBwPn4c++tQlLPesaJ0UzRo6/GoPAZJpj9PSV/UbnctL nvcLHDEzklH40TFyP9Joez38rVxHPimFN5H6EfAyaAAmn9APle8Mmwj7VwIg/pDI jAO/D6aCIPLSekQa3I8lD+bhofJoduAb5xTKDReHCLSwcYjnQHe/JoPe1G3nwYMw I0Jw4Sw3cUXOLI/WI/UuA+G+v8XjsQo1mhSX4JgOcS3Gi4d5bSj3YLcYPAXJAnoC wZwzvXioRZWLkPFbb33AIQaRfDdDRoMKGiQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgeehgedgfeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhm rghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenuc ggtffrrghtthgvrhhnpedthfdujeejffdvteekudduudehtddtjeeihedvjeefvdeihfeg keefteehueeffeenucffohhmrghinhepphhmugdrnhgvthdpghhithhhuhgsrdgtohhmpd htrhgrtggvrdhphienucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhl fhhrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Jun 2023 20:46:24 -0400 (EDT) From: Thomas Monjalon To: Slava Ovsiienko Cc: "dev@dpdk.org" , Raslan Darawsheh , rjarry@redhat.com, jerinj@marvell.com Subject: Re: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Date: Tue, 27 Jun 2023 02:46:23 +0200 Message-ID: <2676545.uZKlY2gecq@thomas> In-Reply-To: References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230613165845.19109-1-viacheslavo@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 20/06/2023 14:00, Raslan Darawsheh: > Hi, > > > -----Original Message----- > > From: Viacheslav Ovsiienko > > Sent: Tuesday, June 13, 2023 7:59 PM > > To: dev@dpdk.org > > Subject: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing > > > > The mlx5 provides the send scheduling on specific moment of time, > > and for the related kind of applications it would be extremely useful > > to have extra debug information - when and how packets were scheduled > > and when the actual sending was completed by the NIC hardware (it helps > > application to track the internal delay issues). > > > > Because the DPDK tx datapath API does not suppose getting any feedback > > from the driver and the feature looks like to be mlx5 specific, it seems > > to be reasonable to engage exisiting DPDK datapath tracing capability. > > > > The work cycle is supposed to be: > > - compile appplication with enabled tracing > > - run application with EAL parameters configuring the tracing in mlx5 > > Tx datapath > > - store the dump file with gathered tracing information > > - run analyzing scrypt (in Python) to combine related events (packet > > firing and completion) and see the data in human-readable view > > > > Below is the detailed instruction "how to" with mlx5 NIC to gather > > all the debug data including the full timings information. > > > > > > 1. Build DPDK application with enabled datapath tracing > > > > The meson option should be specified: > > --enable_trace_fp=true > > > > The c_args shoudl be specified: > > -DALLOW_EXPERIMENTAL_API > > > > The DPDK configuration examples: > > > > meson configure --buildtype=debug -Denable_trace_fp=true > > -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT - > > DALLOW_EXPERIMENTAL_API' build > > > > meson configure --buildtype=debug -Denable_trace_fp=true > > -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build > > > > meson configure --buildtype=release -Denable_trace_fp=true > > -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build > > > > meson configure --buildtype=release -Denable_trace_fp=true > > -Dc_args='-DALLOW_EXPERIMENTAL_API' build > > > > > > 2. Configuring the NIC > > > > If the sending completion timings are important the NIC should be configured > > to provide realtime timestamps, the REAL_TIME_CLOCK_ENABLE NV settings > > parameter > > should be configured to TRUE, for example with command (and with following > > FW/driver reset): > > > > sudo mlxconfig -d /dev/mst/mt4125_pciconf0 s > > REAL_TIME_CLOCK_ENABLE=1 > > > > > > 3. Run DPDK application to gather the traces > > > > EAL parameters controlling trace capability in runtime > > > > --trace=pmd.net.mlx5.tx - the regular expression enabling the tracepoints > > with matching names at least "pmd.net.mlx5.tx" > > must be enabled to gather all events needed > > to analyze mlx5 Tx datapath and its timings. > > By default all tracepoints are disabled. > > > > --trace-dir=/var/log - trace storing directory > > > > --trace-bufsz=B|K|M - optional, trace data buffer size > > per thread. The default is 1MB. > > > > --trace-mode=overwrite|discard - optional, selects trace data buffer mode. > > > > > > 4. Installing or Building Babeltrace2 Package > > > > The gathered trace data can be analyzed with a developed Python script. > > To parse the trace, the data script uses the Babeltrace2 library. > > The package should be either installed or built from source code as > > shown below: > > > > git clone https://github.com/efficios/babeltrace.git > > cd babeltrace > > ./bootstrap > > ./configure -help > > ./configure --disable-api-doc --disable-man-pages > > --disable-python-bindings-doc --enbale-python-plugins > > --enable-python-binding > > > > 5. Running the Analyzing Script > > > > The analyzing script is located in the folder: ./drivers/net/mlx5/tools > > It requires Python3.6, Babeltrace2 packages and it takes the only parameter > > of trace data file. For example: > > > > ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 > > > > > > 6. Interpreting the Script Output Data > > > > All the timings are given in nanoseconds. > > The list of Tx (and coming Rx) bursts per port/queue is presented in the > > output. > > Each list element contains the list of built WQEs with specific opcodes, and > > each WQE contains the list of the encompassed packets to send. This information should be in the documentation. I think we should request a review of the Python script from people familiar with tracing and from people more familiar with Python scripting for user tools.