From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1FC2442D71; Tue, 27 Jun 2023 13:34:20 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B420840F18; Tue, 27 Jun 2023 13:34:19 +0200 (CEST) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by mails.dpdk.org (Postfix) with ESMTP id EC58540EE1 for <dev@dpdk.org>; Tue, 27 Jun 2023 13:34:17 +0200 (CEST) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 5CFB65C0206; Tue, 27 Jun 2023 07:34:15 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Tue, 27 Jun 2023 07:34:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1687865655; x=1687952055; bh=BZLGvS7Fx6k0zzbCXhgkVQ6hy5Bbsk3YpK5 FW1s3OhE=; b=ULrJyW0nzXQFY9gtK/+spSZ+AuwbkLzRGDj/K1jTdbhW4jKypXS 2FoKb8n7tW0Bx9VPwgIvjqbSB+5cMDviMt2sAzYqbj6XH4mofS5c+Lj7+kTzerDK 4/QuUM56uh2qxd98AKWW7YNORkzQF5B1EQPQYlrtI+XYEZUL9i0yI714iPd+gUqO ORATMrbZv7I22yN8NtJSlVDtMKd7rf2WJDZ818rkGT1Y7y1RriRaV/5FqPJ5pTwp ufNNLW8FqOICN2Pd+6wiil0PcCQwxTdsuThW9ARoMM/dO7wyEQkxKW9P9gDNN9MI 6Ut88YJQ90wBmIqsA4IF9pf/EDWFFQ2orvQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1687865655; x=1687952055; bh=BZLGvS7Fx6k0zzbCXhgkVQ6hy5Bbsk3YpK5 FW1s3OhE=; b=QPpST7pZRuYYJxw7WFChsIZFQ0OkNp2nZWT/X19jcC17OSXGreL KrQnZe+il/o5+C/IQw7vYEhumQ7LNhVDpNnDW7YnX4zhWEZUnNhCWc7t0Z8oLUtK DHkaj2HQ4sFk60QyKpQ7aygs9QFy/GfU90pigpJMtl5lMNOS5haWrWp8SwEUL1mb +mxuoVJklW1lbU70MycM2JEPTMhXIqG8IuSZFPqncSEZwei6XIsu996H8Br7BBKo sxUxF26LnYx0DVPAOYdJT+/32DMDIoaHgzMTREcU43foP7+hIuAEJaJXWouIRr02 L1JtzPTa80sN/XxgRxSCkCGuZu6CrdrVeFQ== X-ME-Sender: <xms:NsmaZGRLBKMkqZ2fQQjz1Gws6V1asBl2JLnyxxQ7LgYj1-ErFoQpnw> <xme:NsmaZLwNtsTD_uRYeyPbuSFXrkN9MUOwfY6CAufqLNY1YUqARvBSWIV_TUrIVL6EL 6gRRiCInTGV2Qj4-A> X-ME-Received: <xmr:NsmaZD0Zi_rORBmQ03JUH6C3MODt7Dv-TnmDLMd2g3dXXPfnLsS3san7XkULrnGetzKnm9Kz5_o6WulyYbvD63ALVw> X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgeehhedgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhm rghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenuc ggtffrrghtthgvrhhnpedthfdujeejffdvteekudduudehtddtjeeihedvjeefvdeihfeg keefteehueeffeenucffohhmrghinhepphhmugdrnhgvthdpghhithhhuhgsrdgtohhmpd htrhgrtggvrdhphienucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhl fhhrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: <xmx:NsmaZCD5BRXRTGU3Ev17ZsOCg59Wbtln7QsE5sNT7to-cfOi8TTfBw> <xmx:NsmaZPiWIfcgMZ_Lz-tv7PzN-OitjwXv93fsmewHjx5NvKC4k76hkg> <xmx:NsmaZOqzdiSDfgF56HlfHrP3R4KuQMtw_hlanLEc4ofEk5bf2n6igw> <xmx:N8maZPfTe66_1fwTXF-Ed6OdpcBz3L8YB2FcdGDuUAKTSAX5Csr36A> Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 27 Jun 2023 07:34:13 -0400 (EDT) From: Thomas Monjalon <thomas@monjalon.net> To: Slava Ovsiienko <viacheslavo@nvidia.com>, "rjarry@redhat.com" <rjarry@redhat.com> Cc: "dev@dpdk.org" <dev@dpdk.org>, Raslan Darawsheh <rasland@nvidia.com>, "jerinj@marvell.com" <jerinj@marvell.com>, david.marchand@redhat.com Subject: Re: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Date: Tue, 27 Jun 2023 13:34:11 +0200 Message-ID: <2310682.ECZNHGQPT7@thomas> In-Reply-To: <DM6PR12MB3753CB7534423178B902296ADF27A@DM6PR12MB3753.namprd12.prod.outlook.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <2676545.uZKlY2gecq@thomas> <DM6PR12MB3753CB7534423178B902296ADF27A@DM6PR12MB3753.namprd12.prod.outlook.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org 27/06/2023 13:24, Slava Ovsiienko: > > > -----Original Message----- > > From: Thomas Monjalon <thomas@monjalon.net> > > Sent: Tuesday, June 27, 2023 3:46 AM > > To: Slava Ovsiienko <viacheslavo@nvidia.com> > > Cc: dev@dpdk.org; Raslan Darawsheh <rasland@nvidia.com>; > > rjarry@redhat.com; jerinj@marvell.com > > Subject: Re: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing > > > > 20/06/2023 14:00, Raslan Darawsheh: > > > Hi, > > > > > > > -----Original Message----- > > > > From: Viacheslav Ovsiienko <viacheslavo@nvidia.com> > > > > Sent: Tuesday, June 13, 2023 7:59 PM > > > > To: dev@dpdk.org > > > > Subject: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing > > > > > > > > The mlx5 provides the send scheduling on specific moment of time, > > > > and for the related kind of applications it would be extremely > > > > useful to have extra debug information - when and how packets were > > > > scheduled and when the actual sending was completed by the NIC > > > > hardware (it helps application to track the internal delay issues). > > > > > > > > Because the DPDK tx datapath API does not suppose getting any > > > > feedback from the driver and the feature looks like to be mlx5 > > > > specific, it seems to be reasonable to engage exisiting DPDK datapath > > tracing capability. > > > > > > > > The work cycle is supposed to be: > > > > - compile appplication with enabled tracing > > > > - run application with EAL parameters configuring the tracing in mlx5 > > > > Tx datapath > > > > - store the dump file with gathered tracing information > > > > - run analyzing scrypt (in Python) to combine related events (packet > > > > firing and completion) and see the data in human-readable view > > > > > > > > Below is the detailed instruction "how to" with mlx5 NIC to gather > > > > all the debug data including the full timings information. > > > > > > > > > > > > 1. Build DPDK application with enabled datapath tracing > > > > > > > > The meson option should be specified: > > > > --enable_trace_fp=true > > > > > > > > The c_args shoudl be specified: > > > > -DALLOW_EXPERIMENTAL_API > > > > > > > > The DPDK configuration examples: > > > > > > > > meson configure --buildtype=debug -Denable_trace_fp=true > > > > -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT - > > > > DALLOW_EXPERIMENTAL_API' build > > > > > > > > meson configure --buildtype=debug -Denable_trace_fp=true > > > > -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' > > > > build > > > > > > > > meson configure --buildtype=release -Denable_trace_fp=true > > > > -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' > > > > build > > > > > > > > meson configure --buildtype=release -Denable_trace_fp=true > > > > -Dc_args='-DALLOW_EXPERIMENTAL_API' build > > > > > > > > > > > > 2. Configuring the NIC > > > > > > > > If the sending completion timings are important the NIC should be > > > > configured to provide realtime timestamps, the > > > > REAL_TIME_CLOCK_ENABLE NV settings parameter should be configured > > to > > > > TRUE, for example with command (and with following FW/driver reset): > > > > > > > > sudo mlxconfig -d /dev/mst/mt4125_pciconf0 s > > > > REAL_TIME_CLOCK_ENABLE=1 > > > > > > > > > > > > 3. Run DPDK application to gather the traces > > > > > > > > EAL parameters controlling trace capability in runtime > > > > > > > > --trace=pmd.net.mlx5.tx - the regular expression enabling the tracepoints > > > > with matching names at least "pmd.net.mlx5.tx" > > > > must be enabled to gather all events needed > > > > to analyze mlx5 Tx datapath and its timings. > > > > By default all tracepoints are disabled. > > > > > > > > --trace-dir=/var/log - trace storing directory > > > > > > > > --trace-bufsz=<val>B|<val>K|<val>M - optional, trace data buffer size > > > > per thread. The default is 1MB. > > > > > > > > --trace-mode=overwrite|discard - optional, selects trace data buffer > > mode. > > > > > > > > > > > > 4. Installing or Building Babeltrace2 Package > > > > > > > > The gathered trace data can be analyzed with a developed Python script. > > > > To parse the trace, the data script uses the Babeltrace2 library. > > > > The package should be either installed or built from source code as > > > > shown below: > > > > > > > > git clone https://github.com/efficios/babeltrace.git > > > > cd babeltrace > > > > ./bootstrap > > > > ./configure -help > > > > ./configure --disable-api-doc --disable-man-pages > > > > --disable-python-bindings-doc --enbale-python-plugins > > > > --enable-python-binding > > > > > > > > 5. Running the Analyzing Script > > > > > > > > The analyzing script is located in the folder: > > > > ./drivers/net/mlx5/tools It requires Python3.6, Babeltrace2 packages > > > > and it takes the only parameter of trace data file. For example: > > > > > > > > ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 > > > > > > > > > > > > 6. Interpreting the Script Output Data > > > > > > > > All the timings are given in nanoseconds. > > > > The list of Tx (and coming Rx) bursts per port/queue is presented in > > > > the output. > > > > Each list element contains the list of built WQEs with specific > > > > opcodes, and each WQE contains the list of the encompassed packets to > > send. > > > > This information should be in the documentation. > OK, should we make this cover-letter part of mlx5.rst? Kind of, yes. > > I think we should request a review of the Python script from people familiar > > with tracing and from people more familiar with Python scripting for user > > tools. > Would be very helpful, could you recommend/ask someone? Jerin, what do you think of such a script? Robin, would you have time to look at this trace processing script please?