From: Raslan Darawsheh <rasland@nvidia.com>
To: Slava Ovsiienko <viacheslavo@nvidia.com>, "dev@dpdk.org" <dev@dpdk.org>
Cc: "jerinj@marvell.com" <jerinj@marvell.com>
Subject: RE: [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing
Date: Thu, 6 Jul 2023 16:27:41 +0000 [thread overview]
Message-ID: <BYAPR12MB30784CAEDE994A32D0F0CF44CF2CA@BYAPR12MB3078.namprd12.prod.outlook.com> (raw)
In-Reply-To: <20230705153125.4657-1-viacheslavo@nvidia.com>
Hi,
> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Wednesday, July 5, 2023 6:31 PM
> To: dev@dpdk.org
> Cc: jerinj@marvell.com; Raslan Darawsheh <rasland@nvidia.com>
> Subject: [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing
>
> The mlx5 provides the send scheduling on specific moment of time, and for
> the related kind of applications it would be extremely useful to have extra
> debug information - when and how packets were scheduled and when the
> actual sending was completed by the NIC hardware (it helps application to
> track the internal delay issues).
>
> Because the DPDK tx datapath API does not suppose getting any feedback
> from the driver and the feature looks like to be mlx5 specific, it seems to be
> reasonable to engage exisiting DPDK datapath tracing capability.
>
> The work cycle is supposed to be:
> - compile appplication with enabled tracing
> - run application with EAL parameters configuring the tracing in mlx5
> Tx datapath
> - store the dump file with gathered tracing information
> - run analyzing scrypt (in Python) to combine related events (packet
> firing and completion) and see the data in human-readable view
>
> Below is the detailed instruction "how to" with mlx5 NIC to gather all the
> debug data including the full timings information.
>
>
> 1. Build DPDK application with enabled datapath tracing
>
> The meson option should be specified:
> --enable_trace_fp=true
>
> The c_args shoudl be specified:
> -DALLOW_EXPERIMENTAL_API
>
> The DPDK configuration examples:
>
> meson configure --buildtype=debug -Denable_trace_fp=true
> -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT -
> DALLOW_EXPERIMENTAL_API' build
>
> meson configure --buildtype=debug -Denable_trace_fp=true
> -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build
>
> meson configure --buildtype=release -Denable_trace_fp=true
> -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build
>
> meson configure --buildtype=release -Denable_trace_fp=true
> -Dc_args='-DALLOW_EXPERIMENTAL_API' build
>
>
> 2. Configuring the NIC
>
> If the sending completion timings are important the NIC should be configured
> to provide realtime timestamps, the REAL_TIME_CLOCK_ENABLE NV settings
> parameter should be configured to TRUE, for example with command (and
> with following FW/driver reset):
>
> sudo mlxconfig -d /dev/mst/mt4125_pciconf0 s
> REAL_TIME_CLOCK_ENABLE=1
>
>
> 3. Run DPDK application to gather the traces
>
> EAL parameters controlling trace capability in runtime
>
> --trace=pmd.net.mlx5.tx - the regular expression enabling the tracepoints
> with matching names at least "pmd.net.mlx5.tx"
> must be enabled to gather all events needed
> to analyze mlx5 Tx datapath and its timings.
> By default all tracepoints are disabled.
>
> --trace-dir=/var/log - trace storing directory
>
> --trace-bufsz=<val>B|<val>K|<val>M - optional, trace data buffer size
> per thread. The default is 1MB.
>
> --trace-mode=overwrite|discard - optional, selects trace data buffer mode.
>
>
> 4. Installing or Building Babeltrace2 Package
>
> The gathered trace data can be analyzed with a developed Python script.
> To parse the trace, the data script uses the Babeltrace2 library.
> The package should be either installed or built from source code as shown
> below:
>
> git clone https://github.com/efficios/babeltrace.git
> cd babeltrace
> ./bootstrap
> ./configure -help
> ./configure --disable-api-doc --disable-man-pages
> --disable-python-bindings-doc --enbale-python-plugins
> --enable-python-binding
>
> 5. Running the Analyzing Script
>
> The analyzing script is located in the folder: ./drivers/net/mlx5/tools It requires
> Python3.6, Babeltrace2 packages and it takes the only parameter of trace data
> file. For example:
>
> ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39
>
>
> 6. Interpreting the Script Output Data
>
> All the timings are given in nanoseconds.
> The list of Tx (and coming Rx) bursts per port/queue is presented in the
> output.
> Each list element contains the list of built WQEs with specific opcodes, and
> each WQE contains the list of the encompassed packets to send.
>
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
>
> --
> v2: - comment addressed: "dump_trace" command is replaced with
> "save_trace"
> - Windows build failure addressed, Windows does not support tracing
>
> v3: - tracepoint routines are moved to the net folder, no need to export
> - documentation added
> - testpmd patches moved out from series to the dedicated patches
>
> v4: - Python comments addressed
> - codestyle issues fixed
>
> v5: - traces are moved to the dedicated files, otherwise registration
> header caused wrong code generation for 3rd party files/objects
> and resulted in performance drop
>
> Viacheslav Ovsiienko (4):
> net/mlx5: introduce tracepoints for mlx5 drivers
> net/mlx5: add comprehensive send completion trace
> net/mlx5: add Tx datapath trace analyzing script
> doc: add mlx5 datapath tracing feature description
>
> doc/guides/nics/mlx5.rst | 78 +++++++
> drivers/net/mlx5/linux/mlx5_verbs.c | 8 +-
> drivers/net/mlx5/meson.build | 1 +
> drivers/net/mlx5/mlx5_devx.c | 8 +-
> drivers/net/mlx5/mlx5_rx.h | 19 --
> drivers/net/mlx5/mlx5_rxtx.h | 19 ++
> drivers/net/mlx5/mlx5_trace.c | 25 +++
> drivers/net/mlx5/mlx5_trace.h | 73 +++++++
> drivers/net/mlx5/mlx5_tx.c | 9 +
> drivers/net/mlx5/mlx5_tx.h | 89 +++++++-
> drivers/net/mlx5/tools/mlx5_trace.py | 307
> +++++++++++++++++++++++++++
> 11 files changed, 607 insertions(+), 29 deletions(-) create mode 100644
> drivers/net/mlx5/mlx5_trace.c create mode 100644
> drivers/net/mlx5/mlx5_trace.h create mode 100755
> drivers/net/mlx5/tools/mlx5_trace.py
>
> --
> 2.18.1
Applied first two patches to next-net-mlx,
Script + doc will be considered for RC4
Kindest regards,
Raslan Darawsheh
next prev parent reply other threads:[~2023-07-06 16:27 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-04-20 10:07 [RFC 0/5] " Viacheslav Ovsiienko
2023-04-20 10:07 ` [RFC 1/5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-04-20 10:13 ` Jerin Jacob
2023-04-20 10:08 ` [RFC 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-04-20 10:11 ` Jerin Jacob
2023-06-13 15:50 ` Slava Ovsiienko
2023-06-13 15:53 ` Jerin Jacob
2023-06-13 15:59 ` Slava Ovsiienko
2023-06-13 16:01 ` Jerin Jacob
2023-06-27 0:39 ` Thomas Monjalon
2023-06-27 6:15 ` Slava Ovsiienko
2023-06-27 7:28 ` Thomas Monjalon
2023-06-27 8:19 ` Slava Ovsiienko
2023-06-27 9:33 ` Thomas Monjalon
2023-06-27 9:43 ` Slava Ovsiienko
2023-06-27 11:36 ` Thomas Monjalon
2023-04-20 10:08 ` [RFC 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-04-20 10:08 ` [RFC 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-04-20 10:08 ` [RFC 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 0/5] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 1/5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-13 16:58 ` [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-13 16:58 ` [PATCH v2 1/5] app/testpmd: add trace save command Viacheslav Ovsiienko
2023-06-21 11:15 ` Ferruh Yigit
2023-06-23 8:00 ` Slava Ovsiienko
2023-06-23 11:52 ` Ferruh Yigit
2023-06-23 12:03 ` Jerin Jacob
2023-06-23 12:14 ` Slava Ovsiienko
2023-06-23 12:23 ` Ferruh Yigit
2023-06-13 16:58 ` [PATCH v2 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-13 16:58 ` [PATCH v2 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-06-13 16:58 ` [PATCH v2 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-13 16:58 ` [PATCH v2 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-20 12:00 ` [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Raslan Darawsheh
2023-06-27 0:46 ` Thomas Monjalon
2023-06-27 11:24 ` Slava Ovsiienko
2023-06-27 11:34 ` Thomas Monjalon
2023-06-28 14:18 ` Robin Jarry
2023-06-29 7:16 ` Slava Ovsiienko
2023-06-29 9:08 ` Robin Jarry
2023-06-26 11:06 ` [PATCH] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-26 11:07 ` [PATCH v3] " Viacheslav Ovsiienko
2023-06-26 11:57 ` [PATCH v4] " Viacheslav Ovsiienko
2023-06-27 11:34 ` Ferruh Yigit
2023-06-27 11:39 ` Slava Ovsiienko
2023-06-27 11:58 ` Ferruh Yigit
2023-06-27 14:44 ` [PATCH] app/testpmd: add dump command help message Viacheslav Ovsiienko
2023-06-27 18:03 ` Ferruh Yigit
2023-06-28 9:54 ` [PATCH v2] " Viacheslav Ovsiienko
2023-06-28 13:18 ` Ferruh Yigit
2023-06-27 13:09 ` [PATCH v5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-27 15:18 ` Ferruh Yigit
2023-06-28 11:09 ` [PATCH v3 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-28 11:09 ` [PATCH v3 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-28 11:09 ` [PATCH v3 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-28 11:09 ` [PATCH v3 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-28 11:09 ` [PATCH v3 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-05 11:10 ` [PATCH v4 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-07-05 11:10 ` [PATCH v4 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-07-05 11:10 ` [PATCH v4 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-07-05 11:10 ` [PATCH v4 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-05 11:10 ` [PATCH v4 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-05 15:31 ` [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-07-05 15:31 ` [PATCH v5 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-07-05 15:31 ` [PATCH v5 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-07-05 15:31 ` [PATCH v5 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-05 15:31 ` [PATCH v5 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-06 16:27 ` Raslan Darawsheh [this message]
2023-07-11 15:15 ` [PATCH v6 0/2] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-07-11 15:15 ` [PATCH v6 1/2] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-11 15:15 ` [PATCH v6 2/2] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-27 10:52 ` [PATCH v6 0/2] net/mlx5: introduce Tx datapath tracing Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=BYAPR12MB30784CAEDE994A32D0F0CF44CF2CA@BYAPR12MB3078.namprd12.prod.outlook.com \
--to=rasland@nvidia.com \
--cc=dev@dpdk.org \
--cc=jerinj@marvell.com \
--cc=viacheslavo@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).