DPDK patches and discussions
 help / color / mirror / Atom feed
From: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
To: <dev@dpdk.org>
Cc: <jerinj@marvell.com>, <rasland@nvidia.com>
Subject: [PATCH v4 0/4] net/mlx5: introduce Tx datapath tracing
Date: Wed, 5 Jul 2023 14:10:34 +0300	[thread overview]
Message-ID: <20230705111038.9935-1-viacheslavo@nvidia.com> (raw)
In-Reply-To: <20230420100803.494-1-viacheslavo@nvidia.com>

The mlx5 provides the send scheduling on specific moment of time,
and for the related kind of applications it would be extremely useful
to have extra debug information - when and how packets were scheduled
and when the actual sending was completed by the NIC hardware (it helps
application to track the internal delay issues).

Because the DPDK tx datapath API does not suppose getting any feedback
from the driver and the feature looks like to be mlx5 specific, it seems
to be reasonable to engage exisiting DPDK datapath tracing capability.

The work cycle is supposed to be:
  - compile appplication with enabled tracing
  - run application with EAL parameters configuring the tracing in mlx5
    Tx datapath
  - store the dump file with gathered tracing information
  - run analyzing scrypt (in Python) to combine related events (packet
    firing and completion) and see the data in human-readable view

Below is the detailed instruction "how to" with mlx5 NIC to gather
all the debug data including the full timings information.


1. Build DPDK application with enabled datapath tracing

The meson option should be specified:
   --enable_trace_fp=true

The c_args shoudl be specified:
   -DALLOW_EXPERIMENTAL_API

The DPDK configuration examples:

  meson configure --buildtype=debug -Denable_trace_fp=true
        -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build

  meson configure --buildtype=debug -Denable_trace_fp=true
        -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build

  meson configure --buildtype=release -Denable_trace_fp=true
        -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build

  meson configure --buildtype=release -Denable_trace_fp=true
        -Dc_args='-DALLOW_EXPERIMENTAL_API' build


2. Configuring the NIC

If the sending completion timings are important the NIC should be configured
to provide realtime timestamps, the REAL_TIME_CLOCK_ENABLE NV settings parameter
should be configured to TRUE, for example with command (and with following
FW/driver reset):

  sudo mlxconfig -d /dev/mst/mt4125_pciconf0 s REAL_TIME_CLOCK_ENABLE=1


3. Run DPDK application to gather the traces

EAL parameters controlling trace capability in runtime

  --trace=pmd.net.mlx5.tx - the regular expression enabling the tracepoints
                            with matching names at least "pmd.net.mlx5.tx"
                            must be enabled to gather all events needed
                            to analyze mlx5 Tx datapath and its timings.
                            By default all tracepoints are disabled.

  --trace-dir=/var/log - trace storing directory

  --trace-bufsz=<val>B|<val>K|<val>M - optional, trace data buffer size
                                       per thread. The default is 1MB.

  --trace-mode=overwrite|discard  - optional, selects trace data buffer mode.


4. Installing or Building Babeltrace2 Package

The gathered trace data can be analyzed with a developed Python script.
To parse the trace, the data script uses the Babeltrace2 library.
The package should be either installed or built from source code as
shown below:

  git clone https://github.com/efficios/babeltrace.git
  cd babeltrace
  ./bootstrap
  ./configure -help
  ./configure --disable-api-doc --disable-man-pages
              --disable-python-bindings-doc --enbale-python-plugins
              --enable-python-binding

5. Running the Analyzing Script

The analyzing script is located in the folder: ./drivers/net/mlx5/tools
It requires Python3.6, Babeltrace2 packages and it takes the only parameter
of trace data file. For example:

   ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39


6. Interpreting the Script Output Data

All the timings are given in nanoseconds.
The list of Tx (and coming Rx) bursts per port/queue is presented in the output.
Each list element contains the list of built WQEs with specific opcodes, and
each WQE contains the list of the encompassed packets to send.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

--
v2: - comment addressed: "dump_trace" command is replaced with "save_trace"
    - Windows build failure addressed, Windows does not support tracing

v3: - tracepoint routines are moved to the net folder, no need to export
    - documentation added
    - testpmd patches moved out from series to the dedicated patches

v4: - Python comments addressed
    - codestyle issues fixed

Viacheslav Ovsiienko (4):
  net/mlx5: introduce tracepoints for mlx5 drivers
  net/mlx5: add comprehensive send completion trace
  net/mlx5: add Tx datapath trace analyzing script
  doc: add mlx5 datapath tracing feature description

 doc/guides/nics/mlx5.rst             |  78 +++++++
 drivers/net/mlx5/linux/mlx5_verbs.c  |   8 +-
 drivers/net/mlx5/mlx5_devx.c         |   8 +-
 drivers/net/mlx5/mlx5_rx.h           |  19 --
 drivers/net/mlx5/mlx5_rxtx.h         |  19 ++
 drivers/net/mlx5/mlx5_tx.c           |  29 +++
 drivers/net/mlx5/mlx5_tx.h           | 135 +++++++++++-
 drivers/net/mlx5/tools/mlx5_trace.py | 307 +++++++++++++++++++++++++++
 8 files changed, 574 insertions(+), 29 deletions(-)
 create mode 100755 drivers/net/mlx5/tools/mlx5_trace.py

-- 
2.18.1


  parent reply	other threads:[~2023-07-05 11:11 UTC|newest]

Thread overview: 76+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-04-20 10:07 [RFC 0/5] " Viacheslav Ovsiienko
2023-04-20 10:07 ` [RFC 1/5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-04-20 10:13   ` Jerin Jacob
2023-04-20 10:08 ` [RFC 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-04-20 10:11   ` Jerin Jacob
2023-06-13 15:50     ` Slava Ovsiienko
2023-06-13 15:53       ` Jerin Jacob
2023-06-13 15:59         ` Slava Ovsiienko
2023-06-13 16:01           ` Jerin Jacob
2023-06-27  0:39             ` Thomas Monjalon
2023-06-27  6:15               ` Slava Ovsiienko
2023-06-27  7:28                 ` Thomas Monjalon
2023-06-27  8:19                   ` Slava Ovsiienko
2023-06-27  9:33                     ` Thomas Monjalon
2023-06-27  9:43                       ` Slava Ovsiienko
2023-06-27 11:36                         ` Thomas Monjalon
2023-04-20 10:08 ` [RFC 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-04-20 10:08 ` [RFC 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-04-20 10:08 ` [RFC 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 0/5] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 1/5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-13 16:58 ` [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 1/5] app/testpmd: add trace save command Viacheslav Ovsiienko
2023-06-21 11:15     ` Ferruh Yigit
2023-06-23  8:00       ` Slava Ovsiienko
2023-06-23 11:52         ` Ferruh Yigit
2023-06-23 12:03           ` Jerin Jacob
2023-06-23 12:14             ` Slava Ovsiienko
2023-06-23 12:23             ` Ferruh Yigit
2023-06-13 16:58   ` [PATCH v2 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-20 12:00   ` [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Raslan Darawsheh
2023-06-27  0:46     ` Thomas Monjalon
2023-06-27 11:24       ` Slava Ovsiienko
2023-06-27 11:34         ` Thomas Monjalon
2023-06-28 14:18           ` Robin Jarry
2023-06-29  7:16             ` Slava Ovsiienko
2023-06-29  9:08               ` Robin Jarry
2023-06-26 11:06 ` [PATCH] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-26 11:07 ` [PATCH v3] " Viacheslav Ovsiienko
2023-06-26 11:57 ` [PATCH v4] " Viacheslav Ovsiienko
2023-06-27 11:34   ` Ferruh Yigit
2023-06-27 11:39     ` Slava Ovsiienko
2023-06-27 11:58       ` Ferruh Yigit
2023-06-27 14:44     ` [PATCH] app/testpmd: add dump command help message Viacheslav Ovsiienko
2023-06-27 18:03       ` Ferruh Yigit
2023-06-28  9:54         ` [PATCH v2] " Viacheslav Ovsiienko
2023-06-28 13:18           ` Ferruh Yigit
2023-06-27 13:09 ` [PATCH v5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-27 15:18   ` Ferruh Yigit
2023-06-28 11:09 ` [PATCH v3 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-05 11:10 ` Viacheslav Ovsiienko [this message]
2023-07-05 11:10   ` [PATCH v4 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-07-05 11:10   ` [PATCH v4 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-07-05 11:10   ` [PATCH v4 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-05 11:10   ` [PATCH v4 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-05 15:31 ` [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-06 16:27   ` [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing Raslan Darawsheh
2023-07-11 15:15 ` [PATCH v6 0/2] " Viacheslav Ovsiienko
2023-07-11 15:15   ` [PATCH v6 1/2] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-11 15:15   ` [PATCH v6 2/2] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-27 10:52   ` [PATCH v6 0/2] net/mlx5: introduce Tx datapath tracing Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230705111038.9935-1-viacheslavo@nvidia.com \
    --to=viacheslavo@nvidia.com \
    --cc=dev@dpdk.org \
    --cc=jerinj@marvell.com \
    --cc=rasland@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).