DPDK patches and discussions
 help / color / mirror / Atom feed
From: Slava Ovsiienko <viacheslavo@nvidia.com>
To: "NBU-Contact-Thomas Monjalon (EXTERNAL)" <thomas@monjalon.net>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	Raslan Darawsheh <rasland@nvidia.com>,
	"rjarry@redhat.com" <rjarry@redhat.com>,
	"jerinj@marvell.com" <jerinj@marvell.com>
Subject: RE: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing
Date: Tue, 27 Jun 2023 11:24:14 +0000	[thread overview]
Message-ID: <DM6PR12MB3753CB7534423178B902296ADF27A@DM6PR12MB3753.namprd12.prod.outlook.com> (raw)
In-Reply-To: <2676545.uZKlY2gecq@thomas>



> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, June 27, 2023 3:46 AM
> To: Slava Ovsiienko <viacheslavo@nvidia.com>
> Cc: dev@dpdk.org; Raslan Darawsheh <rasland@nvidia.com>;
> rjarry@redhat.com; jerinj@marvell.com
> Subject: Re: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing
> 
> 20/06/2023 14:00, Raslan Darawsheh:
> > Hi,
> >
> > > -----Original Message-----
> > > From: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> > > Sent: Tuesday, June 13, 2023 7:59 PM
> > > To: dev@dpdk.org
> > > Subject: [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing
> > >
> > > The mlx5 provides the send scheduling on specific moment of time,
> > > and for the related kind of applications it would be extremely
> > > useful to have extra debug information - when and how packets were
> > > scheduled and when the actual sending was completed by the NIC
> > > hardware (it helps application to track the internal delay issues).
> > >
> > > Because the DPDK tx datapath API does not suppose getting any
> > > feedback from the driver and the feature looks like to be mlx5
> > > specific, it seems to be reasonable to engage exisiting DPDK datapath
> tracing capability.
> > >
> > > The work cycle is supposed to be:
> > >   - compile appplication with enabled tracing
> > >   - run application with EAL parameters configuring the tracing in mlx5
> > >     Tx datapath
> > >   - store the dump file with gathered tracing information
> > >   - run analyzing scrypt (in Python) to combine related events (packet
> > >     firing and completion) and see the data in human-readable view
> > >
> > > Below is the detailed instruction "how to" with mlx5 NIC to gather
> > > all the debug data including the full timings information.
> > >
> > >
> > > 1. Build DPDK application with enabled datapath tracing
> > >
> > > The meson option should be specified:
> > >    --enable_trace_fp=true
> > >
> > > The c_args shoudl be specified:
> > >    -DALLOW_EXPERIMENTAL_API
> > >
> > > The DPDK configuration examples:
> > >
> > >   meson configure --buildtype=debug -Denable_trace_fp=true
> > >         -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT -
> > > DALLOW_EXPERIMENTAL_API' build
> > >
> > >   meson configure --buildtype=debug -Denable_trace_fp=true
> > >         -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API'
> > > build
> > >
> > >   meson configure --buildtype=release -Denable_trace_fp=true
> > >         -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API'
> > > build
> > >
> > >   meson configure --buildtype=release -Denable_trace_fp=true
> > >         -Dc_args='-DALLOW_EXPERIMENTAL_API' build
> > >
> > >
> > > 2. Configuring the NIC
> > >
> > > If the sending completion timings are important the NIC should be
> > > configured to provide realtime timestamps, the
> > > REAL_TIME_CLOCK_ENABLE NV settings parameter should be configured
> to
> > > TRUE, for example with command (and with following FW/driver reset):
> > >
> > >   sudo mlxconfig -d /dev/mst/mt4125_pciconf0 s
> > > REAL_TIME_CLOCK_ENABLE=1
> > >
> > >
> > > 3. Run DPDK application to gather the traces
> > >
> > > EAL parameters controlling trace capability in runtime
> > >
> > >   --trace=pmd.net.mlx5.tx - the regular expression enabling the tracepoints
> > >                             with matching names at least "pmd.net.mlx5.tx"
> > >                             must be enabled to gather all events needed
> > >                             to analyze mlx5 Tx datapath and its timings.
> > >                             By default all tracepoints are disabled.
> > >
> > >   --trace-dir=/var/log - trace storing directory
> > >
> > >   --trace-bufsz=<val>B|<val>K|<val>M - optional, trace data buffer size
> > >                                        per thread. The default is 1MB.
> > >
> > >   --trace-mode=overwrite|discard  - optional, selects trace data buffer
> mode.
> > >
> > >
> > > 4. Installing or Building Babeltrace2 Package
> > >
> > > The gathered trace data can be analyzed with a developed Python script.
> > > To parse the trace, the data script uses the Babeltrace2 library.
> > > The package should be either installed or built from source code as
> > > shown below:
> > >
> > >   git clone https://github.com/efficios/babeltrace.git
> > >   cd babeltrace
> > >   ./bootstrap
> > >   ./configure -help
> > >   ./configure --disable-api-doc --disable-man-pages
> > >               --disable-python-bindings-doc --enbale-python-plugins
> > >               --enable-python-binding
> > >
> > > 5. Running the Analyzing Script
> > >
> > > The analyzing script is located in the folder:
> > > ./drivers/net/mlx5/tools It requires Python3.6, Babeltrace2 packages
> > > and it takes the only parameter of trace data file. For example:
> > >
> > >    ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39
> > >
> > >
> > > 6. Interpreting the Script Output Data
> > >
> > > All the timings are given in nanoseconds.
> > > The list of Tx (and coming Rx) bursts per port/queue is presented in
> > > the output.
> > > Each list element contains the list of built WQEs with specific
> > > opcodes, and each WQE contains the list of the encompassed packets to
> send.
> 
> This information should be in the documentation.
OK, should we make this cover-letter part of mlx5.rst?

> 
> I think we should request a review of the Python script from people familiar
> with tracing and from people more familiar with Python scripting for user
> tools.
Would be very helpful, could you recommend/ask someone?

With best regards,
Slava



> 


  reply	other threads:[~2023-06-27 11:24 UTC|newest]

Thread overview: 76+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-04-20 10:07 [RFC " Viacheslav Ovsiienko
2023-04-20 10:07 ` [RFC 1/5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-04-20 10:13   ` Jerin Jacob
2023-04-20 10:08 ` [RFC 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-04-20 10:11   ` Jerin Jacob
2023-06-13 15:50     ` Slava Ovsiienko
2023-06-13 15:53       ` Jerin Jacob
2023-06-13 15:59         ` Slava Ovsiienko
2023-06-13 16:01           ` Jerin Jacob
2023-06-27  0:39             ` Thomas Monjalon
2023-06-27  6:15               ` Slava Ovsiienko
2023-06-27  7:28                 ` Thomas Monjalon
2023-06-27  8:19                   ` Slava Ovsiienko
2023-06-27  9:33                     ` Thomas Monjalon
2023-06-27  9:43                       ` Slava Ovsiienko
2023-06-27 11:36                         ` Thomas Monjalon
2023-04-20 10:08 ` [RFC 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-04-20 10:08 ` [RFC 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-04-20 10:08 ` [RFC 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-09 15:28 ` [PATCH 0/5] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 1/5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-09 15:28   ` [PATCH 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-13 16:58 ` [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 1/5] app/testpmd: add trace save command Viacheslav Ovsiienko
2023-06-21 11:15     ` Ferruh Yigit
2023-06-23  8:00       ` Slava Ovsiienko
2023-06-23 11:52         ` Ferruh Yigit
2023-06-23 12:03           ` Jerin Jacob
2023-06-23 12:14             ` Slava Ovsiienko
2023-06-23 12:23             ` Ferruh Yigit
2023-06-13 16:58   ` [PATCH v2 2/5] common/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 3/5] net/mlx5: add Tx datapath tracing Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 4/5] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-13 16:58   ` [PATCH v2 5/5] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-20 12:00   ` [PATCH v2 0/5] net/mlx5: introduce Tx datapath tracing Raslan Darawsheh
2023-06-27  0:46     ` Thomas Monjalon
2023-06-27 11:24       ` Slava Ovsiienko [this message]
2023-06-27 11:34         ` Thomas Monjalon
2023-06-28 14:18           ` Robin Jarry
2023-06-29  7:16             ` Slava Ovsiienko
2023-06-29  9:08               ` Robin Jarry
2023-06-26 11:06 ` [PATCH] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-26 11:07 ` [PATCH v3] " Viacheslav Ovsiienko
2023-06-26 11:57 ` [PATCH v4] " Viacheslav Ovsiienko
2023-06-27 11:34   ` Ferruh Yigit
2023-06-27 11:39     ` Slava Ovsiienko
2023-06-27 11:58       ` Ferruh Yigit
2023-06-27 14:44     ` [PATCH] app/testpmd: add dump command help message Viacheslav Ovsiienko
2023-06-27 18:03       ` Ferruh Yigit
2023-06-28  9:54         ` [PATCH v2] " Viacheslav Ovsiienko
2023-06-28 13:18           ` Ferruh Yigit
2023-06-27 13:09 ` [PATCH v5] app/testpmd: add trace dump command Viacheslav Ovsiienko
2023-06-27 15:18   ` Ferruh Yigit
2023-06-28 11:09 ` [PATCH v3 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-06-28 11:09   ` [PATCH v3 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-05 11:10 ` [PATCH v4 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-07-05 11:10   ` [PATCH v4 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-07-05 11:10   ` [PATCH v4 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-07-05 11:10   ` [PATCH v4 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-05 11:10   ` [PATCH v4 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-05 15:31 ` [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 2/4] net/mlx5: add comprehensive send completion trace Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 3/4] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-05 15:31   ` [PATCH v5 4/4] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-06 16:27   ` [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing Raslan Darawsheh
2023-07-11 15:15 ` [PATCH v6 0/2] " Viacheslav Ovsiienko
2023-07-11 15:15   ` [PATCH v6 1/2] net/mlx5: add Tx datapath trace analyzing script Viacheslav Ovsiienko
2023-07-11 15:15   ` [PATCH v6 2/2] doc: add mlx5 datapath tracing feature description Viacheslav Ovsiienko
2023-07-27 10:52   ` [PATCH v6 0/2] net/mlx5: introduce Tx datapath tracing Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM6PR12MB3753CB7534423178B902296ADF27A@DM6PR12MB3753.namprd12.prod.outlook.com \
    --to=viacheslavo@nvidia.com \
    --cc=dev@dpdk.org \
    --cc=jerinj@marvell.com \
    --cc=rasland@nvidia.com \
    --cc=rjarry@redhat.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).