RE: [RFC PATCH 0/5] Introduce mempool object new debug capabilities

[Shani Peretz <shperetz@nvidia.com>]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Monday, 16 June 2025 18:30
> To: Shani Peretz <shperetz@nvidia.com>
> Cc: dev@dpdk.org
> Subject: Re: [RFC PATCH 0/5] Introduce mempool object new debug
> capabilities
> 
> External email: Use caution opening links or attachments
> 
> 
> On Mon, 16 Jun 2025 10:29:05 +0300
> Shani Peretz <shperetz@nvidia.com> wrote:
> 
> > This feature is designed to monitor the lifecycle of mempool objects
> > as they move between the application and the PMD.
> >
> > It will allow us to track the operations and transitions of each
> > mempool object throughout the system, helping in debugging and
> understanding objects flow.
> >
> > The implementation include several key components:
> > 1. Added a bitmap to mempool's header (rte_mempool_objhdr)
> >    that represent the operations history.
> > 2. Added functions that allow marking operations on an
> >    mempool objects.
> > 3. Dumps the history to a file or the console
> >    (rte_mempool_objects_dump).
> > 4. Added python script that can parse, analyze the data and
> >    present it in an human readable format.
> > 5. Added compilation flag to enable the feature.
> >
> > Shani Peretz (5):
> >   mempool: record mempool objects operations history
> >   drivers: add mempool history compilation flag
> >   net/mlx5: mark an operation in mempool object's history
> >   app/testpmd: add testpmd command to dump mempool history
> >   usertool: add a script to parse mempool history dump
> >
> >  app/test-pmd/cmdline.c                        |  59 +++++++-
> >  config/meson.build                            |   1 +
> >  drivers/meson.build                           |   7 +
> >  drivers/net/af_packet/meson.build             |   1 +
> >  drivers/net/af_xdp/meson.build                |   1 +
> >  drivers/net/ark/meson.build                   |   2 +
> >  drivers/net/atlantic/meson.build              |   2 +
> >  drivers/net/avp/meson.build                   |   2 +
> >  drivers/net/axgbe/meson.build                 |   2 +
> >  drivers/net/bnx2x/meson.build                 |   1 +
> >  drivers/net/bnxt/meson.build                  |   2 +
> >  drivers/net/bonding/meson.build               |   1 +
> >  drivers/net/cnxk/meson.build                  |   1 +
> >  drivers/net/cxgbe/meson.build                 |   2 +
> >  drivers/net/dpaa/meson.build                  |   2 +
> >  drivers/net/dpaa2/meson.build                 |   2 +
> >  drivers/net/ena/meson.build                   |   2 +
> >  drivers/net/enetc/meson.build                 |   2 +
> >  drivers/net/enetfec/meson.build               |   2 +
> >  drivers/net/enic/meson.build                  |   2 +
> >  drivers/net/failsafe/meson.build              |   1 +
> >  drivers/net/gve/meson.build                   |   2 +
> >  drivers/net/hinic/meson.build                 |   2 +
> >  drivers/net/hns3/meson.build                  |   1 +
> >  drivers/net/intel/cpfl/meson.build            |   2 +
> >  drivers/net/intel/e1000/meson.build           |   2 +
> >  drivers/net/intel/fm10k/meson.build           |   2 +
> >  drivers/net/intel/i40e/meson.build            |   2 +
> >  drivers/net/intel/iavf/meson.build            |   2 +
> >  drivers/net/intel/ice/meson.build             |   1 +
> >  drivers/net/intel/idpf/meson.build            |   2 +
> >  drivers/net/intel/ixgbe/meson.build           |   2 +
> >  drivers/net/ionic/meson.build                 |   2 +
> >  drivers/net/mana/meson.build                  |   2 +
> >  drivers/net/memif/meson.build                 |   1 +
> >  drivers/net/mlx4/meson.build                  |   2 +
> >  drivers/net/mlx5/meson.build                  |   1 +
> >  drivers/net/mlx5/mlx5_rx.c                    |   9 ++
> >  drivers/net/mlx5/mlx5_rx.h                    |   2 +
> >  drivers/net/mlx5/mlx5_rxq.c                   |   9 +-
> >  drivers/net/mlx5/mlx5_rxtx_vec.c              |   6 +
> >  drivers/net/mlx5/mlx5_tx.h                    |   7 +
> >  drivers/net/mlx5/mlx5_txq.c                   |   1 +
> >  drivers/net/mvneta/meson.build                |   2 +
> >  drivers/net/mvpp2/meson.build                 |   2 +
> >  drivers/net/netvsc/meson.build                |   2 +
> >  drivers/net/nfb/meson.build                   |   2 +
> >  drivers/net/nfp/meson.build                   |   2 +
> >  drivers/net/ngbe/meson.build                  |   2 +
> >  drivers/net/ntnic/meson.build                 |   4 +
> >  drivers/net/null/meson.build                  |   1 +
> >  drivers/net/octeon_ep/meson.build             |   2 +
> >  drivers/net/octeontx/meson.build              |   2 +
> >  drivers/net/pcap/meson.build                  |   1 +
> >  drivers/net/pfe/meson.build                   |   2 +
> >  drivers/net/qede/meson.build                  |   2 +
> >  drivers/net/r8169/meson.build                 |   4 +-
> >  drivers/net/ring/meson.build                  |   1 +
> >  drivers/net/sfc/meson.build                   |   2 +
> >  drivers/net/softnic/meson.build               |   2 +
> >  drivers/net/tap/meson.build                   |   1 +
> >  drivers/net/thunderx/meson.build              |   2 +
> >  drivers/net/txgbe/meson.build                 |   2 +
> >  drivers/net/vdev_netvsc/meson.build           |   2 +
> >  drivers/net/vhost/meson.build                 |   2 +
> >  drivers/net/virtio/meson.build                |   2 +
> >  drivers/net/vmxnet3/meson.build               |   2 +
> >  drivers/net/xsc/meson.build                   |   2 +
> >  drivers/net/zxdh/meson.build                  |   4 +
> >  lib/ethdev/rte_ethdev.h                       |  14 ++
> >  lib/mempool/rte_mempool.c                     | 111 +++++++++++++++
> >  lib/mempool/rte_mempool.h                     | 106 ++++++++++++++
> >  meson_options.txt                             |   2 +
> >  .../dpdk-mempool_object_history_parser.py     | 129
> ++++++++++++++++++
> >  74 files changed, 571 insertions(+), 4 deletions(-)  create mode
> > 100755 usertools/dpdk-mempool_object_history_parser.py
> >
> 
> Could this not already be done with tracing infrastructure?

Hey,
We did consider tracing but:
	- It has limited capacity, which will result in older mbufs being lost in the tracing output while they are still in use 
	- Some operations may be lost, and we might not capture the complete picture due to trace misses caused by the performance overhead of tracking on the datapath as far as I understand
WDYT?