DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH] net/mlx5: enable PCI related counters
@ 2024-02-09 20:41 Wathsala Vithanage
  2024-02-09 21:16 ` Honnappa Nagarahalli
                   ` (4 more replies)
  0 siblings, 5 replies; 10+ messages in thread
From: Wathsala Vithanage @ 2024-02-09 20:41 UTC (permalink / raw)
  To: Thomas Monjalon, Dariusz Sosnowski, Viacheslav Ovsiienko,
	Ori Kam, Suanming Mou, Matan Azrad
  Cc: dev, nd, Wathsala Vithanage, Honnappa Nagarahalli

Versions of Mellanox NICs starting from CX5 have device counters
related to PCI. These counters are helpful in debugging IO
bottlenecks. For instance, the outbound_pci_stalled_rd and
outbound_pci_stalled_wr counters can help with identifying NIC
stalls due to insufficient PCI credits, which otherwise would
have required a PCI analyzer or a sophisticated PCI root port
with a PMU.
Currently none of these are available in the MLX5 PMD even
though ethtool is capable of reading some of them.
Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
reads via the kernel driver it is possible to add support with
ease.
There is one more PCI related counter and a device counter that
aren't implemented in the Linux driver at the moment. These two
are named outbound_pci_buffer_overflow and dev_out_of_buffer
respectively. As per Nvidia's documentation these two counters
can tell the number of packets dropped due to pci buffer
overflow and the number of times the device owned queue had not
enough buffers allocated.

Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                                |  1 +
 drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33 +++++++++++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/.mailmap b/.mailmap
index aa569ff456..f57415f7a1 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
 Wang Sheng-Hui <shhuiw@gmail.com>
 Wangyu (Eric) <seven.wangyu@huawei.com>
 Waterman Cao <waterman.cao@intel.com>
+Wathsala Vithanage <wathsala.vithanage@arm.com>
 Weichun Chen <weichunx.chen@intel.com>
 Wei Dai <wei.dai@intel.com>
 Weifeng Li <liweifeng96@126.com>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index dd5a0c546d..8f1567f6a7 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
 		.dpdk_name = "tx_vport_bytes",
 		.ctr_name = "vport_tx_bytes",
 	},
+	/* Device counters */
+	{
+		.dpdk_name = "rx_pci_signal_integrity",
+		.ctr_name = "rx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "tx_pci_signal_integrity",
+		.ctr_name = "tx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "outbound_pci_buffer_overflow",
+		.ctr_name = "outbound_pci_buffer_overflow",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd",
+		.ctr_name = "outbound_pci_stalled_rd",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr",
+		.ctr_name = "outbound_pci_stalled_wr",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd_events",
+		.ctr_name = "outbound_pci_stalled_rd_events",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr_events",
+		.ctr_name = "outbound_pci_stalled_wr_events",
+	},
+	{
+		.dpdk_name = "dev_out_of_buffer",
+		.ctr_name = "dev_out_of_buffer",
+	},
 };
 
 static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
-- 
2.25.1


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] net/mlx5: enable PCI related counters
  2024-02-09 20:41 [PATCH] net/mlx5: enable PCI related counters Wathsala Vithanage
@ 2024-02-09 21:16 ` Honnappa Nagarahalli
  2024-02-10  1:32 ` Stephen Hemminger
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 10+ messages in thread
From: Honnappa Nagarahalli @ 2024-02-09 21:16 UTC (permalink / raw)
  To: Wathsala Wathawana Vithanage
  Cc: thomas, Dariusz Sosnowski, Viacheslav Ovsiienko, Ori Kam,
	Suanming Mou, Matan Azrad, dev, nd



> On Feb 9, 2024, at 2:41 PM, Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:
> 
> Versions of Mellanox NICs starting from CX5 have device counters
> related to PCI. These counters are helpful in debugging IO
> bottlenecks. For instance, the outbound_pci_stalled_rd and
> outbound_pci_stalled_wr counters can help with identifying NIC
> stalls due to insufficient PCI credits, which otherwise would
> have required a PCI analyzer or a sophisticated PCI root port
> with a PMU.
> Currently none of these are available in the MLX5 PMD even
> though ethtool is capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
> reads via the kernel driver it is possible to add support with
> ease.
> There is one more PCI related counter and a device counter that
> aren't implemented in the Linux driver at the moment. These two
> are named outbound_pci_buffer_overflow and dev_out_of_buffer
> respectively. As per Nvidia's documentation these two counters
> can tell the number of packets dropped due to pci buffer
> overflow and the number of times the device owned queue had not
> enough buffers allocated.
It would be good to see more of the PCI counters added in other NIC drivers as well. It helps significantly with debugging.

> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
> .mailmap                                |  1 +
> drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33 +++++++++++++++++++++++++
> 2 files changed, 34 insertions(+)
> 
> diff --git a/.mailmap b/.mailmap
> index aa569ff456..f57415f7a1 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
> Wang Sheng-Hui <shhuiw@gmail.com>
> Wangyu (Eric) <seven.wangyu@huawei.com>
> Waterman Cao <waterman.cao@intel.com>
> +Wathsala Vithanage <wathsala.vithanage@arm.com>
> Weichun Chen <weichunx.chen@intel.com>
> Wei Dai <wei.dai@intel.com>
> Weifeng Li <liweifeng96@126.com>
> diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> index dd5a0c546d..8f1567f6a7 100644
> --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> @@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
> .dpdk_name = "tx_vport_bytes",
> .ctr_name = "vport_tx_bytes",
> },
> + /* Device counters */
> + {
> + .dpdk_name = "rx_pci_signal_integrity",
> + .ctr_name = "rx_pci_signal_integrity",
> + },
> + {
> + .dpdk_name = "tx_pci_signal_integrity",
> + .ctr_name = "tx_pci_signal_integrity",
> + },
> + {
> + .dpdk_name = "outbound_pci_buffer_overflow",
> + .ctr_name = "outbound_pci_buffer_overflow",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_rd",
> + .ctr_name = "outbound_pci_stalled_rd",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_wr",
> + .ctr_name = "outbound_pci_stalled_wr",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_rd_events",
> + .ctr_name = "outbound_pci_stalled_rd_events",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_wr_events",
> + .ctr_name = "outbound_pci_stalled_wr_events",
> + },
> + {
> + .dpdk_name = "dev_out_of_buffer",
> + .ctr_name = "dev_out_of_buffer",
> + },
> };
> 
> static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
> -- 
> 2.25.1
> 


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] net/mlx5: enable PCI related counters
  2024-02-09 20:41 [PATCH] net/mlx5: enable PCI related counters Wathsala Vithanage
  2024-02-09 21:16 ` Honnappa Nagarahalli
@ 2024-02-10  1:32 ` Stephen Hemminger
  2024-02-13 16:13   ` Dariusz Sosnowski
  2024-02-13 13:12 ` Slava Ovsiienko
                   ` (2 subsequent siblings)
  4 siblings, 1 reply; 10+ messages in thread
From: Stephen Hemminger @ 2024-02-10  1:32 UTC (permalink / raw)
  To: Wathsala Vithanage
  Cc: Thomas Monjalon, Dariusz Sosnowski, Viacheslav Ovsiienko,
	Ori Kam, Suanming Mou, Matan Azrad, dev, nd,
	Honnappa Nagarahalli

On Fri,  9 Feb 2024 20:41:42 +0000
Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:

> Versions of Mellanox NICs starting from CX5 have device counters
> related to PCI. These counters are helpful in debugging IO
> bottlenecks. For instance, the outbound_pci_stalled_rd and
> outbound_pci_stalled_wr counters can help with identifying NIC
> stalls due to insufficient PCI credits, which otherwise would
> have required a PCI analyzer or a sophisticated PCI root port
> with a PMU.
> Currently none of these are available in the MLX5 PMD even
> though ethtool is capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
> reads via the kernel driver it is possible to add support with
> ease.
> There is one more PCI related counter and a device counter that
> aren't implemented in the Linux driver at the moment. These two
> are named outbound_pci_buffer_overflow and dev_out_of_buffer
> respectively. As per Nvidia's documentation these two counters
> can tell the number of packets dropped due to pci buffer
> overflow and the number of times the device owned queue had not
> enough buffers allocated.
> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>

Would it be possible to do this at PCI bus layer so all PCI devices have
that feature?

^ permalink raw reply	[flat|nested] 10+ messages in thread

* RE: [PATCH] net/mlx5: enable PCI related counters
  2024-02-09 20:41 [PATCH] net/mlx5: enable PCI related counters Wathsala Vithanage
  2024-02-09 21:16 ` Honnappa Nagarahalli
  2024-02-10  1:32 ` Stephen Hemminger
@ 2024-02-13 13:12 ` Slava Ovsiienko
  2024-02-14  1:50   ` Honnappa Nagarahalli
  2024-02-13 16:17 ` Dariusz Sosnowski
  2024-02-14 20:14 ` [PATCH v2] " Wathsala Vithanage
  4 siblings, 1 reply; 10+ messages in thread
From: Slava Ovsiienko @ 2024-02-13 13:12 UTC (permalink / raw)
  To: Wathsala Vithanage, NBU-Contact-Thomas Monjalon (EXTERNAL),
	Dariusz Sosnowski, Ori Kam, Suanming Mou, Matan Azrad
  Cc: dev, nd, Honnappa Nagarahalli

Hi,

Regarding "dev_out_of_buffer" - it is global counter, relates to the whole device port,
Including queues not managed by DPDK application - Mellanox/Nvidia NICs operate
In "bifurcated mode" - there might be queues managed by kernel or another DPDK
application. Not sure it makes a lot of sense, but I have no strong objections.

The PCI related counters are also global ones and reflect statistics, impacted by
PCI activity of the whole physical device, including all the network ports located
on the same NIC board (and, sometimes, by internal activity in BlueField).

As I said, no objections from my side:

Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

With best regards,
Slava

> -----Original Message-----
> From: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Sent: Friday, February 9, 2024 10:42 PM
> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: dev@dpdk.org; nd@arm.com; Wathsala Vithanage
> <wathsala.vithanage@arm.com>; Honnappa Nagarahalli
> <honnappa.nagarahalli@arm.com>
> Subject: [PATCH] net/mlx5: enable PCI related counters
> 
> Versions of Mellanox NICs starting from CX5 have device counters related to PCI.
> These counters are helpful in debugging IO bottlenecks. For instance, the
> outbound_pci_stalled_rd and outbound_pci_stalled_wr counters can help with
> identifying NIC stalls due to insufficient PCI credits, which otherwise would have
> required a PCI analyzer or a sophisticated PCI root port with a PMU.
> Currently none of these are available in the MLX5 PMD even though ethtool is
> capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads via the
> kernel driver it is possible to add support with ease.
> There is one more PCI related counter and a device counter that aren't
> implemented in the Linux driver at the moment. These two are named
> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As per
> Nvidia's documentation these two counters can tell the number of packets
> dropped due to pci buffer overflow and the number of times the device owned
> queue had not enough buffers allocated.
> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
>  .mailmap                                |  1 +
>  drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33
> +++++++++++++++++++++++++
>  2 files changed, 34 insertions(+)
> 
> diff --git a/.mailmap b/.mailmap
> index aa569ff456..f57415f7a1 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
> Wang Sheng-Hui <shhuiw@gmail.com>  Wangyu (Eric)
> <seven.wangyu@huawei.com>  Waterman Cao <waterman.cao@intel.com>
> +Wathsala Vithanage <wathsala.vithanage@arm.com>
>  Weichun Chen <weichunx.chen@intel.com>
>  Wei Dai <wei.dai@intel.com>
>  Weifeng Li <liweifeng96@126.com>
> diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> index dd5a0c546d..8f1567f6a7 100644
> --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> @@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl
> mlx5_counters_init[] = {
>  		.dpdk_name = "tx_vport_bytes",
>  		.ctr_name = "vport_tx_bytes",
>  	},
> +	/* Device counters */
> +	{
> +		.dpdk_name = "rx_pci_signal_integrity",
> +		.ctr_name = "rx_pci_signal_integrity",
> +	},
> +	{
> +		.dpdk_name = "tx_pci_signal_integrity",
> +		.ctr_name = "tx_pci_signal_integrity",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_buffer_overflow",
> +		.ctr_name = "outbound_pci_buffer_overflow",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_rd",
> +		.ctr_name = "outbound_pci_stalled_rd",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_wr",
> +		.ctr_name = "outbound_pci_stalled_wr",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_rd_events",
> +		.ctr_name = "outbound_pci_stalled_rd_events",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_wr_events",
> +		.ctr_name = "outbound_pci_stalled_wr_events",
> +	},
> +	{
> +		.dpdk_name = "dev_out_of_buffer",
> +		.ctr_name = "dev_out_of_buffer",
> +	},
>  };
> 
>  static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
> --
> 2.25.1


^ permalink raw reply	[flat|nested] 10+ messages in thread

* RE: [PATCH] net/mlx5: enable PCI related counters
  2024-02-10  1:32 ` Stephen Hemminger
@ 2024-02-13 16:13   ` Dariusz Sosnowski
  2024-02-14  0:52     ` Honnappa Nagarahalli
  0 siblings, 1 reply; 10+ messages in thread
From: Dariusz Sosnowski @ 2024-02-13 16:13 UTC (permalink / raw)
  To: Stephen Hemminger, Wathsala Vithanage
  Cc: NBU-Contact-Thomas Monjalon (EXTERNAL),
	Slava Ovsiienko, Ori Kam, Suanming Mou, Matan Azrad, dev, nd,
	Honnappa Nagarahalli

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Saturday, February 10, 2024 02:33
> To: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>;
> dev@dpdk.org; nd@arm.com; Honnappa Nagarahalli
> <honnappa.nagarahalli@arm.com>
> Subject: Re: [PATCH] net/mlx5: enable PCI related counters
> 
> On Fri,  9 Feb 2024 20:41:42 +0000
> Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:
> 
> > Versions of Mellanox NICs starting from CX5 have device counters
> > related to PCI. These counters are helpful in debugging IO
> > bottlenecks. For instance, the outbound_pci_stalled_rd and
> > outbound_pci_stalled_wr counters can help with identifying NIC stalls
> > due to insufficient PCI credits, which otherwise would have required a
> > PCI analyzer or a sophisticated PCI root port with a PMU.
> > Currently none of these are available in the MLX5 PMD even though
> > ethtool is capable of reading some of them.
> > Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads
> > via the kernel driver it is possible to add support with ease.
> > There is one more PCI related counter and a device counter that aren't
> > implemented in the Linux driver at the moment. These two are named
> > outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As
> > per Nvidia's documentation these two counters can tell the number of
> > packets dropped due to pci buffer overflow and the number of times the
> > device owned queue had not enough buffers allocated.
> >
> > Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> 
> Would it be possible to do this at PCI bus layer so all PCI devices have that
> feature?
PCIe performance counters mentioned here are exposed by the NIC itself and mlx5 kernel driver just passes them to userspace.
If such a feature would be added at PCI bus layer, we would need to use (or add) some additional infrastructure.
I'm not familiar with what Linux kernel exposes in terms of PCI counters. It's worth looking into.
I'd assume such data can probably be extracted through PMU.

Best regards,
Dariusz Sosnowski

^ permalink raw reply	[flat|nested] 10+ messages in thread

* RE: [PATCH] net/mlx5: enable PCI related counters
  2024-02-09 20:41 [PATCH] net/mlx5: enable PCI related counters Wathsala Vithanage
                   ` (2 preceding siblings ...)
  2024-02-13 13:12 ` Slava Ovsiienko
@ 2024-02-13 16:17 ` Dariusz Sosnowski
  2024-02-14 20:14 ` [PATCH v2] " Wathsala Vithanage
  4 siblings, 0 replies; 10+ messages in thread
From: Dariusz Sosnowski @ 2024-02-13 16:17 UTC (permalink / raw)
  To: Wathsala Vithanage, NBU-Contact-Thomas Monjalon (EXTERNAL),
	Slava Ovsiienko, Ori Kam, Suanming Mou, Matan Azrad
  Cc: dev, nd, Honnappa Nagarahalli

> -----Original Message-----
> From: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Sent: Friday, February 9, 2024 21:42
> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: dev@dpdk.org; nd@arm.com; Wathsala Vithanage
> <wathsala.vithanage@arm.com>; Honnappa Nagarahalli
> <honnappa.nagarahalli@arm.com>
> Subject: [PATCH] net/mlx5: enable PCI related counters
> 
> Versions of Mellanox NICs starting from CX5 have device counters related to
> PCI. These counters are helpful in debugging IO bottlenecks. For instance, the
> outbound_pci_stalled_rd and outbound_pci_stalled_wr counters can help
> with identifying NIC stalls due to insufficient PCI credits, which otherwise
> would have required a PCI analyzer or a sophisticated PCI root port with a
> PMU.
> Currently none of these are available in the MLX5 PMD even though ethtool is
> capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads via
> the kernel driver it is possible to add support with ease.
> There is one more PCI related counter and a device counter that aren't
> implemented in the Linux driver at the moment. These two are named
> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As per
> Nvidia's documentation these two counters can tell the number of packets
> dropped due to pci buffer overflow and the number of times the device
> owned queue had not enough buffers allocated.
> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>

Best regards,
Dariusz Sosnowski

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] net/mlx5: enable PCI related counters
  2024-02-13 16:13   ` Dariusz Sosnowski
@ 2024-02-14  0:52     ` Honnappa Nagarahalli
  0 siblings, 0 replies; 10+ messages in thread
From: Honnappa Nagarahalli @ 2024-02-14  0:52 UTC (permalink / raw)
  To: Dariusz Sosnowski
  Cc: Stephen Hemminger, Wathsala Wathawana Vithanage, thomas,
	Slava Ovsiienko, Ori Kam, Suanming Mou, Matan Azrad, dev, nd



> On Feb 13, 2024, at 10:13 AM, Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:
> 
>> -----Original Message-----
>> From: Stephen Hemminger <stephen@networkplumber.org>
>> Sent: Saturday, February 10, 2024 02:33
>> To: Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
>> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
>> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
>> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>;
>> dev@dpdk.org; nd@arm.com; Honnappa Nagarahalli
>> <honnappa.nagarahalli@arm.com>
>> Subject: Re: [PATCH] net/mlx5: enable PCI related counters
>> 
>> On Fri,  9 Feb 2024 20:41:42 +0000
>> Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:
>> 
>>> Versions of Mellanox NICs starting from CX5 have device counters
>>> related to PCI. These counters are helpful in debugging IO
>>> bottlenecks. For instance, the outbound_pci_stalled_rd and
>>> outbound_pci_stalled_wr counters can help with identifying NIC stalls
>>> due to insufficient PCI credits, which otherwise would have required a
>>> PCI analyzer or a sophisticated PCI root port with a PMU.
>>> Currently none of these are available in the MLX5 PMD even though
>>> ethtool is capable of reading some of them.
>>> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads
>>> via the kernel driver it is possible to add support with ease.
>>> There is one more PCI related counter and a device counter that aren't
>>> implemented in the Linux driver at the moment. These two are named
>>> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As
>>> per Nvidia's documentation these two counters can tell the number of
>>> packets dropped due to pci buffer overflow and the number of times the
>>> device owned queue had not enough buffers allocated.
>>> 
>>> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> 
>> Would it be possible to do this at PCI bus layer so all PCI devices have that
>> feature?
> PCIe performance counters mentioned here are exposed by the NIC itself and mlx5 kernel driver just passes them to userspace.
> If such a feature would be added at PCI bus layer, we would need to use (or add) some additional infrastructure.
> I'm not familiar with what Linux kernel exposes in terms of PCI counters. It's worth looking into.
> I'd assume such data can probably be extracted through PMU.
In our investigation, we did not find anything that Linux provides in terms of PCIe PMUs on PCIe root port. The best we found was these PCIe counters as seen by NIC.

It would be good to see other NICs providing similar and additional counters if any.



> 
> Best regards,
> Dariusz Sosnowski


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] net/mlx5: enable PCI related counters
  2024-02-13 13:12 ` Slava Ovsiienko
@ 2024-02-14  1:50   ` Honnappa Nagarahalli
  0 siblings, 0 replies; 10+ messages in thread
From: Honnappa Nagarahalli @ 2024-02-14  1:50 UTC (permalink / raw)
  To: Slava Ovsiienko
  Cc: Wathsala Wathawana Vithanage, thomas, Dariusz Sosnowski, Ori Kam,
	Suanming Mou, Matan Azrad, dev, nd



> On Feb 13, 2024, at 7:12 AM, Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> 
> Hi,
> 
> Regarding "dev_out_of_buffer" - it is global counter, relates to the whole device port,
> Including queues not managed by DPDK application - Mellanox/Nvidia NICs operate
> In "bifurcated mode" - there might be queues managed by kernel or another DPDK
> application. Not sure it makes a lot of sense, but I have no strong objections.
These are still helpful to debug in lab environment. But, it would be good to document these.

> 
> The PCI related counters are also global ones and reflect statistics, impacted by
> PCI activity of the whole physical device, including all the network ports located
> on the same NIC board (and, sometimes, by internal activity in BlueField).
> 
> As I said, no objections from my side:
> 
> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> 
> With best regards,
> Slava
> 
>> -----Original Message-----
>> From: Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Sent: Friday, February 9, 2024 10:42 PM
>> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
>> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
>> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
>> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
>> Cc: dev@dpdk.org; nd@arm.com; Wathsala Vithanage
>> <wathsala.vithanage@arm.com>; Honnappa Nagarahalli
>> <honnappa.nagarahalli@arm.com>
>> Subject: [PATCH] net/mlx5: enable PCI related counters
>> 
>> Versions of Mellanox NICs starting from CX5 have device counters related to PCI.
>> These counters are helpful in debugging IO bottlenecks. For instance, the
>> outbound_pci_stalled_rd and outbound_pci_stalled_wr counters can help with
>> identifying NIC stalls due to insufficient PCI credits, which otherwise would have
>> required a PCI analyzer or a sophisticated PCI root port with a PMU.
>> Currently none of these are available in the MLX5 PMD even though ethtool is
>> capable of reading some of them.
>> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads via the
>> kernel driver it is possible to add support with ease.
>> There is one more PCI related counter and a device counter that aren't
>> implemented in the Linux driver at the moment. These two are named
>> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As per
>> Nvidia's documentation these two counters can tell the number of packets
>> dropped due to pci buffer overflow and the number of times the device owned
>> queue had not enough buffers allocated.
>> 
>> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> ---
>> .mailmap                                |  1 +
>> drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33
>> +++++++++++++++++++++++++
>> 2 files changed, 34 insertions(+)
>> 
>> diff --git a/.mailmap b/.mailmap
>> index aa569ff456..f57415f7a1 100644
>> --- a/.mailmap
>> +++ b/.mailmap
>> @@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
>> Wang Sheng-Hui <shhuiw@gmail.com>  Wangyu (Eric)
>> <seven.wangyu@huawei.com>  Waterman Cao <waterman.cao@intel.com>
>> +Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Weichun Chen <weichunx.chen@intel.com>
>> Wei Dai <wei.dai@intel.com>
>> Weifeng Li <liweifeng96@126.com>
>> diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> index dd5a0c546d..8f1567f6a7 100644
>> --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> @@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl
>> mlx5_counters_init[] = {
>> .dpdk_name = "tx_vport_bytes",
>> .ctr_name = "vport_tx_bytes",
>> },
>> + /* Device counters */
>> + {
>> + .dpdk_name = "rx_pci_signal_integrity",
>> + .ctr_name = "rx_pci_signal_integrity",
>> + },
>> + {
>> + .dpdk_name = "tx_pci_signal_integrity",
>> + .ctr_name = "tx_pci_signal_integrity",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_buffer_overflow",
>> + .ctr_name = "outbound_pci_buffer_overflow",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_rd",
>> + .ctr_name = "outbound_pci_stalled_rd",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_wr",
>> + .ctr_name = "outbound_pci_stalled_wr",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_rd_events",
>> + .ctr_name = "outbound_pci_stalled_rd_events",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_wr_events",
>> + .ctr_name = "outbound_pci_stalled_wr_events",
>> + },
>> + {
>> + .dpdk_name = "dev_out_of_buffer",
>> + .ctr_name = "dev_out_of_buffer",
>> + },
>> };
>> 
>> static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
>> --
>> 2.25.1
> 


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v2] net/mlx5: enable PCI related counters
  2024-02-09 20:41 [PATCH] net/mlx5: enable PCI related counters Wathsala Vithanage
                   ` (3 preceding siblings ...)
  2024-02-13 16:17 ` Dariusz Sosnowski
@ 2024-02-14 20:14 ` Wathsala Vithanage
  2024-02-15 18:26   ` [PATCH v3] " Wathsala Vithanage
  4 siblings, 1 reply; 10+ messages in thread
From: Wathsala Vithanage @ 2024-02-14 20:14 UTC (permalink / raw)
  To: Thomas Monjalon, Dariusz Sosnowski, Viacheslav Ovsiienko,
	Ori Kam, Suanming Mou, Matan Azrad
  Cc: dev, nd, Wathsala Vithanage, Honnappa Nagarahalli

Versions of Mellanox NICs starting from CX5 have device counters
related to PCI. These counters are helpful in debugging IO
bottlenecks. For instance, the outbound_pci_stalled_rd and
outbound_pci_stalled_wr counters can help with identifying NIC
stalls due to insufficient PCI credits, which otherwise would
have required a PCI analyzer or a sophisticated PCI root port
with a PMU.
Currently none of these are available in the MLX5 PMD even
though ethtool is capable of reading some of them.
Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
reads via the kernel driver it is possible to add support with
ease.
There is one more PCI related counter and a device counter that
aren't implemented in the Linux driver at the moment. These two
are named outbound_pci_buffer_overflow and dev_out_of_buffer
respectively. As per Nvidia's documentation these two counters
can tell the number of packets dropped due to pci buffer
overflow and the number of times the device owned queue had not
enough buffers allocated.

Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 .mailmap                                |  1 +
 drivers/net/mlx5/linux/mlx5_ethdev_os.c | 41 +++++++++++++++++++++++++
 2 files changed, 42 insertions(+)

diff --git a/.mailmap b/.mailmap
index aa569ff456..f57415f7a1 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
 Wang Sheng-Hui <shhuiw@gmail.com>
 Wangyu (Eric) <seven.wangyu@huawei.com>
 Waterman Cao <waterman.cao@intel.com>
+Wathsala Vithanage <wathsala.vithanage@arm.com>
 Weichun Chen <weichunx.chen@intel.com>
 Wei Dai <wei.dai@intel.com>
 Weifeng Li <liweifeng96@126.com>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index dd5a0c546d..c837c862a8 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -1574,6 +1574,47 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
 		.dpdk_name = "tx_vport_bytes",
 		.ctr_name = "vport_tx_bytes",
 	},
+	/**
+	 * Device counters: These counters are for the
+	 * entire PCI device (NIC). These counters are
+	 * not counting on a per port/queue basis.
+	 * Values reported by these counters may not be
+	 * useful if the device is bifurcated and queues
+	 * are shared with the kernel or other DPDK
+	 * applications.
+	 */
+	{
+		.dpdk_name = "rx_pci_signal_integrity",
+		.ctr_name = "rx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "tx_pci_signal_integrity",
+		.ctr_name = "tx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "outbound_pci_buffer_overflow",
+		.ctr_name = "outbound_pci_buffer_overflow",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd",
+		.ctr_name = "outbound_pci_stalled_rd",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr",
+		.ctr_name = "outbound_pci_stalled_wr",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd_events",
+		.ctr_name = "outbound_pci_stalled_rd_events",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr_events",
+		.ctr_name = "outbound_pci_stalled_wr_events",
+	},
+	{
+		.dpdk_name = "dev_out_of_buffer",
+		.ctr_name = "dev_out_of_buffer",
+	},
 };
 
 static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
-- 
2.25.1


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v3] net/mlx5: enable PCI related counters
  2024-02-14 20:14 ` [PATCH v2] " Wathsala Vithanage
@ 2024-02-15 18:26   ` Wathsala Vithanage
  0 siblings, 0 replies; 10+ messages in thread
From: Wathsala Vithanage @ 2024-02-15 18:26 UTC (permalink / raw)
  To: Thomas Monjalon, Dariusz Sosnowski, Viacheslav Ovsiienko,
	Ori Kam, Suanming Mou, Matan Azrad
  Cc: dev, nd, Wathsala Vithanage, Honnappa Nagarahalli

Versions of Mellanox NICs starting from CX5 have device counters
related to PCI. These counters are helpful in debugging IO
bottlenecks. For instance, the outbound_pci_stalled_rd and
outbound_pci_stalled_wr counters can help with identifying NIC
stalls due to insufficient PCI credits, which otherwise would
have required a PCI analyzer or a sophisticated PCI root port
with a PMU.
Currently none of these are available in the MLX5 PMD even
though ethtool is capable of reading some of them.
Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
reads via the kernel driver it is possible to add support with
ease.
There is one more PCI related counter and a device counter that
aren't implemented in the Linux driver at the moment. These two
are named outbound_pci_buffer_overflow and dev_out_of_buffer
respectively. As per Nvidia's documentation these two counters
can tell the number of packets dropped due to pci buffer
overflow and the number of times the device owned queue had not
enough buffers allocated.

Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 .mailmap                                |  1 +
 drivers/net/mlx5/linux/mlx5_ethdev_os.c | 37 +++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/.mailmap b/.mailmap
index aa569ff456..f57415f7a1 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
 Wang Sheng-Hui <shhuiw@gmail.com>
 Wangyu (Eric) <seven.wangyu@huawei.com>
 Waterman Cao <waterman.cao@intel.com>
+Wathsala Vithanage <wathsala.vithanage@arm.com>
 Weichun Chen <weichunx.chen@intel.com>
 Wei Dai <wei.dai@intel.com>
 Weifeng Li <liweifeng96@126.com>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index dd5a0c546d..92c47a3b3d 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -1574,6 +1574,43 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
 		.dpdk_name = "tx_vport_bytes",
 		.ctr_name = "vport_tx_bytes",
 	},
+	/**
+	 * Device counters: These counters are for the
+	 * entire PCI device (NIC). These counters are
+	 * not counting on a per port/queue basis.
+	 */
+	{
+		.dpdk_name = "rx_pci_signal_integrity",
+		.ctr_name = "rx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "tx_pci_signal_integrity",
+		.ctr_name = "tx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "outbound_pci_buffer_overflow",
+		.ctr_name = "outbound_pci_buffer_overflow",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd",
+		.ctr_name = "outbound_pci_stalled_rd",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr",
+		.ctr_name = "outbound_pci_stalled_wr",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd_events",
+		.ctr_name = "outbound_pci_stalled_rd_events",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr_events",
+		.ctr_name = "outbound_pci_stalled_wr_events",
+	},
+	{
+		.dpdk_name = "dev_out_of_buffer",
+		.ctr_name = "dev_out_of_buffer",
+	},
 };
 
 static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
-- 
2.25.1


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2024-02-15 18:26 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-09 20:41 [PATCH] net/mlx5: enable PCI related counters Wathsala Vithanage
2024-02-09 21:16 ` Honnappa Nagarahalli
2024-02-10  1:32 ` Stephen Hemminger
2024-02-13 16:13   ` Dariusz Sosnowski
2024-02-14  0:52     ` Honnappa Nagarahalli
2024-02-13 13:12 ` Slava Ovsiienko
2024-02-14  1:50   ` Honnappa Nagarahalli
2024-02-13 16:17 ` Dariusz Sosnowski
2024-02-14 20:14 ` [PATCH v2] " Wathsala Vithanage
2024-02-15 18:26   ` [PATCH v3] " Wathsala Vithanage

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).