[PATCH] net/mlx5: enable PCI related counters

[PATCH] net/mlx5: enable PCI related counters

[Wathsala Vithanage]

Versions of Mellanox NICs starting from CX5 have device counters
related to PCI. These counters are helpful in debugging IO
bottlenecks. For instance, the outbound_pci_stalled_rd and
outbound_pci_stalled_wr counters can help with identifying NIC
stalls due to insufficient PCI credits, which otherwise would
have required a PCI analyzer or a sophisticated PCI root port
with a PMU.
Currently none of these are available in the MLX5 PMD even
though ethtool is capable of reading some of them.
Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
reads via the kernel driver it is possible to add support with
ease.
There is one more PCI related counter and a device counter that
aren't implemented in the Linux driver at the moment. These two
are named outbound_pci_buffer_overflow and dev_out_of_buffer
respectively. As per Nvidia's documentation these two counters
can tell the number of packets dropped due to pci buffer
overflow and the number of times the device owned queue had not
enough buffers allocated.

Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 .mailmap                                |  1 +
 drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33 +++++++++++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/.mailmap b/.mailmap
index aa569ff456..f57415f7a1 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
 Wang Sheng-Hui <shhuiw@gmail.com>
 Wangyu (Eric) <seven.wangyu@huawei.com>
 Waterman Cao <waterman.cao@intel.com>
+Wathsala Vithanage <wathsala.vithanage@arm.com>
 Weichun Chen <weichunx.chen@intel.com>
 Wei Dai <wei.dai@intel.com>
 Weifeng Li <liweifeng96@126.com>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index dd5a0c546d..8f1567f6a7 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
 		.dpdk_name = "tx_vport_bytes",
 		.ctr_name = "vport_tx_bytes",
 	},
+	/* Device counters */
+	{
+		.dpdk_name = "rx_pci_signal_integrity",
+		.ctr_name = "rx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "tx_pci_signal_integrity",
+		.ctr_name = "tx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "outbound_pci_buffer_overflow",
+		.ctr_name = "outbound_pci_buffer_overflow",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd",
+		.ctr_name = "outbound_pci_stalled_rd",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr",
+		.ctr_name = "outbound_pci_stalled_wr",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd_events",
+		.ctr_name = "outbound_pci_stalled_rd_events",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr_events",
+		.ctr_name = "outbound_pci_stalled_wr_events",
+	},
+	{
+		.dpdk_name = "dev_out_of_buffer",
+		.ctr_name = "dev_out_of_buffer",
+	},
 };
 
 static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
-- 
2.25.1

Re: [PATCH] net/mlx5: enable PCI related counters

[Honnappa Nagarahalli]

> On Feb 9, 2024, at 2:41 PM, Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:
> 
> Versions of Mellanox NICs starting from CX5 have device counters
> related to PCI. These counters are helpful in debugging IO
> bottlenecks. For instance, the outbound_pci_stalled_rd and
> outbound_pci_stalled_wr counters can help with identifying NIC
> stalls due to insufficient PCI credits, which otherwise would
> have required a PCI analyzer or a sophisticated PCI root port
> with a PMU.
> Currently none of these are available in the MLX5 PMD even
> though ethtool is capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
> reads via the kernel driver it is possible to add support with
> ease.
> There is one more PCI related counter and a device counter that
> aren't implemented in the Linux driver at the moment. These two
> are named outbound_pci_buffer_overflow and dev_out_of_buffer
> respectively. As per Nvidia's documentation these two counters
> can tell the number of packets dropped due to pci buffer
> overflow and the number of times the device owned queue had not
> enough buffers allocated.
It would be good to see more of the PCI counters added in other NIC drivers as well. It helps significantly with debugging.

> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
> .mailmap                                |  1 +
> drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33 +++++++++++++++++++++++++
> 2 files changed, 34 insertions(+)
> 
> diff --git a/.mailmap b/.mailmap
> index aa569ff456..f57415f7a1 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
> Wang Sheng-Hui <shhuiw@gmail.com>
> Wangyu (Eric) <seven.wangyu@huawei.com>
> Waterman Cao <waterman.cao@intel.com>
> +Wathsala Vithanage <wathsala.vithanage@arm.com>
> Weichun Chen <weichunx.chen@intel.com>
> Wei Dai <wei.dai@intel.com>
> Weifeng Li <liweifeng96@126.com>
> diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> index dd5a0c546d..8f1567f6a7 100644
> --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> @@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
> .dpdk_name = "tx_vport_bytes",
> .ctr_name = "vport_tx_bytes",
> },
> + /* Device counters */
> + {
> + .dpdk_name = "rx_pci_signal_integrity",
> + .ctr_name = "rx_pci_signal_integrity",
> + },
> + {
> + .dpdk_name = "tx_pci_signal_integrity",
> + .ctr_name = "tx_pci_signal_integrity",
> + },
> + {
> + .dpdk_name = "outbound_pci_buffer_overflow",
> + .ctr_name = "outbound_pci_buffer_overflow",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_rd",
> + .ctr_name = "outbound_pci_stalled_rd",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_wr",
> + .ctr_name = "outbound_pci_stalled_wr",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_rd_events",
> + .ctr_name = "outbound_pci_stalled_rd_events",
> + },
> + {
> + .dpdk_name = "outbound_pci_stalled_wr_events",
> + .ctr_name = "outbound_pci_stalled_wr_events",
> + },
> + {
> + .dpdk_name = "dev_out_of_buffer",
> + .ctr_name = "dev_out_of_buffer",
> + },
> };
> 
> static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
> -- 
> 2.25.1
> 

Re: [PATCH] net/mlx5: enable PCI related counters

[Stephen Hemminger]

On Fri,  9 Feb 2024 20:41:42 +0000
Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:

> Versions of Mellanox NICs starting from CX5 have device counters
> related to PCI. These counters are helpful in debugging IO
> bottlenecks. For instance, the outbound_pci_stalled_rd and
> outbound_pci_stalled_wr counters can help with identifying NIC
> stalls due to insufficient PCI credits, which otherwise would
> have required a PCI analyzer or a sophisticated PCI root port
> with a PMU.
> Currently none of these are available in the MLX5 PMD even
> though ethtool is capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
> reads via the kernel driver it is possible to add support with
> ease.
> There is one more PCI related counter and a device counter that
> aren't implemented in the Linux driver at the moment. These two
> are named outbound_pci_buffer_overflow and dev_out_of_buffer
> respectively. As per Nvidia's documentation these two counters
> can tell the number of packets dropped due to pci buffer
> overflow and the number of times the device owned queue had not
> enough buffers allocated.
> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>

Would it be possible to do this at PCI bus layer so all PCI devices have
that feature?

RE: [PATCH] net/mlx5: enable PCI related counters

[Dariusz Sosnowski]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Saturday, February 10, 2024 02:33
> To: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>;
> dev@dpdk.org; nd@arm.com; Honnappa Nagarahalli
> <honnappa.nagarahalli@arm.com>
> Subject: Re: [PATCH] net/mlx5: enable PCI related counters
> 
> On Fri,  9 Feb 2024 20:41:42 +0000
> Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:
> 
> > Versions of Mellanox NICs starting from CX5 have device counters
> > related to PCI. These counters are helpful in debugging IO
> > bottlenecks. For instance, the outbound_pci_stalled_rd and
> > outbound_pci_stalled_wr counters can help with identifying NIC stalls
> > due to insufficient PCI credits, which otherwise would have required a
> > PCI analyzer or a sophisticated PCI root port with a PMU.
> > Currently none of these are available in the MLX5 PMD even though
> > ethtool is capable of reading some of them.
> > Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads
> > via the kernel driver it is possible to add support with ease.
> > There is one more PCI related counter and a device counter that aren't
> > implemented in the Linux driver at the moment. These two are named
> > outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As
> > per Nvidia's documentation these two counters can tell the number of
> > packets dropped due to pci buffer overflow and the number of times the
> > device owned queue had not enough buffers allocated.
> >
> > Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> 
> Would it be possible to do this at PCI bus layer so all PCI devices have that
> feature?
PCIe performance counters mentioned here are exposed by the NIC itself and mlx5 kernel driver just passes them to userspace.
If such a feature would be added at PCI bus layer, we would need to use (or add) some additional infrastructure.
I'm not familiar with what Linux kernel exposes in terms of PCI counters. It's worth looking into.
I'd assume such data can probably be extracted through PMU.

Best regards,
Dariusz Sosnowski

Re: [PATCH] net/mlx5: enable PCI related counters

[Honnappa Nagarahalli]

> On Feb 13, 2024, at 10:13 AM, Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:
> 
>> -----Original Message-----
>> From: Stephen Hemminger <stephen@networkplumber.org>
>> Sent: Saturday, February 10, 2024 02:33
>> To: Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
>> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
>> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
>> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>;
>> dev@dpdk.org; nd@arm.com; Honnappa Nagarahalli
>> <honnappa.nagarahalli@arm.com>
>> Subject: Re: [PATCH] net/mlx5: enable PCI related counters
>> 
>> On Fri,  9 Feb 2024 20:41:42 +0000
>> Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:
>> 
>>> Versions of Mellanox NICs starting from CX5 have device counters
>>> related to PCI. These counters are helpful in debugging IO
>>> bottlenecks. For instance, the outbound_pci_stalled_rd and
>>> outbound_pci_stalled_wr counters can help with identifying NIC stalls
>>> due to insufficient PCI credits, which otherwise would have required a
>>> PCI analyzer or a sophisticated PCI root port with a PMU.
>>> Currently none of these are available in the MLX5 PMD even though
>>> ethtool is capable of reading some of them.
>>> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads
>>> via the kernel driver it is possible to add support with ease.
>>> There is one more PCI related counter and a device counter that aren't
>>> implemented in the Linux driver at the moment. These two are named
>>> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As
>>> per Nvidia's documentation these two counters can tell the number of
>>> packets dropped due to pci buffer overflow and the number of times the
>>> device owned queue had not enough buffers allocated.
>>> 
>>> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> 
>> Would it be possible to do this at PCI bus layer so all PCI devices have that
>> feature?
> PCIe performance counters mentioned here are exposed by the NIC itself and mlx5 kernel driver just passes them to userspace.
> If such a feature would be added at PCI bus layer, we would need to use (or add) some additional infrastructure.
> I'm not familiar with what Linux kernel exposes in terms of PCI counters. It's worth looking into.
> I'd assume such data can probably be extracted through PMU.
In our investigation, we did not find anything that Linux provides in terms of PCIe PMUs on PCIe root port. The best we found was these PCIe counters as seen by NIC.

It would be good to see other NICs providing similar and additional counters if any.



> 
> Best regards,
> Dariusz Sosnowski

RE: [PATCH] net/mlx5: enable PCI related counters

[Slava Ovsiienko]

Hi,

Regarding "dev_out_of_buffer" - it is global counter, relates to the whole device port,
Including queues not managed by DPDK application - Mellanox/Nvidia NICs operate
In "bifurcated mode" - there might be queues managed by kernel or another DPDK
application. Not sure it makes a lot of sense, but I have no strong objections.

The PCI related counters are also global ones and reflect statistics, impacted by
PCI activity of the whole physical device, including all the network ports located
on the same NIC board (and, sometimes, by internal activity in BlueField).

As I said, no objections from my side:

Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

With best regards,
Slava

> -----Original Message-----
> From: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Sent: Friday, February 9, 2024 10:42 PM
> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: dev@dpdk.org; nd@arm.com; Wathsala Vithanage
> <wathsala.vithanage@arm.com>; Honnappa Nagarahalli
> <honnappa.nagarahalli@arm.com>
> Subject: [PATCH] net/mlx5: enable PCI related counters
> 
> Versions of Mellanox NICs starting from CX5 have device counters related to PCI.
> These counters are helpful in debugging IO bottlenecks. For instance, the
> outbound_pci_stalled_rd and outbound_pci_stalled_wr counters can help with
> identifying NIC stalls due to insufficient PCI credits, which otherwise would have
> required a PCI analyzer or a sophisticated PCI root port with a PMU.
> Currently none of these are available in the MLX5 PMD even though ethtool is
> capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads via the
> kernel driver it is possible to add support with ease.
> There is one more PCI related counter and a device counter that aren't
> implemented in the Linux driver at the moment. These two are named
> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As per
> Nvidia's documentation these two counters can tell the number of packets
> dropped due to pci buffer overflow and the number of times the device owned
> queue had not enough buffers allocated.
> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
>  .mailmap                                |  1 +
>  drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33
> +++++++++++++++++++++++++
>  2 files changed, 34 insertions(+)
> 
> diff --git a/.mailmap b/.mailmap
> index aa569ff456..f57415f7a1 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
> Wang Sheng-Hui <shhuiw@gmail.com>  Wangyu (Eric)
> <seven.wangyu@huawei.com>  Waterman Cao <waterman.cao@intel.com>
> +Wathsala Vithanage <wathsala.vithanage@arm.com>
>  Weichun Chen <weichunx.chen@intel.com>
>  Wei Dai <wei.dai@intel.com>
>  Weifeng Li <liweifeng96@126.com>
> diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> index dd5a0c546d..8f1567f6a7 100644
> --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> @@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl
> mlx5_counters_init[] = {
>  		.dpdk_name = "tx_vport_bytes",
>  		.ctr_name = "vport_tx_bytes",
>  	},
> +	/* Device counters */
> +	{
> +		.dpdk_name = "rx_pci_signal_integrity",
> +		.ctr_name = "rx_pci_signal_integrity",
> +	},
> +	{
> +		.dpdk_name = "tx_pci_signal_integrity",
> +		.ctr_name = "tx_pci_signal_integrity",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_buffer_overflow",
> +		.ctr_name = "outbound_pci_buffer_overflow",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_rd",
> +		.ctr_name = "outbound_pci_stalled_rd",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_wr",
> +		.ctr_name = "outbound_pci_stalled_wr",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_rd_events",
> +		.ctr_name = "outbound_pci_stalled_rd_events",
> +	},
> +	{
> +		.dpdk_name = "outbound_pci_stalled_wr_events",
> +		.ctr_name = "outbound_pci_stalled_wr_events",
> +	},
> +	{
> +		.dpdk_name = "dev_out_of_buffer",
> +		.ctr_name = "dev_out_of_buffer",
> +	},
>  };
> 
>  static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
> --
> 2.25.1

Re: [PATCH] net/mlx5: enable PCI related counters

[Honnappa Nagarahalli]

> On Feb 13, 2024, at 7:12 AM, Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> 
> Hi,
> 
> Regarding "dev_out_of_buffer" - it is global counter, relates to the whole device port,
> Including queues not managed by DPDK application - Mellanox/Nvidia NICs operate
> In "bifurcated mode" - there might be queues managed by kernel or another DPDK
> application. Not sure it makes a lot of sense, but I have no strong objections.
These are still helpful to debug in lab environment. But, it would be good to document these.

> 
> The PCI related counters are also global ones and reflect statistics, impacted by
> PCI activity of the whole physical device, including all the network ports located
> on the same NIC board (and, sometimes, by internal activity in BlueField).
> 
> As I said, no objections from my side:
> 
> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> 
> With best regards,
> Slava
> 
>> -----Original Message-----
>> From: Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Sent: Friday, February 9, 2024 10:42 PM
>> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
>> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
>> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
>> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
>> Cc: dev@dpdk.org; nd@arm.com; Wathsala Vithanage
>> <wathsala.vithanage@arm.com>; Honnappa Nagarahalli
>> <honnappa.nagarahalli@arm.com>
>> Subject: [PATCH] net/mlx5: enable PCI related counters
>> 
>> Versions of Mellanox NICs starting from CX5 have device counters related to PCI.
>> These counters are helpful in debugging IO bottlenecks. For instance, the
>> outbound_pci_stalled_rd and outbound_pci_stalled_wr counters can help with
>> identifying NIC stalls due to insufficient PCI credits, which otherwise would have
>> required a PCI analyzer or a sophisticated PCI root port with a PMU.
>> Currently none of these are available in the MLX5 PMD even though ethtool is
>> capable of reading some of them.
>> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads via the
>> kernel driver it is possible to add support with ease.
>> There is one more PCI related counter and a device counter that aren't
>> implemented in the Linux driver at the moment. These two are named
>> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As per
>> Nvidia's documentation these two counters can tell the number of packets
>> dropped due to pci buffer overflow and the number of times the device owned
>> queue had not enough buffers allocated.
>> 
>> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> ---
>> .mailmap                                |  1 +
>> drivers/net/mlx5/linux/mlx5_ethdev_os.c | 33
>> +++++++++++++++++++++++++
>> 2 files changed, 34 insertions(+)
>> 
>> diff --git a/.mailmap b/.mailmap
>> index aa569ff456..f57415f7a1 100644
>> --- a/.mailmap
>> +++ b/.mailmap
>> @@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
>> Wang Sheng-Hui <shhuiw@gmail.com>  Wangyu (Eric)
>> <seven.wangyu@huawei.com>  Waterman Cao <waterman.cao@intel.com>
>> +Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Weichun Chen <weichunx.chen@intel.com>
>> Wei Dai <wei.dai@intel.com>
>> Weifeng Li <liweifeng96@126.com>
>> diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> index dd5a0c546d..8f1567f6a7 100644
>> --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
>> @@ -1574,6 +1574,39 @@ static const struct mlx5_counter_ctrl
>> mlx5_counters_init[] = {
>> .dpdk_name = "tx_vport_bytes",
>> .ctr_name = "vport_tx_bytes",
>> },
>> + /* Device counters */
>> + {
>> + .dpdk_name = "rx_pci_signal_integrity",
>> + .ctr_name = "rx_pci_signal_integrity",
>> + },
>> + {
>> + .dpdk_name = "tx_pci_signal_integrity",
>> + .ctr_name = "tx_pci_signal_integrity",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_buffer_overflow",
>> + .ctr_name = "outbound_pci_buffer_overflow",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_rd",
>> + .ctr_name = "outbound_pci_stalled_rd",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_wr",
>> + .ctr_name = "outbound_pci_stalled_wr",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_rd_events",
>> + .ctr_name = "outbound_pci_stalled_rd_events",
>> + },
>> + {
>> + .dpdk_name = "outbound_pci_stalled_wr_events",
>> + .ctr_name = "outbound_pci_stalled_wr_events",
>> + },
>> + {
>> + .dpdk_name = "dev_out_of_buffer",
>> + .ctr_name = "dev_out_of_buffer",
>> + },
>> };
>> 
>> static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
>> --
>> 2.25.1
> 

RE: [PATCH] net/mlx5: enable PCI related counters

[Dariusz Sosnowski]

> -----Original Message-----
> From: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Sent: Friday, February 9, 2024 21:42
> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: dev@dpdk.org; nd@arm.com; Wathsala Vithanage
> <wathsala.vithanage@arm.com>; Honnappa Nagarahalli
> <honnappa.nagarahalli@arm.com>
> Subject: [PATCH] net/mlx5: enable PCI related counters
> 
> Versions of Mellanox NICs starting from CX5 have device counters related to
> PCI. These counters are helpful in debugging IO bottlenecks. For instance, the
> outbound_pci_stalled_rd and outbound_pci_stalled_wr counters can help
> with identifying NIC stalls due to insufficient PCI credits, which otherwise
> would have required a PCI analyzer or a sophisticated PCI root port with a
> PMU.
> Currently none of these are available in the MLX5 PMD even though ethtool is
> capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads via
> the kernel driver it is possible to add support with ease.
> There is one more PCI related counter and a device counter that aren't
> implemented in the Linux driver at the moment. These two are named
> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As per
> Nvidia's documentation these two counters can tell the number of packets
> dropped due to pci buffer overflow and the number of times the device
> owned queue had not enough buffers allocated.
> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>

Best regards,
Dariusz Sosnowski

[PATCH v2] net/mlx5: enable PCI related counters

[Wathsala Vithanage]

Versions of Mellanox NICs starting from CX5 have device counters
related to PCI. These counters are helpful in debugging IO
bottlenecks. For instance, the outbound_pci_stalled_rd and
outbound_pci_stalled_wr counters can help with identifying NIC
stalls due to insufficient PCI credits, which otherwise would
have required a PCI analyzer or a sophisticated PCI root port
with a PMU.
Currently none of these are available in the MLX5 PMD even
though ethtool is capable of reading some of them.
Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
reads via the kernel driver it is possible to add support with
ease.
There is one more PCI related counter and a device counter that
aren't implemented in the Linux driver at the moment. These two
are named outbound_pci_buffer_overflow and dev_out_of_buffer
respectively. As per Nvidia's documentation these two counters
can tell the number of packets dropped due to pci buffer
overflow and the number of times the device owned queue had not
enough buffers allocated.

Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 .mailmap                                |  1 +
 drivers/net/mlx5/linux/mlx5_ethdev_os.c | 41 +++++++++++++++++++++++++
 2 files changed, 42 insertions(+)

diff --git a/.mailmap b/.mailmap
index aa569ff456..f57415f7a1 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
 Wang Sheng-Hui <shhuiw@gmail.com>
 Wangyu (Eric) <seven.wangyu@huawei.com>
 Waterman Cao <waterman.cao@intel.com>
+Wathsala Vithanage <wathsala.vithanage@arm.com>
 Weichun Chen <weichunx.chen@intel.com>
 Wei Dai <wei.dai@intel.com>
 Weifeng Li <liweifeng96@126.com>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index dd5a0c546d..c837c862a8 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -1574,6 +1574,47 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
 		.dpdk_name = "tx_vport_bytes",
 		.ctr_name = "vport_tx_bytes",
 	},
+	/**
+	 * Device counters: These counters are for the
+	 * entire PCI device (NIC). These counters are
+	 * not counting on a per port/queue basis.
+	 * Values reported by these counters may not be
+	 * useful if the device is bifurcated and queues
+	 * are shared with the kernel or other DPDK
+	 * applications.
+	 */
+	{
+		.dpdk_name = "rx_pci_signal_integrity",
+		.ctr_name = "rx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "tx_pci_signal_integrity",
+		.ctr_name = "tx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "outbound_pci_buffer_overflow",
+		.ctr_name = "outbound_pci_buffer_overflow",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd",
+		.ctr_name = "outbound_pci_stalled_rd",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr",
+		.ctr_name = "outbound_pci_stalled_wr",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd_events",
+		.ctr_name = "outbound_pci_stalled_rd_events",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr_events",
+		.ctr_name = "outbound_pci_stalled_wr_events",
+	},
+	{
+		.dpdk_name = "dev_out_of_buffer",
+		.ctr_name = "dev_out_of_buffer",
+	},
 };
 
 static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
-- 
2.25.1

[PATCH v3] net/mlx5: enable PCI related counters

[Wathsala Vithanage]

Versions of Mellanox NICs starting from CX5 have device counters
related to PCI. These counters are helpful in debugging IO
bottlenecks. For instance, the outbound_pci_stalled_rd and
outbound_pci_stalled_wr counters can help with identifying NIC
stalls due to insufficient PCI credits, which otherwise would
have required a PCI analyzer or a sophisticated PCI root port
with a PMU.
Currently none of these are available in the MLX5 PMD even
though ethtool is capable of reading some of them.
Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and
reads via the kernel driver it is possible to add support with
ease.
There is one more PCI related counter and a device counter that
aren't implemented in the Linux driver at the moment. These two
are named outbound_pci_buffer_overflow and dev_out_of_buffer
respectively. As per Nvidia's documentation these two counters
can tell the number of packets dropped due to pci buffer
overflow and the number of times the device owned queue had not
enough buffers allocated.

Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 .mailmap                                |  1 +
 drivers/net/mlx5/linux/mlx5_ethdev_os.c | 37 +++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/.mailmap b/.mailmap
index aa569ff456..f57415f7a1 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1510,6 +1510,7 @@ Walter Heymans <walter.heymans@corigine.com>
 Wang Sheng-Hui <shhuiw@gmail.com>
 Wangyu (Eric) <seven.wangyu@huawei.com>
 Waterman Cao <waterman.cao@intel.com>
+Wathsala Vithanage <wathsala.vithanage@arm.com>
 Weichun Chen <weichunx.chen@intel.com>
 Wei Dai <wei.dai@intel.com>
 Weifeng Li <liweifeng96@126.com>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index dd5a0c546d..92c47a3b3d 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -1574,6 +1574,43 @@ static const struct mlx5_counter_ctrl mlx5_counters_init[] = {
 		.dpdk_name = "tx_vport_bytes",
 		.ctr_name = "vport_tx_bytes",
 	},
+	/**
+	 * Device counters: These counters are for the
+	 * entire PCI device (NIC). These counters are
+	 * not counting on a per port/queue basis.
+	 */
+	{
+		.dpdk_name = "rx_pci_signal_integrity",
+		.ctr_name = "rx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "tx_pci_signal_integrity",
+		.ctr_name = "tx_pci_signal_integrity",
+	},
+	{
+		.dpdk_name = "outbound_pci_buffer_overflow",
+		.ctr_name = "outbound_pci_buffer_overflow",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd",
+		.ctr_name = "outbound_pci_stalled_rd",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr",
+		.ctr_name = "outbound_pci_stalled_wr",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_rd_events",
+		.ctr_name = "outbound_pci_stalled_rd_events",
+	},
+	{
+		.dpdk_name = "outbound_pci_stalled_wr_events",
+		.ctr_name = "outbound_pci_stalled_wr_events",
+	},
+	{
+		.dpdk_name = "dev_out_of_buffer",
+		.ctr_name = "dev_out_of_buffer",
+	},
 };
 
 static const unsigned int xstats_n = RTE_DIM(mlx5_counters_init);
-- 
2.25.1

RE: [PATCH v3] net/mlx5: enable PCI related counters

[Raslan Darawsheh]

Hi,
> -----Original Message-----
> From: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Sent: Thursday, February 15, 2024 8:27 PM
> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: dev@dpdk.org; nd@arm.com; Wathsala Vithanage
> <wathsala.vithanage@arm.com>; Honnappa Nagarahalli
> <honnappa.nagarahalli@arm.com>
> Subject: [PATCH v3] net/mlx5: enable PCI related counters
> 
> Versions of Mellanox NICs starting from CX5 have device counters related to
> PCI. These counters are helpful in debugging IO bottlenecks. For instance, the
> outbound_pci_stalled_rd and outbound_pci_stalled_wr counters can help
> with identifying NIC stalls due to insufficient PCI credits, which otherwise
> would have required a PCI analyzer or a sophisticated PCI root port with a
> PMU.
> Currently none of these are available in the MLX5 PMD even though ethtool is
> capable of reading some of them.
> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads via
> the kernel driver it is possible to add support with ease.
> There is one more PCI related counter and a device counter that aren't
> implemented in the Linux driver at the moment. These two are named
> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As per
> Nvidia's documentation these two counters can tell the number of packets
> dropped due to pci buffer overflow and the number of times the device
> owned queue had not enough buffers allocated.
> 
> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Patch applied to next-net-mlx,
Kindest regards
Raslan Darawsheh