From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
To: Dariusz Sosnowski <dsosnowski@nvidia.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>,
Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com>,
"thomas@monjalon.net" <thomas@monjalon.net>,
Slava Ovsiienko <viacheslavo@nvidia.com>,
Ori Kam <orika@nvidia.com>, Suanming Mou <suanmingm@nvidia.com>,
Matan Azrad <matan@nvidia.com>, "dev@dpdk.org" <dev@dpdk.org>,
nd <nd@arm.com>
Subject: Re: [PATCH] net/mlx5: enable PCI related counters
Date: Wed, 14 Feb 2024 00:52:48 +0000 [thread overview]
Message-ID: <9E1E7584-1D70-4A7A-9C75-6EF027371398@arm.com> (raw)
In-Reply-To: <IA1PR12MB83117C4ED19A0B29E122B9B8A44F2@IA1PR12MB8311.namprd12.prod.outlook.com>
> On Feb 13, 2024, at 10:13 AM, Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:
>
>> -----Original Message-----
>> From: Stephen Hemminger <stephen@networkplumber.org>
>> Sent: Saturday, February 10, 2024 02:33
>> To: Wathsala Vithanage <wathsala.vithanage@arm.com>
>> Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
>> Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko
>> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
>> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>;
>> dev@dpdk.org; nd@arm.com; Honnappa Nagarahalli
>> <honnappa.nagarahalli@arm.com>
>> Subject: Re: [PATCH] net/mlx5: enable PCI related counters
>>
>> On Fri, 9 Feb 2024 20:41:42 +0000
>> Wathsala Vithanage <wathsala.vithanage@arm.com> wrote:
>>
>>> Versions of Mellanox NICs starting from CX5 have device counters
>>> related to PCI. These counters are helpful in debugging IO
>>> bottlenecks. For instance, the outbound_pci_stalled_rd and
>>> outbound_pci_stalled_wr counters can help with identifying NIC stalls
>>> due to insufficient PCI credits, which otherwise would have required a
>>> PCI analyzer or a sophisticated PCI root port with a PMU.
>>> Currently none of these are available in the MLX5 PMD even though
>>> ethtool is capable of reading some of them.
>>> Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads
>>> via the kernel driver it is possible to add support with ease.
>>> There is one more PCI related counter and a device counter that aren't
>>> implemented in the Linux driver at the moment. These two are named
>>> outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As
>>> per Nvidia's documentation these two counters can tell the number of
>>> packets dropped due to pci buffer overflow and the number of times the
>>> device owned queue had not enough buffers allocated.
>>>
>>> Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>>
>> Would it be possible to do this at PCI bus layer so all PCI devices have that
>> feature?
> PCIe performance counters mentioned here are exposed by the NIC itself and mlx5 kernel driver just passes them to userspace.
> If such a feature would be added at PCI bus layer, we would need to use (or add) some additional infrastructure.
> I'm not familiar with what Linux kernel exposes in terms of PCI counters. It's worth looking into.
> I'd assume such data can probably be extracted through PMU.
In our investigation, we did not find anything that Linux provides in terms of PCIe PMUs on PCIe root port. The best we found was these PCIe counters as seen by NIC.
It would be good to see other NICs providing similar and additional counters if any.
>
> Best regards,
> Dariusz Sosnowski
next prev parent reply other threads:[~2024-02-14 0:53 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-09 20:41 Wathsala Vithanage
2024-02-09 21:16 ` Honnappa Nagarahalli
2024-02-10 1:32 ` Stephen Hemminger
2024-02-13 16:13 ` Dariusz Sosnowski
2024-02-14 0:52 ` Honnappa Nagarahalli [this message]
2024-02-13 13:12 ` Slava Ovsiienko
2024-02-14 1:50 ` Honnappa Nagarahalli
2024-02-13 16:17 ` Dariusz Sosnowski
2024-02-14 20:14 ` [PATCH v2] " Wathsala Vithanage
2024-02-15 18:26 ` [PATCH v3] " Wathsala Vithanage
2024-02-27 16:14 ` Raslan Darawsheh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9E1E7584-1D70-4A7A-9C75-6EF027371398@arm.com \
--to=honnappa.nagarahalli@arm.com \
--cc=dev@dpdk.org \
--cc=dsosnowski@nvidia.com \
--cc=matan@nvidia.com \
--cc=nd@arm.com \
--cc=orika@nvidia.com \
--cc=stephen@networkplumber.org \
--cc=suanmingm@nvidia.com \
--cc=thomas@monjalon.net \
--cc=viacheslavo@nvidia.com \
--cc=wathsala.vithanage@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).