From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 81E0743B0C; Tue, 13 Feb 2024 17:13:10 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 521EC40A4B; Tue, 13 Feb 2024 17:13:10 +0100 (CET) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2066.outbound.protection.outlook.com [40.107.102.66]) by mails.dpdk.org (Postfix) with ESMTP id 156EC402D8 for <dev@dpdk.org>; Tue, 13 Feb 2024 17:13:08 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fhy9AaekaS633HVYL1iEzCXyrJtSwah2bSpMFddBYBTEnu8MLI9khImZ6enE9CSuPYOXKlE7kS5DHK+hweUKWWVlRn5rxWP90RBsbs89i8VRSTyysRnclfPlwX/sQzv8So0o+NNsN+LjRThm7+kGElTYrMo77r+GNYf99A7CJzxml7r98tWrP9oZ4B4lsASGWAMG3np5NxRGM06Qzf6YX0vcWY5py1G4S39K69e7+apzoI/Mo21V1YjmW6ieB/QDXOHvOjkTeYAITMVb31SqqKUS4HFRjA1wBYbOiSCNDHC3wsj+2glqg0aXwMjLuWyxo1isOR06v9BLk+5n5EjhPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Zb8AizkgmqkpEymUExNfWf/SC8C/CTHN63TDfA08eAU=; b=P6OZJbAe0+yRgEi8r9MRt49QgRig7CUFIqNvo8AaKYj8NwJLaWc0vPL8agev0XRP1eopZxk0NWYyGVjYu493J6If8Simb7V2xA2xvnPgY/3+5dTuXsF7TEBEQUSViDExIOBPK70inpAU8x4ECo87Ygoa9MRiAr5A51eI0tiPlSSfc/dyBaUmGpgLBCRvImH0J5E8L2atQLACmaDNd6PsGORNperV4pBlTRDUkHVRhOO9xcmF5bTAtrXIHrzloY7EoXNp/Yn/IlBKD8+PJ3GLWcsWLVP5uVaw5fuWWaKmjKK0aUMqfKN1rQv0V0B9KkACfKyXe8T8KxgDwXpg7bMgCw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zb8AizkgmqkpEymUExNfWf/SC8C/CTHN63TDfA08eAU=; b=O1UL7vwxl834ZADLHxoOSztS78RRsGkh31f2bbmPWdAor6HAOi9QorP/Ve7JKr0+57JCwCtZfs/f3vanQ52JVENDnh9euHSkhcK/AqV36UQ7d+aKyiEwK6Vi3lBgwcY/5W50NRLg3Sj4a/dApkdqXQGosnZJe7b1S3qwIBGxso7/KxNbvJCOz5UcffetEYqsORcBrHPAudSCBLOIeajma1niruWEa7T2fdOteXkHILAns5ocgXP0GmvA/5TGUqExwoMZEMW7KpD1Zh/qyD7tnNemd4ng636GLnZr9lIy1VnOKISXSXZdxsLnKo+B5Re0inPt3rIpK0LRw840MAmR/w== Received: from IA1PR12MB8311.namprd12.prod.outlook.com (2603:10b6:208:3fa::12) by MW4PR12MB7215.namprd12.prod.outlook.com (2603:10b6:303:228::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.18; Tue, 13 Feb 2024 16:13:05 +0000 Received: from IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::b013:88f8:c1df:9ce1]) by IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::b013:88f8:c1df:9ce1%7]) with mapi id 15.20.7292.022; Tue, 13 Feb 2024 16:13:05 +0000 From: Dariusz Sosnowski <dsosnowski@nvidia.com> To: Stephen Hemminger <stephen@networkplumber.org>, Wathsala Vithanage <wathsala.vithanage@arm.com> CC: "NBU-Contact-Thomas Monjalon (EXTERNAL)" <thomas@monjalon.net>, Slava Ovsiienko <viacheslavo@nvidia.com>, Ori Kam <orika@nvidia.com>, Suanming Mou <suanmingm@nvidia.com>, Matan Azrad <matan@nvidia.com>, "dev@dpdk.org" <dev@dpdk.org>, "nd@arm.com" <nd@arm.com>, Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Subject: RE: [PATCH] net/mlx5: enable PCI related counters Thread-Topic: [PATCH] net/mlx5: enable PCI related counters Thread-Index: AQHaW5hwFmfBSDEVBE2sHS1aOTL5BrECyuCAgAWP2lA= Date: Tue, 13 Feb 2024 16:13:05 +0000 Message-ID: <IA1PR12MB83117C4ED19A0B29E122B9B8A44F2@IA1PR12MB8311.namprd12.prod.outlook.com> References: <20240209204142.1148790-1-wathsala.vithanage@arm.com> <20240209173239.72497ffb@hermes.local> In-Reply-To: <20240209173239.72497ffb@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA1PR12MB8311:EE_|MW4PR12MB7215:EE_ x-ms-office365-filtering-correlation-id: 73ff2742-8635-49a7-2ae7-08dc2caea8fd x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: lJO9slKv9m2+KHpEb3jxACgUn2PcLgtimiK4uDqw1ijmOoiwUu1itmT1Rs6l5ejUmZxcy8FsUvuotTeV6lJlH7m8F3GCPSsyyAIfEUghujMgK96oBbzHKIXjiq3C/XSx/OTy9J9dJC+FrYWCsEYX0ThUErcbonEkCiOZjx1ALTGvUSWEpaFwY/6OMTGo4sOGHmLSrkPIg3lMHZbvUQA/RKtJLby5imwyXBPvS7HfFuVHV80jK/voG7kTNs007Q1uj88v2bXbg80l/XJHdjNxvEQppuPCseNg1OLadDpa8vYUXTL0BidyHEHLbK2TXOLWTyY3kW4QuYooSD5kNSZMkqI6yGxk5fyBFcC4a9WdZDU9Y4g4ppqFJd44uX+KpsMsmamxLl/Eux2Z707uh6G7nV2bCJOVEt8Uvt95JZe54LIyXFNxeVic1etmfmT6/vZ835JGg7F6/O1Mx/HGyjpBZMD2q/tQe8d9iOcMKe17SOg1u5w70iSgWM6y+9BNRocq2sKeKlFidhd9OVZ2tXD38GVFQgU/KqgWQ9NfejNZ83a7k5Ir7i0ebl5X+/MM8X+/FydmoVMG/IOhPQOsufyuGJjZtLQnpOtVunJd6mILQgDdwxvbshDDaq9vuVFv3uwU x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA1PR12MB8311.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376002)(346002)(396003)(136003)(366004)(39860400002)(230273577357003)(230922051799003)(451199024)(64100799003)(1800799012)(186009)(66556008)(83380400001)(86362001)(71200400001)(66946007)(66476007)(38100700002)(55236004)(478600001)(54906003)(7696005)(122000001)(9686003)(110136005)(6506007)(53546011)(76116006)(26005)(316002)(38070700009)(33656002)(5660300002)(4326008)(8936002)(52536014)(8676002)(55016003)(64756008)(41300700001)(66446008)(2906002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?qXfsv4ZX8e1GDWhIpfOZ4cAZ0Fl0AA0uIWDMEpml4OKYUZhT/kOMC3rNEVyy?= =?us-ascii?Q?S8AEIDB6x/qYiXrNIzfc1d+C6OOdf0OMo519h7ekIEypxhPy+DC4/eUNHAUi?= =?us-ascii?Q?0QHA1C61RmPVJT61RVhsLWJTdnV4qspLo4OcW9g0fBrWQwmHTNKwCCt2Qqa5?= =?us-ascii?Q?B9vo8LtfihfbvuM8PqGJi8lCUPjyWg1n6JRhyjsH/J45tJpdBKdgX3JNfL68?= =?us-ascii?Q?omtUNevgHi7ECrpQUS7r0C3bjpj+TE315trBRiAlEftHhDbYvOnHoc7fN8FS?= =?us-ascii?Q?zXaq/FMVk2cdmW7MUK3mlJZeEOJLqxarlB13IIF8Ppy0sV/uaN3LWuJ4/8UT?= =?us-ascii?Q?HTDMPD9uLEBPn2whmkXnYxzWD/DUOUide79czeodraeMfzcXhqvPy/e4F8xE?= =?us-ascii?Q?EnMeb39N7qyqupY0/IJnc7ByYXEGfNTgleQlMBbVtix6iLsinvxttVGBQzCS?= =?us-ascii?Q?98ZMm0rmW9u1hxwme8lWI/bWgK79mP3KZc9v9XSyezJce+HC1vchH50AUzS+?= =?us-ascii?Q?WjUa2qMyBwscV3QjTSVo+/0QNZgyUbZP3ikAv7aGgr+zTZjqbKG3UVsIkynX?= =?us-ascii?Q?QaHvTnIFgRJpJGEbijXw/g+QXNcaDQyCPWCwjBJglvWyySy1XnqQVz7K1bbh?= =?us-ascii?Q?7JHj3cYTdybumREh5FbvWykAbTqUK+TRZ0lu5mCzkwQIxTFiXdzYU04vEGcY?= =?us-ascii?Q?blxeO1Rn5pIWQVKmL7xxhZ7fP7bWulVa9So3BdOMG/G9yN5oOXHfUNN6Y5Pn?= =?us-ascii?Q?CS3YANcqRPZjDnj7ENjgQAfCaxBzkdfjMIZv7NKKnddL6kT+7bBik+nTPhou?= =?us-ascii?Q?TVe8hWOslad4wlP7HL3/PFVXN9KdDG8+tqtMHxb6vgQWpviaHAUpBO53G369?= =?us-ascii?Q?sL6gtNcnzRBkt8rINROtfP+zoww8t4p+87xfQ9mTZSk6Uro46QNIzVZwWulT?= =?us-ascii?Q?1FEfkH67p1gRQ75CZhct234RLWnitYE6HWN+BbvCLqpnTiQDeIXIcVlkSoDi?= =?us-ascii?Q?evZcwUT09OG3bOmxM5NL1KK61R6JtkXdfjVBor8AFj6dt8/U00NQIFt0SfpT?= =?us-ascii?Q?vy5TrTfPmacOHYPQsucRM3wwVUzfNREBixoAOV9XE+nhDof5+Q8nl6h89zBO?= =?us-ascii?Q?OzZVE2fKr5weNxJ7nL8aopF1mXXNkWeSFtQ/L/Oy9FqBQFTxh70bE60ZPUFu?= =?us-ascii?Q?D8i4qKL3R+RcfmcryqmLYgFLZrC+1cjkC+9ANcT+0oPox2mtNs5kYXtNf1SC?= =?us-ascii?Q?fxH1WSGzywWFyOptUfSCBkV9WO8YhkY5N4Un8DJfqvaxJDpW51IAQu4I5bnA?= =?us-ascii?Q?ptq1QmZU5p0dv8MWDOZkHjqRCYcCTJgeedPbHrmydoEM7MOoDHeb3D80nePQ?= =?us-ascii?Q?uXzDWEmcTB/Xc5PRkCd3ZnG3tzKqCV1a7oRVfkeIgO3AkhYnUVO8yxZowzj4?= =?us-ascii?Q?VYKSWe4d5W3gj5/M38GdW/26IFQGK/UQ4M0wOdmQbMYVAARaEw4VhvuW6eTW?= =?us-ascii?Q?JwIGHZDjTDwHfLYlUopfqP8ujD0sI9H3mEdQ9sRm3m4nPHumfQAanKtPST5R?= =?us-ascii?Q?VBHo5XUxEbQKvakkUbKeyn6eU1VVX8V4BIQGPpXh?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB8311.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 73ff2742-8635-49a7-2ae7-08dc2caea8fd X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Feb 2024 16:13:05.0585 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: /vXGAbO5mpKODNu7I14kluNy4f9VrGiZt28fiqSEYbbAF0v0FHGD0amZVkmBuBtAkARt1UFUW+4S8j45R2rUqw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7215 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Stephen Hemminger <stephen@networkplumber.org> > Sent: Saturday, February 10, 2024 02:33 > To: Wathsala Vithanage <wathsala.vithanage@arm.com> > Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; > Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava Ovsiienko > <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou > <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>; > dev@dpdk.org; nd@arm.com; Honnappa Nagarahalli > <honnappa.nagarahalli@arm.com> > Subject: Re: [PATCH] net/mlx5: enable PCI related counters >=20 > On Fri, 9 Feb 2024 20:41:42 +0000 > Wathsala Vithanage <wathsala.vithanage@arm.com> wrote: >=20 > > Versions of Mellanox NICs starting from CX5 have device counters > > related to PCI. These counters are helpful in debugging IO > > bottlenecks. For instance, the outbound_pci_stalled_rd and > > outbound_pci_stalled_wr counters can help with identifying NIC stalls > > due to insufficient PCI credits, which otherwise would have required a > > PCI analyzer or a sophisticated PCI root port with a PMU. > > Currently none of these are available in the MLX5 PMD even though > > ethtool is capable of reading some of them. > > Since PMD uses the same ioctl used by ethtool (SIOCETHTOOL) and reads > > via the kernel driver it is possible to add support with ease. > > There is one more PCI related counter and a device counter that aren't > > implemented in the Linux driver at the moment. These two are named > > outbound_pci_buffer_overflow and dev_out_of_buffer respectively. As > > per Nvidia's documentation these two counters can tell the number of > > packets dropped due to pci buffer overflow and the number of times the > > device owned queue had not enough buffers allocated. > > > > Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com> > > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> >=20 > Would it be possible to do this at PCI bus layer so all PCI devices have = that > feature? PCIe performance counters mentioned here are exposed by the NIC itself and = mlx5 kernel driver just passes them to userspace. If such a feature would be added at PCI bus layer, we would need to use (or= add) some additional infrastructure. I'm not familiar with what Linux kernel exposes in terms of PCI counters. I= t's worth looking into. I'd assume such data can probably be extracted through PMU. Best regards, Dariusz Sosnowski