From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CB579454EF for ; Tue, 25 Jun 2024 12:43:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AA74C40696; Tue, 25 Jun 2024 12:43:44 +0200 (CEST) Received: from us-smtp-delivery-119.mimecast.com (us-smtp-delivery-119.mimecast.com [170.10.129.119]) by mails.dpdk.org (Postfix) with ESMTP id C4B07402C4 for ; Tue, 25 Jun 2024 12:42:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=progress.com; s=mimecast2021; t=1719312159; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=hst0+4BmQTlo9jryKoD/SAK3uZIuV+xRealNgsqeoeI=; b=QI379FOO/zGWCoR3aNJlTz1lbpHalXOkaIMlno5Y/0e6IE878R7gFFhOsYV7UFOjfb4ze0 nQMjmjuQ2LTwJDHdJkNMhYi+oXom2HjSA7zgFyoyK9EMRIcavUIP4r8C7gram1iqu6CVkg eUu6zZbeJwPSA5qNFO0valoRyzJW6J7SI6DUgUUdNZYiuW+V3alRrNbWaARw667RZJZ/G2 0khRu9zVhL4gX7XgZqrNAiBZHpP6ztifHdFuL1SToLDz8R2Lh73meu8DCv7hqYR2gKq5GA 2GWIfj5Xvp8boa0bFZuc2qZAMkfIZQ2XwrVWrAu9zSvpvYlRRG6lQZ1eNS2Crg== Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2101.outbound.protection.outlook.com [104.47.58.101]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-201-Mi9CO9cgPNWF4gCbn9n7-Q-1; Tue, 25 Jun 2024 06:42:37 -0400 X-MC-Unique: Mi9CO9cgPNWF4gCbn9n7-Q-1 Received: from SA1PR13MB4925.namprd13.prod.outlook.com (2603:10b6:806:1a7::8) by DS1PR13MB7121.namprd13.prod.outlook.com (2603:10b6:8:211::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.30; Tue, 25 Jun 2024 10:42:35 +0000 Received: from SA1PR13MB4925.namprd13.prod.outlook.com ([fe80::1cbe:a23a:c147:2f3d]) by SA1PR13MB4925.namprd13.prod.outlook.com ([fe80::1cbe:a23a:c147:2f3d%4]) with mapi id 15.20.7698.025; Tue, 25 Jun 2024 10:42:35 +0000 From: Tomas Jansky To: "users@dpdk.org" Subject: Unable to start mlx5 PMD on NUMA node 1 Thread-Topic: Unable to start mlx5 PMD on NUMA node 1 Thread-Index: AQHaxuwa5EgT7ncHb027CBVr4YUhkw== Date: Tue, 25 Jun 2024 10:42:35 +0000 Message-ID: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SA1PR13MB4925:EE_|DS1PR13MB7121:EE_ x-ms-office365-filtering-correlation-id: edd020e7-5876-4102-bc90-08dc95038680 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230037|366013|376011|1800799021|38070700015 x-microsoft-antispam-message-info: =?iso-8859-1?Q?qxOslUwqBI9V19kv4qlM3Cw5Pf3GMykKd6HsAdLyV2niPyrjUSW1clTmB3?= =?iso-8859-1?Q?ZMrxIAL5uzhVrCwGryyMRMPQ6nENkmUlcc7PAzXufSmRNGpsgtjV2dn2zv?= =?iso-8859-1?Q?BsfG0Szco/R9wS8hLmxVHnHT1VH02EgPDoO3fi18QE/nLG8q4g2TbVBUG/?= =?iso-8859-1?Q?dI/z692v916/wxhrH/ctDWfsyJYMM9fHvBfYkoxegX3mxPObW7fVPQ0xl2?= =?iso-8859-1?Q?YvBEA8VskOFTI6Ug7zRYbl0RTfw3XTrrAvG21ZP/415B13c29sVr4pJ4Lq?= =?iso-8859-1?Q?M+SXLr+EDhSq0Zdb0yaXOqueZ00miOOScM2HpHF+yUTyKnZqfNE9jK5oLS?= =?iso-8859-1?Q?ium6VAhrCrHKoVigopR91d7f+x/0cCvj3tAdYE+gAr2V7BjytPZVErIGlj?= =?iso-8859-1?Q?tsej9IA34H/9/a770r+1Kus9i2b6XNnpOv6mYSvmITzwj9n7EhOY4O5Rnp?= =?iso-8859-1?Q?mqVUPJHfu9jNhfasL8DcvAarbI7dR5cr0VYuCCBubtsnQtQG5uXGWBaClJ?= =?iso-8859-1?Q?mzwKL+x/cQS9vC128Z7c8junuTtbviLof2LYdOQoU7kDwXNK7D4V8reVv3?= =?iso-8859-1?Q?s6CjZnC/7UiMOGm4aPGZKMih8rUre0bcmkQs/Tr98Vvyr6TVNPIYJanhDg?= =?iso-8859-1?Q?HYweJ33we+/4DtJvVp5rmDKV02CzlSJHU7OLnyqudUVrFx7akkCsZyq4Md?= =?iso-8859-1?Q?LsvLyFTG5SNzrMtJfgmeWBhEwIBdST+CuOroUpqZwig9c9BnPJVu18Eb9e?= =?iso-8859-1?Q?W6jd/27FqmlABBd26w3t3Udo/2bYvjUIPPi88YXRQBmLaD7wc/nfKWGnXw?= =?iso-8859-1?Q?2viysQ4+Ge86numMk3jfkB1asgkauPu5KJFU4DMIIY1GHCdhVSJWr6yKhM?= =?iso-8859-1?Q?n2XGGHWKdVx3S6D5cLxEdaMB9M8FXHJUisvWWG43U+m2SLbE2k0AsporUz?= =?iso-8859-1?Q?eQFf6xSm8v2reio9JayWYVf6AeMa0yMVEpARsZmJ+/VL1BWfkMZPJNNdlv?= =?iso-8859-1?Q?hiT/4HNfrTp140NbGARRJcGnNIqsPGWrbjTfwBILzlqBpMVXqspuxJlYbr?= =?iso-8859-1?Q?7JMdLqDtUbe5MUB2P7EobT8/GzraBuK4uyP5BVWjz+rHji1PHOUoRIe+dd?= =?iso-8859-1?Q?qcvX0mCzROIQbGn1yeUVBtnpwNijFjWEpCKZAv3NvFLeZAeCJTK9jePPQe?= =?iso-8859-1?Q?25mgh365EKtT/GE60s/K5EEHMSt/rUdhlhPj5l5pYSIdTc4yxE+hFH72sA?= =?iso-8859-1?Q?hNjNN90dY0BDh91d207B+cigJsrJ2L8i/fWWrXi+0hBYykRKles1YmXvhU?= =?iso-8859-1?Q?ky0ZvxGyGtgAyHzqmxjduvQUz5wmNXmtEDCDwU07q9yJ1FSqVxibi3bhM2?= =?iso-8859-1?Q?ICKW0MjFKxpjrjSyvzahPPr2W0jUm8xm0ojy4Zx+Mz77xP4iw5eoliLlDB?= =?iso-8859-1?Q?UOGjBfI1EIsUK+iqFYyVpT+1wtz8vC3v89iMgA=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SA1PR13MB4925.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230037)(366013)(376011)(1800799021)(38070700015); DIR:OUT; SFP:1102 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?HdcC8sfktgfgEEOo516wqC8N6Uuhcufh4yDdd+jh/PrcHQXoiydAGbEzJN?= =?iso-8859-1?Q?GN4wDiKCLP8rpR2FUwlZFos4A24Of3nLgTBXqfQgXaASjidG1y8nw+DGMs?= =?iso-8859-1?Q?DIA/BW1T9el0Tp8QMhul8eQB2FjjxiS+3l+zePJaue6WRc+FHnMirjpZMY?= =?iso-8859-1?Q?yGDSaWH53+T6sYVC9oELPCtB6yFbeqKDWd/VslIncaxpbrFuUky/GW2Sk5?= =?iso-8859-1?Q?GaGPJ6V3cIQWJUMR8/FQWRj1u/aZS/FdcsuRE9E+zvH5iVKfXcD8r95MAj?= =?iso-8859-1?Q?mdEnaH/wqXgHtFWyYduCqhEO7GfQtpqzloadPxGCgbMWiet936sxCdpNGw?= =?iso-8859-1?Q?N8r9TlFpwQbm2BaC2NNtByasA0toEfpyC4q4oWuS0Sph4KNvMmlOHYw+yn?= =?iso-8859-1?Q?8+/D/MET8inrHhoHeexDG+Eh6dZbjbrss9JaszUlcYxAc4T8gZfpHbPKgX?= =?iso-8859-1?Q?n4JLVnvxhGTlKTtx1s+W6w97Rfhsu7tHqO39/QqZwpvaIIK0MHW0qm1nQ6?= =?iso-8859-1?Q?oGOevQ2KMsmrKlbVOFWmPvvmDUCd1Tt2CxLZ1ueuqH7mOExc9yKIuGb5Y9?= =?iso-8859-1?Q?wDQgFUUrtaPX+i2WUYYXLZBUabXQ373CwiYGQNhsggEuRs56QqZ7dLiH99?= =?iso-8859-1?Q?lmcPniCikRUy2F6Ijb7fb/NxseL7a2OziAaUEWuMUeJGvyO1DkfARTrbY+?= =?iso-8859-1?Q?gRoJlS+Ek0Gjw7pZZI3GaldMvkx0BT8m5SoTIgI4ppvznHy7KKEtccfqyJ?= =?iso-8859-1?Q?2n/qlr14hjl33QAcQp19JEIIu5fvEksyplksgxMvfD5++kzsQkCg5Y7GJN?= =?iso-8859-1?Q?eVb0TD+c+kJjyT8zf8FZuZJLVC40fyt/GR/+LCl1365yFOH36wa5u6VE6G?= =?iso-8859-1?Q?ZfxOD8GKY1OE+bYho+a9Ne2n/2nYfZZNBewp+qSN8HaLpUO2IqzKMu85fL?= =?iso-8859-1?Q?fQJiqZrrUyv6dNRdaBDuHoo2oa8R2CEjO5MgYi10aULM65Wh2EBa9xCXdW?= =?iso-8859-1?Q?Mtdh+5r9FduXQ/GigLIFP2nigBN2IU9H0zq2XoPHuyyXD9p2e2dA2xfSaK?= =?iso-8859-1?Q?3nRX+iU92KJc2QQ6eZjKQ3HOWSSS3tY4E6GnBiY4UVOHcwoRsdLNmUWjUz?= =?iso-8859-1?Q?edTYIxhoHPvXs6Pw1e7f6XT4ztlVoFXRm00mtcAZkVu3IVQsa505/Az+Xa?= =?iso-8859-1?Q?Zdr84nNDlcrbJCIji5Kx9VZJ+SQE0NIOu550cIGTyjhgA2bqZr5CGw2kTh?= =?iso-8859-1?Q?/1VnbZLjwtNPNcpiuI5TscL9h7jfRVv62KKMe9AI5l2ZDYtn01HOgNSdGI?= =?iso-8859-1?Q?5jTyqinTKoFs/N4+uqbq5ubeA/r87bMTALhfCpuC3vmsJepQnRE91gpdXm?= =?iso-8859-1?Q?NrNz0lUlSfxJVQCV6PvSszo3fpb45r2+H0/5axA+OIsiwvIZqHlEKzK/bp?= =?iso-8859-1?Q?KbpswiQU0ZhDOFBH/c3Nes0R+cAv6yuIMZ0LbEg2M1Wjk/N+zhrTK9SMus?= =?iso-8859-1?Q?qLp/CK4M8TITpISqczXPuMje/C6mvQdhOwpq8cUWc3gbUKn1tExauGgHh+?= =?iso-8859-1?Q?xoFjg3byn/9O53qdvPqtahpvNZbt4umwy5b4+0TCnfbq7Nduhsx2IquZwd?= =?iso-8859-1?Q?xx5biyHB9qioTBF5kuQuEEtFdtNVz0apu6?= MIME-Version: 1.0 X-OriginatorOrg: progress.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA1PR13MB4925.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: edd020e7-5876-4102-bc90-08dc95038680 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Jun 2024 10:42:35.3519 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: db266a67-cbe0-4d26-ae1a-d0581fe03535 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: TKwIMSVB/vArvnsEqfe0w9U6bARdoizexvep/SakX9I4rmAlq4c5ImFZEG2NvAewGo3Q5E1vYJ1hQ84SlBstHw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS1PR13MB7121 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: progress.com Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_SA1PR13MB492543CEAA58C87A607EA484E6D52SA1PR13MB4925namp_" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_SA1PR13MB492543CEAA58C87A607EA484E6D52SA1PR13MB4925namp_ Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Hello, I am experiencing issues with DPDK (21.11) mlx5 PMD driver when allocating = hugepages on NUMA node 1. Card info: Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] Subsystem: Mellanox Technologies Device 0020 NUMA node: 1 Driver: mlx5_core Version: 5.7-1.0.2 Firmware-version: 26.36.1010 (DEL0000000031) The card is clearly linked to NUMA node 1, so I assigned some hugepages onl= y to the NUMA node 1. cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepag= es 0 cat /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepag= es 4 However, when I run my DPDK application, it presents with the following out= put and fails: EAL: Detected CPU lcores: 24 EAL: Detected NUMA nodes: 2 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/0000:98:00.0/mp_socket EAL: Selected IOVA mode 'PA' EAL: No free 1048576 kB hugepages reported on node 0 EAL: No free 2048 kB hugepages reported on node 0 EAL: No free 2048 kB hugepages reported on node 1 EAL: No available 2048 kB hugepages reported EAL: Probe PCI driver: mlx5_pci (15b3:101f) device: 0000:98:00.0 (socket 1) mlx5_net: Failed to create ASO bits mem for MR. EAL: Error: Invalid memory mlx5_net: probe of PCI device 0000:98:00.0 aborted after encountering an er= ror: Operation not permitted mlx5_common: Failed to load driver mlx5_eth EAL: Requested device 0000:98:00.0 cannot be used EAL: Bus (pci) probe failed. If I allocate hugepages only for NUMA node 0, it fails with: mlx5_common: Failed to initialize global MR share cache. So the only working solution for me currently is to have hugepages allocate= d for both NUMA nodes, which is weird, considering that e.g., i40e PMD work= s completely fine with having hugepages only on a single NUMA node. Is there a way to force the mlx5 PMD to allocate everything on a single spe= cific NUMA node? Any advice is appreciated Tomas --_000_SA1PR13MB492543CEAA58C87A607EA484E6D52SA1PR13MB4925namp_ Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Hel= lo,

I am experiencing issues with DPDK (21.11) mlx5 PMD driver when allocating = hugepages on NUMA node 1.

Card info:
  Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6= Lx]
  Subsystem: Mellanox Technologies Device 0020
  NUMA node: 1
  Driver: mlx5_core
  Version: 5.7-1.0.2
  Firmware-version: 26.36.1010 (DEL0000000031)

The card is clearly linked to NUMA node 1, so I assigned some hugepages onl= y to the NUMA node 1.
cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepag= es 
0
cat /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepag= es 
4

However, when I run my DPDK application, it presents with the following out= put and fails:
EAL: Detected CPU lcores: 24
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/0000:98:00.0/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free 1048576 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: No available 2048 kB hugepages reported
EAL: Probe PCI driver: mlx5_pci (15b3:101f) device: 0000:98:00.0 (socket 1)=
mlx5_net: Failed to create ASO bits mem for MR.
EAL: Error: Invalid memory
mlx5_net: probe of PCI device 0000:98:00.0 aborted after encountering an er= ror: Operation not permitted
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:98:00.0 cannot be used
EAL: Bus (pci) probe failed.

If I allocate hugepages only for NUMA node 0, it fails with:
mlx5_common: Failed to initialize global MR share cache.

So the only working solution for me currently is to have hugepages allocate= d for both NUMA nodes, which is weird, considering that e.g., i40e PMD work= s completely fine with having hugepages only on a single NUMA node.

Is there a way to force the mlx5 PMD to allocate everything on a single spe= cific NUMA node?

Any advice is appreciated
Tomas

--_000_SA1PR13MB492543CEAA58C87A607EA484E6D52SA1PR13MB4925namp_--