From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7EACE454F1 for ; Tue, 25 Jun 2024 16:29:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5BCAE4069F; Tue, 25 Jun 2024 16:29:29 +0200 (CEST) Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) by mails.dpdk.org (Postfix) with ESMTP id A0A17402D0 for ; Tue, 25 Jun 2024 16:29:27 +0200 (CEST) Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-57d3125e0baso601444a12.0 for ; Tue, 25 Jun 2024 07:29:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719325767; x=1719930567; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=sNWkUJ5tBv+yJAj/I3QRW5E9uHy1H2Lf5YgAbTSpsAY=; b=i28g+mdhJVTQXLm5VPeznBOzezD0/Y766NOAR3blpp8ThFc/A8eg8OtJ3Uo3azV8Ki 3o0O7JoWWgO9epW5bA2P10D8rvFDSVH21lCxonuXBojX/UPr+tH3wjGXq33VEVGYEVee xCce6DFdP+9LIZwwU55wux2NQQ/MceBeYotb2WzAYZ0dnegFrqNPROdt8CK7AVLCi5E+ fTZX7j6yF6vsb2xSLR0sF+WB7hwtR2hrqrjXFM1W7UyntrlZ7yoqvBlM9WT0kSn8WlLk nzLsnzO1/u/8lSG7SkYKIbsArOF8E/kfmRayjVROvhXMDHdH3zmcuso2pre4t/yrTWgE 2Sog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719325767; x=1719930567; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sNWkUJ5tBv+yJAj/I3QRW5E9uHy1H2Lf5YgAbTSpsAY=; b=Z8zOWdJPe2mrxf0a7OIwbJRV8oMa9+fgDMq2wIR1CGWkI5S7ZPhz13TxFjyv0kcxqz SEtjbgDkq+crZEwIbaBGB+yfoDBgnf08bHY7QeFWQeYILyIS/rLuN9JLKDRFEvUK2siT SKqTjTBOpZQAxyCh0aGgdsY+iGQemHgdR/SbnRZUQZi4GDh5evE/h1zZ9xbdXZ66w4YY nCHFzPL2Xr3gx7n3eowiKkhQRcr18v77N4SEXz2bRztN84eIOXjQJMKa1wnU627K3jbM y0tqsl2eqYVxvVuWyzqROGXD3tapUYAm5HuYn4beclp3XKaoYzflmSeS4qKysfMJqOYR CTjQ== X-Gm-Message-State: AOJu0YwbI12uugtVSeuuu1AJPd43Jvx/HHg/98syNw0LNYHw+kac/qt/ cShhM1umQ9TS3d4ZgrKN5u7dVy23Sum/jginVK2ccqYVIhl8vXbw4wpEcI+R+SItPFJsQOYREy0 f0UfX8VDvcJcVf2cFmdhFaxSXtGo= X-Google-Smtp-Source: AGHT+IGG2eHesIO03wyrJmIBlj0i+0rbc81t36ZcM8AzjwaEmIc3Vor9IM35IyY/8SFRVwqQdrkO5Qyh74usufAM3Bo= X-Received: by 2002:a50:d54b:0:b0:57c:5cf8:69db with SMTP id 4fb4d7f45d1cf-57d41f68b49mr4839856a12.4.1719325766636; Tue, 25 Jun 2024 07:29:26 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Erez Ferber Date: Tue, 25 Jun 2024 16:29:13 +0200 Message-ID: Subject: Re: Unable to start mlx5 PMD on NUMA node 1 To: Tomas Jansky Cc: "users@dpdk.org" Content-Type: multipart/alternative; boundary="00000000000092db70061bb7b9a8" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --00000000000092db70061bb7b9a8 Content-Type: text/plain; charset="UTF-8" Make sure you're running with the below fix which should've been merged in v21.11.1 and up. https://mails.dpdk.org/archives/stable/2022-February/036254.html Regards, Erez On Tue, 25 Jun 2024 at 12:52, Tomas Jansky wrote: > Hello, > > I am experiencing issues with DPDK (21.11) mlx5 PMD driver when allocating > hugepages on NUMA node 1. > > Card info: > Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] > Subsystem: Mellanox Technologies Device 0020 > NUMA node: 1 > Driver: mlx5_core > Version: 5.7-1.0.2 > Firmware-version: 26.36.1010 (DEL0000000031) > > The card is clearly linked to NUMA node 1, so I assigned some hugepages > only to the NUMA node 1. > cat > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages > 0 > cat > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages > 4 > > However, when I run my DPDK application, it presents with the following > output and fails: > EAL: Detected CPU lcores: 24 > EAL: Detected NUMA nodes: 2 > EAL: Detected shared linkage of DPDK > EAL: Multi-process socket /var/run/dpdk/0000:98:00.0/mp_socket > EAL: Selected IOVA mode 'PA' > EAL: No free 1048576 kB hugepages reported on node 0 > EAL: No free 2048 kB hugepages reported on node 0 > EAL: No free 2048 kB hugepages reported on node 1 > EAL: No available 2048 kB hugepages reported > EAL: Probe PCI driver: mlx5_pci (15b3:101f) device: 0000:98:00.0 (socket 1) > mlx5_net: Failed to create ASO bits mem for MR. > EAL: Error: Invalid memory > mlx5_net: probe of PCI device 0000:98:00.0 aborted after encountering an > error: Operation not permitted > mlx5_common: Failed to load driver mlx5_eth > EAL: Requested device 0000:98:00.0 cannot be used > EAL: Bus (pci) probe failed. > > If I allocate hugepages only for NUMA node 0, it fails with: > mlx5_common: Failed to initialize global MR share cache. > > So the only working solution for me currently is to have hugepages > allocated for both NUMA nodes, which is weird, considering that e.g., i40e > PMD works completely fine with having hugepages only on a single NUMA node. > > Is there a way to force the mlx5 PMD to allocate everything on a single > specific NUMA node? > > Any advice is appreciated > Tomas > > --00000000000092db70061bb7b9a8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Make sure you're running=C2=A0with the below fix which= should've been merged in v21.11.1 and up.


Regards,
Erez

On Tue, 25 Jun 2024 at 12:52= , Tomas Jansky <Tomas.Jansk= y@progress.com> wrote:
Hello,

I am experiencing issues with DPDK (21.11) mlx5 PMD driver when allocating = hugepages on NUMA node 1.

Card info:
=C2=A0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6= Lx]
=C2=A0 Subsystem: Mellanox Technologies Device 0020
=C2=A0 NUMA node: 1
=C2=A0 Driver: mlx5_core
=C2=A0 Version: 5.7-1.0.2
=C2=A0 Firmware-version: 26.36.1010 (DEL0000000031)

The card is clearly linked to NUMA node 1, so I assigned some hugepages onl= y to the NUMA node 1.
cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepag= es=C2=A0
0
cat /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepag= es=C2=A0
4

However, when I run my DPDK application, it presents with the following out= put and fails:
EAL: Detected CPU lcores: 24
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/0000:98:00.0/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free 1048576 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: No available 2048 kB hugepages reported
EAL: Probe PCI driver: mlx5_pci (15b3:101f) device: 0000:98:00.0 (socket 1)=
mlx5_net: Failed to create ASO bits mem for MR.
EAL: Error: Invalid memory
mlx5_net: probe of PCI device 0000:98:00.0 aborted after encountering an er= ror: Operation not permitted
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:98:00.0 cannot be used
EAL: Bus (pci) probe failed.

If I allocate hugepages only for NUMA node 0, it fails with:
mlx5_common: Failed to initialize global MR share cache.

So the only working solution for me currently is to have hugepages allocate= d for both NUMA nodes, which is weird, considering that e.g., i40e PMD work= s completely fine with having hugepages only on a single NUMA node.

Is there a way to force the mlx5 PMD to allocate everything on a single spe= cific NUMA node?

Any advice is appreciated
Tomas

--00000000000092db70061bb7b9a8--