From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9F7A543DB2 for ; Tue, 2 Apr 2024 09:15:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1493F4028C; Tue, 2 Apr 2024 09:15:13 +0200 (CEST) Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by mails.dpdk.org (Postfix) with ESMTP id D29A84026F for ; Tue, 2 Apr 2024 09:15:11 +0200 (CEST) Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-69625f89aa2so39119006d6.3 for ; Tue, 02 Apr 2024 00:15:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712042111; x=1712646911; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=GGfIegds3tNSNNj7BHoz8HI9q2ThTFJq+bNK1WJ6DQw=; b=DuiKBZT6EjgkJyMWiBBIWqR7Cj/ttwiTNWkpA5o3y0BQlAzpBQfiwrkHeN172VcmWm I4kpy1zRhikBZM1SmUqcgZeH/VJ9nZrigClytnbrXoUSdK8AtJsRUSd3bIt23mDVF4aN J04icDIjbEO1+VLJDhPirA7EPRmf1gW0Hn28d8DtMpFjOGY14KTSb74iKcP6lxSbIY7K hXXW/f5V7zpDRQie8YzL8JICVjD/Aut87PhR5ruT+U8fi12jwet6zlLvPu1Vwxi1afD2 RFAOIvFGAlPU/b6yr6L6kF/WleRlnO20RWjntDbvTZhQGSjBrBgqrDyON049fhQm8aqU oppg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712042111; x=1712646911; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GGfIegds3tNSNNj7BHoz8HI9q2ThTFJq+bNK1WJ6DQw=; b=s1mVS0IgYO3FHSKJZ3s2dV9+e3zAWGf7+CrHZYdSUabCbm1fURDsYuvyPGclgaB5J4 RivF8ZWGUqKbKqlVus37+BUhHzIHXY7ircEo/yQktifrsgOvqfGEXedjYFbLKHDDvVrm cFYPWzPFE6WYRZbqtnyw846V+gk3/DjZdwncDOzb+C49enKoigj5QgFPMX4dS48WRzXc o/BZAdjnABFhITHXUgrMqdSSUz6GTIeITJCAip10/VH+dOUrOR9suUXV0liWK7YzuZT2 oJr0wQ93Lg1kz5XAAp0zZzLDEp5sGNCOtH2SCZaIutxMOLkiAWtZukB/wsxS+TT/RkmK hb+A== X-Gm-Message-State: AOJu0YzdXUCFXeBiGVUmDWlPB1Q7BiP9yeXnGzlsOyWeKag/YynOwS6I eo9teXVGabFT1SiCuozokjjbtnLnd265o1HXmRIGSgIl9lwqLi3Lu7/dcx3DN1NKJhy3QrtYRqP Ofyh0WDYQE7xct8uTpn1ItbHlVxwzhRz+aoQ= X-Google-Smtp-Source: AGHT+IHlX82NGfxfUfnM7ZOqRw/B1Gz8FWTHBOzFn7GGK5iIOyYC3e39+GWaLx/T1iXbkCLAJgiIVxSOUerkPN/WpuA= X-Received: by 2002:a05:6214:5ed1:b0:699:1fcc:ca3d with SMTP id mn17-20020a0562145ed100b006991fccca3dmr174948qvb.38.1712042110855; Tue, 02 Apr 2024 00:15:10 -0700 (PDT) MIME-Version: 1.0 References: <20240331120633.05256df7@hermes.local> In-Reply-To: From: Lokesh Chakka Date: Tue, 2 Apr 2024 12:44:59 +0530 Message-ID: Subject: Re: No free hugepages reported To: Stephen Hemminger Cc: users Content-Type: multipart/alternative; boundary="000000000000dbb252061517dd7b" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000dbb252061517dd7b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable hi, To add more information, the server I'm using has two CPU sockets and two NUMA nodes. One is numbered as Node 2 and other one as Node 6 one more observation is the following command is getting executed successfully. $ sudo dpdk-hugepages.py -p 1G --setup 2G -n 2 also the following one $ sudo dpdk-hugepages.py -p 1G --setup 2G -n 6 After executing the first command, 1G huge pages are getting created. After executing the second command, huge pages under node 2 are getting deleted. Following is the output of dpdk-testpmd command ***************************************************************************= ************************* EAL: Detected CPU lcores: 128 EAL: Detected NUMA nodes: 8 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No free 1048576 kB hugepages reported on node 0 EAL: No free 1048576 kB hugepages reported on node 1 EAL: No free 1048576 kB hugepages reported on node 3 EAL: No free 1048576 kB hugepages reported on node 4 EAL: No free 1048576 kB hugepages reported on node 5 EAL: No free 1048576 kB hugepages reported on node 6 EAL: No free 1048576 kB hugepages reported on node 7 EAL: VFIO support initialized set_mempolicy: Invalid argument set_mempolicy: Invalid argument set_mempolicy: Invalid argument set_mempolicy: Invalid argument EAL: Using IOMMU type 1 (Type 1) set_mempolicy: Invalid argument set_mempolicy: Invalid argument EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:63:00.0 (socket 3) set_mempolicy: Invalid argument PANIC in eth_dev_shared_data_prepare(): Cannot allocate ethdev shared data 0: /lib/x86_64-linux-gnu/librte_eal.so.23 (rte_dump_stack+0x41) [7dbb0fe000b1] 1: /lib/x86_64-linux-gnu/librte_eal.so.23 (__rte_panic+0xc1) [7dbb0fde11c7] 2: /lib/x86_64-linux-gnu/librte_ethdev.so.23 (7dbb0fedb000+0x8b16) [7dbb0fee3b16] 3: /lib/x86_64-linux-gnu/librte_ethdev.so.23 (rte_eth_dev_allocate+0x31) [7dbb0feef971] 4: /usr/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ice.so.23 (7dbb0f70e000+0x67465) [7dbb0f775465] 5: /usr/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_pci.so.23 (7dbb0fc64000+0x4c76) [7dbb0fc68c76] 6: /usr/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_pci.so.23 (7dbb0fc64000+0x8af4) [7dbb0fc6caf4] 7: /lib/x86_64-linux-gnu/librte_eal.so.23 (rte_bus_probe+0x23) [7dbb0fdeeab3] 8: /lib/x86_64-linux-gnu/librte_eal.so.23 (7dbb0fdd4000+0x123bf) [7dbb0fde63bf] 9: dpdk-testpmd (5813e0022000+0x45150) [5813e0067150] 10: /lib/x86_64-linux-gnu/libc.so.6 (7dbb0ec00000+0x28150) [7dbb0ec28150] 11: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0x89) [7dbb0ec28209] 12: dpdk-testpmd (5813e0022000+0x48e55) [5813e006ae55] Aborted ***************************************************************************= ************************* Thanks & Regards -- Lokesh Chakka. On Mon, Apr 1, 2024 at 2:50=E2=80=AFAM Lokesh Chakka wrote: > hi Stephen, > > Thanks for the reply. Following is the observation... > > ************************************************************* > $ dpdk-hugepages.py -s > Node Pages Size Total > 2 512 2Mb 1Gb > 6 512 2Mb 1Gb > > Hugepages mounted on /dev/hugepages /mnt/huge > > $ sudo dpdk-hugepages.py -p 1G --setup 2G > Unable to set pages (0 instead of 2 in > /sys/devices/system/node/node4/hugepages/hugepages-1048576kB/nr_hugepages= ). > ************************************************************* > > > Regards > -- > Lokesh Chakka. > > > On Mon, Apr 1, 2024 at 12:36=E2=80=AFAM Stephen Hemminger < > stephen@networkplumber.org> wrote: > >> On Sun, 31 Mar 2024 16:28:19 +0530 >> Lokesh Chakka wrote: >> >> > Hello, >> > >> > I've installed dpdk in Ubuntu 23.10 with the command "sudo apt -y >> install >> > dpdk*" >> > >> > added "nodev /mnt/huge hugetlbfs pagesize=3D1GB 0 0" in /etc/fstab >> > added "vm.nr_hugepages=3D1024" in /etc/sysctl.conf >> > >> > rebooted the machine and then did devbind using the following command: >> > >> > sudo modprobe vfio-pci && sudo dpdk-devbind.py --bind=3Dvfio-pci 63:00= .0 >> > 63:00.1 >> > >> > Huge page info is as follows : >> > >> > ************************************************* >> > $ cat /proc/meminfo | grep Huge >> > AnonHugePages: 6144 kB >> > ShmemHugePages: 0 kB >> > FileHugePages: 0 kB >> > HugePages_Total: 1024 >> > HugePages_Free: 1023 >> > HugePages_Rsvd: 0 >> > HugePages_Surp: 0 >> > Hugepagesize: 2048 kB >> > Hugetlb: 2097152 kB >> > ************************************************* >> >> Your hugepages are not setup correctly. The mount is for 1G pages >> and the sysctl entry makes 2M pages. >> >> Did you try using the dpdk-hugepages script? >> > --000000000000dbb252061517dd7b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
hi,

To add more information,= the server I'm using has two CPU sockets and two NUMA nodes. One is nu= mbered as Node 2 and other one as Node 6
one more observation= is the following command is getting executed successfully.
$= sudo dpdk-hugepages.py -p 1G --setup 2G -n 2
also the following = one
$ sudo dpdk-hugepa= ges.py -p 1G --setup 2G -n 6
After executing the first command, 1= G huge pages are getting created. After executing the second command, huge = pages under node 2 are getting deleted.

Following is the output of dpdk-testpmd command

<= div>
******************************************************************= **********************************
EAL: Detected CPU lcores: = 128
EAL: Detected NUMA nodes: 8
EAL: Detected s= hared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk= /rte/mp_socket
EAL: Selected IOVA mode 'VA'
=
EAL: No free 1048576 kB hugepages reported on node 0
EAL= : No free 1048576 kB hugepages reported on node 1
EAL: No fre= e 1048576 kB hugepages reported on node 3
EAL: No free 104857= 6 kB hugepages reported on node 4
EAL: No free 1048576 kB hug= epages reported on node 5
EAL: No free 1048576 kB hugepages r= eported on node 6
EAL: No free 1048576 kB hugepages reported = on node 7
EAL: VFIO support initialized
set_mem= policy: Invalid argument
set_mempolicy: Invalid argument
<= /div>
set_mempolicy: Invalid argument
set_mempolicy: Inva= lid argument
EAL: Using IOMMU type 1 (Type 1)
s= et_mempolicy: Invalid argument
set_mempolicy: Invalid argumen= t
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:63:= 00.0 (socket 3)
set_mempolicy: Invalid argument
PANIC in eth_dev_shared_data_prepare():
Cannot allocate ethd= ev shared data
0: /lib/x86_64-linux-gnu/librte_eal.so.23 (rte= _dump_stack+0x41) [7dbb0fe000b1]
1: /lib/x86_64-linux-gnu/lib= rte_eal.so.23 (__rte_panic+0xc1) [7dbb0fde11c7]
2: /lib/x86_6= 4-linux-gnu/librte_ethdev.so.23 (7dbb0fedb000+0x8b16) [7dbb0fee3b16]
3: /lib/x86_64-linux-gnu/librte_ethdev.so.23 (rte_eth_dev_allocate+= 0x31) [7dbb0feef971]
4: /usr/lib/x86_64-linux-gnu/dpdk/pmds-2= 3.0/librte_net_ice.so.23 (7dbb0f70e000+0x67465) [7dbb0f775465]
5: /usr/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_pci.so.23 (7dbb0fc= 64000+0x4c76) [7dbb0fc68c76]
6: /usr/lib/x86_64-linux-gnu/dpd= k/pmds-23.0/librte_bus_pci.so.23 (7dbb0fc64000+0x8af4) [7dbb0fc6caf4]
7: /lib/x86_64-linux-gnu/librte_eal.so.23 (rte_bus_probe+0x23) [7d= bb0fdeeab3]
8: /lib/x86_64-linux-gnu/librte_eal.so.23 (7dbb0f= dd4000+0x123bf) [7dbb0fde63bf]
9: dpdk-testpmd (5813e0022000+= 0x45150) [5813e0067150]
10: /lib/x86_64-linux-gnu/libc.so.6 (= 7dbb0ec00000+0x28150) [7dbb0ec28150]
11: /lib/x86_64-linux-gn= u/libc.so.6 (__libc_start_main+0x89) [7dbb0ec28209]
12: dpdk-= testpmd (5813e0022000+0x48e55) [5813e006ae55]
Aborted

*****************************************************= ***********************************************



Th= anks & Regards
--
Lokesh Chakka.


On Mon, Apr 1, 2024 at 2:50=E2= =80=AFAM Lokesh Chakka <lvenkatakumarchakka@gmail.com> wrote:
hi Stephen,
<= br>
Thanks for the reply. Following is the observation...

*********************************************************= ****
$ dpdk-hugepages.py -s
Node Pages Size Total
2 =C2=A0 = =C2=A0512 =C2=A0 2Mb =C2=A0 =C2=A01Gb
6 =C2=A0 =C2=A0512 =C2=A0 2Mb =C2= =A0 =C2=A01Gb

Hugepages mounted on /dev/hugepages /mnt/huge

$ sudo dpdk-hugepages.py -p 1G --setup 2G
Unable to= set pages (0 instead of 2 in /sys/devices/system/node/node4/hugepages/huge= pages-1048576kB/nr_hugepages).
******************************= *******************************


Regards
--<= br>Lokesh Chakka.


On Mon, Apr 1, 2024 at 12:36=E2=80=AFAM Stephen Hemminger= <stephe= n@networkplumber.org> wrote:
On Sun, 31 Mar 2024 16:28:19 +0530
Lokesh Chakka <lvenkatakumarchakka@gmail.com> wrote:

> Hello,
>
> I've installed dpdk in Ubuntu 23.10 with the command "sudo ap= t -y install
> dpdk*"
>
> added=C2=A0 "nodev /mnt/huge hugetlbfs pagesize=3D1GB 0 0" i= n /etc/fstab
> added "vm.nr_hugepages=3D1024" in /etc/sysctl.conf
>
> rebooted the machine and then did devbind using the following command:=
>
> sudo modprobe vfio-pci && sudo dpdk-devbind.py --bind=3Dvfio-p= ci 63:00.0
> 63:00.1
>
> Huge page info is as follows :
>
> *************************************************
> $ cat /proc/meminfo | grep Huge
> AnonHugePages:=C2=A0 =C2=A0 =C2=A0 6144 kB
> ShmemHugePages:=C2=A0 =C2=A0 =C2=A0 =C2=A0 0 kB
> FileHugePages:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 kB
> HugePages_Total:=C2=A0 =C2=A0 1024
> HugePages_Free:=C2=A0 =C2=A0 =C2=A01023
> HugePages_Rsvd:=C2=A0 =C2=A0 =C2=A0 =C2=A0 0
> HugePages_Surp:=C2=A0 =C2=A0 =C2=A0 =C2=A0 0
> Hugepagesize:=C2=A0 =C2=A0 =C2=A0 =C2=A02048 kB
> Hugetlb:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02097152 kB
> *************************************************

Your hugepages are not setup correctly. The mount is for 1G pages
and the sysctl entry makes 2M pages.

Did you try using the dpdk-hugepages script?
--000000000000dbb252061517dd7b--