From: Gabriel Danjon <gabriel.danjon@cybertestsystems.com.sg>
To: users@dpdk.org
Cc: Alexis DANJON <alexis.danjon@cybertestsystems.com.sg>,
Antoine LORIN <antoine.lorin@cybertestsystems.com.sg>,
Laurent CHABENET <laurent.chabenet@cybertestsystems.com.sg>,
"gregory.fresnais@cybertestsystems.com.sg"
<gregory.fresnais@cybertestsystems.com.sg>,
Julien RAMET <julien.ramet@cybertestsystems.com.sg>
Subject: [dpdk-users] Unable to setup hugepages
Date: Mon, 31 May 2021 17:35:38 +0200 [thread overview]
Message-ID: <48a96b06-3ffd-97df-5176-97096a0758bc@cybertestsystems.com.sg> (raw)
Hello,
After successfully installed the DPDK 20.11 on my Centos 8-Stream
(minimal), I am trying to configure the hugepages but encounters a lot
of difficulties.
I am trying to reserve 4 hugepages of 1GB.
Here the steps I have done following the documentation
(https://doc.dpdk.org/guides-20.11/linux_gsg/sys_reqs.html):
Additional information about meminfo :
cat /proc/meminfo
MemTotal: 32619404 kB
MemFree: 27331024 kB
MemAvailable: 27415524 kB
Buffers: 4220 kB
Cached: 328628 kB
SwapCached: 0 kB
Active: 194828 kB
Inactive: 210156 kB
Active(anon): 1744 kB
Inactive(anon): 83384 kB
Active(file): 193084 kB
Inactive(file): 126772 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16474108 kB
SwapFree: 16474108 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 72136 kB
Mapped: 84016 kB
Shmem: 12992 kB
KReclaimable: 211956 kB
Slab: 372852 kB
SReclaimable: 211956 kB
SUnreclaim: 160896 kB
KernelStack: 9120 kB
PageTables: 6852 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 30686656 kB
Committed_AS: 270424 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 28416 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 4194304 kB
DirectMap4k: 225272 kB
DirectMap2M: 4919296 kB
DirectMap1G: 30408704 kB
1 Step follow documentation
bash -c 'echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages'
As we're working on a NUMA machine we do this too. (We even do the
previous step because without it, it provides more errors)
bash -c 'echo 2048 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages' && \
bash -c 'echo 2048 >
/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages'
mkdir /mnt/huge
mount -t hugetlbfs pagesize=1GB /mnt/huge
bash -c 'echo nodev /mnt/huge hugetlbfs pagesize=1GB 0 0 >> /etc/fstab'
So here the result of my meminfo (cat /proc/meminfo | grep Huge) :
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 4194304 kB
It looks strange that there is no total and free hugepages.
I tried the dpdk-testpmd using the DPDK documentation : dpdk-testpmd -l
0-3 -n 4 -- -i --nb-cores=2
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs
found for that size
EAL: No free hugepages reported in hugepages-1048576kB
EAL: No free hugepages reported in hugepages-1048576kB
EAL: No available hugepages reported in hugepages-1048576kB
EAL: FATAL: Cannot get hugepage information.
EAL: Cannot get hugepage information.
EAL: Error - exiting with code: 1
Cause: Cannot init EAL: Permission denied
So I checked in the /mnt/huge to look if files had been created (ls
/mnt/huge/ -la) : Empty folder
Then I checked if my folder was correctly mounted : mount | grep huge
pagesize=1GB on /mnt/huge type hugetlbfs
(rw,relatime,seclabel,pagesize=1024M)
Then I tried the helloworld example (make clean && make &&
./build/helloworld):
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs
found for that size
EAL: No free 1048576 kB hugepages reported on node 0
EAL: No free 1048576 kB hugepages reported on node 1
EAL: No available 1048576 kB hugepages reported
EAL: FATAL: Cannot get hugepage information.
EAL: Cannot get hugepage information.
PANIC in main():
Cannot init EAL
5: [./build/helloworld() [0x40079e]]
4: [/lib64/libc.so.6(__libc_start_main+0xf3) [0x7ff43a6f6493]]
3: [./build/helloworld() [0x4006e6]]
2: [/usr/local/lib64/librte_eal.so.21(__rte_panic+0xba) [0x7ff43aaa4b93]]
1: [/usr/local/lib64/librte_eal.so.21(rte_dump_stack+0x1b) [0x7ff43aac79fb]]
Aborted (core dumped)
So I guessed the problem came from the : Hugepagesize: 1048576 kB
(from cat /proc/meminfo | grep Huge).
2 Step adapt documentation
Then I decided to set the values for 1048576KB:
bash -c 'echo 4 >
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages'
bash -c 'echo 4 >
/sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages'
bash -c 'echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'
So here the result of my meminfo (cat /proc/meminfo | grep Huge) :
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 4
HugePages_Free: 4
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 8388608 kB
So here I have my 4 pages sat.
Then I retried the previous steps and here what I got :
dpdk-testpmd -l 0-3 -n 4 -- -i --nb-cores=2
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs
found for that size
EAL: Probing VFIO support...
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd>
Bye...
make clean && make && ./build/helloworld
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs
found for that size
TELEMETRY: No legacy callbacks, legacy socket not created
cat /proc/meminfo | grep Huge
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 4
HugePages_Free: 3
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 8388608 kB
One huge page looks like have been used.
ls -l /mnt/huge/
total 1048576
1073741824 rtemap_0
So yes one has been created, but 2048 hugepages of size 2097152
reserved, but no mounted hugetlbfs found for that size, happens.
So to try to understand what happens I reset
hugepages-2048kB/nr_hugepages to 0 :
bash -c 'echo 0 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages' && \
bash -c 'echo 0 >
/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages' && \
bash -c 'echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages'
but : dpdk-testpmd -l 0-3 -n 4 -- -i --nb-cores=2EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
So here on this part I really don't understand, my /proc/meminfo tells
me I need to use the 1048576kB part but dpdk-testpmd the 2048kB.
Then I searched an alternative to these commands in your documentation
and found : dpdk-hugpages.py
3 Step alternative
https://doc.dpdk.org/guides-20.11/tools/hugepages.html
(There is an error on the documentation : dpdk-hugpages instead of
dpdk-hugepages)
So I reset every files, and removed my mount and my folder.
umount /mnt/huge
rm -rf /mnt/huge
bash -c 'echo 0 >
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages'
bash -c 'echo 0 >
/sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages'
bash -c 'echo 0 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'
cat /proc/meminfo | grep Huge
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 0 kB
dpdk-hugpages.py -s
Node Pages Size Total
Hugepages not mounted
So here I have an cleaned hugepage environment.
Then I tried to reallocate hugepages with the python script:
dpdk-hugepages.py -p 1G --setup 4G
dpdk-hugepages.py -s
Node Pages Size Total
0 4 1Gb 4Gb
1 4 1Gb 4G
So I got my 4 pages of 1GB and retried the previous steps :
cat /proc/meminfo | grep Huge
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 8
HugePages_Free: 8
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 8388608 kB
Here it says I got 8 hugepages of 1GB so I don't understand why, because
the python script tells the opposite.
dpdk-testpmd -l 0-3 -n 4 -- -i --nb-cores=2EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
Same for the helloworld.
Then I cleared my environment:
dpdk-hugepages.py -u && dpdk-hugepages.py -c && dpdk-hugepages.py -s
Node Pages Size Total
Hugepages not mounted
Then as the error says that there are no available hugepages reported in
hugepages-2048kB i tried with Mb :
dpdk-hugepages.py -p 1024M --setup 4G && dpdk-hugepages.py -s
Node Pages Size Total
0 4 1Gb 4Gb
1 4 1Gb 4Gb
But same error happened.
4 Question
So I don't succeed to resolve this issue of testing the DPDK with
helloworld and dpdk-testpmd.
Have I missed something in the creation of the hugepages ?
Could you please, provide help ?
Best,
--
Gabriel Danjon
next reply other threads:[~2021-06-01 7:45 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-31 15:35 Gabriel Danjon [this message]
2021-06-01 7:58 ` Thomas Monjalon
2021-06-02 15:35 ` Gabriel Danjon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=48a96b06-3ffd-97df-5176-97096a0758bc@cybertestsystems.com.sg \
--to=gabriel.danjon@cybertestsystems.com.sg \
--cc=alexis.danjon@cybertestsystems.com.sg \
--cc=antoine.lorin@cybertestsystems.com.sg \
--cc=gregory.fresnais@cybertestsystems.com.sg \
--cc=julien.ramet@cybertestsystems.com.sg \
--cc=laurent.chabenet@cybertestsystems.com.sg \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).