[dpdk-users] DPDK application is hunged in eal_hugepage_info_init

[dpdk-users] DPDK application is hunged in eal_hugepage_info_init

[Jatin Sahu]

Hi,

We are using DPDK libraries with huge pages enabled.
We observed that the application rte_eal_init() function is stuck in
flock() call.

Please advise how to resolve this.

DPDK Version: dpdk-stable-18.11.6/

[root@loadtestsrv helloworld]# ./build/helloworld -l 0-3 -n 4
EAL: Detected 72 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

Stack Trace:
Thread 3 (Thread 0x7f783c34e700 (LWP 1034)):
#0  0x00007f783c664113 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000558724 in eal_intr_thread_main ()
#2  0x00007f783c939dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f783c663b3d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f783bb4d700 (LWP 1035)):
#0  0x00007f783c940bfd in recvmsg () from /lib64/libpthread.so.0
#1  0x0000000000565bae in mp_handle ()
#2  0x00007f783c939dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f783c663b3d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f783d66dc00 (LWP 1033)):
#0  0x00007f783c655167 in flock () from /lib64/libc.so.6
#1  0x000000000054e068 in eal_hugepage_info_init ()
#2  0x000000000054ccfe in rte_eal_init ()
#3  0x00000000004712a6 in main ()
[root@loadtestsrv user]#

HugePage details:

[root@loadtestsrv roamware]# cat /proc/meminfo | grep -i huge
AnonHugePages:    112640 kB
HugePages_Total:      32
HugePages_Free:       32
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

[roamware@loadtestsrv ~]$ cat
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
512
[roamware@loadtestsrv ~]$ cat
/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
512

[root@loadtestsrv roamware]# cat /proc/mounts | grep -i huge
cgroup /sys/fs/cgroup/hugetlb cgroup
rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0
nodev /mnt/huge hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0
nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0

Regards,
Jatin

Re: [dpdk-users] DPDK application is hunged in eal_hugepage_info_init

[Stephen Hemminger]

On Wed, 12 Aug 2020 16:59:33 +0530
Jatin Sahu <jatin152@gmail.com> wrote:

> Hi,
> 
> We are using DPDK libraries with huge pages enabled.
> We observed that the application rte_eal_init() function is stuck in
> flock() call.
> 
> Please advise how to resolve this.
> 
> DPDK Version: dpdk-stable-18.11.6/
> 
> [root@loadtestsrv helloworld]# ./build/helloworld -l 0-3 -n 4
> EAL: Detected 72 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> 
> Stack Trace:
> Thread 3 (Thread 0x7f783c34e700 (LWP 1034)):
> #0  0x00007f783c664113 in epoll_wait () from /lib64/libc.so.6
> #1  0x0000000000558724 in eal_intr_thread_main ()
> #2  0x00007f783c939dd5 in start_thread () from /lib64/libpthread.so.0
> #3  0x00007f783c663b3d in clone () from /lib64/libc.so.6
> Thread 2 (Thread 0x7f783bb4d700 (LWP 1035)):
> #0  0x00007f783c940bfd in recvmsg () from /lib64/libpthread.so.0
> #1  0x0000000000565bae in mp_handle ()
> #2  0x00007f783c939dd5 in start_thread () from /lib64/libpthread.so.0
> #3  0x00007f783c663b3d in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x7f783d66dc00 (LWP 1033)):
> #0  0x00007f783c655167 in flock () from /lib64/libc.so.6
> #1  0x000000000054e068 in eal_hugepage_info_init ()
> #2  0x000000000054ccfe in rte_eal_init ()
> #3  0x00000000004712a6 in main ()
> [root@loadtestsrv user]#
> 
> HugePage details:
> 
> [root@loadtestsrv roamware]# cat /proc/meminfo | grep -i huge
> AnonHugePages:    112640 kB
> HugePages_Total:      32
> HugePages_Free:       32
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:    1048576 kB
> 
> [roamware@loadtestsrv ~]$ cat
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> 512
> [roamware@loadtestsrv ~]$ cat
> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
> 512
> 
> [root@loadtestsrv roamware]# cat /proc/mounts | grep -i huge
> cgroup /sys/fs/cgroup/hugetlb cgroup
> rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
> nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0
> nodev /mnt/huge hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0
> nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0
> 
> Regards,
> Jatin

Maybe in this case Flock blocks if other process has the file locked?
Is there another process which has flocked the file?
Or you have a buggy kernel?