[dpdk-users] No free hugepages after process restart

[dpdk-users] No free hugepages after process restart

[Michael K]

Hi,

I have a hugepages-related problem with a DPDK process not being able to
run after it was terminated and then restarted again.
When the process is restarted, the rte_eal_init() call fails and the terror
message is:

    EAL: No free hugepages reported in hugepages-2048kB
    EAL: FATAL: Cannot get hugepage information.

More details below.

DPDK version is 17.11.3. OS is Ubuntu 14/3.13.0-32-generic.

Hugepages are 2MB large. hugetlbfs is mounted in the process's startup
script, the amount of hugepages is set by
writing to
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages.

The amount of memory for DPDK to allocate is passed during the
initialization with the "--socket-mem" parameter.

After the initial start, DPDK works without any problems.
When I stop the process (with kill/TERM) and start it again, DPDK
initialization fails as I wrote above.
After the failure /proc/meminfo shows zero free hugepages.

So, since DPDK is complaining on "No free hugepages reported", I have tried
to find what other processes could be
using hugepages:
- transparent hugepages are disabled via the kernel's commandline.
- ipcs -a shows nothing
- there are no other hugetlbfs mounts on the system
- running lsof and trying to find files open in the hugetlbfs mount dir
gives nothing
- grepping 'KernelPageSize:     2048 kB' in /proc/*/smaps gives nothing
- grep huge /proc/*/numa_maps gives nothing

Basically I've tried every "find the hugepage user" trick I could find on
the net.

If after the failure I add more hugepages via
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages,
the process starts successfully and everything is fine again, until I kill
it and then it's not able to run again
with the same error message. And /proc/meminfo shows no free hugepages
again.


One suspicious detail: the problem happens only on machines with Docker
running. But I'm not sure whether this
is related or not.

Would appreciate any ideas.

Thanks,
Michael