DPDK usage discussions
 help / color / Atom feed
From: Michael K <mkatworkcn@gmail.com>
To: users@dpdk.org
Subject: [dpdk-users] No free hugepages after process restart
Date: Thu, 30 Jul 2020 12:41:35 +0300
Message-ID: <CAPH7zrYJdmg4Y4N=oiZYZH=qVbjb5J9VKqica2govww-ziAM6g@mail.gmail.com> (raw)

Hi,

I have a hugepages-related problem with a DPDK process not being able to
run after it was terminated and then restarted again.
When the process is restarted, the rte_eal_init() call fails and the terror
message is:

    EAL: No free hugepages reported in hugepages-2048kB
    EAL: FATAL: Cannot get hugepage information.

More details below.

DPDK version is 17.11.3. OS is Ubuntu 14/3.13.0-32-generic.

Hugepages are 2MB large. hugetlbfs is mounted in the process's startup
script, the amount of hugepages is set by
writing to
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages.

The amount of memory for DPDK to allocate is passed during the
initialization with the "--socket-mem" parameter.

After the initial start, DPDK works without any problems.
When I stop the process (with kill/TERM) and start it again, DPDK
initialization fails as I wrote above.
After the failure /proc/meminfo shows zero free hugepages.

So, since DPDK is complaining on "No free hugepages reported", I have tried
to find what other processes could be
using hugepages:
- transparent hugepages are disabled via the kernel's commandline.
- ipcs -a shows nothing
- there are no other hugetlbfs mounts on the system
- running lsof and trying to find files open in the hugetlbfs mount dir
gives nothing
- grepping 'KernelPageSize:     2048 kB' in /proc/*/smaps gives nothing
- grep huge /proc/*/numa_maps gives nothing

Basically I've tried every "find the hugepage user" trick I could find on
the net.

If after the failure I add more hugepages via
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages,
the process starts successfully and everything is fine again, until I kill
it and then it's not able to run again
with the same error message. And /proc/meminfo shows no free hugepages
again.


One suspicious detail: the problem happens only on machines with Docker
running. But I'm not sure whether this
is related or not.

Would appreciate any ideas.

Thanks,
Michael

                 reply index

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAPH7zrYJdmg4Y4N=oiZYZH=qVbjb5J9VKqica2govww-ziAM6g@mail.gmail.com' \
    --to=mkatworkcn@gmail.com \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK usage discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/ public-inbox