Dpdk allocates more memory, than available physically (hugepages)

Dpdk allocates more memory, than available physically (hugepages)

[Szymon Szozda]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

Hey,
I'm running dpdk on a machine with 64GB of RAM. It is configured, so 16GiB
(16 x 1GiB chunks) of hugepage memory is reserved on boot. I was expecting
dpdk to consume only those 16GiB, but it seems it gets more than 30GiB of
virtual memory ( I base it on memory VSZ output of top command ). The
machine is 1 NUMA, 1 NIC. I did some debugging and I do not see any logic
which limits the memory consumption, basically it seems that
eal_dynmem_memseg_lists_init() will allocate the same amount, no matter how
much RAM is physically available.

Is it expected? How to know that setup will not crash due to
insufficient memory available? How to limit those memory consumption.by
dpdk?

[-- Attachment #2: Type: text/html, Size: 757 bytes --]

Re: Dpdk allocates more memory, than available physically (hugepages)

[Dmitry Kozlyuk]

2023-02-08 03:43 (UTC+0100), Szymon Szozda:
> Hey,
> I'm running dpdk on a machine with 64GB of RAM. It is configured, so 16GiB
> (16 x 1GiB chunks) of hugepage memory is reserved on boot. I was expecting
> dpdk to consume only those 16GiB, but it seems it gets more than 30GiB of
> virtual memory ( I base it on memory VSZ output of top command ). The
> machine is 1 NUMA, 1 NIC. I did some debugging and I do not see any logic
> which limits the memory consumption, basically it seems that
> eal_dynmem_memseg_lists_init() will allocate the same amount, no matter how
> much RAM is physically available.
> 
> Is it expected? How to know that setup will not crash due to
> insufficient memory available? How to limit those memory consumption.by
> dpdk?

Hi,

DPDK always reserves a large chunk of virtual address space,
but this costs almost nothing and does not need to be limited.
Then DPDK maps and unmaps actual pages to those addresses as needed.
DPDK does not crash if it runs out of hugepages reserved in the system
but merely returns NULL from its allocation API (rte_malloc).
Real memory consumption can be limited with --socket-limit EAL option.
See also --socket-mem and -m to reserve hugepages at DPDK startup.

Re: Dpdk allocates more memory, than available physically (hugepages)

[Stephen Hemminger]

On Mon, 13 Feb 2023 18:46:14 +0300
Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:

> 2023-02-08 03:43 (UTC+0100), Szymon Szozda:
> > Hey,
> > I'm running dpdk on a machine with 64GB of RAM. It is configured, so 16GiB
> > (16 x 1GiB chunks) of hugepage memory is reserved on boot. I was expecting
> > dpdk to consume only those 16GiB, but it seems it gets more than 30GiB of
> > virtual memory ( I base it on memory VSZ output of top command ). The
> > machine is 1 NUMA, 1 NIC. I did some debugging and I do not see any logic
> > which limits the memory consumption, basically it seems that
> > eal_dynmem_memseg_lists_init() will allocate the same amount, no matter how
> > much RAM is physically available.
> > 
> > Is it expected? How to know that setup will not crash due to
> > insufficient memory available? How to limit those memory consumption.by
> > dpdk?  
> 
> Hi,
> 
> DPDK always reserves a large chunk of virtual address space,
> but this costs almost nothing and does not need to be limited.
> Then DPDK maps and unmaps actual pages to those addresses as needed.
> DPDK does not crash if it runs out of hugepages reserved in the system
> but merely returns NULL from its allocation API (rte_malloc).
> Real memory consumption can be limited with --socket-limit EAL option.
> See also --socket-mem and -m to reserve hugepages at DPDK startup.

Hugepages are mainly used for buffers and shared driver resources.
The normal text and data are not in hugepages (unless you figure
out how to use hugelbfs). Memory consumption limiting is best
done with cgroups. But most applications die if they can't get the
memory they need.  You can see where memory is used with /proc/XXX/maps