Re: Need help with reducing VIRT memory

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

2024-05-03 14:48 (UTC+0000), Lombardo, Ed:
> Hi Dmitry,
> I am not clear on the DPDK memory layout and how to tweak these #define values.  

Given your statement:

> Currently we configure 2x1G hugepages and single socket.

I suggest the following:

#define RTE_MAX_MEMSEG_LISTS       2  // see item 5 below
#define RTE_MAX_MEM_MB_PER_LIST 2024  // see item 4 below
#define RTE_MAX_MEMSEG_PER_LIST 1024  // ditto
#define RTE_MAX_MEM_MB_PER_TYPE 2048  // see item 2 below
#define RTE_MAX_MEMSEG_PER_TYPE 1024  // see item 3 below

Explanation:

1. There is one memory type: socket 0 + 1G hugepages.

2. There are 2 GB = 2048 MB memory of this type.

3. On x86, there may be 1G or 2M hugepages.
   If the latter size is ever used, 2 GB need 1024 of 2M hugepages.
   The cost of allowing 1024 memsegs instead of 2 is negligible.

4. Each hugepage is represented by a "memory segment" in DPDK.
   Memory segment list (MSL) is a virtually continuous span
   of memory segments with equal size.
   DPDK cannot allocate a block larger than an entire MSL.
   In order to be able to allocate a block of any size
   (subject to the limitations of the only memory type = 2GB),
   MSL must be as large as the amount of memory of this type.
   Hence, MSL maximums = memory type maximums.
   
5. There may be two memory types (socket 0 + 2M, socket 0 + 1G),
   so you need at least two MSLs available.

In my tests, VIRT = 3142 MB for testpmd with these settings.

> I want to limit how much DPDK grabs for memory, but grabs what it
> absolutely needs for our application.  I don't want DPDK to plan for any
> hot-plug or dynamic adjustments of hugepages.  Our configuration is static,
> we allocate hugepages, discover ports and initialize couple ring buffers
> and 1 Rx Q and 4 Tx Qs per port.  Max of 4 ports.

This is all irrelevant to EAL reservations of address space.

> My goal is to reduce our application VIRT memory by 80%.  Not clear which config variables to adjust.

I hope that the above addresses your need.
However, I suggest considering what me and Bruce have been explaining:
VIRT usage is not RAM usage and high VIRT is not an issue by itself.
It is not recent DPDK that is broken or unsuitable for your hardware.
It is something in your application or setup
that is incompatible with large virtual address space reservations.
If you could provide more details about what is failing for you
(which system call, which what parameters, etc.),
it would be much appreciated, because the issue is rare
and maybe DPDK could account for this case and work out of the box in future.