RE: Need help with reducing VIRT memory

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 3 May 2024 17.52
> 
> On Fri, May 03, 2024 at 04:27:39PM +0100, Bruce Richardson wrote:
> > On Fri, May 03, 2024 at 02:48:12PM +0000, Lombardo, Ed wrote:
> > > Hi Dmitry,
> > > I am not clear on the DPDK memory layout and how to tweak these
> #define values.
> > >
> > > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> > >
> > > I want to limit how much DPDK grabs for memory, but grabs what it
> absolutely needs for our application.
> >
> > Hi,
> >
> > This is what DPDK does. What is being shown in the VIRT figures is the
> > address space reservation DPDK has made, but not what memory it
> actually uses.
> > Only sufficient hugepage memory to meet the demands of your app should
> be
> > mapped by DPDK, the rest is unused address space that is not taking up
> any
> > actual memory.
> >

On DPDK 17.11, DPDK allocates the command line specified amount of hugemem (--socket-mem <megabytes>).
E.g. if a hardware appliance with 8 GB RAM running Linux has been boot-time configured with 6 GB hugemem (e.g. 4 * 1 GB gigantic hugepages and 1024 * 2 MB hugepages), and DPDK allocates all 6 GB hugemem at EAL init, no hugepages will be available for other applications, regardless if DPDK actually uses this memory or not.
This is especially relevant for "embedded" systems with only FLASH and RAM, and no swap.

I don't know if other DPDK versions behave differently. I haven't looked into this in detail.
I certainly don't hope recent DPDK versions assume swap is available, and blindly allocate obscene amounts of memory.

Configuring overcommitted hugepages might help with the problem of DPDK allocating all available hugepages:

Instead of reserving a fixed number of hugepages at boot-time, by setting nr_hugepages, set nr_overcommit_hugepages to allow applications to dynamically allocate hugepages, where the number of hugepages in the system grows and shrinks with the amount of hugemem allocated by applications. Just set it to a sufficiently large number.
Please note that the Linux kernel does not support overcommitting 1 GB gigantic hugepages, only 2 MB hugepages.

> By way of illustration, here is the memory output for a testpmd process
> on
> my system. I got this by running "top -b -p <testpmd-PID>"
> 
>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
> 2336969 bruce     20   0  256.2g  26432  19712 S  93.8   0.0   5:28.13
> dpdk-testpmd
> 
> If we look at the memory relevant columns, indeed VIRT shows a huge
> value -
> 256G in my case. However, the actual RAM used by testpmd is given in the
> "RES" (resident??) column, showing that testpmd actually is only using
> 26,432kB of memory in this instance, of which 19,712kB is shared memory
> (mostly hugepages). In fact, testpmd actually has even more hugepage
> memory
> than that mapped into it, but they must not be actually in use.
> [Anatoly,
> can you confirm that this would be the case when using vfio-pci i.e. no
> physical addresses to query?]
> 
> Regards,
> /Bruce