RE: Need help with reducing VIRT memory

["Lombardo, Ed" <Ed.Lombardo@netscout.com>]

Hi Dmitry,
I am not clear on the DPDK memory layout and how to tweak these #define values.  

#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

I want to limit how much DPDK grabs for memory, but grabs what it absolutely needs for our application.  I don't want DPDK to plan for any hot-plug or dynamic adjustments of hugepages.  Our configuration is static, we allocate hugepages, discover ports and initialize couple ring buffers and 1 Rx Q and 4 Tx Qs per port.  Max of 4 ports.
My goal is to reduce our application VIRT memory by 80%.  Not clear which config variables to adjust.

What do you suggest, is there a formula?

Regards,
Ed


-----Original Message-----
From: Lombardo, Ed 
Sent: Thursday, May 2, 2024 4:59 PM
To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Cc: dev@dpdk.org
Subject: RE: Need help with reducing VIRT memory

Hi Dmitry,
Yes.  I will look into 

The VM gives the following:
[root@vSTREAMO9-234 bin]# prlimit -v
RESOURCE DESCRIPTION              SOFT      	HARD		 UNITS
AS       	address space limit 	unlimited 	unlimited	 bytes

I saw these optional configurations in my google searching, but was hesitant because of unknown impact.  I will try this path next.

It would be valuable to have meson be able to configure DPDK for low memory devices.

Thank you for replying.

Regards,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Sent: Thursday, May 2, 2024 4:04 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: dev@dpdk.org
Subject: Re: Need help with reducing VIRT memory

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Ed,

I presume it's a revival of this thread:

	https://urldefense.com/v3/__http://inbox.dpdk.org/users/CH3PR01MB8470C9675763E14954D6E3B88F292@CH3PR01MB8470.prod.exchangelabs.com/__;!!Nzg7nt7_!EBENSvDUv9LcvbAw6vLkHgh11JdjksApffakP2JU30OJaPLCkZPZVzk4U8rRTJsxJVCeRubpFQy7TB2-R6jDbrk4QpU$ 

2024-05-02 19:05 (UTC+0000), Lombardo, Ed:
[...]
> My situation is as follows:
> We were on DPDK 17.11.6 and upgraded to DPDK22.11.2 to support the Intel E810.  Also upgraded from CentOS7 to Oracle 91.
> In DPDK 22.11.2 the VIRT memory has skyrocketed and causes major issues with our application that runs on 16 GB of memory (virtual appliances and low end remote appliances).
> 
> I have also tried DPDK 23.11 and 24.03 and I also see across the DPDK 
> versions that the VIRT memory has increase by a factor of 5 over DPDK
> 17.11
> 
> Our application process shows VIRT memory is 7.6 GB with DPDK 17.11.6, 
> and with DPDK 22.11.2, 23.11.2 and 24.03 the VIRT memory is ~36.6GB
> 
> Our application minimum configuration needs to run with 16 GB memory when DPDK is enabled.

As explained in the linked thread,
high reserved VIRT does not mean high memory demand.
Running DPDK on a 16 GB machine is perfectly viable.
Something in your app or setup conflicts with the high VIRT reservation.

> Our application uses memory locking and rlimit settings to get optimal performance.

Does it use address space limit (prlimit -v)?
If so, this limit may be hit.
With default build options, DPDK reserves at least 32 GB of VIRT (one list) per memory type (a combination of NUMA node + hugepage size).
This is controlled by defines in <rte_config.h>:

#define RTE_MAX_MEMSEG_LISTS 128
#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

Try reducing those and rebuilding DPDK if the above is your case.
Reducing these values will limit the amount of memory available to DPDK allocator in your app.