Need help with reducing VIRT memory

Need help with reducing VIRT memory

[Lombardo, Ed]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

Hi,
I have exhausted all my options and want to reach out to the dpdk developers for guidance.

My situation is as follows:
We were on DPDK 17.11.6 and upgraded to DPDK22.11.2 to support the Intel E810.  Also upgraded from CentOS7 to Oracle 91.
In DPDK 22.11.2 the VIRT memory has skyrocketed and causes major issues with our application that runs on 16 GB of memory (virtual appliances and low end remote appliances).

I have also tried DPDK 23.11 and 24.03 and I also see across the DPDK versions that the VIRT memory has increase by a factor of 5 over DPDK 17.11

Our application process shows VIRT memory is 7.6 GB with DPDK 17.11.6,
and with DPDK 22.11.2, 23.11.2 and 24.03 the VIRT memory is ~36.6GB

Our application minimum configuration needs to run with 16 GB memory when DPDK is enabled.  Our application uses memory locking and rlimit settings to get optimal performance.

I understand that DPDK memory design has changed since DPDK 18.xx.

We don't use hot-plug feature, encryption, security, telemetry, ipsec, or graph.
Currently we configure 2x1G hugepages and single socket.
I have tried to disable telemetry, cryptodev, etc in meson and only to find they can not be disabled, they are mandatory.

rte_eal_init() has argument "--legacy-mem", which brought down the VIRT memory to 36.6GB, but not enough.
Is there a developer guide to attack this problem in very low risk and timely fashion?

Thank you,
Ed

[-- Attachment #2: Type: text/html, Size: 3867 bytes --]

Re: Need help with reducing VIRT memory

[Dmitry Kozlyuk]

Hi Ed,

I presume it's a revival of this thread:

	http://inbox.dpdk.org/users/CH3PR01MB8470C9675763E14954D6E3B88F292@CH3PR01MB8470.prod.exchangelabs.com/

2024-05-02 19:05 (UTC+0000), Lombardo, Ed:
[...]
> My situation is as follows:
> We were on DPDK 17.11.6 and upgraded to DPDK22.11.2 to support the Intel E810.  Also upgraded from CentOS7 to Oracle 91.
> In DPDK 22.11.2 the VIRT memory has skyrocketed and causes major issues with our application that runs on 16 GB of memory (virtual appliances and low end remote appliances).
> 
> I have also tried DPDK 23.11 and 24.03 and I also see across the DPDK versions that the VIRT memory has increase by a factor of 5 over DPDK 17.11
> 
> Our application process shows VIRT memory is 7.6 GB with DPDK 17.11.6,
> and with DPDK 22.11.2, 23.11.2 and 24.03 the VIRT memory is ~36.6GB
> 
> Our application minimum configuration needs to run with 16 GB memory when DPDK is enabled.

As explained in the linked thread,
high reserved VIRT does not mean high memory demand.
Running DPDK on a 16 GB machine is perfectly viable.
Something in your app or setup conflicts with the high VIRT reservation.

> Our application uses memory locking and rlimit settings to get optimal performance.

Does it use address space limit (prlimit -v)?
If so, this limit may be hit.
With default build options, DPDK reserves at least 32 GB of VIRT (one list)
per memory type (a combination of NUMA node + hugepage size).
This is controlled by defines in <rte_config.h>:

#define RTE_MAX_MEMSEG_LISTS 128
#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

Try reducing those and rebuilding DPDK if the above is your case.
Reducing these values will limit the amount of memory available to DPDK
allocator in your app.

RE: Need help with reducing VIRT memory

[Lombardo, Ed]

Hi Dmitry,
Yes.  I will look into 

The VM gives the following:
[root@vSTREAMO9-234 bin]# prlimit -v
RESOURCE DESCRIPTION              SOFT      	HARD		 UNITS
AS       	address space limit 	unlimited 	unlimited	 bytes

I saw these optional configurations in my google searching, but was hesitant because of unknown impact.  I will try this path next.

It would be valuable to have meson be able to configure DPDK for low memory devices.

Thank you for replying.

Regards,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Thursday, May 2, 2024 4:04 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: dev@dpdk.org
Subject: Re: Need help with reducing VIRT memory

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Ed,

I presume it's a revival of this thread:

	https://urldefense.com/v3/__http://inbox.dpdk.org/users/CH3PR01MB8470C9675763E14954D6E3B88F292@CH3PR01MB8470.prod.exchangelabs.com/__;!!Nzg7nt7_!EBENSvDUv9LcvbAw6vLkHgh11JdjksApffakP2JU30OJaPLCkZPZVzk4U8rRTJsxJVCeRubpFQy7TB2-R6jDbrk4QpU$ 

2024-05-02 19:05 (UTC+0000), Lombardo, Ed:
[...]
> My situation is as follows:
> We were on DPDK 17.11.6 and upgraded to DPDK22.11.2 to support the Intel E810.  Also upgraded from CentOS7 to Oracle 91.
> In DPDK 22.11.2 the VIRT memory has skyrocketed and causes major issues with our application that runs on 16 GB of memory (virtual appliances and low end remote appliances).
> 
> I have also tried DPDK 23.11 and 24.03 and I also see across the DPDK 
> versions that the VIRT memory has increase by a factor of 5 over DPDK 
> 17.11
> 
> Our application process shows VIRT memory is 7.6 GB with DPDK 17.11.6, 
> and with DPDK 22.11.2, 23.11.2 and 24.03 the VIRT memory is ~36.6GB
> 
> Our application minimum configuration needs to run with 16 GB memory when DPDK is enabled.

As explained in the linked thread,
high reserved VIRT does not mean high memory demand.
Running DPDK on a 16 GB machine is perfectly viable.
Something in your app or setup conflicts with the high VIRT reservation.

> Our application uses memory locking and rlimit settings to get optimal performance.

Does it use address space limit (prlimit -v)?
If so, this limit may be hit.
With default build options, DPDK reserves at least 32 GB of VIRT (one list) per memory type (a combination of NUMA node + hugepage size).
This is controlled by defines in <rte_config.h>:

#define RTE_MAX_MEMSEG_LISTS 128
#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

Try reducing those and rebuilding DPDK if the above is your case.
Reducing these values will limit the amount of memory available to DPDK allocator in your app.

RE: Need help with reducing VIRT memory

[Lombardo, Ed]

Hi Dmitry,
I am not clear on the DPDK memory layout and how to tweak these #define values.  

#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

I want to limit how much DPDK grabs for memory, but grabs what it absolutely needs for our application.  I don't want DPDK to plan for any hot-plug or dynamic adjustments of hugepages.  Our configuration is static, we allocate hugepages, discover ports and initialize couple ring buffers and 1 Rx Q and 4 Tx Qs per port.  Max of 4 ports.
My goal is to reduce our application VIRT memory by 80%.  Not clear which config variables to adjust.

What do you suggest, is there a formula?

Regards,
Ed


-----Original Message-----
From: Lombardo, Ed 
Sent: Thursday, May 2, 2024 4:59 PM
To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Cc: dev@dpdk.org
Subject: RE: Need help with reducing VIRT memory

Hi Dmitry,
Yes.  I will look into 

The VM gives the following:
[root@vSTREAMO9-234 bin]# prlimit -v
RESOURCE DESCRIPTION              SOFT      	HARD		 UNITS
AS       	address space limit 	unlimited 	unlimited	 bytes

I saw these optional configurations in my google searching, but was hesitant because of unknown impact.  I will try this path next.

It would be valuable to have meson be able to configure DPDK for low memory devices.

Thank you for replying.

Regards,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Sent: Thursday, May 2, 2024 4:04 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: dev@dpdk.org
Subject: Re: Need help with reducing VIRT memory

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Ed,

I presume it's a revival of this thread:

	https://urldefense.com/v3/__http://inbox.dpdk.org/users/CH3PR01MB8470C9675763E14954D6E3B88F292@CH3PR01MB8470.prod.exchangelabs.com/__;!!Nzg7nt7_!EBENSvDUv9LcvbAw6vLkHgh11JdjksApffakP2JU30OJaPLCkZPZVzk4U8rRTJsxJVCeRubpFQy7TB2-R6jDbrk4QpU$ 

2024-05-02 19:05 (UTC+0000), Lombardo, Ed:
[...]
> My situation is as follows:
> We were on DPDK 17.11.6 and upgraded to DPDK22.11.2 to support the Intel E810.  Also upgraded from CentOS7 to Oracle 91.
> In DPDK 22.11.2 the VIRT memory has skyrocketed and causes major issues with our application that runs on 16 GB of memory (virtual appliances and low end remote appliances).
> 
> I have also tried DPDK 23.11 and 24.03 and I also see across the DPDK 
> versions that the VIRT memory has increase by a factor of 5 over DPDK
> 17.11
> 
> Our application process shows VIRT memory is 7.6 GB with DPDK 17.11.6, 
> and with DPDK 22.11.2, 23.11.2 and 24.03 the VIRT memory is ~36.6GB
> 
> Our application minimum configuration needs to run with 16 GB memory when DPDK is enabled.

As explained in the linked thread,
high reserved VIRT does not mean high memory demand.
Running DPDK on a 16 GB machine is perfectly viable.
Something in your app or setup conflicts with the high VIRT reservation.

> Our application uses memory locking and rlimit settings to get optimal performance.

Does it use address space limit (prlimit -v)?
If so, this limit may be hit.
With default build options, DPDK reserves at least 32 GB of VIRT (one list) per memory type (a combination of NUMA node + hugepage size).
This is controlled by defines in <rte_config.h>:

#define RTE_MAX_MEMSEG_LISTS 128
#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

Try reducing those and rebuilding DPDK if the above is your case.
Reducing these values will limit the amount of memory available to DPDK allocator in your app.

Re: Need help with reducing VIRT memory

[Bruce Richardson]

On Fri, May 03, 2024 at 02:48:12PM +0000, Lombardo, Ed wrote:
> Hi Dmitry,
> I am not clear on the DPDK memory layout and how to tweak these #define values.  
> 
> #define RTE_MAX_MEMSEG_PER_LIST 8192
> #define RTE_MAX_MEM_MB_PER_LIST 32768
> #define RTE_MAX_MEMSEG_PER_TYPE 32768
> #define RTE_MAX_MEM_MB_PER_TYPE 65536
> 
> I want to limit how much DPDK grabs for memory, but grabs what it absolutely needs for our application.

Hi,

This is what DPDK does. What is being shown in the VIRT figures is the
address space reservation DPDK has made, but not what memory it actually uses.
Only sufficient hugepage memory to meet the demands of your app should be
mapped by DPDK, the rest is unused address space that is not taking up any
actual memory.

Adding Anatoly to the thread, as he is probably best able to suggest what
settings above to tweak to reduce this address-space allocation if it is a
problem for you. If it is an issue, I'd be curious to know what the
underlying cause is, in case there is something we need to fix.

/Bruce

>  I don't want DPDK to plan for any hot-plug or dynamic adjustments of hugepages.  Our configuration is static, we allocate hugepages, discover ports and initialize couple ring buffers and 1 Rx Q and 4 Tx Qs per port.  Max of 4 ports.
> My goal is to reduce our application VIRT memory by 80%.  Not clear which config variables to adjust.
> 
> What do you suggest, is there a formula?
> 
> Regards,
> Ed
> 

Re: Need help with reducing VIRT memory

[Bruce Richardson]

On Fri, May 03, 2024 at 04:27:39PM +0100, Bruce Richardson wrote:
> On Fri, May 03, 2024 at 02:48:12PM +0000, Lombardo, Ed wrote:
> > Hi Dmitry,
> > I am not clear on the DPDK memory layout and how to tweak these #define values.  
> > 
> > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> > 
> > I want to limit how much DPDK grabs for memory, but grabs what it absolutely needs for our application.
> 
> Hi,
> 
> This is what DPDK does. What is being shown in the VIRT figures is the
> address space reservation DPDK has made, but not what memory it actually uses.
> Only sufficient hugepage memory to meet the demands of your app should be
> mapped by DPDK, the rest is unused address space that is not taking up any
> actual memory.
> 
By way of illustration, here is the memory output for a testpmd process on
my system. I got this by running "top -b -p <testpmd-PID>"

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2336969 bruce     20   0  256.2g  26432  19712 S  93.8   0.0   5:28.13 dpdk-testpmd

If we look at the memory relevant columns, indeed VIRT shows a huge value -
256G in my case. However, the actual RAM used by testpmd is given in the
"RES" (resident??) column, showing that testpmd actually is only using
26,432kB of memory in this instance, of which 19,712kB is shared memory
(mostly hugepages). In fact, testpmd actually has even more hugepage memory
than that mapped into it, but they must not be actually in use. [Anatoly,
can you confirm that this would be the case when using vfio-pci i.e. no
physical addresses to query?]

Regards,
/Bruce

RE: Need help with reducing VIRT memory

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 3 May 2024 17.52
> 
> On Fri, May 03, 2024 at 04:27:39PM +0100, Bruce Richardson wrote:
> > On Fri, May 03, 2024 at 02:48:12PM +0000, Lombardo, Ed wrote:
> > > Hi Dmitry,
> > > I am not clear on the DPDK memory layout and how to tweak these
> #define values.
> > >
> > > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> > >
> > > I want to limit how much DPDK grabs for memory, but grabs what it
> absolutely needs for our application.
> >
> > Hi,
> >
> > This is what DPDK does. What is being shown in the VIRT figures is the
> > address space reservation DPDK has made, but not what memory it
> actually uses.
> > Only sufficient hugepage memory to meet the demands of your app should
> be
> > mapped by DPDK, the rest is unused address space that is not taking up
> any
> > actual memory.
> >

On DPDK 17.11, DPDK allocates the command line specified amount of hugemem (--socket-mem <megabytes>).
E.g. if a hardware appliance with 8 GB RAM running Linux has been boot-time configured with 6 GB hugemem (e.g. 4 * 1 GB gigantic hugepages and 1024 * 2 MB hugepages), and DPDK allocates all 6 GB hugemem at EAL init, no hugepages will be available for other applications, regardless if DPDK actually uses this memory or not.
This is especially relevant for "embedded" systems with only FLASH and RAM, and no swap.

I don't know if other DPDK versions behave differently. I haven't looked into this in detail.
I certainly don't hope recent DPDK versions assume swap is available, and blindly allocate obscene amounts of memory.

Configuring overcommitted hugepages might help with the problem of DPDK allocating all available hugepages:

Instead of reserving a fixed number of hugepages at boot-time, by setting nr_hugepages, set nr_overcommit_hugepages to allow applications to dynamically allocate hugepages, where the number of hugepages in the system grows and shrinks with the amount of hugemem allocated by applications. Just set it to a sufficiently large number.
Please note that the Linux kernel does not support overcommitting 1 GB gigantic hugepages, only 2 MB hugepages.

> By way of illustration, here is the memory output for a testpmd process
> on
> my system. I got this by running "top -b -p <testpmd-PID>"
> 
>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
> 2336969 bruce     20   0  256.2g  26432  19712 S  93.8   0.0   5:28.13
> dpdk-testpmd
> 
> If we look at the memory relevant columns, indeed VIRT shows a huge
> value -
> 256G in my case. However, the actual RAM used by testpmd is given in the
> "RES" (resident??) column, showing that testpmd actually is only using
> 26,432kB of memory in this instance, of which 19,712kB is shared memory
> (mostly hugepages). In fact, testpmd actually has even more hugepage
> memory
> than that mapped into it, but they must not be actually in use.
> [Anatoly,
> can you confirm that this would be the case when using vfio-pci i.e. no
> physical addresses to query?]
> 
> Regards,
> /Bruce

RE: Need help with reducing VIRT memory

[Lombardo, Ed]

Hi Bruce,
For the heck of it I tried to build DPDK .a and .h files with rte_config.h changes of these parmeters.

1st build- change RTE_MAX_MEMSEG_PER_TYPE:
/*-orig: #define RTE_MAX_MEMSEG_PER_TYPE 32768 */
#define RTE_MAX_MEMSEG_PER_TYPE 16384         <<<< reduced by 1/2

And executed ninja clean and ninja.
Copied the .h and .a files to our application build environment and built our application.
Htop showed our application process VIRT value did not change, still 36.6G.

2nd build- change RTE_MAX_MEMSEG_PER_TYPE and RTE_MAX_MEMSEG_PER_TYPE together:
/*-orig: #define RTE_MAX_MEMSEG_LISTS 128 */
#define RTE_MAX_MEMSEG_LISTS 64                    <<<< reduced by 1/2
#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
/*-orig: #define RTE_MAX_MEMSEG_PER_TYPE 32768 */
#define RTE_MAX_MEMSEG_PER_TYPE 16384    <<<< reduced by 1/2
#define RTE_MAX_MEM_MB_PER_TYPE 65536

Htop showed our application process VIRT value did not change, still 36.6G.  RES value also did not change.

Also tried same above but with --legacy-mem removed from EAL init argument and VIRT jumped up to 99G.  So it is a must to have --legacy-mem.

Our design 5-6 yrs mature, and we had to upgrade DPDK version to support Intel E810 NIC.  Our boot-up (not grub, but early script) first allocates the 2x1G hugepages on Oracle9 and 1024x2M on CentOS7.  We do not change hugepages once we boot up and go into DPDK mode.


I was hoping this path of changing the rte_config.h for MEMSEG would be a viable path.


Regards,
Ed


-----Original Message-----
From: Morten Brørup <mb@smartsharesystems.com> 
Sent: Friday, May 3, 2024 2:58 PM
To: Bruce Richardson <bruce.richardson@intel.com>; Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; dev@dpdk.org; anatoly.burakov@intel.com
Subject: RE: Need help with reducing VIRT memory

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 3 May 2024 17.52
> 
> On Fri, May 03, 2024 at 04:27:39PM +0100, Bruce Richardson wrote:
> > On Fri, May 03, 2024 at 02:48:12PM +0000, Lombardo, Ed wrote:
> > > Hi Dmitry,
> > > I am not clear on the DPDK memory layout and how to tweak these
> #define values.
> > >
> > > #define RTE_MAX_MEMSEG_PER_LIST 8192 #define 
> > > RTE_MAX_MEM_MB_PER_LIST 32768 #define RTE_MAX_MEMSEG_PER_TYPE 
> > > 32768 #define RTE_MAX_MEM_MB_PER_TYPE 65536
> > >
> > > I want to limit how much DPDK grabs for memory, but grabs what it
> absolutely needs for our application.
> >
> > Hi,
> >
> > This is what DPDK does. What is being shown in the VIRT figures is 
> > the address space reservation DPDK has made, but not what memory it
> actually uses.
> > Only sufficient hugepage memory to meet the demands of your app 
> > should
> be
> > mapped by DPDK, the rest is unused address space that is not taking 
> > up
> any
> > actual memory.
> >

On DPDK 17.11, DPDK allocates the command line specified amount of hugemem (--socket-mem <megabytes>).
E.g. if a hardware appliance with 8 GB RAM running Linux has been boot-time configured with 6 GB hugemem (e.g. 4 * 1 GB gigantic hugepages and 1024 * 2 MB hugepages), and DPDK allocates all 6 GB hugemem at EAL init, no hugepages will be available for other applications, regardless if DPDK actually uses this memory or not.
This is especially relevant for "embedded" systems with only FLASH and RAM, and no swap.

I don't know if other DPDK versions behave differently. I haven't looked into this in detail.
I certainly don't hope recent DPDK versions assume swap is available, and blindly allocate obscene amounts of memory.

Configuring overcommitted hugepages might help with the problem of DPDK allocating all available hugepages:

Instead of reserving a fixed number of hugepages at boot-time, by setting nr_hugepages, set nr_overcommit_hugepages to allow applications to dynamically allocate hugepages, where the number of hugepages in the system grows and shrinks with the amount of hugemem allocated by applications. Just set it to a sufficiently large number.
Please note that the Linux kernel does not support overcommitting 1 GB gigantic hugepages, only 2 MB hugepages.

> By way of illustration, here is the memory output for a testpmd 
> process on my system. I got this by running "top -b -p <testpmd-PID>"
> 
>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
> 2336969 bruce     20   0  256.2g  26432  19712 S  93.8   0.0   5:28.13
> dpdk-testpmd
> 
> If we look at the memory relevant columns, indeed VIRT shows a huge 
> value - 256G in my case. However, the actual RAM used by testpmd is 
> given in the "RES" (resident??) column, showing that testpmd actually 
> is only using 26,432kB of memory in this instance, of which 19,712kB 
> is shared memory (mostly hugepages). In fact, testpmd actually has 
> even more hugepage memory than that mapped into it, but they must not 
> be actually in use.
> [Anatoly,
> can you confirm that this would be the case when using vfio-pci i.e. 
> no physical addresses to query?]
> 
> Regards,
> /Bruce

Re: Need help with reducing VIRT memory

[Dmitry Kozlyuk]

2024-05-03 14:48 (UTC+0000), Lombardo, Ed:
> Hi Dmitry,
> I am not clear on the DPDK memory layout and how to tweak these #define values.  

Given your statement:

> Currently we configure 2x1G hugepages and single socket.

I suggest the following:

#define RTE_MAX_MEMSEG_LISTS       2  // see item 5 below
#define RTE_MAX_MEM_MB_PER_LIST 2024  // see item 4 below
#define RTE_MAX_MEMSEG_PER_LIST 1024  // ditto
#define RTE_MAX_MEM_MB_PER_TYPE 2048  // see item 2 below
#define RTE_MAX_MEMSEG_PER_TYPE 1024  // see item 3 below

Explanation:

1. There is one memory type: socket 0 + 1G hugepages.

2. There are 2 GB = 2048 MB memory of this type.

3. On x86, there may be 1G or 2M hugepages.
   If the latter size is ever used, 2 GB need 1024 of 2M hugepages.
   The cost of allowing 1024 memsegs instead of 2 is negligible.

4. Each hugepage is represented by a "memory segment" in DPDK.
   Memory segment list (MSL) is a virtually continuous span
   of memory segments with equal size.
   DPDK cannot allocate a block larger than an entire MSL.
   In order to be able to allocate a block of any size
   (subject to the limitations of the only memory type = 2GB),
   MSL must be as large as the amount of memory of this type.
   Hence, MSL maximums = memory type maximums.
   
5. There may be two memory types (socket 0 + 2M, socket 0 + 1G),
   so you need at least two MSLs available.

In my tests, VIRT = 3142 MB for testpmd with these settings.

> I want to limit how much DPDK grabs for memory, but grabs what it
> absolutely needs for our application.  I don't want DPDK to plan for any
> hot-plug or dynamic adjustments of hugepages.  Our configuration is static,
> we allocate hugepages, discover ports and initialize couple ring buffers
> and 1 Rx Q and 4 Tx Qs per port.  Max of 4 ports.

This is all irrelevant to EAL reservations of address space.

> My goal is to reduce our application VIRT memory by 80%.  Not clear which config variables to adjust.

I hope that the above addresses your need.
However, I suggest considering what me and Bruce have been explaining:
VIRT usage is not RAM usage and high VIRT is not an issue by itself.
It is not recent DPDK that is broken or unsuitable for your hardware.
It is something in your application or setup
that is incompatible with large virtual address space reservations.
If you could provide more details about what is failing for you
(which system call, which what parameters, etc.),
it would be much appreciated, because the issue is rare
and maybe DPDK could account for this case and work out of the box in future.

Re: Need help with reducing VIRT memory

[Dmitry Kozlyuk]

2024-05-03 23:54 (UTC+0300), Dmitry Kozlyuk:
> #define RTE_MAX_MEM_MB_PER_LIST 2024  // see item 4 below

Typo: 2048

RE: Need help with reducing VIRT memory

[Lombardo, Ed]

Hi Dmitry,
Thank you, I will give this a try.

Another angle is how can I calculate or determine when DPDK is initialized how much total memory DPDK will use.  Then I can provide this to our application core and then they can take the DPDK memory max requirement and subtract it from the total memory and this amount of memory can be locked down by the app core.  The application does not want the locked down memory to be used by anything else.  (just an idea I thought of).

Thanks,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Friday, May 3, 2024 4:57 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: dev@dpdk.org
Subject: Re: Need help with reducing VIRT memory

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

2024-05-03 23:54 (UTC+0300), Dmitry Kozlyuk:
> #define RTE_MAX_MEM_MB_PER_LIST 2024  // see item 4 below

Typo: 2048

RE: Need help with reducing VIRT memory

[Morten Brørup]

> From: Lombardo, Ed [mailto:Ed.Lombardo@netscout.com]
> Sent: Friday, 3 May 2024 23.19
> 
> Hi Dmitry,
> Thank you, I will give this a try.

Many arrays in DPDK are statically allocated at the build-time max array size, so tweaking DPDK's build-time configuration parameters will reduce the memory consumption. E.g. we tweak these (using DPDK 17.11):
CONFIG_RTE_MAX_NUMA_NODES
CONFIG_RTE_MAX_LCORE
CONFIG_RTE_MAX_ETHPORTS
CONFIG_RTE_MAX_QUEUES_PER_PORT

> 
> Another angle is how can I calculate or determine when DPDK is
> initialized how much total memory DPDK will use.  Then I can provide
> this to our application core and then they can take the DPDK memory max
> requirement and subtract it from the total memory and this amount of
> memory can be locked down by the app core.  The application does not
> want the locked down memory to be used by anything else.  (just an idea
> I thought of).

I would love having this ability too. However, it depends on too many parameters, both build time and run-time, to be truly predictable. You have to run your application's DPDK initialization and configuration stages, and then measure it. That will help provide an estimate.

You can check the memory consumption after calling rte_eal_init() and at different stages of application initialization, like this:

struct rte_malloc_socket_stats stats, stats_sum;
unsigned int socket;

memset(&stats_sum, 0, sizeof(stats_sum));
/* Iterate through all initialized heaps */
for (socket=0; socket< RTE_MAX_NUMA_NODES; socket++) {
    if (rte_malloc_get_socket_stats(socket, &stats) < 0)
        continue;

    stats_sum.heap_totalsz_bytes += stats.heap_totalsz_bytes;
    stats_sum.heap_freesz_bytes += stats.heap_freesz_bytes;
    stats_sum.heap_allocsz_bytes += stats.heap_allocsz_bytes;
}
printf("DPDK hugemem used after rte_eal_init(): %.01f MB\n",
        (float)stats_sum.heap_allocsz_bytes / (float)(1024 * 1024));

> 
> Thanks,
> Ed
> 
> -----Original Message-----
> From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> Sent: Friday, May 3, 2024 4:57 PM
> 
> 2024-05-03 23:54 (UTC+0300), Dmitry Kozlyuk:
> > #define RTE_MAX_MEM_MB_PER_LIST 2024  // see item 4 below
> 
> Typo: 2048

RE: Need help with reducing VIRT memory

[Lombardo, Ed]

Hi Dmitry,
I tried your settings tonight and the application VIRT memory is now 7.9G and is in the ball park, you are amazing.

#define RTE_MAX_MEMSEG_LISTS 2
#define RTE_MAX_MEMSEG_PER_LIST 1024
#define RTE_MAX_MEM_MB_PER_LIST 2048
#define RTE_MAX_MEMSEG_PER_TYPE 1024
#define RTE_MAX_MEM_MB_PER_TYPE 2048

There is no possibility to change these settings at run-time?  Maybe I asking too much?  Maybe would be something DPDK developers can consider?

Thanks,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Friday, May 3, 2024 4:57 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: dev@dpdk.org
Subject: Re: Need help with reducing VIRT memory

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

2024-05-03 23:54 (UTC+0300), Dmitry Kozlyuk:
> #define RTE_MAX_MEM_MB_PER_LIST 2024  // see item 4 below

Typo: 2048

Re: Need help with reducing VIRT memory

[Dmitry Kozlyuk]

2024-05-05 03:25 (UTC+0000), Lombardo, Ed:
> Hi Dmitry,
> I tried your settings tonight and the application VIRT memory is now 7.9G and is in the ball park, you are amazing.
> 
> #define RTE_MAX_MEMSEG_LISTS 2
> #define RTE_MAX_MEMSEG_PER_LIST 1024
> #define RTE_MAX_MEM_MB_PER_LIST 2048
> #define RTE_MAX_MEMSEG_PER_TYPE 1024
> #define RTE_MAX_MEM_MB_PER_TYPE 2048
> 
> There is no possibility to change these settings at run-time?  Maybe I
> asking too much?  Maybe would be something DPDK developers can consider?

DPDK has a general aim to convert options from compile-time to run-time.
However, these options would be considered in the last place if ever,
because high VIRT consumption has never been considered harmful before.
If you provide a detailed description of your use case
and explain how and why this interferes with your application needs,
the community can consider this rationale.
Technically, these options are easy to be made dynamic.