[dpdk-users] Memory allocation limits

[dpdk-users] Memory allocation limits

[Mohammad Masumi]

Hi

I have HP server with 768GB memory 384GB in each Numa but I can't allocate
more than 64GB by rte_malloc by changing some parameters in rte_config.h it
increased to 128GB
How to increase heap size?

Re: [dpdk-users] Memory allocation limits

[Thomas Monjalon]

26/09/2021 17:52, Mohammad Masumi:
> Hi
> 
> I have HP server with 768GB memory 384GB in each Numa but I can't allocate
> more than 64GB by rte_malloc by changing some parameters in rte_config.h it
> increased to 128GB
> How to increase heap size?

adding people Cc to help

RE: [dpdk-users] Memory allocation limits

[Burakov, Anatoly]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 10:59 AM
> To: Mohammad Masumi <mmasumi@gmail.com>
> Cc: users@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>;
> dmitry.kozliuk@gmail.com; Richardson, Bruce
> <bruce.richardson@intel.com>; stephen@networkplumber.org
> Subject: Re: [dpdk-users] Memory allocation limits
> 
> 26/09/2021 17:52, Mohammad Masumi:
> > Hi
> >
> > I have HP server with 768GB memory 384GB in each Numa but I can't
> > allocate more than 64GB by rte_malloc by changing some parameters in
> > rte_config.h it increased to 128GB How to increase heap size?
> 
> adding people Cc to help
> 

Hi,

This is intentional. In order to increase the amount of contiguous allocation possible to perform in DPDK, you need to adjust the following values in rte_config.h:

#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

I do not recommend arbitrarily changing them as this is untested, but increasing them proportionally (e.g. multiply all of them by 2 or 4) should not break anything.

Thanks,
Anatoly

Re: [dpdk-users] Memory allocation limits

[Thomas Monjalon]

29/09/2021 12:14, Burakov, Anatoly:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 26/09/2021 17:52, Mohammad Masumi:
> > > Hi
> > >
> > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > allocate more than 64GB by rte_malloc by changing some parameters in
> > > rte_config.h it increased to 128GB How to increase heap size?
> > 
> > adding people Cc to help
> > 
> 
> Hi,
> 
> This is intentional. In order to increase the amount of contiguous allocation possible to perform in DPDK, you need to adjust the following values in rte_config.h:
> 
> #define RTE_MAX_MEMSEG_PER_LIST 8192
> #define RTE_MAX_MEM_MB_PER_LIST 32768
> #define RTE_MAX_MEMSEG_PER_TYPE 32768
> #define RTE_MAX_MEM_MB_PER_TYPE 65536
> 
> I do not recommend arbitrarily changing them as this is untested, but increasing them proportionally (e.g. multiply all of them by 2 or 4) should not break anything.

It looks to be something to add in docs, right?

RE: [dpdk-users] Memory allocation limits

[Burakov, Anatoly]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 12:12 PM
> To: Mohammad Masumi <mmasumi@gmail.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>
> Cc: users@dpdk.org; dmitry.kozliuk@gmail.com; Richardson, Bruce
> <bruce.richardson@intel.com>; stephen@networkplumber.org
> Subject: Re: [dpdk-users] Memory allocation limits
> 
> 29/09/2021 12:14, Burakov, Anatoly:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 26/09/2021 17:52, Mohammad Masumi:
> > > > Hi
> > > >
> > > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > > allocate more than 64GB by rte_malloc by changing some parameters
> > > > in rte_config.h it increased to 128GB How to increase heap size?
> > >
> > > adding people Cc to help
> > >
> >
> > Hi,
> >
> > This is intentional. In order to increase the amount of contiguous allocation
> possible to perform in DPDK, you need to adjust the following values in
> rte_config.h:
> >
> > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> >
> > I do not recommend arbitrarily changing them as this is untested, but
> increasing them proportionally (e.g. multiply all of them by 2 or 4) should not
> break anything.
> 
> It looks to be something to add in docs, right?
> 

[[AB]] 
Yes, which is why I already have 😃

http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#memory-mapping-discovery-and-memory-reservation

There's a section on "Maximum amount of memory" there.

Re: [dpdk-users] Memory allocation limits

[Thomas Monjalon]

29/09/2021 14:43, Burakov, Anatoly:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 29/09/2021 12:14, Burakov, Anatoly:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 26/09/2021 17:52, Mohammad Masumi:
> > > > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > > > allocate more than 64GB by rte_malloc by changing some parameters
> > > > > in rte_config.h it increased to 128GB How to increase heap size?
> > >
> > > This is intentional. In order to increase the amount of contiguous allocation
> > possible to perform in DPDK, you need to adjust the following values in
> > rte_config.h:
> > >
> > > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> > >
> > > I do not recommend arbitrarily changing them as this is untested, but
> > increasing them proportionally (e.g. multiply all of them by 2 or 4) should not
> > break anything.
> > 
> > It looks to be something to add in docs, right?
> 
> [[AB]] 
> Yes, which is why I already have 😃
> 
> http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#memory-mapping-discovery-and-memory-reservation
> 
> There's a section on "Maximum amount of memory" there.

It says "Normally, these options do not need to be changed."
Would it be meaningful to add how we can increase (by multiplying all of them)?

Re: [dpdk-users] Memory allocation limits

[Dave Sclarsky]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

Hi, I'm have a similar problem (but with a dell server with 512GB memory, 256 in each numa).
I can only allocate 128GB, 64 from each NUMA.  If I try 130GB (65 from each numa), I get "eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list".
I tried changing the config variables mentioned in "3.1.4.5 Maximum Amount of Memory" , but no matter what I do to the RTE_MAX_MEMxxx values in rte_config.h the limit seems to be the same.  I tried multiplying all of them by 4,8,16...nothing helped. I also see that RTE_MAX_MEM_MB is "hard coded" to 524288 for ARCH_64 in meson.build, which should be enough so I didn't try to change that one.

Has anyone succeeded in allocating more than 128GB?

[-- Attachment #2: Type: text/html, Size: 2360 bytes --]

Re: [dpdk-users] Memory allocation limits

[Dave Sclarsky]

[-- Attachment #1: Type: text/plain, Size: 223 bytes --]

I apologize, I misread some numbers - I'm actually getting 128GB per numa!
I would still like to increase to get ~192GB per numa if possible!
BTW, I forgot to mention that I'm using DPDK 21.11...
Thanks for any advice!

[-- Attachment #2: Type: text/html, Size: 1796 bytes --]