DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] Memory allocation limits
@ 2021-09-26 15:52 Mohammad Masumi
  2021-09-29  9:58 ` Thomas Monjalon
  0 siblings, 1 reply; 8+ messages in thread
From: Mohammad Masumi @ 2021-09-26 15:52 UTC (permalink / raw)
  To: users

Hi

I have HP server with 768GB memory 384GB in each Numa but I can't allocate
more than 64GB by rte_malloc by changing some parameters in rte_config.h it
increased to 128GB
How to increase heap size?

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-users] Memory allocation limits
  2021-09-26 15:52 [dpdk-users] Memory allocation limits Mohammad Masumi
@ 2021-09-29  9:58 ` Thomas Monjalon
  2021-09-29 10:14   ` Burakov, Anatoly
  0 siblings, 1 reply; 8+ messages in thread
From: Thomas Monjalon @ 2021-09-29  9:58 UTC (permalink / raw)
  To: Mohammad Masumi
  Cc: users, anatoly.burakov, dmitry.kozliuk, bruce.richardson, stephen

26/09/2021 17:52, Mohammad Masumi:
> Hi
> 
> I have HP server with 768GB memory 384GB in each Numa but I can't allocate
> more than 64GB by rte_malloc by changing some parameters in rte_config.h it
> increased to 128GB
> How to increase heap size?

adding people Cc to help



^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: [dpdk-users] Memory allocation limits
  2021-09-29  9:58 ` Thomas Monjalon
@ 2021-09-29 10:14   ` Burakov, Anatoly
  2021-09-29 11:12     ` Thomas Monjalon
  0 siblings, 1 reply; 8+ messages in thread
From: Burakov, Anatoly @ 2021-09-29 10:14 UTC (permalink / raw)
  To: Thomas Monjalon, Mohammad Masumi
  Cc: users, dmitry.kozliuk, Richardson, Bruce, stephen

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 10:59 AM
> To: Mohammad Masumi <mmasumi@gmail.com>
> Cc: users@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>;
> dmitry.kozliuk@gmail.com; Richardson, Bruce
> <bruce.richardson@intel.com>; stephen@networkplumber.org
> Subject: Re: [dpdk-users] Memory allocation limits
> 
> 26/09/2021 17:52, Mohammad Masumi:
> > Hi
> >
> > I have HP server with 768GB memory 384GB in each Numa but I can't
> > allocate more than 64GB by rte_malloc by changing some parameters in
> > rte_config.h it increased to 128GB How to increase heap size?
> 
> adding people Cc to help
> 

Hi,

This is intentional. In order to increase the amount of contiguous allocation possible to perform in DPDK, you need to adjust the following values in rte_config.h:

#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

I do not recommend arbitrarily changing them as this is untested, but increasing them proportionally (e.g. multiply all of them by 2 or 4) should not break anything.

Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-users] Memory allocation limits
  2021-09-29 10:14   ` Burakov, Anatoly
@ 2021-09-29 11:12     ` Thomas Monjalon
  2021-09-29 12:43       ` Burakov, Anatoly
  0 siblings, 1 reply; 8+ messages in thread
From: Thomas Monjalon @ 2021-09-29 11:12 UTC (permalink / raw)
  To: Mohammad Masumi, Burakov, Anatoly
  Cc: users, dmitry.kozliuk, Richardson, Bruce, stephen

29/09/2021 12:14, Burakov, Anatoly:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 26/09/2021 17:52, Mohammad Masumi:
> > > Hi
> > >
> > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > allocate more than 64GB by rte_malloc by changing some parameters in
> > > rte_config.h it increased to 128GB How to increase heap size?
> > 
> > adding people Cc to help
> > 
> 
> Hi,
> 
> This is intentional. In order to increase the amount of contiguous allocation possible to perform in DPDK, you need to adjust the following values in rte_config.h:
> 
> #define RTE_MAX_MEMSEG_PER_LIST 8192
> #define RTE_MAX_MEM_MB_PER_LIST 32768
> #define RTE_MAX_MEMSEG_PER_TYPE 32768
> #define RTE_MAX_MEM_MB_PER_TYPE 65536
> 
> I do not recommend arbitrarily changing them as this is untested, but increasing them proportionally (e.g. multiply all of them by 2 or 4) should not break anything.

It looks to be something to add in docs, right?



^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: [dpdk-users] Memory allocation limits
  2021-09-29 11:12     ` Thomas Monjalon
@ 2021-09-29 12:43       ` Burakov, Anatoly
  2021-09-29 12:55         ` Thomas Monjalon
  0 siblings, 1 reply; 8+ messages in thread
From: Burakov, Anatoly @ 2021-09-29 12:43 UTC (permalink / raw)
  To: Thomas Monjalon, Mohammad Masumi
  Cc: users, dmitry.kozliuk, Richardson, Bruce, stephen

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 12:12 PM
> To: Mohammad Masumi <mmasumi@gmail.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>
> Cc: users@dpdk.org; dmitry.kozliuk@gmail.com; Richardson, Bruce
> <bruce.richardson@intel.com>; stephen@networkplumber.org
> Subject: Re: [dpdk-users] Memory allocation limits
> 
> 29/09/2021 12:14, Burakov, Anatoly:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 26/09/2021 17:52, Mohammad Masumi:
> > > > Hi
> > > >
> > > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > > allocate more than 64GB by rte_malloc by changing some parameters
> > > > in rte_config.h it increased to 128GB How to increase heap size?
> > >
> > > adding people Cc to help
> > >
> >
> > Hi,
> >
> > This is intentional. In order to increase the amount of contiguous allocation
> possible to perform in DPDK, you need to adjust the following values in
> rte_config.h:
> >
> > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> >
> > I do not recommend arbitrarily changing them as this is untested, but
> increasing them proportionally (e.g. multiply all of them by 2 or 4) should not
> break anything.
> 
> It looks to be something to add in docs, right?
> 

[[AB]] 
Yes, which is why I already have 😃

http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#memory-mapping-discovery-and-memory-reservation

There's a section on "Maximum amount of memory" there.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-users] Memory allocation limits
  2021-09-29 12:43       ` Burakov, Anatoly
@ 2021-09-29 12:55         ` Thomas Monjalon
  0 siblings, 0 replies; 8+ messages in thread
From: Thomas Monjalon @ 2021-09-29 12:55 UTC (permalink / raw)
  To: Burakov, Anatoly
  Cc: Mohammad Masumi, users, dmitry.kozliuk, Richardson, Bruce, stephen

29/09/2021 14:43, Burakov, Anatoly:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 29/09/2021 12:14, Burakov, Anatoly:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 26/09/2021 17:52, Mohammad Masumi:
> > > > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > > > allocate more than 64GB by rte_malloc by changing some parameters
> > > > > in rte_config.h it increased to 128GB How to increase heap size?
> > >
> > > This is intentional. In order to increase the amount of contiguous allocation
> > possible to perform in DPDK, you need to adjust the following values in
> > rte_config.h:
> > >
> > > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> > >
> > > I do not recommend arbitrarily changing them as this is untested, but
> > increasing them proportionally (e.g. multiply all of them by 2 or 4) should not
> > break anything.
> > 
> > It looks to be something to add in docs, right?
> 
> [[AB]] 
> Yes, which is why I already have 😃
> 
> http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#memory-mapping-discovery-and-memory-reservation
> 
> There's a section on "Maximum amount of memory" there.

It says "Normally, these options do not need to be changed."
Would it be meaningful to add how we can increase (by multiplying all of them)?



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-users] Memory allocation limits
@ 2022-04-01 21:42 Dave Sclarsky
  0 siblings, 0 replies; 8+ messages in thread
From: Dave Sclarsky @ 2022-04-01 21:42 UTC (permalink / raw)
  To: users

[-- Attachment #1: Type: text/plain, Size: 223 bytes --]

I apologize, I misread some numbers - I'm actually getting 128GB per numa!
I would still like to increase to get ~192GB per numa if possible!
BTW, I forgot to mention that I'm using DPDK 21.11...
Thanks for any advice!

[-- Attachment #2: Type: text/html, Size: 1796 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-users] Memory allocation limits
@ 2022-04-01 20:43 Dave Sclarsky
  0 siblings, 0 replies; 8+ messages in thread
From: Dave Sclarsky @ 2022-04-01 20:43 UTC (permalink / raw)
  To: users

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

Hi, I'm have a similar problem (but with a dell server with 512GB memory, 256 in each numa).
I can only allocate 128GB, 64 from each NUMA.  If I try 130GB (65 from each numa), I get "eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list".
I tried changing the config variables mentioned in "3.1.4.5 Maximum Amount of Memory" , but no matter what I do to the RTE_MAX_MEMxxx values in rte_config.h the limit seems to be the same.  I tried multiplying all of them by 4,8,16...nothing helped. I also see that RTE_MAX_MEM_MB is "hard coded" to 524288 for ARCH_64 in meson.build, which should be enough so I didn't try to change that one.

Has anyone succeeded in allocating more than 128GB?

[-- Attachment #2: Type: text/html, Size: 2360 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2022-04-01 21:42 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-26 15:52 [dpdk-users] Memory allocation limits Mohammad Masumi
2021-09-29  9:58 ` Thomas Monjalon
2021-09-29 10:14   ` Burakov, Anatoly
2021-09-29 11:12     ` Thomas Monjalon
2021-09-29 12:43       ` Burakov, Anatoly
2021-09-29 12:55         ` Thomas Monjalon
2022-04-01 20:43 Dave Sclarsky
2022-04-01 21:42 Dave Sclarsky

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).