[PATCH] config: limit lcore variable maximum size to 4k

[PATCH] config: limit lcore variable maximum size to 4k

[David Marchand]

OVS locks all pages to avoid page faults while processing packets.
1M for each lcore translates to allocating 128M with default build
options on x86.
This resulted in OOM while running unit tests in parallel.

At the moment, the more demanding DPDK user of lcore variable is
rte_service, with a 2112 bytes object.

Limit the lcore variable maximum size to 4k which looks more reasonable.

Fixes: 5bce9bed67ad ("eal: add static per-lcore memory allocation facility")

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 config/rte_config.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/config/rte_config.h b/config/rte_config.h
index 498d509244..5f0627679f 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -41,7 +41,7 @@
 /* EAL defines */
 #define RTE_CACHE_GUARD_LINES 1
 #define RTE_MAX_HEAPS 32
-#define RTE_MAX_LCORE_VAR 1048576
+#define RTE_MAX_LCORE_VAR 4096
 #define RTE_MAX_MEMSEG_LISTS 128
 #define RTE_MAX_MEMSEG_PER_LIST 8192
 #define RTE_MAX_MEM_MB_PER_LIST 32768
-- 
2.47.0

RE: [PATCH] config: limit lcore variable maximum size to 4k

[Morten Brørup]

> From: David Marchand [mailto:david.marchand@redhat.com]
> Sent: Friday, 8 November 2024 19.18
> 
> OVS locks all pages to avoid page faults while processing packets.
> 1M for each lcore translates to allocating 128M with default build
> options on x86.
> This resulted in OOM while running unit tests in parallel.
> 
> At the moment, the more demanding DPDK user of lcore variable is
> rte_service, with a 2112 bytes object.
> 
> Limit the lcore variable maximum size to 4k which looks more
> reasonable.

4 KB is not future proof.

Here's an example where 16 KB is cutting it close:
https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35E9F7D0@smartserver.smartshare.dk/

Depends on how we are going to use it. 4 KB suffices if we only want to use it for "small" structures.

Would 64 KB work as a compromise?

RE: [PATCH] config: limit lcore variable maximum size to 4k

[Morten Brørup]

> From: Morten Brørup [mailto:mb@smartsharesystems.com]
> Sent: Friday, 8 November 2024 19.35
> 
> > From: David Marchand [mailto:david.marchand@redhat.com]
> > Sent: Friday, 8 November 2024 19.18
> >
> > OVS locks all pages to avoid page faults while processing packets.

It sounds smart, so I just took a look at how it does this. I'm not sure, but it seems like it only locks pages that are actually mapped (current and future).

> > 1M for each lcore translates to allocating 128M with default build
> > options on x86.
> > This resulted in OOM while running unit tests in parallel.

Is the root cause the lcore variables library itself, or the unit test using a lot of memory for testing the lcore variables?
We don't want to fix the library if the problem is elsewhere.

> >
> > At the moment, the more demanding DPDK user of lcore variable is
> > rte_service, with a 2112 bytes object.
> >
> > Limit the lcore variable maximum size to 4k which looks more
> > reasonable.
> 
> 4 KB is not future proof.
> 
> Here's an example where 16 KB is cutting it close:
> https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35E9F7D0@smart
> server.smartshare.dk/
> 
> Depends on how we are going to use it. 4 KB suffices if we only want to
> use it for "small" structures.
> 
> Would 64 KB work as a compromise?

Re: [PATCH] config: limit lcore variable maximum size to 4k

[Thomas Monjalon]

08/11/2024 20:53, Morten Brørup:
> > From: Morten Brørup [mailto:mb@smartsharesystems.com]
> > Sent: Friday, 8 November 2024 19.35
> > 
> > > From: David Marchand [mailto:david.marchand@redhat.com]
> > > Sent: Friday, 8 November 2024 19.18
> > >
> > > OVS locks all pages to avoid page faults while processing packets.
> 
> It sounds smart, so I just took a look at how it does this. I'm not sure, but it seems like it only locks pages that are actually mapped (current and future).
> 
> > > 1M for each lcore translates to allocating 128M with default build
> > > options on x86.
> > > This resulted in OOM while running unit tests in parallel.
> 
> Is the root cause the lcore variables library itself, or the unit test using a lot of memory for testing the lcore variables?
> We don't want to fix the library if the problem is elsewhere.

The fix works for our urgent issue and we want to make a release candidate soon.


> > > At the moment, the more demanding DPDK user of lcore variable is
> > > rte_service, with a 2112 bytes object.
> > >
> > > Limit the lcore variable maximum size to 4k which looks more
> > > reasonable.
> > 
> > 4 KB is not future proof.
> > 
> > Here's an example where 16 KB is cutting it close:
> > https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35E9F7D0@smart
> > server.smartshare.dk/
> > 
> > Depends on how we are going to use it. 4 KB suffices if we only want to
> > use it for "small" structures.

This is what is stated in the doc:
"Lcore variables are suitable for small objects"
"The amount of data kept in lcore variables is projected to be small"

> > Would 64 KB work as a compromise?

Let's consider based on the need.
The lcore variables are new and we don't want it to degrade the DPDK footprint,
at least not in this first version.
4 KB is a memory page on common systems,
it looks reasonnable and big enough for a "variable".

Applied, thanks.

Re: [PATCH] config: limit lcore variable maximum size to 4k

[Mattias Rönnblom]

On 2024-11-08 23:13, Thomas Monjalon wrote:
> 08/11/2024 20:53, Morten Brørup:
>>> From: Morten Brørup [mailto:mb@smartsharesystems.com]
>>> Sent: Friday, 8 November 2024 19.35
>>>
>>>> From: David Marchand [mailto:david.marchand@redhat.com]
>>>> Sent: Friday, 8 November 2024 19.18
>>>>
>>>> OVS locks all pages to avoid page faults while processing packets.
>>
>> It sounds smart, so I just took a look at how it does this. I'm not sure, but it seems like it only locks pages that are actually mapped (current and future).
>>
>>>> 1M for each lcore translates to allocating 128M with default build
>>>> options on x86.
>>>> This resulted in OOM while running unit tests in parallel.
>>
>> Is the root cause the lcore variables library itself, or the unit test using a lot of memory for testing the lcore variables?
>> We don't want to fix the library if the problem is elsewhere.
> 
> The fix works for our urgent issue and we want to make a release candidate soon.
> 
> 
>>>> At the moment, the more demanding DPDK user of lcore variable is
>>>> rte_service, with a 2112 bytes object.
>>>>
>>>> Limit the lcore variable maximum size to 4k which looks more
>>>> reasonable.
>>>
>>> 4 KB is not future proof.
>>>
>>> Here's an example where 16 KB is cutting it close:
>>> https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35E9F7D0@smart
>>> server.smartshare.dk/
>>>
>>> Depends on how we are going to use it. 4 KB suffices if we only want to
>>> use it for "small" structures.
> 
> This is what is stated in the doc:
> "Lcore variables are suitable for small objects"
> "The amount of data kept in lcore variables is projected to be small"
> >>> Would 64 KB work as a compromise?
> 
> Let's consider based on the need.
> The lcore variables are new and we don't want it to degrade the DPDK footprint,
> at least not in this first version.
> 4 KB is a memory page on common systems,
> it looks reasonnable and big enough for a "variable".
> 
> Applied, thanks.
> 
> 

Why do you have maintainers if you just ignore them?

Re: [PATCH] config: limit lcore variable maximum size to 4k

[Thomas Monjalon]

08/11/2024 23:34, Mattias Rönnblom:
> On 2024-11-08 23:13, Thomas Monjalon wrote:
> > 08/11/2024 20:53, Morten Brørup:
> >>> From: Morten Brørup [mailto:mb@smartsharesystems.com]
> >>> Sent: Friday, 8 November 2024 19.35
> >>>
> >>>> From: David Marchand [mailto:david.marchand@redhat.com]
> >>>> Sent: Friday, 8 November 2024 19.18
> >>>>
> >>>> OVS locks all pages to avoid page faults while processing packets.
> >>
> >> It sounds smart, so I just took a look at how it does this. I'm not sure, but it seems like it only locks pages that are actually mapped (current and future).
> >>
> >>>> 1M for each lcore translates to allocating 128M with default build
> >>>> options on x86.
> >>>> This resulted in OOM while running unit tests in parallel.
> >>
> >> Is the root cause the lcore variables library itself, or the unit test using a lot of memory for testing the lcore variables?
> >> We don't want to fix the library if the problem is elsewhere.
> > 
> > The fix works for our urgent issue and we want to make a release candidate soon.
> > 
> > 
> >>>> At the moment, the more demanding DPDK user of lcore variable is
> >>>> rte_service, with a 2112 bytes object.
> >>>>
> >>>> Limit the lcore variable maximum size to 4k which looks more
> >>>> reasonable.
> >>>
> >>> 4 KB is not future proof.
> >>>
> >>> Here's an example where 16 KB is cutting it close:
> >>> https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35E9F7D0@smart
> >>> server.smartshare.dk/
> >>>
> >>> Depends on how we are going to use it. 4 KB suffices if we only want to
> >>> use it for "small" structures.
> > 
> > This is what is stated in the doc:
> > "Lcore variables are suitable for small objects"
> > "The amount of data kept in lcore variables is projected to be small"
> > >>> Would 64 KB work as a compromise?
> > 
> > Let's consider based on the need.
> > The lcore variables are new and we don't want it to degrade the DPDK footprint,
> > at least not in this first version.
> > 4 KB is a memory page on common systems,
> > it looks reasonnable and big enough for a "variable".
> > 
> > Applied, thanks.
> 
> Why do you have maintainers if you just ignore them?

I didn't receive your replies when I started to write this.
Please be comprehensive.
We are in a hurry to stabilize this before the release candidate which is already late.

I'll change to 128 KB as you recommend before pushing to the repository.

PS: maybe I should not have merged this feature in 24.11.

RE: [PATCH] config: limit lcore variable maximum size to 4k

[Morten Brørup]

> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Friday, 8 November 2024 23.13

> Let's consider based on the need.
> The lcore variables are new and we don't want it to degrade the DPDK
> footprint,
> at least not in this first version.
> 4 KB is a memory page on common systems,
> it looks reasonnable and big enough for a "variable".
> 
> Applied, thanks.

Changing this breaks the API/ABI.

I consider the 4 KB patch a temporary fix, only to make progress on the release candidate, but not the value to go into the final LTS release.
In other words: I formally NAK this patch for the LTS release, but ACK it for the release candidate.

Let's postpone the discussion until after the release candidate.

Re: [PATCH] config: limit lcore variable maximum size to 4k

[Mattias Rönnblom]

On 2024-11-08 20:53, Morten Brørup wrote:
>> From: Morten Brørup [mailto:mb@smartsharesystems.com]
>> Sent: Friday, 8 November 2024 19.35
>>
>>> From: David Marchand [mailto:david.marchand@redhat.com]
>>> Sent: Friday, 8 November 2024 19.18
>>>
>>> OVS locks all pages to avoid page faults while processing packets.
> 
> It sounds smart, so I just took a look at how it does this. I'm not sure, but it seems like it only locks pages that are actually mapped (current and future).
> 

mlockall(MLOCK_CURRENT) will bring in the whole BSS, it seems. Plus all 
the rest like unused parts of the execution stacks, the data section and 
unused code (text) in the binary and all libraries it has linked to.

It makes a simple (e.g., a unit test) DPDK 24.07 program use ~33x more 
residential memory. After lcore variables, the same MLOCK_CURRENT-ed 
program is ~30% larger than before. So, a relatively modest increase.

The numbers are less drastic, obviously, for many real-world programs, 
which have large packet pools and other memory hogs.

>>> 1M for each lcore translates to allocating 128M with default build
>>> options on x86.
>>> This resulted in OOM while running unit tests in parallel.
> 
> Is the root cause the lcore variables library itself, or the unit test using a lot of memory for testing the lcore variables?
> We don't want to fix the library if the problem is elsewhere.
> 
>>>
>>> At the moment, the more demanding DPDK user of lcore variable is
>>> rte_service, with a 2112 bytes object.
>>>
>>> Limit the lcore variable maximum size to 4k which looks more
>>> reasonable.
>>
>> 4 KB is not future proof.
>>
>> Here's an example where 16 KB is cutting it close:
>> https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35E9F7D0@smart
>> server.smartshare.dk/
>>
>> Depends on how we are going to use it. 4 KB suffices if we only want to
>> use it for "small" structures.
>>
>> Would 64 KB work as a compromise?
> 

RE: [PATCH] config: limit lcore variable maximum size to 4k

[Morten Brørup]

> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> Sent: Friday, 8 November 2024 23.23
> 
> On 2024-11-08 20:53, Morten Brørup wrote:
> >> From: Morten Brørup [mailto:mb@smartsharesystems.com]
> >> Sent: Friday, 8 November 2024 19.35
> >>
> >>> From: David Marchand [mailto:david.marchand@redhat.com]
> >>> Sent: Friday, 8 November 2024 19.18
> >>>
> >>> OVS locks all pages to avoid page faults while processing packets.
> >
> > It sounds smart, so I just took a look at how it does this. I'm not
> sure, but it seems like it only locks pages that are actually mapped
> (current and future).
> >
> 
> mlockall(MLOCK_CURRENT) will bring in the whole BSS, it seems. Plus all
> the rest like unused parts of the execution stacks, the data section
> and
> unused code (text) in the binary and all libraries it has linked to.
> 
> It makes a simple (e.g., a unit test) DPDK 24.07 program use ~33x more
> residential memory. After lcore variables, the same MLOCK_CURRENT-ed
> program is ~30% larger than before. So, a relatively modest increase.

Thank you for testing this, Mattias.
What are the absolute numbers, i.e. in KB, to get an idea of the numbers I should be looking for?

I wonder why the footprint grows at all... Intuitively the same variables should consume approximately the same amount of RAM, regardless how they are allocated.
Speculating...
The lcore_states were allocated through rte_calloc() and thus used some space in the already allocated hugepages, so they didn't add more pages to the footprint. But they do when allocated and initialized as lcore variables.
The first lcore variable allocated/initialized uses RTE_MAX_LCORE (128) pages of 4 KB each = 512 KB total. It seems unlikely that adding 512 KB increases the footprint by 30 %.

> 
> The numbers are less drastic, obviously, for many real-world programs,
> which have large packet pools and other memory hogs.

Agree.
However, it would be good to understand why switching to lcore variables has this effect on the footprint when using mlockall() like OVS.

Re: [PATCH] config: limit lcore variable maximum size to 4k

[Mattias Rönnblom]

On 2024-11-08 19:17, David Marchand wrote:
> OVS locks all pages to avoid page faults while processing packets.
> 1M for each lcore translates to allocating 128M with default build
> options on x86.
> This resulted in OOM while running unit tests in parallel.
> 

Could you give some more context. If you get OOM if you add 128 MB of 
RSS, how much memory have you budgeted for the app in total? What are 
the packet mempool sizes, for example?

If you are running tests in parallel it likely means these aren't 
characteristics tests, and thus you can disable the mlockall() call and 
fit many more copies under the OOM ceiling.

Another alternative might be to unlock the lcore variables area, or make 
the whole BSS MLOCK_ONFAULT.

> At the moment, the more demanding DPDK user of lcore variable is
> rte_service, with a 2112 bytes object.
> 

<rte_lcore_var.h> is a public API, so the largest object may well be 
much larger than that.

That said, maybe 1 MB is too large.

> Limit the lcore variable maximum size to 4k which looks more reasonable.
> 

NAK

128 kB?

> Fixes: 5bce9bed67ad ("eal: add static per-lcore memory allocation facility")
> 

I've mentioned this property of lcore variables a couple of times on the 
list, so it should come as no surprise to anyone.

> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>   config/rte_config.h | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/config/rte_config.h b/config/rte_config.h
> index 498d509244..5f0627679f 100644
> --- a/config/rte_config.h
> +++ b/config/rte_config.h
> @@ -41,7 +41,7 @@
>   /* EAL defines */
>   #define RTE_CACHE_GUARD_LINES 1
>   #define RTE_MAX_HEAPS 32
> -#define RTE_MAX_LCORE_VAR 1048576
> +#define RTE_MAX_LCORE_VAR 4096
>   #define RTE_MAX_MEMSEG_LISTS 128
>   #define RTE_MAX_MEMSEG_PER_LIST 8192
>   #define RTE_MAX_MEM_MB_PER_LIST 32768