Re: [PATCH v3] acl: support custom memory allocator

[Stephen Hemminger <stephen@networkplumber.org>]

On Wed, 26 Nov 2025 16:09:20 +0800
"mannywang(王永峰)" <mannywang@tencent.com> wrote:

> Thanks for the follow-up question.
> 
>  > I don't understand the build stage issue and why it needs a custom   
> allocator.
> 
> The fragmentation concern does not come from the amount of address space,
> but from how the underlying heap allocator manages **large / mid-sized
> temporary buffers** that are repeatedly allocated and freed during ACL 
> build.
> 
> ACL build allocates many temporary arrays, tables and sorted structures.
> Some of them are several MB in size. When these allocations are done via
> malloc/calloc, they typically end up in the general heap. Every build
> iteration produces a different allocation pattern and size distribution.
> Even if the allocations are freed at the end, the internal heap layout is
> not restored to a “flat” state. Small holes remain, and future allocation of
> large contiguous blocks may fail even if the total free memory is 
> sufficient.
> 
> This becomes a real operational issue in long-running processes.
> 
>  > What exactly gets fragmented? It is the entire process address space   
> which is practically unlimited?
> 
> It is not the address space that is the limiting factor.
> It is the **allocator's internal arena**.
> 
> Most allocators (glibc malloc, jemalloc, tcmalloc, etc) retain internal
> metadata, bins, and split blocks. Their fragmentation behavior accumulates
> over time. The process may still have hundreds of MB of “free memory”, but
> not in **contiguous regions** that satisfy the next large request.
> 
>  > How does malloc/free overhead compare to overall ACL build time?  
> 
> The cost of malloc/free calls themselves is not the core problem.
> The overhead is small relative to the total build time.
> 
> The risk is that allocator fragmentation increases unpredictably over a long
> deployment, until a large block allocation fails in the data plane.
> 
> Our team has seen this exact behavior in production environments.
> Because we cannot fully control the allocator state, we prefer a model
> with zero dynamic allocation after init:
> 
> * persistent runtime structures → pre-allocated static region
> * temporary build data → resettable memory pool
> 
> This avoids failure modes caused by allocator history and guarantees stable
> latency regardless of system uptime or build frequency.
> 
> On 11/26/2025 3:57 PM, Dmitry Kozlyuk wrote:
> > On 11/26/25 05:44, mannywang(王永峰) wrote:  
> >> Thanks for sharing this suggestion.
> >>
> >> We actually evaluated the heap-based approach before implementing this 
> >> patch.
> >> It can help in some scenarios, but unfortunately it does not fully 
> >> solve our
> >> use cases. Specifically:
> >>
> >> 1. **Heap count / scalability**
> >>    Our application maintains at least ~200 rte_acl_ctx instances (due 
> >> to the
> >>    total rule count and multi-tenant isolation). Allowing a dedicated 
> >> heap per
> >>    context would exceed the practical limits of the current rte_malloc 
> >> heap
> >>    model. The number of heaps that can be created is not unlimited, and
> >>    maintaining hundreds of separate heaps would introduce considerable
> >>    management overhead.  
> > This is a valid point against heaps, thanks.  
> >> 2. **Temporary allocations in build stage**
> >>    During `rte_acl_build`, a significant portion of memory is 
> >> allocated through
> >>    `calloc()` for internal temporary structures. These allocations are 
> >> freed
> >>    right after the build completes. Even if runtime memory could come 
> >> from a
> >>    custom heap, these temporary allocations would still need an 
> >> independent
> >>    allocator or callback mechanism to avoid fragmentation and repeated
> >>    malloc/free cycles.  
> > I don't understand the build stage issue and why it needs a custom 
> > allocator.
> > What exactly gets fragmented?
> > It is the entire process address space which is practically unlimited?
> > How does is malloc/free overhead compare to the overall ACL build time?
> >   

I have seen similar issues in other networking software, mostly it is because
glibc wants to avoid expensive compaction. See https://sourceware.org/glibc/wiki/MallocInternals

The solution was to call malloc_trim() at the end of control transaction.
If ACL library is doing lots of small allocations, then adding it there
would help.

The effect can also be mitigated by using mallopt to adjust MALLOC_TRIM_THRESHOLD.
There is lots of documentation on the Internet on this.

Another option for some workloads is using an alternative library for malloc.
There are lots of benchmarks on glibc vs tcmalloc vs jemalloc.