Re: [PATCH v3] acl: support custom memory allocator

[=?gb18030?B?bWFubnl3YW5nKM3108C35Sk=?= <mannywang@tencent.com>]

Thanks for the follow-up question.

 > I don't understand the build stage issue and why it needs a custom 
allocator.

The fragmentation concern does not come from the amount of address space,
but from how the underlying heap allocator manages **large / mid-sized
temporary buffers** that are repeatedly allocated and freed during ACL 
build.

ACL build allocates many temporary arrays, tables and sorted structures.
Some of them are several MB in size. When these allocations are done via
malloc/calloc, they typically end up in the general heap. Every build
iteration produces a different allocation pattern and size distribution.
Even if the allocations are freed at the end, the internal heap layout is
not restored to a “flat” state. Small holes remain, and future allocation of
large contiguous blocks may fail even if the total free memory is 
sufficient.

This becomes a real operational issue in long-running processes.

 > What exactly gets fragmented? It is the entire process address space 
which is practically unlimited?

It is not the address space that is the limiting factor.
It is the **allocator's internal arena**.

Most allocators (glibc malloc, jemalloc, tcmalloc, etc) retain internal
metadata, bins, and split blocks. Their fragmentation behavior accumulates
over time. The process may still have hundreds of MB of “free memory”, but
not in **contiguous regions** that satisfy the next large request.

 > How does malloc/free overhead compare to overall ACL build time?

The cost of malloc/free calls themselves is not the core problem.
The overhead is small relative to the total build time.

The risk is that allocator fragmentation increases unpredictably over a long
deployment, until a large block allocation fails in the data plane.

Our team has seen this exact behavior in production environments.
Because we cannot fully control the allocator state, we prefer a model
with zero dynamic allocation after init:

* persistent runtime structures → pre-allocated static region
* temporary build data → resettable memory pool

This avoids failure modes caused by allocator history and guarantees stable
latency regardless of system uptime or build frequency.

On 11/26/2025 3:57 PM, Dmitry Kozlyuk wrote:
> On 11/26/25 05:44, mannywang(王永峰) wrote:
>> Thanks for sharing this suggestion.
>>
>> We actually evaluated the heap-based approach before implementing this 
>> patch.
>> It can help in some scenarios, but unfortunately it does not fully 
>> solve our
>> use cases. Specifically:
>>
>> 1. **Heap count / scalability**
>>    Our application maintains at least ~200 rte_acl_ctx instances (due 
>> to the
>>    total rule count and multi-tenant isolation). Allowing a dedicated 
>> heap per
>>    context would exceed the practical limits of the current rte_malloc 
>> heap
>>    model. The number of heaps that can be created is not unlimited, and
>>    maintaining hundreds of separate heaps would introduce considerable
>>    management overhead.
> This is a valid point against heaps, thanks.
>> 2. **Temporary allocations in build stage**
>>    During `rte_acl_build`, a significant portion of memory is 
>> allocated through
>>    `calloc()` for internal temporary structures. These allocations are 
>> freed
>>    right after the build completes. Even if runtime memory could come 
>> from a
>>    custom heap, these temporary allocations would still need an 
>> independent
>>    allocator or callback mechanism to avoid fragmentation and repeated
>>    malloc/free cycles.
> I don't understand the build stage issue and why it needs a custom 
> allocator.
> What exactly gets fragmented?
> It is the entire process address space which is practically unlimited?
> How does is malloc/free overhead compare to the overall ACL build time?
>