Re: [DPDK] heap memory fragmentation issue

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

Hi,

2023-04-12 08:44 (UTC+0000), wuchangsheng (C):
> When using rte_malloc and rte_free to request and release memory repeatedly, the usage of large pages gradually increases.

Do you have a repro?

> Checking the relevant source code shows that memory requests and releases
> are started from the head of the freelist chain list of the heap.
> Memory fragmentation seems to result from this, which is considered because
> the memory recently released may be in the cache, and requesting this
> memory at the time of allocation may achieve higher performance?

Could you please elaborate?
DPDK uses "first fit" algorithm to select the free block,
is that what you mean by "memory fragmentation seems to result from this"?

> How does the community consider the heap's memory fragmentation issue? Is
> there a future plan for memory fragmentation optimization?

From my experience with a production app, fragmentation is indeed an issue.

Unfortunately, fragmentation often happens within a single 1G hugepage.
It is impossible to coalesce free fragments of a hugepage
into a contiguous block, like an OS would do with disjoint pages.

Often the application can move its objects in memory, though.
DPDK could provide means for applications to defragment memory this way.
I don't think DPDK should drive the defragmentation process,
because only the application knows when, which, and how much memory
it is reasonable to move around.
DPDK could provide facilities to analyze memory fragmentation degree
(now possible by tracking objects and inspecting memseg, but hard)
and to allocate memory at the desired range (API missing).