DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] rte_malloc behavior
@ 2021-07-24 14:54 Pavel Vazharov
  0 siblings, 0 replies; only message in thread
From: Pavel Vazharov @ 2021-07-24 14:54 UTC (permalink / raw)
  To: users

Hi,

Short intro to the original cause of my problem.
We have an application where we use the FreeBSD stack running on top of
DPDK. It's based on F-stack <https://github.com/F-Stack/f-stack> but with
lots of modifications at this point. There are situations where we send
lots of TCP data in a short period of time. Lots of 16KB blocks. The
FreeBSD networking stack internally splits these blocks into so-called jumbo
clusters <http://nginx.org/en/docs/freebsd_tuning.html#mbufs> (4K size)
before putting them into the TCP socket buffers. All allocations needed
from the FreeBSD stack are redirected to call rte_malloc. I observed that
during such TCP sends we get peak delays in the sendmsg API call and
tracked these delays down to the rte_malloc calls.

After that I did some tests with the following piece of C++ code trying to
isolate the issue further.
[[gnu::noinline]] static void

test_allocations(int allocations, int size, int align) noexcept

{

    using namespace std::chrono;

    std::vector<void*> mem(allocations);

    const auto beg = high_resolution_clock::now();

    for (int i = 0; i < allocations; ++i) {

        mem[i] = rte_malloc(nullptr, size, align);

        X3ME_ENFORCE(mem[i]);

    }

    const auto end = high_resolution_clock::now();

    fmt::print(

        "Allocations:{} Size:{} Align:{} Time_msecs:{}
Avg_time_usecs:{}\n",
        allocations, size, align,

        duration_cast<milliseconds>(end - beg).count(),

        duration_cast<microseconds>(end - beg).count() / allocations);

    for (void* m : mem) rte_free(m);
}

The results show big delays in the rte_malloc function if we ask for 1K or
4K. These delays are not present if the size is not an exact multiple of 2
like these values.

Allocations:4096 Size:4096 Align:4096 Time_msecs:330 Avg_time_usecs:80
Allocations:16384 Size:4096 Align:4096 Time_msecs:8724 Avg_time_usecs:532
Allocations:32768 Size:4096 Align:4096 Time_msecs:38291 Avg_time_usecs:1168

Allocations:4096 Size:4112 Align:4096 Time_msecs:12 Avg_time_usecs:3
Allocations:16384 Size:4112 Align:4096 Time_msecs:45 Avg_time_usecs:2
Allocations:32768 Size:4112 Align:4096 Time_msecs:83 Avg_time_usecs:2

Allocations:4096 Size:1024 Align:1024 Time_msecs:244 Avg_time_usecs:59
Allocations:16384 Size:1024 Align:1024 Time_msecs:4428 Avg_time_usecs:270
Allocations:32768 Size:1024 Align:1024 Time_msecs:26901 Avg_time_usecs:820

Allocations:4096 Size:1040 Align:1024 Time_msecs:4 Avg_time_usecs:1
Allocations:16384 Size:1040 Align:1024 Time_msecs:16 Avg_time_usecs:1
Allocations:32768 Size:1040 Align:1024 Time_msecs:30 Avg_time_usecs:0

And just for a reference the speed of the allocations using the standard
"aligned_alloc/free" API instead of rte_malloc/rte_free.
Allocations:32768 Size:1024 Align:1024 Time_msecs:66 Avg_time_usecs:2
Allocations:32768 Size:4096 Align:4096 Time_msecs:118 Avg_time_usecs:3

As far as I know some allocators have inefficiencies working with
particular allocation sizes but I haven't expected such a big difference.
Am I missing something from the documentation explaining this behavior?
Should I report it to the devs mailing list?

Regards,
Pavel.

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2021-07-24 14:54 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-24 14:54 [dpdk-users] rte_malloc behavior Pavel Vazharov

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git