From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id C94CC4546F
	for <public@inbox.dpdk.org>; Sun, 23 Jun 2024 01:23:00 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 3AE53402B7;
	Sun, 23 Jun 2024 01:23:00 +0200 (CEST)
Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com
 [209.85.219.181])
 by mails.dpdk.org (Postfix) with ESMTP id 749DD40281
 for <users@dpdk.org>; Sun, 23 Jun 2024 01:22:58 +0200 (CEST)
Received: by mail-yb1-f181.google.com with SMTP id
 3f1490d57ef6-e02bf947545so3047209276.0
 for <users@dpdk.org>; Sat, 22 Jun 2024 16:22:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1719098577; x=1719703377; darn=dpdk.org;
 h=to:subject:message-id:date:from:mime-version:from:to:cc:subject
 :date:message-id:reply-to;
 bh=DDjy0ghoAatmOlCNLLRtvG3Uytl4BpUE+CxfMKjXg3c=;
 b=c+3maWzaGF36CIgriZzrHoFikSSbfGjnJgxSsg2GouUOWajeVgr79fL5GSk5UDQ4dX
 rdPQlVXaMqdhXHRTI2Vk802OGxLVjneVjBxQJLG9CRBSB3pUEFDNk3ON7bRKm4vPT65F
 01+OXuVRa9mdflY6MadAtjRnXZbF8rirGnUYfstXSJ8pw9jyLjHbfIIYYQD8Oh3kyuas
 EaZFkUsVl5RmWUHg5lUB2qR4E8qYMprtWzgB2CwgLyVvj/oUgEn/6ZMBr0+b4Ky78rX/
 /A/4L99jPWoqW4LsbANLWZxGgzmuHgmRlXGCyckHBApZ9bgRSjPJMtUMRH9jYJGYcpHG
 ZL5Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1719098577; x=1719703377;
 h=to:subject:message-id:date:from:mime-version:x-gm-message-state
 :from:to:cc:subject:date:message-id:reply-to;
 bh=DDjy0ghoAatmOlCNLLRtvG3Uytl4BpUE+CxfMKjXg3c=;
 b=ZcdeSUvTCJDw4WXe3yEXbSJg4mBDTF6TxDFGRHuSWdje55lnLL6J+OhQ4AUi4m2x4r
 /EarQDxVmphD6rskaybmdmf0kr6DcU/ScbdzHYUVhGN7+WuEZz1IxjlmXG4mLtifEp8S
 lawPUcu7PzL7xaRVWDbirgCYqZpM1mvrdYXcihqbXsdj2a3X7So4pkUVQjY+qSK8lCRn
 tximmPREHwOX0hAa+n5YtzGHl0G5jGQo+3f0/xQe/zSEZg7qlaH/zUQDfzjKDfqXcQzw
 UYhuKq6+3fF1ar3N9vmqOkcQzmtDrSNuoCSSFlnSQxXu+SDgmcGN1fSVocG9r48aYSu5
 W3Ow==
X-Gm-Message-State: AOJu0YzZK7GXA9W/K4VfHyZhpaInQs1Ce+NDqGAQayTPyFTHwPN5YBT+
 R1h08g4rU0oJCz3caux0FZZmRD8uN7FLX2ImRTQE4V9hWfzuXNxKUXhLPH7BhFLaOw1L93Ivn65
 7IsGZzxxNO4rzL1BHHrnfSnjWX7uePuWK
X-Google-Smtp-Source: AGHT+IGNpf+cF6VQTf7fTnu9gGvuYRUElmNzF+AB6MODC4IqSoUlqEaI3DLBnmP28oflUx64SVmTSf4QouYerw//jII=
X-Received: by 2002:a25:8c8f:0:b0:e02:50f2:784d with SMTP id
 3f1490d57ef6-e0303f5243emr684100276.21.1719098577529; Sat, 22 Jun 2024
 16:22:57 -0700 (PDT)
MIME-Version: 1.0
From: fwefew 4t4tg <7532yahoo@gmail.com>
Date: Sat, 22 Jun 2024 19:22:21 -0400
Message-ID: <CA+Tq66XZr+7AXsgnCfzYkFxKC9Mbrbj19Bu5XB_EjD9_=i_ZzQ@mail.gmail.com>
Subject: Cache misses on allocating mbufs
To: users@dpdk.org
Content-Type: text/plain; charset="UTF-8"
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
 <mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
 <mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org

I happened to be looking into mbuf allocation, and am a little
underwhelmed by the default DPDK performance. There are a lot of
last-level cache misses as measured by Intel's PMU.

I made a single-producer/single-consumer mempool and benchmarked
rte_pktmbuf_alloc and rte_pktmbuf_free using 1Gb huge pages. This ran
on 1 pinned core doing nothing else. A representative loop,

  for (u_int64_t i=0; i<MAX; ++i) {
    data[i] = rte_pktmbuf_alloc(pool);
  }

The salient results are:

- allocating 100,000 mbufs in a tight loop
  * 246.836380 cycles (72.598935 ns) per alloc
  * 797231 LLC cache misses or 90.8% of all LLC references

That's darn near 8 LLC misses per allocation.

Now, we may assume some level of coldness the first time through. If
one frees everything then reallocates 100,000 mbufs on the same pool
the data is slightly better:

- reallocating the same 100,000 mbufs in a tight loop:
  * 221.091480 cycles (65.026906 ns) per alloc
  * 521377 LLC misses 62.8% of all LLC references

It's not so much the allocation of memory; it's the DPDK
initialization of it. Contrast with raw allocations, which skips some
init work rte_pktmbuf_alloc() does. It's an order of 10 better:

- allocate 100,000 mbufs with rte_mbuf_raw_alloc
  * 62.446280 cycles (18.366553 ns) per alloc
  * 8244 LLC misses 14.9% of all LLC references

The bigger problem I have is there seems to be no way around this.
Even if I allocate my own memory with mmap, and add it to the DPDK
heap, I still have to make a mempool on it and still use the same
rte_pktmbuf_alloc calls on it.

Ideally, I want one mempool per TX queue in 1 thread pinned to 1 core
that has NO/NONE/ZERO contention with anything else.

Ideas?

Env:
CPU: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz
Ubuntu 24.04 stock
MLX5 driver
DPDK 24.07-rc1
OFED MLNX_OFED_LINUX-24.04-0.6.6.0-ubuntu24.04-x86_64.tgz

Mempool created like this:

    rte_mempool *mempool = rte_pktmbuf_pool_create_by_ops(
      name,
      102399,
      512,
      8,
      1700,
      0,
      "ring_sp_sc");