From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E3B3746B16 for ; Sun, 6 Jul 2025 18:02:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7A44F4025F; Sun, 6 Jul 2025 18:02:36 +0200 (CEST) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by mails.dpdk.org (Postfix) with ESMTP id 33F1D4021E for ; Sun, 6 Jul 2025 18:02:35 +0200 (CEST) Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-74b54af901bso1352977b3a.2 for ; Sun, 06 Jul 2025 09:02:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1751817754; x=1752422554; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=IU6RwLpxAfpQ6liL0Ze8FDV3BPvrBSjxejneaxMpfXg=; b=kp+9kUS6puN24UAZyve08e2qJ6SQlH/L/ZFebFEM5VI9deZCSmHCiw/95aUk6EK+e6 jKlLWh4jF17Pci8I3Mom7AhkXYfDCIPTz8r49rWkilqIrprREs+ELGFIN2yLbeIuJyb+ Z/ZRzBXJbuVkYUdhXA8D5Ffek6Eo8ThMHYpbsh7sqUmMp9pkUXWShpGBmIziiINg1Gva snCJelYBvirf9Ys1gSG9ORE3i8/JWXJa4LSL8UkL7basXnPzSk/BTf5YPZjh7m30ZXNU AQENsSj3g16MNatWoF0g6pHGpqnpqa0OLZuem1FZQTzP8tmCh+oG3g7Vj2i/PFoyEqWQ TZdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751817754; x=1752422554; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IU6RwLpxAfpQ6liL0Ze8FDV3BPvrBSjxejneaxMpfXg=; b=EaVG6ITYiKmonF7w7vKIvpB8mETs5CzIaDbkaAVVt0JC4cxH4eDcZYMzc2ZexJEsA7 L6tWduW5G+MZLM7emlObUuUlNPesThr88Sn/qcpXnde7eTRpUGFUkmY8bsg/QFrg2LSZ JyESzcgDrh4LjMYAav8Jn3hrO7KPqHk1DEvNtcOPpEG/g137xxQ/o3qyVDT6dTNboLE1 Oz3Jyc6dlLfA239UqciBFyVUUi73tz/zBgwsmlm5dlxVuKUHnbWzQPWcdS2gKepOq8iH dPLnyTvGm4LqK65mGLlcPatDdEoUAa8RtPL/JTvWQGFZehygxcv9zflW1CNQYxrSTgdC 38sA== X-Forwarded-Encrypted: i=1; AJvYcCU2+5L0JascYDadxGBaxD1cNCJcNULUWtW0PKM6qb2eUOUPZwFMZXHDJhuVfg6nlxTiXpYxFw==@dpdk.org X-Gm-Message-State: AOJu0Yy1Dk2Npl9lFltEjrX6802AakNMnUwv7WlHhbu/q1DV8Ws+VU44 D5Gekw9JCqCF6k3+Eib7Qd0SbALW9lxLhFV535cwFJR/CfO5ZkHCczhWefCJwPMYXJo= X-Gm-Gg: ASbGnctzb4ijtMaKHFXpPYjiJnFSOpvjFb77oHl0ii/v99OluFgak1+uR7DWh1um9TT 71Y9IFKjlQhDyHw+ZgVbfZxQQo8hAQitZ4pPvNBeO4GYJFh2Qs3nBKztIk2wPbwy+thgWG1TgW1 xkGu1acpKkz3tQ/qTYAtjFasb4rmG0zmW+tROc1dQ5iZPPIAflPj0YOvF7SFWZHacsJF+KLWzhu OIKJZiInYscY3cHY2eOsr3rD4k9ERH83W9Q1fE5+oW0/Wo/NIJCPoD2z72pw8X/d9hhsvn6oyzU v/8PBDXmBgvH02auOu6c4YFM8BWr5kTZbXQdYtm5A0k7hz7SYU6lqHqb3MGzArT65HVjsurtSQS g/ZvkBRr8Gl3LWjlBwOF/sefrqvOgVCLrSqMJ38s= X-Google-Smtp-Source: AGHT+IH7k3w1CyDMTGRFiS/N70izh2ZppxmNzlFWsp11cw15Kn99xdw5Nd0NldY9VaN9auKktGx1PA== X-Received: by 2002:a05:6a00:3e0f:b0:748:fe1a:3432 with SMTP id d2e1a72fcca58-74cf6f1d2bamr6601689b3a.9.1751817754110; Sun, 06 Jul 2025 09:02:34 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-74ce35cba78sm7213679b3a.57.2025.07.06.09.02.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Jul 2025 09:02:33 -0700 (PDT) Date: Sun, 6 Jul 2025 09:02:32 -0700 From: Stephen Hemminger To: "Lombardo, Ed" Cc: Ivan Malov , users Subject: Re: dpdk Tx falling short Message-ID: <20250706090232.635bd36e@hermes.local> In-Reply-To: References: <20250704074957.5848175a@hermes.local> <20250705120834.78849e56@hermes.local> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Sun, 6 Jul 2025 00:03:16 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > Here are comments to the list of obvious causes of cache misses you mentiond. > > Obvious cache misses. > - passing packets to worker with ring - we use lots of rings to pass mbuf pointers. If I skip the rte_eth_tx_burst() and just free mbuf bulk, the tx ring does not fill up. > - using spinlocks (cost 16ns) - The driver does not use spinlocks, other than what dpdk uses. > - fetching TSC - We don't do this, we let Rx offload timestamp packets. > - syscalls? - No syscalls are done in our driver fast path. > > You mention "passing packets to worker with ring", do you mean using rings to pass mbuf pointers causes cache misses and should be avoided? Rings do cause data to be modified by one core and examined by another so they are a cache miss.