From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D47646AF7 for ; Thu, 3 Jul 2025 23:49:25 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 39DD8402BD; Thu, 3 Jul 2025 23:49:25 +0200 (CEST) Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by mails.dpdk.org (Postfix) with ESMTP id 7AFB040299 for ; Thu, 3 Jul 2025 23:49:23 +0200 (CEST) Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-4a77ea7ed49so19852191cf.0 for ; Thu, 03 Jul 2025 14:49:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1751579362; x=1752184162; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=STWaHoUc0qKKPherS78W1rIgvPzCcuJpUIPz7I723Ig=; b=sUemfGAIhw25YNh/FjMx9vOTQcBbuYni+ra/l/187Kbx01qPuUt4hNWhC8/y76huVl 5aAaTetDiWctPqLgB075xRNCEp3WQ067ZUPwtbE/mKtSOZUN9Zl0jYD7NXzrFqxOQBI5 sXHz/ZH3EA1bsumL8zxjToYiUj8dhWOcXV2Ygtl7XYKoY5sgTsrd0N8cTHqxjnR1v5bJ +/IbZ++dGVzfherOLPREhb6UsteS1+wwnkmZBIswewcVv15XQm9QzTSsK1NZ1SWmQeNu MtdI/U4Nc3bDfqy77BboP3cMv0yhe+B1N9yKyBPSwr4bTrMeJjZSbeTtJBc//u/NGIJT SLyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751579362; x=1752184162; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=STWaHoUc0qKKPherS78W1rIgvPzCcuJpUIPz7I723Ig=; b=TNfF3C1X/qSCAXbb+pmTeIeCe6maKX0tIeRd0RXbROXJlbnsb3B6MqLgFVZZr45XHR C/IvnSKUHlpD5RAPoITrj2SYblsRx4e3yx2+CMinkpyd+u6qQlA/AOpOxf9FaYQkK1G3 uMIXlwRa23v7Qd6DUwSCXW/HmvZaVenwakvEj8Fjsxvlcpv+Xk9KDaJDF4Y+suq3sW0y g6qMr1PDlI6ADXJugf83PLqlD/Q0ETtArn+ABlbA04rAU8ylFirBRuEqsNMazlBhVmQJ 0QZocJvLi3qhG4maOu66UkihCK+Od4VN4T0qtDUUd4PTrdXiIpAkrEwlMM4BG4wCLL7z 5yww== X-Gm-Message-State: AOJu0YzFW2oMq5rw1+pGkZpb+TJu5Mpv4XQAIWrKli/og7MqkWUxywIn lbjzMCJRF5zlS+FwON7Pxny1C675elT5IqmY8V0X10++HquBgk8zmVamoLkXdVMaoac= X-Gm-Gg: ASbGncsIqp6Ud1vQwt0rKvbFZ4FOBISClw3uJT2DUCnzfsokN8PzOC7zN8UQN07Wyl0 T4gODZYJrRjcmCCpddbdUpisn27T08/DWm+hRoUCLW6fTRg/eTMFNf9+92kmc6j7jEJEmJOvdJi fDLt8BDeYD+w5olNKQbfqshHy2drvRXS8ukbq6R0+94KEdjttHjYCkewegxuNSIMbXOQMPh5pkK B6AqFFy9CRJJOnmjSI60/Yqehie2lfVmAjvxPaC1woO/hrBJbNpeXDMQTwRdRlDfcMfdo2qrwG1 oQlX9dsSpJ+exybvE6iGtGGdwVMNaP7tIh46oDUfHI3CMan1/PmXthXucLi4hR1hilkJ4XakvZK 5qE3npoCM+wyZzJdc+nqmXS18bhrof4u2XE2Ouco= X-Google-Smtp-Source: AGHT+IFqvTRAoFt+/pQwPlbXeXaMNOZaexdEU+1quRKNstRRc1OFCDNwXk84IPfbyD9H+3/K2orHrg== X-Received: by 2002:ac8:7d92:0:b0:4a8:191f:1893 with SMTP id d75a77b69052e-4a99580210bmr9022391cf.26.1751579362594; Thu, 03 Jul 2025 14:49:22 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4a994a3093asm4361851cf.36.2025.07.03.14.49.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Jul 2025 14:49:22 -0700 (PDT) Date: Thu, 3 Jul 2025 14:49:19 -0700 From: Stephen Hemminger To: "Lombardo, Ed" Cc: users Subject: Re: dpdk Tx falling short Message-ID: <20250703144919.2fea3261@hermes.local> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Thu, 3 Jul 2025 20:14:59 +0000 "Lombardo, Ed" wrote: > Hi, > I have run out of ideas and thought I would reach out to the dpdk community. > > I have a Sapphire Rapids dual CPU server and one E180 (also tried X710), both are 4x10G NICs. When our application pipeline final stage enqueues mbufs into the tx ring I expect the rte_ring_dequeue_burst() to pull the mbufs from the tx ring and rte_eth_tx_burst() transmit them at line rate. What I see is when there is one interface receiving 64-byte UDP in IPv4 the receive and transmit is at line rate (i.e. packets in one port and out another port of the NIC @14.9 MPPS). > When I turn on another receive port then both transmit ports of the NIC shows Tx performance drops to 5 MPPS. The Tx ring is filling faster than Tx thread can dequeue and transmit mbufs. > > Packets arrive on ports 1 and 3 in my test setup. NIC is on NUMA Node 1. Hugepage memory (6GB, 1GB page size) is on NUMA Node 1. The mbuf size is 9KB. > > Rx Port 1 -> Tx Port 2 > Rx Port 3 -> Tx port 4 > > I monitor the mbufs available and they are: > *** DPDK Mempool Configuration *** > Number Sockets : 1 > Memory/Socket GB : 6 > Hugepage Size MB : 1024 > Overhead/socket MB : 512 > Usable mem/socket MB: 5629 > mbuf size Bytes : 9216 > nb mbufs per socket : 640455 > total nb mbufs : 640455 > hugepages/socket GB : 6 > mempool cache size : 512 > > *** DPDK EAL args *** > EAL lcore arg : -l 36 <<< NUMA Node 1 > EAL socket-mem arg : --socket-mem=0,6144 > > The number of rings in this configuration is 16 and all are the same size (16384 * 8), and there is one mempool. > > The Tx rings are created as SP and SC when created. > > There is one Tx thread per NIC port, where its only task is to dequeue mbufs from the tx ring and call rte_eth_tx_burst() to transmit the mbufs. The dequeue burst size is 512 and tx burst is equal to or less than 512. The rte_eth_tx_burst() never returns less than the bust size given. > > Each Tx thread is on a dedicated CPU core and its sibling is unused. > We use cpushielding to keep noncritical threads from using these CPUs for Tx threads. HTOP shows the Tx threads are the only threads using the carved-out CPUs. > > In the Tx thread it uses the rte_ring_dequeue_burst() to get a burst of mbufs up to 512. > I added debug counters to keep track of how many mbufs are dequeued from the tx ring with rte_ring_dequeue_burst() that equals to the 512 and a counter for less than 512. The dequeue of the tx ring is always 512, never less. > > > Note: if I skip the rte_eth_tx_burst() in the Tx threads and just dequeue the mbufs and bulk free the mbufs from the tx ring I do not see the tx ring fill-up, i.e., it is able to free the mbufs faster than they arrive on the tx ring. > > So, I suspect that the rte_eth_tx_burst() is the bottleneck to investigate, which involves the inner bows of DPDK and Intel NIC architecture. > > > > Any help to resolve my issue is greatly appreciated. > > Thanks, > Ed > > > Do profiling, and look at the number of cache misses. I suspect using an additional ring is causing lots of cache misses. Remember going to memory is really slow on modern processors.