From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 96BA846AF6 for ; Fri, 4 Jul 2025 13:45:00 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 342794028B; Fri, 4 Jul 2025 13:45:00 +0200 (CEST) Received: from agw.arknetworks.am (agw.arknetworks.am [79.141.165.80]) by mails.dpdk.org (Postfix) with ESMTP id 632CD40267 for ; Fri, 4 Jul 2025 13:44:59 +0200 (CEST) Received: from debian (unknown [78.109.66.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by agw.arknetworks.am (Postfix) with ESMTPSA id A95BAE046D; Fri, 4 Jul 2025 15:44:58 +0400 (+04) DKIM-Filter: OpenDKIM Filter v2.11.0 agw.arknetworks.am A95BAE046D DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arknetworks.am; s=default; t=1751629499; bh=CtZKN8VxV27WW1DXvuDPyhycE/mjFjOBBtMtnjAjjbg=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=z1tWAnYqnUkHdmYPunfl/cDrp2ysoSjpy9/dbVUsNThE1rwYMhz8Lrvj/4RgSAIZH zYDHNEptMI1HgRbPH8BppMceKwfHEq2iwi/5J0jILT65ejL3+HtuUM2inepZNpekw/ Tp0ARHfNWCJUv+/mThzJCMhDxgUeiJjQuj+uvxOfKv8F55OH3KPeOxcTUdVSKWoksL pVs01TnzvN/8qkRaQdOJ5SEKLymfE1HrzJN8x2idu8jEOVeguUeuu9FUpoUcyBm8kx ZuxIJ+IWxdzh59P45m5Sruej1dtQKRnsyYhJOuaE5z8dzbCg66VD1U5oHK3RunpzVf CzQfhxz4wwjDA== Date: Fri, 4 Jul 2025 15:44:50 +0400 (+04) From: Ivan Malov To: "Lombardo, Ed" cc: users Subject: Re: dpdk Tx falling short In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="8323328-310902160-1751629498=:8030" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --8323328-310902160-1751629498=:8030 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 8BIT Hi Ed, You say there is only one mempool. Why? Have you tried using dedicated mempools, one per each port pair (0,2), (3,4)? Thank you. On Thu, 3 Jul 2025, Lombardo, Ed wrote: > > Hi, > > I have run out of ideas and thought I would reach out to the dpdk community. > >   > > I have a Sapphire Rapids dual CPU server and one E180 (also tried X710), both are 4x10G NICs.  When our application pipeline final stage enqueues mbufs into the tx ring I expect the > rte_ring_dequeue_burst() to pull the mbufs from the tx ring and rte_eth_tx_burst() transmit them at line rate.  What I see is when there is one interface receiving 64-byte UDP in IPv4 > the receive and transmit is at line rate (i.e. packets in one port and out another port of the NIC @14.9 MPPS). > > When I turn on another receive port then both transmit ports of the NIC shows Tx performance drops to 5 MPPS.  The Tx ring is filling faster than Tx thread can dequeue and transmit > mbufs.  > >   > > Packets arrive on ports 1 and 3 in my test setup.  NIC is on NUMA Node 1.  Hugepage memory (6GB, 1GB page size) is on NUMA Node 1.  The mbuf size is 9KB. > >   > > Rx Port 1 -> Tx Port 2 > > Rx Port 3 -> Tx port 4 > >   > > I monitor the mbufs available and they are: > > *** DPDK Mempool Configuration *** > > Number Sockets      :                    1 > > Memory/Socket GB    :                 6 > > Hugepage Size MB    :                 1024 > > Overhead/socket MB  :              512 > > Usable mem/socket MB:          5629 > > mbuf size Bytes     :                     9216 > > nb mbufs per socket :               640455 > > total nb mbufs      :                      640455 > > hugepages/socket GB :               6 > > mempool cache size  :            512 > >   > > *** DPDK EAL args *** > > EAL lcore arg       : -l 36   <<< NUMA Node 1 > > EAL socket-mem arg  : --socket-mem=0,6144 > >   > > The number of rings in this configuration is 16 and all are the same size (16384 * 8), and there is one mempool. > >   > > The Tx rings are created as SP and SC when created. > >   > > There is one Tx thread per NIC port, where its only task is to dequeue mbufs from the tx ring and call rte_eth_tx_burst() to transmit the mbufs.  The dequeue burst size is 512 and tx > burst is equal to or less than 512.  The rte_eth_tx_burst() never returns less than the bust size given. > >   > > Each Tx thread is on a dedicated CPU core and its sibling is unused. > > We use cpushielding to keep noncritical threads from using these CPUs for Tx threads.  HTOP shows the Tx threads are the only threads using the carved-out CPUs. > >   > > In the Tx thread it uses the rte_ring_dequeue_burst() to get a burst of mbufs up to 512.  > > I added debug counters to keep track of how many mbufs are dequeued from the tx ring with rte_ring_dequeue_burst() that equals to the 512 and a counter for less than 512.  The dequeue of > the tx ring is always 512, never less.  > >   > >   > > Note: if I skip the rte_eth_tx_burst() in the Tx threads and just dequeue the mbufs and bulk free the mbufs from the tx ring I do not see the tx ring fill-up, i.e., it is able to free > the mbufs faster than they arrive on the tx ring. > >   > > So, I suspect that the rte_eth_tx_burst() is the bottleneck to investigate, which involves the inner bows of DPDK and Intel NIC architecture. > >   > >   > >   > > Any help to resolve my issue is greatly appreciated. > >   > > Thanks, > > Ed > >   > >   > >   > > > --8323328-310902160-1751629498=:8030--