From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6DB3B46AF1 for ; Thu, 3 Jul 2025 22:15:08 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 06D9840267; Thu, 3 Jul 2025 22:15:08 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0a-00196b01.pphosted.com [67.231.149.170]) by mails.dpdk.org (Postfix) with ESMTP id AB30A40264 for ; Thu, 3 Jul 2025 22:15:06 +0200 (CEST) Received: from pps.filterd (m0096263.ppops.net [127.0.0.1]) by mx0a-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 563KB6GI014551 for ; Thu, 3 Jul 2025 16:15:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= content-type:date:from:message-id:mime-version:subject:to; s= netscout.com.09.24.2020; bh=yJ8CRatNgwFGqkwz4N+1bWv6jbhvZljhoGxe NQC3Rfs=; b=UbpPCRTNwLjalvhgyOP5gMntdNdbBiF+DMGNLzQHrTg5Wk7EUs4L PXDFVcSOUzgj+1oFe4AVVaiRlb4vuyGAJHzy7ykA52kuZnXJETWiHGT3KNYFhPR8 ym60R5haXouEsaGEUnlc15sWlhkWTAN9qfIRjKP/zKocCjzOUTjpW8zFcPQ2mbgH QePki5ujbh17CEqKN75gDMR4XoC+Z2/LSedDY+XxJrgKbXXH3J7RarkCmyRTMiRo mOcygLM2Bsgv+AcZ05ECRWtSYuFBX7rCuPfNJOKMWFmACYT9mKcpYa+/Rrnix7vV 7r9KwDU8kCaTPBFyW7ywg008j7zEz/HEUw== Received: from nam04-bn8-obe.outbound.protection.outlook.com (mail-bn8nam04lp2046.outbound.protection.outlook.com [104.47.74.46]) by mx0a-00196b01.pphosted.com (PPS) with ESMTPS id 47ntcdgh5g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 03 Jul 2025 16:15:05 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MuZJsVlvbhF1W07ysXIhU4WWSlP/Z8AXIRZnheXdPUv7t/ETI6DyzF6KXixxYDl3+7Xk7U4rFQFO4m5NwA1aImUOIciUmZDk19EWPgLgdxdr4Lms6l6mgKXiaq6wmVP9B+iFwgQzz3XJV5nPS3ooZLWS9XJJ4wy4x8q7Pbe5vmR0RHbWSR967bQzH7e3+fVVRrQxSnyHYpSYoXUD/7S1k2F9toIiCkuIoOSAMjZb/pcpt2ZW3txsH0YXVoYuCqrsWODiOXTg0Ip7C8NAc5SKqEtQNVE+ZuflobyuCzn2kyWqpqKUDI91gDdbY86oG4FbRtes3f4Fwf2cjht9jF0EJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yJ8CRatNgwFGqkwz4N+1bWv6jbhvZljhoGxeNQC3Rfs=; b=P7lLG4BJrbsPfy6i2JB+3QoKFiArp4CfK1Dp9YsPQn9PNZ5ioARe26rIg8Gsi6xBkLJEs2kImQI/eOgcUFvzpWGix+fhSTsZvT7GWIBrs5B9eq0xfrySg+P+YS3ithpeJI73+fC70c9IUBrBoggylz1VnV5Zftlqtlzw+vxxVIB17dT+kvC9rtY6D/S4Muq7l5bYUhlubop9sxx6tWDibR4IVtz9HnMT3UTKmNlypez8PNIsgp1o8hphE0A8r4k6BPizmSIcuHH6MrXRiRKwcM9h2F3qCe18k20cOXGktDOiT01bsY3ckWjnoCcaT3b2E+LI1JVS+lIjQ60xo7/oMQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by PH7PR01MB8465.prod.exchangelabs.com (2603:10b6:510:2f4::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.20; Thu, 3 Jul 2025 20:15:00 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%3]) with mapi id 15.20.8901.018; Thu, 3 Jul 2025 20:14:59 +0000 From: "Lombardo, Ed" To: users Subject: dpdk Tx falling short Thread-Topic: dpdk Tx falling short Thread-Index: AdvsS4XiFhfr9tUWTZG/9hrfPmeo0Q== Date: Thu, 3 Jul 2025 20:14:59 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|PH7PR01MB8465:EE_ x-ms-office365-filtering-correlation-id: e1bce491-decf-40cb-cba1-08ddba6e4971 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|376014|1800799024|366016|38070700018|8096899003; x-microsoft-antispam-message-info: =?us-ascii?Q?cuGqaLwjq1+fKqrUbndOTDPgC8d+jPy0BGqPBHL+ojhBHduoIXswKpKPIcNp?= =?us-ascii?Q?CbwBP871CIrdUkuLKtsQW12oF3Ql+8YCGxso86mbPv6jMIZgSq4XWBi9FDDx?= =?us-ascii?Q?b4W6pSnN1u0NmWHv0ZpKacdnS+WIhS1MMyNbhKkjrmlYJe2VYQ9W9PODkFI4?= =?us-ascii?Q?qu+LscNrjk4zM+1kFWuFMSK4Ys/YMuS+hxckEpd9IwOVGt9WsewHdh3ujYqj?= =?us-ascii?Q?lA36nyYfy/Od6g+p1P5zZjv1dhZtwgXeyGfyJjdc69m8zE8KmhqgtsMwlgzb?= =?us-ascii?Q?7Wt1s4wf0MjrkVKhlnLqpS99abGuhKiYwq8xbJNuFBN3RjQ0IDasTbGsfTcf?= =?us-ascii?Q?qmMpjkV61QQhqrXZ3OyKV7qgO23zR2UyvQcsBDpFiG83WgaGCyediMTjMBNB?= =?us-ascii?Q?5jHJ6esaVDS5GXsYe5J4z2YtYKGaWqELMUaCLJ0Aa2ETXX+VB39OclLzkdYt?= =?us-ascii?Q?aM3qTDqb1hqdtbpDsst2kFOnBUMmhuIBPBb5EVNbkB98bm6nUuELwPbg5X+W?= =?us-ascii?Q?gw1PNxD+bD9IrLSf+dEKeabwXNLfX6m0ovERW1OxxVzEkVA6GFFrgYVLy2YQ?= =?us-ascii?Q?taqwmluwCqCCEpLvL8HqhyOy7QSDojvT4gCKD90W0sjW7QsCnnbP7eAbwhRW?= =?us-ascii?Q?kYvbjI2K6lYIhuQAEqJueggCGiDOcyhn87SOpuOItr53frE+ojauFk44fgN9?= =?us-ascii?Q?a8MTGZZLFczWc79JeBUeFelIKvFLpemehJlRV0IcF/ZHtMIC9weCXZTakyfT?= =?us-ascii?Q?TH9MyW7RVlBUJB8VrPaTSUiBFbCMVHbmnijD0afViMxjsAehKu+Zb2vbnmhg?= =?us-ascii?Q?o8b2YnWK/TOhOcYSQxGkACm8at0VD+/UdWaOJSz2Jh3Gm1akGXYLvqp+fwFQ?= =?us-ascii?Q?hOy6qt+cFCTFNnr3aYJ5htKz9kN8uFDrDynUYoyqRmd70dtfzXujzf9+foN6?= =?us-ascii?Q?FK+fv9/8IAd49Td8psdY6joNeqN+G9KzuUR+v60xX8PGNwXzpRgjCe/H6ZPr?= =?us-ascii?Q?Duj7LJ7Sy1uEUaXRqt8Qtc4H5yGHlIt5OcQlrNYnYy2CSKkB4ZxwxGn+5Q+R?= =?us-ascii?Q?AGP4sv6wfLw7tmhnlwKC4Eq/2as9K2fQ9fm3qmD3sDoHIKaGc9aAaWQ7aofZ?= =?us-ascii?Q?8zycxfB8qlPlFFi8mZrJv551AdqmxGEVPkrAhg6DOwynSjn4gGsYOoXT3M0+?= =?us-ascii?Q?guxp4RDG5PszsEuyUngsSzJ4WcGOtrpUqwqylcRMXfMRQRQENmHzPOBc9u/p?= =?us-ascii?Q?GLAMH6qGW43019sBnIu36HT+gTsb9SvqUu0Qvh3VQj3vzkXP9lTeYqdQnPVh?= =?us-ascii?Q?qKW5lFaPX7yusOg8rNP7UkpXbqWb54rGjCpKpGY1I2OLUGmnI5tH1OG2fOhz?= =?us-ascii?Q?LLlsHvycRH1Es6kFVckMUPbYYRsJDFx3qoJzEJV/Ao+cL7iXchLjKKaMs1Fv?= =?us-ascii?Q?2uZKv//O5IxHmU5f81HbyNTsOcc4lDcoJiBdh8lLOL/WCWCHrcJHEg=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(1800799024)(366016)(38070700018)(8096899003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?v1zgqiaOGkoLKTNcnqoHr22XB3z4lYogNPIGbSrqVSQmKcuYXlHE8rE4/58B?= =?us-ascii?Q?aa3KKY6q5dJpWy5257cWo+dUW1tC0bGj9mlQ83C2rk0fqskB2GKDoiLK5bMv?= =?us-ascii?Q?PPQy1nLnWyiEH7EVLasuSqMkg/2CK9e4eb9o4jvp9nzRelusKxZipwrI4JxI?= =?us-ascii?Q?sytsVYtPvM9x7OBdcKq5LDMjwWPwRiN1FjcdBlxQYR0JtyyqyJrxKnyYO23C?= =?us-ascii?Q?aVRkTLPHAiRPrQ3W8MAY0g8RSfUEddcjuYOiXy0WUmvAawXYe/FJKZirj4hr?= =?us-ascii?Q?gFDwLfcfL4711bbAahhnWyiUUFDTLq23vLyK0rZi/xLsFIrfmkr17Tiyz8vg?= =?us-ascii?Q?HiCItTeQvidMEohtu3hTvoKtauzO12kW3TVCkUhsaAa7mF5tN1WTjTjf16K8?= =?us-ascii?Q?UtW+rk+BsKfBC79Ch/FUMR3h4XH3dRcW5T9RXPRbbQS2LCmBfxIrdkI4oIs2?= =?us-ascii?Q?AhtgQKOIiTbHbTE2saKOesW/MvM7fzIAEkTohc/S+EtHu3XKSmJvjV5LExCf?= =?us-ascii?Q?aHZJhcMALmNlozvacTVXCJyrLwl0xjGgUQDTtx9rdLcjWKLJ8u+Zu7Y7gg0+?= =?us-ascii?Q?MEfnvcr1EfeQcoMm/KHAaaqrWCi2QP1SsMke/nZ8cb1ry5ySF1Tj9hGZi5th?= =?us-ascii?Q?CMRREORsbFQi++ZBPLa6z7rpwo6QYezDWZftHjybmWARab9Ri47wSy0sVdn1?= =?us-ascii?Q?MCWFBTp6eMn7CPcxQrAdZMQkhRqz6Lw4CEtnGSIpgvJuVOs+te8a8tfTAbFY?= =?us-ascii?Q?0V+2N4U0Chba3HYa9O5a1CSilC1UfywkvU87uh7Zcvu5ss5vbQpw21vn6xFQ?= =?us-ascii?Q?f6r/zIqPL06VNwXo3it+fGY5xcxUJbEEgWTuNSvT/H65sD0d3XkLrPg6jiYS?= =?us-ascii?Q?T/aQtF+MvwV9wsWr7/T3y649NwMCq+pX82sfr7EuSD00z8AOrHLkG4cCKnma?= =?us-ascii?Q?F8JlcWleSQ2hRYMYEGsNdpW/vA1g+cs0VjeQmDMjH39QrsU10aL0xSDfsI07?= =?us-ascii?Q?SvLdDq9TeIKWuYXzp3pX0ali4MYBYqyB9sMgUOvpxa2uZsHVkCRFurEP5Mdd?= =?us-ascii?Q?o45GV+cFkjaFWNoGhl2pah2nAvwe95jo/lTTRS6nO/KPtldkymaVee2Avr/J?= =?us-ascii?Q?2HmKTU9boV0U3hpsYMEWkumzRKY5osZEIeTT+rWBLRUQQqMRbYa48RQAuEdM?= =?us-ascii?Q?aeDWxk/qgNQGqKO4o/9P8/iyqsCbk3n+j2Mi01KZdzDfrz/UifqwQ+rbIrAa?= =?us-ascii?Q?zrp5hUzGymu75bPy+ZsPlzQwoO8H+GKTArVrsSF5jcuq5ifBNwVte/uv6sCq?= =?us-ascii?Q?iYpxRO66G5KmMZAwirCJlLcWIbkQDY7Rj9vBFTUH3pN4G5WArpKrsTCd4oav?= =?us-ascii?Q?rDEViJVDhR1X7eNDoo/aNclPJAZ9G2xWbiqgIH+9c57qTEwwWTE+qx4KhzYf?= =?us-ascii?Q?z2MpT9fqKgjrnj50L8lo8azokIQ6dlMd5p7xuEEN1BzsxMfGMrJCtj0dj+kv?= =?us-ascii?Q?+FRKg2Riz5aa64SQ0Omzqv5ZUfFF1Zr88lD/opQTX34YTTPEANwZqUAh+qgJ?= =?us-ascii?Q?OByZV4Pudg8+hYaaThmf4bH05vPTFkSeqaOSW5PI?= Content-Type: multipart/alternative; boundary="_000_CH3PR01MB8470E2030EECEB410B356F878F43ACH3PR01MB8470prod_" MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 3pXKcsiVj407weLTMMTAy4B1fKYtwykoAggDgG8iVfPGR9rdsMXcbIAgOwwnqQiHZwJyYkY08TRudJIbjc0d4rSpmgREGNNtRY4X4CMsQBnd8gOFRgHLwHfqA21yLYHHcKPsf9vU8PiYCAwVvwr4MVO4uGAS1gRcSuecKTUihaKOkjMqfCbGVamKUtY9i4cYgMNvj90eLHOnAeOyn/KHegrp3Eznw7Oz4BPa19qmXuXHabC0dTsRU7/cCb8wXA33sq3V0Qk/tZhOCkW9PTdhEXnPeXju8GWJoGiNpSu+CMlZ40CzCLyKh5b0oaPKlStj3i7OXI/lm/dp8ONWogHfl2tgtsBRUVHcsPviw5Zg9Prv76/tLwQhIDCeIy6oo8Ves5wKkOwmE58/0OaYyaQJY/nfw2WnfEO2bg7R2zo+0+wvFRf0uzufb7wf7EwfPmz25BR6Ko42fp0I3ik56yUpW5QVjIUaiUPu1l6bRQN1I48wWaw4Oga1EXcdYPzesCkdLdPifEgsFAVDZtGsEtgfuGu7yzrkj3j5NFkzrrCUe2Dj2Je454n1TEtf+lUKFC4szOApYTcbnF4qDWFp141WevSALLwNiPSNhZ4joQTFha24vLzQ991/vKfqBab1wZXK X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: e1bce491-decf-40cb-cba1-08ddba6e4971 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jul 2025 20:14:59.7431 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Od7J6rmm/zj5VRihxrZubrF6BuWWnpzOJYPfFaVWdsKu7Ebv+sU0TfNYYUWwnnk3lBxcyLbXsFR+mdE15rVBrg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB8465 X-Authority-Analysis: v=2.4 cv=W/E4VQWk c=1 sm=1 tr=0 ts=6866e4c9 cx=c_pps a=G0EMd8eUBd5ElxF49Cdl+w==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=Wb1JkmetP80A:10 a=1tAdkt49UVCQ2orlFWAA:9 a=CjuIK1q_8ugA:10 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=lTPdD7O4eVHejAUjlcAA:9 a=gKO2Hq4RSVkA:10 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 cc=ntf X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNzAzMDE2NiBTYWx0ZWRfX+IGRpBBTUfTd 6gwRpwWRLzOckRWzvGF19hfVV8nZT20GarcSJbD8vKLCFOIGA8VJIejoTS0cUDK7hkQwNooLwRo 0uxk+LnKB8zqB4UUIdvJjmEVSnFzOCUkov748AFCrpvXLyGYtv+MLbzuV/LFJ1kBJamVqy9E7Mm FIVox+QLZ1Amq90z6DGa9X/lq20TGy1LKutF99Uy6T6sU4p4tdJWjII8P1l2bH07fj5RsnAWtsk Wt/4BYMlO63F843YCSEH8km78FpyxL9LaUTDXsUYJ4oeLslTQm8EAXI+pwVWhYvb+X82Mz2WuBY SjoNK1KQ/f2/4JgD/pJllbG7EJaCzYYJ4QgsWyeWjuvC/hYlLZ8htn2NxYDUA84W8EHGlgU8ox9 s45u3ESCVsWDZIujuywjirY0iCza48R0lu1iQmmIH/i4on2lFqdZGw8n/Iw/cknA4oBnXLg0 X-Proofpoint-ORIG-GUID: zKV5r06rfqqsnJcNGA3k4wdAVlvWZaaE X-Proofpoint-GUID: zKV5r06rfqqsnJcNGA3k4wdAVlvWZaaE X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 adultscore=0 spamscore=0 bulkscore=0 impostorscore=0 priorityscore=1501 clxscore=1015 suspectscore=0 phishscore=0 malwarescore=0 mlxlogscore=984 mlxscore=0 lowpriorityscore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2505280000 definitions=main-2507030166 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_CH3PR01MB8470E2030EECEB410B356F878F43ACH3PR01MB8470prod_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, I have run out of ideas and thought I would reach out to the dpdk community= . I have a Sapphire Rapids dual CPU server and one E180 (also tried X710), bo= th are 4x10G NICs. When our application pipeline final stage enqueues mbuf= s into the tx ring I expect the rte_ring_dequeue_burst() to pull the mbufs = from the tx ring and rte_eth_tx_burst() transmit them at line rate. What I= see is when there is one interface receiving 64-byte UDP in IPv4 the recei= ve and transmit is at line rate (i.e. packets in one port and out another p= ort of the NIC @14.9 MPPS). When I turn on another receive port then both transmit ports of the NIC sho= ws Tx performance drops to 5 MPPS. The Tx ring is filling faster than Tx t= hread can dequeue and transmit mbufs. Packets arrive on ports 1 and 3 in my test setup. NIC is on NUMA Node 1. = Hugepage memory (6GB, 1GB page size) is on NUMA Node 1. The mbuf size is 9= KB. Rx Port 1 -> Tx Port 2 Rx Port 3 -> Tx port 4 I monitor the mbufs available and they are: *** DPDK Mempool Configuration *** Number Sockets : 1 Memory/Socket GB : 6 Hugepage Size MB : 1024 Overhead/socket MB : 512 Usable mem/socket MB: 5629 mbuf size Bytes : 9216 nb mbufs per socket : 640455 total nb mbufs : 640455 hugepages/socket GB : 6 mempool cache size : 512 *** DPDK EAL args *** EAL lcore arg : -l 36 <<< NUMA Node 1 EAL socket-mem arg : --socket-mem=3D0,6144 The number of rings in this configuration is 16 and all are the same size (= 16384 * 8), and there is one mempool. The Tx rings are created as SP and SC when created. There is one Tx thread per NIC port, where its only task is to dequeue mbuf= s from the tx ring and call rte_eth_tx_burst() to transmit the mbufs. The = dequeue burst size is 512 and tx burst is equal to or less than 512. The r= te_eth_tx_burst() never returns less than the bust size given. Each Tx thread is on a dedicated CPU core and its sibling is unused. We use cpushielding to keep noncritical threads from using these CPUs for T= x threads. HTOP shows the Tx threads are the only threads using the carved= -out CPUs. In the Tx thread it uses the rte_ring_dequeue_burst() to get a burst of mbu= fs up to 512. I added debug counters to keep track of how many mbufs are dequeued from th= e tx ring with rte_ring_dequeue_burst() that equals to the 512 and a counte= r for less than 512. The dequeue of the tx ring is always 512, never less. Note: if I skip the rte_eth_tx_burst() in the Tx threads and just dequeue t= he mbufs and bulk free the mbufs from the tx ring I do not see the tx ring = fill-up, i.e., it is able to free the mbufs faster than they arrive on the = tx ring. So, I suspect that the rte_eth_tx_burst() is the bottleneck to investigate,= which involves the inner bows of DPDK and Intel NIC architecture. Any help to resolve my issue is greatly appreciated. Thanks, Ed --_000_CH3PR01MB8470E2030EECEB410B356F878F43ACH3PR01MB8470prod_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

I have run out of ideas and thought I would reach ou= t to the dpdk community.

 

I have a Sapphire Rapids dual CPU server and one E18= 0 (also tried X710), both are 4x10G NICs.  When our application pipeli= ne final stage enqueues mbufs into the tx ring I expect the rte_ring_dequeu= e_burst() to pull the mbufs from the tx ring and rte_eth_tx_burst() transmit them at line rate.  What I see i= s when there is one interface receiving 64-byte UDP in IPv4 the receive and= transmit is at line rate (i.e. packets in one port and out another port of= the NIC @14.9 MPPS).

When I turn on another receive port then both transm= it ports of the NIC shows Tx performance drops to 5 MPPS.  The Tx ring= is filling faster than Tx thread can dequeue and transmit mbufs. 

 

Packets arrive on ports 1 and 3 in my test setup.&nb= sp; NIC is on NUMA Node 1.  Hugepage memory (6GB, 1GB page size) is on= NUMA Node 1.  The mbuf size is 9KB.

 

Rx Port 1 -> Tx Port 2

Rx Port 3 -> Tx port 4

 

I monitor the mbufs available and they are:

*** DPDK Mempool Configuration ***

Number Sockets      : =             &nb= sp;      1

Memory/Socket GB    :  &nbs= p;            &= nbsp; 6

Hugepage Size MB    :  &nbs= p;            &= nbsp; 1024

Overhead/socket MB  :    &n= bsp;         512

Usable mem/socket MB:     &= nbsp;    5629

mbuf size Bytes     :  = ;             &= nbsp;     9216

nb mbufs per socket :     &= nbsp;         640455

total nb mbufs      : =              &n= bsp;       640455

hugepages/socket GB :     &= nbsp;         6

mempool cache size  :    &n= bsp;       512

 

*** DPDK EAL args ***

EAL lcore arg       : = -l 36   <<< NUMA Node 1

EAL socket-mem arg  : --socket-mem=3D0,6144

 

The number of rings in this configuration is 16 and = all are the same size (16384 * 8), and there is one mempool.

 

The Tx rings are created as SP and SC when created.<= o:p>

 

There is one Tx thread per NIC port, where its only = task is to dequeue mbufs from the tx ring and call rte_eth_tx_burst() to tr= ansmit the mbufs.  The dequeue burst size is 512 and tx burst is equal= to or less than 512.  The rte_eth_tx_burst() never returns less than the bust size given.

 

Each Tx thread is on a dedicated CPU core and its si= bling is unused.

We use cpushielding to keep noncritical threads from= using these CPUs for Tx threads.  HTOP shows the Tx threads are the o= nly threads using the carved-out CPUs.

 

In the Tx thread it uses the rte_ring_dequeue_burst(= ) to get a burst of mbufs up to 512. 

I added debug counters to keep track of how many mbu= fs are dequeued from the tx ring with rte_ring_dequeue_burst() that equals = to the 512 and a counter for less than 512.  The dequeue of the tx rin= g is always 512, never less. 

 

 

Note: if I skip the rte_eth_tx_burst() in the Tx thr= eads and just dequeue the mbufs and bulk free the mbufs from the tx ring I = do not see the tx ring fill-up, i.e., it is able to free the mbufs faster t= han they arrive on the tx ring.

 

So, I suspect that the rte_eth_tx_burst() is the bot= tleneck to investigate, which involves the inner bows of DPDK and Intel NIC= architecture.

 

 

 

Any help to resolve my issue is greatly appreciated.=

 

Thanks,

Ed

 

 

 

--_000_CH3PR01MB8470E2030EECEB410B356F878F43ACH3PR01MB8470prod_--