From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4607D46A8F for ; Wed, 9 Jul 2025 03:09:19 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1C3944021E; Wed, 9 Jul 2025 03:09:19 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0a-00196b01.pphosted.com [67.231.149.170]) by mails.dpdk.org (Postfix) with ESMTP id D43A54013F for ; Wed, 9 Jul 2025 03:09:16 +0200 (CEST) Received: from pps.filterd (m0096263.ppops.net [127.0.0.1]) by mx0a-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 568JxwUq006901; Tue, 8 Jul 2025 21:09:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=79FokA83b2OmYQeogfJCvttsc2/SSltnF+mf vGpXaQA=; b=R4iI+SSE4eOnbQPDF9dFqQ2prMB9gMbLrthbpsXNi2SZoFkRKYU5 KyaWJFuR9dIorwgBMrGxbq+LdEaiM3y34whJkevA2QnGLLfKKIaqBwDqDFwgjaQ+ OFQ105h0zz6pqdFMfXBlZTLI8tzMUtnYR3YfASfJMIcgnUZ2wKpeRTTWF7xQ5qOd Oh75mZd+MfR0KV9XTGGRnOtaG9Opr8My0F1iBKf8fyOAImcrzLq2BH1dE54cwP2j MQOekzRt0GA5Ni7RQFY+RzRm1DtJgfcGa8QUr6zudQWYIU6lT+JZ/xTjdWvjek2z I+rvST3yoD8L45GHc7iBlwhTM3nVbpWCCw== Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2175.outbound.protection.outlook.com [104.47.56.175]) by mx0a-00196b01.pphosted.com (PPS) with ESMTPS id 47rf5ma56h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 08 Jul 2025 21:09:13 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rLPNfSUIFh8X8ygj/xjeR1ogTBA410Iag/8usXIl5pkk8/lGd4GMvBm9j6uT6KjcR37JKGaE6571LxxDivzPyHLGYdZYCJuv4L/ZNxnVLYQOVVMhg8MqW9yzJrR6nsdiVlJ+qsdd/3QukPToZbHERPLKP0Apr7XKE4F8xawGteqKqeT4FhrKH5HFovSIj72vGoj5h2tl9FrBeXSEtUhHS+mBDxS23uj5es9JBBwaJH6Zmk3V8l6RH7YXT0hQ1QCEGTIQnozU9laY0lhzWTwhRHxaRclN7mx1hMrBeGP4GWVg+iFyhMwLbYyYam5S9J7Ic0VKYeFjs5XRWlM7i0yuug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=79FokA83b2OmYQeogfJCvttsc2/SSltnF+mfvGpXaQA=; b=epOOdOsJL7geKYId0/y2Z4q8Awaer6otAVjwUO98OChW4R4TVhAtsHIv5C5tHGr0Qlr9xdL7WowCyfabdEk1hmyursGGt4KYCPg3JlsehO86BMT995CCJZ8KnK/7FxwyiXGi/RLW1qAYgraX0WrwycJ51T4br/sUfYLJvhOs0xjfl+JzYVte4GSEw+Qt5lg6xAyzCd821wtE6UfrWpWm0yqpYGMgZhHiecRqAhVbMVBe450fAYr0XwrObtFj8K0hF5T89NOO0Bju2Dw7/E3iarAYpkwMsZ+it8crLif06NSL2gx4Vetym6kJTtq2JIgDIhJt+voI0GU7TSDsJmhZjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by BL1PR01MB7554.prod.exchangelabs.com (2603:10b6:208:385::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.29; Wed, 9 Jul 2025 01:09:09 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%3]) with mapi id 15.20.8901.021; Wed, 9 Jul 2025 01:09:09 +0000 From: "Lombardo, Ed" To: Ivan Malov CC: Stephen Hemminger , users Subject: RE: dpdk Tx falling short Thread-Topic: dpdk Tx falling short Thread-Index: AdvsS4XiFhfr9tUWTZG/9hrfPmeo0QAjYaoAAAZ3EoAAOAaP4AADTEIAAAkqa1AAIqDrAAADVbOwAC9nmnAAC62DAAACmVaAAApYvfAAFH2agAAANzQwAADljgAAACPVIAAA6lSAAAMFuyAAAU/PgAARDE2w Date: Wed, 9 Jul 2025 01:09:08 +0000 Message-ID: References: <20250705120834.78849e56@hermes.local> <20250706090232.635bd36e@hermes.local> <9ae56e38-0d29-4c7c-0bc2-f92912146da2@arknetworks.am> <20250707160409.75fbc2f1@hermes.local> <20250708064707.583df905@hermes.local> <4b43a1ce-2dc6-5d46-12e0-b26d13a60633@arknetworks.am> <1b7533d3-a3de-b5e9-8838-2d6608f2c8e5@arknetworks.am> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|BL1PR01MB7554:EE_ x-ms-office365-filtering-correlation-id: 676d442a-cdda-4a27-a7c0-08ddbe853544 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|376014|366016|1800799024|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?BDuCtCu34nrXymC8pgpeg4Had4ZmX7o4wtYp3MenkXlqGXKy1h1bUfd/KtAY?= =?us-ascii?Q?oz1o6XsoApNcthwYUcKiSLCvyOFhgDmBJroyBvan6kML0fsqk9UEz6GWJseg?= =?us-ascii?Q?2q7c2DtYOSYldo/d85QjXIF0wIOJSfaOnAdMe2ed8omAr/UOS5NNSv6VWazt?= =?us-ascii?Q?rUBiHx0T56DvCuUSyCl7gQY87jvYweRCMsR7eGUzlRf+Ql9aeeRFL6tlz37m?= =?us-ascii?Q?IFWsGQY8/3utYbpAK0gM0++eOMtfODyPhWIE2eIyyDHsrVvoG18Kck6f5PbE?= =?us-ascii?Q?Hbz7oN3HcDUVExTnFVDa/qczg1rAWCFnAlcFl90Ha4/9nFbsYT46otMPDeAb?= =?us-ascii?Q?5pqdp3xDwbz2dQYlS68+LRurNPqP0yMIydBiGMKjsHyXPiExiUZ+6G5/FPCq?= =?us-ascii?Q?K8EqjmkRqswtTzK5o4wfDcXQKiDBEmbL+iMsRq0WCwIpjITX0y9og/r5wjog?= =?us-ascii?Q?fDQvupnkvBNx90Y/sYAMlp/L7/jGmD+gCYwWK+iX2kgLBjPNetyhOj67LBDJ?= =?us-ascii?Q?WHibdsN2lwbfYdGnahaJZi04abG1PND9PqNn3Ry0rQA8WteTKZRrTdNLhn52?= =?us-ascii?Q?P2gHsEXZZ+qKDW1p3FQ3nuQODvRk+kwfyUHkfIA0rUF2FMhTj+RUMPPTnwj1?= =?us-ascii?Q?HPkfimK6z3ptEoHyNzvSGksLpFkYf26RiY3OrvWEuAWQnhbR10n3AA5Lkde6?= =?us-ascii?Q?LvJ1vARvo8RZ9LqZ1Q4C59ynzcsCtzt1HuBV6cLkzwURKpvZv27vQ67Je5qM?= =?us-ascii?Q?QZ1RR8g2bTzd3b8CCk4d0H6R3tNdNHEHxGydlCITLmCcaut4NX4PQPBwJ3eZ?= =?us-ascii?Q?p5tFyE1eBOOutnhWkJGseXlCOmTtR79Ux9IUXSyVPzhz9o1H8eD+IjvYtx6U?= =?us-ascii?Q?UemCiTKLDx8A7ZHMuT3f8Z3gBpePKHXF5wfgAxQaM892FHVO7aCyAXVAlROp?= =?us-ascii?Q?0NBgY0S6cxZvdefluN9GpliWgJStKkm8qw4qYujMjUWwGRR6ZS2jlRglLuKN?= =?us-ascii?Q?YMdE/bsplDmfhysS6jCOkHpfHWLFsiQtPTiyaQEGY6yrK+cJSwbkEvgI8FLr?= =?us-ascii?Q?uX7xDJqWowcn8rNWXWXkkxbKifb6uP2pim5FYfePIZmvNnSaLidPi/WEG6uY?= =?us-ascii?Q?6nv2rxXpMrSLqPrgAvlu588HN8/tmrfxQqs9uMd651A2PEaTe1ax7V94U9sb?= =?us-ascii?Q?ydxTpi4NFaSHvsBZa0my9DNhBPmWUF/uwGQwnaEInd3+bD6HvQt9RC6Jq9RW?= =?us-ascii?Q?nHHw7bb64wVJsl9x8jggpB7GIVbG96D7j2r/i26ASrJpEKk16hpHkt0WqWor?= =?us-ascii?Q?ouamI6aieM3Ue/BXXFq0rVlJJRMd8YZGXabxG1svT51pAL2QXrmINaf7/1MD?= =?us-ascii?Q?1+osjTrGme3eaIFDdjCWpOJyXbum3DUTZTpGVXNGBx2JsQJEpdv0HM1kMYCe?= =?us-ascii?Q?dzwUvtOYOR/zcU9ZjGrhWaRE1ESXbIdz?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(366016)(1800799024)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?X91tqYKjo1aAhB6LydH5NdZtlPbPJvcO8qywpevrtWdWqrArvfb5KwuxOLw4?= =?us-ascii?Q?9tdF41bSHvC8GELZwG9u41JdW4PbxrkmhS+8XIAZYNLVkLAoeQmxrRr017dF?= =?us-ascii?Q?JPr9o666Dh6G2pbmZzGVXjlwJp1mVLkGuGut3UdZ+yAlZMUsGziuEtejjbva?= =?us-ascii?Q?pHJIxsG2VIuVpp+o61mM/+tiup0IkzWOGg1kznZmFzHM0BOWjY5i3P/QHM7z?= =?us-ascii?Q?/0trAsitaxpEmnmVJx5md5dAJKWbpzVgoIU3VCfesvhJtOv647Sdutk7E98c?= =?us-ascii?Q?S3cbUwTveXnMZ7eAvfUk3c8LO7N5S86aGTrK4WkAfeWkJ01/J5WV/fNal72Z?= =?us-ascii?Q?YSnximkZf3PevrVzODBixo8MQl3lSDLFLmo+tfrgAGm0D5qSoxa0BI0yPIjX?= =?us-ascii?Q?BcC2v+eW65ZF3OqCg5zsSsO8HZnR5bzUXYfSax1G/Xi8mtyW/3Y1KsnDLzmG?= =?us-ascii?Q?zqIyS9ueURSdFrewFJG81l3VRdVlLTHQg/b7DbcOcXXSrVsp1B3SuXRyQ7gN?= =?us-ascii?Q?QhRA8NKul7yazqKGVxc+kwup6wn6Pyh4YwarvqPQvPvhR1xgEe457sJ5pLhR?= =?us-ascii?Q?oUnNrmehydeDchQG4MM0D8MMdSkUsKMx4w5vqjHVzsrvbpBUYEzuJJ3ysj/f?= =?us-ascii?Q?mm0o98Zt/VkX7uNwaschK5RO39QDYraMvO668KWWnOhW1AJ048L7+zLyKCjj?= =?us-ascii?Q?2RhWJAXTzFHkFaJh5g8Zvt0VwasTX5zsXV0ixa7QcVtLgBxRlxWa45rIt//y?= =?us-ascii?Q?fzSSTlUPtHaXTxC9dzL3/vkJwj1PaO78ka6K6undBYReUDqchlcoYNOi8o2h?= =?us-ascii?Q?DY2HbEaSxQEwo+Hr2cNHiPJIny90jkLNAeRfc8dmKUUtl1QvTPtBQkHnMWYb?= =?us-ascii?Q?R0MMubsfUXJk9f45DelyHbJlCYq4D4x+9Nbj1u1t6aKgy3Y/D5A6JhdGG/y9?= =?us-ascii?Q?7Q6aVykqqF5m+dC2m48u2tqYGESgGw4wnw6j0273r83lEjal6tXB+FPUdHXV?= =?us-ascii?Q?Fe3okUGSrnypWwET62OxEO+G1OgiqlQFKhNeeAqQBWknNiXAqyybFlgOFe4x?= =?us-ascii?Q?03CE9mQYuDeQ55yHLNPaAB+Op4UDMgvwREnUHg+wcwtTkyUPmc/chuiNSskv?= =?us-ascii?Q?NShDiC1XH5dCB4AQ1p+pWtsNzzxY+3jGjf6BGGBLBydKzWkVhL8qVp2kXFXh?= =?us-ascii?Q?Y9C+8ZTstAj6Gy3OOYxeUpudHUs7ZKeIJualuteH4/jLZKh78BUvj9cwyDF6?= =?us-ascii?Q?0GKH+gFtT5kg4Gp4pwMrZVGnIXzmZVCcFiEku+snoaD3Yb03/4W1Y87CYkHK?= =?us-ascii?Q?FRsJ46hcivvLIpf3j+tuQRHXZD8oDh/pPZVvUF6LbTt2+rCYIM9xxBnVWX50?= =?us-ascii?Q?1q1o/HQOFeG6k/RhJEtkGV3N880teQrZUIY3lW7gTSeo+Qd1kzqqgSbxFS/O?= =?us-ascii?Q?IpvjT708K8uXNkX9W0Wzk8CWDlvBX4JnnZl/7VXj4uijr/wHcZMcYIYylQW9?= =?us-ascii?Q?Kbv3LIz7yfOGgSW02by0A3wwzPSRuGWkPfYE3UQ5uh43UdLTwat7mFMVb2Q1?= =?us-ascii?Q?WEslnzCqjiVIyMPX461A89+R48QRQH7cm8LzTlBY?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: q8JT4CQ+D3PRQwP34FJ/MIQl2XDA8YB7Q9Kut0EMSZMdNJY0H77UfvN3Ygo6SCF6RYtF+gnyvPFo65QkdvU1BLHmkHYLtDjQf9EJBmKqW6Naa5+R+tTMd/ex0kTZiNP3Y61u+YaJ1qNTgkSm9jXJRq4jhE9XRY/bOcMIPulc74SQ5thz9eKJuEtIYqUd9Bqg5wx6ZBE52cWznvI37NSAUCLx3wmv/9NEQ/GeJQpqaQdDfcp6q/WZpS4WrpZI3SMkFYKf3hgUrNFXjj04tNC5LkitVCCtSXX9xAI4sA9OI9JMALrGmM+tqN79JZyumFIED94OQwAdRCLjkaQp315bRwnFLfWtekZkk9xwNz/XRMGzAAbXxm6vXbjYilBaW5LkEku7KTlzsHmZNLg5LNt//FNQtf+po5Tl8yQGmL3dzNI4ne2Q1e6ilVB8sDz74HCfOH4LIoyai7PZwjpciTSgFERZT2aHD85u1X4XfuGPu0v+iPrgt7QOxmomweilZDuDOy7yVXUjvprh9M9UqNY2PzcyNj1DadSSVPw9vi+EBR/0RytB4JBQ/ayEUbs4Tr1/cq6gFrREjbaUGxI9o+wcSaT4x89lpW4SKCu2M2i3YdlvQlYswEfdrbiY9lhfli0o X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 676d442a-cdda-4a27-a7c0-08ddbe853544 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jul 2025 01:09:08.9739 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +1lQi4jL7PSuEy5GubZM1+odjZbLg2FqGnoSHPKGEvveICUzz48PDeTqZ57fIrQCxs4hEXWQXnj6fRzQ8B4dLQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR01MB7554 X-Authority-Analysis: v=2.4 cv=NInV+16g c=1 sm=1 tr=0 ts=686dc139 cx=c_pps a=AuG0SFjpmAmqNFFXyzUckA==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Wb1JkmetP80A:10 a=uherdBYGAAAA:8 a=8rWy6zfcAAAA:8 a=jQOgFn-ZAAAA:8 a=jZVsG21pAAAA:8 a=YJkDK-Jr7TysNG3Gxl8A:9 a=CjuIK1q_8ugA:10 a=YjdVzJdQTyZRADMV7wFX:22 a=mT82qxFQzDvLIExZS32s:22 a=3Sh2lD0sZASs_lUdrUhf:22 cc=ntf X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNzA5MDAwOCBTYWx0ZWRfX1I0SZPIujltK 0oW4FWCQCBrNGoH3ue3uZHntKZkgUmI6+5ATkvJH10LMN7Wx4aKKO5Zpv09jHH2i9iW4W4ajtXT 0kMuX9w95RdMLwrjsLXud3Om9P1ekEeqhdB0HzoyGf8j7fNDKQXXjz9Fm1r2YG22KqGg2UtnkJE AVaDGzI39hidTCM50VhK4co2/mSx3KNBM3RMC9/LK4DL/60Xq5TA5sIc9s3y02P+KiSFA3zEzHC 71Fk+88jnvwp+03qj36GUh5uDRXNVdbBGqh7b8sIv+5aR2i++ngpZAyyMJjHM/V2HkHJTDGWG4i ry7ZuEjgDGU+RhAYYUVOmU64q7gL2OcuqLhRAf5m4Kh50+ASgc9sZ8QqPu8JyU1QhzM+Jb7Gxju SHgcbnaDqvZmZ5Q7k5etWaijze6n8tI+TK7aQUuluuXqDpO+fJffvvRmkry7D+QEWd6tlRHU X-Proofpoint-GUID: _LxLb96iJum8Hkod9mJ5-fsuc_HewgG2 X-Proofpoint-ORIG-GUID: _LxLb96iJum8Hkod9mJ5-fsuc_HewgG2 X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 lowpriorityscore=0 mlxscore=0 mlxlogscore=999 spamscore=0 adultscore=0 suspectscore=0 impostorscore=0 clxscore=1015 bulkscore=0 malwarescore=0 priorityscore=1501 phishscore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2505280000 definitions=main-2507090008 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Ivan, I added two mempools per port pair as you suggeted. The results of Tx performance improved and when turn on one port pair and i= t does not affect the second port pair. The tx ring no longer fills up but= drains near empty. Improved: Tx - 1.5 Mpps to 8.3 Mpps, 1.5 Mpps to 11.2 Mpps I need to do the perf analysis again but wanted to provide you results. I still need to improve the performance on Tx, but this is much needed brea= k through (with your help). Thanks, Ed -----Original Message----- From: Ivan Malov =20 Sent: Tuesday, July 8, 2025 12:53 PM To: Lombardo, Ed Cc: Stephen Hemminger ; users Subject: RE: dpdk Tx falling short External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. Hi Ed, On Tue, 8 Jul 2025, Lombardo, Ed wrote: > Hi Ivan, > Thanks, this clears up my confusion. Using API[2] to create one mempool = for the network Rx and Tx queues must be MP/MC. The CPU Cycles spent on th= e common_ring_mp_enqueue increase as more ports are transmitting. The tran= smit operation causes the call for Rx and Tx queues results in fight for ac= cess to the mbuf mempool because of one mempool? Not really. Mempools in DPDK in general (and, in particular, as shown in yo= ur monitor printout) have per-lcore object cache, which, if I'm not mistake= n, is to avoid such contention when accessing the pool. And, since only a s= ingle pool is used in your case, the use of MP/MC seems logical, as well as= the use of the per-lcore object cache. But it's not obvious if this is opt= imal in your case. > This is why you suggested creating two mempools, one for each pair of por= ts. It could be a low-hanging fruit to do a quick check with two separate mempo= ols, probably also MP/MC even (allocated via the same API [2]), to know if = it affects performance or not. Again, as Stephen noted, this may even worse= n CPU cache performance, but may be it still pays to do a quick check after= all. > If I go this route what are the precautions I need to take? > > I will try RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE offload flag first. This is somehow unrelated to pools and rings, yet it should enable the PMD'= s internal Tx handling to accumulate bulks of mbufs to be freed upon transm= ission via bulk operations that, akin Tx and Rx bursts, may also improve CP= U cache utilisation and overall performance. The only prerequisite - all mb= ufs passed to a given Tx queue have to come from the same mempool. Hopefull= y this holds for you, if the logic does not intermix packets from 2 pools i= nto the same Tx queue. May be Stephen's suggestion to use a Tx buffer API is also worth the shot. Thank you. > > Thanks, > Ed > > -----Original Message----- > From: Ivan Malov > Sent: Tuesday, July 8, 2025 10:49 AM > To: Lombardo, Ed > Cc: Stephen Hemminger ; users=20 > > Subject: RE: dpdk Tx falling short > > External Email: This message originated outside of NETSCOUT. Do not click= links or open attachments unless you recognize the sender and know the con= tent is safe. > > On Tue, 8 Jul 2025, Lombardo, Ed wrote: > >> Hi Ivan, >> Yes, only the user space created rings. >> Can you add more to your thoughts? > > I was seeking to address the probable confusion here. If the application = creates a SC / MP ring for its own pipiline logic using API [1] and then in= vokes another API [2] to create a common "mbuf mempool" to be used with Rx = and Tx queues of the network ports, then the observed appearance of "common= _ring_mp_enqueue" is likely attributed to the fact that API [2] creates a r= ing-based mempool internally, and in MP / MC mode by default. And the latte= r ring is not the same as the one created by the application logic. These a= re two independent rings. > > BTW, does your application set RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE offload = flag when configuring Tx port/queue offloads on the network ports? > > Thank you. > > [1]=20 > https://urldefense.com/v3/__https://doc.dpdk.org/api-25.03/rte__ring_8 > h.html*a155cb48ef311eddae9b2e34808338b17__;Iw!!Nzg7nt7_!GXTS2DQR0JZFGh > dahtcpSBjmoh-AZ4dzS73R_J9A1I0JxlORLHvylHea80X_KHTZRcZV4qcMEvJd7Z7izij4 > 0zap9fvA$ [2]=20 > https://urldefense.com/v3/__https://doc.dpdk.org/api-25.03/rte__mbuf_8 > h.html*a8f4abb0d54753d2fde515f35c1ba402a__;Iw!!Nzg7nt7_!GXTS2DQR0JZFGh > dahtcpSBjmoh-AZ4dzS73R_J9A1I0JxlORLHvylHea80X_KHTZRcZV4qcMEvJd7Z7izij4 > 07rwGv1P$ [3]=20 > https://urldefense.com/v3/__https://doc.dpdk.org/api-25.03/rte__mempoo > l_8h.html*a0b64d611bc140a4d2a0c94911580efd5__;Iw!!Nzg7nt7_!GXTS2DQR0JZ > FGhdahtcpSBjmoh-AZ4dzS73R_J9A1I0JxlORLHvylHea80X_KHTZRcZV4qcMEvJd7Z7iz > ij402Z4uOww$ > >> >> Ed >> >> -----Original Message----- >> From: Ivan Malov >> Sent: Tuesday, July 8, 2025 10:19 AM >> To: Lombardo, Ed >> Cc: Stephen Hemminger ; users=20 >> >> Subject: RE: dpdk Tx falling short >> >> External Email: This message originated outside of NETSCOUT. Do not clic= k links or open attachments unless you recognize the sender and know the co= ntent is safe. >> >> Hi Ed, >> >> On Tue, 8 Jul 2025, Lombardo, Ed wrote: >> >>> Hi Stephen, >>> When I replace rte_eth_tx_burst() with mbuf free bulk I do not see the = tx ring fill up. I think this is valuable information. Also, perf analysi= s of the tx thread shows common_ring_mp_enqueue and rte_atomic32_cmpset, wh= ere I did not expect to see if I created all the Tx rings as SP and SC (an= d the workers and ack rings as well, essentially all the 16 rings). >>> >>> Perf report snippet: >>> + 57.25% DPDK_TX_1 test [.] common_ring_mp_enqueue >>> + 25.51% DPDK_TX_1 test [.] rte_atomic32_cmpset >>> + 9.13% DPDK_TX_1 test [.] i40e_xmit_pkts >>> + 6.50% DPDK_TX_1 test [.] rte_pause >>> 0.21% DPDK_TX_1 test [.] rte_mempool_ops_enqueue_bu= lk.isra.0 >>> 0.20% DPDK_TX_1 test [.] dpdk_tx_thread >>> >>> The traffic load is constant 10 Gbps 84 bytes packets with no idles. T= he burst size of 512 is a desired burst of mbufs, however the tx thread wil= l transmit what ever it can get from the Tx ring. >>> >>> I think if resolving why the perf analysis shows ring is MP when it has= been created as SP / SC should resolve this issue. >> >> The 'common_ring_mp_enqueue' is the enqueue method of mempool variant 'r= ing', that is, based on RTE Ring internally. When you say that ring has bee= n created as SP / SC you seemingly refer to the regular RTE ring created by= your application logic, not the internal ring of the mempool. Am I missing= something? >> >> Thank you. >> >>> >>> Thanks, >>> ed >>> >>> -----Original Message----- >>> From: Stephen Hemminger >>> Sent: Tuesday, July 8, 2025 9:47 AM >>> To: Lombardo, Ed >>> Cc: Ivan Malov ; users >>> Subject: Re: dpdk Tx falling short >>> >>> External Email: This message originated outside of NETSCOUT. Do not cli= ck links or open attachments unless you recognize the sender and know the c= ontent is safe. >>> >>> On Tue, 8 Jul 2025 04:10:05 +0000 >>> "Lombardo, Ed" wrote: >>> >>>> Hi Stephen, >>>> I ensured that in every pipeline stage that enqueue or dequeues mbufs = it uses the burst version, perf showed the repercussions of doing one mbuf = dequeue and enqueue. >>>> For the receive stage rte_eth_rx_burst() is used and Tx stage we use r= te_eth_tx_burst(). The burst size used in tx_thread for dequeue burst is 5= 12 Mbufs. >>> >>> You might try buffering like rte_eth_tx_buffer does. >>> Need to add an additional mechanism to ensure that buffer gets flushed = when you detect idle period. >>> >> >