From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4612946B2A for ; Tue, 8 Jul 2025 16:03:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1A781402A0; Tue, 8 Jul 2025 16:03:16 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com [67.231.157.166]) by mails.dpdk.org (Postfix) with ESMTP id 426D14025E for ; Tue, 8 Jul 2025 16:03:14 +0200 (CEST) Received: from pps.filterd (m0072399.ppops.net [127.0.0.1]) by mx0b-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 568C10PZ032025; Tue, 8 Jul 2025 10:03:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=rqlwT5UwMu0Zn2OTqLcwnv15r6HkT81ed87f bOO91YM=; b=Zc1z62u3lhRg4+1LYBc1ojUW0n/Hkwi2JXLmuANqOyBjXOkekn+a R/tIDJC9FhoqT7ZL4ixJqHQyZWCN3D66UwYYEf6ALxsmhlN3pFTxXd8kFPjbqJ1v jX9oo6wFNdvY5wn4nGZfhMya9IG9f0MsCiBkP3fxZ/BEVtyssVq4wmnSlJlf1mJ+ +8Cu/AXeVCkyYBQer6QpBGwJAqhmlrSarWWoMBgFJ+UxefMmyxVprfUsNTSC1BKl 9OI3Fa27Y86kB5JfpjVdmDkY0MFSi0C98yU5+erIksnlIXqAADq/UnlLW5TFxHlE /84emyYBI8pcH777FruFysqTUIYabamowg== Received: from bn8pr05cu002.outbound.protection.outlook.com (mail-eastus2azlp17011027.outbound.protection.outlook.com [40.93.12.27]) by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 47reme1eht-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 08 Jul 2025 10:03:04 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PmmD6n2YjLnYUS5BHlbA7x4jRSlkZA60rjh4EGJiST7D5q9glZ3wzGMC+0gwc3KG1HE0iEU2UAdc4LYnmV5zENPadXS5r5akpkv/hwfPvAvZ9WkRqZHNaiPMANZzNw+gG7W/HNLUsSZxbLE9OKwVO3hKcGAM/9EP8ydojHfDv9rQ8hlLNpgD8N2Q2K6ixc98bDXyqPLmZsfRzePOz2f1tWjkOV2qtybyIzTsKpg3JaBQ21EvmikxMLcWwqpK7I+hyGL3TupIeL02fqigZH+yghS0HesU1cPYvWNSRIdLqIVb44j1NfTJQV0VE7uIMvumnv1NjC+CXS9J6TjmvP+c6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rqlwT5UwMu0Zn2OTqLcwnv15r6HkT81ed87fbOO91YM=; b=G8edZttLd0aarWg57XFShRGNTupVnZhj2NOR6slH5n4/0tjblHSXzsCFkBfY97XwW4STQQ5poFRekI+ZHmRfOEVlyAkZJAgfIuLeitEKfmOiZI8RXaOULNnh/l1ZjXpdsL5r6icfHkq+Z3L0FhzEFjLAnVHX5viOtnkoeKzS4mPoC+8ESCSt0Mmpbi7gC+qLVnybxGpzogSQKaNkpSkBHD6MBD/3PeX8MqmVavi5QbsQzeIU63lHPaZtW5lL9OznmtAyoF99A3BZmpIm4ZZZmeSnwMn/UiD61TnBmvUyWvQH/QvBTokw53VPgpNf0cNkrDAqqPmPI1FC1yWUoSm9Vg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by CO1PR01MB9035.prod.exchangelabs.com (2603:10b6:303:275::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.24; Tue, 8 Jul 2025 14:03:01 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%3]) with mapi id 15.20.8901.021; Tue, 8 Jul 2025 14:03:00 +0000 From: "Lombardo, Ed" To: Stephen Hemminger CC: Ivan Malov , users Subject: RE: dpdk Tx falling short Thread-Topic: dpdk Tx falling short Thread-Index: AdvsS4XiFhfr9tUWTZG/9hrfPmeo0QAjYaoAAAZ3EoAAOAaP4AADTEIAAAkqa1AAIqDrAAADVbOwAC9nmnAAC62DAAACmVaAAApYvfAAFH2agAAANzQw Date: Tue, 8 Jul 2025 14:03:00 +0000 Message-ID: References: <20250704074957.5848175a@hermes.local> <20250705120834.78849e56@hermes.local> <20250706090232.635bd36e@hermes.local> <9ae56e38-0d29-4c7c-0bc2-f92912146da2@arknetworks.am> <20250707160409.75fbc2f1@hermes.local> <20250708064707.583df905@hermes.local> In-Reply-To: <20250708064707.583df905@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|CO1PR01MB9035:EE_ x-ms-office365-filtering-correlation-id: f83ff5c9-4181-4e39-40ee-08ddbe28265e x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?BWq0KcFDjtcZnuALRGQd7HiJzsgP1+MbYUnFI/xK88kgetrJKSmSj93ZAet1?= =?us-ascii?Q?hzabaG0o22AjHfVZolQ5JyEpSWUQHLw0wGmxAqhGJxQIW03iddmi0lbMi5fd?= =?us-ascii?Q?l11uDjf3G1DVmiOt+6T/fNOK5Q59TcssFjnQegMOWPIWMS1Nr8IDJEd7DGkp?= =?us-ascii?Q?PDU6lZuUp/T6+Tbktl4AFLfRm0g73A7mbZ4O7u8LWXhmy7SciyLDLEIqyWKu?= =?us-ascii?Q?9IZ9VuGITtYiNtuwqC/nGxvmUWWFmXJA/YvJhMpgTmtnR7nTPulnbVwlEjvV?= =?us-ascii?Q?zNEmr9kCSxOvvToaWEwAy+6JKWj7Ey5T4vbIo2rHsrfn+d7VMU4bjGnf8Fwz?= =?us-ascii?Q?2uPsQKFKfg/efzzT0hiK25KpURMPh3yMs+u3SOdjR4rDnZaHfdNgNVrdxZ3d?= =?us-ascii?Q?wWuLiqje8tQU+i2GGZOFY/zf615mRHiY41TWjedNt0UscdCz9AV624utKI4C?= =?us-ascii?Q?hnKD+BU62lbNFFVrKzwiCAjn/GanfvbVE1vAIDgEQSVM/hJzvfeHQNowG9pr?= =?us-ascii?Q?6AF4FF+XpPeOEQgshVb+bSRf7lJhCh4N2xH0AHUm1IL5aFgo/mFGYinhvMtK?= =?us-ascii?Q?51BfUcCy0MtYIlNl0ZLNe6gXfD4QfUXMi8z4zzCfJAlOBsmnHneaD/VXQ7Rv?= =?us-ascii?Q?le19A9bO9XEfOQ4n2UPhGDaSCvifCbNROQd4HkxUBAJpcyp+9znjAMuSIa2+?= =?us-ascii?Q?VL+P5FtHd+JDRFoZP/cLeHBgYW/r4uSwTtX6LvAHI8kkUQf5k3aUFoIgsGZA?= =?us-ascii?Q?rfsx8O6dSPpO37VJo/eAgYJlnILAqoXwF5J6c0L63ABIJ2ILn9wUbhnrPT0t?= =?us-ascii?Q?mjrnitq28/Xrf7U46o02GxBbcja+R9aqpphZtKRX+yGiGEtE/PghR3sanTGi?= =?us-ascii?Q?SUP8SZFI+r5EQ6OTAVvXcDEP4+yflX2fmawFqhXoyz6dDfXSkDF6sB4qCDsJ?= =?us-ascii?Q?RhhDvuRI6DldWlvyqio5hEOLKz8zjYsLxXckaQ1xvHnGQi/l8PILjHbek1Es?= =?us-ascii?Q?199C8dqI8YEzhylryuXHuMG2Dpboo+vKcYIIbBoeJiLhS3H0uDjXVuL9yWSE?= =?us-ascii?Q?Ok0bBLWCEHjkHcyokJtS7CaWt8aE38qij9/zNV4ULlZ6H+DIAmYpncAplYNU?= =?us-ascii?Q?QLR0PVTZmC/M6lUSpaUnA4+kfJUMoGuMi21/+s47SNHBYePSWx9EYpIoddjS?= =?us-ascii?Q?Iwh9Iiyd7lH2drfguHWW8tLytYvY2M7skGBwf5LfJ4GA1Gtubfbp+Sdf204C?= =?us-ascii?Q?g2RrIueqpn8t5U911tCOrlDgzalN8OxRd8Cy+qlIVSDtgs+i6J64yBD8TBEH?= =?us-ascii?Q?v5EVMkMSoGpQCNQrdi2w8YBG0LJ3wsX/4jOlCPMLrpZ/k5eMgiFrwH3DgQkj?= =?us-ascii?Q?Pk3VgWx4lgNYNqlMSQ3XOeCAqwWqEa21wNtes1E6Xh/jb9cVPpJiwRnp0Dec?= =?us-ascii?Q?xXuXL1ypgndE5RTlb19ob5RwMsoJQU6qbpbdGlo8pLTfytlXiFhhsackW4aU?= =?us-ascii?Q?PeJXKg99TQDPLLg=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rjrBQsUQHTL4pUiWilS9I5I7M0DCIe8qDmFPY4gnuB4A8u5jJnUc+eTUxYWr?= =?us-ascii?Q?8xfxGEQBwwGm8uz61wZWC98SoOgNT2qVMKzT5xEDPELsEdZcJiEQBcLFbm06?= =?us-ascii?Q?6mV+/+0LXcn6J0vwJ0GyrXLcoBa2UW2xx9bkVfaeQh24v/Dhqe+qViZKnT/f?= =?us-ascii?Q?43ccI3hErwWhT99mfR3JOqnU+EmMiIlsl1hdymP0e5MGxf/BNu7IyEWzvPd5?= =?us-ascii?Q?2tNM7sX/K5qsgB5MF8joK6n2E/Fpi5w0sOtyxz+X3zu5Gi5TTyVWPs68jBJq?= =?us-ascii?Q?GeBMOAPID8yTcMywhcMjJWYpQ0uDOCAvuS/IsWQEwnviVzFRdEP24F2j1M6E?= =?us-ascii?Q?TvYNZbN7FxZO3HNvoSuOk2XZYGQlzRaFVixFpl0iqkrjsM5NndhyRFFV6g2g?= =?us-ascii?Q?XKIfZtf290mdsH3pD6UkiAjpmQuFyfT//awXzI9Jbe6THL95YwDW1c9HbQ3r?= =?us-ascii?Q?NyG0Iby2KSYe/O1JqHb/dxqek7sjlBnapswtzQvEn/oq2rFE/fw8ldvK+uH9?= =?us-ascii?Q?/nLKXfZLhNfrI+TLe1pIw74Cw7uLjQArE1vmMDU2Fj8NZFX0OGXozG2Jq+ZB?= =?us-ascii?Q?9k14NBuERlj2CzT85IlnKC6e7wji88D55T1rKWb/apw+QLmBQl2zWRB9UEzu?= =?us-ascii?Q?Mb6WUQy464Tnx2VL6/YXh3C7V//YezRzKXLasjm/mvpJ0oK6tG9b0kwYIjEI?= =?us-ascii?Q?K5B6ChhTtFBO1+XySEVfTBMqkHIwURiKlZs1djxELppHSLCr7h506poy26ef?= =?us-ascii?Q?HWMR1X8K6IEohCsfJYa+obXpTQXpYD38sGTfMV9+YdCo9OEl+yBj39C5kKn7?= =?us-ascii?Q?C8Mlc/QOjMVGmzXUsgq4nS2mG8lqU6pT79kOw9gvhtHBWCIzIWKO62LDmy2Y?= =?us-ascii?Q?8mcJI7BYlN1iutPfGBTg+QU8xOj2m29cQ5MTR3CDy/dzk2oLrw2UiPrUl5GT?= =?us-ascii?Q?+6K+VY8co3PkwOI04fBLW5H4Ij/yDimwSYIbH2UbyRE76DZGrKDZq8fEabIo?= =?us-ascii?Q?38S3+LtkuJZo61vmzgesZUcwiUG19sMb/IzQ5GWZ0OenYjHay4FiVqo5ww8N?= =?us-ascii?Q?+tarHTj7QuHgCVJ9e2IQZQq+Kde4NIPwcgpcqKWZrQMbyKWWpPE0/i/jFyzk?= =?us-ascii?Q?xl3b7fCna4A88jjxW235ycrSZu5izbiJpbEnS9u88WZHrkOaaixuxGLmGPzk?= =?us-ascii?Q?5MAbD5JQT+6oN/VvVPFmIcTrZfCz5sc4neM7pP3HAx2grDaW1vaHtloBiXEa?= =?us-ascii?Q?RsJhAae0UiUS8ia7hL+ilaAvviRVG0gr5ZU3qAbhbw/czMW00uYdDNTwmwst?= =?us-ascii?Q?6HxQVAwEbJhNcIIer+D+5hwvDy5ol/UzAL7O7D7Wla0DnjbE+myUW7K7VuTY?= =?us-ascii?Q?ppydWBbibWvgOaHhLzIBkfvLs8CdPytdT2+NF+EkExeGBrM7+iBa+kM347Zq?= =?us-ascii?Q?khKjBmMjcxlWzGNdq7jBKxXhk5DFioBU+Es4Ox3I5736ieUHoL0FO5s89dn1?= =?us-ascii?Q?8MQg6vDqta2BPo3fJ9DdhoKIxlJAzXdTDwpjQBEFLsZgrT4a7VTlV7uUyr0n?= =?us-ascii?Q?ku/YoQrDNRSiK7lwVxdvmiuENZO74d67i4k+tKFn?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: NIympnbWQqdXLqyXokr7KeugDf8eqlXF/tMkAZsmRQfoGbfqbeAIuYUJc/JtGd0FCNfFdE4YJigmisSKYpa8AqehUQ1YagASTkhxFKeqNeQ0e3cWd+pccldZWmDEsSTR5yAsTWJBd0vfF7FuvpFGOH3N0mHnbqw6zsqHTUns/szwol12aT2xQ3HuHPWosHMjVLjkc+NkD82wXSVXOAvIE49EY2exIh8F4O4BScWeBbhhGxkP2fO0OTLMhsW67Tbmv/6PJEe9eXjatfS2ncujpilMxtutsQk/+2uxoGO+eBpoie8NHjZGdlxY5s2yy+ijyrixhXSmhPhh1w/1BzOguE5VQaUKutzGatkz0YAB1nxH0l4KCRZvrLHp6ZX2mAVBM5xTJtOkRFXbgjIfjuS75+Bex0UeYriBpgRfXfA4j6kZ8DOfK0xDeVJPnFJyShWpfyFftnasIBhXZBtLzuplv76dhmVz5qXlLDiiu3AaU1HRREocwcjEpY4PWrMDP5mQ2IUo1MCWMs2ZOY7JnWYaZWrJ2rVrk8Ycwb85GwHJSBVAhjCoZw/A9VWqXMJ3Ko0VJt+aZizwtgHpASLxZDTb43DqSKyU64qDrPRabMDoufABpFaoMOg3PjncjyGxaNE5 X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: f83ff5c9-4181-4e39-40ee-08ddbe28265e X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jul 2025 14:03:00.7600 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: BuVl0TJWTRrdtsNcM72FuJTPJyKOwkR73hn67TU3GgfytT/Xs5gXtX41MmbKWdn22nstCnuHSCaJV0ulUHBAEw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR01MB9035 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNzA4MDExNyBTYWx0ZWRfXxsgg6bt38lcM QdepyLReLeomskR12dj2dYNxDTAvKc9VK/dkKY2giycf+0/fRpUMIJF9542ROxPH9JsNtFIWw8+ 7/g6K0mMXvTjwpHgfBAJKX8V8LKazGp/esUUXYtS5DqNnSxkXJBYL9CuM/vO6BftdqJl0Pwwk8Y 3ZCcc7cxXF+JaxvpydcusdTIGQDW6uOQi1z1nIn6THWpKHnguAO07CLHrkPO1SQwKgO6WxlXLs5 5BM10lPp2cnx9BH9C0BwbBtXNJwolZ87Rf9xTFP0VkRqcwqW8cGHCVl2/G8Qzu5UzL3husn1oao xJHNcNj//n7VkF0IEszgCN1UyCn4ltPw18H5fw/fybIODtvPCW/vuixRdez+sGocpRti0hlQYCY M8iGI+xMT7vnjdOxTy7mNnoO3rjXAG5glj+DaBpXRKL9kSpE9UpBitOa2L3NvRD8ULpY1pAG X-Authority-Analysis: v=2.4 cv=KuFN2XWN c=1 sm=1 tr=0 ts=686d2518 cx=c_pps a=k6qe+EuqS5agFzeLFj3oqg==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Wb1JkmetP80A:10 a=jZVsG21pAAAA:8 a=jQOgFn-ZAAAA:8 a=8rWy6zfcAAAA:8 a=20hULY_7Zm33HNLB-kAA:9 a=CjuIK1q_8ugA:10 a=3Sh2lD0sZASs_lUdrUhf:22 a=mT82qxFQzDvLIExZS32s:22 a=YjdVzJdQTyZRADMV7wFX:22 cc=ntf X-Proofpoint-GUID: n_RLuAbf7qTCbvvLR47xtmjv1KZqXPuT X-Proofpoint-ORIG-GUID: n_RLuAbf7qTCbvvLR47xtmjv1KZqXPuT X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 clxscore=1015 mlxscore=0 phishscore=0 bulkscore=0 adultscore=0 suspectscore=0 lowpriorityscore=0 mlxlogscore=813 spamscore=0 malwarescore=0 priorityscore=1501 impostorscore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2505280000 definitions=main-2507080117 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Stephen, When I replace rte_eth_tx_burst() with mbuf free bulk I do not see the tx r= ing fill up. I think this is valuable information. Also, perf analysis of= the tx thread shows common_ring_mp_enqueue and rte_atomic32_cmpset, where = I did not expect to see if I created all the Tx rings as SP and SC (and th= e workers and ack rings as well, essentially all the 16 rings). Perf report snippet: + 57.25% DPDK_TX_1 test [.] common_ring_mp_enqueue=20 + 25.51% DPDK_TX_1 test [.] rte_atomic32_cmpset=20 + 9.13% DPDK_TX_1 test [.] i40e_xmit_pkts=20 + 6.50% DPDK_TX_1 test [.] rte_pause=20 0.21% DPDK_TX_1 test [.] rte_mempool_ops_enqueue_bulk.= isra.0=20 0.20% DPDK_TX_1 test [.] dpdk_tx_thread = =20 The traffic load is constant 10 Gbps 84 bytes packets with no idles. The b= urst size of 512 is a desired burst of mbufs, however the tx thread will tr= ansmit what ever it can get from the Tx ring. I think if resolving why the perf analysis shows ring is MP when it has bee= n created as SP / SC should resolve this issue. Thanks, ed -----Original Message----- From: Stephen Hemminger =20 Sent: Tuesday, July 8, 2025 9:47 AM To: Lombardo, Ed Cc: Ivan Malov ; users Subject: Re: dpdk Tx falling short External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Tue, 8 Jul 2025 04:10:05 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > I ensured that in every pipeline stage that enqueue or dequeues mbufs it = uses the burst version, perf showed the repercussions of doing one mbuf deq= ueue and enqueue. > For the receive stage rte_eth_rx_burst() is used and Tx stage we use rte_= eth_tx_burst(). The burst size used in tx_thread for dequeue burst is 512 = Mbufs. You might try buffering like rte_eth_tx_buffer does. Need to add an additional mechanism to ensure that buffer gets flushed when= you detect idle period.