From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 359AD46AD9 for ; Mon, 7 Jul 2025 23:00:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D1EBA40287; Mon, 7 Jul 2025 23:00:17 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0a-00196b01.pphosted.com [67.231.149.170]) by mails.dpdk.org (Postfix) with ESMTP id 9F15A4025D for ; Mon, 7 Jul 2025 23:00:15 +0200 (CEST) Received: from pps.filterd (m0072398.ppops.net [127.0.0.1]) by mx0a-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 567JZcVq004439; Mon, 7 Jul 2025 17:00:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=IJtI0ONlTrq0PLe9JXhwZRqxLmVT2ehUuBBg mo4nx3U=; b=ZJdNCSJ7KAvxD0gxNNirPBEsN4R0c69RYHAYQNOGYXlcHpbpDonT sJSDmiw9dRlaFELtttgPnOFyc9zkq2PKGUTrtzW45/KJkSVpnVyYSC/dnUCPgjSk wJkk6Em2CcYR3Q9i54YKr7xkvOBKei6H2UJ7IPLvCgqSboyqwlVYTZ294IWpmu7J dbfsMYKJZc8Bt7TFzCaxjO4ZdI+C3UUDXtL7XZiHYmvZ+6QsW8NGrteIspLVSo6S qkp0vq4ug5XoenHcCx4Y5G5vB23QM33rY8Ym/eL9WkKRZmwAyYx+DjLyPJ5FjBe0 beUbDZ7CH88ysaaU5PTel+NCl0znJUVAvA== Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2041.outbound.protection.outlook.com [104.47.58.41]) by mx0a-00196b01.pphosted.com (PPS) with ESMTPS id 47rfbkrfr2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 07 Jul 2025 17:00:12 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Pi8LnObJ4SA6gGDQOY5iSzy2ahjrrlxLnulAvB01wncF00wBEayiubT8TBQwGlMZho5W6jGM6kcOQhfcEureNKSuK/roMvCy5uvaYJYkEiOmkzHErq4ubXo1x+3Ah2sXRE8coC6oYbK/Xn+GA4Z51M0vx4ZmUqZ5wuJJ6tYkuRmnNmdhEP+rXrxAC2IAPFA/gUg99SlZC4OvaDMr+Um55bn1Pdpb10l4c1doIjTe2gQ2VwP76sGl8JpOvL+mi+2KzoDtiGCBebJKRGuk51a4UvFFGjEdZ5JYBphW7s4KHIv0TymhzSPJQ+l10QSc+9EK8BgYV1RUUyrSDXDcYVSl0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IJtI0ONlTrq0PLe9JXhwZRqxLmVT2ehUuBBgmo4nx3U=; b=rHlgvs+7Enyl8wz6iITrDGw/NCYjBDFmqBkshvtQyZIl1VUOt8KUF7Q48iqiS28127gtm77YPRD82fP+ScWb2QuHWjROIZZmhVoVbKIo5jYVCRMu8HiBYLBtpHGgyPbc+/yzuuL1GU7eVhtKhPXp8PopZ1MNKHqLS/qKkbvMhr9UrQnCdHslf9tLj1gkki53B1TOb0xN5WGIhUahAVTb7G2hswvT9JZ7Rm733IwwTdoZt98J0MQPYiUndoINdEMUbuleFjMQh6nDWgGWVWKRvJaP93/S6xUIOaQfD8+CmDQxZv5dHTMhxbQjVz3CCMXZIalifUrCX65I8TqO1ThuNA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by BL3PR01MB7057.prod.exchangelabs.com (2603:10b6:208:35c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.26; Mon, 7 Jul 2025 21:00:10 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%3]) with mapi id 15.20.8901.021; Mon, 7 Jul 2025 21:00:09 +0000 From: "Lombardo, Ed" To: Stephen Hemminger CC: Ivan Malov , users Subject: RE: dpdk Tx falling short Thread-Topic: dpdk Tx falling short Thread-Index: AdvsS4XiFhfr9tUWTZG/9hrfPmeo0QAjYaoAAAZ3EoAAOAaP4AADTEIAAAkqa1AAIqDrAAADVbOwAC9nmnAACXQlYA== Date: Mon, 7 Jul 2025 21:00:09 +0000 Message-ID: References: <20250704074957.5848175a@hermes.local> <20250705120834.78849e56@hermes.local> <20250706090232.635bd36e@hermes.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|BL3PR01MB7057:EE_ x-ms-office365-filtering-correlation-id: 77bf088c-9de1-4d2a-bf9b-08ddbd994220 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|376014|1800799024|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?3+fdSl6NyHbNnZ9aOyO6bAwBSYcaYjrw8cmxgrxi0y7x4xPE5rBpAGF6/uEM?= =?us-ascii?Q?+DrOEgwmaEp3JhfuHCAVNVVIVo5tW0aLBW40+LaLrYOYF+hGsluDaykS+LTJ?= =?us-ascii?Q?+sjg82bfBHj1lJSwL+eV4o7DHCvWC7CN/e+Q1aD603Y64bvzwJGfiu2btaEA?= =?us-ascii?Q?lFISwdqfisU8i3+O8nKwsngNJytAU+FREEquCjyfFuebh1goiZ0naj70qkGn?= =?us-ascii?Q?TStMfK+gflamYiaeX0IaUBu18RcGP8uIOmAfdJRCEYJ2A8zlhhNzqEbPqS4E?= =?us-ascii?Q?kPS1HvOkdSJarcNgyXrwh++qWToNDiyiy7ntzg5zA7BM3Lu76ePNJN7DfKnV?= =?us-ascii?Q?xUneqjBrE1L1dWGGkfbEbtWziW+SCwN6gMkQPddG/FFS7t+o8H7bC3V4UnkS?= =?us-ascii?Q?hach7BGjw+YczgnpSV++UkMUlxREybVr7qPXBJGM893RX89ZZJyXHE0vNxgI?= =?us-ascii?Q?ab9BJxMnvdTLtctIXw/iyX5wL9/fU80526O/JC+EAhvK41mscqP/CotkNqgD?= =?us-ascii?Q?ARNm/4YAt8BbzxMNLtL1gaz/mhi7pnNx6OjkPea3boHEP0Q3SQYmldY4unMv?= =?us-ascii?Q?GzcigE30yd7S3wrjpKLbkOJ9jYRWJBHx9O36Bp5Of4tASphmJTXZi83TM43n?= =?us-ascii?Q?MEzxZEwJI81IbTZVduRgZ6N+VGLMLEUJiPIW7NuDyR32cYDgKx4PEOFQKgad?= =?us-ascii?Q?OnOFU4LJTzTcSBFweZ/4QW0ckQ3MSD+Ly4xejPKknadgPm/8Eedpaky+5O4W?= =?us-ascii?Q?7uRCQX2Sxxa4T33ClEe+ZkgoMN06Vq2rwKhgJtcfqyLZqaQ7sYtlmgIfLRsN?= =?us-ascii?Q?kNP/c3sEu6ZzX/W5wYrIsaSxGiiZAG+Bb9pUHjnLxe1Fvq7vL+enOZrcZ2+w?= =?us-ascii?Q?YNW+ly5A0fhUQ2uIzPlKX3aU7XjygdvZl4tynUzlR/HsERdXYYtGoYvzSfob?= =?us-ascii?Q?RfHTImwicIMTmjd5inrqmQaKaRp/w+G54GVemjJD3yF61Fn/ntqv74q1eca4?= =?us-ascii?Q?u1aCtjujAvB4woonRHaddv/jx81vMZBOlySQQIQekC3yib26AU7tpRZ+4KWT?= =?us-ascii?Q?AbwMpdHAvSGVwiONDBzPFpZpLmE2ptI3hjAP6Oxm/BWjfQZq+D5WANRC2z+F?= =?us-ascii?Q?lLyrXlhIQ0oQEOym0snKnkoA3b/teNxV3F28cNrtSj5GIyiBLlYR9jGb/iL7?= =?us-ascii?Q?DVeF9mjyUYfJfLyR669i5sXPr+PWxWSaHZ1j9peRgLY1+/dr9df4mtLDsfzf?= =?us-ascii?Q?DjA+AjpN1QZL0s8DrN2MunyZ+7Bts497FCSGLkXCg+AU0CPYt/mhK42W1OoE?= =?us-ascii?Q?FhuUk2Hj4bPwJmSO91JTYE/uvyH6s9rQzYXDga6rriQWxBuwT2EdVKzTUT8/?= =?us-ascii?Q?FunFHfV8byVUzlpc84Tioa6K0cc/wgFjNF349vl+rBfEBT2dAAqZd70/v4Ev?= =?us-ascii?Q?Ebc/CeQELdjKeR6av5qSLB7SaIVpt5zjarJ1KkP4uMxzkK1fdNmaFw=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(376014)(1800799024)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?k2+BXfiywYPZ8f/TTdIRFLAB5o1WrakmUJdrRu3xX0/Dg3HHEFh8QWMM7ZBc?= =?us-ascii?Q?9FqpYk4I3/xiFHrDKO2Ivd1hWk47fdpymmJyQri+ty3ngnWHXgNA7nrRUChe?= =?us-ascii?Q?ldrZ4GYXWHyHGrulPV9KOTw+hxaLDO+rfgAztr3EvLXJmYdbWq9LaJJxlbVV?= =?us-ascii?Q?zkB6z7ZxbDAO7bIHfkbnhaeqLi0gQPyovD/gQwI14NQSq+MGWvOrUM+H0XeF?= =?us-ascii?Q?PCNHmw1xkBIiXCtQqcJ6c9RfmLdqZ4OerYcEUKCw+cYnT8UIClNGy2qqyBdi?= =?us-ascii?Q?8mkjEOTVa+ItkQ3Z2NpCCkJoxfepsTzwQkXJGPLolRtuH4worIamcC/9kQNw?= =?us-ascii?Q?8gYBOQj80OIb4IvIO3s/UBBZ5gWZtk1DTgjBrZOgiXIt4f+8V5dUHTTpYrZA?= =?us-ascii?Q?wG/5wreg9WQQXSRI3wCAofNIDKGRA0jesIptE2uZsmO1h8NR8OU4PGm0jJSY?= =?us-ascii?Q?vLJtGkA7Dsm6b2V7FiVN/VAwyz1hS2HS7+29Vvt0PuoGw5UQsPSGp53jxii8?= =?us-ascii?Q?+6oj1wfH7UIlZ3Xfwx0bNPbwwg3PhmwbwJwNpIlXKunCg28nbWXIAh/nyoOi?= =?us-ascii?Q?bxM6Y2/k4KUsTrFcakyI+LDvvXmSdMeBdQscOt/lkL+r3YT7EEIXKzumYpL6?= =?us-ascii?Q?QCjvi7PtKi86k2q2XAKFVdmGPf8Qp+WKviaCnt9fIAdGsK51LKPlFq056pto?= =?us-ascii?Q?JSWqCmg1AzU/Ftn6u6DmMWpKTy3hqj5AqUvfaRhacAkG04wOOQcR7v7Fm/7c?= =?us-ascii?Q?KifLbzwatmffjNKZ0C8THutQnzdV8YVMBMIaesMqPGGwgkOPdbxFLNJZPWMp?= =?us-ascii?Q?9eob8dUJhodI+3Lbu/7Ahwwph0bmfepiGcWocvGf2fE2aP4+7yb1dbiCbhEg?= =?us-ascii?Q?zcQygyr+sGVuZ2MlGMSW5ZpB61CTWdImtEJ5z22kMF2+KCNFE4jl3jzo4o3f?= =?us-ascii?Q?awMqBBVAq1zmGoPph3mFpm7JkT7qv4FfAOgJ+/bzSR4wNRzb7pPHwy5wc4he?= =?us-ascii?Q?LhDJrCUusDqI0Zh7o+lahSI15zrdP4D4sJVsqDiKl1dsBjFx1K/s8GAtF0yK?= =?us-ascii?Q?bzB1jGwyxM6f8uyS6fFA0GVXvTV/ujEBl2O+JHou8682Rif7/n+kzhySIDsM?= =?us-ascii?Q?8M/p/uhHV0EabwbrDb1MNubnx+qfRg7cNYqRBxTI+UoWTbERqTmWAgXlW2cw?= =?us-ascii?Q?maRKaotfoSq8P4+K2eNbW/mgefS1Q2Q5juRTFFiU+oKtpmu9k9DndPRNeGOM?= =?us-ascii?Q?JrZA5RvBxC7CtcdPzrwP1pxuvm9Hf7Bww3BSv4YeoN40xQzp08aOlfyMjtPo?= =?us-ascii?Q?oQ1joRJOEX+/i1u8KTcNIOheQpuxjuO83STIpOtwNwIuh4nosrl0Vr5nGAhv?= =?us-ascii?Q?UIN4c1M5u8+POh4ADdkYTLEM28GTv42rMwB78sPUh/Vfmn2TlQqevurfEk43?= =?us-ascii?Q?WoOgai6e85/cHuiuLx2cYNnIl8JEgPYrgiAwUHJYnRdQvzQZ4vDBfMNOIUh7?= =?us-ascii?Q?6U8cSOqZGBTTUkQTPtJTz8+88J+wF1HNAfUWGyrz1nJzaxnhr9kRc+enarEA?= =?us-ascii?Q?GoFxoEFSCmrH442mDrzFGvbMAcJ+DIJCk++6lvT0?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 51s1fgv/TOUiKZjpEhyIj0avN/ICTJIhGRTKkDYuxMp/aNV0A5G4HBQF+H8Um49f3l8ez49uuyi6xcyA/DgleDmoCzGPEsuGH6OxBEqTQ76M/5DZhN+61H1nIVpbDUvm7aRfeGO3xkbcd6hWDCn13BdY1w2cjXTBGFtWx1pd+xQ5ZpDwXl08UHrS9BxZam0CEA7/BijN/EEBkGANelN/AdrJMdtFnsYfkSCcmq9m6I1/80jfC42fFm+VUpN0rsInQ4gXrf1Lj+ZNcGGU7otLWueQPJiQhIZdV7OxdwVlv2zlkG9YncZfozoYVCzEZMuhc4qPs/1+GBa2HUy0M0kxSEBzV2UwRrMw1+ujVXZUh899gWfq7zZvkZpy4k1kiqZBPkKNP0hnuxx0p3fLCmcIuMvz0Esds8m78SbmJHFCMK5IVPhc0niS7eFOsQ0pN4WRDJTOnbnezPqKmkFNoO+OLTMQE3lc208ziKRkvPkO/JyXJ1iw40iWdzkrf8QCxH1Hi7WeAjkrNfpnoc9nLuloE4VCQ3qU5IqIcE5HBw4M+H+K1AQDWn5ZQBL5o6ChHcigk+7hLCuQ/uwAzOFGvlo0bjURnCBoFhbUMhjcw8nyVcmVt5+PylhP9IBg6DpZzomY X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 77bf088c-9de1-4d2a-bf9b-08ddbd994220 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Jul 2025 21:00:09.3319 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: qqIvrPDELumFcwjr9E7NbgjMnqVBATh5vOzdGMHxVA8OJScHcrdh8FaD1HPeq71+KP3L5udw5rHNqJwm7G3ZhA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR01MB7057 X-Proofpoint-ORIG-GUID: AOH8Kh20UAD9mfraT2pRI8ouQ8bML1_N X-Authority-Analysis: v=2.4 cv=LPFmQIW9 c=1 sm=1 tr=0 ts=686c355c cx=c_pps a=OnljjeCONrlUuPUItWmgXA==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Wb1JkmetP80A:10 a=jZVsG21pAAAA:8 a=8rWy6zfcAAAA:8 a=jQOgFn-ZAAAA:8 a=Ou4HicH6dnXl745DGx8A:9 a=CjuIK1q_8ugA:10 a=3Sh2lD0sZASs_lUdrUhf:22 a=YjdVzJdQTyZRADMV7wFX:22 a=mT82qxFQzDvLIExZS32s:22 cc=ntf X-Proofpoint-GUID: AOH8Kh20UAD9mfraT2pRI8ouQ8bML1_N X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNzA3MDE0MSBTYWx0ZWRfX1T5fZRDhnn8f 0CLngDV58CDmUCgIJqrPzaPks1blgloHsY8gHgv+mQzPFJp/4cPo8nhmC/nW3lm73NJTzVOMtlh vkg+HaYe0xaQnwab/y8WKHa7BYOAWlIMsEH5Ev2hic5mgSj6LqWsrIscLLjyPyCUMuIhHDuTIXg OlbEFqKxslp+H5Q7QjbjQS3+MBuwXo4r3CbX+F5C+RYb7xZVt0e4iPKbvyhiJ7jnwImBFQLmzCM bIc53mV23hTyWXkzEaUlbnY1ZpL/LXI+xO0qez7lkDLtYpqtAJDB/mtBMqZh/SuVo/JvqMIgS4v SqpjfhuWjIpyDBrj3v7qCw3jrY4zMET54f328cL8GE/vJ5vPcJu7hpGaHV6Hn5c22VT5RrXBwZJ hDKna2Zk85NP2M9J5AuuYYZFQp1iHHa6eSqLyB9iBiVjpOgvkYeV5LtvCEnqfn4tWsRGZ2Rj X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 phishscore=0 impostorscore=0 adultscore=0 clxscore=1015 mlxscore=0 suspectscore=0 bulkscore=0 mlxlogscore=999 spamscore=0 lowpriorityscore=0 malwarescore=0 priorityscore=1501 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2505280000 definitions=main-2507070141 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Stephen, I created debug dpdk libs and built our application. I ensured all 16 rings in the driver are SP / SC created and at runtime the= ring flags have value of 3 confirming they are. The perf output continues to show the symbol common_ring_mp_enqueue() and r= te_atomic32_cmpset(). My understanding is in SP / SC rings they don't util= ize: 1. atomic operation 2. spin-wait 3. rte_wait_until_equal_32() Thus, should be lower overhead, no cache-line bouncing, no memory stalls an= d no spinning from 3. Perf report snippit: + 57.25% DPDK_TX_1 test [.] common_ring_mp_enqueue=20 + 25.51% DPDK_TX_1 test [.] rte_atomic32_cmpset=20 + 9.13% DPDK_TX_1 test [.] i40e_xmit_pkts=20 + 6.50% DPDK_TX_1 test [.] rte_pause=20 0.21% DPDK_TX_1 test [.] rte_mempool_ops_enqueue_bulk.= isra.0=20 0.20% DPDK_TX_1 test [.] dpdk_tx_thread = =20 Any suggestions is very appreciated. Thanks, Ed -----Original Message----- From: Lombardo, Ed=20 Sent: Monday, July 7, 2025 12:27 PM To: Stephen Hemminger Cc: Ivan Malov ; users Subject: RE: dpdk Tx falling short Hi Stephen, I ran a perf diff on two perf records and reveals the real problem with the= tx thread in transmitting packets. The comparison is traffic received on ifn3 and transmit ifn4 to traffic rec= eived on ifn3, ifn5 and transmit on ifn4, ifn6. When transmit packets on one port the performance is better, however when t= ransmit on two ports the performance across the two drops dramatically. There is increase of 55.29% of the CPU spent in common_ring_mp_enqueue and = 54.18% less time in i40e_xmit_pkts (was E810 tried x710). The common_ring_mp_enqueue is multi-producer, is the enqueue of mbuf point= ers passed in to rte_eth_tx_burst() have to be multi-producer? Is there a way to change dpdk to use single-producer? # Event 'cycles' # # Baseline Delta Abs Shared Object Symbol # ........ ......... ................. .................................= ..... # 36.37% +55.29% test [.] common_ring_mp_enque= ue 62.36% -54.18% test [.] i40e_xmit_pkts 1.10% -0.94% test [.] dpdk_tx_thread 0.01% -0.01% [kernel.kallsyms] [k] native_sched_clock +0.00% [kernel.kallsyms] [k] fill_pmd +0.00% [kernel.kallsyms] [k] perf_sample_event_too= k 0.00% +0.00% [kernel.kallsyms] [k] __flush_smp_call_function_q= ueue 0.02% [kernel.kallsyms] [k] __intel_pmu_enable_a= ll.constprop.0 0.02% [kernel.kallsyms] [k] native_irq_return_ir= et 0.02% [kernel.kallsyms] [k] native_tss_update_io= _bitmap 0.01% [kernel.kallsyms] [k] ktime_get 0.01% [kernel.kallsyms] [k] perf_adjust_freq_unt= hr_context 0.01% [kernel.kallsyms] [k] __update_blocked_fai= r 0.01% [kernel.kallsyms] [k] perf_adjust_freq_unt= hr_events Thanks, Ed -----Original Message----- From: Lombardo, Ed=20 Sent: Sunday, July 6, 2025 1:45 PM To: Stephen Hemminger Cc: Ivan Malov ; users Subject: RE: dpdk Tx falling short Hi Stephen, If using dpdk rings comes with this penalty then what should I use, is ther= e an alterative to rings. We do not want to use shared memory and do buffe= r copies? Thanks, Ed -----Original Message----- From: Stephen Hemminger =20 Sent: Sunday, July 6, 2025 12:03 PM To: Lombardo, Ed Cc: Ivan Malov ; users Subject: Re: dpdk Tx falling short External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Sun, 6 Jul 2025 00:03:16 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > Here are comments to the list of obvious causes of cache misses you menti= ond. >=20 > Obvious cache misses. > - passing packets to worker with ring - we use lots of rings to pass mbu= f pointers. If I skip the rte_eth_tx_burst() and just free mbuf bulk, the = tx ring does not fill up. > - using spinlocks (cost 16ns) - The driver does not use spinlocks, othe= r than what dpdk uses. > - fetching TSC - We don't do this, we let Rx offload timestamp packets. > - syscalls? - No syscalls are done in our driver fast path. >=20 > You mention "passing packets to worker with ring", do you mean using ring= s to pass mbuf pointers causes cache misses and should be avoided? Rings do cause data to be modified by one core and examined by another so t= hey are a cache miss.