From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D0A4946B17 for ; Sun, 6 Jul 2025 19:45:04 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 779A94025F; Sun, 6 Jul 2025 19:45:04 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com [67.231.157.166]) by mails.dpdk.org (Postfix) with ESMTP id D34734021E for ; Sun, 6 Jul 2025 19:45:03 +0200 (CEST) Received: from pps.filterd (m0072399.ppops.net [127.0.0.1]) by mx0b-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 566GYgci020710; Sun, 6 Jul 2025 13:44:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=odoMPTnst6d/oSBW0/wuxKsfIEAs42atlzHh PYOQras=; b=Kh8paHSC5ZktxxCZbqLVYqNIcXM9WSohyeQzLwsw9q3nyZ9+oe1Z MbesEn1yjAapy8tpcZpeQgc5GTF/NmuaB1wJB3f/B8+NJAaovj2079bnX/b+sGa0 YTI0eN7hs8zi5NjGwkQCsmYh1lWj67TwhePnqPn/bVL1bXRI/SGNE0upBtnI/BMZ GndkgWwMjS+1YCu+a1F6HuUBKfiYG5L0tA2wMyNNXp1514oLwVPZMNY7rrmMzhfB 7RLZ9J2zzehOxnq+pAduaDyQFtTG+pHVxAmLSiQuthoQ2SUewx5vwe1NQhklz2oL WxihjJ9BrAZDbUux9yNTFYnF365tH9/uPg== Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2175.outbound.protection.outlook.com [104.47.55.175]) by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 47qvsk0226-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 06 Jul 2025 13:44:54 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=F0eqxnkVbtElGvDm8vPtPQbIhXPmybYGisc1FZeX9JR7qD1cQgXY4WX+J6DddVdtfeg1hJBzniQ0vf4ew+sBrDVUG6kPxF3ysyNIydLu9tby56rNvXNBQ1nvfyvH7Eu5oA1tSdya5gB2Wxny5EkGGE3xaLn6CkOr7ugeym39gNXU9tNCRGHHIlnBcG6brlnC4yEKOTop2zmL9PLfi06LPA9EGS9qZ/AK0AVD2KgFghKdy3Npai8BDX2qv+aqReqvjNboiLW22J1K9NNGdTfzQ7aB5lXBqU9iHXBY/w+IzGLFuR+nqrCfILl4BOFuXGuAXkZgLenrl9p2zQKMydqYlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=odoMPTnst6d/oSBW0/wuxKsfIEAs42atlzHhPYOQras=; b=JjSSr6NvSQtI+P/f/5NdD9pvyMkLBia3+lShNu/xntG8XOgOVjpSdQDkLDpAAHxUYNXw1h0mAtqMJJJ9kBOL69dPt4lf3nqk5oHBesverkRQUF/G85EIrOpSDO8J6uLUI5R+kRx0NQk9O5/akNTmlAHjf4MPyRXtkF4TlpO0LL40V2DP8R470yul/1J/ODbIBmpEzV3/vIdp6Lr+yEBGQuq00TfxlAbnUUCK6B5O9YdpmACLTyn0I2+DX+4BQDVQMRijObcqiYNZEd0afOkn+O+EZ/oDE/yXWpQuLFcCD7+N0ak/sh9JM4CZo+PFGwqcbCBnoBLU4tljw4wK2+7zIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by LV2PR01MB7718.prod.exchangelabs.com (2603:10b6:408:174::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.22; Sun, 6 Jul 2025 17:44:50 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%3]) with mapi id 15.20.8901.021; Sun, 6 Jul 2025 17:44:49 +0000 From: "Lombardo, Ed" To: Stephen Hemminger CC: Ivan Malov , users Subject: RE: dpdk Tx falling short Thread-Topic: dpdk Tx falling short Thread-Index: AdvsS4XiFhfr9tUWTZG/9hrfPmeo0QAjYaoAAAZ3EoAAOAaP4AADTEIAAAkqa1AAIqDrAAADVbOw Date: Sun, 6 Jul 2025 17:44:49 +0000 Message-ID: References: <20250704074957.5848175a@hermes.local> <20250705120834.78849e56@hermes.local> <20250706090232.635bd36e@hermes.local> In-Reply-To: <20250706090232.635bd36e@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|LV2PR01MB7718:EE_ x-ms-office365-filtering-correlation-id: 9a13ab0e-846c-4f0b-b1ba-08ddbcb4cdfd x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|376014|1800799024|366016|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?TOfebuIu8liJPpuZydEJnLpbE7ZJv215zLv5gZ6iQF59HYe3Z3Ci4YzMxXwP?= =?us-ascii?Q?pBTxIW16eCClvdMOvJHtIs8L556izvNGFA/Xa8dmyLPLvQyo9FoT9Wk2T+c5?= =?us-ascii?Q?d+n5uKJbND47mz5c0bE2TWmH18qmQcwZGBkxIORqTAzfHQ7jQ+Ds6yCcSBRg?= =?us-ascii?Q?urMlhRpyvzzvIH5OLXKmCTqJMrQh96NP72fglYRmb9+zdeKew+PpWnPHKwah?= =?us-ascii?Q?t/kaoTXjjLTEyOmxE1yAAtZvTOriCQc+nKgZ0X0iMjEsg6laIFdrrxy1FaRs?= =?us-ascii?Q?oXH2zxcRib6aT+PlFEc0iqmTXa15Pl2oM8mTOHzq2VWUUEx/0rfec2KooxPa?= =?us-ascii?Q?Fhd3z0F7WS7jnDcA+mmcsAgHnhjTwajv/aYvvjCMllEWaB1eXmCqspc5ZbHx?= =?us-ascii?Q?2LL+yOy7rmaCU98pwv7AQgGKbJhICawz5qpEXFaoVVFcuwgkP1TL0zCULvx+?= =?us-ascii?Q?6w2nNsLkiKvgajEzmwP0iUZtp6iruq5nqC7dm1vOXlJWrzAaBET28w6plH2u?= =?us-ascii?Q?19Jeire3Z0JaP8dGJsXmZd1C7vpnsRxgVyTYBIO8gEaYkOXl34+vpKh6WzEx?= =?us-ascii?Q?o/72U2ZrrcErCS+4siWxsyRq1PBIle92/anTFcEtCrgUXVm3mPqrkxyxcbhW?= =?us-ascii?Q?Lic6qaHFatooYP1vF0e4t0UPElT73QDUy1rPHj926XSFq6/rn8uR5PYEtadu?= =?us-ascii?Q?s4c5Un2sP4lwmVtat2MJ8KhvEUIbQtqKiUSSzU4thdnbnyeQ5lg1utb9uuzO?= =?us-ascii?Q?gtLHfqiOoJK0oM+223ZCCp18PL4aIaFfNSrx5hHzCMMgU+UQbRVdSLcnlIIb?= =?us-ascii?Q?ubUfue+U+ALmYDG6qnERzta+w6lmNGXDAk/q+ZuiHlTVunkDwy+L3/B+9LSQ?= =?us-ascii?Q?wP3tlmPVyLBIofGmWa1eSD1A605/iXZOX61l8DD3rEUguCT3a43Gu7kftgFN?= =?us-ascii?Q?ELjV1C8iKBJqrshdIg/e4jGpAWDFkMGCbgSd/UT7FvivA9Yl70MjbG2Mg0e8?= =?us-ascii?Q?+k87nVfjFceL9aJKl4BKCNz9wz/E3RScDaY4ftWR0tS2Bvr6sKxkkCWl9WhT?= =?us-ascii?Q?NKK/1eNO3hsCiPoI3pxvnjGCwNCib5o4kNrml8Op2Qp9mwHAcZE0vBuoC7/c?= =?us-ascii?Q?G6OvpTl4/udzko3lu8ZOydP2Ym+tKaUAhrAt3Tp6nC/rVGNbea5cNB5lIX4H?= =?us-ascii?Q?Kg2tUfU9Zk/WEBHR86fuFQZBPDrturhbVWOz4a39cp6xW/AtrQJ9COcjcU02?= =?us-ascii?Q?PtoEIqgS2ljgPggw5NqdVaG+8DC84ruG2xsIKjx0U+xVu9nILN1FcVMUKyAJ?= =?us-ascii?Q?iKiH3q0PUf1MMlENK/dGYIpSIYekWCJdOzQ1YtCh3dXVT9IhIXffzEzCOkB5?= =?us-ascii?Q?kG0MzOtuHeCWSpgKZ9HYpOTDvMazwma+gNkNbA5zNBgFtO2J2Z2p6qAufx1H?= =?us-ascii?Q?sTz7b69t597KzLUHVS6G7AleWZeULasqHwXU0a5/R5fcDpjPM0+p8A=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(1800799024)(366016)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?iUxB/F+2jo3fhUW3nVryAldUMnb9ks0UGHgdRjiApLIrcahtjF041T3A9N45?= =?us-ascii?Q?dyUDhOQXcn6EhI74OS+aNjMcuzzNT9yax9kuPKYpGXpSol0ELIyM0k7pJUI1?= =?us-ascii?Q?hwCzA/JykSDDpzAmvbyooqy3zBgUssTcSvhF+lbO4Kf5BcbjFiGXQ2V3fndx?= =?us-ascii?Q?cqIeZP7EkXqFCLH/NbkGHZFenWDezPxke73Z7P/YgzEJALdOqMBAea+BUhFp?= =?us-ascii?Q?EL1hdgJLhJaJ2P2i1CFf0RAFNI+p9ZqX8cOISIoVfDrUzhoNOaDaIZltPyix?= =?us-ascii?Q?suurjLuZyjTpzxIu8FpLSR2PoYO4+RqpKpb1vurRKJXxQ/YLNeYDurfr6x+R?= =?us-ascii?Q?iwd60oN0slbvuSEJplkOVHfmJT/ZMvdqTbs9RFEd1GiVDAi1f7fW0uRLaelp?= =?us-ascii?Q?ONbc6M0tRcIR/HgJOrDsKgKASti0YNVT9f+4PMoiJK05o0GhpWPyU0ojnOj4?= =?us-ascii?Q?2lg/Fu4/d0thE3m+YsMuaUxJXSCjn/tw841Et24wYr5+S23tRvzFWfQNRgw9?= =?us-ascii?Q?a/Sz8P7NSjlmS/QTpCxTWVyKqWHMJh6nAjouLWJks0ihpVOdNj0Z+kHZoDgV?= =?us-ascii?Q?8YNlvoN/QTpX3xGSDA/ltk3V1bNxWLbTpqcdS3kXIBYrY5CQ2lnZishOuArD?= =?us-ascii?Q?xowSxjT3M060Bx18qRU5+KnZeOfvvDlj59N7QUT98iNYvswSUDGHOJTGkA6r?= =?us-ascii?Q?87hD/3Qiu2jZrSRI3RGzo+tr1z3v7N75GIvx2LWnb3IwqE0tLjPdJhJFqb8u?= =?us-ascii?Q?dU1n8uSk6DlzC3QYve3gin49FHhlKnejSmvaeAMR5Jn0x4rCICNbXY/QT/LI?= =?us-ascii?Q?l3DCjIB7SAC4r9ZoQIJA8GYZBHJtAHmnXzFFyItvCT0CkMIhhHB1GgI0+P/W?= =?us-ascii?Q?ov5HHERtemK9JbaFqfIEvgd5BJx+3Z57v0ZopLjghkcjo3+fn8nrZ/+tl4NJ?= =?us-ascii?Q?u9+fNWyIFj8jw5iDhFHwtNeA38Vs7xMmFBfdBFOQIHuWTSmBuZzqduZGTaYp?= =?us-ascii?Q?SNE+zSSYjqE9F7Egp0yI7Uy+f4+lmFWzF3VZRao1IMOfASIkTUld5dAMS5Ms?= =?us-ascii?Q?3dEd4CsO0qanihXt2VlATlpfIklQjcYL66//oSEqPWCpIsSWr+i/nYNpRGTj?= =?us-ascii?Q?uDS9lyh/5BT/cQABw9wfCUR2LOl8UvZkpczvcfaIkjlrsHJejJIbX8hJeZiR?= =?us-ascii?Q?9gB45V9PTla02P3cLZ4cPqJWVrVQqldWlHi0VQ8BvAIgXwZTUUfPRCeiX33H?= =?us-ascii?Q?ISYxpL7s8tnPONKSZNEw+B+IqhvbGqNvmpiQzzjJqK2zZz4Z9fvi2fsD/jh2?= =?us-ascii?Q?JORzPP0ER5jYojE1eJDAWW5+8RmYGFGauRJZvoMvgcK0An7i+3oMvElWq3jU?= =?us-ascii?Q?NHeZO1w/rTyHZ/zklRdaxSfXbNqnmgwnzU0I96JGQqo5YTfP3KWshuaIkEBi?= =?us-ascii?Q?Q7GBUuk9uI0jUuevJn2qDSKZE9WrkinHKgGdS1KT+f4prvmvNNTLej9zCyC4?= =?us-ascii?Q?+yzCyDOCFh/cQPVXC+XtwBNnwMiyXqzZkiqysnnMKkiGyd9iemI0kpWOaYM3?= =?us-ascii?Q?EEKWlWQPVUpD/crCHbxWc6Mpdbz+G0UC8UnZYzkL?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: OYWZdytG7N1r6FxtLsVTV6quM/pmEhfHSPqVEUUg4IvnSimEqcw+1mwpRebbcX55TpbD6vroNsjoNtZ9W1xzrawZ3bdPMovDVHuJzKeeEFOzZOCYSAlKYRkE7Hh135cW24UvKNYMPq5AwUyls5BJXGvlg1yZhIFYOgXw62sT+3f67NCvDYLwwDi/e56UZ3ySP626uuVqtpCImX6UqcJiOiC0sfb66oj0rh39yZhQ21xGG37dd8MCHI+x97ycSLe6dk4m9d/PkCEEj/0orvmqpuHb9lAHvZ0xdAG49ApyamhTP9hSyjLro9cRvJO7rEafPZ4MrTju2l2nqE7UBtQZHAvGnQqnIYK9gVCSfCXg6dgjBklcr5MMCaOuiXR2Hhm9vQEpUceEUTGEHJ67+GqL5Erk/JR7thW1/XyMm30P75Ls1yNICXWxlde/Vs6+4OkyejA0P/f1H6uHsbfEmPqO4C9vZY0XCPXR+daHYBAfWZ7E2JDCBU8X2okPEFUqrytfGzGq8ED/MT9HSAX0qPcn0ZcvtkWyLdTSS0lMKYHsaQx0cq2nWdMmSyWLtf6Jyz6BKXX+uBa6HJQ0sg4EBvk8ln+qZXBmtjmWX/8CFs6lRwgLYoYCIFJa16TgssLm9Pqu X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9a13ab0e-846c-4f0b-b1ba-08ddbcb4cdfd X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Jul 2025 17:44:49.1966 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 3vX9WyVCHL0zp1jl7kDKZN6oI7hcQ7BgToy/jelTyC2psGxZIhJqChqOUQV4+67YDujBqY0lR2zPNb2W5tLP8Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7718 X-Proofpoint-ORIG-GUID: rGpNvdvVVEnPUiP1BjnGXjN-1S1Jl-eY X-Proofpoint-GUID: rGpNvdvVVEnPUiP1BjnGXjN-1S1Jl-eY X-Authority-Analysis: v=2.4 cv=fbGty1QF c=1 sm=1 tr=0 ts=686ab616 cx=c_pps a=F+2k2gSOfOtDHduSTNWrfg==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Wb1JkmetP80A:10 a=jZVsG21pAAAA:8 a=jQOgFn-ZAAAA:8 a=8rWy6zfcAAAA:8 a=h9dsk8424Rh_5D8AfQUA:9 a=CjuIK1q_8ugA:10 a=3Sh2lD0sZASs_lUdrUhf:22 a=mT82qxFQzDvLIExZS32s:22 a=YjdVzJdQTyZRADMV7wFX:22 cc=ntf X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNzA2MDExMyBTYWx0ZWRfX6tuw6kgw0BtD o+zzRAx344jM93g6idoCaWsoiozl6wB/NtRrUEMkpgEsbUaufy+qHhnIQbGyDjeo5e1Qg0Xpbyn Ys3bUyUaKXWQfKeNy+aHO2VvoJEiLK2Xr30cJqD0pCAul8pqP2pQANSRo84dyG7sw8lvtcA+3wm o/9emTi/m9RThYC2HbGZh1RKS/KxEr9sZmrAYGPwodeHfVRV8mQJBThjI0wpIHWtj5GIMLQ7iAa kbABY94fHOYntIWq/0D+n9ERDOSdkDFb629ae0SY6LqudrmMumcduvQi3gjQMD1T1yzBGVJBE+z kypyfN2c/knYGZkMxHPs/C3F93x1SqaU1II4FeJP1swdmKqea43cVQka801XfRZCsj//S5GF8PV jUVwDqWxjQQQxXv23g0W3thfCLh0m0CHyUx59jZsEn3AGAJuQo1rVnyuEs8RycG5Z5HqutY7 X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 mlxlogscore=766 phishscore=0 lowpriorityscore=0 priorityscore=1501 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 clxscore=1015 malwarescore=0 adultscore=0 impostorscore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2505280000 definitions=main-2507060113 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Stephen, If using dpdk rings comes with this penalty then what should I use, is ther= e an alterative to rings. We do not want to use shared memory and do buffe= r copies? Thanks, Ed -----Original Message----- From: Stephen Hemminger =20 Sent: Sunday, July 6, 2025 12:03 PM To: Lombardo, Ed Cc: Ivan Malov ; users Subject: Re: dpdk Tx falling short External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Sun, 6 Jul 2025 00:03:16 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > Here are comments to the list of obvious causes of cache misses you menti= ond. >=20 > Obvious cache misses. > - passing packets to worker with ring - we use lots of rings to pass mbu= f pointers. If I skip the rte_eth_tx_burst() and just free mbuf bulk, the = tx ring does not fill up. > - using spinlocks (cost 16ns) - The driver does not use spinlocks, othe= r than what dpdk uses. > - fetching TSC - We don't do this, we let Rx offload timestamp packets. > - syscalls? - No syscalls are done in our driver fast path. >=20 > You mention "passing packets to worker with ring", do you mean using ring= s to pass mbuf pointers causes cache misses and should be avoided? Rings do cause data to be modified by one core and examined by another so t= hey are a cache miss.