From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 906A746AD2 for ; Sun, 6 Jul 2025 02:03:27 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 268D640285; Sun, 6 Jul 2025 02:03:27 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com [67.231.157.166]) by mails.dpdk.org (Postfix) with ESMTP id 4FB494028B for ; Sun, 6 Jul 2025 02:03:25 +0200 (CEST) Received: from pps.filterd (m0072399.ppops.net [127.0.0.1]) by mx0b-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 565JCCHm016363; Sat, 5 Jul 2025 20:03:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=BmRZcKtxE8OI9GxQgwqabqgY9mjbb6u7NN7j Qa2ziM8=; b=viCYyb5lSzbA/+2FCxRiMq52kNcLsdSxhBEcRVg6XuC9tF2PYhcg C5wmpb0O3hwT3sUQbKqRaPubbt2Xc6tn1N+n6ynytHmCwslRE15Qnh63IAgSDyck 4mHgiQAFfon8U1i9GyqjKitOvT7EPz3q6AoWeHxOn+SJi85q6035Um11SVXb391r Lzi8wI9T4WyhK3ocMS3u7MJyev/1sKLUaum3Yx6n7SlTjNxfHCLLjQOXY7Gc4Dys grDPB8gmAJKj0CYI5/BJyh1ycwRAbcLBME12gKLIeFt3bAMbsw2z0+VsCQSfw6r/ LPCRHmJOCCNOWUXXF04lfMaEluRT9jgIYQ== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2047.outbound.protection.outlook.com [104.47.70.47]) by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 47pw9xrmtq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 05 Jul 2025 20:03:22 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XzxW+V/OWYCdL2euYWedmaBsskPjW+bpS2ximlgsp+o51ABllq6jxrli2TLodqQi/HzLovq8Zr+J0+76IXdO3sgFid30qwbyZ1Uj+kQr3SNxJO1jRCdBO0RjXv7368ka7x31SK4K9cCCuPfZj7MIvLRAUVoO0SartLcCG7wfYLIzWeIejoEFA07Cl/DNo4/gShGKtoqCFZuUn3c8c3hxc7zzUs0rr4d2oj38sWxNxyhBwbgtm4eueMSSP1vnuY3KFIi4OXfeyw9lkDN5nqQrrTCLp47cMZEjZoDRO9gKyQi5N76zltMN+8/ifdgdJeWjfUvh2SGUhH1Lxg/J8Y3PDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BmRZcKtxE8OI9GxQgwqabqgY9mjbb6u7NN7jQa2ziM8=; b=ZW3yEt7S+KDear/o2yO/dHSBQKh/J5JH9pEJsnSTcJm8yeOEwg6Scw1NudhKnHvBhheMqUtFyNrxoAl9vBR+6ffvAsT1wFmj+9MFbTvnAfJmpm0S0psdx1kaRiagOVckZZiAHRQGTAv1QyfC4hkSWBy0uSOxO2UWo9rckacyZHOLqmy7bUxNltYHeaAcr2jsr5BfJ82l0qxd7KWkNrizWSqvQnu50vZq0Nx4PU2b6exJU08MKmW0ELAVkC8lsvOjiumdh0fhRMv8FNxF9151GKCvmbZDxJpwfjUN8ZtWxn5sdhqIfar3+55qABgv1HO1u3dTegkjPoRgy1PFuompDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by DS4PR01MB9201.prod.exchangelabs.com (2603:10b6:8:27e::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.22; Sun, 6 Jul 2025 00:03:17 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%3]) with mapi id 15.20.8901.021; Sun, 6 Jul 2025 00:03:16 +0000 From: "Lombardo, Ed" To: Stephen Hemminger CC: Ivan Malov , users Subject: RE: dpdk Tx falling short Thread-Topic: dpdk Tx falling short Thread-Index: AdvsS4XiFhfr9tUWTZG/9hrfPmeo0QAjYaoAAAZ3EoAAOAaP4AADTEIAAAkqa1A= Date: Sun, 6 Jul 2025 00:03:16 +0000 Message-ID: References: <20250704074957.5848175a@hermes.local> <20250705120834.78849e56@hermes.local> In-Reply-To: <20250705120834.78849e56@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|DS4PR01MB9201:EE_ x-ms-office365-filtering-correlation-id: d20fb5cc-46ac-4f07-3a27-08ddbc208226 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?zI+ALFC8VwIiADEW/cGrlPHlMUpi7G943FI2N1FSAzVjG81zwIynbqlwHcDO?= =?us-ascii?Q?GkIlQiavpeX3NW8IAMsH1c2OdLcKl0VdmYTOZvJA0LHVXX/UPX/BOkwTqkDA?= =?us-ascii?Q?Avpk2v6VQBUy7/XU2flSj32f9xUjwB0FPKzx4sbJPJnE6e7kzEbeWHd7ywmM?= =?us-ascii?Q?fppoF0qIAjsETQvCe42E32Ck8FOZ/jXwT3k+5g6xgogMU+8ELRFGMtO4uIuX?= =?us-ascii?Q?hcofP/4cdfTRn+6Un0/rKasXClrUx2L4XU0esWW+AFBmnswpjGV42L/cTKmI?= =?us-ascii?Q?xzzUtQeP+p0JcQd+nNIEDDvOjfcOMwq9IYfYlhcKjY1ok6Vll87F71x9rRuD?= =?us-ascii?Q?eLZdTF30mVq9+eU7zKN1UEUb43TBQS9HgyOsIG6LRY3N0f/7xlsh/3285DWB?= =?us-ascii?Q?uHV2GksiRats9EAb0Ee86qmS6tOCfblftiHI37l1W9uDQFc7HNgIDR+XutO1?= =?us-ascii?Q?z59iJn25ReX/OSbn1aCZGHXHWQSAVPhYd0/2D6npVM4ynT7XT7/cWrHxWVyU?= =?us-ascii?Q?uJCz0jjRPm3YcHgwm/Z0WRBLzjCnu5pDkBHnf30eGKSnSlvwL/sAaencrhBO?= =?us-ascii?Q?xy57erz6R+Tp89jlevKLcF3YjBxF0pKF+lPsQkzUNgUCDkAS3ESqdjCuU+fJ?= =?us-ascii?Q?d0hm7I4r2/BBLeS8ovgEEnhzxVWlRWmp4UO0os7qMmATHHRm9m4zWkmqvj4P?= =?us-ascii?Q?PKTdCHPQ56XcouCCYO82gp1MOIQ1PP/dDSAYYOxQNfGM2NcK2WOe19zBPzBu?= =?us-ascii?Q?Dfn0DYdLrSOeHnTIKquOFZJ1+ifUHa6TOdDxXDHYxIwfHWt1KnZfsQOCnf/g?= =?us-ascii?Q?dRpmwWwhAV5z/XPP7C5uvR42goB11MnHbrlTj8YbpBLCfsqrtYmoYot/xHB3?= =?us-ascii?Q?8bs7pbolwgnjf7KhHp0qDP+F13BAVOuXDmVldkbrNkFRY2CzcQgy6glqpQWm?= =?us-ascii?Q?cPb2u708IiKPonwyZZcs8ypcq9yN9Fi3/r5mbSiE5zvUtV2FsMpI6SRU3YmV?= =?us-ascii?Q?1CwRj+c18S1yAV6MmetTJOSrKJ5q8WWrRtEGH71LldQa/98ZXLiyKoxt/bK6?= =?us-ascii?Q?0B09RnuRCmE3XLPEEcNSyPXOPTQKSCfYyR57qRAFvUnGGsBEIUSuQRlxbyjA?= =?us-ascii?Q?UH/Zdn1WTM9a8fsChUgwqWAba9OiJhrdCGBM67vBap+Yj3sjD13U2Q9cH/vC?= =?us-ascii?Q?KuUDAtjgtGry+NodNXBFbNhrIIxbOoRk5/aV6rF9Qd1enh/L5T+bckCVA7MT?= =?us-ascii?Q?JK7aKx8rTSf0KPqfIDBYznrCooe2pCnRUByuXGqgXasp79ScBvqe+3tHAq3G?= =?us-ascii?Q?gYMzkXE6shwX9rT0NOewc+Am2BmHswD2YThva/BMBfF9K899Saelu4jMP1uv?= =?us-ascii?Q?E1bhAS178ryIPogX7fK4+opDTVvgD6NxYMpPSy5bnfcSfhsDDpKnIStKnM5C?= =?us-ascii?Q?DhBNwu12cac=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?zUAbnQV1FYbqT8iVtboEDmXkb6R5c+Ac5neq/vsWPwe/q14OTIYoD3qUnXj5?= =?us-ascii?Q?72IqC2AGhPPLD/LfJShi0fvFUM1YmyfLLJTe3w0acBBCRmFTXTDWvw0PAVEM?= =?us-ascii?Q?70TvkXHibfDeUpOeK7lwgKxUU1rcWjOJ8nfsBLb61S3MZyUDX2Mif77UMMZ2?= =?us-ascii?Q?HUurYt4bWcmPpIMF22Pi7LzoNt5uIxrBofyEzeNhduAPB5fFr2XSO5dWMny7?= =?us-ascii?Q?E4/dD4xNy/DNd4SWDO1xppL9Zu6X2cRgS9QxFqiYVbo0FG+5W8CIwGlRXcqw?= =?us-ascii?Q?xILcxZHPPla6ng9nnm0feC2PFlsjA/Khw1fQG8R9/AqNbw9yLR5kRILLVtxG?= =?us-ascii?Q?9Fyf2E6hXejm8uaI0TWGgu0y+9cfMNYnuw26jTCD4ueAEkvmTucFcMd7zzUz?= =?us-ascii?Q?rSaA0QGLGlJ9qnuPMPS/kc7BXxLsVWPGw5U5fs5F7fmpEYoIGC7dFgBDJQDZ?= =?us-ascii?Q?UqLmy3Ea1GNuJsLmiqLVXgYki8JXJh/SjPnlTtVgVoCk3b4hetNU2/1FrVcV?= =?us-ascii?Q?XaYPT9LbDfj4HI8MmNV63nsKpqBbn60ldW6K7lkAU+HIlrRyfpDha0xo0fXW?= =?us-ascii?Q?+CVUCe2oxBvkgcCbRkbvfWDdtdQXLft5OnfMSxUf2OsIFL8bJrsd7VQ07mE/?= =?us-ascii?Q?U+qdUtBIs4Ldiy6vb83m2Xrmq2Ej80bU3Tb7XdiTYbXzlliEuk7HhTjlfGVe?= =?us-ascii?Q?o+xLs2K/RQBKo0oKa/XWeFY0D+JTs4fNkNO8ByM37dRVPQLMBC18b9f/04dz?= =?us-ascii?Q?xscT2iriliWBF/ksIXoEe8b0u3UbdzNsftv4NvjVNdLd5v2t+XuGQ4AeY+EB?= =?us-ascii?Q?+p9LT4ACLH3LRitlPq/ltNCgYNrxIBUicyaJCWKcF7ob2DxPFSc+cMJW9jSp?= =?us-ascii?Q?0fmauZeswYctNuZsP4nMzilnZ3Om+jpoQ70mYCOJCuR89Zk9X50QwRHlFiYy?= =?us-ascii?Q?PyN9GiB7kBspV2SFdrFKVOsH2dvJH8lS/bFNhi+3//nHdXpTubLcgCROGugC?= =?us-ascii?Q?77oRi5KPqR3EornImqB8mhJivrF1nmIvQyvjA3IaBHad7d/P4XXLexprECOk?= =?us-ascii?Q?EJQgrdNwJzhxhQ4zXNKoP7sC+98DaiUweQA+b2+xJd2OTP1gNHO9v+c/pwB5?= =?us-ascii?Q?1BGvOvJhqt/tLe31yCFher/E7DWxZpgBGRKI9FLAN1H5w1ZnimKB/0CtbVGz?= =?us-ascii?Q?tl4te6U4EQ4RhVuEKFoxi2GlvmdRIvJA92fLXPatqxF4kl86Z4LZsFVro7RO?= =?us-ascii?Q?CByTDO6szq+r1RHaKiyH4JEcbjk8TZK8ruWiZk/1yPQufM1XX5HxiMX4OI1V?= =?us-ascii?Q?6/B9tFbsGCf6TYucbWTEtyoY4UV5L02zFcO+ZTmm/PP/FUWYrz6JN5hc3mbL?= =?us-ascii?Q?W0pYNjREeVQOpRbm90En0vlMaQ2LnHF9yLu2La9ko/5ObKg9bukW61KU4wgN?= =?us-ascii?Q?6bWEZm8TprJZDDZZIYYdg8JXHX9XhKlLSMeaOLE/kLgquaPAp+wxTmYtweqB?= =?us-ascii?Q?ScQhJi8ahslYD6E1jTolm+k6jg4qfJDeAfj3V/VRto+y96uT1emCaa4sIEV5?= =?us-ascii?Q?R7xxG4kai6MinrZ3rTH11xObcSNAfE+dt04OmP8x?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 5aU4cEy6esNyyGpfwzoLgTNw/7xcYNtbqOEMCTIRHuYLoTFKHNLYfAenKedmtyVSGPT6zu8gM0VccKSIRDaY5+4SKA09rsdkSRuWPLZcF4Px0oPkoSa7VE3Elsb5VKwxSkOqKVZOL9xwvmetSy3x/pgHsydOe0fi6pFLCpjE/e0iKXvvf7tBZ/8fk2XmIaMO/iDb/b/eVstVwEX/eAcVFp/RXTXMADADL2vzxPw0rYLOrRfq+Z2TTLwJwctKduIKzcBjQxkUGsfKbhhst66i5oHtJsW6zvB9gOUByK/aimU9PmHDB8mklKUBSS038a1ThmUEGAN5cGdPox7o5nu1GwzG180CFJ4WB7ppz0/HtiwJiP5Nk88ObuSMy/G/j0b2n3Vx5rymh6ULctfGYRsIHhmkCGKu8Y5nCGWzVX+mM13iqvIFMwtCsDdPF27HNrFQMGlkUo24dPXPD2DOYaNpF4bjmps52gnkQ9QZ8BSjo3fbUxIKhtJgsLL0cmsIFOqzvpy4/8uyKhbflHQKXniE7mt8y51tqbC2Cdp0nhkJcf6YYf8jeftjI0MszeP+JGnxqRoKth6LABkrB1fqJyIh3hT1mt0QRPz/Y5G9T6bmpsVsmVgwEgpCdlKpG6E+LVmV X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: d20fb5cc-46ac-4f07-3a27-08ddbc208226 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Jul 2025 00:03:16.4384 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: axuI6jydvBsq27TQhU1cIDwBwOSPy5nods2IMcwr2qNh1oqB4DGcOgpVGIvDTGQrYxdA6GSur1PQm4iyyu8SlA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PR01MB9201 X-Proofpoint-GUID: OkRXVOGS3sVpxNMtsKYmOiYOinpCrkFP X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNzA1MDE2MCBTYWx0ZWRfX5iUXS1QFlEXa QHVzpQqXgdg5MlUgWPpPp2t/lnXcI/jHxIihr8y734ZFX+kAFclHtRPJ7900qb053YJRvAIOEew Lmi1w6MpLawJCmUJ5osVnkTuijjJNehjDN5ZAy5/We67z3QSmJxNgWpWGq9Zsu2vW8zfUnByQVi Ob0Esx8nlljNWRe3AZs05hhzgIvtkPABHT3crZAFDZU63e4U3p4gUBqn6ho82UfhdZHZcKr/SxP uhxfEPSAENhpXo6XjQlhlcmGWYgKxLDlB4bwP6oV/JcFb3JD3Ski3rL3tHf9U0yTw/lYizWKam7 mffOhowgvQCAevo/eo8fRsbuMzaMjl26cCDO/nkAsCgCYEfwVSiwwzE14YQRmTxDBCBk4OrI97M zEVSvXTDhmDTrO2uirGDI5HbwR9nlxnqaGdUVdJ2YlvR6Mwy1182IxTCx4Sc9h2OxLLO+ntl X-Authority-Analysis: v=2.4 cv=LYU86ifi c=1 sm=1 tr=0 ts=6869bd4a cx=c_pps a=l9lnEPKonMfu/vbXsUzXcw==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Wb1JkmetP80A:10 a=3HDBlxybAAAA:8 a=jZVsG21pAAAA:8 a=jQOgFn-ZAAAA:8 a=8rWy6zfcAAAA:8 a=FTuzVoTcQQB-fN73zyUA:9 a=CjuIK1q_8ugA:10 a=laEoCiVfU_Unz3mSdgXN:22 a=3Sh2lD0sZASs_lUdrUhf:22 a=mT82qxFQzDvLIExZS32s:22 a=YjdVzJdQTyZRADMV7wFX:22 cc=ntf X-Proofpoint-ORIG-GUID: OkRXVOGS3sVpxNMtsKYmOiYOinpCrkFP X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 impostorscore=0 mlxscore=0 bulkscore=0 clxscore=1015 spamscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=999 lowpriorityscore=0 malwarescore=0 adultscore=0 phishscore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2505280000 definitions=main-2507050160 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Stephen, Here are comments to the list of obvious causes of cache misses you mention= d. Obvious cache misses. - passing packets to worker with ring - we use lots of rings to pass mbuf = pointers. If I skip the rte_eth_tx_burst() and just free mbuf bulk, the tx= ring does not fill up. - using spinlocks (cost 16ns) - The driver does not use spinlocks, other = than what dpdk uses. - fetching TSC - We don't do this, we let Rx offload timestamp packets. - syscalls? - No syscalls are done in our driver fast path. You mention "passing packets to worker with ring", do you mean using rings = to pass mbuf pointers causes cache misses and should be avoided? Thanks, Ed Also, never ever use floating point. -----Original Message----- From: Stephen Hemminger =20 Sent: Saturday, July 5, 2025 3:09 PM To: Lombardo, Ed Cc: Ivan Malov ; users Subject: Re: dpdk Tx falling short External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Sat, 5 Jul 2025 17:36:08 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > I saw your response to more mempools and cache behavior. >=20 > I have a goal to support 2x100G next, and if I can't get 10G with DPDK th= en something is seriously wrong. >=20 > Should I build the dpdk static libraries with LTO? >=20 > Thanks, > Ed Are you doing anything in the fast path that is an obvious cache miss. at 10Gbit/sec and size of 84 bytes =3D 67.2ns CPU's haven't got that much f= aster 3G cpu that is 201 cycles. Single cache miss is 32ns, so two cache misses means per-packet budget is g= one. Obvious cache misses. - passing packets to worker with ring - using spinlocks (cost 16ns) - fetching TSC - syscalls? Also, never ever use floating point. Kernel related and older but worth looking at: https://urldefense.com/v3/__https://people.netfilter.org/hawk/presentations= /LCA2015/net_stack_challenges_100G_LCA2015.pdf__;!!Nzg7nt7_!GXQ2fZd0SInSDkG= mvq3j3Kk78iP6qOBV37umb2lNgU1Lo7VoBJ40ZTYOK0LfS3o3Lq64NXnFrTyRMJXxISJyNSqhy_= c$=20