From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 227314659A for ; Tue, 15 Apr 2025 15:10:26 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0CF5440289; Tue, 15 Apr 2025 15:10:26 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com [67.231.157.166]) by mails.dpdk.org (Postfix) with ESMTP id 25D0D40263 for ; Tue, 15 Apr 2025 15:10:24 +0200 (CEST) Received: from pps.filterd (m0096262.ppops.net [127.0.0.1]) by mx0b-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53F8RvMi028832; Tue, 15 Apr 2025 09:10:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=BJRXMFU6DKIS5C18mJN4175xD9pvH8hQx7DY iUhDxRQ=; b=QP0w1HwYYROgE2Y1ifPslO2cmaqSIAZdTZquoG5f6Lv4PGok0RgA CaF/P7FMn+FnYMpgtau2Vy94G3tBT/0GAGGakQt2QoqYY0qje9tX+F9HkvVKFvk/ gmv7wmji8iqpiFKfRdq6wM+MiXkpv+0mi1NPgSantB6gf8/ZWYVO+mBdxKFXC/iQ MmD0lpqUo69b2cucJskrWqU3+gF6ByThCF4G+oSV94G+mkU+J5zTS4m0NEaH3p+c 09T2VV0/UIb3Pjppg9ASM8a1w0EpThtbbJIAmz76SLRiVfHoxFO4bt0kP2AUXmpU qD45HJBsuHyyhQ3iMv02FwHM83oPYadOGQ== Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 461kyc07we-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 15 Apr 2025 09:10:23 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hjpEicEEdReZILpNBldKqINRZnjOmUcyUEVDcZxsvxdR1anKdBOkEXHBEaGmf8JcG/XrLz4M6xjHY4HhtVxwCvaA9g2CR1nIgBvYQ5uE4OaN4CbU55C4ccbHHrES5XIBto7cRZc6mXv55A9+VMlEwZJ/h5hWUC2fiV/4Z70MCjouJt9UwDuU+1bwwp4O2AwxAVp8rTTRLX/ykjjQJZH8B9KstcWcJnNfiFjzod87nF7vgwX/2k8kAbZN3bMCdTUtthG9jjjWKYneyW6G7aNpDoTR3EqZ1Kmb9BMXgfXvznB1iitWvjM0E9Yn01YCr1ZhzPziskP3VSXx/+Y452dwqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BJRXMFU6DKIS5C18mJN4175xD9pvH8hQx7DYiUhDxRQ=; b=B9dGw7Nt1ybB9ZISp0mdh7xnm8iHqi4ow7tblNlnp/yzFNkrBD1sWt4qMjpBNhCwXBJeyBnk6IuDmafNSl7wr/BFHfLaC+CTFTUE7MbaMJa9j3NGo0sl2NtMfeujGVqr47Q/Zp4gsLU+yt4c4O+ig/VmqPCFzS7mP3COV4dbwgASrfwLTzBIpMuGsxd7ZMRZLzLHbhX943H1HE/LFylypRFAlOnnzwfjomDmE6kxYMxKm0dhzB+8WTwb+hFqPBNYznjeksgaBVI2Bm7peFhm17mjz/6UYxwlwZTcIRiGTMwJaayPS8hYtCcseUicxlkIE14UEirpzc5MAJ4Y6oL5dQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by DM4PR01MB7594.prod.exchangelabs.com (2603:10b6:8:6d::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.34; Tue, 15 Apr 2025 13:10:18 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%5]) with mapi id 15.20.8632.035; Tue, 15 Apr 2025 13:10:18 +0000 From: "Lombardo, Ed" To: Stephen Hemminger CC: Dmitry Kozlyuk , "users@dpdk.org" Subject: RE: mbuf refcnt issue Thread-Topic: mbuf refcnt issue Thread-Index: AdulnN5IfTxeCHcXQsa3xaXr7ECZhQAFBrWAAGffZNAAOS6kYAAzcomAAAGN4hAAGKknAAALaIUgAAuwQwAAAPmcQAAYOK0wADIJNVA= Date: Tue, 15 Apr 2025 13:10:18 +0000 Message-ID: References: <20250408205341.68d37d0a@hermes.local> <20250409092418.70c5eb3d@hermes.local> <20250409202538.46a08ba8@hermes.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|DM4PR01MB7594:EE_ x-ms-office365-filtering-correlation-id: 29c2144e-6545-4c4a-f951-08dd7c1edeb9 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|1800799024|376014|366016|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?pB66M7xisyBz4SnZZnOoSDY4u2qxsGBLCbqwqMaRkNcSRINMldKSB2bPlRkA?= =?us-ascii?Q?+C7BpjYvB2tMdiOR58mknmqwjHJBKG3YK4iqfY/mT5PsbMNbcTVaKZ3fi0/Q?= =?us-ascii?Q?TZViMg03xZljyTpAWlFsL1rtOcPu+8YBV5ZA7wnoNPQR2ybKlnxz+26OUSB5?= =?us-ascii?Q?T5qD+k3is23tse97qQNIef8T3kN6XmsGJgPrKsm635ONAp7YbzsMJiIQ8PkJ?= =?us-ascii?Q?yqD5v8OVWIDt+z7VZAk3uXjuyPt+94m5iNi2FgDbAnpdaeEB5Fw6Nj7j9eOQ?= =?us-ascii?Q?eK+AtKS4q/2mfGAwQBQJJNwz3RcHwQILojOarfIVeZ46IZZROnGd0jWc7eeS?= =?us-ascii?Q?wti1wfO+kl3qwsBbGwR/NxFmXWH70WnodQ8Rq574aJX9GnixS8Xqs1i8Jnk1?= =?us-ascii?Q?8gzUv263iZ8+FJoKR+//31lumyjVu1z/JCCX0Ahtpgz/f9DYvb9A84cDqykO?= =?us-ascii?Q?lXgSttpfj2qRCjXL/SktZiXB5z/8P7ef6ngqqPrtvyZqII+lgWevgYCcZ55E?= =?us-ascii?Q?LdWWQY0mGKU8ccerjvFo9eUh+UQInqR4OjB2U0X3pGo5AIvh3WCsJ8JWBGkN?= =?us-ascii?Q?zYQAbEsRumrlgM9o2hfjqE7oUKPwIm3iSlS6GXItLriA9fRCmibIs6xx96Sk?= =?us-ascii?Q?c8zX+N9QlrMReKgc1BxV3kz2LxRU2QdJumCUUtzbFJA9zBcXfX6FCwrJgLS2?= =?us-ascii?Q?CGtOtPWdW7+mmS6r5MGRGWRrz7S+mqNAVMTxKNpCxAcxaTNwDjAgy0kCyYNg?= =?us-ascii?Q?bU0DwFGGWZ8vJggjP1UNbafjme4UzvbQEIGBjhZkeiLcHOIAW8h1sqeT8Fyj?= =?us-ascii?Q?V8daXu5NIJFqE8cevYVY+ofE45ZUV4Mozmk+k44O470XU6h9/Mfn6fDU7Kfp?= =?us-ascii?Q?y/WGP7J5TJuSHZrx7CBvuuG9bg1yH8a6QaQK/G8qonAfTIY4pxCVeKsaTWR1?= =?us-ascii?Q?hYgjCXbK4LDngKU/kk7ybHzI7ldCRuWLiYKGu3J7yr2O4/DrYz3QX7ebgzma?= =?us-ascii?Q?9Ybhpl2KaCl9pfKAa5d/y2GkGV5QhF4Vr9wLqgZena1Vz1mGduylnzwILs0C?= =?us-ascii?Q?sCbL0Emq8rMJi5zP90poPdtzosa9AwsE7dvN/N7nDHfVBOQdgUA/j38AaNeU?= =?us-ascii?Q?93iMZED7y726stojB8cIs8QtBvTrrETiYcxV9zsJWI3cSZJxgK43TA8exVxv?= =?us-ascii?Q?/8wRcNuep1wR0slch9G1786CRHYlY0fAOS0W89GoIie8huSVwI1dkMAU5W3L?= =?us-ascii?Q?rH3gC9+JC/ES+rifnzlKrTARY8TzseHiR9f1o5J3xGNhdFMOe89EwOWgcBkx?= =?us-ascii?Q?K4ovBn9cUb4qnCrioZiTeYxzqKSi5U/s5t/QKBrNbX1A6d41cxhD26NhHFbp?= =?us-ascii?Q?MItgaNNbtFjJ10jFM31FjuFJm0JPpZP1jp7WryGIRpWlNDFz4XitJdZuyZHo?= =?us-ascii?Q?fxdbn46IJH7DBZi6MPqgSlt2kXeu2y/LsiDvIvLbbA/gmX/H+NLd2Q=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(376014)(366016)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?8p5j7VESrjhkJTKaXKsZrjDyUBARdr1HCeT9U2/0Udo/SmV1NmldExblTcNr?= =?us-ascii?Q?FKjo/45Dz0kV2OgN/Kke4ijiDZyoIo4PhOLlHAK2qXxZDyzowTU3pRjN0ZFM?= =?us-ascii?Q?i5g4glNxc2xRABqwAzv4dkKbejagcCc/2AgQNDkVf+L+y/BT4opM21OVoYSZ?= =?us-ascii?Q?IYn1oxuxSUUZteBFMiYalg3Qw1oA4KZ+CU3bEWWK4A6F6gSQsmlDALjxLIjZ?= =?us-ascii?Q?00aLhGtHraXws8N+cagiohT3XrZ1snZzthq2te8B517DjB1JW34YDeT0w1aY?= =?us-ascii?Q?4rwciyYaJVKP984UocQGtqAnYLaVDq955VfFv6AEhNQE5QSz/0M89G302dWF?= =?us-ascii?Q?czjf0KNzeEXr6oMY9k4Oa5zPYuOyGOoWAkIuaW5P+bw6IOveZ0N49xLJ2JuP?= =?us-ascii?Q?tcfMLKab4gI6QOBgwlC8510GKiKVYWnQtrHB7FxXTO1U+3LS5kQVASoUR8/i?= =?us-ascii?Q?gNEiAoxt6FaEF0Pz1qAEMTD39o6akkZ2RbERbETNaD6xPOwQ2xFlj7HFZDAv?= =?us-ascii?Q?EBRrG4QNYLcnTej5BftrawN6kcioMmkdINDhim4Q6+94oX13bHZhhm9R7o9m?= =?us-ascii?Q?ek4ggzqmCcqrHazG69n61h1wPLQ2V4L5HD/ZDurDMA+ehtBxdjcF8oaKcvm7?= =?us-ascii?Q?2G/FNVih9f7djHCMOuPvdXLSG06obUD0u6oqp47+8tLCtt8cuOzVzfHMWAlz?= =?us-ascii?Q?eoMkUyriNU9WWpNp/WZ+6Tlgh8qhHUNnpS2OgplhD+0ewpuVxZ4AAiuB4uKH?= =?us-ascii?Q?zS3tewCSx2dULwaQTnZPRI9NVNVqye3uIUO4CX63FVLCRhhZumzsKLFTFh10?= =?us-ascii?Q?vI3RvZY0Dv15oHsRdUi6x2d4rUg1JWN9Erjmdu3rLYsl69wBrv/799tEtbtw?= =?us-ascii?Q?CPz7M03YWXi+jFZ69kjQMGaa/VV/1m2M2H077g/VF2OHrvzW8khvkeYgF40+?= =?us-ascii?Q?eZL4LMOD6f5f3xfE6lNEswEm4yNTqBRvQZ36lG+vtsGDZo1HOeVhztXhzhTQ?= =?us-ascii?Q?vYA21fJkFdCJIhCv1EtiaL7e/abeBqhRc4CUBihDy+hkOcnrYruNvCg35ENH?= =?us-ascii?Q?bQPl/rSsmdGWMLxxQhoL0Sge6PVnkgCeu/jU8JQVJS4/BdXeCLF6ipZ7H+Fj?= =?us-ascii?Q?VpZPf3t8meyIRsD1eLBJFpItIKDc9e+VMIrZOlEkLPB4hOQVqC2YadkSiEEZ?= =?us-ascii?Q?rasMo5d3XhtaRMfZyGUxnenQJ73l+fVjYiOk66awt36m1Okw+xs9/G/BVt9+?= =?us-ascii?Q?8zTmqW/nugz8ARku0A/iJcJ/ywhRKc9YlrWRtCT0yKpMgqg6AFf3jb6xkKgc?= =?us-ascii?Q?V3oDNwdN4pEkl9HzH9GtbfFaFd2lY4/0wM8iXgGubupqZdTHiUhUFhU5uNkw?= =?us-ascii?Q?5SGyuc/p5E8dF7WYB7+lqKaHw/KbMQLcyk5NuIFXwweE38soJE+QAAAemw3l?= =?us-ascii?Q?vyuTPaDdZTckyZ3/3tCrJdtLVbsUMU/uIvm6EGe1xsT6FvKAwhudtGZ/4Zwy?= =?us-ascii?Q?SGOhF18wQVa4RCluUq9FSMSmhKedC7i+fNswU91WcGA9nKSHmmyeyl8LfQ2Q?= =?us-ascii?Q?EOqGAf+Ktg6n0gj/JHiGm9j/0MxiEAT64NTjPMjr?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: NxF5FxutcI3vOLwcwCjPEblN4B/ObS6+iO1j4N11OFPuS7LPQ8137wWoM13hGGGaGJymxdlYhDtt1bGOmqjwYXbbD+NcOpu9I6hrmwzT06yIuvXlToesMXFm2CPBCmoPkLSTkoVFXDSZu/nxUP94vr07hSMl0XeCBeYHpaATmEi7h4KLg3mmE/PZzWnhiYiwgtaGES6z2oLSz4GMQ7wT40qJ44XBkrc2DiS62y7/bnozrIPo1cmnEJstGBriGuOjUtvTv3uYpYw0nfBYgksbJs3lA4JojLBnsMHqIzxiYnpDctWX0+r6utZx/YZ5uMGKmV/Oo77P2bKyOFpzJMKbgqKlCgrwfxBDfgMZmy4X8XCzaccQZhAPLIAoNHWn6q3M0IKANHFjc9ToIqDtH9kwdOCkt0oMtNGbypxPHV7swxXrfbGI48d31tVx5TpSTU+W+9Mr3Da89TdC8AMLHKXOMGheQ5mr5ZIzaZlQO8+REG7P6gznMA6sdkZtwL/Smp21BQ54RY/59/A/8w8HTMns4fYWBEfm6ZWJk5wWoar9U4wSiL3wG5w/EvNTjrfcGEoC9swYIF5YSnU3D2BOigsnTmA0UhDYD+iBU2yyI2TsTF8otNUgM36RthbLKmfqasu4 X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 29c2144e-6545-4c4a-f951-08dd7c1edeb9 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Apr 2025 13:10:18.3784 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: HHwVuZSvWjtHz4GZ5NiByeDnFrPlpXeOOapLN7m0M1ImiyYa7vnzuomox5kjcyOkPF2XmRjJhcFi5fQKArCIBQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR01MB7594 X-Proofpoint-ORIG-GUID: Ga_rrIf5SoT6GfpIaxl3prFbjoFSYGup X-Proofpoint-GUID: Ga_rrIf5SoT6GfpIaxl3prFbjoFSYGup X-Authority-Analysis: v=2.4 cv=FcY3xI+6 c=1 sm=1 tr=0 ts=67fe5abf cx=c_pps a=o9WQ8H7iXVZ6wSn1fOU0uA==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=XR8D0OoHHMoA:10 a=jZVsG21pAAAA:8 a=pGLkceISAAAA:8 a=8rWy6zfcAAAA:8 a=jQOgFn-ZAAAA:8 a=w-PyxuGWdSLh-FnGY3AA:9 a=CjuIK1q_8ugA:10 a=3Sh2lD0sZASs_lUdrUhf:22 a=YjdVzJdQTyZRADMV7wFX:22 a=mT82qxFQzDvLIExZS32s:22 cc=ntf X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 mlxlogscore=999 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 lowpriorityscore=0 priorityscore=1501 spamscore=0 impostorscore=0 adultscore=0 clxscore=1015 malwarescore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2502280000 definitions=main-2504150093 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, I enabled more logging, and I see the following ICE Driver Malicious Driver= Detection event 6 and 7, in the dpdk_log.txt when failure occurs. EAL: Allocated 2112 bytes of per-lcore data with a 64-byte alignment ICE_INIT: ice_load_pkg_type(): Active package is: 1.3.35.0, ICE OS Default = Package (single VLAN mode) ICE_INIT: ice_load_pkg_type(): Active package is: 1.3.35.0, ICE OS Default = Package (single VLAN mode) MBUF: Registered dynamic field rte_dynfield_timestamp (sz=3D8, al=3D8, fl= =3D0x0) -> 96 MBUF: Registered dynamic flag rte_dynflag_rx_timestamp (fl=3D0x0) -> 23 MBUF: Registered dynamic field rte_mbuf_dynfield_udata64 (sz=3D8, al=3D8, f= l=3D0x0) -> 104 ICE_DRIVER: ice_interrupt_handler(): OICR: MDD event ICE_DRIVER: ice_interrupt_handler(): Malicious Driver Detection event 7 by = TCLAN on TX queue 2 PF# 0 ICE_DRIVER: ice_interrupt_handler(): Malicious Driver Detection event 6 by = TDPU on TX queue 2 PF# 0 Are malformed packets handled by DPDK, can malformed packets corrupt the mb= ufs? Thanks, Ed -----Original Message----- From: Lombardo, Ed=20 Sent: Thursday, April 10, 2025 5:16 PM To: Stephen Hemminger Cc: Dmitry Kozlyuk ; users@dpdk.org Subject: RE: mbuf refcnt issue Hi Stephen, The overnight tx test passed with no problems. I modified our application = DPDK driver this morning, I set the mbuf refcnt in the second stage of our = pipeline, I then enqueued the mbuf pointers to the ack ring. Then I added = the same mbufs to the tx ring. =20 Then I provided Mobile traffic and the same MEMPOOL log was seen but in add= ition it failed to transmit. I checked the rte_mempool_create() has the flags setting to value of 0. Th= e flags setting to 0 is not an issue for my usecase? I removed the high layer application, if it was corrupting the mbufs and th= is has been eliminated. I then tested the multi-thread tx with rte_eth_tx_burst() and second thread= doing the rte_pktmbuf_free() and I see problems again. =20 Pretty convincing that these two DPDK APIs can not be called by multiple th= reads and maybe the rte_pktmbuf_free() must be called after the mbuf is tra= nsmitted. If this is the case I am back to square one. I verified that when rte_pktmbuf_free() on a mbuf that has refcnt=3D=3D 2 i= s decremented to refcnt=3D=3D1. The rte_eth_tx_burst() should also decrement the mbuf refcnt by 1 if seen a= s 2. In theory I believe that the last thread that decrements the mbuf refcnt wh= en seen as value of 1 should trigger that mbuf to be returned to the mempoo= l. Is my understanding incorrect? Thanks, Ed -----Original Message----- From: Lombardo, Ed Sent: Wednesday, April 9, 2025 11:58 PM To: Stephen Hemminger Cc: Dmitry Kozlyuk ; users@dpdk.org Subject: RE: mbuf refcnt issue Hi Stephen, I should mention that I added mbuf refcnt checks at all pipeline stages, an= d what caught my eye was the first stage of the pipeline the mbuf provided = by the rte_eth_rx_burst() I logged mbuf refcnt set to 2 when should be 1. = S,o something went bad after the event that triggered the error to cause th= is. The mbufs kept arriving so the mempool was making available the mbufs = for receive of packets from the NIC interface. So weird. I will try the patch your provided. -----Original Message----- From: Stephen Hemminger Sent: Wednesday, April 9, 2025 11:26 PM To: Lombardo, Ed Cc: Dmitry Kozlyuk ; users@dpdk.org Subject: Re: mbuf refcnt issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Wed, 9 Apr 2025 23:22:50 +0000 "Lombardo, Ed" wrote: > Hi, > I just finished modifying and testing our application to just do transmit= of packets received on an NIC interface and let the rte_eth_tx_burst () fr= ee the mbuf and all works fine for both traffic types. This proves to me t= hat my implementation of processing the packets and queueing them to tx rin= g and transmit from the tx ring is not buggy, which I had carefully verifie= d in gdb early on. I still believe there is a problem with our application= with many threads that can do rte_pktmbuf_free() on the same mbuf. =20 >=20 > I added these lines in my driver source file: > #define RTE_LITRTE_MBUF_DEBUG 1 > #define RTE_LIBRTE_MEMPOOL_DEBUG 1 > #define RTE_ENABLE_ASSERT 1 >=20 > I don't see any asserts occur during my tx packet testing. >=20 > The dpdk header files show the Atomic ifdef checks=20 > rte_build_config.h:#define RTE_MBUF_REFCNT_ATOMIC > rte_mbuf_core.h: * or non-atomic) is controlled b= y the RTE_MBUF_REFCNT_ATOMIC flag. > rte_mbuf.h:#ifdef RTE_MBUF_REFCNT_ATOMIC rte_mbuf.h:#else /* !=20 > RTE_MBUF_REFCNT_ATOMIC */ rte_mbuf.h:#endif /* RTE_MBUF_REFCNT_ATOMIC=20 > */ >=20 > I verified in building our application with DPDK rte_mbuf.h header file t= hat the atomic functions for mbuf refcnt read/writes are turned ON. I adde= d junk characters, and the compiler spotted syntax errors. >=20 > So, I am back to the question as to why I get mbuf issue with multiple th= reads processing the same mbuf? >=20 > Any more suggestions. >=20 > Thanks, > Ed >=20 > -----Original Message----- > From: Stephen Hemminger > Sent: Wednesday, April 9, 2025 12:24 PM > To: Lombardo, Ed > Cc: Dmitry Kozlyuk ; users@dpdk.org > Subject: Re: mbuf refcnt issue >=20 > External Email: This message originated outside of NETSCOUT. Do not click= links or open attachments unless you recognize the sender and know the con= tent is safe. >=20 > On Wed, 9 Apr 2025 04:46:09 +0000 > "Lombardo, Ed" wrote: >=20 > > Hi Stephen, > > I am looking a the rte_mbuf.h file for rte_pktmbuf_free() and it is not= clear to me that it checks if the mbuf refcnt is 1 before decrementing it = and allowing the mbuf and segments (if any) to be returned to free pool. > >=20 > > Could my application issue be I have tx threads that transmit packets a= nd does rte_pktmbuf_free(), while one other thread will perform rte_pktmbuf= _free() on the same mbuf? I ensured I bump the mbuf refcnt to 2 before oth= er threads can process the same mbuf. > >=20 > > Thanks, > > Ed >=20 > It doesn't need to check refcnt there. The check is done later (since mbu= f can be multi segment). >=20 > rte_pktmbuf_free > -> rte_pktmbuf_free_seg > -> rte_pktmbuf_prefree_seg > =09 > static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(stru= ct rte_mbuf *m) { > __rte_mbuf_sanity_check(m, 0); >=20 > if (likely(rte_mbuf_refcnt_read(m) =3D=3D 1)) { > normal fast path. breaks the chain. > } else if (__rte_mbuf_refcnt_update(m, -1) =3D=3D 0) { > refcnt > 1 logic >=20 > Note, the refcnt doesn't always go to zero when the mbuf is put back in t= he pool. > The refcnt for a freed mbuf (in the pool) doesn't matter, it is free, it = is dead. > The refcnt is reset to 1 when mbuf is extracted from the pool. >=20 >=20 >=20 You might find something by poisoning the mbuf when it is freed, so that an= y attempt to use the data would get junk. Also, add check at start of rte_p= ktmbuf_free_seg() to catch dup free. Something like this, but only compile tested. It will break some of the functional tests, because they tend to make bogus= dummy mbufs. diff --git a/lib/mbuf/rte_mbuf.h b/lib/mbuf/rte_mbuf.h index 06ab7502a5..60= 88b34506 100644 --- a/lib/mbuf/rte_mbuf.h +++ b/lib/mbuf/rte_mbuf.h @@ -1423,10 +1423,11 @@ static inline int __rte_pktmbuf_pinned_extbuf_decre= f(struct rte_mbuf *m) static __rte_always_inline struct rte_mbuf * rte_pk= tmbuf_prefree_seg(struct rte_mbuf *m) { - __rte_mbuf_sanity_check(m, 0); =20 - if (likely(rte_mbuf_refcnt_read(m) =3D=3D 1)) { + __rte_mbuf_sanity_check(m, 0); =20 + unsigned int refcnt =3D rte_mbuf_refcnt_read(m); + if (likely(refcnt =3D=3D 1)) { if (!RTE_MBUF_DIRECT(m)) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && @@ -1435,13 +1436,15 @@ rte_pktmbuf_prefree_seg(struct rte_mbuf *m) return NULL; } =20 + m->refcnt =3D 0; if (m->next !=3D NULL) m->next =3D NULL; if (m->nb_segs !=3D 1) m->nb_segs =3D 1; =20 return m; - + } else if (unlikely(refcnt =3D=3D 0 || refcnt >=3D UINT16_MAX - 1)) { + rte_panic("mbuf refcnt underflow %u\n", refcnt); } else if (__rte_mbuf_refcnt_update(m, -1) =3D=3D 0) { =20 if (!RTE_MBUF_DIRECT(m)) { @@ -1452,6 +1455,7 @@ rte_pktmbuf_prefree_seg(struct rte_mbuf *m) return NULL; } =20 + m->refcnt =3D 0; if (m->next !=3D NULL) m->next =3D NULL; if (m->nb_segs !=3D 1)