From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2205D46548 for ; Thu, 10 Apr 2025 01:22:57 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9DC17402B7; Thu, 10 Apr 2025 01:22:56 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com [67.231.157.166]) by mails.dpdk.org (Postfix) with ESMTP id 7776140288 for ; Thu, 10 Apr 2025 01:22:55 +0200 (CEST) Received: from pps.filterd (m0072399.ppops.net [127.0.0.1]) by mx0b-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 539KbE6g032148; Wed, 9 Apr 2025 19:22:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=KRh+QYJc1+YZ20+CthVBzNEU/0U+0hwPcrEF UZy9meY=; b=yPAsbJJcrAG1O3n8Md5omGD9xfodVmhB/S3Lz2IVKDq7wq4tI6jc nHymAWlaRI+159hV5xz+kPe4UklIvJbjEtSLDax0mjBS3PILIS0jprn24a/V5GAi Uc1uMi91JrUoMQ9oPF1YalL++TK9zFUH6UwfSb52AKAZVIXQuG8UPkpTMObNQm7c 7/148J96jNLmueoTkScfYarWJJVdbkDcNJl+f4BpsQXY6ar36oAhlNhhO1I1k4Su 5y2KlmixMM0OEvGw4T9wOTdXNVjyfykGP3cBJOpP938JWpD7QgVP6Ky6Qk/oPWgS OzEF5gkQskZpqXy37m3i8SBpglK+Yp+1XQ== Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam04lp2041.outbound.protection.outlook.com [104.47.73.41]) by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 45vh96uc1e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Apr 2025 19:22:54 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UyQZlELpG1qJZRBOdTV7Vp8zH5f0fnhlFJTMWelVSVo3rDFflq4k263Ts3n8SvJRmTDYEnExtn9jb5u8exWUE6O4JIuthRtAnk+D6UXxfnY3HBleg7HYB/0RJUiQV+9hqwb+EUFQIv3ABvVLFarqYf8UVc6jASL2sTDVSk1vqBA8NZ8xtNe+kyGam7+IHsmMtMvdRdYX26+55X+C6tyes0jAg8167s1geUgfkNLqYflZQVWlnHVHlA3+1DBOZeaAb+U82zzUtRuiXPOMldg2Y+Nxoe5CnCZWO8LWqJN1xUNfRjmxXwrheRyKsw3Q9vMFglPwflnzAjZUxtt7GDnSHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KRh+QYJc1+YZ20+CthVBzNEU/0U+0hwPcrEFUZy9meY=; b=EKKQcHXtXr1dPZf9IwHMKSV5qH+BiBbx9pO6iQgPdeQAHxXE2UFAtqWC9Ao4e2vfy6BbVGTQScbG5LNzYQ24VyKudR9v8qQY/HslEBEl5YsE0g6qBjQNfL04uxDkJvAkEnofwCwdQc6UTVYzKcFnkJEBpBCoR2smucUi35vLJrmA8uYF5zIlxpCm266cDZ4bM24p1N3Mp6dBSSPxLqCgwNvvsgw0cC4xypmjwcedaK2UrNaTg89alwKvaM69+bymA48Lo5pfJVNCFLNUup+5BZE2UlnRKHV9QBQQE269W3L0ji1/Z+naWUeGlFFqUTxL7iqKwUbWi9C06b+Z2GCQdA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by SA1PR01MB6752.prod.exchangelabs.com (2603:10b6:806:1a3::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8606.29; Wed, 9 Apr 2025 23:22:51 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%5]) with mapi id 15.20.8606.029; Wed, 9 Apr 2025 23:22:50 +0000 From: "Lombardo, Ed" To: Stephen Hemminger CC: Dmitry Kozlyuk , "users@dpdk.org" Subject: RE: mbuf refcnt issue Thread-Topic: mbuf refcnt issue Thread-Index: AdulnN5IfTxeCHcXQsa3xaXr7ECZhQAFBrWAAGffZNAAOS6kYAAzcomAAAGN4hAAGKknAAALaIUg Date: Wed, 9 Apr 2025 23:22:50 +0000 Message-ID: References: <20250408205341.68d37d0a@hermes.local> <20250409092418.70c5eb3d@hermes.local> In-Reply-To: <20250409092418.70c5eb3d@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|SA1PR01MB6752:EE_ x-ms-office365-filtering-correlation-id: e52cd72c-856d-43b9-b2bb-08dd77bd7273 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|376014|366016|1800799024|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?p7r/skbY5ov/6PKlgmD1MniZXsK89XKYpp1Qe3ZH+Mcee99AteLLo1Atvgcb?= =?us-ascii?Q?DkpgphlxAJmhCfXGtz6fMXhJm7Mxq+GvCzi/5M+I/5Hfemrv9XVHkA/McsGD?= =?us-ascii?Q?Deon3fzqtLCkZovxAS9NbkhOdBJQXC9uj5iLpcv6maZTV09JGaVX/0Fdbl1S?= =?us-ascii?Q?5P9IEIi96EYfPBoyw10aRy5SaE473KsL+IbdHfUwUUrYCUv2HVyZqm3vY8uK?= =?us-ascii?Q?Nbb8nAzy7RV/C/l9AmRIwgjJ/1ro+m/AkoqIs7id0B4p3QXnlFflMGn6H9/U?= =?us-ascii?Q?t40V6GjOBCkYQSnupujBOnrkt15Cqtz/R8YfU6Un5qNgVQ7Pt25wKv2azjOm?= =?us-ascii?Q?ZKzzbnNn5wuxgsao6N8w2y4LcMDYR37aHVmnW9OmPO+SCdacnm4gfAw/jlYc?= =?us-ascii?Q?wVNwxHGkBUcJKtA3y+5Y9UuDBOylVe48RD3CC2NKayDQg5k3toSHzTqQdGHE?= =?us-ascii?Q?t1+jpQcFfYmL+u1V1LuHuqh0KM7XNCEfhyQm/c4H0G5dWGjTwlnbu4PSh7VR?= =?us-ascii?Q?HJwi9MqszTj1+1RESdppD2zpgxgVaqmCTzh0ZYa03s9gzYrkLuRhj/OYfKjs?= =?us-ascii?Q?9YeKqsIrizl6VtDTYGffNitjbtLZ98ZelhXQw+Xb1xh5AVYiwR9QWoErNnjg?= =?us-ascii?Q?y1gRH0sy/47nRlf8TKjXoLzJA0JVJFY3elNZ86WNH/VVQYMU4AOCbFScZoZy?= =?us-ascii?Q?MgNxIAomkmT2tXe6wbrCLm2dRnuN34wwjsvQQLj63ZZuzecx6CRmqR1F3evD?= =?us-ascii?Q?EZumV2v0fk256Aumv8MoKHCnpDsvN/FqvVDQqa2/0xcYYG7n/hTLtFoj/sH7?= =?us-ascii?Q?Ensn5n5wP0g53fjKNeQHkYWE7sRZ0TB7BfyGmuFo1rZtWzyXwhTOiE/sfFk3?= =?us-ascii?Q?cFZstBkIqBm0RuW+r/lMtAx6LNKJYsB7frlPTq59fCj6ygKiklSDtQn/YrSm?= =?us-ascii?Q?3kOibStFGV3xD4TFlkcPzE/FfFozArIVAxLhTr5Ta2feqtiKUl5xc7RJhAJh?= =?us-ascii?Q?xT32f8Hu8nJNg8et8C17hB2bws8ouaV+652LmD7LtJnh5Jd+Wzo4U/1sLZ1c?= =?us-ascii?Q?SN24Vl9NsNaYRMZlq3t5m5RkXQ3+CtkDeAYUbAInDlzqePD5JT+9Th9WA/jb?= =?us-ascii?Q?EcQR1EZ/hW6ZxUc7sMDKlz5VHTmphxM+NaB3NwXmB8oO1bm009t6f69ejGjP?= =?us-ascii?Q?dkX9/ExkviWFlOao5xv5ESTaziV+Kfx7wqHaNIAoCm+Zlb5J1AbUkdaWwohK?= =?us-ascii?Q?HCAnlzbF/qF3LU8mt7b/E39NUr7HUikptpz30l9aD+uVf0q08kfCiPA+TYIf?= =?us-ascii?Q?akOg7w6EIDgQinVLYGuOpf2D5yvXlc9+Nhe6bzzipDLThG6n2ypW7kslPSuQ?= =?us-ascii?Q?jrBA4K/5v/Y/+8kRrKrJ22Ywoo7DtgN2rSgNaEYheMqXwi6rcfjTSA4SmguD?= =?us-ascii?Q?TNQFQvQgr/dtDixW0v0RQa7oFmH7cj0g?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(366016)(1800799024)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?8yxh3QTlkb/SGGjQNsOMDs1e60ZBoZ2HtRlbpgBhiUpKeB7WQob1sgNy9mk0?= =?us-ascii?Q?0MMaD8D1JHDWn8bjBZHnmAa/Xlnn5ACVrxioKsnE237DK2EtyUTDJOsx6O8S?= =?us-ascii?Q?2BLqKLNUyeRNlZqjkkOyHuOckDl/tpkKvsTCHN5KaNCH11NkbZqfwhUxgLix?= =?us-ascii?Q?+bTamy6EMCC5dB3f4hVJmUfD7YxNIX+wN0YSNy86anOPap9D+vgn+rhkvpXX?= =?us-ascii?Q?hN7JXgkenKrMxPyw03h3rqpZ6QJih9G1fmOugvHpKi/8IMDknn6bYU7iqurT?= =?us-ascii?Q?MAsjVKqkFIsXmpL5EIxZbuLwHZv3zrPNPksNu8WHVr1ptXASBSh8+MEfpe1Z?= =?us-ascii?Q?IRosktNrUCCWUaKKwJXrsTHcyaXfyI2DOCY/tn/kFusokRabnh4tN++sfWO9?= =?us-ascii?Q?Tood6FyQfw75vrcE9hZG/9o4gkng7lvYmuyWgBuZ52ZxoMqquUNegvfVNNht?= =?us-ascii?Q?/VJwZmRvzq0xL/ILiK0LJyjg2h2DilisE98Jyuvb52d/U7856036U/+/3PqD?= =?us-ascii?Q?q0BjSMhlQeTrpymFcgKX5/vUm/1WhR9xgW77kvFxsAc19457oAQnfX2CK2ia?= =?us-ascii?Q?9JDvRlb9CuBbekrMjBVoBxlr6kOEyD0tcUdQb7hOd7Dl93YEl+u8JoWap++A?= =?us-ascii?Q?xFWqQIipDSiqq1mzRPiHq3h4rkCNN3OmeNYF3+EpP0q126Q/3JucISSFq+0q?= =?us-ascii?Q?mnB5rcroJ+yWq5XY1MPnYUGzi95qAo4/cOgKJMrgSUyaRvdKtVfTCbkEGOZz?= =?us-ascii?Q?d8ivQawOS+i0xR3hmwRFC4WchfenvHT5oHgvnVBmjYXFCrA+8/550OQQSVDW?= =?us-ascii?Q?Zrn9IdNrItWSk0phMgqbZTTwti0o3l+0RMNqaLn92Qe6RNboJum1vq+Ygipz?= =?us-ascii?Q?rgSAoCOKYz9wEsVB/8D3YQdH/WR2Q7pGsjTeGLVP/01FO+ZPKlQjaNS7f1wL?= =?us-ascii?Q?SlBLboKxqe91i3qqIkT7gbM04aP1DBXsR28DXItUewmB0ku5A2mMgQeNvQuS?= =?us-ascii?Q?NwGRRxx3GhphEyuEHwo8vOMmqI8HbvdAlGv3xeqwZiRRYUZVXDbrHgCOX+Px?= =?us-ascii?Q?j1fg7vYc5aspdshnYZ73X6gtnB8kWyMhXH62hLzAPDavlUrvwSzqp055eA1y?= =?us-ascii?Q?lbKBqMMegduB0u4CRLRUgpxPehmI/L9FKP8mqrXJxt6KAqdV3T0xDaycAopB?= =?us-ascii?Q?Csz1z9Pf9OXTyl+Q31VmXfDe6vYAeN+/quzSiIWwx7AYkUZ6ghIDnrhhbWcu?= =?us-ascii?Q?ARAYPpOBpkGLKXKfMGu5y72+UdKP9tgoCLPseKQzrM9GMOIT1lakA0GzMAUb?= =?us-ascii?Q?p/SGXb8mkvphmY1/dkx+R5ucKB4ti5hIxT23WKxJYuPFOXMcr+GYagSRbihs?= =?us-ascii?Q?SsU0cFq6HBMM1vToGYD7UZ3aAGYNCQTOUZyUTNZKGGTA2K1gldiRjHSZwBXW?= =?us-ascii?Q?uots6nXvSdnakxYvNIJ0Eq0k1++JnkfWJM54SqQJ3a+zVPjA4R11JBkkmLY7?= =?us-ascii?Q?Cs1IUVURs64xG0RYy5JfC4zSAal5vWhphGmWixgsARnXxVy3H/EMQzyAeAAC?= =?us-ascii?Q?SSCMpvQqfPz8P+utrBwhN1G4rNFO3zBxouEZuoEO?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: gqSwIOuCUIpQ5ff/g+SwgKq3URvt4pB+ipM8ayzjoecriwt1bP2R1ITdyq7w2U+AGl+wAIDkssrvhPlaKQe+mFZWb39ppnmY8eJc7f/hZrefRM/frSE9t1BXX3nPOJeLgIm4nG4NwuHVYwxLYaJ6qwBHTaRwz+MgHM1b1eQ1s+S40FwEGsufJiNk3siT2Y2poUIQzB1lVhn8ONagQT9/OV6BB9OidODVwMokFNTg/8JgwnfcgZDkaqPYM0Eh2nlmPSt2bskn0GBnMU36J4WKUP+Ir+JOGdufWU8SYY+gfiywSvweI0t8MgoXdHD03Y1Blpo+865/FDiVbPYodOfPJatjXktBh3XfktKGRikZ+jFwr9LnuCxX86gWH6EJTPaKZeUJqga7mLqp82gL328CKAmCu/v0cRFIeHHs82dhsNlRK6bw14Ve0+VgC0ple93/B5phJtGIEvhXQVVXT7SO1830RgP7prRnCvYQp5QQGE/YdKZH2ZrL9tsF4J3pPx458r6g0Hn9Ldx8NdguxmgPUdgHVDtdmNr0/cIje3mAzM+6xKGy+2K57yHwkGqn0hDo4gOynTSAXWcEI0wf5xTW3Q8oUTggbdv+3QdTfgvFdP25jLmoiyXsevtTrH7cqOhp X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: e52cd72c-856d-43b9-b2bb-08dd77bd7273 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Apr 2025 23:22:50.8952 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: M+Y1yxb/3J+2bK6X9/9gHzFUTKuNfKqF6QwGg0ltkt22vqS3sOUDHAawh3gYe6xWrbUQMSY3LaIo6paxeqXpYw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR01MB6752 X-Authority-Analysis: v=2.4 cv=GZAXnRXL c=1 sm=1 tr=0 ts=67f7014e cx=c_pps a=50c+v8Vu7SRZtPLyLAZfNQ==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=XR8D0OoHHMoA:10 a=jZVsG21pAAAA:8 a=jQOgFn-ZAAAA:8 a=pGLkceISAAAA:8 a=8rWy6zfcAAAA:8 a=mzpY7wCNXwZviOgiD2wA:9 a=CjuIK1q_8ugA:10 a=3Sh2lD0sZASs_lUdrUhf:22 a=mT82qxFQzDvLIExZS32s:22 a=YjdVzJdQTyZRADMV7wFX:22 cc=ntf X-Proofpoint-ORIG-GUID: zHSNgGbHeeyl7AOsB6oT-5rtfxef6sNT X-Proofpoint-GUID: zHSNgGbHeeyl7AOsB6oT-5rtfxef6sNT X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 adultscore=0 impostorscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 spamscore=0 lowpriorityscore=0 mlxlogscore=978 clxscore=1015 phishscore=0 suspectscore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2502280000 definitions=main-2504090161 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, I just finished modifying and testing our application to just do transmit o= f packets received on an NIC interface and let the rte_eth_tx_burst () free= the mbuf and all works fine for both traffic types. This proves to me tha= t my implementation of processing the packets and queueing them to tx ring = and transmit from the tx ring is not buggy, which I had carefully verified = in gdb early on. I still believe there is a problem with our application w= ith many threads that can do rte_pktmbuf_free() on the same mbuf. =20 I added these lines in my driver source file: #define RTE_LITRTE_MBUF_DEBUG 1 #define RTE_LIBRTE_MEMPOOL_DEBUG 1 #define RTE_ENABLE_ASSERT 1 I don't see any asserts occur during my tx packet testing. The dpdk header files show the Atomic ifdef checks rte_build_config.h:#define RTE_MBUF_REFCNT_ATOMIC rte_mbuf_core.h: * or non-atomic) is controlled by = the RTE_MBUF_REFCNT_ATOMIC flag. rte_mbuf.h:#ifdef RTE_MBUF_REFCNT_ATOMIC rte_mbuf.h:#else /* ! RTE_MBUF_REFCNT_ATOMIC */ rte_mbuf.h:#endif /* RTE_MBUF_REFCNT_ATOMIC */ I verified in building our application with DPDK rte_mbuf.h header file tha= t the atomic functions for mbuf refcnt read/writes are turned ON. I added = junk characters, and the compiler spotted syntax errors. So, I am back to the question as to why I get mbuf issue with multiple thre= ads processing the same mbuf? Any more suggestions. Thanks, Ed -----Original Message----- From: Stephen Hemminger =20 Sent: Wednesday, April 9, 2025 12:24 PM To: Lombardo, Ed Cc: Dmitry Kozlyuk ; users@dpdk.org Subject: Re: mbuf refcnt issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Wed, 9 Apr 2025 04:46:09 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > I am looking a the rte_mbuf.h file for rte_pktmbuf_free() and it is not c= lear to me that it checks if the mbuf refcnt is 1 before decrementing it an= d allowing the mbuf and segments (if any) to be returned to free pool. >=20 > Could my application issue be I have tx threads that transmit packets and= does rte_pktmbuf_free(), while one other thread will perform rte_pktmbuf_f= ree() on the same mbuf? I ensured I bump the mbuf refcnt to 2 before other= threads can process the same mbuf. >=20 > Thanks, > Ed It doesn't need to check refcnt there. The check is done later (since mbuf = can be multi segment). rte_pktmbuf_free -> rte_pktmbuf_free_seg -> rte_pktmbuf_prefree_seg =09 static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct= rte_mbuf *m) { __rte_mbuf_sanity_check(m, 0); if (likely(rte_mbuf_refcnt_read(m) =3D=3D 1)) { normal fast path. breaks the chain. } else if (__rte_mbuf_refcnt_update(m, -1) =3D=3D 0) { refcnt > 1 logic Note, the refcnt doesn't always go to zero when the mbuf is put back in the= pool. The refcnt for a freed mbuf (in the pool) doesn't matter, it is free, it is= dead. The refcnt is reset to 1 when mbuf is extracted from the pool.