From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C077546468 for ; Mon, 24 Mar 2025 17:39:11 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9865D40A7F; Mon, 24 Mar 2025 17:39:11 +0100 (CET) Received: from mx0a-00196b01.pphosted.com (mx0a-00196b01.pphosted.com [67.231.149.170]) by mails.dpdk.org (Postfix) with ESMTP id 2FFDA40A71 for ; Mon, 24 Mar 2025 17:39:10 +0100 (CET) Received: from pps.filterd (m0096263.ppops.net [127.0.0.1]) by mx0a-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 52OFOGV7020137; Mon, 24 Mar 2025 12:39:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=R+cx3g/UgBZ/wyrv6vq5W2SK6ApzUGFZ1QhY 6ZN2kTI=; b=kTH6aC3kqsahGwNC6+ktnZCXF/MXsKrXh3T2gg/TPEfY0xGM4Jdc ykwsCYRBdraB9eI8QIANoz0xSLRI+xRuM6uTeYFtwT3kslPRvwhkglYPs+wlhgdF G7rk971bumAgQEKosUTFfKJNUAeJTkL4tqX1wfBpGccAY27ORAYwR8zMZaBxhe6r Bwxieuy2QIh2traM3/RUOqEAc+igsexxIDNtEERIj+X5JOQIRryheY1dOdNkTbDR /gX9dQWSKiAF+4T7OM0ta+v4+xJB6BWx/y38+G/7Cqc5OCrm/L8kNVwJs3Je2rLq BVoQ9YmkRhTpNvFJFm8dTe8gDmVKfxWtKA== Received: from sn4pr2101cu001.outbound.protection.outlook.com (mail-southcentralusazlp17012010.outbound.protection.outlook.com [40.93.14.10]) by mx0a-00196b01.pphosted.com (PPS) with ESMTPS id 45ka0b02r7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Mar 2025 12:39:06 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nhQNaj/1G2lWN9NUxmhd42r37/9lkl0fo39OKOo2Iwqmg5klQvbbzX0LN5sUXDCgvVQgg3M+Ltq8rHk/cs6WoKruwylsTfD9i0W/oBBNHC6NP4yn28mo4gbejaAwXs1Sw4NU3lCkYlWT8gnzB5WUo7H/JYp2ZRXJ7RPpDtUNO9Uovztf+g+8jW+7XzLjESrjVDTqJR7oz72/Wp6kUNVbMnHIJqodGWSh4B7u3ZJw0Mrs2RWc3F4BEF4i9XLBArm/kwhwDbR5EhMeaPvpdQ7SmOMu5VBqETJJ6s3YXHoYmXQOZi5qFhEZxr4lxuOj5oOlvV4o6/DpmutAhtWBtC0YBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R+cx3g/UgBZ/wyrv6vq5W2SK6ApzUGFZ1QhY6ZN2kTI=; b=EqOevbP0C906EuFC5swhNhGumkeQlwt/hBYej6DllWG18bqqgVlV5NbDjxcdqM9M5jksVBoPZYRN3U8Nd6933/wO1ZAq8d97d/+q741MblpXyN6/CzVjLi8xumalpmGF3zGWkL04sBcSkn7IhWf/nLKBzArfvb/meSkzm6/Jc/Og0A8TJEKjX/hmDpDsfj+6+14a1j9xq+1UKb1JcTPLaskSEpHtobN2DKFRJqD3lVVu/Ivi8HG0ozKOGR8mwHtM9q24jtZBll6frSyCF3rWUqpZssBm6TZlxqK7+yIbPpZSp8iqKQedyrE7aZBBxwHDLwh8f40rpQDqd4b7U20Cfg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by SN7PR01MB8136.prod.exchangelabs.com (2603:10b6:806:35b::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.42; Mon, 24 Mar 2025 16:39:02 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%4]) with mapi id 15.20.8534.040; Mon, 24 Mar 2025 16:39:01 +0000 From: "Lombardo, Ed" To: "Kompella V, Purnima" , Stephen Hemminger CC: "users@dpdk.org" Subject: RE: tailqs issue Thread-Topic: tailqs issue Thread-Index: AduY9RY0Q86kHIIzTwW3RWYKe22BSAAF7xuAAAKqzZAAA2FugABZg8GQAHdI0PAAGJ/dMAAD+cag Date: Mon, 24 Mar 2025 16:39:01 +0000 Message-ID: References: <20250319132349.5ff339a7@hermes.local> <20250319161659.573e9660@hermes.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|SN7PR01MB8136:EE_ x-ms-office365-filtering-correlation-id: 626403fe-3ad3-40cc-d4ab-08dd6af26236 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|1800799024|376014|366016|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?oVzKSeurSvkZnW9e4mUt6oA3cftEFUPV/ZmsS16/7Uw3IqgMRWDPHzQjcr/R?= =?us-ascii?Q?Jrv/qz4W/EJXJrqVIY0zhXRy/Fw9GgZSybSFMwfulDP32E1AjG3LYFrtjfsr?= =?us-ascii?Q?FCARx2byvGMqldIrc7UMo0vq45QlmhjkIHFkmA07a627jgbtX5YtK4FzRvBI?= =?us-ascii?Q?tp+aZa4dGEYRqGwf2qXouC1KghNN98gH0K0UdoejaeIx4+e1aBLoRaVmnrqt?= =?us-ascii?Q?1u+yDw508EXmB8obk1IumTW8IOKJ9V/wH+uVaZPv6HFoEJczyF8XLU6ZDZfZ?= =?us-ascii?Q?I+WK2dBwSf8/T5VnV+x4V1anBsrk6i+frrjj0j2LnRZrJZ5lrAjaL56/mWBA?= =?us-ascii?Q?3LiaqS2v8h+P2IiPvpLYv+qEbgc1zLwX4+juZasSLJas7SrqBOk2e4wUWvSz?= =?us-ascii?Q?VUEY5k+KWDNQZXB7NVga3U2qAyiNPLUdebZNi/ize93BaLm0IhY9WxTuMc9c?= =?us-ascii?Q?W0Xzg0VNfMyE0Wx41Kz3PGpFOQc3g1QIbbDQNIBM6PlMnFGu+mbYdvbyoj1G?= =?us-ascii?Q?/W8TsfUTOv1oEDSEaKTrcGctuXObhlClG1hTQqC3Nvjq4EorSLewddI3YpLm?= =?us-ascii?Q?BT8LeT4sKA5iyhhekYPEqScUxiKm3dXQTlTEQFXQmmP55xz6IoSxnURx88Ro?= =?us-ascii?Q?PUaADiXB+JZzuZzyEt6PzA3ucXN/TMB5F6b2XBiPtlBFy0s5BEC6+A1TUU7c?= =?us-ascii?Q?s+ZvdbrURGyw+JhLq62ZOQ9Az4KxzHEr1oFvkjQFuh1t+DLZgYhnidHFUaZd?= =?us-ascii?Q?ho1Qd3aRVgAxJUe/tYhCXnMqoZZaQKDk9jJDz2tyJwR6EsttbJYUmPcaw8WV?= =?us-ascii?Q?jWqqyZfKXAOXlWl5sWDAu9PD1/YNK1kG4ECHxLOyo5m0S5c3GKwN+VacwhHS?= =?us-ascii?Q?CvWlF6ikWeVJfzUVLVAFm0+31k3qukcoWB/+zJVHO3OA3g25fS2CWm39cRnk?= =?us-ascii?Q?otHpjnl1pOaBV3EuzV2HGLz7kzlMnQhvPnVNBGy4OEwjcV4/rdI9eehcGR4E?= =?us-ascii?Q?62UHw+jN6R/we8DNkS0EoQYvMoRXPIDxuw/+P8vLqLnKa0S0zRqeHrkG8P//?= =?us-ascii?Q?HTqx2QMjTTkh9Op8iudA6Nz2yDhjf+8ruXC5rlk17IL+oT2+CrHRmuK+tB3d?= =?us-ascii?Q?RCcDXauWTPIgJ9sAHgSvIfZ1DNrx66zZgJv5kL3TBpYx4bx1eAaIXQ2jUn4o?= =?us-ascii?Q?mWEYTJXqvDfGP7K8kibpNQvPguYe/5lwJSwEdlHqIXm3t3WRVbedCUE+Hr1N?= =?us-ascii?Q?DodSmZ2/KFX/76yi1fctVfHzMwUxrtbzJ3n+hvQlRBoKcNksdvL808r8y3xo?= =?us-ascii?Q?aEKVEh5wu/bn14yHgOSpiSp7D8B2SBzd4jJlxMTaivLSAG766VAqR9s3MTst?= =?us-ascii?Q?+0i6h6j/uY3Ab2TZeIjJMrHfsAUPDImAewVqAsjwcAmowh+VnKhHzlZTxG0x?= =?us-ascii?Q?TvQq6DMXIKvK5jlGUW1HdMdXpXUaAzkA?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(376014)(366016)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?gEoKJd/WoXgIAmiM+i5YuVZw74Bd5iFRAUYk2R7aXYy+aE4lrM/A+RnyC5tQ?= =?us-ascii?Q?5cOae9kMfho2nYlttMRKT3xS+0lX7os0H3mZLPI5acbvwJwaGjz2rgt7/u0z?= =?us-ascii?Q?Sli03XlFMvVaxCVdyz3KDvKIOwalrMkC75AM8Gso0tiJcfCZP9ciyDShPphB?= =?us-ascii?Q?ApcxZ/Y2z22RO4eGZif2k+18RasRF74Ofq2LDQmmiEJXYfARz02R+ynwZWMJ?= =?us-ascii?Q?g/kwan+exocMlDO0KYoDeAMrLROCVcaKAk2m4nwX2BHl/wfFMBQEr/gjq3hL?= =?us-ascii?Q?AJim79TWFzM1m3m0fAL/+Nf1BTWlby+VMTVQHc9m+ee3S1WpCaRhxRI4Kxy8?= =?us-ascii?Q?AO5Pa+gu6EGY9hHdMEPQ+lk4SP6+o4eRHN8Pw34cF9GCZhTBzMlrN3NKaTER?= =?us-ascii?Q?13QkKmbxIspwqGaF6VFBijlfEnfdHN23Iza1BeZtlFlF4FvjGuSqHu9XGcuW?= =?us-ascii?Q?CH+W//Lgeb/bd+CpwFXLkJCd5KRmPbe0ubD5o1oXf7aAd6lbJCTa0Iscgf4L?= =?us-ascii?Q?K4N+WINKI0JILF7I75WorgfDV7vwEwbRTcN5msp9Gt/OsAV0+9RWT455gARO?= =?us-ascii?Q?L7Sf638dNsW81K3PYg+o9yQ3x8SE4QVBWhietIGYgF4xmjbbNmDQXWRPTeuY?= =?us-ascii?Q?cq1/ZHIWP92MW/HYCsfIA7fM/WODA2VnXkxz2dfMJk5G6RlDp/nq93dFUlQH?= =?us-ascii?Q?lwKhHWOhzb/g+j5eoqNb3lT8sdpDsbrdcWNQ1H+P7Y+EeHursaOyV51NJtnX?= =?us-ascii?Q?YrNbrh0mbm2LP1xz55TbeYFTxpY4uefCyK5CBg6Lc+wbwaynAiLW0bS71DhT?= =?us-ascii?Q?tlnKyq8j+18LFY0YpeeHnTRj0lzy45v22SefDE5MPjj+Oru+Qlnbcagn5L0k?= =?us-ascii?Q?7fn1vE7tb1O9U6ecuWTtWHa2CeSfXUCGMdOSGsWVo280FJZvF1HJDyYbPdV5?= =?us-ascii?Q?jVg1PxeUER23D6tBXt8FXKKTRIKBNpA5xpd3pXFknhqohPE1eF/r7XvDztqq?= =?us-ascii?Q?lJextSMH3b1npMdb0MfIHl3h5H/638f9OldA7U+RfKD3W+X+Jam/mIeLejz7?= =?us-ascii?Q?mhy+Yxj7eOCByx46DT3Zw1vdottorVHPZUEsyOD0IkuGPGQlvHDUxCv0RuBv?= =?us-ascii?Q?Ti7PkCOGS542lkiyHmixprj3/fERGle7GTXjaWMxZlPx8sTz8EZPnryYmoi9?= =?us-ascii?Q?fVBkPSIznVNERQLPOUtXO+D1KDZlhWdvbtfbTZVdqHTK4+acPCUgVDJ7h2tu?= =?us-ascii?Q?4rAVgXdF1pkpnVS/1/AlfOBUedfftJcQrVc9CAcparXzb8E/2VGhSSGflr6K?= =?us-ascii?Q?vR/YSnc2jStA3FbOwhxXESEs3VklJBAi7VwdPOzd31ap6OqGF0RQoXRSuj7S?= =?us-ascii?Q?1X4rVNLa8nKe7gDDuPnltRiLhqeWJcHNwv0AOG6Y0cnmQYI/X1iFjTiGZeu/?= =?us-ascii?Q?K3Z+WUL+Z8pewZO6iQ0oDZknM4g+ikrzmY7vXBFbwBB0FjqLpYRGAsmDnp54?= =?us-ascii?Q?1+TjXUrVltXtn2qwwbFm319xrhxQ8nGwMUk5N64HfD/MEJ+a9LPtq4wqmocF?= =?us-ascii?Q?Q/oA1ZjdOHMMx9vPzNG5lKgIYecEjDcKZ9MhwFke?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: S+KgviY2kGuN2YaO6YGtLrspOo5Sm1lUBoaBBqASQJjplZYIRDwtSOE+skJiQNMIL1NC60Uiqs9mUHaHBGZLsMJQw2gliAfVB2JSx8hGeqmXaU9S2fRHkrHB5g2wP+4P6ligeipXYp5LTbjHqJV47+Ht5XJ1Joa+nVNwYMCr7SohLXoaWVW2wlGoMtTC2mUlv02dGa9/rQpzMG1i7G52DG290Qjv+s4aHU/uONIRb6F4cjGaWC9Sfq9LBX0MvM+R0GR2D3OHfbGXErokfwJWpF2kWbW9fSRXdInWoS/cY0/MHD/Zop2GDx93aySEFPYJBNj8pQNoX/F7R8wDP+/zKBD1ycNodWIe6YkmHcrGFlyx/i23/wkwToiUUeBjH52YIp2dx1wtUw3zp42qlcY6QscR2a0+8OIm3/+9/Ezhb/TZkrwr9kvQWOlbZ206FKEjYZ5xmRlmTpXBOZ8O8cqHuyQ1C9YpoXgFL5jUNGdoTxEV3PrsdS5I2uORFMoMMaa/AbdPwRNXNhWp/HczKKwbLmGo8t7Y3Id3fZUmuNhVRkqyiL4upBCYVVI2iC5E2OL+7KQPvpXnpWxvEH5l9sRFMpBQ7zJbz7uNQHcutbMeWbCTPnbqjPiKfP+QlNxum2Ho X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 626403fe-3ad3-40cc-d4ab-08dd6af26236 X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Mar 2025 16:39:01.8659 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: m+CfPEMnKDfGlcReNreIPXywWLidYmvvHSoBUYVlOhcdVyBlUwZwse0Ypk7tmyJSD4Xsykx3+o5JGRjcOHNmEA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR01MB8136 X-Proofpoint-ORIG-GUID: JyvNxwFJ2MgAlap4e8gOocatff5BSoJI X-Proofpoint-GUID: JyvNxwFJ2MgAlap4e8gOocatff5BSoJI X-Authority-Analysis: v=2.4 cv=DslW+H/+ c=1 sm=1 tr=0 ts=67e18aaa cx=c_pps a=5L1ZokDb34JZJib1CqSWiA==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Vs1iUdzkB0EA:10 a=H5OGdu5hBBwA:10 a=QyXUC8HyAAAA:8 a=T2Tk2prtAAAA:8 a=jQOgFn-ZAAAA:8 a=jZVsG21pAAAA:8 a=8rWy6zfcAAAA:8 a=LEE67eovihbJKGrO1zMA:9 a=CjuIK1q_8ugA:10 a=vKVseUoFqw0A:10 a=-3WWkWTrn60A:10 a=Q9KPJbe3kf6KyAHjlbO0:22 a=mT82qxFQzDvLIExZS32s:22 a=3Sh2lD0sZASs_lUdrUhf:22 a=YjdVzJdQTyZRADMV7wFX:22 cc=ntf X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 mlxscore=0 impostorscore=0 phishscore=0 mlxlogscore=999 lowpriorityscore=0 priorityscore=1501 adultscore=0 suspectscore=0 bulkscore=0 spamscore=0 clxscore=1011 malwarescore=0 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2502280000 definitions=main-2503240120 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, I have both the primary and secondary ops_name (s) and they are different. The two files are identical, except for these: PRIMARY: ops_name: common_pool_count=3D1024 SECONDARY: ops_name: common_pool_count=3D0 [root@localhost ~]# diff prim_mempool_dump.txt ex_second_mempool_dump.txt 15c15 < ops_name: --- > ops_name: 149c149 < common_pool_count=3D1024 --- > common_pool_count=3D0 Should the ops_name be the same among the primary and secondary processes? I build out application in our build environment and the dpdk-simple_mp is = from the dpdk sdk (meson/ninja). Thanks, Ed -----Original Message----- From: Kompella V, Purnima =20 Sent: Monday, March 24, 2025 11:00 AM To: Lombardo, Ed ; Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. Hi, I am not sure if this could be your problem, but we had run into a similar = issue with mbuf pools. Do you list the DPDK libs in the exact same order when linking Primary and = Secondary process? If not,=20 the OPs to which the ops_index is pointing to in Primary and in secondary = is different. Due to this, rte_mempool_ops_dequeue_bulk called from Secondary does not m= atch the OPs that Primary had used to populate this memPool (during create= ) So, accessing the pool from Secondary appears like the pool is empty but n= on-zero in-use Count. This happens because each DPDK lib "appends" to the OPs list on the fly as = it is linked by the linker.=20 So the order of OPs in the OPs database is a mismatch between Primary and S= econdary. Primary and Secondary share information about OPs using ops_index= , but since the order is different, ops_index=3DX in Primary not the same O= Ps in Secondary. And this leads to the segmentation fault.=20 One of your processes is showing ops_index=3D1, ops_name: . Can you confirm it is the same in the other process also? By OPs, and ops_index, I am referring to this internal data structure of DP= DK fw.=20 struct rte_mempool_ops { char name[RTE_MEMPOOL_OPS_NAMESIZE]; /**< Name of mempool ops struct. */ rte_mempool_alloc_t alloc; /**< Allocate private data. */ rte_mempool_free_t free; /**< Free the external pool. */ rte_mempool_enqueue_t enqueue; /**< Enqueue an object. */ rte_mempool_dequeue_t dequeue; /**< Dequeue an object. */ rte_mempool_get_count get_count; /**< Get qty of available objs. */ /** * Optional callback to calculate memory size required to * store specified number of objects. */ rte_mempool_calc_mem_size_t calc_mem_size; /** * Optional callback to populate mempool objects using * provided memory chunk. */ rte_mempool_populate_t populate; /** * Get mempool info */ rte_mempool_get_info_t get_info; /** * Dequeue a number of contiguous object blocks. */ rte_mempool_dequeue_contig_blocks_t dequeue_contig_blocks; } __rte_cache_a= ligned; Also, the link provides some insight into how OPs is significant. https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/develop= er/articles/technical/optimize-memory-usage-in-multi-threaded-data-plane-de= velopment-kit-dpdk-applications.html__;!!Nzg7nt7_!ElXZ6jVzd3ey8732oohMXbI9i= JA_a5BF67ev7sypu4Vf8LAfVMQzZiEReuDIb-jaLhNO6PswM4RTXVt-qxmR3d_493_Z2SSC$=20 Regards, Purnima -----Original Message----- From: Lombardo, Ed Sent: Monday, March 24, 2025 10:32 AM To: Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue CAUTION: This message originated from an External Source outside of CommSco= pe.com. This may be a phishing email that can result in unauthorized access= to CommScope. Please use caution when opening attachments, clicking links,= scanning QR codes, or responding. You can report suspicious emails directl= y in Microsoft Outlook. Hi, Further debugging has left me clueless as to the problem with getting the f= irst Obj from the mempool on the secondary process to send to the primary p= rocess (my Application). PRIMARY: My application side (primary process) EAL Arguments: "--proc-type=3Dprimary", "--file-prefix=3Ddpdk_shared", "-l 25,26,27,28", "= -n4" , "--socket-mem=3D2048", "--legacy-mem", Create the _MSG_POOL t_message_pool =3D rte_mempool_create(_MSG_POOL, // Pool name t_pool_size, // Number of = elements in the pool STR_TOKEN_SIZE, // Size of ea= ch message t_pool_cache, // Cache size t_priv_data_sz, // Private da= ta size NULL, // mp_init NULL, // mp_init_ar= g NULL, // obj_init NULL, // obj_init_a= rg 0, // rte_so= cket_id(), // socket_id t_flags // flags ); The t_message_pool pointer value matches the secondary process when execute= " message_pool =3D rte_mempool_lookup(_MSG_POOL);" NTSMON: send ring (0x1bfb43480), recv ring (0x1bfb45a00) and message pool (= 0x14f570ac0) are created successfully. SECONDARY: Secondary process I execute: " # ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file= -prefix=3Ddpdk_shared --proc-type=3Dsecondary --" Notes: * hugepages are created 2x1G on NUMA Node 0 [root@localhost ~]# /opt/dpdk/dpdk-hugepages.py -s Node Pages Size Total 0 2 1Gb 2Gb * --file-prefix=3Ddpdk_shared is provided to both primary and secondary * --proc-type is correctly defined for both primary and secondary process. * The secondary process reports correct socket=3D0 (See dump command output= below) * The secondary process showed Available mempool count is 0 and in-use coun= t is 1024 (which looks incorrect). * Secondary process reports mempool is empty * Secondary audit passed (rte_mempool_audit()), no panic occurred. * Tried disabling ASLR * Tried turning off legacy-mem EXECUTE Secondary process application "dpdk-simple_mp_dbg3" [root@localhost ~]# ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file-prefix=3Ddpd= k_shared --proc-type=3Dsecondary --legacy-mem -- EAL: Detected CPU lcores: 128 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/dpdk_shared/mp_socket_221890_1ba056= 2a6ae95 EAL: Selected IOVA mode 'PA' APP: Finished Process Init. APP: Availble mempool count is 0, in-use count is 1024, (mempool ptr=3D0x14= f570ac0) APP: is mempool empty (1) or full (0)? APP: check audit on message_pool APP: Secondary mempool dump file write APP: NO Objs in message pool (_MSG_POOL), exit the app simple_mp > Starting core 31 simple_mp > send hello message_pool pointer is 0x14f570ac0 Failed to get msg obj from mem pool: Success (ret=3D-2) EAL: PANIC in cmd_send_parsed(): Failed to get message buffer 0: ./dpdk-simple_mp_dbg3 (rte_dump_stack+0x2b) [b1ee1b] 1: ./dpdk-simple_mp_dbg3 (__rte_panic+0xbd) [525ede] 2: ./dpdk-simple_mp_dbg3 (cmd_send_parsed+0x2d8) [7ceb68] 3: ./dpdk-simple_mp_dbg3 (400000+0x6a5f46) [aa5f46] 4: ./dpdk-simple_mp_dbg3 (400000+0x6a4f20) [aa4f20] 5: ./dpdk-simple_mp_dbg3 (rdline_char_in+0x34b) [aa841b] 6: ./dpdk-simple_mp_dbg3 (cmdline_in+0x71) [aa4ff1] 7: ./dpdk-simple_mp_dbg3 (cmdline_interact+0x30) [aa5110] 8: ./dpdk-simple_mp_dbg3 (400000+0xfe5cb) [4fe5cb] 9: /lib64/libc.so.6 (7f2d76600000+0x3feb0) [7f2d7663feb0] 10: /lib64/libc.so.6 (__libc_start_main+0x80) [7f2d7663ff60] 11: ./dpdk-simple_mp_dbg3 (_start+0x25) [7ce795] Aborted (core dumped) The rte_mempool_dump() is: mempool @0x14f570ac0 flags=3D1c socket_id=3D0 pool=3D0x14f56c8c0 iova=3D0x1cf570ac0 nrte_mempool_getb_mem_chunks=3D1 size=3D1024 populated_size=3D1024 header_size=3D64 elt_size=3D64 trailer_size=3D64 total_obj_size=3D192 private_data_size=3D0 ops_index=3D1 ops_name: memory chunk at 0x1bfb43180, addr=3D0x14f53c780, iova=3D0x1cf53c780, len= =3D196800 avg bytes/object=3D192.187500 internal cache infos: cache_size=3D32 cache_count[0]=3D0 ..... cache_count[127]=3D0 total_cache_count=3D0 common_pool_count=3D1024 no statistics available Any guidance is appreciated. Thanks, Ed -----Original Message----- From: Lombardo, Ed Sent: Friday, March 21, 2025 2:19 PM To: Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue Hi Sephen, Thank you for your help. I made good progress up to now. When I try to use the dpdk_simple_mp application to send a message to my ap= plication I get a Segmentation fault. First, I re-verified the dpdk_simple_mp process=3Dprimary and dpdk_simple_m= p process-secondary does pass messages successfully. So, my hugepages are = created and DPDK initializes successfully on both at startup. In my application I created the send and recv rings and message_pool as the= primary process. The logs I added do not show any errors. Once my application starts and settles I started the dpdk_simple_mp applica= tion: # ./dpdk-simple_mp_dbg -l 30-31 -n 4 --legacy-mem --proc-type seconda= ry -- However, on the dpdk_simple_mp side do "send hello" and I then get a segmen= tation fault. The debugger takes me deep within the dpdk libraries which I am not too fam= iliar with. The rte_ring_elem.h file function: rte_ring_dequeue_build_elem() is where I= end up with segmentation fault. I notice that the variables are optimized= out, not sure why since I built the dpdk libraries with debug flag. Here is the back trace and could you point me in the direction to look. # gdb dpdk-simple_mp /core/core.dpdk-simple_mp.241269 warning: Unexpected size of section `.reg-xstate/241269' in core file. [Thread debugging using libthread_db enabled] Using host libthread_db libra= ry "/lib64/libthread_db.so.1". Core was generated by `./dpdk-simple_mp -l 30-31 -n 4 --legacy-mem --proc-t= ype secondary --'. Program terminated with signal SIGSEGV, Segmentation fault. warning: Unexpected size of section `.reg-xstate/241269' in core file. #0 0x0000000000cf446a in bucket_dequeue () [Current thread is 1 (Thread 0x= 7f946f835c00 (LWP 241269))] Missing separate debuginfos, use: dnf debuginfo= -install elfutils-libelf-0.189-3.el9.x86_64 glibc-2.34-83.0.1.el9_3.7.x86_6= 4 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libpcap-1.10.0-4.e= l9.x86_64 libzstd-1.5.1-2.el9.x86_64 numactl-libs-2.0.16-1.el9.x86_64 opens= sl-libs-3.0.7-25.0.1.el9_3.x86_64 zlib-1.2.11-40.el9.x86_64 (gdb) bt #0 0x0000000000cf446a in bucket_dequeue () #1 0x00000000007ce77d in cmd_send_parsed () #2 0x0000000000aa5d96 in __cmdline_parse () #3 0x0000000000aa4d70 in cmdline_valid_buffer () #4 0x0000000000aa826b in rdline_char_in () #5 0x0000000000aa4e41 in cmdline_in () #6 0x0000000000aa4f60 in cmdline_interact () #7 0x00000000004fe47a in main.cold () #8 0x00007f946f03feb0 in __libc_start_call_main () from /lib64/libc.so.6 #9 0x00007f946f03ff60 in __libc_start_main_impl () from /lib64/libc.so.6 #10 0x00000000007ce605 in _start () Gdb - stepping through the code, gdb attached to dpdk_simple_mp_debug (gdb) 0x0000000000cf42c5 in rte_ring_dequeue_bulk_elem (available=3D, n=3D, esize=3D, obj_table=3D, r=3D) at ../lib/ring/rte_ri= ng_elem.h:375 375 ../lib/ring/rte_ring_elem.h: No such file or directory. (gdb) p r $17 =3D (gdb) p obj_table $18 =3D (gdb) p available $19 =3D (gdb) n Thread 1 "dpdk-simple_mp_" received signal SIGSEGV, Segmentation fault. bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0x14f= 05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 191 ../drivers/mempool/bucket/rte_mempool_bucket.c: No such file or dir= ectory. (gdb) bt #0 bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0= x14f05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 #1 bucket_dequeue (mp=3D, obj_table=3D0x14f09b5c0, n=3D33) = at ../drivers/mempool/bucket/rte_mempool_bucket.c:289 #2 0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=3D= , obj_table=3D0x14f09b5c0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:793 #3 rte_mempool_do_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7= fff8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1570 #4 rte_mempool_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7fff= 8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1649 #5 rte_mempool_get_bulk (n=3D1, obj_table=3D0x7fff8df066f0, mp=3D0x14f05ed= 40) at ../lib/mempool/rte_mempool.h:1684 #6 rte_mempool_get (obj_p=3D0x7fff8df066f0, mp=3D0x14f05ed40) at ../lib/me= mpool/rte_mempool.h:1710 #7 cmd_send_parsed (parsed_result=3Dparsed_result@entry=3D0x7fff8df06790, = cl=3Dcl@entry=3D0x2f73220, data=3Ddata@entry=3D0x0) at ../examples/multi_process/simple_mp/mp_commands.c:18 #8 0x0000000000aa5d96 in __cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf= =3D0x2f73268 "send hello\n", call_fn=3Dcall_fn@entry=3Dtrue) at ../lib/cmdline/cmdline_parse.c:294 #9 0x0000000000aa5f1a in cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf=3D<= optimized out>) at ../lib/cmdline/cmdline_parse.c:302 #10 0x0000000000aa4d70 in cmdline_valid_buffer (rdl=3D, buf= =3D, size=3D) at ../lib/cmdline/cmdline.c:24 #11 0x0000000000aa826b in rdline_char_in (rdl=3Drdl@entry=3D0x2f73230, c=3D= ) at ../lib/cmdline/cmdline_rdline.c:444 #12 0x0000000000aa4e41 in cmdline_in (size=3D, buf=3D, cl=3D) at ../lib/cmdline/cmdline.c:146 #13 cmdline_in (cl=3D0x2f73220, buf=3D0x7fff8df0c89f "\n\200", size=3D) at ../lib/cmdline/cmdline.c:135 #14 0x0000000000aa4f60 in cmdline_interact (cl=3Dcl@entry=3D0x2f73220) at .= ./lib/cmdline/cmdline.c:192 #15 0x00000000004fe47a in main (argc=3D, argv=3D) at ../examples/multi_process/simple_mp/main.c:122 Appreciate if you can help. Thanks, Ed -----Original Message----- From: Stephen Hemminger Sent: Wednesday, March 19, 2025 7:17 PM To: Lombardo, Ed Cc: users@dpdk.org Subject: Re: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Wed, 19 Mar 2025 21:52:39 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > I added the fib library, but I now see there are many more dpdk libraries= I need to add. Is this typically the case with the example files working = with primary DPDK application? > > I am using meson and ninja to build the examples, but I don't know how to= know the library dependencies. > > How do I learn ahead of building my Application as to what extra librarie= s I need to include for the DPDK example to work? > > I am doing incremental build-test-find_missing_library. > > So far, I needed to add these: -lrte_fib -lrte_rib -lrte_stack=20 > -lrte_member -lrte_efd > > Thanks, > Ed The typical case is to make sure that primary and secondary are built with = the same libraries.