From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E69BD4647B for ; Tue, 25 Mar 2025 15:39:24 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B96B340277; Tue, 25 Mar 2025 15:39:24 +0100 (CET) Received: from mx0a-00196b01.pphosted.com (mx0a-00196b01.pphosted.com [67.231.149.170]) by mails.dpdk.org (Postfix) with ESMTP id A7E2A40276 for ; Tue, 25 Mar 2025 15:39:22 +0100 (CET) Received: from pps.filterd (m0096263.ppops.net [127.0.0.1]) by mx0a-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 52PEU2pA010338; Tue, 25 Mar 2025 10:39:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=1fzZFzEYn3fVP3YXDMNpEctz8qnCW0fv/LLq /Y/O2gA=; b=I2/Bu7yoGa3+wBpDYBZrmlLIA2QNSHvXlipvV+3jWEPMtowMaPv4 M9BsgjTetVYQmZeYNfgKWIcdwwwXQpHbbXDO/z3jl3A4F1r5kRhxC9qDlc9Up5qd GGaE6nXEeh5jVVHwXSeVyj6EH1Cbf2JMIlveTdt1HwjDfSBJVjXEeb1ETQoS7O7L mJtV8jvsPhkWrNmMY9Fhx4v2C15V0BVUOGT+AQ+X1637uLwmksK2AVZHjteHZrV/ 8mtXIUY3D1UIjdiVaKCLfkJNytzj6DusDqZIhw1llPhKyjDz29yKPLLfcEogIA1A qNLye+56aSgshO8GZ6MnG/1QuHOTP0Wuwg== Received: from cy7pr03cu001.outbound.protection.outlook.com (mail-westcentralusazlp17012034.outbound.protection.outlook.com [40.93.6.34]) by mx0a-00196b01.pphosted.com (PPS) with ESMTPS id 45kx9n00ce-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 25 Mar 2025 10:39:19 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BvwSzNApz9GGnIgCgwai2ySqj03huJsFsjJr3KTdhJzhVIKusmIgcyb7FhOo40R70u4ArasScHOM0IyGhJXBakZovu8TAbZfPK1suSCB2FCkF76aNocT0j7Ig3zT7E3JPrUmMJrCWmTg+GiZ/IdMLmCKb1xRiyyl7n5rMLmhkaNkZIFr3Sey9NBWoZ87KqlXun/MfOHZIDvwbSY7BD/+LromQfsTC+wDYeYqyaljTkW960iZdJELuGcoj3Epmq3xuivHVm+GzI5Smw2e5uaQluRxf0OzDa4D0wyQ5JcBdY8JeCbg6/x/d+HbYMpSgC0dCKSmqLyvbPj9K1ikoC0BNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1fzZFzEYn3fVP3YXDMNpEctz8qnCW0fv/LLq/Y/O2gA=; b=kxv+JHAmEPsfTYr6P9Jlp158wcM/QTVLy+Tf6JEiEqFg10y4F7H/RRh3xL0IQMA6521isAHSMcEMm61hanTZket2Q1v+G9jyADHAusGSMW2ivURJ9Kym5Iyk8ZPkOq/5ULxJp9P1qO/WBqRGcajfot9h4oEgsLY1oGsa/T+758lgI4j/ZMqzdFIh/nRLT6NrBcu9jvdldO40JsT6fy4kEJgLG7R/AGVji3o/t2r2o7blQaN6gJWrMVmTbijDGMPQxjPFyYEtGcZ37HeBG6usN6Ke34TdtjTmxdwnx6hzbrRxwxHK4nzSCglD1t4TNJgGfqIXEn5PlfCXqZG9v08tnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by BN0PR01MB6830.prod.exchangelabs.com (2603:10b6:408:14a::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.43; Tue, 25 Mar 2025 14:39:08 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%4]) with mapi id 15.20.8534.042; Tue, 25 Mar 2025 14:39:08 +0000 From: "Lombardo, Ed" To: "Kompella V, Purnima" , Stephen Hemminger CC: "users@dpdk.org" Subject: RE: tailqs issue Thread-Topic: tailqs issue Thread-Index: AduY9RY0Q86kHIIzTwW3RWYKe22BSAAF7xuAAAKqzZAAA2FugABZg8GQAHdI0PAAGJ/dMAAD+cagAB3RdvAAEF8XwA== Date: Tue, 25 Mar 2025 14:39:08 +0000 Message-ID: References: <20250319132349.5ff339a7@hermes.local> <20250319161659.573e9660@hermes.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|BN0PR01MB6830:EE_ x-ms-office365-filtering-correlation-id: 9c5250a4-02a8-4eee-9868-08dd6baaccfc x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|376014|1800799024|366016|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?zKI0uv9RpfA3ugA8p5kkrdSh3s8ZvcM4lzmirBfqS5whl1BUjoYKQUjkYPjK?= =?us-ascii?Q?EqyvFNyhj3f33IOjeczr6FHEDOoe9XElPYqh2JaqQaNakQmyHgHcT8ZWNJS4?= =?us-ascii?Q?jtr6TKQ29dbhkvBe50G2O2U8VGwmgJ/TeRkEIRAYeLZMi+NLEs0oRjSvzvSk?= =?us-ascii?Q?ESB9eQPWW7EAUc2PLi73G7n0+07XRuHv1WVcF32AFWiCfM/hT5E6927b1H27?= =?us-ascii?Q?xZlfiO1gMI2KaaL7cD4Ge/WWZeSwJu7O00LmlaU4PAtzT/Z1jYpqarSBycTY?= =?us-ascii?Q?piOX03DlIB4ID19MqmdusFEvF+m++i1WojDzaliX+Wz0CjjiNXelHqsZ14ns?= =?us-ascii?Q?PckHLRaQ1kN1+Om+S2qi2ZXyLQaWE4ijxsB3zDao0TY00F/6OcoOPaBdWzP8?= =?us-ascii?Q?m5KBdQ9H1lH/Foqp5dodjoR80un4VYMyhyBw9u59qeQw3AcwvUDXUXe/7i1R?= =?us-ascii?Q?GW775MNS503c7Aq+qoUyNHMWpMY+3Qc7IVIfO+8xRu+UzJs/neZ47vRnBpmF?= =?us-ascii?Q?Q7qixBCcxIx15Fhq3jj+9kkY09AOsSKHfgWXF7BQcnkR23W93c7TGJwL5B9I?= =?us-ascii?Q?CbqJaVVayZb6Onp8PGyZpHATa/CGfHVuANt67CuUnEwjblTyi232RuQ+h7q3?= =?us-ascii?Q?qmZu0BJzTSlI0xnNyt14/zZw3bIZAeoPCCj8VCcyDtrxj4R8tDRJZWqR7Z0+?= =?us-ascii?Q?c6ESdXFIqc0MX8ccSRvCHDvYzPzTIKjduvqcbePrnmHUFQErJSFtOu5xFcrl?= =?us-ascii?Q?Lf8bMaCBGwOkhjpXwty0j5CY/dywYdOHbi2AbRk1HaX8QVBRgne0nlWMi/uW?= =?us-ascii?Q?CszyUdpMtSB9/Ws390/4l21L1PH5pSxPo1T1J9iGazeuoqMvHpqZB4cu32pO?= =?us-ascii?Q?AE+2aeVQRMxgci+cc4z/M11JQFiQSTK/5gpLosL5jFeUwHy0MdrqZ6cbqWAE?= =?us-ascii?Q?LHTrRREF5vLYTvokOL3dMJFWM1tAM/oDfLL5inMzf3JGyt5eWL129oPQroZR?= =?us-ascii?Q?fFhhfAMBSewwcONOAfkZGw/Y3Rf8Wpg3mW52oNXfFPIvI3p7IBcPafw27vJh?= =?us-ascii?Q?SYs0gsp8M1OPcEkKl19kejeXZFj3n4rAf88K8tNvRA93Sg1aNzujc2A7Jmed?= =?us-ascii?Q?7XYe6FkNNcej4SNTnaqxDZVYR9g14XsZjQVH678vglAG7jyhFss8Y65cX7+N?= =?us-ascii?Q?jwBjLS8V8K7XKpY7NfoZwN2wPFjtbCs9WLqHkkgpZjk/R00dUIqMxEo1p+EM?= =?us-ascii?Q?xv+CUGpVGytVfgYbfsqTt61hu3OfmKtShszFBjWO9HMZOwWYaTmgwmGs2xgJ?= =?us-ascii?Q?OoWwdavZ2Ko2JTQJ4tfU6UoQnl3tZfXPGs9e8cuhjvZ88ll7p31DU3Q8sKwm?= =?us-ascii?Q?VXJ+Nm4CZNy4zuMg8MhLwwqvNn5Iw67PNI7gYBDutoE9MgxvtOjvk9UABlcK?= =?us-ascii?Q?9tZjlSm01clfbzV/A5aWe+V8FfSd5fDm?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(1800799024)(366016)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?VMe/lntjyprMCFwjmiQFKQsOsKhcuEtzyOaEcDaIn8qMLP0jqc2gGuzKMbee?= =?us-ascii?Q?YvgtMhHJBB/AcyvTPfvS6gZTdAW41OVefO0Lg4ZRRPWOYcyQ2+WfOZuCaHEX?= =?us-ascii?Q?wXa/wB83hGjWgrI2rCWm/ocfFSVlPRNHz+xM17+4qm0cF2xblgrE8Y/aS9LQ?= =?us-ascii?Q?2umXxrAzpVo2nmTytiaXbgWfvY0eboh24SqhM2Eg2KhcgaKLSYENQLSdwJnz?= =?us-ascii?Q?IXawmHeb09BP1kOuXK5awCFQHD+NHtRlP3Caq4kAB6Zsgcve2iH1Sy6wDU1R?= =?us-ascii?Q?AppVHFVQf6NMDcfOE+KpCTFzwvt0uhqmGHwaiisBqouKxKrSiWq6jq5mHaAX?= =?us-ascii?Q?SN33U5n00Bri4qAjHwew5ihy72lXxWkmhYzufKgYiWGGYdyzhcyoVjmGnw93?= =?us-ascii?Q?NINoAASdVbGtaMvHhnu+S5UQtDomeTt4EcBGNzcPPi30Gq/lDW60Q9UiASgH?= =?us-ascii?Q?dsTtk51abpwMj4ORp0LibqTq3eIUlxhk32AHVwI182Zyr+JlFi8EDF6W67R6?= =?us-ascii?Q?2pheJWhpDZIiVeLnNrJV5xDefSyu/rNmTR/g+SmCrQG+/PFfpzAvJsDVSNn3?= =?us-ascii?Q?xCEV0AtL3OHS0brLrW6V16FhKyHD3O0g2kjhq7e1WyfmntqbeX6nE+ezh5Ht?= =?us-ascii?Q?FmGM7IyZdhhpdxF2zQWd3EYmT1vY8vzcFzhUxk3fmXF9ZI56fExhBB+va7Qq?= =?us-ascii?Q?7V6Wj0DppqO8eb1VEU7bW/FJfh1pnnKDvfcWfIJnGcGTx1eWjiwhh5EtmCfD?= =?us-ascii?Q?SiQTn0ZyjDer9ngp/Cj/BpaP9iI1GLE8pwTYDDoel33aP4X7ioomzsJ5Y+IC?= =?us-ascii?Q?hsn4Z6UFybdao4q+M3snhy9RgBmRq4lAlNTvjAPu0VAuM/ED1cjnGHi1iiQu?= =?us-ascii?Q?DC8vqWnOJdAk1RL01OY/y4VtOXFIuqkvLUiBAnKXEYABPk+dMnLTbRP8ayw9?= =?us-ascii?Q?3FSFMrGJJsqj52CQreJBiapZA0fuoqYoVyL1IGZ1B09zNC32ocKvhTwxCBoO?= =?us-ascii?Q?2V54OUGbaCbMvfwqu+G0JofMuZ34j7e1IHugYhuhsi/e1BJwPzVuyt64nxAs?= =?us-ascii?Q?BlpZgq1QiwiP3OjWUvs6dIt8RiCstrEeOiLF02Lrldded+/IwDJOcubYS4Qk?= =?us-ascii?Q?wVF1unS28Z6UAGzQNRmww5JoahalAN12WgwfdxAT32jESRPfq0mpSL/cQVVQ?= =?us-ascii?Q?rekzwpNrW9dy93QJQ0zithpjb2qqo7QbMojRbSk6G6HRm9e0CvX8tja/bAho?= =?us-ascii?Q?M0BhHbKc23qemf1+xHm7KHiFqJNLvn1zJOYOX02O0h52oEFafGKu50cyAeAO?= =?us-ascii?Q?zzoZXVmCr7M0+Xb1NLAIxWItIs3jLCmA/iBgmXrFYKuRrLiQqBxYXtH3U7vz?= =?us-ascii?Q?oC7SV71vyp49X7xGaQcXsNcDZ7BBypa3IPwkaRezDOlhhpbG3J6pGwPSAv3H?= =?us-ascii?Q?h92xPaL9HxrA8QQnrKXERjogL7ua7RZlUpEWN0IrGeEwWp6jqhS8Zd2KUjwx?= =?us-ascii?Q?ilSJns9IxSosN7OebzUyG2y0ydl1qVLMXBO0XorDd2hl+hpU/FCrJ9VDhtMy?= =?us-ascii?Q?VQ+xntSt5tSAPfIVfRKX+0YdXn4unX+CxZu4RvR6?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: rkkR7RDxnCVMkotRoc4P72BqmU3ubHl67y2IxxlB/37JrZB4RhS24i8+TQBM55jrOsZ1YYFBDCXnYAkIkQFaTYIqcCc+vboRWZA2aDBdsjYEg+H4UKv1DqUDjTlrXuSURVI21e7Xt7riWfskFQWpFnrgQkovZWI9QH2aLGP9jQP8PjSbX5Q4Alt6CTLkL7g1jW3c/2B/yVVVG+e1Lu/q8JJs84mSYtqkJjyXObgqyRuDLh9I481FViDZVu+rLjIbkd8IhiKlBd/ZRvPAy+Zp28Idw342mqe9RYFwOd/7HkPHpKyU2ZSTO03k9VSywLBFRzsl0kHrx91tPvO/l79PDg+658uLe1ecw3EYX4fN0zZlFl74sXMkJGwQk2NHb1nK5T7NSoaPd5KiWEpgow7SOZ06QkJo9pmEhiZfSGqoP1Ui9MAm0SWGcaFpk524scbfYMWI43MTzzFiCZSKqds0DZqfeWWDcEMRCo9M+Q9puPokgA0CQMIxEimZtnKRphM7h8pViRYzV2U4iCDjsZxfOlgeh/w7OW2r0zNXBpJG74Ambz9twMVwFS0OKmeOVLjbqprXssT2XA3f+VfuN5PrCzkLegarvbR23jL8TUlGGx7B4PFbRZyqBwY5Zab6Uqdh X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9c5250a4-02a8-4eee-9868-08dd6baaccfc X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Mar 2025 14:39:08.3806 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: UEd2CTNAvLnBR9JgtO3dkQvj3lVoSqDaAQCVEUmhPWjiuYoxTN+PvjNtJRsr+0mV68PlnZG5W0JdT0rqV/BjUQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR01MB6830 X-Authority-Analysis: v=2.4 cv=JrzxrN4C c=1 sm=1 tr=0 ts=67e2c017 cx=c_pps a=jgLrbg7RBTw9j6qPRKhXvw==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Vs1iUdzkB0EA:10 a=H5OGdu5hBBwA:10 a=QyXUC8HyAAAA:8 a=T2Tk2prtAAAA:8 a=jQOgFn-ZAAAA:8 a=jZVsG21pAAAA:8 a=8rWy6zfcAAAA:8 a=vhVYTUtfZsunyh5bEfsA:9 a=CjuIK1q_8ugA:10 a=vKVseUoFqw0A:10 a=-3WWkWTrn60A:10 a=Q9KPJbe3kf6KyAHjlbO0:22 a=mT82qxFQzDvLIExZS32s:22 a=3Sh2lD0sZASs_lUdrUhf:22 a=YjdVzJdQTyZRADMV7wFX:22 cc=ntf X-Proofpoint-ORIG-GUID: DHZNFtSSLspXbHNtqXzIKqpKaQR_wgwW X-Proofpoint-GUID: DHZNFtSSLspXbHNtqXzIKqpKaQR_wgwW X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 lowpriorityscore=0 impostorscore=0 suspectscore=0 phishscore=0 priorityscore=1501 mlxscore=0 mlxlogscore=999 bulkscore=0 spamscore=0 malwarescore=0 adultscore=0 clxscore=1015 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2502280000 definitions=main-2503250103 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Purnima, I will try your suggestion, but this seems weird. What if I have a 3rd par= ty application that I want to integrate with our application. This could b= e impossible to coordinate this requirement. Thanks, Ed -----Original Message----- From: Kompella V, Purnima =20 Sent: Tuesday, March 25, 2025 6:26 AM To: Lombardo, Ed ; Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. Hi Ed, GDB BT seems to indicate one of your processes (that one that suffered segf= ault) uses OP, not or as you have= listed! #0 bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0= x14f05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 #1 bucket_dequeue (mp=3D, obj_table=3D0x14f09b5c0, n=3D33) = at ../drivers/mempool/bucket/rte_mempool_bucket.c:289 #2 0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=3D= , obj_table=3D0x14f09b5c0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:793 #3 rte_mempool_do_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7= fff8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1570 In any case, can you try by keeping the "order of dpdk libs" exactly the sa= me when compiling the Primary process and the Secondary process? This way, both processes will get the same OPs database and hence ops_index= =3DX will refer to same OPs from either of them and hopefully things should= work fine. Thanks, Purnima -----Original Message----- From: Lombardo, Ed Sent: Monday, March 24, 2025 10:09 PM To: Kompella V, Purnima ; Stephen Hemminger= Cc: users@dpdk.org Subject: RE: tailqs issue CAUTION: This message originated from an External Source outside of CommSco= pe.com. This may be a phishing email that can result in unauthorized access= to CommScope. Please use caution when opening attachments, clicking links,= scanning QR codes, or responding. You can report suspicious emails directl= y in Microsoft Outlook. Hi, I have both the primary and secondary ops_name (s) and they are different. The two files are identical, except for these: PRIMARY: ops_name: common_pool_count=3D1024 SECONDARY: ops_name: common_pool_count=3D0 [root@localhost ~]# diff prim_mempool_dump.txt ex_second_mempool_dump.txt 15c15 < ops_name: --- > ops_name: 149c149 < common_pool_count=3D1024 --- > common_pool_count=3D0 Should the ops_name be the same among the primary and secondary processes? I build out application in our build environment and the dpdk-simple_mp is = from the dpdk sdk (meson/ninja). Thanks, Ed -----Original Message----- From: Kompella V, Purnima Sent: Monday, March 24, 2025 11:00 AM To: Lombardo, Ed ; Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. Hi, I am not sure if this could be your problem, but we had run into a similar = issue with mbuf pools. Do you list the DPDK libs in the exact same order when linking Primary and = Secondary process? If not, the OPs to which the ops_index is pointing to in Primary and in sec= ondary is different. Due to this, rte_mempool_ops_dequeue_bulk called from Secondary doe= s not match the OPs that Primary had used to populate this memPool (during= create) So, accessing the pool from Secondary appears like the pool is empt= y but non-zero in-use Count. This happens because each DPDK lib "appends" to the OPs list on the fly as = it is linked by the linker. So the order of OPs in the OPs database is a mismatch between Primary and S= econdary. Primary and Secondary share information about OPs using ops_index= , but since the order is different, ops_index=3DX in Primary not the same O= Ps in Secondary. And this leads to the segmentation fault. One of your processes is showing ops_index=3D1, ops_name: . Can you confirm it is the same in the other process also? By OPs, and ops_index, I am referring to this internal data structure of DP= DK fw. struct rte_mempool_ops { char name[RTE_MEMPOOL_OPS_NAMESIZE]; /**< Name of mempool ops struc= t. */ rte_mempool_alloc_t alloc; /**< Allocate private data. */ rte_mempool_free_t free; /**< Free the external pool. */ rte_mempool_enqueue_t enqueue; /**< Enqueue an object. */ rte_mempool_dequeue_t dequeue; /**< Dequeue an object. */ rte_mempool_get_count get_count; /**< Get qty of available objs. */ /** * Optional callback to calculate memory size required to * store specified number of objects. */ rte_mempool_calc_mem_size_t calc_mem_size; /** * Optional callback to populate mempool objects using * provided memory chunk. */ rte_mempool_populate_t populate; /** * Get mempool info */ rte_mempool_get_info_t get_info; /** * Dequeue a number of contiguous object blocks. */ rte_mempool_dequeue_contig_blocks_t dequeue_contig_blocks; } __rte_= cache_aligned; Also, the link provides some insight into how OPs is significant. https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/develop= er/articles/technical/optimize-memory-usage-in-multi-threaded-data-plane-de= velopment-kit-dpdk-applications.html__;!!Nzg7nt7_!ElXZ6jVzd3ey8732oohMXbI9i= JA_a5BF67ev7sypu4Vf8LAfVMQzZiEReuDIb-jaLhNO6PswM4RTXVt-qxmR3d_493_Z2SSC$ Regards, Purnima -----Original Message----- From: Lombardo, Ed Sent: Monday, March 24, 2025 10:32 AM To: Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue CAUTION: This message originated from an External Source outside of CommSco= pe.com. This may be a phishing email that can result in unauthorized access= to CommScope. Please use caution when opening attachments, clicking links,= scanning QR codes, or responding. You can report suspicious emails directl= y in Microsoft Outlook. Hi, Further debugging has left me clueless as to the problem with getting the f= irst Obj from the mempool on the secondary process to send to the primary p= rocess (my Application). PRIMARY: My application side (primary process) EAL Arguments: "--proc-type=3Dprimary", "--file-prefix=3Ddpdk_shared", "-l 25,26,27,28", "= -n4" , "--socket-mem=3D2048", "--legacy-mem", Create the _MSG_POOL t_message_pool =3D rte_mempool_create(_MSG_POOL, // Pool name t_pool_size, // Number of = elements in the pool STR_TOKEN_SIZE, // Size of ea= ch message t_pool_cache, // Cache size t_priv_data_sz, // Private da= ta size NULL, // mp_init NULL, // mp_init_ar= g NULL, // obj_init NULL, // obj_init_a= rg 0, // rte_so= cket_id(), // socket_id t_flags // flags ); The t_message_pool pointer value matches the secondary process when execute= " message_pool =3D rte_mempool_lookup(_MSG_POOL);" NTSMON: send ring (0x1bfb43480), recv ring (0x1bfb45a00) and message pool (= 0x14f570ac0) are created successfully. SECONDARY: Secondary process I execute: " # ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file= -prefix=3Ddpdk_shared --proc-type=3Dsecondary --" Notes: * hugepages are created 2x1G on NUMA Node 0 [root@localhost ~]# /opt/dpdk/dpdk-hugepages.py -s Node Pages Size Total 0 2 1Gb 2Gb * --file-prefix=3Ddpdk_shared is provided to both primary and secondary * --proc-type is correctly defined for both primary and secondary process. * The secondary process reports correct socket=3D0 (See dump command output= below) * The secondary process showed Available mempool count is 0 and in-use coun= t is 1024 (which looks incorrect). * Secondary process reports mempool is empty * Secondary audit passed (rte_mempool_audit()), no panic occurred. * Tried disabling ASLR * Tried turning off legacy-mem EXECUTE Secondary process application "dpdk-simple_mp_dbg3" [root@localhost ~]# ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file-prefix=3Ddpd= k_shared --proc-type=3Dsecondary --legacy-mem -- EAL: Detected CPU lcores: 128 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/dpdk_shared/mp_socket_221890_1ba056= 2a6ae95 EAL: Selected IOVA mode 'PA' APP: Finished Process Init. APP: Availble mempool count is 0, in-use count is 1024, (mempool ptr=3D0x14= f570ac0) APP: is mempool empty (1) or full (0)? APP: check audit on message_pool APP: Secondary mempool dump file write APP: NO Objs in message pool (_MSG_POOL), exit the app simple_mp > Starting core 31 simple_mp > send hello message_pool pointer is 0x14f570ac0 Failed to get msg obj from mem pool: Success (ret=3D-2) EAL: PANIC in cmd_send_parsed(): Failed to get message buffer 0: ./dpdk-simple_mp_dbg3 (rte_dump_stack+0x2b) [b1ee1b] 1: ./dpdk-simple_mp_dbg3 (__rte_panic+0xbd) [525ede] 2: ./dpdk-simple_mp_dbg3 (cmd_send_parsed+0x2d8) [7ceb68] 3: ./dpdk-simple_mp_dbg3 (400000+0x6a5f46) [aa5f46] 4: ./dpdk-simple_mp_dbg3 (400000+0x6a4f20) [aa4f20] 5: ./dpdk-simple_mp_dbg3 (rdline_char_in+0x34b) [aa841b] 6: ./dpdk-simple_mp_dbg3 (cmdline_in+0x71) [aa4ff1] 7: ./dpdk-simple_mp_dbg3 (cmdline_interact+0x30) [aa5110] 8: ./dpdk-simple_mp_dbg3 (400000+0xfe5cb) [4fe5cb] 9: /lib64/libc.so.6 (7f2d76600000+0x3feb0) [7f2d7663feb0] 10: /lib64/libc.so.6 (__libc_start_main+0x80) [7f2d7663ff60] 11: ./dpdk-simple_mp_dbg3 (_start+0x25) [7ce795] Aborted (core dumped) The rte_mempool_dump() is: mempool @0x14f570ac0 flags=3D1c socket_id=3D0 pool=3D0x14f56c8c0 iova=3D0x1cf570ac0 nrte_mempool_getb_mem_chunks=3D1 size=3D1024 populated_size=3D1024 header_size=3D64 elt_size=3D64 trailer_size=3D64 total_obj_size=3D192 private_data_size=3D0 ops_index=3D1 ops_name: memory chunk at 0x1bfb43180, addr=3D0x14f53c780, iova=3D0x1cf53c780, len= =3D196800 avg bytes/object=3D192.187500 internal cache infos: cache_size=3D32 cache_count[0]=3D0 ..... cache_count[127]=3D0 total_cache_count=3D0 common_pool_count=3D1024 no statistics available Any guidance is appreciated. Thanks, Ed -----Original Message----- From: Lombardo, Ed Sent: Friday, March 21, 2025 2:19 PM To: Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue Hi Sephen, Thank you for your help. I made good progress up to now. When I try to use the dpdk_simple_mp application to send a message to my ap= plication I get a Segmentation fault. First, I re-verified the dpdk_simple_mp process=3Dprimary and dpdk_simple_m= p process-secondary does pass messages successfully. So, my hugepages are = created and DPDK initializes successfully on both at startup. In my application I created the send and recv rings and message_pool as the= primary process. The logs I added do not show any errors. Once my application starts and settles I started the dpdk_simple_mp applica= tion: # ./dpdk-simple_mp_dbg -l 30-31 -n 4 --legacy-mem --proc-type seconda= ry -- However, on the dpdk_simple_mp side do "send hello" and I then get a segmen= tation fault. The debugger takes me deep within the dpdk libraries which I am not too fam= iliar with. The rte_ring_elem.h file function: rte_ring_dequeue_build_elem() is where I= end up with segmentation fault. I notice that the variables are optimized= out, not sure why since I built the dpdk libraries with debug flag. Here is the back trace and could you point me in the direction to look. # gdb dpdk-simple_mp /core/core.dpdk-simple_mp.241269 warning: Unexpected size of section `.reg-xstate/241269' in core file. [Thread debugging using libthread_db enabled] Using host libthread_db libra= ry "/lib64/libthread_db.so.1". Core was generated by `./dpdk-simple_mp -l 30-31 -n 4 --legacy-mem --proc-t= ype secondary --'. Program terminated with signal SIGSEGV, Segmentation fault. warning: Unexpected size of section `.reg-xstate/241269' in core file. #0 0x0000000000cf446a in bucket_dequeue () [Current thread is 1 (Thread 0x= 7f946f835c00 (LWP 241269))] Missing separate debuginfos, use: dnf debuginfo= -install elfutils-libelf-0.189-3.el9.x86_64 glibc-2.34-83.0.1.el9_3.7.x86_6= 4 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libpcap-1.10.0-4.e= l9.x86_64 libzstd-1.5.1-2.el9.x86_64 numactl-libs-2.0.16-1.el9.x86_64 opens= sl-libs-3.0.7-25.0.1.el9_3.x86_64 zlib-1.2.11-40.el9.x86_64 (gdb) bt #0 0x0000000000cf446a in bucket_dequeue () #1 0x00000000007ce77d in cmd_send_parsed () #2 0x0000000000aa5d96 in __cmdline_parse () #3 0x0000000000aa4d70 in cmdline_valid_buffer () #4 0x0000000000aa826b in rdline_char_in () #5 0x0000000000aa4e41 in cmdline_in () #6 0x0000000000aa4f60 in cmdline_interact () #7 0x00000000004fe47a in main.cold () #8 0x00007f946f03feb0 in __libc_start_call_main () from /lib64/libc.so.6 #9 0x00007f946f03ff60 in __libc_start_main_impl () from /lib64/libc.so.6 #10 0x00000000007ce605 in _start () Gdb - stepping through the code, gdb attached to dpdk_simple_mp_debug (gdb) 0x0000000000cf42c5 in rte_ring_dequeue_bulk_elem (available=3D, n=3D, esize=3D, obj_table=3D, r=3D) at ../lib/ring/rte_ri= ng_elem.h:375 375 ../lib/ring/rte_ring_elem.h: No such file or directory. (gdb) p r $17 =3D (gdb) p obj_table $18 =3D (gdb) p available $19 =3D (gdb) n Thread 1 "dpdk-simple_mp_" received signal SIGSEGV, Segmentation fault. bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0x14f= 05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 191 ../drivers/mempool/bucket/rte_mempool_bucket.c: No such file or dir= ectory. (gdb) bt #0 bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0= x14f05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 #1 bucket_dequeue (mp=3D, obj_table=3D0x14f09b5c0, n=3D33) = at ../drivers/mempool/bucket/rte_mempool_bucket.c:289 #2 0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=3D= , obj_table=3D0x14f09b5c0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:793 #3 rte_mempool_do_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7= fff8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1570 #4 rte_mempool_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7fff= 8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1649 #5 rte_mempool_get_bulk (n=3D1, obj_table=3D0x7fff8df066f0, mp=3D0x14f05ed= 40) at ../lib/mempool/rte_mempool.h:1684 #6 rte_mempool_get (obj_p=3D0x7fff8df066f0, mp=3D0x14f05ed40) at ../lib/me= mpool/rte_mempool.h:1710 #7 cmd_send_parsed (parsed_result=3Dparsed_result@entry=3D0x7fff8df06790, = cl=3Dcl@entry=3D0x2f73220, data=3Ddata@entry=3D0x0) at ../examples/multi_process/simple_mp/mp_commands.c:18 #8 0x0000000000aa5d96 in __cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf= =3D0x2f73268 "send hello\n", call_fn=3Dcall_fn@entry=3Dtrue) at ../lib/cmdline/cmdline_parse.c:294 #9 0x0000000000aa5f1a in cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf=3D<= optimized out>) at ../lib/cmdline/cmdline_parse.c:302 #10 0x0000000000aa4d70 in cmdline_valid_buffer (rdl=3D, buf= =3D, size=3D) at ../lib/cmdline/cmdline.c:24 #11 0x0000000000aa826b in rdline_char_in (rdl=3Drdl@entry=3D0x2f73230, c=3D= ) at ../lib/cmdline/cmdline_rdline.c:444 #12 0x0000000000aa4e41 in cmdline_in (size=3D, buf=3D, cl=3D) at ../lib/cmdline/cmdline.c:146 #13 cmdline_in (cl=3D0x2f73220, buf=3D0x7fff8df0c89f "\n\200", size=3D) at ../lib/cmdline/cmdline.c:135 #14 0x0000000000aa4f60 in cmdline_interact (cl=3Dcl@entry=3D0x2f73220) at .= ./lib/cmdline/cmdline.c:192 #15 0x00000000004fe47a in main (argc=3D, argv=3D) at ../examples/multi_process/simple_mp/main.c:122 Appreciate if you can help. Thanks, Ed -----Original Message----- From: Stephen Hemminger Sent: Wednesday, March 19, 2025 7:17 PM To: Lombardo, Ed Cc: users@dpdk.org Subject: Re: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Wed, 19 Mar 2025 21:52:39 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > I added the fib library, but I now see there are many more dpdk libraries= I need to add. Is this typically the case with the example files working = with primary DPDK application? > > I am using meson and ninja to build the examples, but I don't know how to= know the library dependencies. > > How do I learn ahead of building my Application as to what extra librarie= s I need to include for the DPDK example to work? > > I am doing incremental build-test-find_missing_library. > > So far, I needed to add these: -lrte_fib -lrte_rib -lrte_stack=20 > -lrte_member -lrte_efd > > Thanks, > Ed The typical case is to make sure that primary and secondary are built with = the same libraries.