From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A3F846460 for ; Mon, 24 Mar 2025 06:01:58 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1E5CD4029A; Mon, 24 Mar 2025 06:01:58 +0100 (CET) Received: from mx0a-00196b01.pphosted.com (mx0a-00196b01.pphosted.com [67.231.149.170]) by mails.dpdk.org (Postfix) with ESMTP id 6726740277 for ; Mon, 24 Mar 2025 06:01:56 +0100 (CET) Received: from pps.filterd (m0096263.ppops.net [127.0.0.1]) by mx0a-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 52NJRiC1009187; Mon, 24 Mar 2025 01:01:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= netscout.com.09.24.2020; bh=ARJOYWNWJBxhtIjIrblFHTBYttmhNx57XZIA +F/eIoU=; b=Ll/04fDl1iDoWv9uFge7Jt+P4KmocaWkdxnVlX5oHOotLMJOFoG0 Hx9ylr1olgt5So8WwUWvOf6MtH5GX91nFUE810uGSsqyDgVdbiMQSZrlIPemQqR5 VACa1t4FTFfBv1PDDObe3gtaDiGW94y2XGyeSVNiE8fiZ+gs5n+/azoX9zvtbk/4 LhrGqeGZJ6BqZC17plZk9vopo8uohJwY44zlKMD+212j+HWJBM0FfH8FmAxB8k2l h275jd+6L+GVMPcr7VAw7eNw8Fxf0sGZm8J9vINY+vYyhAtjVw0KoL9u9jjEjcn2 iMVjswYwgjXsTGk9T+4WPjExQYmk3s8Egw== Received: from ch5pr02cu005.outbound.protection.outlook.com (mail-northcentralusazlp17012054.outbound.protection.outlook.com [40.93.20.54]) by mx0a-00196b01.pphosted.com (PPS) with ESMTPS id 45hsxehhpp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Mar 2025 01:01:55 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=pnJorCxJTIybjnmQ+mXSKX8lzIM67hJ/e0aXG6ResZEIqJqgkXqw5YkE6fMs0yr/+V8vffOF87hXWDxRrfjsVOEoZvo10KGiyJO19s7NSZ6cWxbJCXnZodYccFFU9q+UF5TFU1mOZb4qGy5rTYKZKmFbEAR9Lec3wRqZbrq2DpQR+iHckprgHcXfVocAivSYIqtLGVkfU9Alvy3hzXW9r70EFAv+9ur4IkvetbYoPVXvhgwtRcPmqyIcPdxnS3qfmnrurqB3H+LcJ8TkbMzfVRGv3Ph1qJUFlcgTU6TyAjdEP+e+ha7nNS4S0M3gRJSPLEvs4cv/TFX5iOdJYCIfVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ARJOYWNWJBxhtIjIrblFHTBYttmhNx57XZIA+F/eIoU=; b=yD0xFplPRXbt8NJnga0TvSKQcEIBIJJMafO+AS0JgYyB1T2CPiapLs+RJajZkcWdp3KmikwrlNCnYiCRp+nQ1oogLW147VcBaBDlRmGhlyRRMErWaBJyqDSzZv2WiJrzXrHTA9Xxb7nfsVvj/e7QiagFHxQlMlUxeyPt9i5J3nz5Iy3DVp11DvyJIZv5b7Uz7aeQAEEfQ6OTkUPemn8cEun7GEH2jZZNJCp2VC+oZ3aNGxsU0LREd9q4n/bG4/Bk/QWCZyyhmjWmnbZYLNLuGbQ1d9f7dIO9A5kIMm1PeMjBtU7NgONuz44dII+8VH0zEFXdtsjS+Mk83VeoltGEpw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by SA1PR01MB8356.prod.exchangelabs.com (2603:10b6:806:386::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.42; Mon, 24 Mar 2025 05:01:52 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::80c4:7216:f070:e5fd%4]) with mapi id 15.20.8534.040; Mon, 24 Mar 2025 05:01:51 +0000 From: "Lombardo, Ed" To: Stephen Hemminger CC: "users@dpdk.org" Subject: RE: tailqs issue Thread-Topic: tailqs issue Thread-Index: AduY9RY0Q86kHIIzTwW3RWYKe22BSAAF7xuAAAKqzZAAA2FugABZg8GQAHdI0PA= Date: Mon, 24 Mar 2025 05:01:51 +0000 Message-ID: References: <20250319132349.5ff339a7@hermes.local> <20250319161659.573e9660@hermes.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|SA1PR01MB8356:EE_ x-ms-office365-filtering-correlation-id: 77743aa1-dae0-4f26-76aa-08dd6a90fd8f x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?VhSP20LXQtGFT7C4Fn7w3bPtxoVD9CzEScDZlV8zUqdrt9jGRPsDwEK5mUs9?= =?us-ascii?Q?oLShJ8KZaXpDk1DJfHmv0AOud74LZNuSd6Jdmtafd3CH8nLJLMrXeiExYkm9?= =?us-ascii?Q?mgvcQTBSdbjsOnWHFmUDMTqq6u3aiD3dedD9WuUdcH8FKgpC0VieBhF4b2bF?= =?us-ascii?Q?8INIXaCpejrDNv1HuZnpaoW4CCJMsX78kPTU5ExVXjyHcniuZQ7BrE4bwzEb?= =?us-ascii?Q?yiKLGKTMZX3sQvPSDhC7wZAOHzp8W1D1kubxeQs3Vfk3Ss04OzGOu6Vbu76+?= =?us-ascii?Q?q5AIZtE+cFWEuchxGKaIqeLmPmduwtMPMjmu8uACHJAXHE5Bit5QSiDmK4zg?= =?us-ascii?Q?HLc1EQMwuBzGFNMnZdFxgNEtX4IDLzvh6BVqhr+tKDtgIvCfkkgjLCQs+xwU?= =?us-ascii?Q?SehUowPZf3yTBjDgPOBJdtW232j8EoJo3Ru2z2vtx+Pt8RVHYFGIQWsHEUHR?= =?us-ascii?Q?g0dHYS+u0VVbG4ss05yogrMVsUC1U8qvyYBWQjd4we+D9byy1l2EcAYZ/UH5?= =?us-ascii?Q?xG9PXUk75nRGagRsFWwU6BWE/LYANea5ZNfN+bBlPWVt8sQKc1FjFbVkaG23?= =?us-ascii?Q?ji2dXtqFX67RKQoQcqaW6u30PjVSF+H+3jfUuC+Ij4h4v8wA0S6jnD74itsS?= =?us-ascii?Q?WLALNz8+7nUWaJUfsQryuQNPV9gtcbXY/xvAQvg+1Bvo4X3CWboqeG2aBI9Z?= =?us-ascii?Q?FbDWCfjBMENrA2+ERkByCA5ahR1TnOGwENE/utAvC8QjoSyg8Qh+T1WN5181?= =?us-ascii?Q?xsN/gMknslG4GlI6z9t+GKnLYH9m3JKPaMzrELU1OrF4CK2G+m+MePrOiT5v?= =?us-ascii?Q?yz2hVvnoC8FPwHRCGiTLLwn3gxXQ9K+mNw7Mn93Xa8N3xGjgNbJPvgi4s9e3?= =?us-ascii?Q?UapaSSKuIHlKfoCA0kj4gwDqihiiMj9kt+vV7iA0Ckbs4UAAca0VDGglTOYj?= =?us-ascii?Q?sRSAAEPv4Tn0OBnc5UhSX7H37+snGVL96mRGynZEVxvRk8FeM2YsAZZL1l/O?= =?us-ascii?Q?NpoXt5TDyVxkWf/5bSvyphSzb5H5gMsDjJBWVilrdAl3wkdjVhrp0ryuW1Kd?= =?us-ascii?Q?u07T6Re5VJ/2Io+SzMD1ksFvUc859hPj/mxWkEM7WfO71JS+P7r4jYktPe6M?= =?us-ascii?Q?MSS+Gq+u/3TwtR1QLs+DgISyWeVmzPT9eTRnLHRWKrmnh8S6zm7BaXSVdoyG?= =?us-ascii?Q?UoKDyA15z+6bv1km37NDLJsGPxbId6OM5IZ1Xj1UfSQKRdNscWq349kU+deC?= =?us-ascii?Q?daBR4fEgsCzPJ6ZoJ9wdgXZTqymxYd33MHrVFlyD9V22DqskaQ1BSKBkw617?= =?us-ascii?Q?Fz+nD68BRQkZwGPbDkkiCxWs+pOsxeJIUgUM1RkL2BnQ7QQeDgf05AbnWPk4?= =?us-ascii?Q?R7umukC0O3Zo++NWxkiF2QYAmEsEK26v1+R2WHnCX0h0dyIqOEj7srmqAqPU?= =?us-ascii?Q?3xP4hGzn4x0PEsnDInTB/FVu0yB4DApH?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?gxjVANYs6Ibk99sOTotQ5eHOnbyJnyWodxYeXLmYCwPmymg5OdVx5aeZpTIz?= =?us-ascii?Q?qvHXdwAiCEDosmPbhEHR/ibIXZxQDh/uWqWBtzXv6v8iO4C0egJ8OEBgC1ut?= =?us-ascii?Q?VS1NS4tGiKlfG7f9gkvabSX4+bcZZ1eOZbXAzeJLPuFh2YVY5MPrMd0IArHD?= =?us-ascii?Q?UNf0dVl1IOFHPChNsPAw9Unh42pgLQnkls6wPGlIhuIVIgQaMAohVYf9n85/?= =?us-ascii?Q?hVh3Pg3Osv1y5YcKiF3w3PzYCg/A2buQrt0fKZOH5VWjcDFW+R+JYUOdYsLp?= =?us-ascii?Q?+GWSFAVgAcc1qDh3l50sJc7yC+U22s6MVFs5w4n92OqUJ0xR4SKasZiyWtaI?= =?us-ascii?Q?UIYls+eZYUS2iH1jinwO1t85VfR0EFsDnT36RzZCG9HlMHDzjR//sqWIxvyE?= =?us-ascii?Q?Al8KrsUiuI/zbI+4tFB++k9b2TUuxVO2e01CwUQpt39fH7v+voZqRDaEIwUE?= =?us-ascii?Q?KkVEYOEN3RAdrM+WVfF8fYQOKWaUyvmS1vJxidrWBZlEz7oLxpotWd28a3M1?= =?us-ascii?Q?V7P99a2eX+5oJEEe/2D8fDD+B8EnNwiv+Be3jSrZprsmLNl/Qtgrz6o0oFr9?= =?us-ascii?Q?mkpQjELvXlB0y9dCsLbz9vV+Fw1VHO7QrXs7mlllJKZZUvhE/cNrX6Ng1fUB?= =?us-ascii?Q?6p73Q5yivtGAYYtB7gi5yHqWnOx3PChGLjP4QAoLuFPWd/BS5RtmZL7/A8eB?= =?us-ascii?Q?r39yj3SyXEwJUOkQ59inAOxl8URTYHuN8ndQhXw+wmAT1oK4r3xqtzydbDJE?= =?us-ascii?Q?Ftfw8HoJQnbeCXBnIL51daPq8mk9ulJ6nD5mbvPlGBvw/AxpqS2K8VC5LdPH?= =?us-ascii?Q?FTj2QAiK1k+vi8Bi4iWuD6stYk/+tYoEoQDJpBpRLhtHngPNPLfhl6u+7kqQ?= =?us-ascii?Q?TME/h7vSL7akZxIG1QnzAgFYCaCDVeoZamTECQOYZ8YbrBYU4o8/JKs0qaR6?= =?us-ascii?Q?6Yznw3QFQx8819E45sc0UpvWKPc6qUJSkwipPjsQCqbhAs2pfwdxu9ZH2zqa?= =?us-ascii?Q?lzzFoUgz1oYzu4ajSKUZ64zzy5mKvCExHOyhkQ1f4S9l2NSbfhzOVlAmZdIT?= =?us-ascii?Q?UBrb62fECyC3Zp6aUpgdPqHOOGlZRUc0k10e41bN1+CqIopWENTajq38+CfT?= =?us-ascii?Q?o+7OkhtbcH7u1gt3cK1IUOR27hz2OH7haGUGBmh6c4bO09cpvKupV8ONFJzI?= =?us-ascii?Q?VKL9+U6Op5F+DW0uE8bFEi+aaYVd73XdhUr0I+bKMzbZnTKCEd1xCMSmwfW+?= =?us-ascii?Q?tcjZ72kEOdqv35HIrUWE+RDWBhKt41muyizs+H+kN5cqNQSky4zA4K24Bwu+?= =?us-ascii?Q?KXT0EDkWAzuKCB9S+P4hEqLhSVTJnD0AkuamStmXLC3gAaM6VcV2wyPkOV1g?= =?us-ascii?Q?p8JMErmyzxteUBPjwOEPx32rgAawvR0Rvq19PAyrQl/43ui+XtrUdkYtz0vT?= =?us-ascii?Q?eIvcQV0ChXbBOeU+xT99bzEN7whwJPMYa8qtS+legYKzU4JQE6XVW8k7SkkZ?= =?us-ascii?Q?c+fI78VixeGXIg7KVia5WCrHFULgafg9taXwIBlmqkFRpnvlGA3YVqZiI9uS?= =?us-ascii?Q?LJ/tHNZnX+oW/hDLgjS29mrR31kzru8SWtTvYtZj?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 89Nbz16BBDrtG/5hW6y1WG7VDLV1PnlV6HHphaz46fLioq/dyw38GlzgeRwMXboTaFXjm4kKp8ryibpZLySF/bgctq2mnN1o5E6eBDiKZ1YrhR1s7Hk5j424nsniPZ2lFHCT/MEsCE7DfiyhXrDCY+ONJIOgrJoWGxW6+4rqRGFXYrAe2iMdZoZJkSrkCpaXd1/RNHqiu92sDitAzxSuwJKHbCptKY1xI07rvXFEFTdpgRXWxOthjNvgLAkikBc7WqoTZE4CSJ27lkoM1lzCYaSc39194aQ6Xu8YizWglXf+idxC5G0MtbaXor3o6JdoQPbXvbqjgf6D5ELt0qPBwcmi3AhDZjtL3D5jgKUkzLmmO3woBteG1m7ttHNFbDZKfSAYzE5aLHkpurxVyvkpn/dQ2C9zmkxHx4YKu67VJ26W/ZDYklFTxYK0MWxUsCHV/ogiLCulxfRoQP9uhaxybjx5OWhcoZUYLQ9Z9aJgHxNDn9WLFRb5MmVoQitYLBMrEYYI97tkTn4yidgtlog5NBgjN6Jzf81DIHgI6mdiz2qta/UG5fpQXEVvUHc6Rd9ulQYesJG82TpUF1z3V7hG+VyEhDg1qk+KQcVeScz9oL72Q1y1j7gsJoFYf5bNkuZU X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 77743aa1-dae0-4f26-76aa-08dd6a90fd8f X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Mar 2025 05:01:51.8214 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: rdosQGQ9jsSOmZhllmD1FTs8JB+kn7AZ8WP/hJo17VwP9kFJN+a9vmNBGeATeswKGEovnornHSpK9isLIwB2Dg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR01MB8356 X-Proofpoint-ORIG-GUID: IO2isfYRtH39zc1BzXYCed0EfgdPBz9W X-Authority-Analysis: v=2.4 cv=CfkI5Krl c=1 sm=1 tr=0 ts=67e0e743 cx=c_pps a=ZeLi2qfhioKtEj0V5xjDvA==:117 a=lCpzRmAYbLLaTzLvsPZ7Mbvzbb8=:19 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=kj9zAlcOel0A:10 a=Vs1iUdzkB0EA:10 a=H5OGdu5hBBwA:10 a=jZVsG21pAAAA:8 a=8rWy6zfcAAAA:8 a=jQOgFn-ZAAAA:8 a=7yplnqxgdl-nhmj3K14A:9 a=CjuIK1q_8ugA:10 a=3Sh2lD0sZASs_lUdrUhf:22 a=YjdVzJdQTyZRADMV7wFX:22 a=mT82qxFQzDvLIExZS32s:22 cc=ntf X-Proofpoint-GUID: IO2isfYRtH39zc1BzXYCed0EfgdPBz9W X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 bulkscore=0 mlxscore=0 clxscore=1015 phishscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 lowpriorityscore=0 impostorscore=0 spamscore=0 malwarescore=0 priorityscore=1501 classifier=spam authscore=0 authtc=n/a authcc=notification route=outbound adjust=0 reason=mlx scancount=1 engine=8.21.0-2502280000 definitions=main-2503240035 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, Further debugging has left me clueless as to the problem with getting the f= irst Obj from the mempool on the secondary process to send to the primary p= rocess (my Application). PRIMARY: My application side (primary process) EAL Arguments: "--proc-type=3Dprimary", "--file-prefix=3Ddpdk_shared", "-l 25,26,27,28", "= -n4" , "--socket-mem=3D2048", "--legacy-mem", Create the _MSG_POOL t_message_pool =3D rte_mempool_create(_MSG_POOL, // Pool name t_pool_size, // Number of = elements in the pool STR_TOKEN_SIZE, // Size of ea= ch message t_pool_cache, // Cache size t_priv_data_sz, // Private da= ta size NULL, // mp_init NULL, // mp_init_ar= g NULL, // obj_init NULL, // obj_init_a= rg 0, // rte_so= cket_id(), // socket_id t_flags // flags ); The t_message_pool pointer value matches the secondary process when execute= " message_pool =3D rte_mempool_lookup(_MSG_POOL);" NTSMON: send ring (0x1bfb43480), recv ring (0x1bfb45a00) and message pool (= 0x14f570ac0) are created successfully. SECONDARY: Secondary process I execute: " # ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file= -prefix=3Ddpdk_shared --proc-type=3Dsecondary --" Notes:=20 * hugepages are created 2x1G on NUMA Node 0 [root@localhost ~]# /opt/dpdk/dpdk-hugepages.py -s Node Pages Size Total 0 2 1Gb 2Gb * --file-prefix=3Ddpdk_shared is provided to both primary and secondary * --proc-type is correctly defined for both primary and secondary process. * The secondary process reports correct socket=3D0 (See dump command output= below) * The secondary process showed Available mempool count is 0 and in-use coun= t is 1024 (which looks incorrect). * Secondary process reports mempool is empty * Secondary audit passed (rte_mempool_audit()), no panic occurred. * Tried disabling ASLR * Tried turning off legacy-mem EXECUTE Secondary process application "dpdk-simple_mp_dbg3" [root@localhost ~]# ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file-prefix=3Ddpd= k_shared --proc-type=3Dsecondary --legacy-mem -- EAL: Detected CPU lcores: 128 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/dpdk_shared/mp_socket_221890_1ba056= 2a6ae95 EAL: Selected IOVA mode 'PA' APP: Finished Process Init. APP: Availble mempool count is 0, in-use count is 1024, (mempool ptr=3D0x14= f570ac0) APP: is mempool empty (1) or full (0)? APP: check audit on message_pool APP: Secondary mempool dump file write APP: NO Objs in message pool (_MSG_POOL), exit the app simple_mp > Starting core 31 simple_mp > send hello message_pool pointer is 0x14f570ac0 Failed to get msg obj from mem pool: Success (ret=3D-2) EAL: PANIC in cmd_send_parsed(): Failed to get message buffer 0: ./dpdk-simple_mp_dbg3 (rte_dump_stack+0x2b) [b1ee1b] 1: ./dpdk-simple_mp_dbg3 (__rte_panic+0xbd) [525ede] 2: ./dpdk-simple_mp_dbg3 (cmd_send_parsed+0x2d8) [7ceb68] 3: ./dpdk-simple_mp_dbg3 (400000+0x6a5f46) [aa5f46] 4: ./dpdk-simple_mp_dbg3 (400000+0x6a4f20) [aa4f20] 5: ./dpdk-simple_mp_dbg3 (rdline_char_in+0x34b) [aa841b] 6: ./dpdk-simple_mp_dbg3 (cmdline_in+0x71) [aa4ff1] 7: ./dpdk-simple_mp_dbg3 (cmdline_interact+0x30) [aa5110] 8: ./dpdk-simple_mp_dbg3 (400000+0xfe5cb) [4fe5cb] 9: /lib64/libc.so.6 (7f2d76600000+0x3feb0) [7f2d7663feb0] 10: /lib64/libc.so.6 (__libc_start_main+0x80) [7f2d7663ff60] 11: ./dpdk-simple_mp_dbg3 (_start+0x25) [7ce795] Aborted (core dumped) The rte_mempool_dump() is: mempool @0x14f570ac0 flags=3D1c socket_id=3D0 pool=3D0x14f56c8c0 iova=3D0x1cf570ac0 nrte_mempool_getb_mem_chunks=3D1 size=3D1024 populated_size=3D1024 header_size=3D64 elt_size=3D64 trailer_size=3D64 total_obj_size=3D192 private_data_size=3D0 ops_index=3D1 ops_name: memory chunk at 0x1bfb43180, addr=3D0x14f53c780, iova=3D0x1cf53c780, len= =3D196800 avg bytes/object=3D192.187500 internal cache infos: cache_size=3D32 cache_count[0]=3D0 ..... cache_count[127]=3D0 total_cache_count=3D0 common_pool_count=3D1024 no statistics available Any guidance is appreciated. Thanks, Ed -----Original Message----- From: Lombardo, Ed=20 Sent: Friday, March 21, 2025 2:19 PM To: Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue Hi Sephen, Thank you for your help. I made good progress up to now. When I try to use the dpdk_simple_mp application to send a message to my ap= plication I get a Segmentation fault. First, I re-verified the dpdk_simple_mp process=3Dprimary and dpdk_simple_m= p process-secondary does pass messages successfully. So, my hugepages are = created and DPDK initializes successfully on both at startup. In my application I created the send and recv rings and message_pool as the= primary process. The logs I added do not show any errors. =20 Once my application starts and settles I started the dpdk_simple_mp applica= tion: # ./dpdk-simple_mp_dbg -l 30-31 -n 4 --legacy-mem --proc-type seconda= ry -- However, on the dpdk_simple_mp side do "send hello" and I then get a segmen= tation fault. The debugger takes me deep within the dpdk libraries which I am not too fam= iliar with. The rte_ring_elem.h file function: rte_ring_dequeue_build_elem() is where I= end up with segmentation fault. I notice that the variables are optimized= out, not sure why since I built the dpdk libraries with debug flag. Here is the back trace and could you point me in the direction to look. =20 # gdb dpdk-simple_mp /core/core.dpdk-simple_mp.241269 warning: Unexpected size of section `.reg-xstate/241269' in core file. [Thread debugging using libthread_db enabled] Using host libthread_db libra= ry "/lib64/libthread_db.so.1". Core was generated by `./dpdk-simple_mp -l 30-31 -n 4 --legacy-mem --proc-t= ype secondary --'. Program terminated with signal SIGSEGV, Segmentation fault. warning: Unexpected size of section `.reg-xstate/241269' in core file. #0 0x0000000000cf446a in bucket_dequeue () [Current thread is 1 (Thread 0x= 7f946f835c00 (LWP 241269))] Missing separate debuginfos, use: dnf debuginfo= -install elfutils-libelf-0.189-3.el9.x86_64 glibc-2.34-83.0.1.el9_3.7.x86_6= 4 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libpcap-1.10.0-4.e= l9.x86_64 libzstd-1.5.1-2.el9.x86_64 numactl-libs-2.0.16-1.el9.x86_64 opens= sl-libs-3.0.7-25.0.1.el9_3.x86_64 zlib-1.2.11-40.el9.x86_64 (gdb) bt #0 0x0000000000cf446a in bucket_dequeue () #1 0x00000000007ce77d in cmd_send_parsed () #2 0x0000000000aa5d96 in __cmdline_parse () #3 0x0000000000aa4d70 in cmdline_valid_buffer () #4 0x0000000000aa826b in rdline_char_in () #5 0x0000000000aa4e41 in cmdline_in () #6 0x0000000000aa4f60 in cmdline_interact () #7 0x00000000004fe47a in main.cold () #8 0x00007f946f03feb0 in __libc_start_call_main () from /lib64/libc.so.6 #9 0x00007f946f03ff60 in __libc_start_main_impl () from /lib64/libc.so.6 #10 0x00000000007ce605 in _start () Gdb - stepping through the code, gdb attached to dpdk_simple_mp_debug (gdb) 0x0000000000cf42c5 in rte_ring_dequeue_bulk_elem (available=3D, n=3D, esize=3D, obj_table=3D, r=3D) at ../lib/ring/rte_ri= ng_elem.h:375 375 ../lib/ring/rte_ring_elem.h: No such file or directory. (gdb) p r $17 =3D (gdb) p obj_table $18 =3D (gdb) p available $19 =3D (gdb) n Thread 1 "dpdk-simple_mp_" received signal SIGSEGV, Segmentation fault. bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0x14f= 05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 191 ../drivers/mempool/bucket/rte_mempool_bucket.c: No such file or dir= ectory. (gdb) bt #0 bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0= x14f05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 #1 bucket_dequeue (mp=3D, obj_table=3D0x14f09b5c0, n=3D33) = at ../drivers/mempool/bucket/rte_mempool_bucket.c:289 #2 0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=3D= , obj_table=3D0x14f09b5c0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:793 #3 rte_mempool_do_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7= fff8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1570 #4 rte_mempool_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7fff= 8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1649 #5 rte_mempool_get_bulk (n=3D1, obj_table=3D0x7fff8df066f0, mp=3D0x14f05ed= 40) at ../lib/mempool/rte_mempool.h:1684 #6 rte_mempool_get (obj_p=3D0x7fff8df066f0, mp=3D0x14f05ed40) at ../lib/me= mpool/rte_mempool.h:1710 #7 cmd_send_parsed (parsed_result=3Dparsed_result@entry=3D0x7fff8df06790, = cl=3Dcl@entry=3D0x2f73220, data=3Ddata@entry=3D0x0) at ../examples/multi_process/simple_mp/mp_commands.c:18 #8 0x0000000000aa5d96 in __cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf= =3D0x2f73268 "send hello\n", call_fn=3Dcall_fn@entry=3Dtrue) at ../lib/cmdline/cmdline_parse.c:294 #9 0x0000000000aa5f1a in cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf=3D<= optimized out>) at ../lib/cmdline/cmdline_parse.c:302 #10 0x0000000000aa4d70 in cmdline_valid_buffer (rdl=3D, buf= =3D, size=3D) at ../lib/cmdline/cmdline.c:24 #11 0x0000000000aa826b in rdline_char_in (rdl=3Drdl@entry=3D0x2f73230, c=3D= ) at ../lib/cmdline/cmdline_rdline.c:444 #12 0x0000000000aa4e41 in cmdline_in (size=3D, buf=3D, cl=3D) at ../lib/cmdline/cmdline.c:146 #13 cmdline_in (cl=3D0x2f73220, buf=3D0x7fff8df0c89f "\n\200", size=3D) at ../lib/cmdline/cmdline.c:135 #14 0x0000000000aa4f60 in cmdline_interact (cl=3Dcl@entry=3D0x2f73220) at .= ./lib/cmdline/cmdline.c:192 #15 0x00000000004fe47a in main (argc=3D, argv=3D) at ../examples/multi_process/simple_mp/main.c:122 Appreciate if you can help. Thanks, Ed -----Original Message----- From: Stephen Hemminger Sent: Wednesday, March 19, 2025 7:17 PM To: Lombardo, Ed Cc: users@dpdk.org Subject: Re: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Wed, 19 Mar 2025 21:52:39 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > I added the fib library, but I now see there are many more dpdk libraries= I need to add. Is this typically the case with the example files working = with primary DPDK application? >=20 > I am using meson and ninja to build the examples, but I don't know how to= know the library dependencies. >=20 > How do I learn ahead of building my Application as to what extra librarie= s I need to include for the DPDK example to work? >=20 > I am doing incremental build-test-find_missing_library. >=20 > So far, I needed to add these: -lrte_fib -lrte_rib -lrte_stack=20 > -lrte_member -lrte_efd >=20 > Thanks, > Ed The typical case is to make sure that primary and secondary are built with = the same libraries.