From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6743D46477 for ; Tue, 25 Mar 2025 11:25:45 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2A31C40B9C; Tue, 25 Mar 2025 11:25:45 +0100 (CET) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2086.outbound.protection.outlook.com [40.107.101.86]) by mails.dpdk.org (Postfix) with ESMTP id C148B4027C for ; Tue, 25 Mar 2025 11:25:43 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=sEGrtIxUK21wrzhUtWqgwv3K7pF4uw2dyqGPMfJe4OaSAPRtr5KQnq2Oa1I2uJboYVghTCKlcvGFGXruLByfu9NbeIPhy8ZVJSx7iIu86OP/VR+lnm1liFgF8pTx3Jg98ineEa3ocX6z905tDH2VHUX78IlOpyBnbXfhziC4wVTLfvhmW7KjT3gHpfSsBsFO3iHZKuxxWTlc8StB+Ms0Y7C5dtCbBtbihC0Ic61cSBHFan8XV5riZ/HKxsrRLJa3LX31uDiafTmQbB3Hdrp7h7clRVt7vOufv0a0ULZaDwpYfE/QKhaZ6cDvPVKuON1u+9HojCN40VG6HmjjIPyLIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CVb4Z0ysU3gQJBPzqP2yYMkaQPkoa695Lka6et33KIE=; b=ZJt9Qf61BGl9csnQKTddgZjrLTjx10Gl2T72eSKUgNM/sw+4a7YlG8vFlTRa72S8/xG3qBH1h8o5CBtnj1FlrNXVa9sjPnqgZiwr6sHZFORbzZIaZmfXtS8wFI6nLva1kWwUKWw5LHl9A1KiWzsKImAQ+XjLZpOd50gvPVuYFxvx4/Rs7otfxEkI+drfRsyQaxpr6wEQZZlvYm7t5fv4rBEF2+KbB3UuWV5OwlbF2Wnz6cbjubUfTOEFvAI8AxoHqHIp6i3HgF3Xf27FfvUxvuQkGevXeJ4xQ6HWMWLIJ56UdAlp4sLaIEzoiBKyoFq//aSZ67shE3DeISyk66Z4Dw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=commscope.com; dmarc=pass action=none header.from=commscope.com; dkim=pass header.d=commscope.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=commscope.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CVb4Z0ysU3gQJBPzqP2yYMkaQPkoa695Lka6et33KIE=; b=YODCGLJ5BMcb+RZqXqrlt5T6xFRBj2jrz+T8ui10INKVAWI6fDHtbYxId0sMDpNpZRcbNJ64RZ61E1m7s2fz8WHLeNIm1qiB06hNldUfBapGlBHKXU/+KCBjc4nCKeFbB96L2EMHcwxZZHWhE23b2at+d70LAAePb8geUO59tpA= Received: from DM6PR14MB3597.namprd14.prod.outlook.com (2603:10b6:5:20c::16) by PH7PR14MB7417.namprd14.prod.outlook.com (2603:10b6:510:1b0::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.42; Tue, 25 Mar 2025 10:25:41 +0000 Received: from DM6PR14MB3597.namprd14.prod.outlook.com ([fe80::8cb:da9e:5ec2:1c7a]) by DM6PR14MB3597.namprd14.prod.outlook.com ([fe80::8cb:da9e:5ec2:1c7a%3]) with mapi id 15.20.8534.040; Tue, 25 Mar 2025 10:25:41 +0000 From: "Kompella V, Purnima" To: "Lombardo, Ed" , Stephen Hemminger CC: "users@dpdk.org" Subject: RE: tailqs issue Thread-Topic: tailqs issue Thread-Index: AduY9RY0Q86kHIIzTwW3RWYKe22BSAAF7xuAAAKqzZAAA2FugABZg8GQAHdI0PAAGJ/dMAAD+cagAB3RdvA= Date: Tue, 25 Mar 2025 10:25:41 +0000 Message-ID: References: <20250319132349.5ff339a7@hermes.local> <20250319161659.573e9660@hermes.local> In-Reply-To: Accept-Language: en-IN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=commscope.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR14MB3597:EE_|PH7PR14MB7417:EE_ x-ms-office365-filtering-correlation-id: f5d934e1-c45d-4e80-3a7c-08dd6b8764b4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?ajau7xm1dI0/S/zs4EZydNeZvDzYh6/hNRiCZ4/FKLlHksYWXw/Pxip3Uoml?= =?us-ascii?Q?Nq3T34A5zksTpiGQy/5jugIJcrwtvriE7AUsGEDkKrm4Qt029s/46kohagF/?= =?us-ascii?Q?tjgi3BH/WB3GsmU0ywVmGklYzI64vlLq8CMnrhCTMUDlUqTAArAblOD59qhH?= =?us-ascii?Q?qbYC/0hSERgWzE1y8OxEzjVDK482xuEM6G3Rvmnq+vYx/DLBU1FZSbeHHyYq?= =?us-ascii?Q?NQl5K8ypXgIHLZuPnkNtiJ88OLXmcddNGXMDU0jvYZ2xlyU3NlsvzOkFd3R6?= =?us-ascii?Q?vvjJlJz5AhKh+FH+Zkw7KQ5pBy6oL/3U2mH7zbfD7GB0GqgnxC9PC0UgFM1u?= =?us-ascii?Q?NthP5QpQbSj6wjFTdRShcRfDgjyIi7fbjK4GpGuv6xdJ9E6JjMkm0d9SDlzL?= =?us-ascii?Q?bS8khde7Cp1EcS/hhD4l82Uv2gAaG89afIgj04SVUa/VK6If78SmGIJ891R+?= =?us-ascii?Q?BX8I0cjtmcRUnDE6iHHvxxHe4qPVVpM1Npqil+LqcVlhC6sm+naSfnTC8P6i?= =?us-ascii?Q?iVlIfqFnh90YVN1bB0nUZAP8QXqJgJwaynTUYqxUOmiAvf+2zAWO8TZojP9f?= =?us-ascii?Q?ZvR+IDuGUVtr79hW6s2RMcMmcDJBf/rRXV7WwNjtpg1fW2/0z3Hz3kIYuX1u?= =?us-ascii?Q?5kBT4qh/Fl6MhhbNAt4BEeikif0sn07F3MyqzMTfoSgBnXkTuqZ4Cd8nJEWt?= =?us-ascii?Q?/ub/vZ3RFkedgyitDEmEyCNM2oH7xvc2FQUiGrvNuU1H2lVCrXO4Y/ghNvkd?= =?us-ascii?Q?6SJ/3lVRHv9UpKrrvUh9BT1/8hjQxxJfQpSFwNkC8o9GE8exaRMtobpd1TaJ?= =?us-ascii?Q?BxcVe12waMjabwZSODDZnZPvSEo4u2RaOhOcYJUgsR/MUd6vU36XaGkVV7Zl?= =?us-ascii?Q?1vPFn2+mphL03RX8F6GnC/hUcN1Yk7s7Y62W1GAIMfYCZCYtudS919t2+miC?= =?us-ascii?Q?z429AWFhCjgnvxfAe7RjhrgQJVamfnmdKFlN7PlGMFUa2sq95pn5jo7CwNBu?= =?us-ascii?Q?AIq77Vx4bWMnHr5BVxbsHb301eJ4kH0+4lomqcrnbWOPYbm5kwUVZXpB8ipE?= =?us-ascii?Q?YukA6QKWYcsN++duQ1hwrtw37XHkoJV/HgrR3FmyqgG55eY6QTrSf2Pd8BAd?= =?us-ascii?Q?w1Q3gky85jgsxSO+SMIRaZtSeqjfaX1FJtJFC0FJ4F/n3cjHSLU82b8U408Q?= =?us-ascii?Q?x5FVsCXslW3L1JcZECH96yPiVsIUYJnt5gl7GJvwF1fufObQWfJKWP2taYpL?= =?us-ascii?Q?bXb6j/PB7z1N8K05LwMK1K8uUt+pLDbEXEgpFDWCHAgA2S+pu31OxoW02Lxw?= =?us-ascii?Q?dzGMuHKtJ3NT+TF4LQCadWOzrV+Y4Av1Zi1TnQqPMokG7KqET+j6MTUoMp97?= =?us-ascii?Q?dl1bksfBrm8EvRBVBGrH8x/LX0inkOz5PgFSt7bwtO1TMHVXCyEUn+W8EMh2?= =?us-ascii?Q?xWZ9VJUNM6hG+8YugSGiIScnAArCoIwB?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR14MB3597.namprd14.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(38070700018); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?AA2yFVFdr+Q/gMrY+LNbz2q+9AH3KblUa1z4OCemwN/WpQJRYeqnD1oE8r1/?= =?us-ascii?Q?WHmVHaVVLkV7KlgZgo//+DI/ZiJ1n7uxcGZXHMiUgrW3NPhQQFWL/4yAs87g?= =?us-ascii?Q?PlDCJ3x8cyoIWU57Z6ZeO/26yp675wa3yMTVAQboEkj3NxUI4K36cTViVQUy?= =?us-ascii?Q?4HdntXadFB+SwSL7/CkBUrNMCrHkyqylJFBHMTXHPRkSYNrduMT+vm+F/bEi?= =?us-ascii?Q?6//SfyNgYA9+NYwJFVWrnTYTmpnq3hiMBiCeerI9n8ND9mqPSdce4KeYq/xk?= =?us-ascii?Q?5bC8LggFwbmjjgNI1f2BsBRD9ME9bZ4Eoc9N180nK54SY/hFNigbHH1j0i1u?= =?us-ascii?Q?z7kWWVv4ImpwafK0SSix4r6d0zybNWxp4u9P2FfeIQxxeoueRqMU7Hf0Zwpw?= =?us-ascii?Q?PuKBfLNWpJjkCcLFQLKPpDmUn2o/a7n6CE2hecPPj136ph5bvCmHUZ7z/8Pr?= =?us-ascii?Q?XWaWAJeVrfVdk32A+Kox4ohhFDVED7ieAwCpSvT21DLwHRy0fEt3s+YdQwxO?= =?us-ascii?Q?vSSwCqaRi9bWrj7Zxn/eRvXw9r7D7cLRGRUddbGCm/Titw94NsDjLHiztESs?= =?us-ascii?Q?YFHukuQBzMRmzWM3l+cMNuZB60efrjEVSJskhdj+zMWPpPpdPRYd9IG8JHXp?= =?us-ascii?Q?DaF/q0a/EJAlosothtgvpTcSzOTFYITI6A74/w+OELGt02xJJEV9Cvs7SeW8?= =?us-ascii?Q?1kCbWOxoIq6oznEdNtyRWa5vsm6kZZLqge2X1+z+DQeh3calU2St6RhycSCt?= =?us-ascii?Q?Q0pniMm8GHtU5hXLf/KVz4zJOm/Fi3cCKCQtc0Uuo3NuYB95D0+ej4n9ME2x?= =?us-ascii?Q?G2AFn0LkC2BGK+lEyJ6VKWymLiWqd/iTzj/a0SQEjlhfnrNge95PaGT00jNr?= =?us-ascii?Q?tuSCCrouieuwLYTJxpzHfED86LePrqXyosuPl5eLOpoIO2M5kgf/r1obXQpO?= =?us-ascii?Q?bsNBZv1rpdNw74EWYym0D6UaJdTF4tavHLF5GAhsuv/8meUxOBE8VdFi1CBG?= =?us-ascii?Q?UDizckwW3SCHxGGju5LsYmX+g0RYCeMsrc0MRLoLYAGl04+dvcvl5WcYTdct?= =?us-ascii?Q?WOhzjeai9V5hYH7X5acIx/dNQgbIPIjGMRRbtGWwyjHBFbwSFiC9MWoJZXxh?= =?us-ascii?Q?amk0MKyHWztYJeFke1i8y/wHTGnlLUckrncixDnowsqCvqBSesgQ+iPSxBMh?= =?us-ascii?Q?HLwPa8Srh8F0mFTiBGoclno+rejH+YopjaEjWOu+A+mEcpRhhhESDqAOLKKT?= =?us-ascii?Q?tmLhI3xdPK6756ji37KrglHmcNPaVlH+2x9GZ79TG5ozuwY1UE5RdxY3H+0h?= =?us-ascii?Q?icReDUusaI0tRbZk++bQGU3UGDgRqS/2BxE5RDc4MM3XuYYhpZoNvKXDQ1D0?= =?us-ascii?Q?dMDGIhet7zkatTt6/Ny1BGRgwPwghgYopXZl1U/4pHlh61BHk4kCNrVjvbhc?= =?us-ascii?Q?zg/fNLG/Ix4bW6dhnQ9B8fgMsxbxxmxLj/eMsb2Xh0XexRiVtc+HTOAfPSuc?= =?us-ascii?Q?yNdA8oRq5J/8k/FUy8Zoj1pA+KgOfWIB94DH/+cI5I8jrIRywuPjRCeAxip0?= =?us-ascii?Q?HuSO9pjbCGmJPHtslPKeahkIZI8OQzd0yT8uTkLwY2u2tjVibpw+aTVUC02M?= =?us-ascii?Q?Dg=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: commscope.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR14MB3597.namprd14.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f5d934e1-c45d-4e80-3a7c-08dd6b8764b4 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Mar 2025 10:25:41.0617 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 31472f81-8fe4-49ec-8bc3-fa1c295640d7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: yOJcD4+pydb9mAiVCjNSINxq8MM1ZFYMTC+5QN76iZHZExsw45n/T9L74Pi/sts3vsRsPci7hImo7dAOrbb6wg0mZDszQotCseq4MuAurZw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR14MB7417 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Ed, GDB BT seems to indicate one of your processes (that one that suffered segf= ault) uses OP, not or as you have= listed! #0 bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0= x14f05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 #1 bucket_dequeue (mp=3D, obj_table=3D0x14f09b5c0, n=3D33) = at ../drivers/mempool/bucket/rte_mempool_bucket.c:289 #2 0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=3D= , obj_table=3D0x14f09b5c0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:793 #3 rte_mempool_do_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7= fff8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1570 In any case, can you try by keeping the "order of dpdk libs" exactly the sa= me when compiling the Primary process and the Secondary process? This way, both processes will get the same OPs database and hence ops_index= =3DX will refer to same OPs from either of them and hopefully things should= work fine. Thanks, Purnima -----Original Message----- From: Lombardo, Ed Sent: Monday, March 24, 2025 10:09 PM To: Kompella V, Purnima ; Stephen Hemminger= Cc: users@dpdk.org Subject: RE: tailqs issue CAUTION: This message originated from an External Source outside of CommSco= pe.com. This may be a phishing email that can result in unauthorized access= to CommScope. Please use caution when opening attachments, clicking links,= scanning QR codes, or responding. You can report suspicious emails directl= y in Microsoft Outlook. Hi, I have both the primary and secondary ops_name (s) and they are different. The two files are identical, except for these: PRIMARY: ops_name: common_pool_count=3D1024 SECONDARY: ops_name: common_pool_count=3D0 [root@localhost ~]# diff prim_mempool_dump.txt ex_second_mempool_dump.txt 15c15 < ops_name: --- > ops_name: 149c149 < common_pool_count=3D1024 --- > common_pool_count=3D0 Should the ops_name be the same among the primary and secondary processes? I build out application in our build environment and the dpdk-simple_mp is = from the dpdk sdk (meson/ninja). Thanks, Ed -----Original Message----- From: Kompella V, Purnima Sent: Monday, March 24, 2025 11:00 AM To: Lombardo, Ed ; Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. Hi, I am not sure if this could be your problem, but we had run into a similar = issue with mbuf pools. Do you list the DPDK libs in the exact same order when linking Primary and = Secondary process? If not, the OPs to which the ops_index is pointing to in Primary and in sec= ondary is different. Due to this, rte_mempool_ops_dequeue_bulk called from Secondary doe= s not match the OPs that Primary had used to populate this memPool (during= create) So, accessing the pool from Secondary appears like the pool is empt= y but non-zero in-use Count. This happens because each DPDK lib "appends" to the OPs list on the fly as = it is linked by the linker. So the order of OPs in the OPs database is a mismatch between Primary and S= econdary. Primary and Secondary share information about OPs using ops_index= , but since the order is different, ops_index=3DX in Primary not the same O= Ps in Secondary. And this leads to the segmentation fault. One of your processes is showing ops_index=3D1, ops_name: . Can you confirm it is the same in the other process also? By OPs, and ops_index, I am referring to this internal data structure of DP= DK fw. struct rte_mempool_ops { char name[RTE_MEMPOOL_OPS_NAMESIZE]; /**< Name of mempool ops struc= t. */ rte_mempool_alloc_t alloc; /**< Allocate private data. */ rte_mempool_free_t free; /**< Free the external pool. */ rte_mempool_enqueue_t enqueue; /**< Enqueue an object. */ rte_mempool_dequeue_t dequeue; /**< Dequeue an object. */ rte_mempool_get_count get_count; /**< Get qty of available objs. */ /** * Optional callback to calculate memory size required to * store specified number of objects. */ rte_mempool_calc_mem_size_t calc_mem_size; /** * Optional callback to populate mempool objects using * provided memory chunk. */ rte_mempool_populate_t populate; /** * Get mempool info */ rte_mempool_get_info_t get_info; /** * Dequeue a number of contiguous object blocks. */ rte_mempool_dequeue_contig_blocks_t dequeue_contig_blocks; } __rte_= cache_aligned; Also, the link provides some insight into how OPs is significant. https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/develop= er/articles/technical/optimize-memory-usage-in-multi-threaded-data-plane-de= velopment-kit-dpdk-applications.html__;!!Nzg7nt7_!ElXZ6jVzd3ey8732oohMXbI9i= JA_a5BF67ev7sypu4Vf8LAfVMQzZiEReuDIb-jaLhNO6PswM4RTXVt-qxmR3d_493_Z2SSC$ Regards, Purnima -----Original Message----- From: Lombardo, Ed Sent: Monday, March 24, 2025 10:32 AM To: Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue CAUTION: This message originated from an External Source outside of CommSco= pe.com. This may be a phishing email that can result in unauthorized access= to CommScope. Please use caution when opening attachments, clicking links,= scanning QR codes, or responding. You can report suspicious emails directl= y in Microsoft Outlook. Hi, Further debugging has left me clueless as to the problem with getting the f= irst Obj from the mempool on the secondary process to send to the primary p= rocess (my Application). PRIMARY: My application side (primary process) EAL Arguments: "--proc-type=3Dprimary", "--file-prefix=3Ddpdk_shared", "-l 25,26,27,28", "= -n4" , "--socket-mem=3D2048", "--legacy-mem", Create the _MSG_POOL t_message_pool =3D rte_mempool_create(_MSG_POOL, // Pool name t_pool_size, // Number of = elements in the pool STR_TOKEN_SIZE, // Size of ea= ch message t_pool_cache, // Cache size t_priv_data_sz, // Private da= ta size NULL, // mp_init NULL, // mp_init_ar= g NULL, // obj_init NULL, // obj_init_a= rg 0, // rte_so= cket_id(), // socket_id t_flags // flags ); The t_message_pool pointer value matches the secondary process when execute= " message_pool =3D rte_mempool_lookup(_MSG_POOL);" NTSMON: send ring (0x1bfb43480), recv ring (0x1bfb45a00) and message pool (= 0x14f570ac0) are created successfully. SECONDARY: Secondary process I execute: " # ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file= -prefix=3Ddpdk_shared --proc-type=3Dsecondary --" Notes: * hugepages are created 2x1G on NUMA Node 0 [root@localhost ~]# /opt/dpdk/dpdk-hugepages.py -s Node Pages Size Total 0 2 1Gb 2Gb * --file-prefix=3Ddpdk_shared is provided to both primary and secondary * --proc-type is correctly defined for both primary and secondary process. * The secondary process reports correct socket=3D0 (See dump command output= below) * The secondary process showed Available mempool count is 0 and in-use coun= t is 1024 (which looks incorrect). * Secondary process reports mempool is empty * Secondary audit passed (rte_mempool_audit()), no panic occurred. * Tried disabling ASLR * Tried turning off legacy-mem EXECUTE Secondary process application "dpdk-simple_mp_dbg3" [root@localhost ~]# ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file-prefix=3Ddpd= k_shared --proc-type=3Dsecondary --legacy-mem -- EAL: Detected CPU lcores: 128 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/dpdk_shared/mp_socket_221890_1ba056= 2a6ae95 EAL: Selected IOVA mode 'PA' APP: Finished Process Init. APP: Availble mempool count is 0, in-use count is 1024, (mempool ptr=3D0x14= f570ac0) APP: is mempool empty (1) or full (0)? APP: check audit on message_pool APP: Secondary mempool dump file write APP: NO Objs in message pool (_MSG_POOL), exit the app simple_mp > Starting core 31 simple_mp > send hello message_pool pointer is 0x14f570ac0 Failed to get msg obj from mem pool: Success (ret=3D-2) EAL: PANIC in cmd_send_parsed(): Failed to get message buffer 0: ./dpdk-simple_mp_dbg3 (rte_dump_stack+0x2b) [b1ee1b] 1: ./dpdk-simple_mp_dbg3 (__rte_panic+0xbd) [525ede] 2: ./dpdk-simple_mp_dbg3 (cmd_send_parsed+0x2d8) [7ceb68] 3: ./dpdk-simple_mp_dbg3 (400000+0x6a5f46) [aa5f46] 4: ./dpdk-simple_mp_dbg3 (400000+0x6a4f20) [aa4f20] 5: ./dpdk-simple_mp_dbg3 (rdline_char_in+0x34b) [aa841b] 6: ./dpdk-simple_mp_dbg3 (cmdline_in+0x71) [aa4ff1] 7: ./dpdk-simple_mp_dbg3 (cmdline_interact+0x30) [aa5110] 8: ./dpdk-simple_mp_dbg3 (400000+0xfe5cb) [4fe5cb] 9: /lib64/libc.so.6 (7f2d76600000+0x3feb0) [7f2d7663feb0] 10: /lib64/libc.so.6 (__libc_start_main+0x80) [7f2d7663ff60] 11: ./dpdk-simple_mp_dbg3 (_start+0x25) [7ce795] Aborted (core dumped) The rte_mempool_dump() is: mempool @0x14f570ac0 flags=3D1c socket_id=3D0 pool=3D0x14f56c8c0 iova=3D0x1cf570ac0 nrte_mempool_getb_mem_chunks=3D1 size=3D1024 populated_size=3D1024 header_size=3D64 elt_size=3D64 trailer_size=3D64 total_obj_size=3D192 private_data_size=3D0 ops_index=3D1 ops_name: memory chunk at 0x1bfb43180, addr=3D0x14f53c780, iova=3D0x1cf53c780, len= =3D196800 avg bytes/object=3D192.187500 internal cache infos: cache_size=3D32 cache_count[0]=3D0 ..... cache_count[127]=3D0 total_cache_count=3D0 common_pool_count=3D1024 no statistics available Any guidance is appreciated. Thanks, Ed -----Original Message----- From: Lombardo, Ed Sent: Friday, March 21, 2025 2:19 PM To: Stephen Hemminger Cc: users@dpdk.org Subject: RE: tailqs issue Hi Sephen, Thank you for your help. I made good progress up to now. When I try to use the dpdk_simple_mp application to send a message to my ap= plication I get a Segmentation fault. First, I re-verified the dpdk_simple_mp process=3Dprimary and dpdk_simple_m= p process-secondary does pass messages successfully. So, my hugepages are = created and DPDK initializes successfully on both at startup. In my application I created the send and recv rings and message_pool as the= primary process. The logs I added do not show any errors. Once my application starts and settles I started the dpdk_simple_mp applica= tion: # ./dpdk-simple_mp_dbg -l 30-31 -n 4 --legacy-mem --proc-type seconda= ry -- However, on the dpdk_simple_mp side do "send hello" and I then get a segmen= tation fault. The debugger takes me deep within the dpdk libraries which I am not too fam= iliar with. The rte_ring_elem.h file function: rte_ring_dequeue_build_elem() is where I= end up with segmentation fault. I notice that the variables are optimized= out, not sure why since I built the dpdk libraries with debug flag. Here is the back trace and could you point me in the direction to look. # gdb dpdk-simple_mp /core/core.dpdk-simple_mp.241269 warning: Unexpected size of section `.reg-xstate/241269' in core file. [Thread debugging using libthread_db enabled] Using host libthread_db libra= ry "/lib64/libthread_db.so.1". Core was generated by `./dpdk-simple_mp -l 30-31 -n 4 --legacy-mem --proc-t= ype secondary --'. Program terminated with signal SIGSEGV, Segmentation fault. warning: Unexpected size of section `.reg-xstate/241269' in core file. #0 0x0000000000cf446a in bucket_dequeue () [Current thread is 1 (Thread 0x= 7f946f835c00 (LWP 241269))] Missing separate debuginfos, use: dnf debuginfo= -install elfutils-libelf-0.189-3.el9.x86_64 glibc-2.34-83.0.1.el9_3.7.x86_6= 4 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libpcap-1.10.0-4.e= l9.x86_64 libzstd-1.5.1-2.el9.x86_64 numactl-libs-2.0.16-1.el9.x86_64 opens= sl-libs-3.0.7-25.0.1.el9_3.x86_64 zlib-1.2.11-40.el9.x86_64 (gdb) bt #0 0x0000000000cf446a in bucket_dequeue () #1 0x00000000007ce77d in cmd_send_parsed () #2 0x0000000000aa5d96 in __cmdline_parse () #3 0x0000000000aa4d70 in cmdline_valid_buffer () #4 0x0000000000aa826b in rdline_char_in () #5 0x0000000000aa4e41 in cmdline_in () #6 0x0000000000aa4f60 in cmdline_interact () #7 0x00000000004fe47a in main.cold () #8 0x00007f946f03feb0 in __libc_start_call_main () from /lib64/libc.so.6 #9 0x00007f946f03ff60 in __libc_start_main_impl () from /lib64/libc.so.6 #10 0x00000000007ce605 in _start () Gdb - stepping through the code, gdb attached to dpdk_simple_mp_debug (gdb) 0x0000000000cf42c5 in rte_ring_dequeue_bulk_elem (available=3D, n=3D, esize=3D, obj_table=3D, r=3D) at ../lib/ring/rte_ri= ng_elem.h:375 375 ../lib/ring/rte_ring_elem.h: No such file or directory. (gdb) p r $17 =3D (gdb) p obj_table $18 =3D (gdb) p available $19 =3D (gdb) n Thread 1 "dpdk-simple_mp_" received signal SIGSEGV, Segmentation fault. bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0x14f= 05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 191 ../drivers/mempool/bucket/rte_mempool_bucket.c: No such file or dir= ectory. (gdb) bt #0 bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0= x14f05aa80) at ../drivers/mempool/bucket/rte_mempool_bucket.c:191 #1 bucket_dequeue (mp=3D, obj_table=3D0x14f09b5c0, n=3D33) = at ../drivers/mempool/bucket/rte_mempool_bucket.c:289 #2 0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=3D= , obj_table=3D0x14f09b5c0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:793 #3 rte_mempool_do_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7= fff8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1570 #4 rte_mempool_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7fff= 8df066f0, mp=3D0x14f05ed40) at ../lib/mempool/rte_mempool.h:1649 #5 rte_mempool_get_bulk (n=3D1, obj_table=3D0x7fff8df066f0, mp=3D0x14f05ed= 40) at ../lib/mempool/rte_mempool.h:1684 #6 rte_mempool_get (obj_p=3D0x7fff8df066f0, mp=3D0x14f05ed40) at ../lib/me= mpool/rte_mempool.h:1710 #7 cmd_send_parsed (parsed_result=3Dparsed_result@entry=3D0x7fff8df06790, = cl=3Dcl@entry=3D0x2f73220, data=3Ddata@entry=3D0x0) at ../examples/multi_process/simple_mp/mp_commands.c:18 #8 0x0000000000aa5d96 in __cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf= =3D0x2f73268 "send hello\n", call_fn=3Dcall_fn@entry=3Dtrue) at ../lib/cmdline/cmdline_parse.c:294 #9 0x0000000000aa5f1a in cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf=3D<= optimized out>) at ../lib/cmdline/cmdline_parse.c:302 #10 0x0000000000aa4d70 in cmdline_valid_buffer (rdl=3D, buf= =3D, size=3D) at ../lib/cmdline/cmdline.c:24 #11 0x0000000000aa826b in rdline_char_in (rdl=3Drdl@entry=3D0x2f73230, c=3D= ) at ../lib/cmdline/cmdline_rdline.c:444 #12 0x0000000000aa4e41 in cmdline_in (size=3D, buf=3D, cl=3D) at ../lib/cmdline/cmdline.c:146 #13 cmdline_in (cl=3D0x2f73220, buf=3D0x7fff8df0c89f "\n\200", size=3D) at ../lib/cmdline/cmdline.c:135 #14 0x0000000000aa4f60 in cmdline_interact (cl=3Dcl@entry=3D0x2f73220) at .= ./lib/cmdline/cmdline.c:192 #15 0x00000000004fe47a in main (argc=3D, argv=3D) at ../examples/multi_process/simple_mp/main.c:122 Appreciate if you can help. Thanks, Ed -----Original Message----- From: Stephen Hemminger Sent: Wednesday, March 19, 2025 7:17 PM To: Lombardo, Ed Cc: users@dpdk.org Subject: Re: tailqs issue External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Wed, 19 Mar 2025 21:52:39 +0000 "Lombardo, Ed" wrote: > Hi Stephen, > I added the fib library, but I now see there are many more dpdk libraries= I need to add. Is this typically the case with the example files working = with primary DPDK application? > > I am using meson and ninja to build the examples, but I don't know how to= know the library dependencies. > > How do I learn ahead of building my Application as to what extra librarie= s I need to include for the DPDK example to work? > > I am doing incremental build-test-find_missing_library. > > So far, I needed to add these: -lrte_fib -lrte_rib -lrte_stack > -lrte_member -lrte_efd > > Thanks, > Ed The typical case is to make sure that primary and secondary are built with = the same libraries.