From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 0AF1B46466
for <public@inbox.dpdk.org>; Mon, 24 Mar 2025 15:59:56 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id BCECB40A7F;
Mon, 24 Mar 2025 15:59:54 +0100 (CET)
Received: from NAM10-DM6-obe.outbound.protection.outlook.com
(mail-dm6nam10on2042.outbound.protection.outlook.com [40.107.93.42])
by mails.dpdk.org (Postfix) with ESMTP id CED3F40A71
for <users@dpdk.org>; Mon, 24 Mar 2025 15:59:53 +0100 (CET)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
b=PnJ50NnyY95FMSuiGXAb98j7fWt69eY5S3IzDmr5QH9s+lS81qiWXuJbwqm4/djZhImhehU2UZYC7RHVNfrjk7JhIH4YqP2GA52E/HHuFsGZUdEzjeSaodrqdrlbHgun5V3MOBHLpeMAmc0zsLarL7VWeLzfHPg87enT++VPnN621oQBxCJhRkN2JqB/ZDY2Zq8prcz/RCeQcsSSUbiXm53DJO23y+NOC6vT+XqU1G4VTgPi0tJqCjsWJSkEhJ7xPh2JRaAnw+MoiMiAfy9B3iWtU22192FBudLdMI+wuPK6KeK62zI8qbGTeTgA37froxVIxmbNaEyTr6ynwKFohA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector10001;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=+LmNiEIDkBZNhIwahnt/rwXfIsWeqVbCC8s3pZpmgFQ=;
b=h05m9/P5tq4GHeN1uIiHf/HpfY/eMwE+/QP7QaPiY2cOwYLC8g7wxZydiTDKa2Bph1W6lDh/nVq+YzLGXujkp/YhRaTecOT96wJD8QT3IHZmr6Re5XjQ8Z0WLtZpI12kVNqvx88DLYG6GIbmSz97KfXOo6lRS2+BxGKPLiZiLHeBzgQAdWhSuOmdo5F2CDia9jWqO9LDA8I9s/g2WOPXbCGMl/qMZuB0DudBZ5wGNKg3qpTKN62ig9LTnyOWLJh7wuEERdUVkiG+oeB5LGLPJ9EEg6qK00v1JpvfgTFsDX4tuRvlOwZCDS2+Q1D798/TWcMu7YyI+VbA13qRDN2GYQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=commscope.com; dmarc=pass action=none
header.from=commscope.com; dkim=pass header.d=commscope.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=commscope.com;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=+LmNiEIDkBZNhIwahnt/rwXfIsWeqVbCC8s3pZpmgFQ=;
b=ROcNz2yciBx7JhKx7jXn597P/rz1TQGiDv5zR076CtK9nBwjfhSwu5u2M385FHACCJG0lVOEWJa1PIaxMiB4gKNz/Pmr0Le5AqAmvr22iv75V0ZCsFl9CzUYCrI43w+1rshZHCp+qbktZPiTIGns5CJIGjhanRPdS0zdHGkl3ps=
Received: from DM6PR14MB3597.namprd14.prod.outlook.com (2603:10b6:5:20c::16)
by SN7PR14MB5937.namprd14.prod.outlook.com (2603:10b6:806:2a0::7) with
Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.42; Mon, 24 Mar
2025 14:59:51 +0000
Received: from DM6PR14MB3597.namprd14.prod.outlook.com
([fe80::8cb:da9e:5ec2:1c7a]) by DM6PR14MB3597.namprd14.prod.outlook.com
([fe80::8cb:da9e:5ec2:1c7a%3]) with mapi id 15.20.8534.040; Mon, 24 Mar 2025
14:59:51 +0000
From: "Kompella V, Purnima" <Kompella.Purnima@commscope.com>
To: "Lombardo, Ed" <Ed.Lombardo@netscout.com>, Stephen Hemminger
<stephen@networkplumber.org>
CC: "users@dpdk.org" <users@dpdk.org>
Subject: RE: tailqs issue
Thread-Topic: tailqs issue
Thread-Index: AduY9RY0Q86kHIIzTwW3RWYKe22BSAAF7xuAAAKqzZAAA2FugABZg8GQAHdI0PAAGJ/dMA==
Date: Mon, 24 Mar 2025 14:59:51 +0000
Message-ID: <DM6PR14MB3597B4F30FD5FBE9314BBE569CA42@DM6PR14MB3597.namprd14.prod.outlook.com>
References: <CH3PR01MB8470C2B4DCBFEC90D7EA856F8FD92@CH3PR01MB8470.prod.exchangelabs.com>
<20250319132349.5ff339a7@hermes.local>
<CH3PR01MB8470103A17A1A2C7D3CC81EB8FD92@CH3PR01MB8470.prod.exchangelabs.com>
<20250319161659.573e9660@hermes.local>
<CH3PR01MB84709474C1DA516FC0B2AD3E8FDB2@CH3PR01MB8470.prod.exchangelabs.com>
<CH3PR01MB847095627410E88255CD26C98FA42@CH3PR01MB8470.prod.exchangelabs.com>
In-Reply-To: <CH3PR01MB847095627410E88255CD26C98FA42@CH3PR01MB8470.prod.exchangelabs.com>
Accept-Language: en-IN, en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: dkim=none (message not signed)
header.d=none;dmarc=none action=none header.from=commscope.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: DM6PR14MB3597:EE_|SN7PR14MB5937:EE_
x-ms-office365-filtering-correlation-id: abbe955b-6766-406e-6813-08dd6ae487ab
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018;
x-microsoft-antispam-message-info: =?us-ascii?Q?6wo0oWgAgXcQ9E9q4+mFt/ONzOrhfwsI+EgPplWOJmmXtb87vWhmPWhxY7el?=
=?us-ascii?Q?E1UywzmhBg4zzqgPASbUmWAOSomTSUEAHOI5Hk08aycj17N1LYR+P+J37O54?=
=?us-ascii?Q?pXVv1yvLmpJX4I5cYPp3OqXItvib5Ux/Fz0kWe6HM7TMsXIwiinWZoJVOVgu?=
=?us-ascii?Q?RlwlpTLaNn25ir/CD8OsaBx1QgWA63QT8j54UHmn2bmhKjjLsGvpeX4tZKVW?=
=?us-ascii?Q?8Chc4GB60ne7f7elqn1Ze0OmcgB2hazdfLg5dX5TY0bDgOojVIv0tsxm65ir?=
=?us-ascii?Q?d8Kzxm84o6dMAAqqzRar9kTQhKoA3FPJS/0TeeEQmH8vtGO5BL6XBNtzv9/C?=
=?us-ascii?Q?lPVggJHrxTC6Hwc6YydJTWQGvw7+9Poqyvnl0oH3HuAGe9DsBdxiPN0aSplH?=
=?us-ascii?Q?/FurUeQZj75gp07f+JGTWle4bn7mEymRLWYTF26neCoOCvnglUPpHmypQA1/?=
=?us-ascii?Q?ToM8WOevTWowILuA48+/HnG+SIgCx4Jf4rEszM2rIeWRbcTMPoZ5t6qzeZDj?=
=?us-ascii?Q?6k5ZlJ2/ILgBNjbQmMyheyhNEn6BgZukDb18wpyOXkgbVm3iteP9l4xdmoNd?=
=?us-ascii?Q?4TTY3MAWQ9FkDsjA4FIgH4riq3qYt+FdVl2OAtt/b71D36LysN8octAB3Yl0?=
=?us-ascii?Q?hIExMKIi9lN/RZ3lG7h5jwluXn7RyluzQ9Kvm+yyP0osFebaxaoizPoYzJqN?=
=?us-ascii?Q?H4rECpvXhCqjiEcfXjscKpFQoJ4iimNv4PJ9EvZZUHN0xRmny4IW+QAkw+nE?=
=?us-ascii?Q?saqimBAdcO6tY9UOQkslU1TZ3ZZUrGAd5CV5SS9w0mEoVfmQR3ucwynQnqZ7?=
=?us-ascii?Q?thBbiZxHjzejygxA6PgS8LWQQ6cVua8CoWZjdQyine85+KMkqoiY2kX6qzna?=
=?us-ascii?Q?1bF1g9tEmwESLPo7Y7B0df04ooRdXcxKtOaPay5v85kr/YO7AeE6qwNGu2pe?=
=?us-ascii?Q?4LfZcuRSsgVujz7pqhSbInmWadyqFa+6bKTNZYaa/RGnOtpEUvlE7w4nOyVU?=
=?us-ascii?Q?UsX9rgNgpHkzQG6h94cp+aLwWBTY6wDVJY/inNa9OrJ20sxo+qtA94kC5Epk?=
=?us-ascii?Q?P+7OMQIf9NK4BQJQK0oqja40EeGgrYrre0tLT3N8azUJ2dBLbCb7qNxreL49?=
=?us-ascii?Q?7fFM9XPXBAsbxjZGKlrqEKB1E/t0MDXTUpdaqeaQghVfuUOizzrcVB2cHlEH?=
=?us-ascii?Q?NfmIUazj06FUfLrrEjuXFwmD3krNGuwwY16+C32AbpBlxHXi+30EJpNzDAth?=
=?us-ascii?Q?ndXT0MCoIXH1okyv3D0TtjvfkSRXwPRwWX+Mjd0qSKr1kFUvLp8GFz+DjsiR?=
=?us-ascii?Q?g0IRXypkwnJZJ/EyZk1PvMdtcxL4cPQxwLNjySHF4VTRi7FXL+msTAoTOUON?=
=?us-ascii?Q?BzvKQSHllvPfuAmPa+gZW8Wr5oyC+DYyDgj0D83Fqdo2oui0KUaKqKy9qxij?=
=?us-ascii?Q?DYd6kiznASgy69OZ7EFgk4ARfUKA7avS?=
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:DM6PR14MB3597.namprd14.prod.outlook.com; PTR:; CAT:NONE;
SFS:(13230040)(366016)(1800799024)(376014)(38070700018); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rhTFrwI/Nbi05hFd2rLOqjUuKHYghyfZ2VNZoSRlkuqz7yj3CpbRc8DofSMP?=
=?us-ascii?Q?z+GFnqQ3m8EifOQEWCtObKl3RWVbgbGP5CQWOKG5n8Q5YLu35IYcG4WOjp41?=
=?us-ascii?Q?pYv8+jN9W4ViH7tgrg5LuKPQ4ISF6UYYBG0Mzi8wMwC0KgRKF2NjP4Dz7Cme?=
=?us-ascii?Q?gvG2essJ0jDUrr9EtcBfDRqmu1AgGIQ11vg79giYVQouLnJN2TDqvO/XUTdk?=
=?us-ascii?Q?VxxM7pSc6ZD0QKTO7dSwJZzg9Zn2D3M+areiml3/9UxZuwUIZT0NmEdDHHOU?=
=?us-ascii?Q?GmIJrLY4wWiUJz5m74LFbe3RdZ2vEAAyOTC24gkQcsaFsrP8MAgGgb1+myi5?=
=?us-ascii?Q?iNzSxnk/WQitpqsB9vfRQZuC81Jdc6Tfl1UDzuxS5+SQvc3caCTFykyix3v+?=
=?us-ascii?Q?Dv0unQLovFls2tI6zRCaGRXc6UI5fH5vND53qaa6sm3dBoEAIBj/GyDB9WMw?=
=?us-ascii?Q?hrUJfaj7CKBOVTBBWLP1BnBUVEWbsxwjpDuSEEW0DlXudonwec+yMOpUOVFT?=
=?us-ascii?Q?OeZagRAM9BXorSJr38WSAase6G+LnOgBieNe2xpOqf9l7RVlnTJYexq/pE2W?=
=?us-ascii?Q?nEOyGKSPquB7rzvQq0u3KMUhY5QaMIknXb/XQj7GVEevF+tSyrvCMVCi97MK?=
=?us-ascii?Q?xKi72/W3TAmto2CM1d6DbvH8+NfMwBdo+hP2NWKDiaR3BD7+Zt7o/tzcDCh1?=
=?us-ascii?Q?T3kaVu4e3Eo/6OQwjfnjZ48U79W5tbN4YSL0+MaD7Pb3zIn7OzLvskfMrAmp?=
=?us-ascii?Q?l8IAzz1JJv6SzcrOd/Klkrql8iv3YPW4AKcvGzZfjIIxZpHhvu9/FZ978fcZ?=
=?us-ascii?Q?X1umbBi6SraUB0nQm9d3E029h9ORmTrQQhOoo8x6DXrWFQAoF2xAgalTO+VY?=
=?us-ascii?Q?PkYGNDN6TqEwPzmNZR8BXlHNpI2mTeINR+7T4k4s0ZQmW/1ZOfmuOHst3f5/?=
=?us-ascii?Q?uiu8Te0f58fPSaF6EEaHzFer6/t0WrDBrbuRpwAueJosI4gl02+j21FOUNip?=
=?us-ascii?Q?zN542wp2xG8+p+fZa/Y9cq+yu+bAzCClKq9Xdtvy518Tl4wcXOysnSe9UMJr?=
=?us-ascii?Q?sn67S5xNwYCMJ/Bv9PrwMLq8RA/+FJoT7BgImIRkrmTE2wUfofElLcXjj05t?=
=?us-ascii?Q?w1ZEicABcd3U3kpPr9yrKdLOWrLEbBHu7nfJhVGqS87WK5DtFSnJIZFMufxP?=
=?us-ascii?Q?nrxnLt3qqSL4ZtUpccS+LW1WIcjILbagroOY7BG+14QjXeK9A8fCwvvOzvkI?=
=?us-ascii?Q?+tvFAo6PTRC7Ac9d1rzL0mAoER8t4LrHinQhIvKknr7PK+iDQqDOeI8WRr6j?=
=?us-ascii?Q?vlbZybBPXG0p8WKV7pLXJQBHpUAwW67mf0P7lZ5YJgGjzO1ZS3B6homJEHXn?=
=?us-ascii?Q?VFwPXBElY+AZknvtzMaXZbhkWIWYTNlExDzoNKgquZQbRxaNkGyqVOfFoHDh?=
=?us-ascii?Q?pFux1rmp0qFfqxKFJKyiNFYw89syOmuGAa7I6UgxT9SbCNHOGk9nJFBv3OxO?=
=?us-ascii?Q?P0XPJTu0miZC5DaiY9njYyesZ3Q/aybXkMA2Ox8Ksg9tLh4Dyji/nTo4zfJP?=
=?us-ascii?Q?gdDvVQ8i61gY9cmIG8s8DrkafHrR0N/kJ7EbS6KPeWRe5kWncR7174whyaWD?=
=?us-ascii?Q?HA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: commscope.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR14MB3597.namprd14.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: abbe955b-6766-406e-6813-08dd6ae487ab
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Mar 2025 14:59:51.7146 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 31472f81-8fe4-49ec-8bc3-fa1c295640d7
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Q6a7vI/ZzKvMltxCVAYW6Mqw+S6NtqxJv5ca2AF6/nsTnfYuAQWxBBwIPf1txeLTtmGRqX8ZmDVu6qPYC0O0De7OuEgo6VLrSK2RYDWCABc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR14MB5937
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
<mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
<mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org
Hi,
I am not sure if this could be your problem, but we had run into a similar =
issue with mbuf pools.
Do you list the DPDK libs in the exact same order when linking Primary and =
Secondary process? If not,=20
the OPs to which the ops_index is pointing to in Primary and in secondary =
is different.
Due to this, rte_mempool_ops_dequeue_bulk called from Secondary does not m=
atch the OPs that Primary had used to populate this memPool (during create=
)
So, accessing the pool from Secondary appears like the pool is empty but n=
on-zero in-use Count.
This happens because each DPDK lib "appends" to the OPs list on the fly as =
it is linked by the linker.=20
So the order of OPs in the OPs database is a mismatch between Primary and S=
econdary. Primary and Secondary share information about OPs using ops_index=
, but since the order is different, ops_index=3DX in Primary not the same O=
Ps in Secondary.
And this leads to the segmentation fault.=20
One of your processes is showing ops_index=3D1, ops_name: <cn9k_mempool_o=
ps>. Can you confirm it is the same in the other process also?
By OPs, and ops_index, I am referring to this internal data structure of DP=
DK fw.=20
struct rte_mempool_ops {
char name[RTE_MEMPOOL_OPS_NAMESIZE]; /**< Name of mempool ops struct. */
rte_mempool_alloc_t alloc; /**< Allocate private data. */
rte_mempool_free_t free; /**< Free the external pool. */
rte_mempool_enqueue_t enqueue; /**< Enqueue an object. */
rte_mempool_dequeue_t dequeue; /**< Dequeue an object. */
rte_mempool_get_count get_count; /**< Get qty of available objs. */
/**
* Optional callback to calculate memory size required to
* store specified number of objects.
*/
rte_mempool_calc_mem_size_t calc_mem_size;
/**
* Optional callback to populate mempool objects using
* provided memory chunk.
*/
rte_mempool_populate_t populate;
/**
* Get mempool info
*/
rte_mempool_get_info_t get_info;
/**
* Dequeue a number of contiguous object blocks.
*/
rte_mempool_dequeue_contig_blocks_t dequeue_contig_blocks;
} __rte_cache_aligned;
Also, the link provides some insight into how OPs is significant.
https://www.intel.com/content/www/us/en/developer/articles/technical/optimi=
ze-memory-usage-in-multi-threaded-data-plane-development-kit-dpdk-applicati=
ons.html
Regards,
Purnima
-----Original Message-----
From: Lombardo, Ed <Ed.Lombardo@netscout.com>=20
Sent: Monday, March 24, 2025 10:32 AM
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: users@dpdk.org
Subject: RE: tailqs issue
CAUTION: This message originated from an External Source outside of CommSco=
pe.com. This may be a phishing email that can result in unauthorized access=
to CommScope. Please use caution when opening attachments, clicking links,=
scanning QR codes, or responding. You can report suspicious emails directl=
y in Microsoft Outlook.
Hi,
Further debugging has left me clueless as to the problem with getting the f=
irst Obj from the mempool on the secondary process to send to the primary p=
rocess (my Application).
PRIMARY:
My application side (primary process) EAL Arguments:
"--proc-type=3Dprimary", "--file-prefix=3Ddpdk_shared", "-l 25,26,27,28", "=
-n4" , "--socket-mem=3D2048", "--legacy-mem",
Create the _MSG_POOL
t_message_pool =3D rte_mempool_create(_MSG_POOL, // Pool name
t_pool_size, // Number of =
elements in the pool
STR_TOKEN_SIZE, // Size of ea=
ch message
t_pool_cache, // Cache size
t_priv_data_sz, // Private da=
ta size
NULL, // mp_init
NULL, // mp_init_ar=
g
NULL, // obj_init
NULL, // obj_init_a=
rg
0, // rte_so=
cket_id(), // socket_id
t_flags // flags
);
The t_message_pool pointer value matches the secondary process when execute=
" message_pool =3D rte_mempool_lookup(_MSG_POOL);"
NTSMON: send ring (0x1bfb43480), recv ring (0x1bfb45a00) and message pool (=
0x14f570ac0) are created successfully.
SECONDARY:
Secondary process I execute: " # ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file=
-prefix=3Ddpdk_shared --proc-type=3Dsecondary --"
Notes:
* hugepages are created 2x1G on NUMA Node 0
[root@localhost ~]# /opt/dpdk/dpdk-hugepages.py -s
Node Pages Size Total
0 2 1Gb 2Gb
* --file-prefix=3Ddpdk_shared is provided to both primary and secondary
* --proc-type is correctly defined for both primary and secondary process.
* The secondary process reports correct socket=3D0 (See dump command output=
below)
* The secondary process showed Available mempool count is 0 and in-use coun=
t is 1024 (which looks incorrect).
* Secondary process reports mempool is empty
* Secondary audit passed (rte_mempool_audit()), no panic occurred.
* Tried disabling ASLR
* Tried turning off legacy-mem
EXECUTE Secondary process application "dpdk-simple_mp_dbg3"
[root@localhost ~]# ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file-prefix=3Ddpd=
k_shared --proc-type=3Dsecondary --legacy-mem --
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/dpdk_shared/mp_socket_221890_1ba056=
2a6ae95
EAL: Selected IOVA mode 'PA'
APP: Finished Process Init.
APP: Availble mempool count is 0, in-use count is 1024, (mempool ptr=3D0x14=
f570ac0)
APP: is mempool empty (1) or full (0)?
APP: check audit on message_pool
APP: Secondary mempool dump file write
APP: NO Objs in message pool (_MSG_POOL), exit the app
simple_mp > Starting core 31
simple_mp > send hello
message_pool pointer is 0x14f570ac0
Failed to get msg obj from mem pool: Success (ret=3D-2)
EAL: PANIC in cmd_send_parsed():
Failed to get message buffer
0: ./dpdk-simple_mp_dbg3 (rte_dump_stack+0x2b) [b1ee1b]
1: ./dpdk-simple_mp_dbg3 (__rte_panic+0xbd) [525ede]
2: ./dpdk-simple_mp_dbg3 (cmd_send_parsed+0x2d8) [7ceb68]
3: ./dpdk-simple_mp_dbg3 (400000+0x6a5f46) [aa5f46]
4: ./dpdk-simple_mp_dbg3 (400000+0x6a4f20) [aa4f20]
5: ./dpdk-simple_mp_dbg3 (rdline_char_in+0x34b) [aa841b]
6: ./dpdk-simple_mp_dbg3 (cmdline_in+0x71) [aa4ff1]
7: ./dpdk-simple_mp_dbg3 (cmdline_interact+0x30) [aa5110]
8: ./dpdk-simple_mp_dbg3 (400000+0xfe5cb) [4fe5cb]
9: /lib64/libc.so.6 (7f2d76600000+0x3feb0) [7f2d7663feb0]
10: /lib64/libc.so.6 (__libc_start_main+0x80) [7f2d7663ff60]
11: ./dpdk-simple_mp_dbg3 (_start+0x25) [7ce795] Aborted (core dumped)
The rte_mempool_dump() is:
mempool <MSG_POOL>@0x14f570ac0
flags=3D1c
socket_id=3D0
pool=3D0x14f56c8c0
iova=3D0x1cf570ac0
nrte_mempool_getb_mem_chunks=3D1
size=3D1024
populated_size=3D1024
header_size=3D64
elt_size=3D64
trailer_size=3D64
total_obj_size=3D192
private_data_size=3D0
ops_index=3D1
ops_name: <cn9k_mempool_ops>
memory chunk at 0x1bfb43180, addr=3D0x14f53c780, iova=3D0x1cf53c780, len=
=3D196800
avg bytes/object=3D192.187500
internal cache infos:
cache_size=3D32 cache_count[0]=3D0
.....
cache_count[127]=3D0
total_cache_count=3D0
common_pool_count=3D1024
no statistics available
Any guidance is appreciated.
Thanks,
Ed
-----Original Message-----
From: Lombardo, Ed
Sent: Friday, March 21, 2025 2:19 PM
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: users@dpdk.org
Subject: RE: tailqs issue
Hi Sephen,
Thank you for your help. I made good progress up to now.
When I try to use the dpdk_simple_mp application to send a message to my ap=
plication I get a Segmentation fault.
First, I re-verified the dpdk_simple_mp process=3Dprimary and dpdk_simple_m=
p process-secondary does pass messages successfully. So, my hugepages are =
created and DPDK initializes successfully on both at startup.
In my application I created the send and recv rings and message_pool as the=
primary process. The logs I added do not show any errors.
Once my application starts and settles I started the dpdk_simple_mp applica=
tion: # ./dpdk-simple_mp_dbg -l 30-31 -n 4 --legacy-mem --proc-type seconda=
ry --
However, on the dpdk_simple_mp side do "send hello" and I then get a segmen=
tation fault.
The debugger takes me deep within the dpdk libraries which I am not too fam=
iliar with.
The rte_ring_elem.h file function: rte_ring_dequeue_build_elem() is where I=
end up with segmentation fault. I notice that the variables are optimized=
out, not sure why since I built the dpdk libraries with debug flag.
Here is the back trace and could you point me in the direction to look.
# gdb dpdk-simple_mp /core/core.dpdk-simple_mp.241269
warning: Unexpected size of section `.reg-xstate/241269' in core file.
[Thread debugging using libthread_db enabled] Using host libthread_db libra=
ry "/lib64/libthread_db.so.1".
Core was generated by `./dpdk-simple_mp -l 30-31 -n 4 --legacy-mem --proc-t=
ype secondary --'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of section `.reg-xstate/241269' in core file.
#0 0x0000000000cf446a in bucket_dequeue () [Current thread is 1 (Thread 0x=
7f946f835c00 (LWP 241269))] Missing separate debuginfos, use: dnf debuginfo=
-install elfutils-libelf-0.189-3.el9.x86_64 glibc-2.34-83.0.1.el9_3.7.x86_6=
4 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libpcap-1.10.0-4.e=
l9.x86_64 libzstd-1.5.1-2.el9.x86_64 numactl-libs-2.0.16-1.el9.x86_64 opens=
sl-libs-3.0.7-25.0.1.el9_3.x86_64 zlib-1.2.11-40.el9.x86_64
(gdb) bt
#0 0x0000000000cf446a in bucket_dequeue ()
#1 0x00000000007ce77d in cmd_send_parsed ()
#2 0x0000000000aa5d96 in __cmdline_parse ()
#3 0x0000000000aa4d70 in cmdline_valid_buffer ()
#4 0x0000000000aa826b in rdline_char_in ()
#5 0x0000000000aa4e41 in cmdline_in ()
#6 0x0000000000aa4f60 in cmdline_interact ()
#7 0x00000000004fe47a in main.cold ()
#8 0x00007f946f03feb0 in __libc_start_call_main () from /lib64/libc.so.6
#9 0x00007f946f03ff60 in __libc_start_main_impl () from /lib64/libc.so.6
#10 0x00000000007ce605 in _start ()
Gdb - stepping through the code, gdb attached to dpdk_simple_mp_debug
(gdb)
0x0000000000cf42c5 in rte_ring_dequeue_bulk_elem (available=3D<optimized ou=
t>, n=3D<optimized out>, esize=3D<optimized out>,
obj_table=3D<optimized out>, r=3D<optimized out>) at ../lib/ring/rte_ri=
ng_elem.h:375
375 ../lib/ring/rte_ring_elem.h: No such file or directory.
(gdb) p r
$17 =3D <optimized out>
(gdb) p obj_table
$18 =3D <optimized out>
(gdb) p available
$19 =3D <optimized out>
(gdb) n
Thread 1 "dpdk-simple_mp_" received signal SIGSEGV, Segmentation fault.
bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0x14f=
05aa80)
at ../drivers/mempool/bucket/rte_mempool_bucket.c:191
191 ../drivers/mempool/bucket/rte_mempool_bucket.c: No such file or dir=
ectory.
(gdb) bt
#0 bucket_dequeue_orphans (n_orphans=3D33, obj_table=3D0x14f09b5c0, bd=3D0=
x14f05aa80)
at ../drivers/mempool/bucket/rte_mempool_bucket.c:191
#1 bucket_dequeue (mp=3D<optimized out>, obj_table=3D0x14f09b5c0, n=3D33) =
at ../drivers/mempool/bucket/rte_mempool_bucket.c:289
#2 0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=3D<optimized out>=
, obj_table=3D0x14f09b5c0, mp=3D0x14f05ed40)
at ../lib/mempool/rte_mempool.h:793
#3 rte_mempool_do_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7=
fff8df066f0, mp=3D0x14f05ed40)
at ../lib/mempool/rte_mempool.h:1570
#4 rte_mempool_generic_get (cache=3D0x14f09b580, n=3D1, obj_table=3D0x7fff=
8df066f0, mp=3D0x14f05ed40)
at ../lib/mempool/rte_mempool.h:1649
#5 rte_mempool_get_bulk (n=3D1, obj_table=3D0x7fff8df066f0, mp=3D0x14f05ed=
40) at ../lib/mempool/rte_mempool.h:1684
#6 rte_mempool_get (obj_p=3D0x7fff8df066f0, mp=3D0x14f05ed40) at ../lib/me=
mpool/rte_mempool.h:1710
#7 cmd_send_parsed (parsed_result=3Dparsed_result@entry=3D0x7fff8df06790, =
cl=3Dcl@entry=3D0x2f73220, data=3Ddata@entry=3D0x0)
at ../examples/multi_process/simple_mp/mp_commands.c:18
#8 0x0000000000aa5d96 in __cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf=
=3D0x2f73268 "send hello\n",
call_fn=3Dcall_fn@entry=3Dtrue) at ../lib/cmdline/cmdline_parse.c:294
#9 0x0000000000aa5f1a in cmdline_parse (cl=3Dcl@entry=3D0x2f73220, buf=3D<=
optimized out>) at ../lib/cmdline/cmdline_parse.c:302
#10 0x0000000000aa4d70 in cmdline_valid_buffer (rdl=3D<optimized out>, buf=
=3D<optimized out>, size=3D<optimized out>)
at ../lib/cmdline/cmdline.c:24
#11 0x0000000000aa826b in rdline_char_in (rdl=3Drdl@entry=3D0x2f73230, c=3D=
<optimized out>)
at ../lib/cmdline/cmdline_rdline.c:444
#12 0x0000000000aa4e41 in cmdline_in (size=3D<optimized out>, buf=3D<optimi=
zed out>, cl=3D<optimized out>)
at ../lib/cmdline/cmdline.c:146
#13 cmdline_in (cl=3D0x2f73220, buf=3D0x7fff8df0c89f "\n\200", size=3D<opti=
mized out>) at ../lib/cmdline/cmdline.c:135
#14 0x0000000000aa4f60 in cmdline_interact (cl=3Dcl@entry=3D0x2f73220) at .=
./lib/cmdline/cmdline.c:192
#15 0x00000000004fe47a in main (argc=3D<optimized out>, argv=3D<optimized o=
ut>)
at ../examples/multi_process/simple_mp/main.c:122
Appreciate if you can help.
Thanks,
Ed
-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Wednesday, March 19, 2025 7:17 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: users@dpdk.org
Subject: Re: tailqs issue
External Email: This message originated outside of NETSCOUT. Do not click l=
inks or open attachments unless you recognize the sender and know the conte=
nt is safe.
On Wed, 19 Mar 2025 21:52:39 +0000
"Lombardo, Ed" <Ed.Lombardo@netscout.com> wrote:
> Hi Stephen,
> I added the fib library, but I now see there are many more dpdk libraries=
I need to add. Is this typically the case with the example files working =
with primary DPDK application?
>
> I am using meson and ninja to build the examples, but I don't know how to=
know the library dependencies.
>
> How do I learn ahead of building my Application as to what extra librarie=
s I need to include for the DPDK example to work?
>
> I am doing incremental build-test-find_missing_library.
>
> So far, I needed to add these: -lrte_fib -lrte_rib -lrte_stack=20
> -lrte_member -lrte_efd
>
> Thanks,
> Ed
The typical case is to make sure that primary and secondary are built with =
the same libraries.