From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A074A4323F; Mon, 30 Oct 2023 12:39:54 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 33AC3402BD; Mon, 30 Oct 2023 12:39:54 +0100 (CET) Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01olkn2084.outbound.protection.outlook.com [40.92.107.84]) by mails.dpdk.org (Postfix) with ESMTP id 5292440285 for ; Mon, 30 Oct 2023 12:39:52 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X0j6z7KVjDy/o1MenIqirpbrVYrjgISlwTWkOKx/0363dx/e/GdjovhZeEZzpgyp9RyM0ka6dGzzO2//ZXeIZfMoyh0YPMFhTZD2Lp8EqoYoM2X1lXfK6vtNc54c1nbSo52AiFdMybb5HbNSvIC/x1e6FBsbpcnJN5nL0w+54Q3bIWA4+tIgF9JgqKpmLLNs6aDuBrKjZGEOheJbaSf6BHVYfDeN1Uerh5VLGygLl9JEcBWIL4FcXXk5/by/g5Um6mUe6avffRu0M9/aq/XJTkcugCf2pO8TfJMs2a1vAUp7sXVcMtVgnPgOdL/q+H2n0jq63GJwUvBpb2f0/JMKzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hnnT8M/3WM0Sy+ubMb/RRzEGIg4eQ2+poz8P+VYyQPs=; b=Vx6kJartIVMqVyc73qRHXC8gi2rKMcUO5GWkatoTQLuuxCcxutadlDHNdD46cROeWMUggerpHKFrvTsMpcNKmJcUKJA6RZ2riKmg31mYDG9dLaQ8KWadcMLT1AxrwSAJldeqpnF9gDOBS/rQeY9NOONnf10fVFB02DnF1F5GE/VtaIaUnwQTfQ2h7UtTwfwYAQFyKaorSVaFVU/z/UuEmn/w8mXaD9hqMcdH1BDpHvDzbcmYBpkEJoJ84BCpmGS/aUraDP4DKIjRFVX2OJQMx27ETXmlZBhps7UO5ZV8DU53KwQdJ1VAsuY8cVGMD7Y4geeAbTpyrGbS/eH2TDznfg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hnnT8M/3WM0Sy+ubMb/RRzEGIg4eQ2+poz8P+VYyQPs=; b=udQhpOM9Z6OF1ttQezT4cFYMF1reyLhqcbTRsR2Qgc8uywoeJ33RqB4My3JnIg2HCQeiEm0fSns+FiRo0/8wN8BMJY/aExuega3HQJoMgTupCBczKNPFL+mqnO/n9cN1C2KFDD8ZMZ4etTIAEwl4jM+msM9+3f/vOyrY4mSe73h+xclMyhKMd3ABygJOTANI7ckB58SMGvYVSE4ePD18Bnav5P/iH+aJCAh+TiH5oNC18O0yqEUD82ygaAZeJ+t8agEqQMmMBeL0cp42XYi8w2qfRhPtZIW8Li4HAaMuGgkf+et0e5kmatn1AqqjapiRHgjOyRflEpVqOCEenIuq0w== Received: from SI2PR06MB4507.apcprd06.prod.outlook.com (2603:1096:4:158::13) by SEYPR06MB6397.apcprd06.prod.outlook.com (2603:1096:101:143::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.27; Mon, 30 Oct 2023 11:39:48 +0000 Received: from SI2PR06MB4507.apcprd06.prod.outlook.com ([fe80::1517:951:b394:2be0]) by SI2PR06MB4507.apcprd06.prod.outlook.com ([fe80::1517:951:b394:2be0%4]) with mapi id 15.20.6933.027; Mon, 30 Oct 2023 11:39:48 +0000 From: Hari Sasank To: Slava Ovsiienko , Matan Azrad , Ori Kam , Suanming Mou CC: "dev@dpdk.org" Subject: Re: [PATCH] net/mlx5: support bus socket with no hugepages Thread-Topic: [PATCH] net/mlx5: support bus socket with no hugepages Thread-Index: AQHZpBJagcgSqkXEKUCQRrbgBvdbGLBi85CAgAAHvMo= Date: Mon, 30 Oct 2023 11:39:48 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [PVvNW2h1AS+UzfvBwEsEaqGpJYzMWva/DwcBDClziF1ibXa+DZVem4fzDdMLNEyIVjYpZ6UD734=] x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2PR06MB4507:EE_|SEYPR06MB6397:EE_ x-ms-office365-filtering-correlation-id: b78ceda7-cc6d-4cd2-ca71-08dbd93cec1e x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Eo6uit/C3mEqqc7lDgKGMSIcAcqIdX2Fyy9xZCyC9WQ9S+tIQeXbZzDPc9vo2FsdcIV1pNVnfTqP1/DnSZ4no2bJo2fWaZFszmEXWLn75esACFj/iS0JdzNDTYF8rq6+UAGew1A1lqlQ87zezX17pjiAEmW54MSRKMwW887HP2BmM+2d4O9GUK/tHhtAU3BB9ajgYinJ/BLDq9pRBgGgnz4x91AoWLT0nZC4dO8ohZMMO8dEHOACdSQaM+V/BLOWkVHAAiYkONRZKDXXIHY2Io0UpiQP69u3UNWKd40fL7tZHBYxKMWP/EeUgBgFGQI9POpLFawjtQHNAfKz1xGSdbwOqMcaN/26EB0UoRwxXRjB7mnh5c81Of+jN2wqbIjo+tjPtgYhgz5HnaQUCDmQwnjTFzFu3vtpQ5WS5kvoAq1x1obJr81vaRJfwsawFMtgDq4u3MhcSwlbRpnFD7WpfAU8qMKFiq5inEGWtuaATpDaaQckzwbB0AC4rWwuXEdAI+HPHVMB13GgZRc9VfHU/NLXuYx+GG2DoNTrgii4mowL0i1UlO3DzP2y0Aa05i13 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ErgqwdGTXKhqZ8IVZMK/u2McDJBfJ9belIJiH2cm4S8m+XiW9Z2nZfnjcMXR?= =?us-ascii?Q?FzRcIUN/x3FoSICf0XJOQhZ51IVNTeJDfvwy8Qldfec3kdaRsAMS8BthBdQK?= =?us-ascii?Q?RblmnRoXXkMvxA2Pi+s2P6wr6F66yoTn1ya+XRTKOx9nZP0ed07L9p711CY4?= =?us-ascii?Q?rm0kzy2J6aQ7Fs2PTkZ0HaUUalhz2OJ1cz2pghiaDVwlPv2isNKU7Jahvkzn?= =?us-ascii?Q?bJf6ftpNBDM2NiLSFsTtskfkZOQ415GUGMKfoaFhDYiUmE397SnHHECZNR0h?= =?us-ascii?Q?8Di0sBVbiGRceiCkuFd70a/ZVtkWpOaOm8DjhvNxmlI7rP0gDeaoAopCyLq2?= =?us-ascii?Q?P/oP0kpcp9qr9N4hOOkIlhVXPeWdFxGkF+4cnRwWAs4Sjt06v427ebbEEK+e?= =?us-ascii?Q?xFqEW/DNngxBnDfXrjXFCd62QISS9JRP9tkQpKxF3JoZ3S3YDaC9a53QjJx9?= =?us-ascii?Q?dlaQuOeTZQpTdInn/un7XGbIiIA9hwy92E8DC5+dd+4cCLPPhdW5BoaHRlhY?= =?us-ascii?Q?rLKRkUZIcWjMs6PJDRU7+lx8SGuk2mzw2M8i+DwVEZCqV7MoZxb0G1n5DIVz?= =?us-ascii?Q?ph7oRbOBYdlK3NSZqFdB3SyrPBUn4QOZXwRWShIbdHvEEESvxiBSeWcb5s+N?= =?us-ascii?Q?89dUzZwIh2YAYqZ4GFLsU3I3g4Yvm90m8k6TMyz6Ci1KNH+N8YsZGgKUdSuu?= =?us-ascii?Q?ZXCME2vjV64lp9j7VqnQ1BruCcaTFSFlxnRSuC/3BPu1PUs+58lqeALELlXl?= =?us-ascii?Q?NnQUaV/6n33/ILCJdRbmDiTtSt6dlP8PsEJZP0LYlpI4Lx7XnQjo/Kz2H/J8?= =?us-ascii?Q?qAFUVMC5vPoTJl6ROzHHy3T6M2dDaztm9UT/5bNGygbgsOGekkvnqklgK+9a?= =?us-ascii?Q?OdJNvi8khSgGQtIoHm9hYpplJyp5GLrM+JfUa9fR/JCOG7uR6zqyfwIYpAb7?= =?us-ascii?Q?WARYWy2GgURO4nPHIuzNVkfR5waMZHeyjgJ1h+zLnE+mPQrLBKbcQrLKBTfo?= =?us-ascii?Q?O1uux+KzTG9SxjN776GWwC7ruECp6XxcCw8WAYP+ukv8K0fCf9K3k5GmVXE4?= =?us-ascii?Q?gjJsaVJSP1OZblehFOlYrf9GBb/bV96m9UmsK1jZPw5GztSgPm0VShr5FhRR?= =?us-ascii?Q?MaHbggFgnI4ir/lNrTzG69X93zUNuGmUEhPgZa7CxhNTgiwrmzp1O16LsI6b?= =?us-ascii?Q?FSFiP4UTIA4GYRshNzTH1SakR+gc+6jo4Z7BZATc98gBLNoU0SE6p0DmhNpR?= =?us-ascii?Q?rdA1Ppi57QOcf2hHXvfAeFu6Kpd5BXBT+Qi0ypeKRg=3D=3D?= Content-Type: multipart/alternative; boundary="_000_SI2PR06MB4507E3615AC7DADB0AC8889ACBA1ASI2PR06MB4507apcp_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2PR06MB4507.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: b78ceda7-cc6d-4cd2-ca71-08dbd93cec1e X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Oct 2023 11:39:48.5513 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEYPR06MB6397 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --_000_SI2PR06MB4507E3615AC7DADB0AC8889ACBA1ASI2PR06MB4507apcp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Slava We have a DPDK application running with all the CPU logical cores belonging= to a single socket on a multi-socket system. But the NIC itself is on a bu= s that is attached to the CPU from a different socket. In this case, the driver is allocating huge pages belonging to a socket tha= t is different from the CPU. But, since huge pages are not available on th= e socket of the NIC card, it crashes the application. (We started this dpdk application saying not to use memory from the socket = where this NIC card is present because that is one of the requirements. We = also probably can't move this NIC from the PCIe bus from this socket as all= the buses from the other socket are in use.). I understand the performance implications of this, so we use this flag to p= ermit the mlx5_malloc to allocate its huge pages from any socket (SOCKET_ID= _ANY seems to try starting with the current cpu lcore socket) only if huge = pages are not available on the NIC socket that the driver is originally cal= led with. I would be happy to discover if there is a better way to solve this. Thanks Hari ________________________________ From: Slava Ovsiienko Sent: Monday, October 30, 2023 11:43 AM To: Hari Sasank ; Matan Azrad ; O= ri Kam ; Suanming Mou Cc: dev@dpdk.org Subject: RE: [PATCH] net/mlx5: support bus socket with no hugepages Hi, Hari As I see almost all updates, using newly introduced MLX5_MEM_FALLBACK_ANY_S= OCKET flag, are related to the memory mapped for the hardware usage (NIC accesses these= areas with DMA over the PCIe bus segment NIC attached to). It means the memory allegiance = to the specific is strong and might be critical for the performance. In general, mlx5 PMD is designed in way requesting the memory on the specif= ic socket only If, and only if it is really needed. Generally speaking, missing huge pages on the socket to which the NIC is at= tached to should be considered as misconfiguration. Could you, please, elaborate a l= ittle bit more, what is a use case for this scenario? With best regards, Slava > -----Original Message----- > From: Hari Sasank > Sent: Wednesday, June 21, 2023 10:31 AM > To: Matan Azrad ; Slava Ovsiienko > ; Ori Kam ; Suanming Mou > > Cc: dev@dpdk.org; Hari Sasank > Subject: [PATCH] net/mlx5: support bus socket with no hugepages > > When a Mellanox NIC is attached to a bus on a numa socket, it tries to > allocate rte memory in that socket. > If hugepages are not configured/available on that rte socket > mlx5_common_pci_probe fails with ENOMEM. > > In this patch, a memflag MLX5_MEM_FALLBACK_ANY_SOCKET is introduced > which when set on mlx5_malloc, will allocate the memory using > SOCKET_ID_ANY if it is not able to allocate memory on the specified socke= t. > This allocates memory on any socket starting with the current thread's > socket. > > Signed-off-by: Hari Sasank > --- > drivers/common/mlx5/mlx5_common_devx.c | 9 ++++++--- > drivers/common/mlx5/mlx5_common_mr.c | 5 +++-- > drivers/common/mlx5/mlx5_malloc.c | 7 +++++++ > drivers/common/mlx5/mlx5_malloc.h | 4 ++++ > drivers/net/mlx5/mlx5.c | 3 ++- > drivers/net/mlx5/mlx5_devx.c | 3 ++- > drivers/net/mlx5/mlx5_rxq.c | 3 ++- > drivers/net/mlx5/mlx5_trigger.c | 6 ++++-- > drivers/net/mlx5/mlx5_txpp.c | 3 ++- > drivers/net/mlx5/mlx5_txq.c | 3 ++- > 10 files changed, 34 insertions(+), 12 deletions(-) > > diff --git a/drivers/common/mlx5/mlx5_common_devx.c > b/drivers/common/mlx5/mlx5_common_devx.c > index 431d8361cebd..122f1c65eab6 100644 > --- a/drivers/common/mlx5/mlx5_common_devx.c > +++ b/drivers/common/mlx5/mlx5_common_devx.c > @@ -107,7 +107,8 @@ mlx5_devx_cq_create(void *ctx, struct > mlx5_devx_cq *cq_obj, uint16_t log_desc_n, > umem_size =3D sizeof(struct mlx5_cqe) * num_of_cqes; > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, socket); > if (!umem_buf) { > DRV_LOG(ERR, "Failed to allocate memory for CQ."); @@ - > 225,7 +226,8 @@ mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq > *sq_obj, uint16_t log_wqbb_n, > umem_size =3D MLX5_WQE_SIZE * num_of_wqbbs; > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, socket); > if (!umem_buf) { > DRV_LOG(ERR, "Failed to allocate memory for SQ."); @@ - > 476,7 +478,8 @@ mlx5_devx_wq_init(void *ctx, uint32_t wqe_size, uint16_t > log_wqbb_n, int socket, > umem_size =3D wqe_size * (1 << log_wqbb_n); > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, socket); > if (!umem_buf) { > DRV_LOG(ERR, "Failed to allocate memory for RQ."); diff -- > git a/drivers/common/mlx5/mlx5_common_mr.c > b/drivers/common/mlx5/mlx5_common_mr.c > index 7b14b0c7bf1e..b2ad6a249732 100644 > --- a/drivers/common/mlx5/mlx5_common_mr.c > +++ b/drivers/common/mlx5/mlx5_common_mr.c > @@ -223,7 +223,8 @@ mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, > int socket) > } > MLX5_ASSERT(!bt->table && !bt->size); > memset(bt, 0, sizeof(*bt)); > - bt->table =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + bt->table =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > sizeof(struct mr_cache_entry) * n, > 0, socket); > if (bt->table =3D=3D NULL) { > @@ -767,7 +768,7 @@ mlx5_mr_create_primary(void *pd, > (void *)addr, data.start, data.end, msl->page_sz, ms_n); > /* Size of memory for bitmap. */ > bmp_size =3D rte_bitmap_get_memory_footprint(ms_n); > - mr =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + mr =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > +MLX5_MEM_FALLBACK_ANY_SOCKET, > RTE_ALIGN_CEIL(sizeof(*mr), > RTE_CACHE_LINE_SIZE) + > bmp_size, RTE_CACHE_LINE_SIZE, msl->socket_id); > if (mr =3D=3D NULL) { > diff --git a/drivers/common/mlx5/mlx5_malloc.c > b/drivers/common/mlx5/mlx5_malloc.c > index c58c41da9266..e109f1bfa994 100644 > --- a/drivers/common/mlx5/mlx5_malloc.c > +++ b/drivers/common/mlx5/mlx5_malloc.c > @@ -182,6 +182,13 @@ mlx5_malloc(uint32_t flags, size_t size, unsigned in= t > align, int socket) > addr =3D rte_zmalloc_socket(NULL, size, align, soc= ket); > else > addr =3D rte_malloc_socket(NULL, size, align, sock= et); > + if (!addr && socket !=3D SOCKET_ID_ANY && > + (flags & MLX5_MEM_FALLBACK_ANY_SOCKET)) { > + if (flags & MLX5_MEM_ZERO) > + addr =3D rte_zmalloc_socket(NULL, size, ali= gn, > SOCKET_ID_ANY); > + else > + addr =3D rte_malloc_socket(NULL, size, alig= n, > SOCKET_ID_ANY); > + } > mlx5_mem_update_msl(addr); > #ifdef RTE_LIBRTE_MLX5_DEBUG > if (addr) > diff --git a/drivers/common/mlx5/mlx5_malloc.h > b/drivers/common/mlx5/mlx5_malloc.h > index 9086a4f3f22e..cd57f95a629e 100644 > --- a/drivers/common/mlx5/mlx5_malloc.h > +++ b/drivers/common/mlx5/mlx5_malloc.h > @@ -28,6 +28,10 @@ enum mlx5_mem_flags { > /* Memory should be allocated from rte hugepage. */ > MLX5_MEM_ZERO =3D 1 << 2, > /* Memory should be cleared to zero. */ > + MLX5_MEM_FALLBACK_ANY_SOCKET =3D 1 << 3, > + /* Memory can be allocated on any socket if > + * it fails to allocate on the given socket. > + */ > }; > > /** > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > f9aea1318736..5b520d468299 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -2063,7 +2063,8 @@ mlx5_proc_priv_init(struct rte_eth_dev *dev) > */ > ppriv_size =3D sizeof(struct mlx5_proc_priv) + > priv->txqs_n * sizeof(struct mlx5_uar_data); > - ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > ppriv_size, > + ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, ppriv_size, > RTE_CACHE_LINE_SIZE, dev->device- > >numa_node); > if (!ppriv) { > rte_errno =3D ENOMEM; > diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c > index 4369d2557e9e..47a925e5913a 100644 > --- a/drivers/net/mlx5/mlx5_devx.c > +++ b/drivers/net/mlx5/mlx5_devx.c > @@ -1285,7 +1285,8 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev > *dev, uint16_t idx) > > RTE_BIT32(host_mem_attr.wq_attr.log_hairpin_num_packets); > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | > MLX5_MEM_ZERO, umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | > MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, priv->sh->numa_node); > if (umem_buf =3D=3D NULL && txq_ctrl- > >hairpin_conf.force_memory) { > DRV_LOG(ERR, "Failed to allocate memory for > hairpin TX queue"); diff --git a/drivers/net/mlx5/mlx5_rxq.c > b/drivers/net/mlx5/mlx5_rxq.c index ad8fd13cbe8e..6bdf1678e499 100644 > --- a/drivers/net/mlx5/mlx5_rxq.c > +++ b/drivers/net/mlx5/mlx5_rxq.c > @@ -1728,7 +1728,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc, > desc >>=3D mprq_log_actual_stride_num; > alloc_size +=3D desc * sizeof(struct mlx5_mprq_buf *); > } > - tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > alloc_size, 0, socket); > + tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, alloc_size, > 0, socket); > if (!tmpl) { > rte_errno =3D ENOMEM; > return NULL; > diff --git a/drivers/net/mlx5/mlx5_trigger.c > b/drivers/net/mlx5/mlx5_trigger.c index bbaa7d2aa021..1a6abdf2b61d > 100644 > --- a/drivers/net/mlx5/mlx5_trigger.c > +++ b/drivers/net/mlx5/mlx5_trigger.c > @@ -55,7 +55,8 @@ mlx5_txq_start(struct rte_eth_dev *dev) > for (i =3D 0; i !=3D priv->txqs_n; ++i) { > struct mlx5_txq_ctrl *txq_ctrl =3D mlx5_txq_get(dev, i); > struct mlx5_txq_data *txq_data =3D &txq_ctrl->txq; > - uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO; > + uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET; > > if (!txq_ctrl) > continue; > @@ -180,7 +181,8 @@ mlx5_rxq_ctrl_prepare(struct rte_eth_dev *dev, > struct mlx5_rxq_ctrl *rxq_ctrl, > return ret; > } > MLX5_ASSERT(!rxq_ctrl->obj); > - rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > sizeof(*rxq_ctrl->obj), 0, > rxq_ctrl->socket); > if (!rxq_ctrl->obj) { > diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c > index 5a5df2d1bb16..c81ae7bfd328 100644 > --- a/drivers/net/mlx5/mlx5_txpp.c > +++ b/drivers/net/mlx5/mlx5_txpp.c > @@ -394,7 +394,8 @@ mlx5_txpp_create_clock_queue(struct > mlx5_dev_ctx_shared *sh) > struct mlx5_txpp_wq *wq =3D &sh->txpp.clock_queue; > int ret; > > - sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > MLX5_TXPP_REARM_SQ_SIZE * > sizeof(struct mlx5_txpp_ts), > 0, sh->numa_node); > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c in= dex > 8cb52b0f7d8e..b83e798544d2 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -1074,7 +1074,8 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc, > struct mlx5_priv *priv =3D dev->data->dev_private; > struct mlx5_txq_ctrl *tmpl; > > - tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > sizeof(*tmpl) + > + tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > sizeof(*tmpl) + > desc * sizeof(struct rte_mbuf *), 0, socket); > if (!tmpl) { > rte_errno =3D ENOMEM; > -- > 2.39.2 --_000_SI2PR06MB4507E3615AC7DADB0AC8889ACBA1ASI2PR06MB4507apcp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Slava

We have a DPDK application running with all the CPU logical cores belonging= to a single socket on a multi-socket system. But the NIC itself is on a bu= s that is attached to the CPU from a different socket. 
In this case, the driver is allocating huge pages belonging to a socket tha= t is different from the CPU.  But, since huge pages are not available = on the socket of the NIC card, it crashes the application.
(We started this dpdk application sa= ying not to use memory from the socket where this NIC card is present becau= se that is one of the requirements. We also probably can't move this NIC from the PCIe bus from this socket as= all the buses from the other socket are in use.).

I understand the performance implications of this, so= we use this flag to permit the mlx5_malloc to allocate its huge pages from any socket (SOCKET_ID_ANY seems to try starting with the cu= rrent cpu lcore socket) only if huge pages are not available on the NIC socket that the driver is originally called with.

I would be happy to discover if= there is a better way to solve this. 

Thanks
Hari
From: Slava Ovsiienko <v= iacheslavo@nvidia.com>
Sent: Monday, October 30, 2023 11:43 AM
To: Hari Sasank <harisasank@outlook.com>; Matan Azrad <mata= n@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou <suanmi= ngm@nvidia.com>
Cc: dev@dpdk.org <dev@dpdk.org>
Subject: RE: [PATCH] net/mlx5: support bus socket with no hugepages<= /font>
 
Hi, Hari

As I see almost all updates, using newly introduced MLX5_MEM_FALLBACK_ANY_S= OCKET flag,
are related to the memory mapped for the hardware usage (NIC accesses these= areas with DMA
over the PCIe bus segment NIC attached to). It means the memory allegiance = to the specific
is strong and might be critical for the performance.

In general, mlx5 PMD is designed in way requesting the memory on the specif= ic socket only
If, and only if it is really needed.

Generally speaking, missing huge pages on the socket to which the NIC is at= tached to
should be considered as misconfiguration.  Could you, please, elaborat= e a little bit more, what is a use case
for this scenario?

With best regards,
Slava

> -----Original Message-----
> From: Hari Sasank <harisasank@outlook.com>
> Sent: Wednesday, June 21, 2023 10:31 AM
> To: Matan Azrad <matan@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suan= ming Mou
> <suanmingm@nvidia.com>
> Cc: dev@dpdk.org; Hari Sasank <harisasank@outlook.com>
> Subject: [PATCH] net/mlx5: support bus socket with no hugepages
>
> When a Mellanox NIC is attached to a bus on a numa socket, it tries to=
> allocate rte memory in that socket.
> If hugepages are not configured/available on that rte socket
> mlx5_common_pci_probe fails with ENOMEM.
>
> In this patch, a memflag MLX5_MEM_FALLBACK_ANY_SOCKET is introduced > which when set on mlx5_malloc, will allocate the memory using
> SOCKET_ID_ANY if it is not able to allocate memory on the specified so= cket.
> This allocates memory on any socket starting with the current thread's=
> socket.
>
> Signed-off-by: Hari Sasank <harisasank@outlook.com>
> ---
>  drivers/common/mlx5/mlx5_common_devx.c | 9 ++++++---
>  drivers/common/mlx5/mlx5_common_mr.c   | 5 +++--
>  drivers/common/mlx5/mlx5_malloc.c      = | 7 +++++++
>  drivers/common/mlx5/mlx5_malloc.h      = | 4 ++++
>  drivers/net/mlx5/mlx5.c       = ;         | 3 ++-
>  drivers/net/mlx5/mlx5_devx.c      =      | 3 ++-
>  drivers/net/mlx5/mlx5_rxq.c      &= nbsp;     | 3 ++-
>  drivers/net/mlx5/mlx5_trigger.c     &nb= sp;  | 6 ++++--
>  drivers/net/mlx5/mlx5_txpp.c      =      | 3 ++-
>  drivers/net/mlx5/mlx5_txq.c      &= nbsp;     | 3 ++-
>  10 files changed, 34 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/common/mlx5/mlx5_common_devx.c
> b/drivers/common/mlx5/mlx5_common_devx.c
> index 431d8361cebd..122f1c65eab6 100644
> --- a/drivers/common/mlx5/mlx5_common_devx.c
> +++ b/drivers/common/mlx5/mlx5_common_devx.c
> @@ -107,7 +107,8 @@ mlx5_devx_cq_create(void *ctx, struct
> mlx5_devx_cq *cq_obj, uint16_t log_desc_n,
>        umem_size =3D sizeof(struct = mlx5_cqe) * num_of_cqes;
>        umem_dbrec =3D RTE_ALIGN(ume= m_size, MLX5_DBR_SIZE);
>        umem_size +=3D MLX5_DBR_SIZE= ;
> -     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO,
> umem_size,
> +     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO |
> +           &nb= sp;            =     MLX5_MEM_FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;      alignment, socket);
>        if (!umem_buf) {
>            = ;    DRV_LOG(ERR, "Failed to allocate memory for CQ.&qu= ot;); @@ -
> 225,7 +226,8 @@ mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq
> *sq_obj, uint16_t log_wqbb_n,
>        umem_size =3D MLX5_WQE_SIZE = * num_of_wqbbs;
>        umem_dbrec =3D RTE_ALIGN(ume= m_size, MLX5_DBR_SIZE);
>        umem_size +=3D MLX5_DBR_SIZE= ;
> -     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO,
> umem_size,
> +     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO |
> +           &nb= sp;            =     MLX5_MEM_FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;      alignment, socket);
>        if (!umem_buf) {
>            = ;    DRV_LOG(ERR, "Failed to allocate memory for SQ.&qu= ot;); @@ -
> 476,7 +478,8 @@ mlx5_devx_wq_init(void *ctx, uint32_t wqe_size, uint16= _t
> log_wqbb_n, int socket,
>        umem_size =3D wqe_size * (1 = << log_wqbb_n);
>        umem_dbrec =3D RTE_ALIGN(ume= m_size, MLX5_DBR_SIZE);
>        umem_size +=3D MLX5_DBR_SIZE= ;
> -     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO,
> umem_size,
> +     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO |
> +           &nb= sp;            =     MLX5_MEM_FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;      alignment, socket);
>        if (!umem_buf) {
>            = ;    DRV_LOG(ERR, "Failed to allocate memory for RQ.&qu= ot;); diff --
> git a/drivers/common/mlx5/mlx5_common_mr.c
> b/drivers/common/mlx5/mlx5_common_mr.c
> index 7b14b0c7bf1e..b2ad6a249732 100644
> --- a/drivers/common/mlx5/mlx5_common_mr.c
> +++ b/drivers/common/mlx5/mlx5_common_mr.c
> @@ -223,7 +223,8 @@ mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n= ,
> int socket)
>        }
>        MLX5_ASSERT(!bt->table &a= mp;& !bt->size);
>        memset(bt, 0, sizeof(*bt));<= br> > -     bt->table =3D mlx5_malloc(MLX5_MEM_RTE | = MLX5_MEM_ZERO,
> +     bt->table =3D mlx5_malloc(MLX5_MEM_RTE | = MLX5_MEM_ZERO |
> +           &nb= sp;            =      MLX5_MEM_FALLBACK_ANY_SOCKET,
>            = ;            &n= bsp;       sizeof(struct mr_cache_entry) * n,=
>            = ;            &n= bsp;       0, socket);
>        if (bt->table =3D=3D NULL= ) {
> @@ -767,7 +768,7 @@ mlx5_mr_create_primary(void *pd,
>            = ;  (void *)addr, data.start, data.end, msl->page_sz, ms_n);
>        /* Size of memory for bitmap= . */
>        bmp_size =3D rte_bitmap_get_= memory_footprint(ms_n);
> -     mr =3D mlx5_malloc(MLX5_MEM_RTE |  MLX5= _MEM_ZERO,
> +     mr =3D mlx5_malloc(MLX5_MEM_RTE |  MLX5= _MEM_ZERO |
> +MLX5_MEM_FALLBACK_ANY_SOCKET,
>            = ;             R= TE_ALIGN_CEIL(sizeof(*mr),
> RTE_CACHE_LINE_SIZE) +
>            = ;             b= mp_size, RTE_CACHE_LINE_SIZE, msl->socket_id);
>        if (mr =3D=3D NULL) {
> diff --git a/drivers/common/mlx5/mlx5_malloc.c
> b/drivers/common/mlx5/mlx5_malloc.c
> index c58c41da9266..e109f1bfa994 100644
> --- a/drivers/common/mlx5/mlx5_malloc.c
> +++ b/drivers/common/mlx5/mlx5_malloc.c
> @@ -182,6 +182,13 @@ mlx5_malloc(uint32_t flags, size_t size, unsigned= int
> align, int socket)
>            = ;            addr = =3D rte_zmalloc_socket(NULL, size, align, socket);
>            = ;    else
>            = ;            addr = =3D rte_malloc_socket(NULL, size, align, socket);
> +           &nb= sp; if (!addr && socket !=3D SOCKET_ID_ANY &&
> +           &nb= sp;     (flags & MLX5_MEM_FALLBACK_ANY_SOCKET)) { > +           &nb= sp;         if (flags & MLX5_ME= M_ZERO)
> +           &nb= sp;            =      addr =3D rte_zmalloc_socket(NULL, size, align,
> SOCKET_ID_ANY);
> +           &nb= sp;         else
> +           &nb= sp;            =      addr =3D rte_malloc_socket(NULL, size, align,
> SOCKET_ID_ANY);
> +           &nb= sp; }
>            = ;    mlx5_mem_update_msl(addr);
>  #ifdef RTE_LIBRTE_MLX5_DEBUG
>            = ;    if (addr)
> diff --git a/drivers/common/mlx5/mlx5_malloc.h
> b/drivers/common/mlx5/mlx5_malloc.h
> index 9086a4f3f22e..cd57f95a629e 100644
> --- a/drivers/common/mlx5/mlx5_malloc.h
> +++ b/drivers/common/mlx5/mlx5_malloc.h
> @@ -28,6 +28,10 @@ enum mlx5_mem_flags {
>        /* Memory should be allocate= d from rte hugepage. */
>        MLX5_MEM_ZERO =3D 1 <<= 2,
>        /* Memory should be cleared = to zero. */
> +     MLX5_MEM_FALLBACK_ANY_SOCKET =3D 1 << = 3,
> +     /* Memory can be allocated on any socket if<= br> > +      * it fails to allocate on the given so= cket.
> +      */
>  };
>
>  /**
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > f9aea1318736..5b520d468299 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -2063,7 +2063,8 @@ mlx5_proc_priv_init(struct rte_eth_dev *dev)
>         */
>        ppriv_size =3D sizeof(struct= mlx5_proc_priv) +
>            = ;         priv->txqs_n * sizeof(= struct mlx5_uar_data);
> -     ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_ME= M_ZERO,
> ppriv_size,
> +     ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_ME= M_ZERO |
> +           &nb= sp;            = MLX5_MEM_FALLBACK_ANY_SOCKET, ppriv_size,
>            = ;            &n= bsp;   RTE_CACHE_LINE_SIZE, dev->device-
> >numa_node);
>        if (!ppriv) {
>            = ;    rte_errno =3D ENOMEM;
> diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx= .c
> index 4369d2557e9e..47a925e5913a 100644
> --- a/drivers/net/mlx5/mlx5_devx.c
> +++ b/drivers/net/mlx5/mlx5_devx.c
> @@ -1285,7 +1285,8 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev
> *dev, uint16_t idx)
>
>        RTE_BIT32(host_mem_attr.wq_a= ttr.log_hairpin_num_packets);
>            = ;    umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE);
>            = ;    umem_size +=3D MLX5_DBR_SIZE;
> -           &nb= sp; umem_buf =3D mlx5_malloc(MLX5_MEM_RTE |
> MLX5_MEM_ZERO, umem_size,
> +           &nb= sp; umem_buf =3D mlx5_malloc(MLX5_MEM_RTE |
> MLX5_MEM_ZERO |
> +           &nb= sp;            =             MLX5_MEM= _FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;            = ;  alignment, priv->sh->numa_node);
>            = ;    if (umem_buf =3D=3D NULL && txq_ctrl-
> >hairpin_conf.force_memory) {
>            = ;            DRV_LOG= (ERR, "Failed to allocate memory for
> hairpin TX queue"); diff --git a/drivers/net/mlx5/mlx5_rxq.c
> b/drivers/net/mlx5/mlx5_rxq.c index ad8fd13cbe8e..6bdf1678e499 100644<= br> > --- a/drivers/net/mlx5/mlx5_rxq.c
> +++ b/drivers/net/mlx5/mlx5_rxq.c
> @@ -1728,7 +1728,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc,
>            = ;    desc >>=3D mprq_log_actual_stride_num;
>            = ;    alloc_size +=3D desc * sizeof(struct mlx5_mprq_buf *);<= br> >        }
> -     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO,
> alloc_size, 0, socket);
> +     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO |
> +           &nb= sp;            MLX5_= MEM_FALLBACK_ANY_SOCKET, alloc_size,
> 0, socket);
>        if (!tmpl) {
>            = ;    rte_errno =3D ENOMEM;
>            = ;    return NULL;
> diff --git a/drivers/net/mlx5/mlx5_trigger.c
> b/drivers/net/mlx5/mlx5_trigger.c index bbaa7d2aa021..1a6abdf2b61d
> 100644
> --- a/drivers/net/mlx5/mlx5_trigger.c
> +++ b/drivers/net/mlx5/mlx5_trigger.c
> @@ -55,7 +55,8 @@ mlx5_txq_start(struct rte_eth_dev *dev)
>        for (i =3D 0; i !=3D priv-&g= t;txqs_n; ++i) {
>            = ;    struct mlx5_txq_ctrl *txq_ctrl =3D mlx5_txq_get(dev, i)= ;
>            = ;    struct mlx5_txq_data *txq_data =3D &txq_ctrl->tx= q;
> -           &nb= sp; uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO;
> +           &nb= sp; uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO |
> +           &nb= sp;            =       MLX5_MEM_FALLBACK_ANY_SOCKET;
>
>            = ;    if (!txq_ctrl)
>            = ;            continu= e;
> @@ -180,7 +181,8 @@ mlx5_rxq_ctrl_prepare(struct rte_eth_dev *dev,
> struct mlx5_rxq_ctrl *rxq_ctrl,
>            = ;            return = ret;
>        }
>        MLX5_ASSERT(!rxq_ctrl->ob= j);
> -     rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RT= E | MLX5_MEM_ZERO,
> +     rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RT= E | MLX5_MEM_ZERO |
> +           &nb= sp;            =          MLX5_MEM_FALLBACK_ANY_SOCK= ET,
>            = ;            &n= bsp;           sizeof(*rx= q_ctrl->obj), 0,
>            = ;            &n= bsp;           rxq_ctrl-&= gt;socket);
>        if (!rxq_ctrl->obj) {
> diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp= .c
> index 5a5df2d1bb16..c81ae7bfd328 100644
> --- a/drivers/net/mlx5/mlx5_txpp.c
> +++ b/drivers/net/mlx5/mlx5_txpp.c
> @@ -394,7 +394,8 @@ mlx5_txpp_create_clock_queue(struct
> mlx5_dev_ctx_shared *sh)
>        struct mlx5_txpp_wq *wq =3D = &sh->txpp.clock_queue;
>        int ret;
>
> -     sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE= | MLX5_MEM_ZERO,
> +     sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE= | MLX5_MEM_ZERO |
> +           &nb= sp;            =         MLX5_MEM_FALLBACK_ANY_SOCKET, >            = ;            &n= bsp;          MLX5_TXPP_REARM_= SQ_SIZE *
>            = ;            &n= bsp;          sizeof(struct ml= x5_txpp_ts),
>            = ;            &n= bsp;          0, sh->numa_n= ode);
> diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c= index
> 8cb52b0f7d8e..b83e798544d2 100644
> --- a/drivers/net/mlx5/mlx5_txq.c
> +++ b/drivers/net/mlx5/mlx5_txq.c
> @@ -1074,7 +1074,8 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc,
>        struct mlx5_priv *priv =3D d= ev->data->dev_private;
>        struct mlx5_txq_ctrl *tmpl;<= br> >
> -     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO,
> sizeof(*tmpl) +
> +     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO |
> +           &nb= sp;            MLX5_= MEM_FALLBACK_ANY_SOCKET,
> sizeof(*tmpl) +
>            = ;            &n= bsp;  desc * sizeof(struct rte_mbuf *), 0, socket);
>        if (!tmpl) {
>            = ;    rte_errno =3D ENOMEM;
> --
> 2.39.2

--_000_SI2PR06MB4507E3615AC7DADB0AC8889ACBA1ASI2PR06MB4507apcp_--