From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3588943244; Mon, 30 Oct 2023 16:45:31 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AF54F40285; Mon, 30 Oct 2023 16:45:30 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2043.outbound.protection.outlook.com [40.107.244.43]) by mails.dpdk.org (Postfix) with ESMTP id 53CEA4026F for ; Mon, 30 Oct 2023 16:45:29 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=csxNMN1XEw3Xltewne1BjhYgQGKs7bKV8XH3dM2TUbEDS0e/cGkIE3z3Vmm0GDpARBpgXaC0UdRUqy7gBVpNyiOX+q9llX323Ldf7DRxulK3U22iVPjwnTZWqL//B13TyAUGRQWIXsRvoxoFS5UGbZPVYtPa5rwPYfDzBjH4QCASCLcrZkLUX0Alm0IiAC1Uj+7U0oaqEUSJBFHq4PvwoR8UPXK+QdjaDl0DFmHjvpbFCZpB5mqKtE3kiU43Qxky9GpZCFJ5kVjiWFDSdQ/uv//0w4RCp3iS5yQOb+MqqbLTdEF3fQmWVjZomDD0yHwmss6h//nTGPMw+3FFVhk9mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KwTFhVmL1b2M0zFGYxc2F6dIp2xRrds47SCfUEiFxFU=; b=ll0r+N5bqwVGif3riamMpMyq4WTKYfUMoiLqqCZZNOvnqn5JOkKbBp+eh88OU32FniKWCIdHwek3DZyWD+dbXSZNY0YvT1ILhOeqc72OmZFADN+FRgH8y8t5A5+Mg9CRzlhJmdYSrlLREy+JF/KMCCJUGO82fgJoFpmrw9JxjqoKyu4y+fvvcIduT3Su5Cx9Q2qVe5RM9qHLgOotR5GVdY15ehXOrmJlm6AI6L6Prqhj1DgqQZkeMGDNnIxy/ABsx4ggQyQGKoPoo/WfsiJHn/6idugIk4Wj89Tgo2WCmqAWlK2+rwRLlb0NtxOL8I5YH5nwJDxZBth6mIsBfjC8cw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KwTFhVmL1b2M0zFGYxc2F6dIp2xRrds47SCfUEiFxFU=; b=Nz0bsRx9ooDW+p4kw3D3rFVF3x5CG9heW6o/ojPP5cH+jEWYmbg1suVIN13+7O0r0Wmglt9AHXGMzkEGR+ukH5T18tLNrue9apuqdywtvdTc83knJTv7faF0c5WGP8FFT+fT5Hleq4HfnoQ3jZN/gD92NK/Oxq00TRlvdGhSRfm/cJlvypWHD8gy3mDtw5Cf7Fc79nLhE4gi6CMlXkg7/68QQAiO50fxJrFa11jlP9UP97lMzWDiIQgSLf8o12FxO9NwlIlRIspo+KP0M9kBfsmE/HBuABunGhHk3tbwd0aFdv5O3Bn/2NpAg7s7491BLu/bFYXneSrgBueuON4NHw== Received: from DM6PR12MB3753.namprd12.prod.outlook.com (2603:10b6:5:1c7::18) by SA1PR12MB7038.namprd12.prod.outlook.com (2603:10b6:806:24d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.27; Mon, 30 Oct 2023 15:45:26 +0000 Received: from DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::5fd9:c0c:398c:7dfb]) by DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::5fd9:c0c:398c:7dfb%2]) with mapi id 15.20.6933.028; Mon, 30 Oct 2023 15:45:26 +0000 From: Slava Ovsiienko To: Hari Sasank , Matan Azrad , Ori Kam , Suanming Mou CC: "dev@dpdk.org" Subject: RE: [PATCH] net/mlx5: support bus socket with no hugepages Thread-Topic: [PATCH] net/mlx5: support bus socket with no hugepages Thread-Index: AQHZpBJgCjjEVoNlmkScjpWY6t+HE7Bi8SzQgAASQQCAAEHKEA== Date: Mon, 30 Oct 2023 15:45:26 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR12MB3753:EE_|SA1PR12MB7038:EE_ x-ms-office365-filtering-correlation-id: 2b6d3792-b288-45d8-ee19-08dbd95f3c77 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: SEzTAzN9BjOValS7R9yy9yV/Q47auOKdIZksesvjyEY/vIYumit75bYNZ9uu3fyns0ADJXkqR1WWzjW2HWQHQvR+JYHuGm6akcp6X/1cxuRIQZmvSL4imN0Ep6ZXbnc4dDH1A03Fa9IzP21KOD17WkTotdBLFgvu12UrATqt5aclLx0Tnio472brqGTj7z/G2W5U/xQ5H9hOj1pIZl6SMEPdVfIBMrVD5hImZ/2GeC1BXglzaxbhCvODy/qPj1IS497qM1EWC2s82D3B0lMsESl2d8/YsmWvwd8vgucZZ2qYMsVjeqaJ5+ul0claPnBHspFTDho2DSnScAluwf++GtNNEUVLIWcXlTgOIvxEqSWzcDnkLd9myGFAz3hmo1InYk4tKsBtdAQsLbndsq8Fha9oo9/fS0xzdFF39R9FSP8+6t8rmnj2dFSflSV8lwsdNgbMYXYy4hvHk+VS3SJ4quvsIgcXBImlrLnR8PnlK6Ki+uQm6rXc5mFhB2FMTMrI4edRz2EolqnaE9dVVsu+2V9+kKXtJZAJXgnkt+W6Df3FVHQZa3v22ksHuJJ1uehwK7JmDmd85siCIPtUzMzez9LwLVbgsxeIrdtp5WJTds3NcVnABLwizBSvcq2E6CPh x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB3753.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376002)(396003)(39860400002)(366004)(346002)(136003)(230922051799003)(186009)(451199024)(64100799003)(1800799009)(83380400001)(38070700009)(38100700002)(122000001)(478600001)(2906002)(30864003)(9686003)(7696005)(6506007)(64756008)(66476007)(45080400002)(8676002)(8936002)(4326008)(76116006)(66556008)(66946007)(110136005)(66446008)(6636002)(316002)(52536014)(33656002)(5660300002)(41300700001)(26005)(55016003)(53546011)(86362001)(71200400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?hYYXdNHN/OxQ05ad/DIwQT1jf+FYQ+mvfaFDNG8Z+dALbJhlAQkSp4Hk84ad?= =?us-ascii?Q?crs0VorFMikm/95kAjICukPlyqXzWj9YStlzhJEbDs7NJ7twUF8cNBARD/m9?= =?us-ascii?Q?2rwPbhldIAqN7x+x8bpwrELSfD241xYECyVHm/nkjnG3O7i11sbqmsRNKZhe?= =?us-ascii?Q?MC9TGVoDWJ3FsjJ/EEdVOZkyiJDN3qlwZ1ta3gJX0vxjdy2MNcQqdDyc/woa?= =?us-ascii?Q?2+aXxV+ts3fOq1F5YFUQRC3Ekl2iViek8Jdigqct+hx4oFUfhILZCK4+434d?= =?us-ascii?Q?Z+z+t+piaFymbJ+wmTvAq5Xvkzynm8Hvaa4V4yTOEzp9yL1TJlzad2Wn6yEa?= =?us-ascii?Q?+FyTI5t0zrbcMJ3MSKUoYJ4tho2Lm0QCNdycGYFhJRsZ9VH6g4Y/WbGCL89k?= =?us-ascii?Q?kKwcXoDJW8IxgiGgXtVUXncF/pcvWdOXsTw38FmgQ/1CWkZtSnc2ZOLtQ+oL?= =?us-ascii?Q?wuCiHztFWOwtYfvMTgNCNzIJAPyeJA7gPuVvIS3rC553YYmp7KYL78/knm6G?= =?us-ascii?Q?nh3RuAlc76gT269ZjzhcbbTF7j3Wmr6FibKjVYNUSRIgeAgHpqhS7et242mS?= =?us-ascii?Q?zIkUDO53c1G0XwErBFVwrSXowLjxQaBu23WMfqFQeowuHgTbaho4aQirsMf+?= =?us-ascii?Q?O1qUsR+kgB+GBGhyWTtgQwTtOFry4legHVTtXIIR5xpdJP+anvh7Y1k2apun?= =?us-ascii?Q?CDEf9SEA6n1DenMabjqwClCXTd8YSeuguVnPqw+We1HzFFcHSh3kLXHWbOp7?= =?us-ascii?Q?JzxFO5zzZAaNgAazHn4BqxOQREPHHWUTnNyUoUZr6yMXZ562vpYEhHBheHRN?= =?us-ascii?Q?Jx3e5bvvbd4L8gAXWWoBz7BGusU2zzckxhT62rvfEtI7qPQKWFiQVm1BSDcN?= =?us-ascii?Q?LhyPf1a82ys4xHStuDym+/w17tOyuybPC2z7J/8qi7lDwXUz6ZwTTCZqDKMZ?= =?us-ascii?Q?8JRvs6xUotm5riJ4uRi4eawD+4Exp8afAxCZP0W5y2+PdoANM2Zpcm+wxjjU?= =?us-ascii?Q?scvk5/ZeFvO+uJgUMPrwdp40fEAp4CXlqtxo1OJ6UgJ2UaMuFLTCRXSKy9yy?= =?us-ascii?Q?6poi6UNLMhrwBps/g4GfqyXvvlby66FqnzVZv9ioxdjEW5hIn1eXtUkcklxn?= =?us-ascii?Q?VwJ/nSfTN9pbFtSEdXTzUGU10kcyjXA3U90WB5c/KFGgJtoMYUp7oDLvWr4a?= =?us-ascii?Q?3o/8i0NbNZJvzqHSLxW6hUOfyx74dcaQ6RxWJB1uOtTibomlkZRr9Xu1bIzU?= =?us-ascii?Q?ye4EhWNCH8zf2OogKNW8v3+K1zx0rLKIpNSYCkKNFXo2wsA45E5UgMnfxdEL?= =?us-ascii?Q?wCSSztsr07dYgHxDmU1mSqG4U3bjMuwHZH5m29GlTI9G11aCvz5vw4cvOVPc?= =?us-ascii?Q?CS686/EmhmSEDshvbhX+nkrqjuzIfosbJjn77yjjRo4xYyQWfq1n67Jo8RT/?= =?us-ascii?Q?s7gDDiZGhFAHt5zP1WgpTGjjGATlxXsBzTqntrrOY+7+xt8BF78eVkouxEZ4?= =?us-ascii?Q?DfhEF9yYzkybv0Oc7ohhq+dV5lxjXn4Oe1jlk2J+fdCUdNiRBDY3QwdZV7sU?= =?us-ascii?Q?iwfc8bTM5FdzNQtfaVFp0WpRKpzDLNPtlqismy+C?= Content-Type: multipart/alternative; boundary="_000_DM6PR12MB3753FCEB708F27B0DAEFD5BFDFA1ADM6PR12MB3753namp_" MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3753.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2b6d3792-b288-45d8-ee19-08dbd95f3c77 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Oct 2023 15:45:26.2582 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ioUOfTKjLpITpXHJJzHmzoCp+sPx7tML+wtXUyTICriNkx7KEt7Y2J7qKDfaf6YGkz3U4rhRpnpog21aP5+HuA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7038 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --_000_DM6PR12MB3753FCEB708F27B0DAEFD5BFDFA1ADM6PR12MB3753namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Hari 1. >> We have a DPDK application running with all the CPU logical cores = belonging to a single socket on a multi-socket system. 2. >> But the NIC itself is on a bus that is attached to the CPU from a = different socket 3. >> socket where this NIC card is present because that is one of the r= equirements This is quite not typical use case - running the driver not close to the H= W. All transactions between the NIC and remote (not on the same socket) memory will cause memory subsystem overhead - QPI (or other inter-socket bu= s) will be highly loaded, MESI protocols will be engaged, and so on. I believe you have strong reasons to choose this configuration, but this is= certainly the mlx5 driver was not designed for. I would prefer to have the way to detect this socket misconfiguration for m= ore common scenarios (now mlx5 just fails). What about introducing the some devargs "allow_any_socket", allowing "wron= g socket" allocation? With best regards, Slava From: Hari Sasank Sent: Monday, October 30, 2023 1:40 PM To: Slava Ovsiienko ; Matan Azrad ; Ori Kam ; Suanming Mou Cc: dev@dpdk.org Subject: Re: [PATCH] net/mlx5: support bus socket with no hugepages Hi Slava We have a DPDK application running with all the CPU logical cores belonging= to a single socket on a multi-socket system. But the NIC itself is on a bu= s that is attached to the CPU from a different socket. In this case, the driver is allocating huge pages belonging to a socket tha= t is different from the CPU. But, since huge pages are not available on th= e socket of the NIC card, it crashes the application. (We started this dpdk application saying not to use memory from the socket = where this NIC card is present because that is one of the requirements. We = also probably can't move this NIC from the PCIe bus from this socket as all= the buses from the other socket are in use.). I understand the performance implications of this, so we use this flag to p= ermit the mlx5_malloc to allocate its huge pages from any socket (SOCKET_ID= _ANY seems to try starting with the current cpu lcore socket) only if huge = pages are not available on the NIC socket that the driver is originally cal= led with. I would be happy to discover if there is a better way to solve this. Thanks Hari ________________________________ From: Slava Ovsiienko > Sent: Monday, October 30, 2023 11:43 AM To: Hari Sasank >; Ma= tan Azrad >; Ori Kam >; Suanming Mou > Cc: dev@dpdk.org > Subject: RE: [PATCH] net/mlx5: support bus socket with no hugepages Hi, Hari As I see almost all updates, using newly introduced MLX5_MEM_FALLBACK_ANY_S= OCKET flag, are related to the memory mapped for the hardware usage (NIC accesses these= areas with DMA over the PCIe bus segment NIC attached to). It means the memory allegiance = to the specific is strong and might be critical for the performance. In general, mlx5 PMD is designed in way requesting the memory on the specif= ic socket only If, and only if it is really needed. Generally speaking, missing huge pages on the socket to which the NIC is at= tached to should be considered as misconfiguration. Could you, please, elaborate a l= ittle bit more, what is a use case for this scenario? With best regards, Slava > -----Original Message----- > From: Hari Sasank > > Sent: Wednesday, June 21, 2023 10:31 AM > To: Matan Azrad >; Slava Ovsiie= nko > >; Ori Kam >; Suanming Mou > > > Cc: dev@dpdk.org; Hari Sasank > > Subject: [PATCH] net/mlx5: support bus socket with no hugepages > > When a Mellanox NIC is attached to a bus on a numa socket, it tries to > allocate rte memory in that socket. > If hugepages are not configured/available on that rte socket > mlx5_common_pci_probe fails with ENOMEM. > > In this patch, a memflag MLX5_MEM_FALLBACK_ANY_SOCKET is introduced > which when set on mlx5_malloc, will allocate the memory using > SOCKET_ID_ANY if it is not able to allocate memory on the specified socke= t. > This allocates memory on any socket starting with the current thread's > socket. > > Signed-off-by: Hari Sasank > > --- > drivers/common/mlx5/mlx5_common_devx.c | 9 ++++++--- > drivers/common/mlx5/mlx5_common_mr.c | 5 +++-- > drivers/common/mlx5/mlx5_malloc.c | 7 +++++++ > drivers/common/mlx5/mlx5_malloc.h | 4 ++++ > drivers/net/mlx5/mlx5.c | 3 ++- > drivers/net/mlx5/mlx5_devx.c | 3 ++- > drivers/net/mlx5/mlx5_rxq.c | 3 ++- > drivers/net/mlx5/mlx5_trigger.c | 6 ++++-- > drivers/net/mlx5/mlx5_txpp.c | 3 ++- > drivers/net/mlx5/mlx5_txq.c | 3 ++- > 10 files changed, 34 insertions(+), 12 deletions(-) > > diff --git a/drivers/common/mlx5/mlx5_common_devx.c > b/drivers/common/mlx5/mlx5_common_devx.c > index 431d8361cebd..122f1c65eab6 100644 > --- a/drivers/common/mlx5/mlx5_common_devx.c > +++ b/drivers/common/mlx5/mlx5_common_devx.c > @@ -107,7 +107,8 @@ mlx5_devx_cq_create(void *ctx, struct > mlx5_devx_cq *cq_obj, uint16_t log_desc_n, > umem_size =3D sizeof(struct mlx5_cqe) * num_of_cqes; > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, socket); > if (!umem_buf) { > DRV_LOG(ERR, "Failed to allocate memory for CQ."); @@ - > 225,7 +226,8 @@ mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq > *sq_obj, uint16_t log_wqbb_n, > umem_size =3D MLX5_WQE_SIZE * num_of_wqbbs; > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, socket); > if (!umem_buf) { > DRV_LOG(ERR, "Failed to allocate memory for SQ."); @@ - > 476,7 +478,8 @@ mlx5_devx_wq_init(void *ctx, uint32_t wqe_size, uint16_t > log_wqbb_n, int socket, > umem_size =3D wqe_size * (1 << log_wqbb_n); > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, socket); > if (!umem_buf) { > DRV_LOG(ERR, "Failed to allocate memory for RQ."); diff -- > git a/drivers/common/mlx5/mlx5_common_mr.c > b/drivers/common/mlx5/mlx5_common_mr.c > index 7b14b0c7bf1e..b2ad6a249732 100644 > --- a/drivers/common/mlx5/mlx5_common_mr.c > +++ b/drivers/common/mlx5/mlx5_common_mr.c > @@ -223,7 +223,8 @@ mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, > int socket) > } > MLX5_ASSERT(!bt->table && !bt->size); > memset(bt, 0, sizeof(*bt)); > - bt->table =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + bt->table =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > sizeof(struct mr_cache_entry) * n, > 0, socket); > if (bt->table =3D=3D NULL) { > @@ -767,7 +768,7 @@ mlx5_mr_create_primary(void *pd, > (void *)addr, data.start, data.end, msl->page_sz, ms_n); > /* Size of memory for bitmap. */ > bmp_size =3D rte_bitmap_get_memory_footprint(ms_n); > - mr =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + mr =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > +MLX5_MEM_FALLBACK_ANY_SOCKET, > RTE_ALIGN_CEIL(sizeof(*mr), > RTE_CACHE_LINE_SIZE) + > bmp_size, RTE_CACHE_LINE_SIZE, msl->socket_id); > if (mr =3D=3D NULL) { > diff --git a/drivers/common/mlx5/mlx5_malloc.c > b/drivers/common/mlx5/mlx5_malloc.c > index c58c41da9266..e109f1bfa994 100644 > --- a/drivers/common/mlx5/mlx5_malloc.c > +++ b/drivers/common/mlx5/mlx5_malloc.c > @@ -182,6 +182,13 @@ mlx5_malloc(uint32_t flags, size_t size, unsigned in= t > align, int socket) > addr =3D rte_zmalloc_socket(NULL, size, align, soc= ket); > else > addr =3D rte_malloc_socket(NULL, size, align, sock= et); > + if (!addr && socket !=3D SOCKET_ID_ANY && > + (flags & MLX5_MEM_FALLBACK_ANY_SOCKET)) { > + if (flags & MLX5_MEM_ZERO) > + addr =3D rte_zmalloc_socket(NULL, size, ali= gn, > SOCKET_ID_ANY); > + else > + addr =3D rte_malloc_socket(NULL, size, alig= n, > SOCKET_ID_ANY); > + } > mlx5_mem_update_msl(addr); > #ifdef RTE_LIBRTE_MLX5_DEBUG > if (addr) > diff --git a/drivers/common/mlx5/mlx5_malloc.h > b/drivers/common/mlx5/mlx5_malloc.h > index 9086a4f3f22e..cd57f95a629e 100644 > --- a/drivers/common/mlx5/mlx5_malloc.h > +++ b/drivers/common/mlx5/mlx5_malloc.h > @@ -28,6 +28,10 @@ enum mlx5_mem_flags { > /* Memory should be allocated from rte hugepage. */ > MLX5_MEM_ZERO =3D 1 << 2, > /* Memory should be cleared to zero. */ > + MLX5_MEM_FALLBACK_ANY_SOCKET =3D 1 << 3, > + /* Memory can be allocated on any socket if > + * it fails to allocate on the given socket. > + */ > }; > > /** > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > f9aea1318736..5b520d468299 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -2063,7 +2063,8 @@ mlx5_proc_priv_init(struct rte_eth_dev *dev) > */ > ppriv_size =3D sizeof(struct mlx5_proc_priv) + > priv->txqs_n * sizeof(struct mlx5_uar_data); > - ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > ppriv_size, > + ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, ppriv_size, > RTE_CACHE_LINE_SIZE, dev->device- > >numa_node); > if (!ppriv) { > rte_errno =3D ENOMEM; > diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c > index 4369d2557e9e..47a925e5913a 100644 > --- a/drivers/net/mlx5/mlx5_devx.c > +++ b/drivers/net/mlx5/mlx5_devx.c > @@ -1285,7 +1285,8 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev > *dev, uint16_t idx) > > RTE_BIT32(host_mem_attr.wq_attr.log_hairpin_num_packets); > umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE); > umem_size +=3D MLX5_DBR_SIZE; > - umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | > MLX5_MEM_ZERO, umem_size, > + umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | > MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > umem_size, > alignment, priv->sh->numa_node); > if (umem_buf =3D=3D NULL && txq_ctrl- > >hairpin_conf.force_memory) { > DRV_LOG(ERR, "Failed to allocate memory for > hairpin TX queue"); diff --git a/drivers/net/mlx5/mlx5_rxq.c > b/drivers/net/mlx5/mlx5_rxq.c index ad8fd13cbe8e..6bdf1678e499 100644 > --- a/drivers/net/mlx5/mlx5_rxq.c > +++ b/drivers/net/mlx5/mlx5_rxq.c > @@ -1728,7 +1728,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc, > desc >>=3D mprq_log_actual_stride_num; > alloc_size +=3D desc * sizeof(struct mlx5_mprq_buf *); > } > - tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > alloc_size, 0, socket); > + tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, alloc_size, > 0, socket); > if (!tmpl) { > rte_errno =3D ENOMEM; > return NULL; > diff --git a/drivers/net/mlx5/mlx5_trigger.c > b/drivers/net/mlx5/mlx5_trigger.c index bbaa7d2aa021..1a6abdf2b61d > 100644 > --- a/drivers/net/mlx5/mlx5_trigger.c > +++ b/drivers/net/mlx5/mlx5_trigger.c > @@ -55,7 +55,8 @@ mlx5_txq_start(struct rte_eth_dev *dev) > for (i =3D 0; i !=3D priv->txqs_n; ++i) { > struct mlx5_txq_ctrl *txq_ctrl =3D mlx5_txq_get(dev, i); > struct mlx5_txq_data *txq_data =3D &txq_ctrl->txq; > - uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO; > + uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET; > > if (!txq_ctrl) > continue; > @@ -180,7 +181,8 @@ mlx5_rxq_ctrl_prepare(struct rte_eth_dev *dev, > struct mlx5_rxq_ctrl *rxq_ctrl, > return ret; > } > MLX5_ASSERT(!rxq_ctrl->obj); > - rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > sizeof(*rxq_ctrl->obj), 0, > rxq_ctrl->socket); > if (!rxq_ctrl->obj) { > diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c > index 5a5df2d1bb16..c81ae7bfd328 100644 > --- a/drivers/net/mlx5/mlx5_txpp.c > +++ b/drivers/net/mlx5/mlx5_txpp.c > @@ -394,7 +394,8 @@ mlx5_txpp_create_clock_queue(struct > mlx5_dev_ctx_shared *sh) > struct mlx5_txpp_wq *wq =3D &sh->txpp.clock_queue; > int ret; > > - sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > + sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > MLX5_TXPP_REARM_SQ_SIZE * > sizeof(struct mlx5_txpp_ts), > 0, sh->numa_node); > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c in= dex > 8cb52b0f7d8e..b83e798544d2 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -1074,7 +1074,8 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc, > struct mlx5_priv *priv =3D dev->data->dev_private; > struct mlx5_txq_ctrl *tmpl; > > - tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, > sizeof(*tmpl) + > + tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO | > + MLX5_MEM_FALLBACK_ANY_SOCKET, > sizeof(*tmpl) + > desc * sizeof(struct rte_mbuf *), 0, socket); > if (!tmpl) { > rte_errno =3D ENOMEM; > -- > 2.39.2 --_000_DM6PR12MB3753FCEB708F27B0DAEFD5BFDFA1ADM6PR12MB3753namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi, Hari

 

  1. &= gt;> We have a DPDK application running with all the CPU logical cores b= elonging to a single socket on a multi-socket system.
  2. &= gt;> But the NIC itself is on a bus that is attached to the CPU from a d= ifferent socket
  3. >> socket where this NIC card is presen= t because that is one of the requirements

 

This is quite not typical use case –= ; running  the driver not close to the HW. All transactions between th= e NIC and remote (not on the same socket)

memory will cause memory subsystem overhe= ad – QPI (or other inter-socket bus) will be highly loaded, MESI prot= ocols will be engaged, and so on.

I believe you have strong reasons to choo= se this configuration, but this is certainly the mlx5 driver was not design= ed for.

I would prefer to have the way to detect = this socket misconfiguration for more common scenarios (now mlx5 just fails= ).
What about introducing the some devargs “allow_any_socket”, &nb= sp;allowing “wrong socket” allocation?

 

With best regards,
Slava

 

From: Hari Sasank <harisasank@outlook.com&= gt;
Sent: Monday, October 30, 2023 1:40 PM
To: Slava Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad <= matan@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou <su= anmingm@nvidia.com>
Cc: dev@dpdk.org
Subject: Re: [PATCH] net/mlx5: support bus socket with no hugepages<= o:p>

 

Hi Slava

 

We have a DPDK application running with a= ll the CPU logical cores belonging to a single socket on a multi-socket sys= tem. But the NIC itself is on a bus that is attached to the CPU from a different socket. 

In this case, the driver is allocating hu= ge pages belonging to a socket that is different from the CPU.  But, s= ince huge pages are not available on the socket of the NIC card, it crashes the application.

(We starte= d this dpdk application saying not to use memory from the socket where this= NIC card is present because that is one of the requirements. We also probably can't move this NIC from the PCIe bus from = this socket as all the buses from the other socket are in use.).

 

I understa= nd the performance implications of this, so we use this = flag to permit the mlx5_malloc to allocate its huge pages from any socket (SOCKET_ID_ANY seems = to try starting with the current cpu lcore socket) only if huge pages= are not available on the NIC socket that the driver is originally called with.

 

I would be happy to di= scover if there is a better way to solve this. 

 

Thanks
Hari

From: Slava Ovsiienko <viacheslavo@nvidia.com>
Sent: Monday, October 30, 2023 11:43 AM
To: Hari Sasank <harisa= sank@outlook.com>; Matan Azrad <matan@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou <suanmingm@nvidia.com>
Cc: dev@dpdk.org <dev@dpdk.org>
Subject: RE: [PATCH] net/mlx5: support bus socket with no hugepages<= /span>

 

Hi, Hari

As I see almost all updates, using newly introduced MLX5_MEM_FALLBACK_ANY_S= OCKET flag,
are related to the memory mapped for the hardware usage (NIC accesses these= areas with DMA
over the PCIe bus segment NIC attached to). It means the memory allegiance = to the specific
is strong and might be critical for the performance.

In general, mlx5 PMD is designed in way requesting the memory on the specif= ic socket only
If, and only if it is really needed.

Generally speaking, missing huge pages on the socket to which the NIC is at= tached to
should be considered as misconfiguration.  Could you, please, elaborat= e a little bit more, what is a use case
for this scenario?

With best regards,
Slava

> -----Original Message-----
> From: Hari Sasank <harisa= sank@outlook.com>
> Sent: Wednesday, June 21, 2023 10:31 AM
> To: Matan Azrad <matan@nvidia.c= om>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com&g= t;
> Cc: dev@dpdk.org; Hari Sasank <= harisasank@outlook.com> > Subject: [PATCH] net/mlx5: support bus socket with no hugepages
>
> When a Mellanox NIC is attached to a bus on a numa socket, it tries to=
> allocate rte memory in that socket.
> If hugepages are not configured/available on that rte socket
> mlx5_common_pci_probe fails with ENOMEM.
>
> In this patch, a memflag MLX5_MEM_FALLBACK_ANY_SOCKET is introduced > which when set on mlx5_malloc, will allocate the memory using
> SOCKET_ID_ANY if it is not able to allocate memory on the specified so= cket.
> This allocates memory on any socket starting with the current thread's=
> socket.
>
> Signed-off-by: Hari Sasank <harisasank@outlook.com>
> ---
>  drivers/common/mlx5/mlx5_common_devx.c | 9 ++++++---
>  drivers/common/mlx5/mlx5_common_mr.c   | 5 +++--
>  drivers/common/mlx5/mlx5_malloc.c      = | 7 +++++++
>  drivers/common/mlx5/mlx5_malloc.h      = | 4 ++++
>  drivers/net/mlx5/mlx5.c       = ;         | 3 ++-
>  drivers/net/mlx5/mlx5_devx.c      =      | 3 ++-
>  drivers/net/mlx5/mlx5_rxq.c      &= nbsp;     | 3 ++-
>  drivers/net/mlx5/mlx5_trigger.c     &nb= sp;  | 6 ++++--
>  drivers/net/mlx5/mlx5_txpp.c      =      | 3 ++-
>  drivers/net/mlx5/mlx5_txq.c      &= nbsp;     | 3 ++-
>  10 files changed, 34 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/common/mlx5/mlx5_common_devx.c
> b/drivers/common/mlx5/mlx5_common_devx.c
> index 431d8361cebd..122f1c65eab6 100644
> --- a/drivers/common/mlx5/mlx5_common_devx.c
> +++ b/drivers/common/mlx5/mlx5_common_devx.c
> @@ -107,7 +107,8 @@ mlx5_devx_cq_create(void *ctx, struct
> mlx5_devx_cq *cq_obj, uint16_t log_desc_n,
>        umem_size =3D sizeof(struct = mlx5_cqe) * num_of_cqes;
>        umem_dbrec =3D RTE_ALIGN(ume= m_size, MLX5_DBR_SIZE);
>        umem_size +=3D MLX5_DBR_SIZE= ;
> -     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO,
> umem_size,
> +     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO |
> +           &nb= sp;            =     MLX5_MEM_FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;      alignment, socket);
>        if (!umem_buf) {
>            = ;    DRV_LOG(ERR, "Failed to allocate memory for CQ.&qu= ot;); @@ -
> 225,7 +226,8 @@ mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq
> *sq_obj, uint16_t log_wqbb_n,
>        umem_size =3D MLX5_WQE_SIZE = * num_of_wqbbs;
>        umem_dbrec =3D RTE_ALIGN(ume= m_size, MLX5_DBR_SIZE);
>        umem_size +=3D MLX5_DBR_SIZE= ;
> -     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO,
> umem_size,
> +     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO |
> +           &nb= sp;            =     MLX5_MEM_FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;      alignment, socket);
>        if (!umem_buf) {
>            = ;    DRV_LOG(ERR, "Failed to allocate memory for SQ.&qu= ot;); @@ -
> 476,7 +478,8 @@ mlx5_devx_wq_init(void *ctx, uint32_t wqe_size, uint16= _t
> log_wqbb_n, int socket,
>        umem_size =3D wqe_size * (1 = << log_wqbb_n);
>        umem_dbrec =3D RTE_ALIGN(ume= m_size, MLX5_DBR_SIZE);
>        umem_size +=3D MLX5_DBR_SIZE= ;
> -     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO,
> umem_size,
> +     umem_buf =3D mlx5_malloc(MLX5_MEM_RTE | MLX5= _MEM_ZERO |
> +           &nb= sp;            =     MLX5_MEM_FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;      alignment, socket);
>        if (!umem_buf) {
>            = ;    DRV_LOG(ERR, "Failed to allocate memory for RQ.&qu= ot;); diff --
> git a/drivers/common/mlx5/mlx5_common_mr.c
> b/drivers/common/mlx5/mlx5_common_mr.c
> index 7b14b0c7bf1e..b2ad6a249732 100644
> --- a/drivers/common/mlx5/mlx5_common_mr.c
> +++ b/drivers/common/mlx5/mlx5_common_mr.c
> @@ -223,7 +223,8 @@ mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n= ,
> int socket)
>        }
>        MLX5_ASSERT(!bt->table &a= mp;& !bt->size);
>        memset(bt, 0, sizeof(*bt));<= br> > -     bt->table =3D mlx5_malloc(MLX5_MEM_RTE | = MLX5_MEM_ZERO,
> +     bt->table =3D mlx5_malloc(MLX5_MEM_RTE | = MLX5_MEM_ZERO |
> +           &nb= sp;            =      MLX5_MEM_FALLBACK_ANY_SOCKET,
>            = ;            &n= bsp;       sizeof(struct mr_cache_entry) * n,=
>            = ;            &n= bsp;       0, socket);
>        if (bt->table =3D=3D NULL= ) {
> @@ -767,7 +768,7 @@ mlx5_mr_create_primary(void *pd,
>            = ;  (void *)addr, data.start, data.end, msl->page_sz, ms_n);
>        /* Size of memory for bitmap= . */
>        bmp_size =3D rte_bitmap_get_= memory_footprint(ms_n);
> -     mr =3D mlx5_malloc(MLX5_MEM_RTE |  MLX5= _MEM_ZERO,
> +     mr =3D mlx5_malloc(MLX5_MEM_RTE |  MLX5= _MEM_ZERO |
> +MLX5_MEM_FALLBACK_ANY_SOCKET,
>            = ;             R= TE_ALIGN_CEIL(sizeof(*mr),
> RTE_CACHE_LINE_SIZE) +
>            = ;             b= mp_size, RTE_CACHE_LINE_SIZE, msl->socket_id);
>        if (mr =3D=3D NULL) {
> diff --git a/drivers/common/mlx5/mlx5_malloc.c
> b/drivers/common/mlx5/mlx5_malloc.c
> index c58c41da9266..e109f1bfa994 100644
> --- a/drivers/common/mlx5/mlx5_malloc.c
> +++ b/drivers/common/mlx5/mlx5_malloc.c
> @@ -182,6 +182,13 @@ mlx5_malloc(uint32_t flags, size_t size, unsigned= int
> align, int socket)
>            = ;            addr = =3D rte_zmalloc_socket(NULL, size, align, socket);
>            = ;    else
>            = ;            addr = =3D rte_malloc_socket(NULL, size, align, socket);
> +           &nb= sp; if (!addr && socket !=3D SOCKET_ID_ANY &&
> +           &nb= sp;     (flags & MLX5_MEM_FALLBACK_ANY_SOCKET)) { > +           &nb= sp;         if (flags & MLX5_ME= M_ZERO)
> +           &nb= sp;            =      addr =3D rte_zmalloc_socket(NULL, size, align,
> SOCKET_ID_ANY);
> +           &nb= sp;         else
> +           &nb= sp;            =      addr =3D rte_malloc_socket(NULL, size, align,
> SOCKET_ID_ANY);
> +           &nb= sp; }
>            = ;    mlx5_mem_update_msl(addr);
>  #ifdef RTE_LIBRTE_MLX5_DEBUG
>            = ;    if (addr)
> diff --git a/drivers/common/mlx5/mlx5_malloc.h
> b/drivers/common/mlx5/mlx5_malloc.h
> index 9086a4f3f22e..cd57f95a629e 100644
> --- a/drivers/common/mlx5/mlx5_malloc.h
> +++ b/drivers/common/mlx5/mlx5_malloc.h
> @@ -28,6 +28,10 @@ enum mlx5_mem_flags {
>        /* Memory should be allocate= d from rte hugepage. */
>        MLX5_MEM_ZERO =3D 1 <<= 2,
>        /* Memory should be cleared = to zero. */
> +     MLX5_MEM_FALLBACK_ANY_SOCKET =3D 1 << = 3,
> +     /* Memory can be allocated on any socket if<= br> > +      * it fails to allocate on the given so= cket.
> +      */
>  };
>
>  /**
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > f9aea1318736..5b520d468299 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -2063,7 +2063,8 @@ mlx5_proc_priv_init(struct rte_eth_dev *dev)
>         */
>        ppriv_size =3D sizeof(struct= mlx5_proc_priv) +
>            = ;         priv->txqs_n * sizeof(= struct mlx5_uar_data);
> -     ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_ME= M_ZERO,
> ppriv_size,
> +     ppriv =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_ME= M_ZERO |
> +           &nb= sp;            = MLX5_MEM_FALLBACK_ANY_SOCKET, ppriv_size,
>            = ;            &n= bsp;   RTE_CACHE_LINE_SIZE, dev->device-
> >numa_node);
>        if (!ppriv) {
>            = ;    rte_errno =3D ENOMEM;
> diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx= .c
> index 4369d2557e9e..47a925e5913a 100644
> --- a/drivers/net/mlx5/mlx5_devx.c
> +++ b/drivers/net/mlx5/mlx5_devx.c
> @@ -1285,7 +1285,8 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev
> *dev, uint16_t idx)
>
>        RTE_BIT32(host_mem_attr.wq_a= ttr.log_hairpin_num_packets);
>            = ;    umem_dbrec =3D RTE_ALIGN(umem_size, MLX5_DBR_SIZE);
>            = ;    umem_size +=3D MLX5_DBR_SIZE;
> -           &nb= sp; umem_buf =3D mlx5_malloc(MLX5_MEM_RTE |
> MLX5_MEM_ZERO, umem_size,
> +           &nb= sp; umem_buf =3D mlx5_malloc(MLX5_MEM_RTE |
> MLX5_MEM_ZERO |
> +           &nb= sp;            =             MLX5_MEM= _FALLBACK_ANY_SOCKET,
> umem_size,
>            = ;            &n= bsp;            = ;  alignment, priv->sh->numa_node);
>            = ;    if (umem_buf =3D=3D NULL && txq_ctrl-
> >hairpin_conf.force_memory) {
>            = ;            DRV_LOG= (ERR, "Failed to allocate memory for
> hairpin TX queue"); diff --git a/drivers/net/mlx5/mlx5_rxq.c
> b/drivers/net/mlx5/mlx5_rxq.c index ad8fd13cbe8e..6bdf1678e499 100644<= br> > --- a/drivers/net/mlx5/mlx5_rxq.c
> +++ b/drivers/net/mlx5/mlx5_rxq.c
> @@ -1728,7 +1728,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc,
>            = ;    desc >>=3D mprq_log_actual_stride_num;
>            = ;    alloc_size +=3D desc * sizeof(struct mlx5_mprq_buf *);<= br> >        }
> -     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO,
> alloc_size, 0, socket);
> +     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO |
> +           &nb= sp;            MLX5_= MEM_FALLBACK_ANY_SOCKET, alloc_size,
> 0, socket);
>        if (!tmpl) {
>            = ;    rte_errno =3D ENOMEM;
>            = ;    return NULL;
> diff --git a/drivers/net/mlx5/mlx5_trigger.c
> b/drivers/net/mlx5/mlx5_trigger.c index bbaa7d2aa021..1a6abdf2b61d
> 100644
> --- a/drivers/net/mlx5/mlx5_trigger.c
> +++ b/drivers/net/mlx5/mlx5_trigger.c
> @@ -55,7 +55,8 @@ mlx5_txq_start(struct rte_eth_dev *dev)
>        for (i =3D 0; i !=3D priv-&g= t;txqs_n; ++i) {
>            = ;    struct mlx5_txq_ctrl *txq_ctrl =3D mlx5_txq_get(dev, i)= ;
>            = ;    struct mlx5_txq_data *txq_data =3D &txq_ctrl->tx= q;
> -           &nb= sp; uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO;
> +           &nb= sp; uint32_t flags =3D MLX5_MEM_RTE | MLX5_MEM_ZERO |
> +           &nb= sp;            =       MLX5_MEM_FALLBACK_ANY_SOCKET;
>
>            = ;    if (!txq_ctrl)
>            = ;            continu= e;
> @@ -180,7 +181,8 @@ mlx5_rxq_ctrl_prepare(struct rte_eth_dev *dev,
> struct mlx5_rxq_ctrl *rxq_ctrl,
>            = ;            return = ret;
>        }
>        MLX5_ASSERT(!rxq_ctrl->ob= j);
> -     rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RT= E | MLX5_MEM_ZERO,
> +     rxq_ctrl->obj =3D mlx5_malloc(MLX5_MEM_RT= E | MLX5_MEM_ZERO |
> +           &nb= sp;            =          MLX5_MEM_FALLBACK_ANY_SOCK= ET,
>            = ;            &n= bsp;           sizeof(*rx= q_ctrl->obj), 0,
>            = ;            &n= bsp;           rxq_ctrl-&= gt;socket);
>        if (!rxq_ctrl->obj) {
> diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp= .c
> index 5a5df2d1bb16..c81ae7bfd328 100644
> --- a/drivers/net/mlx5/mlx5_txpp.c
> +++ b/drivers/net/mlx5/mlx5_txpp.c
> @@ -394,7 +394,8 @@ mlx5_txpp_create_clock_queue(struct
> mlx5_dev_ctx_shared *sh)
>        struct mlx5_txpp_wq *wq =3D = &sh->txpp.clock_queue;
>        int ret;
>
> -     sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE= | MLX5_MEM_ZERO,
> +     sh->txpp.tsa =3D mlx5_malloc(MLX5_MEM_RTE= | MLX5_MEM_ZERO |
> +           &nb= sp;            =         MLX5_MEM_FALLBACK_ANY_SOCKET, >            = ;            &n= bsp;          MLX5_TXPP_REARM_= SQ_SIZE *
>            = ;            &n= bsp;          sizeof(struct ml= x5_txpp_ts),
>            = ;            &n= bsp;          0, sh->numa_n= ode);
> diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c= index
> 8cb52b0f7d8e..b83e798544d2 100644
> --- a/drivers/net/mlx5/mlx5_txq.c
> +++ b/drivers/net/mlx5/mlx5_txq.c
> @@ -1074,7 +1074,8 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t > idx, uint16_t desc,
>        struct mlx5_priv *priv =3D d= ev->data->dev_private;
>        struct mlx5_txq_ctrl *tmpl;<= br> >
> -     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO,
> sizeof(*tmpl) +
> +     tmpl =3D mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM= _ZERO |
> +           &nb= sp;            MLX5_= MEM_FALLBACK_ANY_SOCKET,
> sizeof(*tmpl) +
>            = ;            &n= bsp;  desc * sizeof(struct rte_mbuf *), 0, socket);
>        if (!tmpl) {
>            = ;    rte_errno =3D ENOMEM;
> --
> 2.39.2

--_000_DM6PR12MB3753FCEB708F27B0DAEFD5BFDFA1ADM6PR12MB3753namp_--