From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 738244650E for ; Sat, 5 Apr 2025 09:44:19 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4AB46400EF; Sat, 5 Apr 2025 09:44:19 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2058.outbound.protection.outlook.com [40.107.96.58]) by mails.dpdk.org (Postfix) with ESMTP id 646BF400EF for ; Sat, 5 Apr 2025 09:44:18 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PnGjQ1x5MdQGsyVy0qhPMGfjSGGdDq6dDmabP+yaY82sN3dTryQzV/He1aJ4DtdzCNPRY9tljpV40tSQT4mfVYiGqSMPBgqz30vIxDlBzcM7nwd5gHgxgWz/tcWlY5hUJXnLNYv83UuiZyYUp890OedZicRd0/QbMY8W7f22x4Elkfz/A02zHL/9fpnN2m6HV+uOIqdos5txG7Stir5ohbFvGreei9YFzMRZxtHkWn92bDcRC0IGsPltBFfWwheGeuuNbrl6OlSMYWjaRgPjf+1KUtx4DwqKUyNh6ne1i51/frd5a+6jLpg5cMDM+iVPnLuPi+Ery/J6GtJ9R865cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fxz4LL0QkGTP6XYqPRJZP58G95fustgMD1sWqgVToLk=; b=egJ3nPrIu+uscZ46qIE68fnSFDpNmPPucU7a9tg3B7T+Vd6sJqD3Fd8MU+s6aLnejV3MhE8ncE5iL+7HaRx9A+D7d5bUJwcb9Zg0iLhje9oE58iudK3OEEn7zP0uZmhknvP6RF2digrvrRfrOHtrGfixJHWvL0IgwG9Fz9DxBT/WrT0ecf21btejcimlQiJV7PwZjo2GLYAY2wen9SMFEzZnYsFrAD+lcHs7uEXvexZE6DT7dsxCVTPQW9w04ezAhuJX+1+zkrgyUfr6naDNdsXma4drU+R0pFpCJcFxKxDn/9dK8/5vvaYIdcKmPPx3z6TqtZptW8uhJZbz0a13hQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fxz4LL0QkGTP6XYqPRJZP58G95fustgMD1sWqgVToLk=; b=O2a0kKWnetTv0OX3wjweAlEUU/U5bdD0t7WbgO7YCr/pP8iz1LLLAh2mS/sUfHOOw0s6tZh/P1P7PLqhs8eP7uc6HtptJxkTZ3rk2AhAsJvy6G9qubvStA8xdSw8oCUdis2VGHF75M1bYBJCb+44oHR+Yh9LgrJcFTebETClhQ1CWy2go6aSga89oN2NjgvJJc5sCEC6TYk83ZUST2zUHe/n22ChBAXHQ5f61IK5iqreCcjU7iQKmOagTE05ijfonE4yUAuoEg4wQ+QiGfZmwuOYRmTNGX7JL0L4wb7ZJLUuJUo8Ol5L/QNMlqheCvahCPwkd04ub9oL1skjk37fQg== Received: from CH3PR12MB8658.namprd12.prod.outlook.com (2603:10b6:610:175::8) by CY8PR12MB9036.namprd12.prod.outlook.com (2603:10b6:930:78::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.44; Sat, 5 Apr 2025 07:44:14 +0000 Received: from CH3PR12MB8658.namprd12.prod.outlook.com ([fe80::d5cc:cc84:5e00:2f42]) by CH3PR12MB8658.namprd12.prod.outlook.com ([fe80::d5cc:cc84:5e00:2f42%4]) with mapi id 15.20.8534.048; Sat, 5 Apr 2025 07:44:14 +0000 From: Xueming Li To: Joshua Washington , "stable@dpdk.org" , Junfeng Guo , Jeroen de Borst , Rushil Gupta , Xiaoyun Li CC: Praveen Kaligineedi Subject: Re: [PATCH 23.11] net/gve: allocate Rx QPL pages using malloc Thread-Topic: [PATCH 23.11] net/gve: allocate Rx QPL pages using malloc Thread-Index: AQHbjJDibsKz/UFDtEyuKLkgiW/sRLOU40vE Date: Sat, 5 Apr 2025 07:44:13 +0000 Message-ID: References: <20250303230608.2228640-1-joshwash@google.com> In-Reply-To: <20250303230608.2228640-1-joshwash@google.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR12MB8658:EE_|CY8PR12MB9036:EE_ x-ms-office365-filtering-correlation-id: 4427d641-f653-43bd-fbf6-08dd7415a942 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018|7053199007|8096899003; x-microsoft-antispam-message-info: =?us-ascii?Q?nCMEu85CzwNMslczfHlUGTaZTn+Nb2jPdmul+HwXHe8NKv4IGlTuh6uXydgs?= =?us-ascii?Q?fKS/+iSkH1xgL7Jb1SxpTtr51XN9WIXgH6r3BP75OvQsGZt7PgVnHMmrMvsF?= =?us-ascii?Q?ynVnIif6AZQAKYhA8VZyfBJTpdGuPUErCKPd79enVUzOFnzeh5YiiIkcEHru?= =?us-ascii?Q?XIFrdoX345aIzqFEQvIWh74YiI9wT9JhTOQxcM/4c2VzTmXoNMUM3503Apsd?= =?us-ascii?Q?xmEAVRSeR0GerAS90egcq7u8njhdbGy+tF5ora7Q2TqUZtSom81Bds9WTR1d?= =?us-ascii?Q?Rp6hwEX7M0RkcDmvg842bXFGqgbMzsyjJ2MMU2ClzA6cW77k+eqEUlPekOv4?= =?us-ascii?Q?dKdnComNnbYi/yJeKdMHnq6VR1MZlZsfKFTNHKqfPvBpK3CBr6IXouePHrYT?= =?us-ascii?Q?+r6nkMwtMmI6RuDH3HlCpiROZzgd+NWQy21lh4D96SqnRW1EeoISZWyWNgJM?= =?us-ascii?Q?j7lLHM1FCBYKZTjBU901F2egtmdKAOtMRqOdlDbLivyv4mz7OfaykC9QtttN?= =?us-ascii?Q?q4h5o7st9dYVIQgOkHtHpFTrl3GoPzpo8b/hPmfFI4xqpJSdn6N89QXdV9Xi?= =?us-ascii?Q?ntZBuAFha+lsx3mGkv9iLHtA568YRcNZW9buXfZM27rDeELt9ObDrUlwmkoA?= =?us-ascii?Q?fFtpFdr5MymYnaD2UmwQSI0wfyZwG6tB464bOJXw2zn7fBHlK48i/ZN2d/Nj?= =?us-ascii?Q?ZjbzG7aqOcc9TKJNZquv/jfYLHGzFqvNukwL8OepbtyMEp4OkNtd93Jb8JbV?= =?us-ascii?Q?EZGYl2MGK14lcPoDLC1+FS4LjEvyTJh5BC88AsbsNP61cRNQtDtytIlBAaU3?= =?us-ascii?Q?DdHooTx5oNUN9IZlj78oY37zA0r9J83o3pPqWSAPEUuMrlkOtJPiguaHXH+M?= =?us-ascii?Q?2p1kVLCCRWw0yozdG8zVlE0gImJoCfE5cYJg+jnFeB10BA6CMgW20PjpnQ3V?= =?us-ascii?Q?rZ5kUz3sNgopT2B77SJD+mOrIAa0+78geB8ebhSp1iKkXfksg5oLA/dVmd2P?= =?us-ascii?Q?Lj69X1JPBgWdwo7J/iXVm/Iqxs3RoEx3A4VZ6gyg3oBDfP7KRuj9e2RY0/Ic?= =?us-ascii?Q?KKRrlTTiI97BbU/wvy/3l9uNXarRJ+7/6KD86jWKQTw/2WBYAUHLpvv3oG7T?= =?us-ascii?Q?Ai12B3PhaJ0BBlDdcp9k0aJn1qWNPy2fVabwhOOnRUJimNWWfJtvYMx6QhbS?= =?us-ascii?Q?oDOcgwS12DQQ3yswjBfmlGVbsKZiu6Wbb2YjWHuamSS4e7Vw5v3u0JYm/HGD?= =?us-ascii?Q?Qo2jZxs4Fc+POTZipoK8V7rBL3+gnTYhl4iVBRTvhjieGUQSVgF+/Tii0Qy/?= =?us-ascii?Q?IJvLQFkN1KR80iYt1LT1zmG7yLqFjDM9PtAneyMe30Z9kmV7X6PY2HWYiTWb?= =?us-ascii?Q?iN8V5hBXcg8mgrffoo4iepSEL/EhlSvwYrO4aM+pPQirhV2KohV45PPBy15b?= =?us-ascii?Q?OF3tB6x11bBkcjQfHvxFhqdEa00F2R8m?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR12MB8658.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(38070700018)(7053199007)(8096899003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Hk8/MpoKRF4oi+vNObal6Lzni/fdBMRxJFF2BY+uqA+5LoShcONSlz5DHhZL?= =?us-ascii?Q?lfKny0TKZktZqJ6r+hoA33VbqbsiQDzVsC+oT+KB2AIrIVlwwSjHStmDai0I?= =?us-ascii?Q?lLWZlAPd9Tbw7CTGc1hFCr+n1QHo0zCwZ1PwiLzyMY8k8tmeFOXmVYv6y6Eh?= =?us-ascii?Q?WzvGeKx/KxoQnyg3vLysLwJxhQjaZbTOWm9ZMpykw69xbS+y/8fUID26OoMQ?= =?us-ascii?Q?dK+q5bbfGit+8MCJA3qpkfZ86S190fP3NlkEEOqiU7kyUCDqwS1dM6qstCO0?= =?us-ascii?Q?A1xjLLAM1QcHLr+j0M4/zpaBdrvY27LNlJYj244UbabtQl7rxyDSRUz9ulao?= =?us-ascii?Q?wy0mB4Iki3SsY6wq8zmzISLIT4sDSxZWDHT0nGFahocBuIJ4dKYW9WuNenv9?= =?us-ascii?Q?Ybgo+d2x4mslbrdRJYHWk0zpkpFf55Py+sGjLW9PJ+5cpzOa9smK16bXbbm2?= =?us-ascii?Q?VaeH0Se513osLtorO/WH1C75sQbSBwMAGNcaS1e/Rer3aoQBZqb2XShxqci0?= =?us-ascii?Q?tOZvNf/SptfCa7i/YW6sgpN31k89EXnGDcnSwQCz9PQPA27NWaK37bImMleC?= =?us-ascii?Q?KuIapxJ9/jgFhKDMm/vhzocQwpyFib2vUBDo2+ymIeRm+r9xw+x0/34oInRC?= =?us-ascii?Q?U3gi1+8+7FfNnIlbnVndabz2nSM6MmaOYKs/CORQhHkaTmAhzA/D0y/I/GlK?= =?us-ascii?Q?sv0lgYiFg3V4OgYuAkqDUM33lwWUJn09BGjYhvC2QyeNe4ZdVVCqsH9+zpxo?= =?us-ascii?Q?8o+v9gESrRpS9R7YtMFtvkIFZyq6pS3PCxmAdg5zFs16cDLmatVk8FWTR9yx?= =?us-ascii?Q?/LWxBBy9rhZ668ay4yiLHaUfSSFjwyTaHHczvVTjLBI4HkPkUoTbOkz0wZeT?= =?us-ascii?Q?0KF/PGzn1DkB5fHt66XxE2pVAidV3f2rv3eK3Q0xwkzmooBPPKUzGlAE3h7K?= =?us-ascii?Q?oOkvq2WavwjCSsFdhlO9OaDA8oo7QDK/BHzsBj+E6LHBK9X0flgznWsBHIQb?= =?us-ascii?Q?xTmrlHfKqFTm0B3NwceMtaYlzMFrjHHq0pRRHA6KEee5TOcFa+wd7Bt1F29O?= =?us-ascii?Q?Z4biaD/bXiBN9XUCQoLheuRPk+fEQcCZxD7Kj7WLUC6BNpJpHiDen/agV5dL?= =?us-ascii?Q?wAdoeEDLkccfQ5awWBfxu1ZUg5WaE88QFDlS2TcsjaVrZEbEQOoBd3oSj127?= =?us-ascii?Q?AW59F32ilWJs39Tm0F2Di3JepBz8pjlB9MXPYU+gMy2JWy6Qb7bFRfYyZhcq?= =?us-ascii?Q?CoETtEFMFvEwEvdAFpfMIOanyA0cyX7OM0b2sBfP/rHzipw9lkiHLtDG/LRi?= =?us-ascii?Q?3dAA7pqNCkBYAsDpTfkUHrJnAJScYjNLPc6NYjdutKmA4TB6+cetHd7DwVaf?= =?us-ascii?Q?77c+newJfV2SeMNMnUGhpqW3Pz+MGzAY5qJh34329fquZZPfqrjdJmEsS5Xe?= =?us-ascii?Q?fCKxVEsR88Q212lmFYhHdg6nA17+OxxCt5v4J0HnFTwT1GwhUop76iLQgOM7?= =?us-ascii?Q?pBnXaLuswiQYVtpNdF11Qz3WsTNeJUgzQ/QBl9PhiHDhvsglI6kSo70CNXBE?= =?us-ascii?Q?ewSh60TwRdf45lV1IQOJ/I9oy59Ngio4aHTa4G6b?= Content-Type: multipart/alternative; boundary="_000_CH3PR12MB8658A71EF0474EB94F05AE95A1A82CH3PR12MB8658namp_" MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8658.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4427d641-f653-43bd-fbf6-08dd7415a942 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Apr 2025 07:44:13.9058 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ldgvnLEgSZAPaP9cMvEuq9gvqfqa4bo/3MAG0X1YPhLR9OqFzSkvYPeCAmAS1D+ua01eHxEZo/CMePQv8gqlYw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB9036 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org --_000_CH3PR12MB8658A71EF0474EB94F05AE95A1A82CH3PR12MB8658namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks for the your help, patch queued to 23.11.4 LTS release staging. Regards, Xueming ________________________________ From: Joshua Washington Sent: Tuesday, March 4, 2025 7:06 AM To: stable@dpdk.org ; Junfeng Guo ;= Jeroen de Borst ; Rushil Gupta ; = Joshua Washington ; Xiaoyun Li Cc: Praveen Kaligineedi Subject: [PATCH 23.11] net/gve: allocate Rx QPL pages using malloc From: Praveen Kaligineedi Allocating QPL for an RX queue might fail if enough contiguous IOVA memory cannot be allocated. This can commonly occur when using 2MB huge pages because the 1024 4K buffers are allocated for each RX ring by default, resulting in 4MB for each ring. However, the only requirement for RX QPLs is that each 4K buffer be IOVA contiguous, not the entire QPL. Therefore, malloc will be used to allocate RX QPLs instead. Note that TX queues require the entire QPL to be IOVA contiguous, so it will continue to use the memzone-based allocation. Fixes: a46583cf43c8 ("net/gve: support Rx/Tx") Cc: stable@dpdk.org Signed-off-by: Praveen Kaligineedi Signed-off-by: Joshua Washington --- drivers/net/gve/gve_ethdev.c | 139 +++++++++++++++++++++++++++++------ drivers/net/gve/gve_ethdev.h | 5 +- drivers/net/gve/gve_rx.c | 2 +- 3 files changed, 122 insertions(+), 24 deletions(-) diff --git a/drivers/net/gve/gve_ethdev.c b/drivers/net/gve/gve_ethdev.c index ecd37ff37f..d020e0be66 100644 --- a/drivers/net/gve/gve_ethdev.c +++ b/drivers/net/gve/gve_ethdev.c @@ -20,13 +20,45 @@ gve_write_version(uint8_t *driver_version_register) writeb('\n', driver_version_register); } +static const struct rte_memzone * +gve_alloc_using_mz(const char *name, uint32_t num_pages) +{ + const struct rte_memzone *mz; + mz =3D rte_memzone_reserve_aligned(name, num_pages * PAGE_SIZE, + rte_socket_id(), + RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE= ); + if (mz =3D=3D NULL) + PMD_DRV_LOG(ERR, "Failed to alloc memzone %s.", name); + return mz; +} + static int -gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pag= es) +gve_alloc_using_malloc(void **bufs, uint32_t num_entries) +{ + uint32_t i; + + for (i =3D 0; i < num_entries; i++) { + bufs[i] =3D rte_malloc_socket(NULL, PAGE_SIZE, PAGE_SIZE, r= te_socket_id()); + if (bufs[i] =3D=3D NULL) { + PMD_DRV_LOG(ERR, "Failed to malloc"); + goto free_bufs; + } + } + return 0; + +free_bufs: + while (i > 0) + rte_free(bufs[--i]); + + return -ENOMEM; +} + +static int +gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pag= es, + bool is_rx) { - char z_name[RTE_MEMZONE_NAMESIZE]; struct gve_queue_page_list *qpl; - const struct rte_memzone *mz; - dma_addr_t page_bus; + int err =3D 0; uint32_t i; if (priv->num_registered_pages + pages > @@ -37,31 +69,79 @@ gve_alloc_queue_page_list(struct gve_priv *priv, uint32= _t id, uint32_t pages) return -EINVAL; } qpl =3D &priv->qpl[id]; - snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_dev->dev= ice.name, id); - mz =3D rte_memzone_reserve_aligned(z_name, pages * PAGE_SIZE, - rte_socket_id(), - RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE= ); - if (mz =3D=3D NULL) { - PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name); - return -ENOMEM; - } + qpl->page_buses =3D rte_zmalloc("qpl page buses", pages * sizeof(d= ma_addr_t), 0); if (qpl->page_buses =3D=3D NULL) { PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page buses", id); return -ENOMEM; } - page_bus =3D mz->iova; - for (i =3D 0; i < pages; i++) { - qpl->page_buses[i] =3D page_bus; - page_bus +=3D PAGE_SIZE; + + if (is_rx) { + /* RX QPL need not be IOVA contiguous. + * Allocate 4K size buffers using malloc + */ + qpl->qpl_bufs =3D rte_zmalloc("qpl bufs", + pages * sizeof(void *), 0); + if (qpl->qpl_bufs =3D=3D NULL) { + PMD_DRV_LOG(ERR, "Failed to alloc qpl bufs"); + err =3D -ENOMEM; + goto free_qpl_page_buses; + } + + err =3D gve_alloc_using_malloc(qpl->qpl_bufs, pages); + if (err) + goto free_qpl_page_bufs; + + /* Populate the IOVA addresses */ + for (i =3D 0; i < pages; i++) + qpl->page_buses[i] =3D + rte_malloc_virt2iova(qpl->qpl_bufs[i]); + } else { + char z_name[RTE_MEMZONE_NAMESIZE]; + + snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_= dev->device.name, id); + + /* TX QPL needs to be IOVA contiguous + * Allocate QPL using memzone + */ + qpl->mz =3D gve_alloc_using_mz(z_name, pages); + if (!qpl->mz) { + err =3D -ENOMEM; + goto free_qpl_page_buses; + } + + /* Populate the IOVA addresses */ + for (i =3D 0; i < pages; i++) + qpl->page_buses[i] =3D qpl->mz->iova + i * PAGE_SIZ= E; } + qpl->id =3D id; - qpl->mz =3D mz; qpl->num_entries =3D pages; priv->num_registered_pages +=3D pages; return 0; + +free_qpl_page_bufs: + rte_free(qpl->qpl_bufs); +free_qpl_page_buses: + rte_free(qpl->page_buses); + return err; +} + +/* + * Free QPL bufs in RX QPLs. Should not be used on TX QPLs. + **/ +static void +gve_free_qpl_bufs(struct gve_queue_page_list *qpl) +{ + uint32_t i; + + for (i =3D 0; i < qpl->num_entries; i++) + rte_free(qpl->qpl_bufs[i]); + + rte_free(qpl->qpl_bufs); + qpl->qpl_bufs =3D NULL; } static void @@ -74,9 +154,19 @@ gve_free_qpls(struct gve_priv *priv) if (priv->queue_format !=3D GVE_GQI_QPL_FORMAT) return; - for (i =3D 0; i < nb_txqs + nb_rxqs; i++) { - if (priv->qpl[i].mz !=3D NULL) + /* Free TX QPLs. */ + for (i =3D 0; i < nb_txqs; i++) { + if (priv->qpl[i].mz) { rte_memzone_free(priv->qpl[i].mz); + priv->qpl[i].mz =3D NULL; + } + rte_free(priv->qpl[i].page_buses); + } + + /* Free RX QPLs. */ + for (; i < nb_rxqs; i++) { + if (priv->qpl[i].qpl_bufs) + gve_free_qpl_bufs(&priv->qpl[i]); rte_free(priv->qpl[i].page_buses); } @@ -755,11 +845,16 @@ gve_init_priv(struct gve_priv *priv, bool skip_descri= be_device) } for (i =3D 0; i < priv->max_nb_txq + priv->max_nb_rxq; i++= ) { - if (i < priv->max_nb_txq) + bool is_rx; + + if (i < priv->max_nb_txq) { pages =3D priv->tx_pages_per_qpl; - else + is_rx =3D false; + } else { pages =3D priv->rx_data_slot_cnt; - err =3D gve_alloc_queue_page_list(priv, i, pages); + is_rx =3D true; + } + err =3D gve_alloc_queue_page_list(priv, i, pages, i= s_rx); if (err !=3D 0) { PMD_DRV_LOG(ERR, "Failed to alloc qpl %u."= , i); goto err_qpl; diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h index 58d8943e71..59febc153e 100644 --- a/drivers/net/gve/gve_ethdev.h +++ b/drivers/net/gve/gve_ethdev.h @@ -40,7 +40,10 @@ struct gve_queue_page_list { uint32_t id; /* unique id */ uint32_t num_entries; dma_addr_t *page_buses; /* the dma addrs of the pages */ - const struct rte_memzone *mz; + union { + const struct rte_memzone *mz; /* memzone allocated for TX q= ueue */ + void **qpl_bufs; /* RX qpl-buffer list allocated using mall= oc*/ + }; }; /* A TX desc ring entry */ diff --git a/drivers/net/gve/gve_rx.c b/drivers/net/gve/gve_rx.c index 36a1b73c65..b8ef625b5c 100644 --- a/drivers/net/gve/gve_rx.c +++ b/drivers/net/gve/gve_rx.c @@ -117,7 +117,7 @@ gve_rx_mbuf(struct gve_rx_queue *rxq, struct rte_mbuf *= rxe, uint16_t len, rxq->ctx.mbuf_tail =3D rxe; } if (rxq->is_gqi_qpl) { - addr =3D (uint64_t)(rxq->qpl->mz->addr) + rx_id * PAGE_SIZE= + padding; + addr =3D (uint64_t)rxq->qpl->qpl_bufs[rx_id] + padding; rte_memcpy((void *)((size_t)rxe->buf_addr + rxe->data_off)= , (void *)(size_t)addr, len); } -- 2.48.1.601.g30ceb7b040-goog --_000_CH3PR12MB8658A71EF0474EB94F05AE95A1A82CH3PR12MB8658namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Thanks for the your help, patch queued to 23.11.4 LTS release staging.

Regards,
Xueming
From: Joshua Washington <= ;joshwash@google.com>
Sent: Tuesday, March 4, 2025 7:06 AM
To: stable@dpdk.org <stable@dpdk.org>; Junfeng Guo <junfeng= .guo@intel.com>; Jeroen de Borst <jeroendb@google.com>; Rushil Gup= ta <rushilg@google.com>; Joshua Washington <joshwash@google.com>= ;; Xiaoyun Li <xiaoyun.li@intel.com>
Cc: Praveen Kaligineedi <pkaligineedi@google.com>
Subject: [PATCH 23.11] net/gve: allocate Rx QPL pages using malloc
 
From: Praveen Kaligineedi <pkaligineedi@google.= com>

Allocating QPL for an RX queue might fail if enough contiguous IOVA
memory cannot be allocated. This can commonly occur when using 2MB huge
pages because the 1024 4K buffers are allocated for each RX ring by
default, resulting in 4MB for each ring. However, the only requirement
for RX QPLs is that each 4K buffer be IOVA contiguous, not the entire
QPL. Therefore, malloc will be used to allocate RX QPLs instead.

Note that TX queues require the entire QPL to be IOVA contiguous, so it
will continue to use the memzone-based allocation.

Fixes: a46583cf43c8 ("net/gve: support Rx/Tx")
Cc: stable@dpdk.org

Signed-off-by: Praveen Kaligineedi <pkaligineedi@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
 drivers/net/gve/gve_ethdev.c | 139 +++++++++++++++++++++++++++++-----= -
 drivers/net/gve/gve_ethdev.h |   5 +-
 drivers/net/gve/gve_rx.c     |   2 +-  3 files changed, 122 insertions(+), 24 deletions(-)

diff --git a/drivers/net/gve/gve_ethdev.c b/drivers/net/gve/gve_ethdev.c index ecd37ff37f..d020e0be66 100644
--- a/drivers/net/gve/gve_ethdev.c
+++ b/drivers/net/gve/gve_ethdev.c
@@ -20,13 +20,45 @@ gve_write_version(uint8_t *driver_version_register)
         writeb('\n', driver_versio= n_register);
 }
 
+static const struct rte_memzone *
+gve_alloc_using_mz(const char *name, uint32_t num_pages)
+{
+       const struct rte_memzone *mz;
+       mz =3D rte_memzone_reserve_aligned(na= me, num_pages * PAGE_SIZE,
+            &n= bsp;            = ;            &n= bsp;  rte_socket_id(),
+            &n= bsp;            = ;            &n= bsp;  RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
+       if (mz =3D=3D NULL)
+            &n= bsp;  PMD_DRV_LOG(ERR, "Failed to alloc memzone %s.", name);=
+       return mz;
+}
+
 static int
-gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pag= es)
+gve_alloc_using_malloc(void **bufs, uint32_t num_entries)
+{
+       uint32_t i;
+
+       for (i =3D 0; i < num_entries; i++= ) {
+            &n= bsp;  bufs[i] =3D rte_malloc_socket(NULL, PAGE_SIZE, PAGE_SIZE, rte_so= cket_id());
+            &n= bsp;  if (bufs[i] =3D=3D NULL) {
+            &n= bsp;          PMD_DRV_LOG(ERR,= "Failed to malloc");
+            &n= bsp;          goto free_bufs;<= br> +            &n= bsp;  }
+       }
+       return 0;
+
+free_bufs:
+       while (i > 0)
+            &n= bsp;  rte_free(bufs[--i]);
+
+       return -ENOMEM;
+}
+
+static int
+gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pag= es,
+            &n= bsp;            bool= is_rx)
 {
-       char z_name[RTE_MEMZONE_NAMESIZE];          struct gve_queue_page_list= *qpl;
-       const struct rte_memzone *mz;
-       dma_addr_t page_bus;
+       int err =3D 0;
         uint32_t i;
 
         if (priv->num_registere= d_pages + pages >
@@ -37,31 +69,79 @@ gve_alloc_queue_page_list(struct gve_priv *priv, uint32= _t id, uint32_t pages)
            &nb= sp;    return -EINVAL;
         }
         qpl =3D &priv->qpl[= id];
-       snprintf(z_name, sizeof(z_name), &quo= t;gve_%s_qpl%d", priv->pci_dev->device.name, id);
-       mz =3D rte_memzone_reserve_aligned(z_= name, pages * PAGE_SIZE,
-            &n= bsp;            = ;            &n= bsp;  rte_socket_id(),
-            &n= bsp;            = ;            &n= bsp;  RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
-       if (mz =3D=3D NULL) {
-            &n= bsp;  PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name);
-            &n= bsp;  return -ENOMEM;
-       }
+
         qpl->page_buses =3D rte= _zmalloc("qpl page buses", pages * sizeof(dma_addr_t), 0);
         if (qpl->page_buses =3D= =3D NULL) {
            &nb= sp;    PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page bu= ses", id);
            &nb= sp;    return -ENOMEM;
         }
-       page_bus =3D mz->iova;
-       for (i =3D 0; i < pages; i++) { -            &n= bsp;  qpl->page_buses[i] =3D page_bus;
-            &n= bsp;  page_bus +=3D PAGE_SIZE;
+
+       if (is_rx) {
+            &n= bsp;  /* RX QPL need not be IOVA contiguous.
+            &n= bsp;   * Allocate 4K size buffers using malloc
+            &n= bsp;   */
+            &n= bsp;  qpl->qpl_bufs =3D rte_zmalloc("qpl bufs",
+            &n= bsp;          pages * sizeof(v= oid *), 0);
+            &n= bsp;  if (qpl->qpl_bufs =3D=3D NULL) {
+            &n= bsp;          PMD_DRV_LOG(ERR,= "Failed to alloc qpl bufs");
+            &n= bsp;          err =3D -ENOMEM;=
+            &n= bsp;          goto free_qpl_pa= ge_buses;
+            &n= bsp;  }
+
+            &n= bsp;  err =3D gve_alloc_using_malloc(qpl->qpl_bufs, pages);
+            &n= bsp;  if (err)
+            &n= bsp;          goto free_qpl_pa= ge_bufs;
+
+            &n= bsp;  /* Populate the IOVA addresses */
+            &n= bsp;  for (i =3D 0; i < pages; i++)
+            &n= bsp;          qpl->page_bus= es[i] =3D
+            &n= bsp;            = ;      rte_malloc_virt2iova(qpl->qpl_bufs[i]);<= br> +       } else {
+            &n= bsp;  char z_name[RTE_MEMZONE_NAMESIZE];
+
+            &n= bsp;  snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv-= >pci_dev->device.name, id);
+
+            &n= bsp;  /* TX QPL needs to be IOVA contiguous
+            &n= bsp;   * Allocate QPL using memzone
+            &n= bsp;   */
+            &n= bsp;  qpl->mz =3D gve_alloc_using_mz(z_name, pages);
+            &n= bsp;  if (!qpl->mz) {
+            &n= bsp;          err =3D -ENOMEM;=
+            &n= bsp;          goto free_qpl_pa= ge_buses;
+            &n= bsp;  }
+
+            &n= bsp;  /* Populate the IOVA addresses */
+            &n= bsp;  for (i =3D 0; i < pages; i++)
+            &n= bsp;          qpl->page_bus= es[i] =3D qpl->mz->iova + i * PAGE_SIZE;
         }
+
         qpl->id =3D id;
-       qpl->mz =3D mz;
         qpl->num_entries =3D pa= ges;
 
         priv->num_registered_pa= ges +=3D pages;
 
         return 0;
+
+free_qpl_page_bufs:
+       rte_free(qpl->qpl_bufs);
+free_qpl_page_buses:
+       rte_free(qpl->page_buses);
+       return err;
+}
+
+/*
+ * Free QPL bufs in RX QPLs. Should not be used on TX QPLs.
+ **/
+static void
+gve_free_qpl_bufs(struct gve_queue_page_list *qpl)
+{
+       uint32_t i;
+
+       for (i =3D 0; i < qpl->num_entr= ies; i++)
+            &n= bsp;  rte_free(qpl->qpl_bufs[i]);
+
+       rte_free(qpl->qpl_bufs);
+       qpl->qpl_bufs =3D NULL;
 }
 
 static void
@@ -74,9 +154,19 @@ gve_free_qpls(struct gve_priv *priv)
         if (priv->queue_format = !=3D GVE_GQI_QPL_FORMAT)
            &nb= sp;    return;
 
-       for (i =3D 0; i < nb_txqs + nb_rxq= s; i++) {
-            &n= bsp;  if (priv->qpl[i].mz !=3D NULL)
+       /* Free TX QPLs. */
+       for (i =3D 0; i < nb_txqs; i++) {<= br> +            &n= bsp;  if (priv->qpl[i].mz) {
            &nb= sp;            rte_m= emzone_free(priv->qpl[i].mz);
+            &n= bsp;          priv->qpl[i].= mz =3D NULL;
+            &n= bsp;  }
+            &n= bsp;  rte_free(priv->qpl[i].page_buses);
+       }
+
+       /* Free RX QPLs. */
+       for (; i < nb_rxqs; i++) {
+            &n= bsp;  if (priv->qpl[i].qpl_bufs)
+            &n= bsp;          gve_free_qpl_buf= s(&priv->qpl[i]);
            &nb= sp;    rte_free(priv->qpl[i].page_buses);
         }
 
@@ -755,11 +845,16 @@ gve_init_priv(struct gve_priv *priv, bool skip_descri= be_device)
            &nb= sp;    }
 
            &nb= sp;    for (i =3D 0; i < priv->max_nb_txq + priv->m= ax_nb_rxq; i++) {
-            &n= bsp;          if (i < priv-= >max_nb_txq)
+            &n= bsp;          bool is_rx;
+
+            &n= bsp;          if (i < priv-= >max_nb_txq) {
            &nb= sp;            =         pages =3D priv->tx_pages_per_= qpl;
-            &n= bsp;          else
+            &n= bsp;            = ;      is_rx =3D false;
+            &n= bsp;          } else {
            &nb= sp;            =         pages =3D priv->rx_data_slot_= cnt;
-            &n= bsp;          err =3D gve_allo= c_queue_page_list(priv, i, pages);
+            &n= bsp;            = ;      is_rx =3D true;
+            &n= bsp;          }
+            &n= bsp;          err =3D gve_allo= c_queue_page_list(priv, i, pages, is_rx);
            &nb= sp;            if (e= rr !=3D 0) {
            &nb= sp;            =         PMD_DRV_LOG(ERR, "Failed to= alloc qpl %u.", i);
            &nb= sp;            =         goto err_qpl;
diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h index 58d8943e71..59febc153e 100644
--- a/drivers/net/gve/gve_ethdev.h
+++ b/drivers/net/gve/gve_ethdev.h
@@ -40,7 +40,10 @@ struct gve_queue_page_list {
         uint32_t id; /* unique id = */
         uint32_t num_entries;
         dma_addr_t *page_buses; /*= the dma addrs of the pages */
-       const struct rte_memzone *mz;
+       union {
+            &n= bsp;  const struct rte_memzone *mz; /* memzone allocated for TX queue = */
+            &n= bsp;  void **qpl_bufs; /* RX qpl-buffer list allocated using malloc*/<= br> +       };
 };
 
 /* A TX desc ring entry */
diff --git a/drivers/net/gve/gve_rx.c b/drivers/net/gve/gve_rx.c
index 36a1b73c65..b8ef625b5c 100644
--- a/drivers/net/gve/gve_rx.c
+++ b/drivers/net/gve/gve_rx.c
@@ -117,7 +117,7 @@ gve_rx_mbuf(struct gve_rx_queue *rxq, struct rte_mbuf *= rxe, uint16_t len,
            &nb= sp;    rxq->ctx.mbuf_tail =3D rxe;
         }
         if (rxq->is_gqi_qpl) {<= br> -            &n= bsp;  addr =3D (uint64_t)(rxq->qpl->mz->addr) + rx_id * PAGE_= SIZE + padding;
+            &n= bsp;  addr =3D (uint64_t)rxq->qpl->qpl_bufs[rx_id] + padding;             &nb= sp;    rte_memcpy((void *)((size_t)rxe->buf_addr + rxe-&g= t;data_off),
            &nb= sp;            =             (void *)= (size_t)addr, len);
         }
--
2.48.1.601.g30ceb7b040-goog

--_000_CH3PR12MB8658A71EF0474EB94F05AE95A1A82CH3PR12MB8658namp_--