From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4D7C2457A1 for ; Mon, 12 Aug 2024 15:03:38 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 40C874065A; Mon, 12 Aug 2024 15:03:38 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2066.outbound.protection.outlook.com [40.107.237.66]) by mails.dpdk.org (Postfix) with ESMTP id 19B9F402C3 for ; Mon, 12 Aug 2024 15:03:36 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xI5FRdkMel75mmfG34cx3wXG4wtHFFWgmKen/CDbjP358guCiAhhingcWogJ6wgTjrP3a1NkyJWsK1qyD1ViXRU8pUtBgNcw4WLFnVuB4LTyBa652GBTicv596oB5L1MKD+K0BdbwVf6KY5xuI1JjwX3EZd1SL6UxR8U43co99gKSkdh43P6wyz3D/yIjms5/5cZqleNoPDqHY8RFhiHJPMQdrZ1Jwn+na0RR4++8rW+CtNkqXitUmoOAIFjdCehM2/MssVdlHbU4L78wFo9eelQQNIJn7k5/jJeFMUEfbmurJo4Po6xcQobyOoktH/t3ZFeLSoAiDIs8JVfFrLVNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dQV0tBH/JF1WqcpZt5Rls9gB1bhsOf04HqEit3acos8=; b=xvURK9ynhC9ikgh+KJpH01Cxdy5l8+ywVaWiLwlGD+TKrUZpPurSbFxtrDgz169Mr5gMry6ukiq12q5Q0SJmPZWa9uEExOkqQM7WyDwzty2wGC/1IGFVHdI57Q2kuP0V+UjMol0T+CyzJwRL5bJSm8bJSg29H4yPwWMqgd3s/iGnjovN0c+gx0lzR4NQqKsV4QnEXDRGbzGgNZqdRClfbJBm9bCb2tf/ApNKrVUJw16fcJlGO7LXdB7bQlcBYLbkbGrOpJH6NAOqL5mkurSP+jrKGi6mQ0aGTWCW2MGpT8uZxcJUuZ3TpfTrWk7uhmEx8oCYQUeivAqe548FpYyxNA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dQV0tBH/JF1WqcpZt5Rls9gB1bhsOf04HqEit3acos8=; b=n3cCokC01v+wazN1gZXjehb5Mn9PTMHEfSkKwM9QHz65N7LxH1EfyvEH5d1NHfqD+LfpLtlM44HY/vocczbgqyMzFYRsk9EXxLTr60Epoze5a86TgzI4T6vwOPnz6f6Kn5Ae55vUN6q+4WZyKmANCLb89pqid8UdHw/HbPfA8oHAnZSnyrs3R1f80cFHPRrqJd1UnjHpo9C0PxBOVat/3CgBfbkQcsBzQc3fXfB1+ZYvZ3W+OHgHb1F2tH6dXuGjA8omZqsBqd0z2zVRT4Qd3PJLEAbtUWipH5ImYra5KuVfiX/8g4x7+kbG/z/JpBv+dpKyvyvnkuZ6Pj7diFsvUw== Received: from BY3PR04CA0012.namprd04.prod.outlook.com (2603:10b6:a03:217::17) by SA3PR12MB8000.namprd12.prod.outlook.com (2603:10b6:806:31f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.19; Mon, 12 Aug 2024 13:03:31 +0000 Received: from SJ5PEPF00000205.namprd05.prod.outlook.com (2603:10b6:a03:217:cafe::83) by BY3PR04CA0012.outlook.office365.com (2603:10b6:a03:217::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.22 via Frontend Transport; Mon, 12 Aug 2024 13:03:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SJ5PEPF00000205.mail.protection.outlook.com (10.167.244.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.8 via Frontend Transport; Mon, 12 Aug 2024 13:03:31 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 12 Aug 2024 06:03:13 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 12 Aug 2024 06:03:11 -0700 From: Xueming Li To: Bruce Richardson CC: , Padraig Connolly , Ferruh Yigit , dpdk stable Subject: patch 'ethdev: fix device init without socket-local memory' has been queued to stable release 23.11.2 Date: Mon, 12 Aug 2024 20:50:11 +0800 Message-ID: <20240812125035.389667-135-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812125035.389667-1-xuemingl@nvidia.com> References: <20240712110153.309690-23-xuemingl@nvidia.com> <20240812125035.389667-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000205:EE_|SA3PR12MB8000:EE_ X-MS-Office365-Filtering-Correlation-Id: 401b7452-946f-42a2-f8d2-08dcbacf2a9e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?VygpMr2YYM8wqEowZ+j17nkAMxwVm59DkbNBCXxvPVGtloICSRMcPxSGXCv8?= =?us-ascii?Q?A2reJxvq15RhZ1Tl298sn3zVQLZl17umI5j8oyOv04Uh6BWXpB5BBLKZWK5/?= =?us-ascii?Q?WEfprIeHByWwjNnIXBeePbD/NXFtM58TAyxDcZGxA9XhYfIRRjCZpbrUmgjZ?= =?us-ascii?Q?QYIF2hy/dHEn5UHqKpSsybXDQvwyjTUEcWKcSdP5qM/NHUglIxESBcx9eQQn?= =?us-ascii?Q?J1yQlLjNWQYrHYHIzov8ORcumasp4ld3lq2//HcUM/pzkybTOX9egnV9KD3/?= =?us-ascii?Q?hGFUCbR8AEEkPWzFig9sIui0/cWMRTruUqHP/s9RpLK5YggOcBGm9eyoA6x4?= =?us-ascii?Q?qKKqGtQZNZfMCgZr0YUlyFTVjg8su3kZJtxJNnoA8HgoUbsRRO6QcAOHU7h7?= =?us-ascii?Q?/u7bbizBAFTeZcZad0/O0dVR6FYGUlQw65Doj/z41hqRtfJAJDruKryI/50t?= =?us-ascii?Q?Qc/ytGtotELMM/JrT7mGd1iham5sDQ46DFaTJIn1fVZhC1a2RCSe/L64KFvK?= =?us-ascii?Q?/K6JE8MSsYeM6fhjSuXl/i//wDYeCGrX/sbc6w5FI7jqZrpsIbFJlQagt4z6?= =?us-ascii?Q?Zv5/UaMtBO2E5nUX6/QVwNNmD42NVWGG+pPfSv4f17Yer3rHCzeR0qYYEXn2?= =?us-ascii?Q?Pa5d1T3tBcALv34iQkcj1QxAjMKnsPtdrjWLRGCFHnk7Z1+hsbqLs1Wth4DO?= =?us-ascii?Q?v+6qAHff0tf9RJnoyVjheEwetmWJM3J7QK4OI2jKgNyFTJAWX4PezJSXMFMO?= =?us-ascii?Q?08C+au6POvAcvY1SUwyDWLf/8gP9E0DOUrEMyb3aqMsi23JWav3XpRPITO7W?= =?us-ascii?Q?gQ8uOuiGHbg21Qs4qIFYg7czIktRZ4O6EvlmgHfmkThh1w8lI7E/bf4W6Gug?= =?us-ascii?Q?RidizTkt2AMOTi/XBxaY0VN0ISskhPobuPqdwMftIG8a8dNzR9YddieV3E9Y?= =?us-ascii?Q?HaYGNdg+jvauuGXnzJwq+WwT2ABCk4GzV2c+qp1uyWLHXa7WL81/aBatlubt?= =?us-ascii?Q?1tiDVp8gfq4QBpQgF8UqQlu8ZOi5CbMS8jGTP+zKrTo/3gyfG9jyhWYTrQuo?= =?us-ascii?Q?p7JhYM62zr4ZbKptlUE1hPs7r8Rmg0VDRQXEjsFBudYFFayrmypCXzHd6aeE?= =?us-ascii?Q?A+fE8v/xOBpaDa4CTPjnX0kbGBs9l5gelgyd5Kcf/npJRpW78swO47e4A7Li?= =?us-ascii?Q?JpG15zJc6z4Ik6nNlhGM9DNOKVrWid38WMGqUE0gZIpowQqzM4yVsfolKT5Q?= =?us-ascii?Q?bFe5M4RoGLE775BTXYBhPTnaGXa7Qxn3XfcceX8e6QM1ANa3yD/0fHaFb9Nz?= =?us-ascii?Q?W+N5ki+NuBYsk/bURGPb4Mk4WI3ntMT2GqvNMEvEy0qyAJfRHlHdQ9lIzC9q?= =?us-ascii?Q?/Pb2zYP28UXmj8qDy2sCk1FAN7Ng8NgqaDL89kudMr4aLbabQ4scbdmH6ciW?= =?us-ascii?Q?nybScLjVFw0w1uEoyHudCxLXoyhPEy4l?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2024 13:03:31.5357 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 401b7452-946f-42a2-f8d2-08dcbacf2a9e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000205.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB8000 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.2 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 08/14/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://git.dpdk.org/dpdk-stable/log/?h=23.11-staging This queued commit can be viewed at: https://git.dpdk.org/dpdk-stable/commit/?h=23.11-staging&id=7ff9eeeb710f2d1b441d7db1efca0809c20bebae Thanks. Xueming Li --- >From 7ff9eeeb710f2d1b441d7db1efca0809c20bebae Mon Sep 17 00:00:00 2001 From: Bruce Richardson Date: Mon, 22 Jul 2024 11:02:28 +0100 Subject: [PATCH] ethdev: fix device init without socket-local memory Cc: Xueming Li [ upstream commit ed34d87d9cfbae8b908159f60df2008e45e4c39f ] When allocating memory for an ethdev, the rte_malloc_socket call used only allocates memory on the NUMA node/socket local to the device. This means that even if the user wanted to, they could never use a remote NIC without also having memory on that NIC's socket. For example, if we change examples/skeleton/basicfwd.c to have SOCKET_ID_ANY as the socket_id parameter for Rx and Tx rings, we should be able to run the app cross-numa e.g. as below, where the two PCI devices are on socket 1, and core 1 is on socket 0: ./build/examples/dpdk-skeleton -l 1 --legacy-mem --socket-mem=1024,0 \ -a a8:00.0 -a b8:00.0 This fails however, with the error: ETHDEV: failed to allocate private data PCI_BUS: Requested device 0000:a8:00.0 cannot be used We can remove this restriction by doing a fallback call to general rte_malloc after a call to rte_malloc_socket fails. This should be safe to do because the later ethdev calls to setup Rx/Tx queues all take a socket_id parameter, which can be used by applications to enforce the requirement for local-only memory for a device, if so desired. [If device-local memory is present it will be used as before, while if not present the rte_eth_dev_configure call will now pass, but the subsequent queue setup calls requesting local memory will fail]. Fixes: e489007a411c ("ethdev: add generic create/destroy ethdev APIs") Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") Signed-off-by: Bruce Richardson Signed-off-by: Padraig Connolly Reviewed-by: Ferruh Yigit --- lib/ethdev/ethdev_driver.c | 20 +++++++++++++++----- lib/ethdev/ethdev_pci.h | 20 +++++++++++++++++--- 2 files changed, 32 insertions(+), 8 deletions(-) diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c index 55a9dcc565..47659d5e8c 100644 --- a/lib/ethdev/ethdev_driver.c +++ b/lib/ethdev/ethdev_driver.c @@ -297,15 +297,25 @@ rte_eth_dev_create(struct rte_device *device, const char *name, return -ENODEV; if (priv_data_size) { + /* try alloc private data on device-local node. */ ethdev->data->dev_private = rte_zmalloc_socket( name, priv_data_size, RTE_CACHE_LINE_SIZE, device->numa_node); - if (!ethdev->data->dev_private) { - RTE_ETHDEV_LOG(ERR, - "failed to allocate private data\n"); - retval = -ENOMEM; - goto probe_failed; + /* fall back to alloc on any socket on failure */ + if (ethdev->data->dev_private == NULL) { + ethdev->data->dev_private = rte_zmalloc(name, + priv_data_size, RTE_CACHE_LINE_SIZE); + + if (ethdev->data->dev_private == NULL) { + RTE_ETHDEV_LOG(ERR, "failed to allocate private data\n"); + retval = -ENOMEM; + goto probe_failed; + } + /* got memory, but not local, so issue warning */ + RTE_ETHDEV_LOG(WARNING, + "Private data for ethdev '%s' not allocated on local NUMA node %d\n", + device->name, device->numa_node); } } } else { diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h index ddb559aa95..c40bc2ed02 100644 --- a/lib/ethdev/ethdev_pci.h +++ b/lib/ethdev/ethdev_pci.h @@ -93,12 +93,26 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size) return NULL; if (private_data_size) { + /* Try and alloc the private-data structure on socket local to the device */ eth_dev->data->dev_private = rte_zmalloc_socket(name, private_data_size, RTE_CACHE_LINE_SIZE, dev->device.numa_node); - if (!eth_dev->data->dev_private) { - rte_eth_dev_release_port(eth_dev); - return NULL; + + /* if cannot allocate memory on the socket local to the device + * use rte_malloc to allocate memory on some other socket, if available. + */ + if (eth_dev->data->dev_private == NULL) { + eth_dev->data->dev_private = rte_zmalloc(name, + private_data_size, RTE_CACHE_LINE_SIZE); + + if (eth_dev->data->dev_private == NULL) { + rte_eth_dev_release_port(eth_dev); + return NULL; + } + /* got memory, but not local, so issue warning */ + RTE_ETHDEV_LOG(WARNING, + "Private data for ethdev '%s' not allocated on local NUMA node %d\n", + dev->device.name, dev->device.numa_node); } } } else { -- 2.34.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-08-12 20:44:06.766476861 +0800 +++ 0134-ethdev-fix-device-init-without-socket-local-memory.patch 2024-08-12 20:44:02.545069383 +0800 @@ -1 +1 @@ -From ed34d87d9cfbae8b908159f60df2008e45e4c39f Mon Sep 17 00:00:00 2001 +From 7ff9eeeb710f2d1b441d7db1efca0809c20bebae Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit ed34d87d9cfbae8b908159f60df2008e45e4c39f ] @@ -35 +37,0 @@ -Cc: stable@dpdk.org @@ -46 +48 @@ -index f48c0eb8bc..c335a25a82 100644 +index 55a9dcc565..47659d5e8c 100644 @@ -49 +51 @@ -@@ -303,15 +303,25 @@ rte_eth_dev_create(struct rte_device *device, const char *name, +@@ -297,15 +297,25 @@ rte_eth_dev_create(struct rte_device *device, const char *name, @@ -59,2 +61,2 @@ -- RTE_ETHDEV_LOG_LINE(ERR, -- "failed to allocate private data"); +- RTE_ETHDEV_LOG(ERR, +- "failed to allocate private data\n"); @@ -69 +71 @@ -+ RTE_ETHDEV_LOG_LINE(ERR, "failed to allocate private data"); ++ RTE_ETHDEV_LOG(ERR, "failed to allocate private data\n"); @@ -74,3 +76,3 @@ -+ RTE_ETHDEV_LOG_LINE(WARNING, -+ "Private data for ethdev '%s' not allocated on local NUMA node %d", -+ device->name, device->numa_node); ++ RTE_ETHDEV_LOG(WARNING, ++ "Private data for ethdev '%s' not allocated on local NUMA node %d\n", ++ device->name, device->numa_node); @@ -81 +83 @@ -index 737fff1833..ec4f731270 100644 +index ddb559aa95..c40bc2ed02 100644 @@ -108,3 +110,3 @@ -+ RTE_ETHDEV_LOG_LINE(WARNING, -+ "Private data for ethdev '%s' not allocated on local NUMA node %d", -+ dev->device.name, dev->device.numa_node); ++ RTE_ETHDEV_LOG(WARNING, ++ "Private data for ethdev '%s' not allocated on local NUMA node %d\n", ++ dev->device.name, dev->device.numa_node);