From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E183A455F3; Thu, 11 Jul 2024 14:35:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D025142E50; Thu, 11 Jul 2024 14:35:17 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) by mails.dpdk.org (Postfix) with ESMTP id 357024065B; Thu, 11 Jul 2024 14:35:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1720701316; x=1752237316; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=AvHnRgPr7/MphAXgkmuEUSOP8NE/5bVxNIv3DrEQNc0=; b=fL7NyO2zMVHm27JjuVz53WNWo3/tQcOIwMXIxHKWkrQaW9f33YMiP4x/ 7OOacS+k+HfH3fL4OZ+AZpWdarWTLk/XbvqAQ3/zFjSV6DRQrsSE1zTod 1Td3x055xHsUmhrlB7M5gLcHgx1Wq10B2Wk8O0RUIfsiDsPePgdo2TV7s VguDXJeIOTPNxpxkZsXP9crEtqu8NG9jlv5uqn7Ox6jA9nIzADJbLZO0k NqDWJ0kskbH4ZpMAojqHT3z+qtq9WV3FSPlvZLRH7ii6PS6P8WMKCzhqQ DBPdUlfh7V+Kr6KpGT7GFiapdbt7WpLOTH/QG1rYcvDcsCfo+C76EtbVa Q==; X-CSE-ConnectionGUID: v9juqvkpS2ugFLx2Whwy+Q== X-CSE-MsgGUID: x6Yj2RFqTvKLn9wCbXbhtw== X-IronPort-AV: E=McAfee;i="6700,10204,11129"; a="29228391" X-IronPort-AV: E=Sophos;i="6.09,200,1716274800"; d="scan'208";a="29228391" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jul 2024 05:35:15 -0700 X-CSE-ConnectionGUID: vI3xTBBWSr2Qq8Ct6xbNFQ== X-CSE-MsgGUID: lF8RspQTS3WCiRhDRgZNOQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,200,1716274800"; d="scan'208";a="53488320" Received: from silpixa00399413-oob.ir.intel.com (HELO silpixa00401385.ir.intel.com) ([10.237.214.33]) by orviesa005.jf.intel.com with ESMTP; 11 Jul 2024 05:35:14 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org, Padraig Connolly Subject: [PATCH] ethdev: fix device init without socket-local memory Date: Thu, 11 Jul 2024 13:35:00 +0100 Message-ID: <20240711123500.483119-1-bruce.richardson@intel.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When allocating memory for an ethdev, the rte_malloc_socket call used only allocates memory on the NUMA node/socket local to the device. This means that even if the user wanted to, they could never use a remote NIC without also having memory on that NIC's socket. For example, if we change examples/skeleton/basicfwd.c to have SOCKET_ID_ANY as the socket_id parameter for Rx and Tx rings, we should be able to run the app cross-numa e.g. as below, where the two PCI devices are on socket 1, and core 1 is on socket 0: ./build/examples/dpdk-skeleton -l 1 --legacy-mem --socket-mem=1024,0 \ -a a8:00.0 -a b8:00.0 This fails however, with the error: ETHDEV: failed to allocate private data PCI_BUS: Requested device 0000:a8:00.0 cannot be used We can remove this restriction by doing a fallback call to general rte_malloc after a call to rte_malloc_socket fails. This should be safe to do because the later ethdev calls to setup Rx/Tx queues all take a socket_id parameter, which can be used by applications to enforce the requirement for local-only memory for a device, if so desired. [If device-local memory is present it will be used as before, while if not present the rte_eth_dev_configure call will now pass, but the subsequent queue setup calls requesting local memory will fail]. Fixes: e489007a411c ("ethdev: add generic create/destroy ethdev APIs") Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") Cc: stable@dpdk.org Signed-off-by: Bruce Richardson Signed-off-by: Padraig Connolly --- lib/ethdev/ethdev_driver.c | 7 ++++++- lib/ethdev/ethdev_pci.h | 11 ++++++++++- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c index f48c0eb8bc..f9ce7ec348 100644 --- a/lib/ethdev/ethdev_driver.c +++ b/lib/ethdev/ethdev_driver.c @@ -303,11 +303,16 @@ rte_eth_dev_create(struct rte_device *device, const char *name, return -ENODEV; if (priv_data_size) { + /* try alloc private data on device-local node. */ ethdev->data->dev_private = rte_zmalloc_socket( name, priv_data_size, RTE_CACHE_LINE_SIZE, device->numa_node); + /* fall back to alloc on any socket on failure */ + if (ethdev->data->dev_private == NULL) + ethdev->data->dev_private = rte_zmalloc(name, + priv_data_size, RTE_CACHE_LINE_SIZE); - if (!ethdev->data->dev_private) { + if (ethdev->data->dev_private == NULL) { RTE_ETHDEV_LOG_LINE(ERR, "failed to allocate private data"); retval = -ENOMEM; diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h index 737fff1833..d600d9acbb 100644 --- a/lib/ethdev/ethdev_pci.h +++ b/lib/ethdev/ethdev_pci.h @@ -93,10 +93,19 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size) return NULL; if (private_data_size) { + /* Try and alloc the private-data structure on socket local to the device */ eth_dev->data->dev_private = rte_zmalloc_socket(name, private_data_size, RTE_CACHE_LINE_SIZE, dev->device.numa_node); - if (!eth_dev->data->dev_private) { + + /* if cannot allocate memory on the socket local to the device + * use rte_malloc to allocate memory on some other socket, if available. + */ + if (eth_dev->data->dev_private == NULL) + eth_dev->data->dev_private = rte_zmalloc(name, + private_data_size, RTE_CACHE_LINE_SIZE); + + if (eth_dev->data->dev_private == NULL) { rte_eth_dev_release_port(eth_dev); return NULL; } -- 2.43.0