From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EEDD74569C for ; Wed, 24 Jul 2024 13:34:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E5AD242D89; Wed, 24 Jul 2024 13:34:36 +0200 (CEST) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mails.dpdk.org (Postfix) with ESMTP id CFEDB42D28 for ; Wed, 24 Jul 2024 13:34:33 +0200 (CEST) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-42797289c8bso50551515e9.0 for ; Wed, 24 Jul 2024 04:34:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721820873; x=1722425673; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=upQ9QS2nMgszS54QDrdi2hFgR0+1cQs77lG6imOCyBs=; b=NKK7A1jlW3mqASDHs34TvA5zsOlInFR9AzlFP9mrs1LpgDvUA0z5/S8X3lEmcUG2L0 5ZQi4wr4PNKrunHnKgTIGIpgf7pB/WtZoYL0KAgUQlrGHs3jrae2g9vA++XXcn6GRUn2 zLgSRkpndE2MxLKvSE1Wl9nTrut82ZmNORVbKXGzhLTwfrnnoshTUhddqaW7StTIvbK4 xbuZtnCuc7/hKpq1NsueuRU1R0syWoGasIaIeuW3U3OfcmpCNdOhTeWlmcvjm3XiwtEe Z/jYbEBZd6gMajTVqXhpmhj974/eG0Nq+X9KhUS84QSdz+eH9NsTEp6Qgs8IYfUAiUR7 t8+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721820873; x=1722425673; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=upQ9QS2nMgszS54QDrdi2hFgR0+1cQs77lG6imOCyBs=; b=am9jE5SpgCR7jmf/n99qmhrKawrLE1oCzM9NHhThVxdDpUc4lBb0f+ezPzg/k83nRO sMjIxQfIzgqwLNBSK/v5c14Ko4/M6gKmTGgkv2kDd/XRXmhp/64WpBhPIRDCmk3vFZHY zCRF1G9EcIJDpTYreoBmXK3VoY0yyJwFzC0WQ+Z8eHfSSmvjuHe5cUXuKRWlQfQv4kK4 BTngJtrgJqILZTyK2rbwv14zhDEjR/6wdLgcqwWwBoBLh3r76PIISgIvhqwnVn3EY+km 2wmOzJB1GBshIytADYXgbLcb8KbkKpQZuLhO/AwimOYyscloJL+T7/O1V/BzHCqWk112 Q0mQ== X-Forwarded-Encrypted: i=1; AJvYcCV6nlgpVECQZivve+65QXSKtCgJGub1uA8aBIs6bYsvLmDaPiuU3ZJMNn8nXFW1KD/vEz4juXO3qq31VuJ7YM4= X-Gm-Message-State: AOJu0YxyFX+VIg1sl6K6ZIxd+ed5qO3zbjPJVWvoGyC3yCwHr2EaeTlB L0arEDKIaBMJE1AqORevdO8KFsUNKbcUPq4nLfOFUiVbXJN3MqkDZdLU55T+ X-Google-Smtp-Source: AGHT+IF7SlUayhzV+NTFImrt8wpE71NYhe8t6Xz0igaE/Nd6wuDRLIfrrovB+UfAufnYtsZ/vZWQPQ== X-Received: by 2002:a05:600c:19c6:b0:426:66a2:b200 with SMTP id 5b1f17b1804b1-427f937e296mr12597995e9.0.1721820873297; Wed, 24 Jul 2024 04:34:33 -0700 (PDT) Received: from localhost ([2a01:4b00:d036:ae00:7aef:1aaa:3dff:d546]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427f93e6795sm27289385e9.36.2024.07.24.04.34.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jul 2024 04:34:32 -0700 (PDT) From: luca.boccassi@gmail.com To: Bruce Richardson Cc: Padraig Connolly , Ferruh Yigit , dpdk stable Subject: patch 'ethdev: fix device init without socket-local memory' has been queued to stable release 22.11.6 Date: Wed, 24 Jul 2024 12:33:02 +0100 Message-Id: <20240724113318.616754-24-luca.boccassi@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240724113318.616754-1-luca.boccassi@gmail.com> References: <20240715152704.2229503-86-luca.boccassi@gmail.com> <20240724113318.616754-1-luca.boccassi@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 22.11.6 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 07/26/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/bluca/dpdk-stable This queued commit can be viewed at: https://github.com/bluca/dpdk-stable/commit/1092b5655fd49e38ce26368b379b3ec081f082a4 Thanks. Luca Boccassi --- >From 1092b5655fd49e38ce26368b379b3ec081f082a4 Mon Sep 17 00:00:00 2001 From: Bruce Richardson Date: Mon, 22 Jul 2024 11:02:28 +0100 Subject: [PATCH] ethdev: fix device init without socket-local memory [ upstream commit ed34d87d9cfbae8b908159f60df2008e45e4c39f ] When allocating memory for an ethdev, the rte_malloc_socket call used only allocates memory on the NUMA node/socket local to the device. This means that even if the user wanted to, they could never use a remote NIC without also having memory on that NIC's socket. For example, if we change examples/skeleton/basicfwd.c to have SOCKET_ID_ANY as the socket_id parameter for Rx and Tx rings, we should be able to run the app cross-numa e.g. as below, where the two PCI devices are on socket 1, and core 1 is on socket 0: ./build/examples/dpdk-skeleton -l 1 --legacy-mem --socket-mem=1024,0 \ -a a8:00.0 -a b8:00.0 This fails however, with the error: ETHDEV: failed to allocate private data PCI_BUS: Requested device 0000:a8:00.0 cannot be used We can remove this restriction by doing a fallback call to general rte_malloc after a call to rte_malloc_socket fails. This should be safe to do because the later ethdev calls to setup Rx/Tx queues all take a socket_id parameter, which can be used by applications to enforce the requirement for local-only memory for a device, if so desired. [If device-local memory is present it will be used as before, while if not present the rte_eth_dev_configure call will now pass, but the subsequent queue setup calls requesting local memory will fail]. Fixes: e489007a411c ("ethdev: add generic create/destroy ethdev APIs") Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") Signed-off-by: Bruce Richardson Signed-off-by: Padraig Connolly Reviewed-by: Ferruh Yigit --- lib/ethdev/ethdev_driver.c | 20 +++++++++++++++----- lib/ethdev/ethdev_pci.h | 20 +++++++++++++++++--- 2 files changed, 32 insertions(+), 8 deletions(-) diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c index 0387c7133d..867c5de3bb 100644 --- a/lib/ethdev/ethdev_driver.c +++ b/lib/ethdev/ethdev_driver.c @@ -275,15 +275,25 @@ rte_eth_dev_create(struct rte_device *device, const char *name, return -ENODEV; if (priv_data_size) { + /* try alloc private data on device-local node. */ ethdev->data->dev_private = rte_zmalloc_socket( name, priv_data_size, RTE_CACHE_LINE_SIZE, device->numa_node); - if (!ethdev->data->dev_private) { - RTE_ETHDEV_LOG(ERR, - "failed to allocate private data\n"); - retval = -ENOMEM; - goto probe_failed; + /* fall back to alloc on any socket on failure */ + if (ethdev->data->dev_private == NULL) { + ethdev->data->dev_private = rte_zmalloc(name, + priv_data_size, RTE_CACHE_LINE_SIZE); + + if (ethdev->data->dev_private == NULL) { + RTE_ETHDEV_LOG(ERR, "failed to allocate private data\n"); + retval = -ENOMEM; + goto probe_failed; + } + /* got memory, but not local, so issue warning */ + RTE_ETHDEV_LOG(WARNING, + "Private data for ethdev '%s' not allocated on local NUMA node %d\n", + device->name, device->numa_node); } } } else { diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h index ddb559aa95..fbe74fdefe 100644 --- a/lib/ethdev/ethdev_pci.h +++ b/lib/ethdev/ethdev_pci.h @@ -93,12 +93,26 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size) return NULL; if (private_data_size) { + /* Try and alloc the private-data structure on socket local to the device */ eth_dev->data->dev_private = rte_zmalloc_socket(name, private_data_size, RTE_CACHE_LINE_SIZE, dev->device.numa_node); - if (!eth_dev->data->dev_private) { - rte_eth_dev_release_port(eth_dev); - return NULL; + + /* if cannot allocate memory on the socket local to the device + * use rte_malloc to allocate memory on some other socket, if available. + */ + if (eth_dev->data->dev_private == NULL) { + eth_dev->data->dev_private = rte_zmalloc(name, + private_data_size, RTE_CACHE_LINE_SIZE); + + if (eth_dev->data->dev_private == NULL) { + rte_eth_dev_release_port(eth_dev); + return NULL; + } + /* got memory, but not local, so issue warning */ + RTE_ETHDEV_LOG(WARNING, + "Private data for ethdev '%s' not allocated on local NUMA node %d\n", + dev->device.name, dev->device.numa_node); } } } else { -- 2.39.2 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-07-24 12:29:22.619886438 +0100 +++ 0024-ethdev-fix-device-init-without-socket-local-memory.patch 2024-07-24 12:29:20.787026453 +0100 @@ -1 +1 @@ -From ed34d87d9cfbae8b908159f60df2008e45e4c39f Mon Sep 17 00:00:00 2001 +From 1092b5655fd49e38ce26368b379b3ec081f082a4 Mon Sep 17 00:00:00 2001 @@ -5,0 +6,2 @@ +[ upstream commit ed34d87d9cfbae8b908159f60df2008e45e4c39f ] + @@ -35 +36,0 @@ -Cc: stable@dpdk.org @@ -46 +47 @@ -index f48c0eb8bc..c335a25a82 100644 +index 0387c7133d..867c5de3bb 100644 @@ -49 +50 @@ -@@ -303,15 +303,25 @@ rte_eth_dev_create(struct rte_device *device, const char *name, +@@ -275,15 +275,25 @@ rte_eth_dev_create(struct rte_device *device, const char *name, @@ -59,2 +60,2 @@ -- RTE_ETHDEV_LOG_LINE(ERR, -- "failed to allocate private data"); +- RTE_ETHDEV_LOG(ERR, +- "failed to allocate private data\n"); @@ -69 +70 @@ -+ RTE_ETHDEV_LOG_LINE(ERR, "failed to allocate private data"); ++ RTE_ETHDEV_LOG(ERR, "failed to allocate private data\n"); @@ -74,2 +75,2 @@ -+ RTE_ETHDEV_LOG_LINE(WARNING, -+ "Private data for ethdev '%s' not allocated on local NUMA node %d", ++ RTE_ETHDEV_LOG(WARNING, ++ "Private data for ethdev '%s' not allocated on local NUMA node %d\n", @@ -81 +82 @@ -index 737fff1833..ec4f731270 100644 +index ddb559aa95..fbe74fdefe 100644 @@ -108,2 +109,2 @@ -+ RTE_ETHDEV_LOG_LINE(WARNING, -+ "Private data for ethdev '%s' not allocated on local NUMA node %d", ++ RTE_ETHDEV_LOG(WARNING, ++ "Private data for ethdev '%s' not allocated on local NUMA node %d\n",