From: Bruce Richardson <bruce.richardson@intel.com>
To: dev@dpdk.org
Cc: Bruce Richardson <bruce.richardson@intel.com>,
stable@dpdk.org, Padraig Connolly <padraig.j.connolly@intel.com>
Subject: [PATCH] ethdev: fix device init without socket-local memory
Date: Thu, 11 Jul 2024 13:35:00 +0100 [thread overview]
Message-ID: <20240711123500.483119-1-bruce.richardson@intel.com> (raw)
When allocating memory for an ethdev, the rte_malloc_socket call used
only allocates memory on the NUMA node/socket local to the device. This
means that even if the user wanted to, they could never use a remote NIC
without also having memory on that NIC's socket.
For example, if we change examples/skeleton/basicfwd.c to have
SOCKET_ID_ANY as the socket_id parameter for Rx and Tx rings, we should
be able to run the app cross-numa e.g. as below, where the two PCI
devices are on socket 1, and core 1 is on socket 0:
./build/examples/dpdk-skeleton -l 1 --legacy-mem --socket-mem=1024,0 \
-a a8:00.0 -a b8:00.0
This fails however, with the error:
ETHDEV: failed to allocate private data
PCI_BUS: Requested device 0000:a8:00.0 cannot be used
We can remove this restriction by doing a fallback call to general
rte_malloc after a call to rte_malloc_socket fails. This should be safe
to do because the later ethdev calls to setup Rx/Tx queues all take a
socket_id parameter, which can be used by applications to enforce the
requirement for local-only memory for a device, if so desired. [If
device-local memory is present it will be used as before, while if not
present the rte_eth_dev_configure call will now pass, but the subsequent
queue setup calls requesting local memory will fail].
Fixes: e489007a411c ("ethdev: add generic create/destroy ethdev APIs")
Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers")
Cc: stable@dpdk.org
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Padraig Connolly <padraig.j.connolly@intel.com>
---
lib/ethdev/ethdev_driver.c | 7 ++++++-
lib/ethdev/ethdev_pci.h | 11 ++++++++++-
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index f48c0eb8bc..f9ce7ec348 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -303,11 +303,16 @@ rte_eth_dev_create(struct rte_device *device, const char *name,
return -ENODEV;
if (priv_data_size) {
+ /* try alloc private data on device-local node. */
ethdev->data->dev_private = rte_zmalloc_socket(
name, priv_data_size, RTE_CACHE_LINE_SIZE,
device->numa_node);
+ /* fall back to alloc on any socket on failure */
+ if (ethdev->data->dev_private == NULL)
+ ethdev->data->dev_private = rte_zmalloc(name,
+ priv_data_size, RTE_CACHE_LINE_SIZE);
- if (!ethdev->data->dev_private) {
+ if (ethdev->data->dev_private == NULL) {
RTE_ETHDEV_LOG_LINE(ERR,
"failed to allocate private data");
retval = -ENOMEM;
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 737fff1833..d600d9acbb 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -93,10 +93,19 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size)
return NULL;
if (private_data_size) {
+ /* Try and alloc the private-data structure on socket local to the device */
eth_dev->data->dev_private = rte_zmalloc_socket(name,
private_data_size, RTE_CACHE_LINE_SIZE,
dev->device.numa_node);
- if (!eth_dev->data->dev_private) {
+
+ /* if cannot allocate memory on the socket local to the device
+ * use rte_malloc to allocate memory on some other socket, if available.
+ */
+ if (eth_dev->data->dev_private == NULL)
+ eth_dev->data->dev_private = rte_zmalloc(name,
+ private_data_size, RTE_CACHE_LINE_SIZE);
+
+ if (eth_dev->data->dev_private == NULL) {
rte_eth_dev_release_port(eth_dev);
return NULL;
}
--
2.43.0
next reply other threads:[~2024-07-11 12:35 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-07-11 12:35 Bruce Richardson [this message]
2024-07-19 8:59 ` Ferruh Yigit
2024-07-19 9:57 ` Bruce Richardson
2024-07-19 11:10 ` Ferruh Yigit
2024-07-19 13:22 ` Bruce Richardson
2024-07-19 15:31 ` Ferruh Yigit
2024-07-19 16:10 ` Bruce Richardson
2024-07-21 22:56 ` Ferruh Yigit
2024-07-22 10:06 ` Bruce Richardson
2024-07-19 10:41 ` Bruce Richardson
2024-07-22 10:02 ` [PATCH v2] " Bruce Richardson
2024-07-22 13:24 ` Ferruh Yigit
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240711123500.483119-1-bruce.richardson@intel.com \
--to=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=padraig.j.connolly@intel.com \
--cc=stable@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).