From: Bruce Richardson <bruce.richardson@intel.com>
To: dev@dpdk.org
Cc: Anatoly Burakov <anatoly.burakov@intel.com>,
Bruce Richardson <bruce.richardson@intel.com>
Subject: [PATCH v2] eal: allow swapping of malloc heaps
Date: Fri, 15 Sep 2023 14:10:15 +0100 [thread overview]
Message-ID: <20230915131015.820160-1-bruce.richardson@intel.com> (raw)
In-Reply-To: <20230915122703.475834-1-bruce.richardson@intel.com>
The external memory functions in DPDK allow the addition of externally
access memory to malloc heaps, but with one major restriction - the
memory must be allocated to an application-created heap, not one of the
standard DPDK heaps for a NUMA node.
This restriction makes it difficult - if not impossible - to use
externally allocated memory for DPDK by default. However, even if the
restriction is relaxed, so we can add external memory to e.g. the socket
0 heap, there would be no way to guarantee that the external memory
would be used in preference to the standard DPDK hugepage memory for a
given allocation.
To give appropriately defined behaviour, a better solution is to allow
the application to explicitly swap a pair of heaps. With this one new
API in place, it allows the user to configure a new malloc heap, add
external memory to it, and then replace a standard socket heap with the
newly created one - thereby guaranteeing future allocations from the
external memory.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
V2: Fix doxygen comment issue on doc builds. Mark code snippet using
@code...@endcode to avoid issues with "<" symbols
---
lib/eal/common/malloc_heap.c | 18 ++++++++++++++++++
lib/eal/include/rte_malloc.h | 28 ++++++++++++++++++++++++++++
lib/eal/version.map | 2 ++
3 files changed, 48 insertions(+)
diff --git a/lib/eal/common/malloc_heap.c b/lib/eal/common/malloc_heap.c
index 6b6cf9174c..e5d0ad6b52 100644
--- a/lib/eal/common/malloc_heap.c
+++ b/lib/eal/common/malloc_heap.c
@@ -1320,6 +1320,24 @@ malloc_heap_add_external_memory(struct malloc_heap *heap,
return 0;
}
+int
+rte_malloc_heap_swap_socket(int socket1, int socket2)
+{
+ const int h1 = malloc_socket_to_heap_id(socket1);
+ if (h1 < 0 || h1 > RTE_MAX_HEAPS)
+ return -1;
+
+ const int h2 = malloc_socket_to_heap_id(socket2);
+ if (h2 < 0 || h2 > RTE_MAX_HEAPS)
+ return -1;
+
+ struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+ int tmp = mcfg->malloc_heaps[h1].socket_id;
+ mcfg->malloc_heaps[h1].socket_id = mcfg->malloc_heaps[h2].socket_id;
+ mcfg->malloc_heaps[h2].socket_id = tmp;
+ return 0;
+}
+
int
malloc_heap_remove_external_memory(struct malloc_heap *heap, void *va_addr,
size_t len)
diff --git a/lib/eal/include/rte_malloc.h b/lib/eal/include/rte_malloc.h
index 54a8ac211e..410211aa44 100644
--- a/lib/eal/include/rte_malloc.h
+++ b/lib/eal/include/rte_malloc.h
@@ -490,6 +490,34 @@ rte_malloc_heap_get_socket(const char *name);
int
rte_malloc_heap_socket_is_external(int socket_id);
+/**
+ * Swap the heaps for the given socket ids
+ *
+ * This causes the heaps for the given socket ids to be swapped, allowing
+ * external memory registered as a malloc heap to become the new default memory
+ * for a standard numa node. For example, to have allocations on socket 0 come
+ * from external memory, the following sequence of API calls can be used:
+ * @code
+ * rte_malloc_heap_create(<name>)
+ * rte_malloc_heap_memory_add(<name>,....)
+ * id = rte_malloc_heap_get_socket(<name>)
+ * rte_malloc_heap_swap_socket(0, id)
+ * @endcode
+ *
+ * Following these calls, allocations for the old memory allocated on socket 0,
+ * can be made by passing "id" as the socket_id parameter
+ *
+ * @param socket1
+ * The socket id of the first heap to swap
+ * @param socket2
+ * The socket id of the second heap to swap
+ * @return
+ * 0 on success, -1 on error
+ */
+__rte_experimental
+int
+rte_malloc_heap_swap_socket(int socket1, int socket2);
+
/**
* Dump statistics.
*
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 7940431e5a..b06ee7219e 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -417,6 +417,8 @@ EXPERIMENTAL {
# added in 23.07
rte_memzone_max_get;
rte_memzone_max_set;
+
+ rte_malloc_heap_swap_socket;
};
INTERNAL {
--
2.39.2
next prev parent reply other threads:[~2023-09-15 13:10 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-15 12:27 [PATCH] " Bruce Richardson
2023-09-15 13:10 ` Bruce Richardson [this message]
2023-09-15 15:43 ` Stephen Hemminger
2023-09-18 16:32 ` [PATCH v3 0/2] Enable apps to use external memory as native Bruce Richardson
2023-09-18 16:32 ` [PATCH v3 1/2] eal: add flag to indicate non-EAL malloc heaps Bruce Richardson
2023-09-18 16:32 ` [PATCH v3 2/2] eal: allow swapping of " Bruce Richardson
2023-09-18 16:44 ` [PATCH v3 0/2] Enable apps to use external memory as native Stephen Hemminger
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230915131015.820160-1-bruce.richardson@intel.com \
--to=bruce.richardson@intel.com \
--cc=anatoly.burakov@intel.com \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).