DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH] eal: allow swapping of malloc heaps
@ 2023-09-15 12:27 Bruce Richardson
  2023-09-15 13:10 ` [PATCH v2] " Bruce Richardson
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Bruce Richardson @ 2023-09-15 12:27 UTC (permalink / raw)
  To: dev; +Cc: Bruce Richardson, Anatoly Burakov

The external memory functions in DPDK allow the addition of externally
access memory to malloc heaps, but with one major restriction - the
memory must be allocated to an application-created heap, not one of the
standard DPDK heaps for a NUMA node.

This restriction makes it difficult - if not impossible - to use
externally allocated memory for DPDK by default. However, even if the
restriction is relaxed, so we can add external memory to e.g. the socket
0 heap, there would be no way to guarantee that the external memory
would be used in preference to the standard DPDK hugepage memory for a
given allocation.

To give appropriately defined behaviour, a better solution is to allow
the application to explicitly swap a pair of heaps. With this one new
API in place, it allows the user to configure a new malloc heap, add
external memory to it, and then replace a standard socket heap with the
newly created one - thereby guaranteeing future allocations from the
external memory.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/eal/common/malloc_heap.c | 18 ++++++++++++++++++
 lib/eal/include/rte_malloc.h | 25 +++++++++++++++++++++++++
 lib/eal/version.map          |  2 ++
 3 files changed, 45 insertions(+)

diff --git a/lib/eal/common/malloc_heap.c b/lib/eal/common/malloc_heap.c
index 6b6cf9174c..e5d0ad6b52 100644
--- a/lib/eal/common/malloc_heap.c
+++ b/lib/eal/common/malloc_heap.c
@@ -1320,6 +1320,24 @@ malloc_heap_add_external_memory(struct malloc_heap *heap,
 	return 0;
 }
 
+int
+rte_malloc_heap_swap_socket(int socket1, int socket2)
+{
+	const int h1 = malloc_socket_to_heap_id(socket1);
+	if (h1 < 0 || h1 > RTE_MAX_HEAPS)
+		return -1;
+
+	const int h2 = malloc_socket_to_heap_id(socket2);
+	if (h2 < 0 || h2 > RTE_MAX_HEAPS)
+		return -1;
+
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+	int tmp = mcfg->malloc_heaps[h1].socket_id;
+	mcfg->malloc_heaps[h1].socket_id = mcfg->malloc_heaps[h2].socket_id;
+	mcfg->malloc_heaps[h2].socket_id = tmp;
+	return 0;
+}
+
 int
 malloc_heap_remove_external_memory(struct malloc_heap *heap, void *va_addr,
 		size_t len)
diff --git a/lib/eal/include/rte_malloc.h b/lib/eal/include/rte_malloc.h
index 54a8ac211e..fa70ca7430 100644
--- a/lib/eal/include/rte_malloc.h
+++ b/lib/eal/include/rte_malloc.h
@@ -490,6 +490,31 @@ rte_malloc_heap_get_socket(const char *name);
 int
 rte_malloc_heap_socket_is_external(int socket_id);
 
+/**
+ * Swap the heaps for the given socket ids
+ *
+ * This causes the heaps for the given socket ids to be swapped, allowing
+ * external memory registered as a malloc heap to become the new default memory
+ * for a standard numa node. For example, to have allocations on socket 0 come
+ * from external memory, the following sequence of API calls can be used:
+ *   - rte_malloc_heap_create(<name>)
+ *   - rte_malloc_heap_memory_add(<name>,....)
+ *   - id = rte_malloc_heap_get_socket(<name>)
+ *   - rte_malloc_heap_swap_sockets(0, id)
+ * Following these calls, allocations for the old memory allocated on socket 0,
+ * can be made by passing "id" as the socket_id parameter
+ *
+ * @param socket1
+ *   The socket id of the first heap to swap
+ * @param socket2
+ *   The socket id of the second heap to swap
+ * @return
+ *   0 on success, -1 on error
+ */
+__rte_experimental
+int
+rte_malloc_heap_swap_socket(int socket1, int socket2);
+
 /**
  * Dump statistics.
  *
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 7940431e5a..b06ee7219e 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -417,6 +417,8 @@ EXPERIMENTAL {
 	# added in 23.07
 	rte_memzone_max_get;
 	rte_memzone_max_set;
+
+	rte_malloc_heap_swap_socket;
 };
 
 INTERNAL {
-- 
2.39.2


^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-09-18 16:44 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-15 12:27 [PATCH] eal: allow swapping of malloc heaps Bruce Richardson
2023-09-15 13:10 ` [PATCH v2] " Bruce Richardson
2023-09-15 15:43 ` [PATCH] " Stephen Hemminger
2023-09-18 16:32 ` [PATCH v3 0/2] Enable apps to use external memory as native Bruce Richardson
2023-09-18 16:32   ` [PATCH v3 1/2] eal: add flag to indicate non-EAL malloc heaps Bruce Richardson
2023-09-18 16:32   ` [PATCH v3 2/2] eal: allow swapping of " Bruce Richardson
2023-09-18 16:44   ` [PATCH v3 0/2] Enable apps to use external memory as native Stephen Hemminger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).