From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 69C0A425A5; Fri, 15 Sep 2023 14:27:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0F7DC402AF; Fri, 15 Sep 2023 14:27:14 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 0308E40041 for ; Fri, 15 Sep 2023 14:27:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1694780832; x=1726316832; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=Do0inwSS6N9rSSYK8ZHyTrk/Ona0Y5VsItIdJcizRg4=; b=Dv/UWZ5EclnNDqQ8LupRRomHxz+9FYXOz/UsCS513oEOKyVX7p+y1gf0 U8GGXboD3uj9fpGIfaD0eVCBqb8DpNlAA8EdNdoj71W1sc2exoEhSuTxa yfhGNqbFUyOFMXZNPu0xQpD+njWKYRcTdUy2kMNEk2uz8JrEyL4J614pw Qh/bolfKp/+sPgSxqO/r+n3kMqossYEFvgags51vIdiDD9hFwRUIqtRnB 0gQsXYCvZpZtYIWFBrKtcmxbbNCVNfEnOSUC9IVMbtOe6w9hlEtNBgJKS 3Kcac0ZEIAy69+pvXSR43qjq0JkH5h2Xj1TnpujOQKYZ/NFxfG9gvgQ3d Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10834"; a="369552841" X-IronPort-AV: E=Sophos;i="6.02,149,1688454000"; d="scan'208";a="369552841" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2023 05:27:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10834"; a="1075777924" X-IronPort-AV: E=Sophos;i="6.02,149,1688454000"; d="scan'208";a="1075777924" Received: from silpixa00401385.ir.intel.com ([10.237.214.14]) by fmsmga005.fm.intel.com with ESMTP; 15 Sep 2023 05:27:09 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , Anatoly Burakov Subject: [PATCH] eal: allow swapping of malloc heaps Date: Fri, 15 Sep 2023 13:27:03 +0100 Message-Id: <20230915122703.475834-1-bruce.richardson@intel.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The external memory functions in DPDK allow the addition of externally access memory to malloc heaps, but with one major restriction - the memory must be allocated to an application-created heap, not one of the standard DPDK heaps for a NUMA node. This restriction makes it difficult - if not impossible - to use externally allocated memory for DPDK by default. However, even if the restriction is relaxed, so we can add external memory to e.g. the socket 0 heap, there would be no way to guarantee that the external memory would be used in preference to the standard DPDK hugepage memory for a given allocation. To give appropriately defined behaviour, a better solution is to allow the application to explicitly swap a pair of heaps. With this one new API in place, it allows the user to configure a new malloc heap, add external memory to it, and then replace a standard socket heap with the newly created one - thereby guaranteeing future allocations from the external memory. Signed-off-by: Bruce Richardson --- lib/eal/common/malloc_heap.c | 18 ++++++++++++++++++ lib/eal/include/rte_malloc.h | 25 +++++++++++++++++++++++++ lib/eal/version.map | 2 ++ 3 files changed, 45 insertions(+) diff --git a/lib/eal/common/malloc_heap.c b/lib/eal/common/malloc_heap.c index 6b6cf9174c..e5d0ad6b52 100644 --- a/lib/eal/common/malloc_heap.c +++ b/lib/eal/common/malloc_heap.c @@ -1320,6 +1320,24 @@ malloc_heap_add_external_memory(struct malloc_heap *heap, return 0; } +int +rte_malloc_heap_swap_socket(int socket1, int socket2) +{ + const int h1 = malloc_socket_to_heap_id(socket1); + if (h1 < 0 || h1 > RTE_MAX_HEAPS) + return -1; + + const int h2 = malloc_socket_to_heap_id(socket2); + if (h2 < 0 || h2 > RTE_MAX_HEAPS) + return -1; + + struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config; + int tmp = mcfg->malloc_heaps[h1].socket_id; + mcfg->malloc_heaps[h1].socket_id = mcfg->malloc_heaps[h2].socket_id; + mcfg->malloc_heaps[h2].socket_id = tmp; + return 0; +} + int malloc_heap_remove_external_memory(struct malloc_heap *heap, void *va_addr, size_t len) diff --git a/lib/eal/include/rte_malloc.h b/lib/eal/include/rte_malloc.h index 54a8ac211e..fa70ca7430 100644 --- a/lib/eal/include/rte_malloc.h +++ b/lib/eal/include/rte_malloc.h @@ -490,6 +490,31 @@ rte_malloc_heap_get_socket(const char *name); int rte_malloc_heap_socket_is_external(int socket_id); +/** + * Swap the heaps for the given socket ids + * + * This causes the heaps for the given socket ids to be swapped, allowing + * external memory registered as a malloc heap to become the new default memory + * for a standard numa node. For example, to have allocations on socket 0 come + * from external memory, the following sequence of API calls can be used: + * - rte_malloc_heap_create() + * - rte_malloc_heap_memory_add(,....) + * - id = rte_malloc_heap_get_socket() + * - rte_malloc_heap_swap_sockets(0, id) + * Following these calls, allocations for the old memory allocated on socket 0, + * can be made by passing "id" as the socket_id parameter + * + * @param socket1 + * The socket id of the first heap to swap + * @param socket2 + * The socket id of the second heap to swap + * @return + * 0 on success, -1 on error + */ +__rte_experimental +int +rte_malloc_heap_swap_socket(int socket1, int socket2); + /** * Dump statistics. * diff --git a/lib/eal/version.map b/lib/eal/version.map index 7940431e5a..b06ee7219e 100644 --- a/lib/eal/version.map +++ b/lib/eal/version.map @@ -417,6 +417,8 @@ EXPERIMENTAL { # added in 23.07 rte_memzone_max_get; rte_memzone_max_set; + + rte_malloc_heap_swap_socket; }; INTERNAL { -- 2.39.2