From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8DDD2A0A0C; Fri, 30 Jul 2021 15:56:35 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 79949410DA; Fri, 30 Jul 2021 15:56:35 +0200 (CEST) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mails.dpdk.org (Postfix) with ESMTP id 3274E4003F for ; Fri, 30 Jul 2021 15:56:34 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id C3C715C00BA; Fri, 30 Jul 2021 09:56:33 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 30 Jul 2021 09:56:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=29DJE9mHhUC+S jDNT3ePoHHFPF1FCmJ6HKVlmD0t7jk=; b=hCIG0TTM3B3vCpi1fIXP0aSWkZwFU 8BkktWe4T/ZXxwIs4gNJCkkTqWgiBqJKrIZ/atY9hxwFNlf22W42mSxqJERSINAK KC6WHBFNZwoBykCG4QxjwcxYBTzru2jrG1WoYw2YVtrenFtEH1/m+T0fw/A28NI+ X2O+20y0zT3w9S9GKvPHcIScXfoqsSMPv53uoyGSOs79cMPp2ulwpVelKa11d6lS Owu9nB8W52Bu06vjQaiXfZ98UqRqSk2kZo9lT1gU+w33xmKCddYDtNwjynkPowop LXy9h1GvwPoxc5G9NRQm/a6HEfKt4zcAxdlP8uijyt/6ZG5f22jbBLS4Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=29DJE9mHhUC+SjDNT3ePoHHFPF1FCmJ6HKVlmD0t7jk=; b=gVOCrsC+ U9/91oIP9KUvFaGDuEJFcMUkuzrV0fFplU9DmVc/ho9KBn2qsKtoxszCtRdLRQHp NXjwSMtJJrcQmYbrvQ7VCur2jucUCTueNxwEiJ4RJH1IRy2RgukbjmZtMCNnBbeY VDoiMPv7GSx2iP1wwHm48HDtTF+wIDe1ZMfEONjJIyr+oUDdd1+XZIrmv/JPJ9Mw 5CPtksbSF8NGIbOuLe/xjALnbWdff4jMKYf2X3R5qdSJeZqv6PgN4Ji5RAXZdgip Z9VqJ4FzpzN5sD0xbiFU2b6WfnzVybuGS1OtLLLx065LwhkteSlrASkTxx9MYaPR XkNbJbxiKqzpzg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrheehgdeiiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucggtf frrghtthgvrhhnpedvhefgiedvjeegtdevheefhfetleefgfeivefgffevfeejgedtgfeu tdehtdegveenucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhroh hmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 30 Jul 2021 09:56:32 -0400 (EDT) From: Thomas Monjalon To: dev@dpdk.org Cc: Stephen Hemminger , David Marchand , Andrew Rybchenko , Haiyue Wang , Honnappa Nagarahalli , Jerin Jacob , Ferruh Yigit , Elena Agostini , Ray Kinsella Date: Fri, 30 Jul 2021 15:55:31 +0200 Message-Id: <20210730135533.417611-6-thomas@monjalon.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730135533.417611-1-thomas@monjalon.net> References: <20210602203531.2288645-1-thomas@monjalon.net> <20210730135533.417611-1-thomas@monjalon.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [RFC PATCH v2 5/7] hcdev: add memory API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Elena Agostini In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. Such workload distribution can be achieved by sharing some memory. As a first step, the features are focused on memory management. A function allows to allocate memory inside the device, or in the main (CPU) memory while making it visible for the device. This memory may be used to save packets or for synchronization data. The next step should focus on GPU processing task control. Signed-off-by: Elena Agostini Signed-off-by: Thomas Monjalon --- doc/guides/hcdevs/features/default.ini | 3 + doc/guides/rel_notes/release_21_08.rst | 1 + lib/hcdev/hcdev.c | 88 ++++++++++++++++++++++++++ lib/hcdev/hcdev_driver.h | 9 +++ lib/hcdev/rte_hcdev.h | 53 ++++++++++++++++ lib/hcdev/version.map | 2 + 6 files changed, 156 insertions(+) diff --git a/doc/guides/hcdevs/features/default.ini b/doc/guides/hcdevs/features/default.ini index f988ee73d4..ee32753d94 100644 --- a/doc/guides/hcdevs/features/default.ini +++ b/doc/guides/hcdevs/features/default.ini @@ -8,3 +8,6 @@ ; [Features] Get device info = +Share CPU memory with device = +Allocate device memory = +Free memory = diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst index fb350b4706..e955a331a6 100644 --- a/doc/guides/rel_notes/release_21_08.rst +++ b/doc/guides/rel_notes/release_21_08.rst @@ -58,6 +58,7 @@ New Features * **Introduced Heterogeneous Computing Device library with first features:** * Device information + * Memory management * **Added auxiliary bus support.** diff --git a/lib/hcdev/hcdev.c b/lib/hcdev/hcdev.c index a7badd122b..621e0b99bd 100644 --- a/lib/hcdev/hcdev.c +++ b/lib/hcdev/hcdev.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -501,3 +502,90 @@ rte_hcdev_info_get(int16_t dev_id, struct rte_hcdev_info *info) } return HCDEV_DRV_RET(dev->ops.dev_info_get(dev, info)); } + +#define RTE_HCDEV_MALLOC_FLAGS_ALL \ + RTE_HCDEV_MALLOC_REGISTER_FROM_CPU +#define RTE_HCDEV_MALLOC_FLAGS_RESERVED ~RTE_HCDEV_MALLOC_FLAGS_ALL + +void * +rte_hcdev_malloc(int16_t dev_id, size_t size, uint32_t flags) +{ + struct rte_hcdev *dev; + void *ptr; + int ret; + + dev = hcdev_get_by_id(dev_id); + if (dev == NULL) { + HCDEV_LOG(ERR, "alloc mem for invalid device ID %d", dev_id); + rte_errno = ENODEV; + return NULL; + } + if (flags & RTE_HCDEV_MALLOC_FLAGS_RESERVED) { + HCDEV_LOG(ERR, "alloc mem with reserved flag 0x%x", + flags & RTE_HCDEV_MALLOC_FLAGS_RESERVED); + rte_errno = EINVAL; + return NULL; + } + + if (flags & RTE_HCDEV_MALLOC_REGISTER_FROM_CPU) { + if (dev->ops.mem_register == NULL) { + HCDEV_LOG(ERR, "mem registration not supported"); + rte_errno = ENOTSUP; + return NULL; + } + } else { + if (dev->ops.mem_alloc == NULL) { + HCDEV_LOG(ERR, "mem allocation not supported"); + rte_errno = ENOTSUP; + return NULL; + } + } + + if (size == 0) /* dry-run */ + return NULL; + + if (flags & RTE_HCDEV_MALLOC_REGISTER_FROM_CPU) { + ptr = rte_zmalloc(NULL, size, 0); + if (ptr == NULL) { + HCDEV_LOG(ERR, "cannot allocate CPU memory"); + rte_errno = ENOMEM; + return NULL; + } + ret = dev->ops.mem_register(dev, size, ptr); + } else { + ret = dev->ops.mem_alloc(dev, size, &ptr); + } + /* TODO maintain a table of chunks registered/allocated */ + switch (ret) { + case 0: + return ptr; + case -ENOMEM: + case -E2BIG: + rte_errno = -ret; + return NULL; + default: + rte_errno = EPERM; + return NULL; + } +} + +int +rte_hcdev_free(int16_t dev_id, void *ptr) +{ + struct rte_hcdev *dev; + + dev = hcdev_get_by_id(dev_id); + if (dev == NULL) { + HCDEV_LOG(ERR, "free mem for invalid device ID %d", dev_id); + rte_errno = ENODEV; + return -rte_errno; + } + + if (dev->ops.mem_free == NULL) { + rte_errno = ENOTSUP; + return -rte_errno; + } + return HCDEV_DRV_RET(dev->ops.mem_free(dev, ptr)); + /* TODO unregister callback */ + /* TODO rte_free CPU memory */ +} diff --git a/lib/hcdev/hcdev_driver.h b/lib/hcdev/hcdev_driver.h index f33b56947b..f42f08508f 100644 --- a/lib/hcdev/hcdev_driver.h +++ b/lib/hcdev/hcdev_driver.h @@ -27,12 +27,21 @@ enum rte_hcdev_state { struct rte_hcdev; typedef int (rte_hcdev_close_t)(struct rte_hcdev *dev); typedef int (rte_hcdev_info_get_t)(struct rte_hcdev *dev, struct rte_hcdev_info *info); +typedef int (rte_hcdev_mem_alloc_t)(struct rte_hcdev *dev, size_t size, void **ptr); +typedef int (rte_hcdev_mem_register_t)(struct rte_hcdev *dev, size_t size, void *ptr); +typedef int (rte_hcdev_free_t)(struct rte_hcdev *dev, void *ptr); struct rte_hcdev_ops { /* Get device info. If NULL, info is just copied. */ rte_hcdev_info_get_t *dev_info_get; /* Close device or child context. */ rte_hcdev_close_t *dev_close; + /* Allocate memory in device. */ + rte_hcdev_mem_alloc_t *mem_alloc; + /* Register CPU memory in device. */ + rte_hcdev_mem_register_t *mem_register; + /* Free memory allocated or registered in device. */ + rte_hcdev_free_t *mem_free; }; struct rte_hcdev_mpshared { diff --git a/lib/hcdev/rte_hcdev.h b/lib/hcdev/rte_hcdev.h index c95f37063d..11895d9486 100644 --- a/lib/hcdev/rte_hcdev.h +++ b/lib/hcdev/rte_hcdev.h @@ -9,6 +9,7 @@ #include #include +#include #include /** @@ -293,6 +294,58 @@ int rte_hcdev_callback_unregister(int16_t dev_id, enum rte_hcdev_event event, __rte_experimental int rte_hcdev_info_get(int16_t dev_id, struct rte_hcdev_info *info); +/** Memory allocated on a CPU node and visible by the device. */ +#define RTE_HCDEV_MALLOC_REGISTER_FROM_CPU RTE_BIT32(0) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Allocate a chunk of memory usable by the device. + * + * @param dev_id + * Device ID requiring allocated memory. + * @param size + * Number of bytes to allocate. + * Requesting 0 will do nothing. + * @param flags + * If 0, the default is to allocate in the device memory. + * See flags RTE_HCDEV_MALLOC_* + * + * @return + * A pointer to the allocated memory, otherwise NULL and rte_errno is set: + * - ENODEV if invalid dev_id + * - EINVAL if reserved flags + * - ENOTSUP if operation not supported by the driver + * - E2BIG if size is higher than limit + * - ENOMEM if out of space + * - EPERM if driver error + */ +__rte_experimental +void *rte_hcdev_malloc(int16_t dev_id, size_t size, uint32_t flags) +__rte_alloc_size(2); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Deallocate a chunk of memory allocated with rte_hcdev_malloc(). + * + * @param dev_id + * Reference device ID. + * @param ptr + * Pointer to the memory area to be deallocated. + * NULL is a no-op accepted value. + * + * @return + * 0 on success, -rte_errno otherwise: + * - ENODEV if invalid dev_id + * - ENOTSUP if operation not supported by the driver + * - EPERM if driver error + */ +__rte_experimental +int rte_hcdev_free(int16_t dev_id, void *ptr); + #ifdef __cplusplus } #endif diff --git a/lib/hcdev/version.map b/lib/hcdev/version.map index 450c256527..9195f4f747 100644 --- a/lib/hcdev/version.map +++ b/lib/hcdev/version.map @@ -8,9 +8,11 @@ EXPERIMENTAL { rte_hcdev_close; rte_hcdev_count_avail; rte_hcdev_find_next; + rte_hcdev_free; rte_hcdev_info_get; rte_hcdev_init; rte_hcdev_is_valid; + rte_hcdev_malloc; }; INTERNAL { -- 2.31.1