From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F29F5A0A0C; Fri, 30 Jul 2021 15:56:45 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D9A9A410F6; Fri, 30 Jul 2021 15:56:44 +0200 (CEST) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mails.dpdk.org (Postfix) with ESMTP id 09545410F6 for ; Fri, 30 Jul 2021 15:56:43 +0200 (CEST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id B3E535C00C4; Fri, 30 Jul 2021 09:56:42 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Fri, 30 Jul 2021 09:56:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=IKiqbwgmKuSIp Q9974aAn+RBnCawfSH9+cCtgcvEvM0=; b=3NiDBVumJDBoR5SG+sUE0P3o0lSj+ 9rPjBmsvH88ZDYjqWQGBa821VTPnd/K2jgS33VANDcDDznziXMClB5ATDUcCYMZE SudhvZZvrfx8oSJXb06p55YIYTTaIfnwteCOQK60zcDOkpLi/MU2XmAEF8h1Y3Po vO1/zzAvZ+1JVDFXBmPYFtTq310Y7oJ+hYV/0xMeervrLOE38gPrVOVhjQS6iFUM BkUIszCvcB+g/Fmy143pA+0JEH4OCAGIiTV4IY2UL0oAaEtS7piau1f3jhvRcALK x3Q3fyu8pyhQ/tMGSDuE0m+5h356JMpJv3dVWClbnLmlYCNaB4T9xDyHQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=IKiqbwgmKuSIpQ9974aAn+RBnCawfSH9+cCtgcvEvM0=; b=bYfdTUe1 0x8t/i0aGUlg90IFvl3eoe4H3F66pOj8WbtwagjcHLrn8uuExOlKwyxb20mtI1Wr qHF7Pf+xC4MBD+YxXKYxJ8arnWa2feHmToRIAtjoQ4eSwgv50FFRdbaBTaqYrFMM CvInRTus5QYZbM3C6K1PZrC32dphEbx/QDESfVzID/EG5xI7awRlNAIQ7FOPxSKt kRmPBTbIqCNg7uWRWAbLZYaxn3osyB3RPylImI+hN3b7yw6QRid/kENOLMqypLSm K9PKrSGF2vFPJ5xEnHEf/F8fz1hSZY+E5MO8NsLFdVXPYaD+8wARUdlqU405KOoA k4b4nLD8cmDo5A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrheehgdeiiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucggtf frrghtthgvrhhnpedvhefgiedvjeegtdevheefhfetleefgfeivefgffevfeejgedtgfeu tdehtdegveenucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhroh hmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 30 Jul 2021 09:56:41 -0400 (EDT) From: Thomas Monjalon To: dev@dpdk.org Cc: Stephen Hemminger , David Marchand , Andrew Rybchenko , Haiyue Wang , Honnappa Nagarahalli , Jerin Jacob , Ferruh Yigit , Elena Agostini , Ray Kinsella Date: Fri, 30 Jul 2021 15:55:33 +0200 Message-Id: <20210730135533.417611-8-thomas@monjalon.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730135533.417611-1-thomas@monjalon.net> References: <20210602203531.2288645-1-thomas@monjalon.net> <20210730135533.417611-1-thomas@monjalon.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [RFC PATCH v2 7/7] hcdev: add communication list X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Elena Agostini In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. When mixing network activity with task processing there may be the need to put in communication the CPU with the device in order to synchronize operations. An example could be a receive-and-process application where CPU is responsible for receiving packets in multiple mbufs and the device is responsible for processing the content of those packets. The purpose of this list is to provide a buffer in CPU memory visible from the device that can be treated as a circular buffer to let the CPU provide fondamental info of received packets to the device. A possible use-case is described below. CPU: - Trigger some task on the device - in a loop: - receive a number of packets - provide packets info to the device Device: - Do some pre-processing - Wait to receive a new set of packet to be processed Layout of a communication list would be: ------- | 0 | => pkt_list | status | | #pkts | ------- | 1 | => pkt_list | status | | #pkts | ------- | 2 | => pkt_list | status | | #pkts | ------- | .... | => pkt_list ------- Signed-off-by: Elena Agostini --- lib/hcdev/hcdev.c | 127 ++++++++++++++++++++++++++++++++++++++++ lib/hcdev/meson.build | 2 + lib/hcdev/rte_hcdev.h | 132 ++++++++++++++++++++++++++++++++++++++++++ lib/hcdev/version.map | 4 ++ 4 files changed, 265 insertions(+) diff --git a/lib/hcdev/hcdev.c b/lib/hcdev/hcdev.c index e391988e73..572f1713fc 100644 --- a/lib/hcdev/hcdev.c +++ b/lib/hcdev/hcdev.c @@ -660,3 +660,130 @@ rte_hcdev_comm_get_flag_value(struct rte_hcdev_comm_flag *hcflag, uint32_t *val) return 0; } + +struct rte_hcdev_comm_list * +rte_hcdev_comm_create_list(uint16_t dev_id, + uint32_t num_comm_items) +{ + struct rte_hcdev_comm_list *comm_list; + uint32_t idx_l; + + if (num_comm_items == 0) { + rte_errno = EINVAL; + return NULL; + } + + comm_list = rte_hcdev_malloc(dev_id, + sizeof(struct rte_hcdev_comm_list) * num_comm_items, + RTE_HCDEV_MALLOC_REGISTER_FROM_CPU); + if (comm_list == NULL) { + rte_errno = ENOMEM; + return NULL; + } + + for (idx_l = 0; idx_l < num_comm_items; idx_l++) { + comm_list[idx_l].pkt_list = + rte_hcdev_malloc(dev_id, + sizeof(struct rte_hcdev_comm_pkt) * + RTE_HCDEV_COMM_LIST_PKTS_MAX, + RTE_HCDEV_MALLOC_REGISTER_FROM_CPU); + if (comm_list[idx_l].pkt_list == NULL) { + rte_errno = ENOMEM; + return NULL; + } + + RTE_HCDEV_VOLATILE(comm_list[idx_l].status) = + RTE_HCDEV_COMM_LIST_FREE; + comm_list[idx_l].num_pkts = 0; + } + + return comm_list; +} + +int +rte_hcdev_comm_destroy_list(uint16_t dev_id, + struct rte_hcdev_comm_list *comm_list, + uint32_t num_comm_items) +{ + uint32_t idx_l; + + if (comm_list == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + + for (idx_l = 0; idx_l < num_comm_items; idx_l++) + rte_hcdev_free(dev_id, comm_list[idx_l].pkt_list); + rte_hcdev_free(dev_id, comm_list); + + return 0; +} + +int +rte_hcdev_comm_populate_list_pkts(struct rte_hcdev_comm_list *comm_list_item, + struct rte_mbuf **mbufs, uint32_t num_mbufs) +{ + uint32_t idx; + + if (comm_list_item == NULL || comm_list_item->pkt_list == NULL || + mbufs == NULL || num_mbufs > RTE_HCDEV_COMM_LIST_PKTS_MAX) { + rte_errno = EINVAL; + return -rte_errno; + } + + for (idx = 0; idx < num_mbufs; idx++) { + /* support only unchained mbufs */ + if (unlikely((mbufs[idx]->nb_segs > 1) || + (mbufs[idx]->next != NULL) || + (mbufs[idx]->data_len != mbufs[idx]->pkt_len))) { + rte_errno = ENOTSUP; + return -rte_errno; + } + comm_list_item->pkt_list[idx].addr = + rte_pktmbuf_mtod_offset(mbufs[idx], uintptr_t, 0); + comm_list_item->pkt_list[idx].size = mbufs[idx]->pkt_len; + comm_list_item->pkt_list[idx].opaque = mbufs[idx]; + } + + RTE_HCDEV_VOLATILE(comm_list_item->num_pkts) = num_mbufs; + rte_mb(); + RTE_HCDEV_VOLATILE(comm_list_item->status) = RTE_HCDEV_COMM_LIST_READY; + + return 0; +} + +int +rte_hcdev_comm_cleanup_list(struct rte_hcdev_comm_list *comm_list_item) +{ + struct rte_mbuf *mbufs[RTE_HCDEV_COMM_LIST_PKTS_MAX]; + uint32_t idx = 0; + + if (comm_list_item == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + + if (RTE_HCDEV_VOLATILE(comm_list_item->status) == + RTE_HCDEV_COMM_LIST_READY) { + HCDEV_LOG(ERR, "packet list is still in progress"); + rte_errno = EINVAL; + return -rte_errno; + } + + for (idx = 0; idx < RTE_HCDEV_COMM_LIST_PKTS_MAX; idx++) { + if (comm_list_item->pkt_list[idx].addr == 0) + break; + + comm_list_item->pkt_list[idx].addr = 0; + comm_list_item->pkt_list[idx].size = 0; + mbufs[idx] = (struct rte_mbuf *) comm_list_item->pkt_list[idx].opaque; + } + + rte_pktmbuf_free_bulk(mbufs, idx); + + RTE_HCDEV_VOLATILE(comm_list_item->status) = RTE_HCDEV_COMM_LIST_FREE; + RTE_HCDEV_VOLATILE(comm_list_item->num_pkts) = 0; + rte_mb(); + + return 0; +} diff --git a/lib/hcdev/meson.build b/lib/hcdev/meson.build index 565c3cb623..249849e0cb 100644 --- a/lib/hcdev/meson.build +++ b/lib/hcdev/meson.build @@ -8,3 +8,5 @@ headers = files( sources = files( 'hcdev.c', ) + +deps += ['mbuf'] diff --git a/lib/hcdev/rte_hcdev.h b/lib/hcdev/rte_hcdev.h index 7b58041b3c..78f0e92957 100644 --- a/lib/hcdev/rte_hcdev.h +++ b/lib/hcdev/rte_hcdev.h @@ -9,6 +9,7 @@ #include #include +#include #include #include @@ -41,6 +42,9 @@ extern "C" { /** Access variable as volatile. */ #define RTE_HCDEV_VOLATILE(x) (*(volatile typeof(x)*)&(x)) +/** Max number of packets per communication list. */ +#define RTE_HCDEV_COMM_LIST_PKTS_MAX 1024 + /** Store device info. */ struct rte_hcdev_info { /** Unique identifier name. */ @@ -79,10 +83,47 @@ enum rte_hcdev_comm_flag_type { /** Communication flag to coordinate CPU with the device. */ struct rte_hcdev_comm_flag { + /** Pointer to flag memory area. */ uint32_t *ptr; + /** Type of memory used to allocate the flag. */ enum rte_hcdev_comm_flag_type mtype; }; +/** List of packets shared among CPU and device. */ +struct rte_hcdev_comm_pkt { + /** Address of the packet in memory (e.g. mbuf->buf_addr). */ + uintptr_t addr; + /** Size in byte of the packet. */ + size_t size; + /** Mbuf reference to release it in the rte_hcdev_comm_cleanup_list(). */ + void *opaque; +}; + +/** Possible status for the list of packets shared among CPU and device. */ +enum rte_hcdev_comm_list_status { + /** Packet list can be filled with new mbufs, no one is using it. */ + RTE_HCDEV_COMM_LIST_FREE = 0, + /** Packet list has been filled with new mbufs and it's ready to be used .*/ + RTE_HCDEV_COMM_LIST_READY, + /** Packet list has been processed, it's ready to be freed. */ + RTE_HCDEV_COMM_LIST_DONE, + /** Some error occurred during packet list processing. */ + RTE_HCDEV_COMM_LIST_ERROR, +}; + +/** + * Communication list holding a number of lists of packets + * each having a status flag. + */ +struct rte_hcdev_comm_list { + /** List of packets populated by the CPU with a set of mbufs info. */ + struct rte_hcdev_comm_pkt *pkt_list; + /** Number of packets in the list. */ + uint32_t num_pkts; + /** Status of the list. */ + enum rte_hcdev_comm_list_status status; +}; + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. @@ -453,6 +494,97 @@ __rte_experimental int rte_hcdev_comm_get_flag_value(struct rte_hcdev_comm_flag *hcflag, uint32_t *val); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Create a communication list that can be used to share packets + * between CPU and device. + * Each element of the list contains: + * - a packet list of RTE_HCDEV_COMM_LIST_PKTS_MAX elements + * - number of packets in the list + * - a status flag to communicate if the packet list is FREE, + * READY to be processed, DONE with processing. + * + * The list is allocated in CPU-visible memory. + * At creation time, every list is in FREE state. + * + * @param dev_id + * Reference device ID. + * @param num_comm_items + * Number of items in the communication list. + * + * @return + * A pointer to the allocated list, otherwise NULL and rte_errno is set: + * - EINVAL if invalid input params + */ +__rte_experimental +struct rte_hcdev_comm_list *rte_hcdev_comm_create_list(uint16_t dev_id, + uint32_t num_comm_items); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Destroy a communication list. + * + * @param dev_id + * Reference device ID. + * @param comm_list + * Communication list to be destroyed. + * @param num_comm_items + * Number of items in the communication list. + * + * @return + * 0 on success, -rte_errno otherwise: + * - EINVAL if invalid input params + */ +__rte_experimental +int rte_hcdev_comm_destroy_list(uint16_t dev_id, + struct rte_hcdev_comm_list *comm_list, + uint32_t num_comm_items); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Populate the packets list of the communication item + * with info from a list of mbufs. + * Status flag of that packet list is set to READY. + * + * @param comm_list_item + * Communication list item to fill. + * @param mbufs + * List of mbufs. + * @param num_mbufs + * Number of mbufs. + * + * @return + * 0 on success, -rte_errno otherwise: + * - EINVAL if invalid input params + * - ENOTSUP if mbufs are chained (multiple segments) + */ +__rte_experimental +int rte_hcdev_comm_populate_list_pkts(struct rte_hcdev_comm_list *comm_list_item, + struct rte_mbuf **mbufs, uint32_t num_mbufs); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Reset a communication list item to the original state. + * The status flag set to FREE and mbufs are returned to the pool. + * + * @param comm_list_item + * Communication list item to reset. + * + * @return + * 0 on success, -rte_errno otherwise: + * - EINVAL if invalid input params + */ +__rte_experimental +int rte_hcdev_comm_cleanup_list(struct rte_hcdev_comm_list *comm_list_item); + #ifdef __cplusplus } #endif diff --git a/lib/hcdev/version.map b/lib/hcdev/version.map index da969c7f1f..caa00af647 100644 --- a/lib/hcdev/version.map +++ b/lib/hcdev/version.map @@ -6,9 +6,13 @@ EXPERIMENTAL { rte_hcdev_callback_register; rte_hcdev_callback_unregister; rte_hcdev_close; + rte_hcdev_comm_cleanup_list; rte_hcdev_comm_create_flag; + rte_hcdev_comm_create_list; rte_hcdev_comm_destroy_flag; + rte_hcdev_comm_destroy_list; rte_hcdev_comm_get_flag_value; + rte_hcdev_comm_populate_list_pkts; rte_hcdev_comm_set_flag; rte_hcdev_count_avail; rte_hcdev_find_next; -- 2.31.1