From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 05E6FA0547; Thu, 9 Sep 2021 12:33:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 74E5D40041; Thu, 9 Sep 2021 12:33:09 +0200 (CEST) Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com [66.111.4.221]) by mails.dpdk.org (Postfix) with ESMTP id 9B1CB4003E for ; Thu, 9 Sep 2021 12:33:07 +0200 (CEST) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailnew.nyi.internal (Postfix) with ESMTP id E881D580ABE; Thu, 9 Sep 2021 06:33:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 09 Sep 2021 06:33:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=fm2; bh= uejsBqPerd+zBx2qVONogvL1uaqTCYwy0DMRRtoTZhE=; b=aEM2BT+0bN0g75fQ xkObRag1PhqJ0SCTZFPT1XKAgWXa+sxZdstMK7cDlCzltI5wPJHZmXLsnD818BAp EuOcB62fX3FbTOsnl9SRDZknu60ZIW/LvLQmx+2ogILuMASNloE/PcQVIV2H7ZcO XNBZLl9mIsDG1CcjLseu2tLtTZpHK+RZFu+YQRjzvf+bcpL7+0pJEj1zswu6z23P vqlhPm0H0dcTQHV9SkdU84aEyrlns6NznY0s4i5NFEt5R5rCdzOwzf0VVNcjTYjN QqVmAT+OGQ9lJThZfU4EIXoIdtXi09aWHFrWqAA3yqlXZKHaB8Cv0ci6XB/+ftyd RYwHRg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=uejsBqPerd+zBx2qVONogvL1uaqTCYwy0DMRRtoTZ hE=; b=qs4xCMKwf0aQ+8FDBxKg1cNz4nKV9dmdqnZafuq1YWTkB6iYIc871YmbL zBqk/lPI7K8f2d17hIbDmoAcxqzdvxTsKA4ooVmQXZXNj8uaROsgfMOCJUsy/JD+ 8gL2WmADMqbhIaRXbsaFje3m8Xdd2zJSHXQN0DlldcnfKj3pz7QZ5ct2523mpKdY 6Xai7PGZ2YzA0j+tkafe91jLlUoO6tI36nLD5hklcHH9m3R3bsCCbxjtnRUvQtLq 6OrzcovYCt+f+njee/jg1RSzt2NeZqZzVhI2lsSicRkTWeGItMFZtxmDekI1pp6F HnLm22R1sRWeGK1J0ROvb4H7Rg7xQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrudefledgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtufertddttddvnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecugg ftrfgrthhtvghrnhepffdvffejueetleefieeludduuefgteejleevfeekjeefieegheet ffdvkeefgedunecuffhomhgrihhnpeguphgukhdrohhrghenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghlohhn rdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 9 Sep 2021 06:33:03 -0400 (EDT) From: Thomas Monjalon To: bruce.richardson@intel.com, Chengwen Feng Cc: ferruh.yigit@intel.com, jerinj@marvell.com, jerinjacobk@gmail.com, andrew.rybchenko@oktetlabs.ru, dev@dpdk.org, mb@smartsharesystems.com, nipun.gupta@nxp.com, hemant.agrawal@nxp.com, maxime.coquelin@redhat.com, honnappa.nagarahalli@arm.com, david.marchand@redhat.com, sburla@marvell.com, pkapoor@marvell.com, konstantin.ananyev@intel.com, conor.walsh@intel.com, kevin.laatz@intel.com Date: Thu, 09 Sep 2021 12:33:00 +0200 Message-ID: <1855788.BnNMhiXu0z@thomas> In-Reply-To: <20210907125649.49794-2-fengchengwen@huawei.com> References: <1625231891-2963-1-git-send-email-fengchengwen@huawei.com> <20210907125649.49794-1-fengchengwen@huawei.com> <20210907125649.49794-2-fengchengwen@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-dev] [PATCH v21 1/7] dmadev: introduce DMA device library public APIs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, I am having a surface look at the API. I hope we can do better than previous libs. 07/09/2021 14:56, Chengwen Feng: > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -496,6 +496,10 @@ F: drivers/raw/skeleton/ > F: app/test/test_rawdev.c > F: doc/guides/prog_guide/rawdev.rst > > +DMA device API - EXPERIMENTAL > +M: Chengwen Feng > +F: lib/dmadev/ I think it should before (preferably) or after eventdev, but let's keep rawdev as the last one. Then please apply the same order in other non-alphabetical lists (doc, meson, etc). > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2021 HiSilicon Limited. > + * Copyright(c) 2021 Intel Corporation. > + * Copyright(c) 2021 Marvell International Ltd. > + * Copyright(c) 2021 SmartShare Systems. > + */ No need for final dot in copyright lines. > +#ifndef _RTE_DMADEV_H_ > +#define _RTE_DMADEV_H_ No need for surrounding underscores. > + > +/** > + * @file rte_dmadev.h > + * > + * RTE DMA (Direct Memory Access) device APIs. RTE has no meaning when used in a sentence. And given it is a DPDK library, you don't really need to specify. I would also remove the final "s" as the library is one interface. > + * > + * The DMA framework is built on the following model: > + * > + * --------------- --------------- --------------- > + * | virtual DMA | | virtual DMA | | virtual DMA | > + * | channel | | channel | | channel | > + * --------------- --------------- --------------- > + * | | | > + * ------------------ | > + * | | > + * ------------ ------------ > + * | dmadev | | dmadev | > + * ------------ ------------ > + * | | > + * ------------------ ------------------ > + * | HW-DMA-channel | | HW-DMA-channel | > + * ------------------ ------------------ > + * | | > + * -------------------------------- > + * | > + * --------------------- > + * | HW-DMA-Controller | > + * --------------------- You don't hyphens between the words I think. > + * > + * The DMA controller could have multiple HW-DMA-channels (aka. HW-DMA-queues), > + * each HW-DMA-channel should be represented by a dmadev. > + * > + * The dmadev could create multiple virtual DMA channels, each virtual DMA > + * channel represents a different transfer context. The DMA operation request > + * must be submitted to the virtual DMA channel. e.g. Application could create > + * virtual DMA channel 0 for memory-to-memory transfer scenario, and create > + * virtual DMA channel 1 for memory-to-device transfer scenario. What is the benefit of virtual channels compared to have separate dmadevs for the same HW channel? > + * > + * The dmadev are dynamically allocated by rte_dmadev_pmd_allocate() during the > + * PCI/SoC device probing phase performed at EAL initialization time. And could Not clear what is this phase. Do you mean bus probing? > + * be released by rte_dmadev_pmd_release() during the PCI/SoC device removing > + * phase. Again what is this phase? I think freeing should be done only via the "close" function. > + * > + * This framework uses 'uint16_t dev_id' as the device identifier of a dmadev, > + * and 'uint16_t vchan' as the virtual DMA channel identifier in one dmadev. I think it is better to use signed int16_t so you can express "none" in the API, which can simplify some functions and error management. > + * > + * The functions exported by the dmadev API to setup a device designated by its > + * device identifier must be invoked in the following order: > + * - rte_dmadev_configure() > + * - rte_dmadev_vchan_setup() > + * - rte_dmadev_start() > + * > + * Then, the application can invoke dataplane APIs to process jobs. You mean "dataplane functions". > + * > + * If the application wants to change the configuration (i.e. invoke > + * rte_dmadev_configure() or rte_dmadev_vchan_setup()), it must invoke > + * rte_dmadev_stop() first to stop the device and then do the reconfiguration > + * before invoking rte_dmadev_start() again. The dataplane APIs should not be again, APIs -> functions > + * invoked when the device is stopped. > + * > + * Finally, an application can close a dmadev by invoking the > + * rte_dmadev_close() function. > + * > + * The dataplane APIs include two parts: > + * The first part is the submission of operation requests: > + * - rte_dmadev_copy() > + * - rte_dmadev_copy_sg() > + * - rte_dmadev_fill() > + * - rte_dmadev_submit() > + * > + * These APIs could work with different virtual DMA channels which have > + * different contexts. > + * > + * The first three APIs are used to submit the operation request to the virtual > + * DMA channel, if the submission is successful, an uint16_t ring_idx is > + * returned, otherwise a negative number is returned. unsigned or negative? looks weird. > + * > + * The last API was used to issue doorbell to hardware, and also there are flags > + * (@see RTE_DMA_OP_FLAG_SUBMIT) parameter of the first three APIs could do the > + * same work. I don't understand this sentence. You mean rte_dmadev_submit function? Why past tense "was"? Why having a redundant function? > + * > + * The second part is to obtain the result of requests: > + * - rte_dmadev_completed() > + * - return the number of operation requests completed successfully. > + * - rte_dmadev_completed_status() > + * - return the number of operation requests completed. > + * > + * @note The two completed APIs also support return the last completed > + * operation's ring_idx. Not sure why this note here. > + * @note If the dmadev works in silent mode (@see RTE_DMADEV_CAPA_SILENT), > + * application does not invoke the above two completed APIs. > + * > + * About the ring_idx which enqueue APIs (e.g. rte_dmadev_copy() > + * rte_dmadev_fill()) returned, the rules are as follows: I feel a word is missing above. > + * - ring_idx for each virtual DMA channel are independent. > + * - For a virtual DMA channel, the ring_idx is monotonically incremented, > + * when it reach UINT16_MAX, it wraps back to zero. > + * - This ring_idx can be used by applications to track per-operation > + * metadata in an application-defined circular ring. > + * - The initial ring_idx of a virtual DMA channel is zero, after the > + * device is stopped, the ring_idx needs to be reset to zero. > + * > + * One example: > + * - step-1: start one dmadev > + * - step-2: enqueue a copy operation, the ring_idx return is 0 > + * - step-3: enqueue a copy operation again, the ring_idx return is 1 > + * - ... > + * - step-101: stop the dmadev > + * - step-102: start the dmadev > + * - step-103: enqueue a copy operation, the ring_idx return is 0 > + * - ... > + * - step-x+0: enqueue a fill operation, the ring_idx return is 65535 > + * - step-x+1: enqueue a copy operation, the ring_idx return is 0 > + * - ... > + * > + * The DMA operation address used in enqueue APIs (i.e. rte_dmadev_copy(), > + * rte_dmadev_copy_sg(), rte_dmadev_fill()) defined as rte_iova_t type. The "is" defined > + * dmadev supports two types of address: memory address and device address. Please try to start new sentences on a new line. > + * > + * - memory address: the source and destination address of the memory-to-memory > + * transfer type, or the source address of the memory-to-device transfer type, > + * or the destination address of the device-to-memory transfer type. > + * @note If the device support SVA (@see RTE_DMADEV_CAPA_SVA), the memory > + * address can be any VA address, otherwise it must be an IOVA address. > + * > + * - device address: the source and destination address of the device-to-device > + * transfer type, or the source address of the device-to-memory transfer type, > + * or the destination address of the memory-to-device transfer type. > + * > + * By default, all the functions of the dmadev API exported by a PMD are What do you mean "by default"? Is there some exceptions? > + * lock-free functions which assume to not be invoked in parallel on different > + * logical cores to work on the same target dmadev object. > + * @note Different virtual DMA channels on the same dmadev *DO NOT* support > + * parallel invocation because these virtual DMA channels share the same > + * HW-DMA-channel. > + * > + */ [...] > +#define RTE_DMADEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN Why not using RTE_DEV_NAME_MAX_LEN directly? If you keep it, it should be commented, explaining whether it takes '\0' into account or not. > +__rte_experimental > +bool > +rte_dmadev_is_valid_dev(uint16_t dev_id); I would suggest dropping the final "_dev" in the function name. > +uint16_t > +rte_dmadev_count(void); It would be safer to name it rte_dmadev_count_avail in case we need new kind of device count later. > + > +/* Enumerates DMA device capabilities. */ You should group them with a doxygen group syntax. See https://patches.dpdk.org/project/dpdk/patch/20210830104232.598703-1-thomas@monjalon.net/ > +#define RTE_DMADEV_CAPA_MEM_TO_MEM (1ull << 0) Please use RTE_BIT macro (32 or 64). > +/**< DMA device support memory-to-memory transfer. > + * > + * @see struct rte_dmadev_info::dev_capa > + */ It is preferred to have documentation before the item. [...] > +/** > + * A structure used to retrieve the information of a DMA device. > + */ > +struct rte_dmadev_info { > + struct rte_device *device; /**< Generic Device information. */ Please do not expose this. > + uint64_t dev_capa; /**< Device capabilities (RTE_DMADEV_CAPA_*). */ > + uint16_t max_vchans; > + /**< Maximum number of virtual DMA channels supported. */ > + uint16_t max_desc; > + /**< Maximum allowed number of virtual DMA channel descriptors. */ > + uint16_t min_desc; > + /**< Minimum allowed number of virtual DMA channel descriptors. */ > + uint16_t max_sges; > + /**< Maximum number of source or destination scatter-gather entry > + * supported. > + * If the device does not support COPY_SG capability, this value can be > + * zero. > + * If the device supports COPY_SG capability, then rte_dmadev_copy_sg() > + * parameter nb_src/nb_dst should not exceed this value. > + */ > + uint16_t nb_vchans; /**< Number of virtual DMA channel configured. */ What about adding NUMA node? /* Local NUMA memory ID. -1 if unknown. */ int16_t numa_node; > +}; > +int > +rte_dmadev_info_get(uint16_t dev_id, struct rte_dmadev_info *dev_info); In .h files, the return type should not be on a separate line. > + > +/** > + * A structure used to configure a DMA device. > + */ You should mention where it is used with @see. > +struct rte_dmadev_conf { > + uint16_t nb_vchans; > + /**< The number of virtual DMA channels to set up for the DMA device. > + * This value cannot be greater than the field 'max_vchans' of struct > + * rte_dmadev_info which get from rte_dmadev_info_get(). > + */ > + bool enable_silent; > + /**< Indicates whether to enable silent mode. > + * false-default mode, true-silent mode. > + * This value can be set to true only when the SILENT capability is > + * supported. > + * > + * @see RTE_DMADEV_CAPA_SILENT > + */ > +}; [...] > +#define RTE_DMADEV_ALL_VCHAN 0xFFFFu Please do not add this kind of constant without a doxygen comment. It seems you don't manage the maximum number of devices. It is fixed in the .c file: struct rte_dmadev rte_dmadevices[RTE_DMADEV_MAX_DEVS]; Instead I would suggest a more dynamic approach with an init function, so the application can extend it before calling rte_eal_init. Please see how it is implemented here: https://patches.dpdk.org/project/dpdk/patch/20210730135533.417611-2-thomas@monjalon.net/ Above patch could also inspire you to start some docs in this patch. This series add one file per patch. Instead it would be better to have groups of features per patch, meaning the implementation and the driver interface should be in the same patch. You can split like this: 1/ device allocation 2/ configuration and start/stop 3/ dataplane functions I would suggest 2 more patches: 4/ event notification see https://patches.dpdk.org/project/dpdk/patch/20210730135533.417611-3-thomas@monjalon.net/ 5/ multi-process see https://patches.dpdk.org/project/dpdk/patch/20210730135533.417611-5-thomas@monjalon.net/ Thanks for the work