From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CDC60A0C41; Wed, 23 Jun 2021 11:37:56 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ADC3A4003F; Wed, 23 Jun 2021 11:37:56 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 5CE354003E for ; Wed, 23 Jun 2021 11:37:55 +0200 (CEST) IronPort-SDR: D+xPTlaSl8/NC4EJyuae4vKc76HyFDntIDSXR1r47H2etPF56dtIpMBrHzM0xVqORh4pO5ozPD Hwy82wm5/wYw== X-IronPort-AV: E=McAfee;i="6200,9189,10023"; a="186917151" X-IronPort-AV: E=Sophos;i="5.83,293,1616482800"; d="scan'208";a="186917151" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jun 2021 02:37:54 -0700 IronPort-SDR: QIXGdwwPhiAHKV8wPdMnLszdpUEKPzQ7iycQv76L5mgXGMCmGXknyTwor7OJNfKGhg/8bwPlS0 tMTIuw/nMKAQ== X-IronPort-AV: E=Sophos;i="5.83,293,1616482800"; d="scan'208";a="452971707" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.13.79]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 23 Jun 2021 02:37:50 -0700 Date: Wed, 23 Jun 2021 10:37:45 +0100 From: Bruce Richardson To: Jerin Jacob Cc: fengchengwen , Morten =?iso-8859-1?Q?Br=F8rup?= , Thomas Monjalon , Ferruh Yigit , dpdk-dev , Nipun Gupta , Hemant Agrawal , Maxime Coquelin , Honnappa Nagarahalli , Jerin Jacob , David Marchand , Satananda Burla , Prasun Kapoor Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35C61860@smartserver.smartshare.dk> <3cb0bd01-2b0d-cf96-d173-920947466041@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wed, Jun 23, 2021 at 12:51:07PM +0530, Jerin Jacob wrote: > On Wed, Jun 23, 2021 at 9:00 AM fengchengwen wrote: > > > > > >>> > > >>>> > > >>>>> The above will give better performance and is the best trade-off c > > >>>>> between performance and per transfer variables. > > >>>> > > >>>> We may need to have different APIs for context-aware and context-unaware > > >>>> processing, with which to use determined by the capabilities discovery. > > >>>> Given that for these DMA devices the offload cost is critical, more so than > > >>>> any other dev class I've looked at before, I'd like to avoid having APIs > > >>>> with extra parameters than need to be passed about since that just adds > > >>>> extra CPU cycles to the offload. > > >>> > > >>> If driver does not support additional attributes and/or the > > >>> application does not need it, rte_dmadev_desc_t can be NULL. > > >>> So that it won't have any cost in the datapath. I think, we can go to > > >>> different API > > >>> cases if we can not abstract problems without performance impact. > > >>> Otherwise, it will be too much > > >>> pain for applications. > > >> > > >> Yes, currently we plan to use different API for different case, e.g. > > >> rte_dmadev_memcpy() -- deal with local to local memcopy > > >> rte_dmadev_memset() -- deal with fill with local memory with pattern > > >> maybe: > > >> rte_dmadev_imm_data() --deal with copy very little data > > >> rte_dmadev_p2pcopy() --deal with peer-to-peer copy of diffenet PCIE addr > > >> > > >> These API capabilities will be reflected in the device capability set so that > > >> application could know by standard API. > > > > > > > > > There will be a lot of combination of that it will be like M x N cross > > > base case, It won't scale. > > > > Currently, it is hard to define generic dma descriptor, I think the well-defined > > APIs is feasible. > > I would like to understand why not feasible? if we move the > preparation to the slow path. > > i.e > > struct rte_dmadev_desc defines all the "attributes" of all DMA devices available > using capability. I believe with the scheme, we can scale and > incorporate all features of > all DMA HW without any performance impact. > > something like: > > struct rte_dmadev_desc { > /* Attributes all DMA transfer available for all HW under capability. */ > channel or port; > ops ; // copy, fill etc.. > /* impemention opqueue memory as zero length array, > rte_dmadev_desc_prep() update this memory with HW specific information > */ > uint8_t impl_opq[]; > } > > // allocate the memory for dma decriptor > struct rte_dmadev_desc *rte_dmadev_desc_alloc(devid); > // Convert DPDK specific descriptors to HW specific descriptors in slowpath */ > rte_dmadev_desc_prep(devid, struct rte_dmadev_desc *desc); > // Free dma descriptor memory > rte_dmadev_desc_free(devid, struct rte_dmadev_desc *desc ) > > The above calls in slow path. > > Only below call in fastpath. > // Here desc can be NULL(in case you don't need any specific attribute > attached to transfer, if needed, it can be an object which is gone > through rte_dmadev_desc_prep()) > rte_dmadev_enq(devid, struct rte_dmadev_desc *desc, void *src, void > *dest, unsigned int len, cookie) > The trouble here is the performance penalty due to building up and tearing down structures and passing those structures into functions via function pointer. With the APIs for enqueue/dequeue that have been discussed here, all parameters will be passed in registers, and then each driver can do a write of the actual hardware descriptor straight to cache/memory from registers. With the scheme you propose above, the register contains a pointer to the data which must then be loaded into the CPU before being written out again. This increases our offload cost. However, assuming that the desc_prep call is just for slowpath or initialization time, I'd be ok to have the functions take an extra hw-specific parameter for each call prepared with tx_prep. It would still allow all other parameters to be passed in registers. How much data are you looking to store in this desc struct? It can't all be represented as flags, for example? As for the individual APIs, we could do a generic "enqueue" API, which takes the op as a parameter, I prefer having each operation as a separate function, in order to increase the readability of the code and to reduce the number of parameters needed per function i.e. thereby saving registers needing to be used and potentially making the function calls and offload cost cheaper. Perhaps we can have the "common" ops such as copy, fill, have their own functions, and have a generic "enqueue" function for the less-commonly used or supported ops? /Bruce