From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0488DA0C42; Fri, 18 Jun 2021 07:52:58 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7AB9140150; Fri, 18 Jun 2021 07:52:58 +0200 (CEST) Received: from mail-il1-f171.google.com (mail-il1-f171.google.com [209.85.166.171]) by mails.dpdk.org (Postfix) with ESMTP id 25C4240142 for ; Fri, 18 Jun 2021 07:52:57 +0200 (CEST) Received: by mail-il1-f171.google.com with SMTP id p14so895946ilg.8 for ; Thu, 17 Jun 2021 22:52:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=jJ8BPd8HTwr8GDG62lLXSjX2qzxOy19W3HV09au4X9c=; b=rH2gYnjk13b6KL3zJvBuI+iM+tHLupsZOJmtqEyH3hLKwX34sxfzUppsiIDQMrdRmE 2TLcas6cS2kqWt4ISZg7rqiJYn6oS6Ze7X/G/pUyiij1bdcvcoxm3+YGPoycDjK9AASu ZNgjgjSiMJEcz9rOv16KbAlQvsTacyp3PVAUeMCaN4cQQNNg1SowwrWaWr7HpcZRCoEB QedMFjixnMli6hnudN/OCZ72VMtkB2lj9W/9xSfH6w4Xw5qWxl39b0twk9CfIxIk51sf TmeEwF3I3eLVYXM8Kk9Eu2mlEkZyVao7+p+2eAUrPCJ1JMBpLDUHFrXmjbzi6Wm3o/Id SSAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=jJ8BPd8HTwr8GDG62lLXSjX2qzxOy19W3HV09au4X9c=; b=ChHGy9+Y89eKRJyBkTHxiYKiCilPCfN3LVWuI/TU++tdUexZ2VluC2+E77J0j9v0rP tj+O9y+PaXEEu7REywUaXNeJsE4/T/MFgt/sGgRdtcTX3TBTROmTWBktGLZbOVHc52NE pzPNCLaISyfyJD3v8vr+IYwdubOZ8/m2LzjBkW4/V8Beu9yWKqUeW3aYK5n2E2lCFYj0 nXpSiiU9iA513R334gIFk5w7j/Hxf+b21HKWAnMtoTZ8ltwbCgRFdCz7tV5DFNjA0lgU e7PivAdS3ozDLfO/bwQ87zwQu+98Iqm1rRb2/T/b5YLWCdl6AhQgmcreczDVhQq28oEa N/qg== X-Gm-Message-State: AOAM5305SCTyCpXBWswkNixvFmJYhyaOqumARuN91lKWrC+bRt0XYPrp AP1gDuLwSRCO6OTnGDxCuFRMUXr/mB/xH6JTW5c= X-Google-Smtp-Source: ABdhPJzabbiAerumXC1RuWJC3rcKnbRkdLVBObu5S4fXFdAlRw4+N+Rud/p0lBZYN5zO4uMM20fmZxvyHCGwI7vMgF4= X-Received: by 2002:a05:6e02:1a6a:: with SMTP id w10mr6390810ilv.130.1623995575287; Thu, 17 Jun 2021 22:52:55 -0700 (PDT) MIME-Version: 1.0 References: <1623763327-30987-1-git-send-email-fengchengwen@huawei.com> <98CBD80474FA8B44BF855DF32C47DC35C61860@smartserver.smartshare.dk> In-Reply-To: From: Jerin Jacob Date: Fri, 18 Jun 2021 11:22:28 +0530 Message-ID: To: Bruce Richardson Cc: fengchengwen , =?UTF-8?Q?Morten_Br=C3=B8rup?= , Thomas Monjalon , Ferruh Yigit , dpdk-dev , Nipun Gupta , Hemant Agrawal , Maxime Coquelin , Honnappa Nagarahalli , Jerin Jacob , David Marchand , Satananda Burla , Prasun Kapoor Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, Jun 17, 2021 at 2:46 PM Bruce Richardson wrote: > > On Wed, Jun 16, 2021 at 08:07:26PM +0530, Jerin Jacob wrote: > > On Wed, Jun 16, 2021 at 3:47 PM fengchengwen = wrote: > > > > > > On 2021/6/16 15:09, Morten Br=C3=B8rup wrote: > > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richard= son > > > >> Sent: Tuesday, 15 June 2021 18.39 > > > >> > > > >> On Tue, Jun 15, 2021 at 09:22:07PM +0800, Chengwen Feng wrote: > > > >>> This patch introduces 'dmadevice' which is a generic type of DMA > > > >>> device. > > > >>> > > > >>> The APIs of dmadev library exposes some generic operations which = can > > > >>> enable configuration and I/O with the DMA devices. > > > >>> > > > >>> Signed-off-by: Chengwen Feng > > > >>> --- > > > >> Thanks for sending this. > > > >> > > > >> Of most interest to me right now are the key data-plane APIs. Whil= e we > > > >> are > > > >> still in the prototyping phase, below is a draft of what we are > > > >> thinking > > > >> for the key enqueue/perform_ops/completed_ops APIs. > > > >> > > > >> Some key differences I note in below vs your original RFC: > > > >> * Use of void pointers rather than iova addresses. While using iov= a's > > > >> makes > > > >> sense in the general case when using hardware, in that it can wo= rk > > > >> with > > > >> both physical addresses and virtual addresses, if we change the = APIs > > > >> to use > > > >> void pointers instead it will still work for DPDK in VA mode, wh= ile > > > >> at the > > > >> same time allow use of software fallbacks in error cases, and al= so a > > > >> stub > > > >> driver than uses memcpy in the background. Finally, using iova's > > > >> makes the > > > >> APIs a lot more awkward to use with anything but mbufs or simila= r > > > >> buffers > > > >> where we already have a pre-computed physical address. > > > >> * Use of id values rather than user-provided handles. Allowing the > > > >> user/app > > > >> to manage the amount of data stored per operation is a better > > > >> solution, I > > > >> feel than proscribing a certain about of in-driver tracking. Som= e > > > >> apps may > > > >> not care about anything other than a job being completed, while = other > > > >> apps > > > >> may have significant metadata to be tracked. Taking the user-con= text > > > >> handles out of the API also makes the driver code simpler. > > > >> * I've kept a single combined API for completions, which differs f= rom > > > >> the > > > >> separate error handling completion API you propose. I need to gi= ve > > > >> the > > > >> two function approach a bit of thought, but likely both could wo= rk. > > > >> If we > > > >> (likely) never expect failed ops, then the specifics of error > > > >> handling > > > >> should not matter that much. > > > >> > > > >> For the rest, the control / setup APIs are likely to be rather > > > >> uncontroversial, I suspect. However, I think that rather than xsta= ts > > > >> APIs, > > > >> the library should first provide a set of standardized stats like > > > >> ethdev > > > >> does. If driver-specific stats are needed, we can add xstats later= to > > > >> the > > > >> API. > > > >> > > > >> Appreciate your further thoughts on this, thanks. > > > >> > > > >> Regards, > > > >> /Bruce > > > > > > > > I generally agree with Bruce's points above. > > > > > > > > I would like to share a couple of ideas for further discussion: > > > > > > I believe some of the other requirements and comments for generic DMA w= ill be > > > > 1) Support for the _channel_, Each channel may have different > > capabilities and functionalities. > > Typical cases are, each channel have separate source and destination > > devices like > > DMA between PCIe EP to Host memory, Host memory to Host memory, PCIe > > EP to PCIe EP. > > So we need some notion of the channel in the specification. > > > > Can you share a bit more detail on what constitutes a channel in this cas= e? > Is it equivalent to a device queue (which we are flattening to individual > devices in this API), or to a specific configuration on a queue? It not a queue. It is one of the attributes for transfer. I.e in the same queue, for a given transfer it can specify the different "source" and "destination" device. Like CPU to Sound card, CPU to network card etc. > > > 2) I assume current data plane APIs are not thread-safe. Is it right? > > > Yes. > > > > > 3) Cookie scheme outlined earlier looks good to me. Instead of having > > generic dequeue() API > > > > 4) Can split the rte_dmadev_enqueue_copy(uint16_t dev_id, void * src, > > void * dst, unsigned int length); > > to two stage API like, Where one will be used in fastpath and other > > one will use used in slowpath. > > > > - slowpath API will for take channel and take other attributes for tran= sfer > > > > Example syantx will be: > > > > struct rte_dmadev_desc { > > channel id; > > ops ; // copy, xor, fill etc > > other arguments specific to dma transfer // it can be set > > based on capability. > > > > }; > > > > rte_dmadev_desc_t rte_dmadev_preprare(uint16_t dev_id, struct > > rte_dmadev_desc *dec); > > > > - Fastpath takes arguments that need to change per transfer along with > > slow-path handle. > > > > rte_dmadev_enqueue(uint16_t dev_id, void * src, void * dst, unsigned > > int length, rte_dmadev_desc_t desc) > > > > This will help to driver to > > -Former API form the device-specific descriptors in slow path for a > > given channel and fixed attributes per transfer > > -Later API blend "variable" arguments such as src, dest address with > > slow-path created descriptors > > > > This seems like an API for a context-aware device, where the channel is t= he > config data/context that is preserved across operations - is that correct= ? > At least from the Intel DMA accelerators side, we have no concept of this > context, and each operation is completely self-described. The location or > type of memory for copies is irrelevant, you just pass the src/dst > addresses to reference. it is not context-aware device. Each HW JOB is self-described. You can view it different attributes of transfer. > > > The above will give better performance and is the best trade-off c > > between performance and per transfer variables. > > We may need to have different APIs for context-aware and context-unaware > processing, with which to use determined by the capabilities discovery. > Given that for these DMA devices the offload cost is critical, more so th= an > any other dev class I've looked at before, I'd like to avoid having APIs > with extra parameters than need to be passed about since that just adds > extra CPU cycles to the offload. If driver does not support additional attributes and/or the application does not need it, rte_dmadev_desc_t can be NULL. So that it won't have any cost in the datapath. I think, we can go to different API cases if we can not abstract problems without performance impact. Otherwise, it will be too much pain for applications. Just to understand, I think, we need to HW capabilities and how to have a common API. I assume HW will have some HW JOB descriptors which will be filled in SW and submitted to HW. In our HW, Job descriptor has the following main elements - Channel // We don't expect the application to change per transfer - Source address - It can be scatter-gather too - Will be changed per trans= fer - Destination address - It can be scatter-gather too - Will be changed per transfer - Transfer Length - - It can be scatter-gather too - Will be changed per transfer - IOVA address where HW post Job completion status PER Job descriptor - Will be changed per transfer - Another sideband information related to channel // We don't expect the application to change per transfer - As an option, Job completion can be posted as an event to rte_event_queue too // We don't expect the application to change per transfer @Richardson, Bruce @fengchengwen @Hemant Agrawal Could you share the options for your HW descriptors which you are planning to expose through API like above so that we can easily converge on fastpath API > > /Bruce