From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CB7E8A0C42; Fri, 18 Jun 2021 07:16:37 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 544DF40150; Fri, 18 Jun 2021 07:16:37 +0200 (CEST) Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) by mails.dpdk.org (Postfix) with ESMTP id A945240142 for ; Fri, 18 Jun 2021 07:16:35 +0200 (CEST) Received: by mail-il1-f173.google.com with SMTP id h3so7424620ilc.9 for ; Thu, 17 Jun 2021 22:16:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JvJP2Tgt4pX2ppH9EzhVhChGVgkGEIkwYadFsno5MHg=; b=Le0iJyB7pcLo/t4sQxMyLQmAdhkMND0aKQJUEcQVewxSDXFEuYgMWXNPmVOKQH7TLq +cTah21eYYH7j+nS52ssMtcLdAWYfYPOk3XckNXl8jqWQP0vwUV0qSVqJQ97JVkJ8e4Z Jdpku/G+ZuMoCvb30rbydImE2PbF7YDhcB409s38STE6TU1VPXWCLOyIM8IS7isiUxN7 nd75Q43DsxFj4ak4xRUmm1ozO4Rt3Q0ejbZ0J6U3D+jqKvfXUX8XsLbwIEUhvrHjqNI6 pNSl92ZAiO0PptgCaAfc/AJhHWUE9N18AIutPi4EH2gRGLI0f+dapDL6HNdhC2RbqG7g d/hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JvJP2Tgt4pX2ppH9EzhVhChGVgkGEIkwYadFsno5MHg=; b=WVDe45k2BPTE0ypAqp4N5mQdbhUqI0n/+/ZvMZTDDcOBV5imL7FJBdRump2Dp8POi/ kROcSy70rsWpGbXsPJ8emhiqyBJej4IyoDJG2aguqcgHzZ0ugG8oHNkGyFjtoCOGo/3K +Bo4F30fH26TorW7tT4fioltnhQcfsLejQ/LQ1P+1dXrsuCJis8k08UtsOX1bCWee1e1 OWA6Po8NqrBGjf12CBLpTEQJfxxzXVgCGInCIsaYs25at3yMGcqlH1D0e8UVn1PMv9JX GwoVTA3t1Xxj4TK9wnKJHl/QFNRc2k2twaYTfMPeCEWShWAXbwRPu1XUpkYmMzZrW/8W XIIQ== X-Gm-Message-State: AOAM531v29v94ULX5ayjAbflmp7r6gnupIWwDZfH1Ny4doPaSMOfJvI5 z+QO/j9OlYRxm/VEBcRCgad+m6Kv9PVOViXRnvg= X-Google-Smtp-Source: ABdhPJw6ZgQlmz+hbSrgT19GNOS5g47oOafiC6apHPIBG8eDx1tCq1jDqIPCukmLvcZ9v8LdA3H5eqEofNGdPEMfWzA= X-Received: by 2002:a05:6e02:1b87:: with SMTP id h7mr5787981ili.271.1623993394943; Thu, 17 Jun 2021 22:16:34 -0700 (PDT) MIME-Version: 1.0 References: <1623763327-30987-1-git-send-email-fengchengwen@huawei.com> In-Reply-To: From: Jerin Jacob Date: Fri, 18 Jun 2021 10:46:08 +0530 Message-ID: To: Bruce Richardson Cc: fengchengwen , Thomas Monjalon , Ferruh Yigit , dpdk-dev , Nipun Gupta , Hemant Agrawal , Maxime Coquelin , Honnappa Nagarahalli , Jerin Jacob , David Marchand Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, Jun 17, 2021 at 1:30 PM Bruce Richardson wrote: > > On Thu, Jun 17, 2021 at 01:12:22PM +0530, Jerin Jacob wrote: > > On Thu, Jun 17, 2021 at 12:43 AM Bruce Richardson > > wrote: > > > > > > On Wed, Jun 16, 2021 at 11:38:08PM +0530, Jerin Jacob wrote: > > > > On Wed, Jun 16, 2021 at 11:01 PM Bruce Richardson > > > > wrote: > > > > > > > > > > On Wed, Jun 16, 2021 at 05:41:45PM +0800, fengchengwen wrote: > > > > > > On 2021/6/16 0:38, Bruce Richardson wrote: > > > > > > > On Tue, Jun 15, 2021 at 09:22:07PM +0800, Chengwen Feng wrote: > > > > > > >> This patch introduces 'dmadevice' which is a generic type of DMA > > > > > > >> device. > > > > > > >> > > > > > > >> The APIs of dmadev library exposes some generic operations which can > > > > > > >> enable configuration and I/O with the DMA devices. > > > > > > >> > > > > > > >> Signed-off-by: Chengwen Feng > > > > > > >> --- > > > > > > > Thanks for sending this. > > > > > > > > > > > > > > Of most interest to me right now are the key data-plane APIs. While we are > > > > > > > still in the prototyping phase, below is a draft of what we are thinking > > > > > > > for the key enqueue/perform_ops/completed_ops APIs. > > > > > > > > > > > > > > Some key differences I note in below vs your original RFC: > > > > > > > * Use of void pointers rather than iova addresses. While using iova's makes > > > > > > > sense in the general case when using hardware, in that it can work with > > > > > > > both physical addresses and virtual addresses, if we change the APIs to use > > > > > > > void pointers instead it will still work for DPDK in VA mode, while at the > > > > > > > same time allow use of software fallbacks in error cases, and also a stub > > > > > > > driver than uses memcpy in the background. Finally, using iova's makes the > > > > > > > APIs a lot more awkward to use with anything but mbufs or similar buffers > > > > > > > where we already have a pre-computed physical address. > > > > > > > > > > > > The iova is an hint to application, and widely used in DPDK. > > > > > > If switch to void, how to pass the address (iova or just va ?) > > > > > > this may introduce implementation dependencies here. > > > > > > > > > > > > Or always pass the va, and the driver performs address translation, and this > > > > > > translation may cost too much cpu I think. > > > > > > > > > > > > > > > > On the latter point, about driver doing address translation I would agree. > > > > > However, we probably need more discussion about the use of iova vs just > > > > > virtual addresses. My thinking on this is that if we specify the API using > > > > > iovas it will severely hurt usability of the API, since it forces the user > > > > > to take more inefficient codepaths in a large number of cases. Given a > > > > > pointer to the middle of an mbuf, one cannot just pass that straight as an > > > > > iova but must instead do a translation into offset from mbuf pointer and > > > > > then readd the offset to the mbuf base address. > > > > > > > > > > My preference therefore is to require the use of an IOMMU when using a > > > > > dmadev, so that it can be a much closer analog of memcpy. Once an iommu is > > > > > present, DPDK will run in VA mode, allowing virtual addresses to our > > > > > hugepage memory to be sent directly to hardware. Also, when using > > > > > dmadevs on top of an in-kernel driver, that kernel driver may do all iommu > > > > > management for the app, removing further the restrictions on what memory > > > > > can be addressed by hardware. > > > > > > > > > > > > One issue of keeping void * is that memory can come from stack or heap . > > > > which HW can not really operate it on. > > > > > > when kernel driver is managing the IOMMU all process memory can be worked > > > on, not just hugepage memory, so using iova is wrong in these cases. > > > > But not for stack and heap memory. Right? > > > Yes, even stack and heap can be accessed. The HW device cannot as that memory is NOT mapped to IOMMU. It will result in the transaction fault. At least, In octeon, DMA HW job descriptor will have a pointer (IOVA) which will be updated by _HW_ upon copy job completion. That memory can not be from the heap(malloc()) or stack as those are not mapped by IOMMU. > > > > > > > As I previously said, using iova prevents the creation of a pure software > > > dummy driver too using memcpy in the background. > > > > Why ? the memory alloced uing rte_alloc/rte_memzone etc can be touched by CPU. > > > Yes, but it can't be accessed using physical address, so again only VA mode > where iova's are "void *" make sense. I agree that it should be a physical address. My only concern that void * does not express it can not be from stack/heap. If API tells the memory need to allotted by rte_alloc() or rte_memzone() etc is fine with me. or it may better that. Have separate API to alloc the handle so based on the driver, it can be rte_alloc() or malloc(). It can be burst API in slow path to get number of status pointers > > > Thinking more, Since anyway, we need a separate function for knowing > > the completion status, > > I think, it can be an opaque object as the completion code. Exposing > > directly the status may not help > > . As the driver needs a "context" or "call" to change the > > driver-specific completion code to DPDK completion code. > > > I'm sorry, I didn't follow this. By completion code, you mean the status of > whether a copy job succeeded/failed? Yes, the status of job completion.