From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E84E3A0C4D; Thu, 17 Jun 2021 09:42:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7ACFB4067A; Thu, 17 Jun 2021 09:42:40 +0200 (CEST) Received: from mail-il1-f180.google.com (mail-il1-f180.google.com [209.85.166.180]) by mails.dpdk.org (Postfix) with ESMTP id 3741540150 for <dev@dpdk.org>; Thu, 17 Jun 2021 09:42:39 +0200 (CEST) Received: by mail-il1-f180.google.com with SMTP id w14so4628634ilv.1 for <dev@dpdk.org>; Thu, 17 Jun 2021 00:42:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=p3Wn8y/TpL8Uc2abZloZIfkIxakkN4v9H96IbnTmmVs=; b=BUNen0T/bMEvXI4hLb278eqi3dB0efgemU+cOS2ya7hA6o2dP0w1k8ZVL29p0G3f8q N/oco4zPu/oIajRaNWScPi8hCpFeksAxzRDcZWeqmAnPLGqn7JDQDrqyg03oshncY1e1 jEK205I8luQLsV+z0Km0NTPQ8F9iyqIaf9JStcvOiMuFIheWwA+oA0Wp9vTq4sytMW82 4qNGZA9J4c6Zn+Au4tYcO4+hBHS/P3edDBazrU2UWZ7sAq52k67SyadCG1vtW2Tw5CzS 77itUyrRYmrM9LOT7EQzKXvnMqj5GCHNjjPxe+/DpdlHxsiwymxLl6WTnfIr15Q7q/od qBjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=p3Wn8y/TpL8Uc2abZloZIfkIxakkN4v9H96IbnTmmVs=; b=swQnn/oaqELOJTE9WJ2FPiKzI2r+DaX9bwSILrAOHttY5hTsi0Im21bTXSpSBFYNG3 nZ5R3IhiKpdts/uQ7NJtQFpGxoJpIuz2IX7kwA3/9YlyucOjUilM2a40hLXVIpU5cO4d uXXk0dVoAlEO+RMr5alXcpsP/L2SPsjJNFTa8YGi7ZYoMyN2FAJYTUj+/S+Jjw7AG7g/ XItQPUWXNel1RH3jCc5LSYYMCmg3UALbS+d3hKGh3sU+OV6nSw/am+sDnYyO02QyNnDU TuuB4wil8GAbPFHP3a9c5njIC/fTxufOZRE4uCqkJQaz4ii9pcDepdJYJmftmo3ZSibU 300w== X-Gm-Message-State: AOAM532+TzdngwbBN9TCiitQvP7S7Gck9DD97kTxUl4uI2pT5swlB/3v +A+TR/v6R75Q1FFT7dkUtlQ3b3OyL5y6wvCMPS4= X-Google-Smtp-Source: ABdhPJzhAR7ZPIJ6yQjdQW62fXQmZAfTjnsPU0XIcvzmHUPtdKSleFt47GuAfm2k8LsJfPXZ6AYE6pcJmX62Q7DK6w4= X-Received: by 2002:a92:d08c:: with SMTP id h12mr2483261ilh.294.1623915758503; Thu, 17 Jun 2021 00:42:38 -0700 (PDT) MIME-Version: 1.0 References: <1623763327-30987-1-git-send-email-fengchengwen@huawei.com> <YMjXilFHjCxQ9ViD@bricha3-MOBL.ger.corp.intel.com> <a5dc8da0-bfb6-db19-3567-ecb912c4c6ef@huawei.com> <YMo1V/Trf7WH8dgN@bricha3-MOBL.ger.corp.intel.com> <CALBAE1O62skUmGwyt7=BdsD9dXYKUnXVD-rVCfO55d5DhG1Qfw@mail.gmail.com> <YMpNWu+qUUciExfD@bricha3-MOBL.ger.corp.intel.com> In-Reply-To: <YMpNWu+qUUciExfD@bricha3-MOBL.ger.corp.intel.com> From: Jerin Jacob <jerinjacobk@gmail.com> Date: Thu, 17 Jun 2021 13:12:22 +0530 Message-ID: <CALBAE1PGLv0rn2vntCaO5Th=CU0RaqBchbNYh_rpQEpDfkwLnQ@mail.gmail.com> To: Bruce Richardson <bruce.richardson@intel.com> Cc: fengchengwen <fengchengwen@huawei.com>, Thomas Monjalon <thomas@monjalon.net>, Ferruh Yigit <ferruh.yigit@intel.com>, dpdk-dev <dev@dpdk.org>, Nipun Gupta <nipun.gupta@nxp.com>, Hemant Agrawal <hemant.agrawal@nxp.com>, Maxime Coquelin <maxime.coquelin@redhat.com>, Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>, Jerin Jacob <jerinj@marvell.com>, David Marchand <david.marchand@redhat.com> Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> On Thu, Jun 17, 2021 at 12:43 AM Bruce Richardson <bruce.richardson@intel.com> wrote: > > On Wed, Jun 16, 2021 at 11:38:08PM +0530, Jerin Jacob wrote: > > On Wed, Jun 16, 2021 at 11:01 PM Bruce Richardson > > <bruce.richardson@intel.com> wrote: > > > > > > On Wed, Jun 16, 2021 at 05:41:45PM +0800, fengchengwen wrote: > > > > On 2021/6/16 0:38, Bruce Richardson wrote: > > > > > On Tue, Jun 15, 2021 at 09:22:07PM +0800, Chengwen Feng wrote: > > > > >> This patch introduces 'dmadevice' which is a generic type of DMA > > > > >> device. > > > > >> > > > > >> The APIs of dmadev library exposes some generic operations which can > > > > >> enable configuration and I/O with the DMA devices. > > > > >> > > > > >> Signed-off-by: Chengwen Feng <fengchengwen@huawei.com> > > > > >> --- > > > > > Thanks for sending this. > > > > > > > > > > Of most interest to me right now are the key data-plane APIs. While we are > > > > > still in the prototyping phase, below is a draft of what we are thinking > > > > > for the key enqueue/perform_ops/completed_ops APIs. > > > > > > > > > > Some key differences I note in below vs your original RFC: > > > > > * Use of void pointers rather than iova addresses. While using iova's makes > > > > > sense in the general case when using hardware, in that it can work with > > > > > both physical addresses and virtual addresses, if we change the APIs to use > > > > > void pointers instead it will still work for DPDK in VA mode, while at the > > > > > same time allow use of software fallbacks in error cases, and also a stub > > > > > driver than uses memcpy in the background. Finally, using iova's makes the > > > > > APIs a lot more awkward to use with anything but mbufs or similar buffers > > > > > where we already have a pre-computed physical address. > > > > > > > > The iova is an hint to application, and widely used in DPDK. > > > > If switch to void, how to pass the address (iova or just va ?) > > > > this may introduce implementation dependencies here. > > > > > > > > Or always pass the va, and the driver performs address translation, and this > > > > translation may cost too much cpu I think. > > > > > > > > > > On the latter point, about driver doing address translation I would agree. > > > However, we probably need more discussion about the use of iova vs just > > > virtual addresses. My thinking on this is that if we specify the API using > > > iovas it will severely hurt usability of the API, since it forces the user > > > to take more inefficient codepaths in a large number of cases. Given a > > > pointer to the middle of an mbuf, one cannot just pass that straight as an > > > iova but must instead do a translation into offset from mbuf pointer and > > > then readd the offset to the mbuf base address. > > > > > > My preference therefore is to require the use of an IOMMU when using a > > > dmadev, so that it can be a much closer analog of memcpy. Once an iommu is > > > present, DPDK will run in VA mode, allowing virtual addresses to our > > > hugepage memory to be sent directly to hardware. Also, when using > > > dmadevs on top of an in-kernel driver, that kernel driver may do all iommu > > > management for the app, removing further the restrictions on what memory > > > can be addressed by hardware. > > > > > > One issue of keeping void * is that memory can come from stack or heap . > > which HW can not really operate it on. > > when kernel driver is managing the IOMMU all process memory can be worked > on, not just hugepage memory, so using iova is wrong in these cases. But not for stack and heap memory. Right? > > As I previously said, using iova prevents the creation of a pure software > dummy driver too using memcpy in the background. Why ? the memory alloced uing rte_alloc/rte_memzone etc can be touched by CPU. Thinking more, Since anyway, we need a separate function for knowing the completion status, I think, it can be an opaque object as the completion code. Exposing directly the status may not help . As the driver needs a "context" or "call" to change the driver-specific completion code to DPDK completion code. > > /Bruce