From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CFA7BA0A0F; Sun, 4 Jul 2021 09:43:48 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5B38040040; Sun, 4 Jul 2021 09:43:48 +0200 (CEST) Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) by mails.dpdk.org (Postfix) with ESMTP id 4AE694003F for ; Sun, 4 Jul 2021 09:43:47 +0200 (CEST) Received: by mail-io1-f46.google.com with SMTP id v3so17134683ioq.9 for ; Sun, 04 Jul 2021 00:43:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=hCZKTTSlaR/BlqKzRXHnJJclK/38Kcmy5hJTePwgx7Q=; b=dXwqShiw7beidzc+NyIfPdcFOGATM29ovC/oZ9CccQwl7t3B94R94G/Us9wBbwrroa BwdElsE78kwuXc5mrb9roDggtbeIHApstWT/fSJjECg1ApXCCJOTMhPFnOn+S4mzWQh4 vVuN5VPLg3Lkuyimh/3nghcP2rdC3JNk/Jb9cgJBIJFbWhqlGyyOYcpti0h/xB5NKXQN hE0BQ/a6QFnuI0IICk5XfWNpOoHs5j0SvTvT3t18w9fbRFkrPXwmpwIasjZIhOm33oyN TyHV0khCFrML2a0wMrVeN81wobLbHhEDKXeBlMCJwKotJqFDF2fu/bgqn59932NIL8Ij Y+Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=hCZKTTSlaR/BlqKzRXHnJJclK/38Kcmy5hJTePwgx7Q=; b=AgWCxgiwgowRdLJcOI2a5iRpgTMXHxnXzjrySbzU5OubuaDlsfexaf4ZXSFzHJaVfl SA2zH3IVbTEtMvic8xVGidlH8htQi2RTNH769avBev1xp5AusW8SAcSUFY/lFi3whax9 nthitXJvl0vTHSFBBgubp4Yy7wtUOlfiQcfQZRBmu6baJkTux6xYEFA0oLf2qmnmtpM2 WZnivG0AkvRuY9SjKh2lbiMXpf+bkT41jE94Usf7kOUDDvowUdz9WlOh/WPqv9bObuGi F1aQW1LOVNOtVyI3A7FuhFQMOz7B2dXGXZo1Vhdm0Hato/e7rtrfWmaBCRaP1hZcvNL6 Xz6w== X-Gm-Message-State: AOAM530ui0JV9Td430REfKprSmZqIr/BpmS7UTVkPlYELXXLMJrMrygG bOChgMa0XoSy2IBiLqd7RHdCJKsMSATdOecBn7A= X-Google-Smtp-Source: ABdhPJycEMJrgYKd3Gm6xWdGb7gk4tIKDzrrRYAzJjW6bOh3ej6HqCP4Kud8n2Awu1e5XNin561aJNDNfF5s/Gn8h7I= X-Received: by 2002:a5d:9c43:: with SMTP id 3mr6771265iof.123.1625384626612; Sun, 04 Jul 2021 00:43:46 -0700 (PDT) MIME-Version: 1.0 References: <1623763327-30987-1-git-send-email-fengchengwen@huawei.com> <25d29598-c26d-8497-2867-9b650c79df49@huawei.com> <3db2eda0-4490-2b8f-c65d-636bcf794494@huawei.com> <98CBD80474FA8B44BF855DF32C47DC35C618D9@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35C618DC@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35C618DF@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35C618DF@smartserver.smartshare.dk> From: Jerin Jacob Date: Sun, 4 Jul 2021 13:13:20 +0530 Message-ID: To: =?UTF-8?Q?Morten_Br=C3=B8rup?= Cc: fengchengwen , Bruce Richardson , Jerin Jacob , Nipun Gupta , Thomas Monjalon , Ferruh Yigit , dpdk-dev , Hemant Agrawal , Maxime Coquelin , Honnappa Nagarahalli , David Marchand , Satananda Burla , Prasun Kapoor Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] dmadev discussion summary X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Sat, Jul 3, 2021 at 5:54 PM Morten Br=C3=B8rup wrote: > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob > > Sent: Saturday, 3 July 2021 11.09 > > > > On Sat, Jul 3, 2021 at 2:23 PM Morten Br=C3=B8rup > > wrote: > > > > > > > From: fengchengwen [mailto:fengchengwen@huawei.com] > > > > Sent: Saturday, 3 July 2021 02.32 > > > > > > > > On 2021/7/2 22:57, Morten Br=C3=B8rup wrote: > > > > >> In the DPDK framework, many data-plane API names contain queues. > > > > e.g. > > > > >> eventdev/crypto.. > > > > >> The concept of virt queues has continuity. > > > > > > > > > > I was also wondering about the name "virtual queue". > > > > > > > > > > Usually, something "virtual" would be an abstraction of something > > > > physical, e.g. a software layer on top of something physical. > > > > > > > > > > Back in the days, a "DMA channel" used to mean a DMA engine on a > > CPU. > > > > If a CPU had 2 DMA channels, they could both be set up > > simultaneously. > > > > > > > > > > The current design has the "dmadev" representing a CPU or other > > chip, > > > > which has one or more "HW-queues" representing DMA channels (of the > > > > same type), and then "virt-queue" as a software abstraction on top, > > for > > > > using a DMA channel in different ways through individually > > configured > > > > contexts (virt-queues). > > > > > > > > > > It makes sense to me, although I would consider renaming "HW- > > queue" > > > > to "channel" and perhaps "virt-queue" to "queue". > > > > > > > > The 'DMA channel' is more used than 'DMA queue', at least google > > show > > > > that there are at least 20+ times more. > > > > > > > > It's a good idea build the abstraction layer: queue <> channel <> > > dma- > > > > controller. > > > > In this way, the meaning of each layer is relatively easy to > > > > distinguish literally. > > > > > > > > will fix in V2 > > > > > > > > > > After re-reading all the mails in this thread, I have found one more > > important high level detail still not decided: > > > > > > Bruce had suggested flattening the DMA channels, so each dmadev > > represents a DMA channel. And DMA controllers with multiple DMA > > channels will have to instantiate multiple dmadevs, one for each DMA > > channel. > > > > > > Just like a four port NIC instantiates four ethdevs. > > > > > > Then, like ethdevs, there would only be two abstraction layers: > > dmadev <> queue, where a dmadev is a DMA channel on a DMA controller. > > > > > > However, this assumes that the fast path functions on the individual > > DMA channels of a DMA controller can be accessed completely > > independently and simultaneously by multiple threads. (Otherwise, the > > driver would need to implement critical regions or locking around > > accessing the common registers in the DMA controller shared by the DMA > > channels.) > > > > > > Unless any of the DMA controller vendors claim that this assumption > > about independence of the DMA channels is wrong, I strongly support > > Bruce's flattening suggestion. > > > > It is wrong from alteast octeontx2_dma PoV. > > > > # The PCI device is DMA controller where the driver/device is > > mapped.(As device driver is based on PCI bus, We dont want to have > > vdev for this) > > # The PCI device has HW queue(s) > > # Each HW queue has different channels. > > > > In the current configuration, we have only one queue per device and it > > has 4 channels. 4 channels are not threaded safe as it is based on > > single queue. > > Please clarify "current configuration": Is that a configuration modifiabl= e by changing some software/driver, or is it the chip that was built that w= ay in the RTL code? We have 8 queues per SoC, Based on some of HW versions it can be configured as (a) or (b) using FW settings. a) One PCI devices with 8 Queues b) 8 PCI devices with each one has one queue. Everyone is using mode (b) as it helps 8 different applications to use DMA as if one application binds the PCI device other applications can not use the same PCI device. If one application needs 8 queues, it is possible that 8 dmadevice can be bound to a single application with mode (b). I think, in above way we can flatten to <> > > > > > I think, if we need to flatten it, I think, it makes sense to have > > dmadev <> channel (and each channel can have thread-safe capability > > based on how it mapped on HW queues based on the device driver > > capability). > > The key question is how many threads can independently call data-plane dm= adev functions (rte_dma_copy() etc.) simultaneously. If I understand your e= xplanation correctly, only one - because you only have one DMA device, and = all access to it goes through a single hardware queue. > > I just realized that although you only have one DMA Controller with only = one HW queue, your four DMA channels allows four sequentially initiated tra= nsactions to be running simultaneously. Does the application have any benef= it by knowing that the dmadev can have multiple ongoing transactions, or ca= n the fast-path dmadev API hide that ability? In my view it is better to hide and I have similar proposal at http://mails.dpdk.org/archives/dev/2021-July/213141.html -------------- > 7) Because data-plane APIs are not thread-safe, and user could determin= e > virt-queue to HW-queue's map (at the queue-setup stage), so it is us= er's > duty to ensure thread-safe. +1. But I am not sure how easy for the fast-path application to have this l= ogic, Instead, I think, it is better to tell the capa for queue by driver and in channel configuration, the application can request for requirement (Is multiple producers enq to the same HW queue or not). Based on the request, the implementation can pick the correct function pointer for enq.(lock vs lockless version if HW does not support lockless) ------------------------ >