From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1AD9BA0C47; Tue, 6 Jul 2021 09:11:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9532C40688; Tue, 6 Jul 2021 09:11:23 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id 0BF114067E for ; Tue, 6 Jul 2021 09:11:20 +0200 (CEST) Received: from dggemv711-chm.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4GJttC1G5Cz78Zg; Tue, 6 Jul 2021 15:07:47 +0800 (CST) Received: from dggpeml500024.china.huawei.com (7.185.36.10) by dggemv711-chm.china.huawei.com (10.1.198.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Tue, 6 Jul 2021 15:11:13 +0800 Received: from [127.0.0.1] (10.40.190.165) by dggpeml500024.china.huawei.com (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Tue, 6 Jul 2021 15:11:13 +0800 To: =?UTF-8?Q?Morten_Br=c3=b8rup?= , Jerin Jacob CC: Bruce Richardson , Jerin Jacob , Nipun Gupta , Thomas Monjalon , Ferruh Yigit , dpdk-dev , Hemant Agrawal , Maxime Coquelin , Honnappa Nagarahalli , David Marchand , Satananda Burla , Prasun Kapoor References: <1623763327-30987-1-git-send-email-fengchengwen@huawei.com> <25d29598-c26d-8497-2867-9b650c79df49@huawei.com> <3db2eda0-4490-2b8f-c65d-636bcf794494@huawei.com> <98CBD80474FA8B44BF855DF32C47DC35C618D9@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35C618DC@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35C618DF@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35C618E3@smartserver.smartshare.dk> From: fengchengwen Message-ID: <1d7e9c47-bbda-0395-c0e4-0de7d7511c3d@huawei.com> Date: Tue, 6 Jul 2021 15:11:12 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35C618E3@smartserver.smartshare.dk> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 8bit X-Originating-IP: [10.40.190.165] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpeml500024.china.huawei.com (7.185.36.10) X-CFilter-Loop: Reflected Subject: Re: [dpdk-dev] dmadev discussion summary X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 2021/7/5 18:28, Morten Brørup wrote: >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob >> Sent: Sunday, 4 July 2021 09.43 >> >> On Sat, Jul 3, 2021 at 5:54 PM Morten Brørup >> wrote: >>> >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob >>>> Sent: Saturday, 3 July 2021 11.09 >>>> >>>> On Sat, Jul 3, 2021 at 2:23 PM Morten Brørup >> >>>> wrote: >>>>> >>>>>> From: fengchengwen [mailto:fengchengwen@huawei.com] >>>>>> Sent: Saturday, 3 July 2021 02.32 >>>>>> >>>>>> On 2021/7/2 22:57, Morten Brørup wrote: >>>>>>>> In the DPDK framework, many data-plane API names contain >> queues. >>>>>> e.g. >>>>>>>> eventdev/crypto.. >>>>>>>> The concept of virt queues has continuity. >>>>>>> >>>>>>> I was also wondering about the name "virtual queue". >>>>>>> >>>>>>> Usually, something "virtual" would be an abstraction of >> something >>>>>> physical, e.g. a software layer on top of something physical. >>>>>>> >>>>>>> Back in the days, a "DMA channel" used to mean a DMA engine >> on a >>>> CPU. >>>>>> If a CPU had 2 DMA channels, they could both be set up >>>> simultaneously. >>>>>>> >>>>>>> The current design has the "dmadev" representing a CPU or >> other >>>> chip, >>>>>> which has one or more "HW-queues" representing DMA channels (of >> the >>>>>> same type), and then "virt-queue" as a software abstraction on >> top, >>>> for >>>>>> using a DMA channel in different ways through individually >>>> configured >>>>>> contexts (virt-queues). >>>>>>> >>>>>>> It makes sense to me, although I would consider renaming "HW- >>>> queue" >>>>>> to "channel" and perhaps "virt-queue" to "queue". >>>>>> >>>>>> The 'DMA channel' is more used than 'DMA queue', at least >> google >>>> show >>>>>> that there are at least 20+ times more. >>>>>> >>>>>> It's a good idea build the abstraction layer: queue <> channel >> <> >>>> dma- >>>>>> controller. >>>>>> In this way, the meaning of each layer is relatively easy to >>>>>> distinguish literally. >>>>>> >>>>>> will fix in V2 >>>>>> >>>>> >>>>> After re-reading all the mails in this thread, I have found one >> more >>>> important high level detail still not decided: >>>>> >>>>> Bruce had suggested flattening the DMA channels, so each dmadev >>>> represents a DMA channel. And DMA controllers with multiple DMA >>>> channels will have to instantiate multiple dmadevs, one for each >> DMA >>>> channel. >>>>> >>>>> Just like a four port NIC instantiates four ethdevs. >>>>> >>>>> Then, like ethdevs, there would only be two abstraction layers: >>>> dmadev <> queue, where a dmadev is a DMA channel on a DMA >> controller. >>>>> >>>>> However, this assumes that the fast path functions on the >> individual >>>> DMA channels of a DMA controller can be accessed completely >>>> independently and simultaneously by multiple threads. (Otherwise, >> the >>>> driver would need to implement critical regions or locking around >>>> accessing the common registers in the DMA controller shared by the >> DMA >>>> channels.) >>>>> >>>>> Unless any of the DMA controller vendors claim that this >> assumption >>>> about independence of the DMA channels is wrong, I strongly support >>>> Bruce's flattening suggestion. >>>> >>>> It is wrong from alteast octeontx2_dma PoV. >>>> >>>> # The PCI device is DMA controller where the driver/device is >>>> mapped.(As device driver is based on PCI bus, We dont want to have >>>> vdev for this) >>>> # The PCI device has HW queue(s) >>>> # Each HW queue has different channels. >>>> >>>> In the current configuration, we have only one queue per device and >> it >>>> has 4 channels. 4 channels are not threaded safe as it is based on >>>> single queue. >>> >>> Please clarify "current configuration": Is that a configuration >> modifiable by changing some software/driver, or is it the chip that was >> built that way in the RTL code? >> >> We have 8 queues per SoC, Based on some of HW versions it can be >> configured as (a) or (b) using FW settings. >> a) One PCI devices with 8 Queues >> b) 8 PCI devices with each one has one queue. >> >> Everyone is using mode (b) as it helps 8 different applications to use >> DMA as if one application binds the PCI device other applications can >> not use the same PCI device. >> If one application needs 8 queues, it is possible that 8 dmadevice can >> be bound to a single application with mode (b). >> >> >> I think, in above way we can flatten to <> I just look at dpaa2_qdma driver code, and found it seems OK to setup one xxxdev for one queue. >> >>> >>>> >>>> I think, if we need to flatten it, I think, it makes sense to have >>>> dmadev <> channel (and each channel can have thread-safe capability >>>> based on how it mapped on HW queues based on the device driver >>>> capability). >>> >>> The key question is how many threads can independently call data- >> plane dmadev functions (rte_dma_copy() etc.) simultaneously. If I >> understand your explanation correctly, only one - because you only have >> one DMA device, and all access to it goes through a single hardware >> queue. >>> >>> I just realized that although you only have one DMA Controller with >> only one HW queue, your four DMA channels allows four sequentially >> initiated transactions to be running simultaneously. Does the >> application have any benefit by knowing that the dmadev can have >> multiple ongoing transactions, or can the fast-path dmadev API hide >> that ability? >> >> In my view it is better to hide and I have similar proposal at >> http://mails.dpdk.org/archives/dev/2021-July/213141.html >> -------------- >>> 7) Because data-plane APIs are not thread-safe, and user could >> determine >>> virt-queue to HW-queue's map (at the queue-setup stage), so it >> is user's >>> duty to ensure thread-safe. >> >> +1. But I am not sure how easy for the fast-path application to have >> this logic, >> Instead, I think, it is better to tell the capa for queue by driver >> and in channel configuration, >> the application can request for requirement (Is multiple producers enq >> to the same HW queue or not). >> Based on the request, the implementation can pick the correct function >> pointer for enq.(lock vs lockless version if HW does not support >> lockless) > > +1 to that! > add in channel configuration sound good. >> >> ------------------------ >>> >