From: fengchengwen <fengchengwen@huawei.com>
To: "Morten Brørup" <mb@smartsharesystems.com>,
"Jerin Jacob" <jerinjacobk@gmail.com>
Cc: Bruce Richardson <bruce.richardson@intel.com>,
Jerin Jacob <jerinj@marvell.com>,
Nipun Gupta <nipun.gupta@nxp.com>,
Thomas Monjalon <thomas@monjalon.net>,
Ferruh Yigit <ferruh.yigit@intel.com>, dpdk-dev <dev@dpdk.org>,
Hemant Agrawal <hemant.agrawal@nxp.com>,
Maxime Coquelin <maxime.coquelin@redhat.com>,
Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>,
David Marchand <david.marchand@redhat.com>,
Satananda Burla <sburla@marvell.com>,
Prasun Kapoor <pkapoor@marvell.com>
Subject: Re: [dpdk-dev] dmadev discussion summary
Date: Tue, 6 Jul 2021 15:11:12 +0800 [thread overview]
Message-ID: <1d7e9c47-bbda-0395-c0e4-0de7d7511c3d@huawei.com> (raw)
In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35C618E3@smartserver.smartshare.dk>
On 2021/7/5 18:28, Morten Brørup wrote:
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
>> Sent: Sunday, 4 July 2021 09.43
>>
>> On Sat, Jul 3, 2021 at 5:54 PM Morten Brørup <mb@smartsharesystems.com>
>> wrote:
>>>
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
>>>> Sent: Saturday, 3 July 2021 11.09
>>>>
>>>> On Sat, Jul 3, 2021 at 2:23 PM Morten Brørup
>> <mb@smartsharesystems.com>
>>>> wrote:
>>>>>
>>>>>> From: fengchengwen [mailto:fengchengwen@huawei.com]
>>>>>> Sent: Saturday, 3 July 2021 02.32
>>>>>>
>>>>>> On 2021/7/2 22:57, Morten Brørup wrote:
>>>>>>>> In the DPDK framework, many data-plane API names contain
>> queues.
>>>>>> e.g.
>>>>>>>> eventdev/crypto..
>>>>>>>> The concept of virt queues has continuity.
>>>>>>>
>>>>>>> I was also wondering about the name "virtual queue".
>>>>>>>
>>>>>>> Usually, something "virtual" would be an abstraction of
>> something
>>>>>> physical, e.g. a software layer on top of something physical.
>>>>>>>
>>>>>>> Back in the days, a "DMA channel" used to mean a DMA engine
>> on a
>>>> CPU.
>>>>>> If a CPU had 2 DMA channels, they could both be set up
>>>> simultaneously.
>>>>>>>
>>>>>>> The current design has the "dmadev" representing a CPU or
>> other
>>>> chip,
>>>>>> which has one or more "HW-queues" representing DMA channels (of
>> the
>>>>>> same type), and then "virt-queue" as a software abstraction on
>> top,
>>>> for
>>>>>> using a DMA channel in different ways through individually
>>>> configured
>>>>>> contexts (virt-queues).
>>>>>>>
>>>>>>> It makes sense to me, although I would consider renaming "HW-
>>>> queue"
>>>>>> to "channel" and perhaps "virt-queue" to "queue".
>>>>>>
>>>>>> The 'DMA channel' is more used than 'DMA queue', at least
>> google
>>>> show
>>>>>> that there are at least 20+ times more.
>>>>>>
>>>>>> It's a good idea build the abstraction layer: queue <> channel
>> <>
>>>> dma-
>>>>>> controller.
>>>>>> In this way, the meaning of each layer is relatively easy to
>>>>>> distinguish literally.
>>>>>>
>>>>>> will fix in V2
>>>>>>
>>>>>
>>>>> After re-reading all the mails in this thread, I have found one
>> more
>>>> important high level detail still not decided:
>>>>>
>>>>> Bruce had suggested flattening the DMA channels, so each dmadev
>>>> represents a DMA channel. And DMA controllers with multiple DMA
>>>> channels will have to instantiate multiple dmadevs, one for each
>> DMA
>>>> channel.
>>>>>
>>>>> Just like a four port NIC instantiates four ethdevs.
>>>>>
>>>>> Then, like ethdevs, there would only be two abstraction layers:
>>>> dmadev <> queue, where a dmadev is a DMA channel on a DMA
>> controller.
>>>>>
>>>>> However, this assumes that the fast path functions on the
>> individual
>>>> DMA channels of a DMA controller can be accessed completely
>>>> independently and simultaneously by multiple threads. (Otherwise,
>> the
>>>> driver would need to implement critical regions or locking around
>>>> accessing the common registers in the DMA controller shared by the
>> DMA
>>>> channels.)
>>>>>
>>>>> Unless any of the DMA controller vendors claim that this
>> assumption
>>>> about independence of the DMA channels is wrong, I strongly support
>>>> Bruce's flattening suggestion.
>>>>
>>>> It is wrong from alteast octeontx2_dma PoV.
>>>>
>>>> # The PCI device is DMA controller where the driver/device is
>>>> mapped.(As device driver is based on PCI bus, We dont want to have
>>>> vdev for this)
>>>> # The PCI device has HW queue(s)
>>>> # Each HW queue has different channels.
>>>>
>>>> In the current configuration, we have only one queue per device and
>> it
>>>> has 4 channels. 4 channels are not threaded safe as it is based on
>>>> single queue.
>>>
>>> Please clarify "current configuration": Is that a configuration
>> modifiable by changing some software/driver, or is it the chip that was
>> built that way in the RTL code?
>>
>> We have 8 queues per SoC, Based on some of HW versions it can be
>> configured as (a) or (b) using FW settings.
>> a) One PCI devices with 8 Queues
>> b) 8 PCI devices with each one has one queue.
>>
>> Everyone is using mode (b) as it helps 8 different applications to use
>> DMA as if one application binds the PCI device other applications can
>> not use the same PCI device.
>> If one application needs 8 queues, it is possible that 8 dmadevice can
>> be bound to a single application with mode (b).
>>
>>
>> I think, in above way we can flatten to <device> <> <channel/queue>
I just look at dpaa2_qdma driver code, and found it seems OK to setup
one xxxdev for one queue.
>>
>>>
>>>>
>>>> I think, if we need to flatten it, I think, it makes sense to have
>>>> dmadev <> channel (and each channel can have thread-safe capability
>>>> based on how it mapped on HW queues based on the device driver
>>>> capability).
>>>
>>> The key question is how many threads can independently call data-
>> plane dmadev functions (rte_dma_copy() etc.) simultaneously. If I
>> understand your explanation correctly, only one - because you only have
>> one DMA device, and all access to it goes through a single hardware
>> queue.
>>>
>>> I just realized that although you only have one DMA Controller with
>> only one HW queue, your four DMA channels allows four sequentially
>> initiated transactions to be running simultaneously. Does the
>> application have any benefit by knowing that the dmadev can have
>> multiple ongoing transactions, or can the fast-path dmadev API hide
>> that ability?
>>
>> In my view it is better to hide and I have similar proposal at
>> http://mails.dpdk.org/archives/dev/2021-July/213141.html
>> --------------
>>> 7) Because data-plane APIs are not thread-safe, and user could
>> determine
>>> virt-queue to HW-queue's map (at the queue-setup stage), so it
>> is user's
>>> duty to ensure thread-safe.
>>
>> +1. But I am not sure how easy for the fast-path application to have
>> this logic,
>> Instead, I think, it is better to tell the capa for queue by driver
>> and in channel configuration,
>> the application can request for requirement (Is multiple producers enq
>> to the same HW queue or not).
>> Based on the request, the implementation can pick the correct function
>> pointer for enq.(lock vs lockless version if HW does not support
>> lockless)
>
> +1 to that!
>
add in channel configuration sound good.
>>
>> ------------------------
>>>
>
next prev parent reply other threads:[~2021-07-06 7:11 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-15 13:22 [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library Chengwen Feng
2021-06-15 16:38 ` Bruce Richardson
2021-06-16 7:09 ` Morten Brørup
2021-06-16 10:17 ` fengchengwen
2021-06-16 12:09 ` Morten Brørup
2021-06-16 13:06 ` Bruce Richardson
2021-06-16 14:37 ` Jerin Jacob
2021-06-17 9:15 ` Bruce Richardson
2021-06-18 5:52 ` Jerin Jacob
2021-06-18 9:41 ` fengchengwen
2021-06-22 17:25 ` Jerin Jacob
2021-06-23 3:30 ` fengchengwen
2021-06-23 7:21 ` Jerin Jacob
2021-06-23 9:37 ` Bruce Richardson
2021-06-23 11:40 ` Jerin Jacob
2021-06-23 14:19 ` Bruce Richardson
2021-06-24 6:49 ` Jerin Jacob
2021-06-23 9:41 ` Bruce Richardson
2021-06-23 10:10 ` Morten Brørup
2021-06-23 11:46 ` Jerin Jacob
2021-06-23 14:22 ` Bruce Richardson
2021-06-18 9:55 ` Bruce Richardson
2021-06-22 17:31 ` Jerin Jacob
2021-06-22 19:17 ` Bruce Richardson
2021-06-23 7:00 ` Jerin Jacob
2021-06-16 9:41 ` fengchengwen
2021-06-16 17:31 ` Bruce Richardson
2021-06-16 18:08 ` Jerin Jacob
2021-06-16 19:13 ` Bruce Richardson
2021-06-17 7:42 ` Jerin Jacob
2021-06-17 8:00 ` Bruce Richardson
2021-06-18 5:16 ` Jerin Jacob
2021-06-18 10:03 ` Bruce Richardson
2021-06-22 17:36 ` Jerin Jacob
2021-06-17 9:48 ` fengchengwen
2021-06-17 11:02 ` Bruce Richardson
2021-06-17 14:18 ` Bruce Richardson
2021-06-18 8:52 ` fengchengwen
2021-06-18 9:30 ` Bruce Richardson
2021-06-22 17:51 ` Jerin Jacob
2021-06-23 3:50 ` fengchengwen
2021-06-23 11:00 ` Jerin Jacob
2021-06-23 14:56 ` Bruce Richardson
2021-06-24 12:19 ` fengchengwen
2021-06-26 3:59 ` [dpdk-dev] dmadev discussion summary fengchengwen
2021-06-28 10:00 ` Bruce Richardson
2021-06-28 11:14 ` Ananyev, Konstantin
2021-06-28 12:53 ` Bruce Richardson
2021-07-02 13:31 ` fengchengwen
2021-07-01 15:01 ` Jerin Jacob
2021-07-01 16:33 ` Bruce Richardson
2021-07-02 7:39 ` Morten Brørup
2021-07-02 10:05 ` Bruce Richardson
2021-07-02 13:45 ` fengchengwen
2021-07-02 14:57 ` Morten Brørup
2021-07-03 0:32 ` fengchengwen
2021-07-03 8:53 ` Morten Brørup
2021-07-03 9:08 ` Jerin Jacob
2021-07-03 12:24 ` Morten Brørup
2021-07-04 7:43 ` Jerin Jacob
2021-07-05 10:28 ` Morten Brørup
2021-07-06 7:11 ` fengchengwen [this message]
2021-07-03 9:45 ` fengchengwen
2021-07-03 12:00 ` Morten Brørup
2021-07-04 7:34 ` Jerin Jacob
2021-07-02 7:07 ` Liang Ma
2021-07-02 13:59 ` fengchengwen
2021-06-24 7:03 ` [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library Jerin Jacob
2021-06-24 7:59 ` Morten Brørup
2021-06-24 8:05 ` Jerin Jacob
2021-06-23 5:34 ` Hu, Jiayu
2021-06-23 11:07 ` Jerin Jacob
2021-06-16 2:17 ` Wang, Haiyue
2021-06-16 8:04 ` Bruce Richardson
2021-06-16 8:16 ` Wang, Haiyue
2021-06-16 12:14 ` David Marchand
2021-06-16 13:11 ` Bruce Richardson
2021-06-16 16:48 ` Honnappa Nagarahalli
2021-06-16 19:10 ` Bruce Richardson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1d7e9c47-bbda-0395-c0e4-0de7d7511c3d@huawei.com \
--to=fengchengwen@huawei.com \
--cc=bruce.richardson@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=hemant.agrawal@nxp.com \
--cc=honnappa.nagarahalli@arm.com \
--cc=jerinj@marvell.com \
--cc=jerinjacobk@gmail.com \
--cc=maxime.coquelin@redhat.com \
--cc=mb@smartsharesystems.com \
--cc=nipun.gupta@nxp.com \
--cc=pkapoor@marvell.com \
--cc=sburla@marvell.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).