DPDK patches and discussions
 help / color / mirror / Atom feed
From: fengchengwen <fengchengwen@huawei.com>
To: "Morten Brørup" <mb@smartsharesystems.com>,
	"Jerin Jacob" <jerinjacobk@gmail.com>
Cc: Bruce Richardson <bruce.richardson@intel.com>,
	Jerin Jacob <jerinj@marvell.com>,
	Nipun Gupta <nipun.gupta@nxp.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	Ferruh Yigit <ferruh.yigit@intel.com>, dpdk-dev <dev@dpdk.org>,
	Hemant Agrawal <hemant.agrawal@nxp.com>,
	Maxime Coquelin <maxime.coquelin@redhat.com>,
	Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>,
	David Marchand <david.marchand@redhat.com>,
	Satananda Burla <sburla@marvell.com>,
	Prasun Kapoor <pkapoor@marvell.com>
Subject: Re: [dpdk-dev] dmadev discussion summary
Date: Tue, 6 Jul 2021 15:11:12 +0800	[thread overview]
Message-ID: <1d7e9c47-bbda-0395-c0e4-0de7d7511c3d@huawei.com> (raw)
In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35C618E3@smartserver.smartshare.dk>



On 2021/7/5 18:28, Morten Brørup wrote:
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
>> Sent: Sunday, 4 July 2021 09.43
>>
>> On Sat, Jul 3, 2021 at 5:54 PM Morten Brørup <mb@smartsharesystems.com>
>> wrote:
>>>
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
>>>> Sent: Saturday, 3 July 2021 11.09
>>>>
>>>> On Sat, Jul 3, 2021 at 2:23 PM Morten Brørup
>> <mb@smartsharesystems.com>
>>>> wrote:
>>>>>
>>>>>> From: fengchengwen [mailto:fengchengwen@huawei.com]
>>>>>> Sent: Saturday, 3 July 2021 02.32
>>>>>>
>>>>>> On 2021/7/2 22:57, Morten Brørup wrote:
>>>>>>>> In the DPDK framework, many data-plane API names contain
>> queues.
>>>>>> e.g.
>>>>>>>> eventdev/crypto..
>>>>>>>> The concept of virt queues has continuity.
>>>>>>>
>>>>>>> I was also wondering about the name "virtual queue".
>>>>>>>
>>>>>>> Usually, something "virtual" would be an abstraction of
>> something
>>>>>> physical, e.g. a software layer on top of something physical.
>>>>>>>
>>>>>>> Back in the days, a "DMA channel" used to mean a DMA engine
>> on a
>>>> CPU.
>>>>>> If a CPU had 2 DMA channels, they could both be set up
>>>> simultaneously.
>>>>>>>
>>>>>>> The current design has the "dmadev" representing a CPU or
>> other
>>>> chip,
>>>>>> which has one or more "HW-queues" representing DMA channels (of
>> the
>>>>>> same type), and then "virt-queue" as a software abstraction on
>> top,
>>>> for
>>>>>> using a DMA channel in different ways through individually
>>>> configured
>>>>>> contexts (virt-queues).
>>>>>>>
>>>>>>> It makes sense to me, although I would consider renaming "HW-
>>>> queue"
>>>>>> to "channel" and perhaps "virt-queue" to "queue".
>>>>>>
>>>>>> The 'DMA channel' is more used than 'DMA queue', at least
>> google
>>>> show
>>>>>> that there are at least 20+ times more.
>>>>>>
>>>>>> It's a good idea build the abstraction layer: queue <> channel
>> <>
>>>> dma-
>>>>>> controller.
>>>>>> In this way, the meaning of each layer is relatively easy to
>>>>>> distinguish literally.
>>>>>>
>>>>>> will fix in V2
>>>>>>
>>>>>
>>>>> After re-reading all the mails in this thread, I have found one
>> more
>>>> important high level detail still not decided:
>>>>>
>>>>> Bruce had suggested flattening the DMA channels, so each dmadev
>>>> represents a DMA channel. And DMA controllers with multiple DMA
>>>> channels will have to instantiate multiple dmadevs, one for each
>> DMA
>>>> channel.
>>>>>
>>>>> Just like a four port NIC instantiates four ethdevs.
>>>>>
>>>>> Then, like ethdevs, there would only be two abstraction layers:
>>>> dmadev <> queue, where a dmadev is a DMA channel on a DMA
>> controller.
>>>>>
>>>>> However, this assumes that the fast path functions on the
>> individual
>>>> DMA channels of a DMA controller can be accessed completely
>>>> independently and simultaneously by multiple threads. (Otherwise,
>> the
>>>> driver would need to implement critical regions or locking around
>>>> accessing the common registers in the DMA controller shared by the
>> DMA
>>>> channels.)
>>>>>
>>>>> Unless any of the DMA controller vendors claim that this
>> assumption
>>>> about independence of the DMA channels is wrong, I strongly support
>>>> Bruce's flattening suggestion.
>>>>
>>>> It is wrong from alteast octeontx2_dma PoV.
>>>>
>>>> # The PCI device is DMA controller where the driver/device is
>>>> mapped.(As device driver is based on PCI bus, We dont want to have
>>>> vdev for this)
>>>> # The PCI device has HW queue(s)
>>>> # Each HW queue has different channels.
>>>>
>>>> In the current configuration, we have only one queue per device and
>> it
>>>> has 4 channels. 4 channels are not threaded safe as it is based on
>>>> single queue.
>>>
>>> Please clarify "current configuration": Is that a configuration
>> modifiable by changing some software/driver, or is it the chip that was
>> built that way in the RTL code?
>>
>> We have 8 queues per SoC, Based on some of HW versions it can be
>> configured as (a) or (b) using FW settings.
>> a) One PCI devices with 8 Queues
>> b) 8 PCI devices with each one has one queue.
>>
>> Everyone is using mode (b) as it helps 8 different applications to use
>> DMA as if one application binds the PCI device other applications can
>> not use the same PCI device.
>> If one application needs 8 queues, it is possible that 8 dmadevice can
>> be bound to a single application with mode (b).
>>
>>
>> I think, in above way we can flatten to <device> <> <channel/queue>

I just look at dpaa2_qdma driver code, and found it seems OK to setup
one xxxdev for one queue.

>>
>>>
>>>>
>>>> I think, if we need to flatten it, I think, it makes sense to have
>>>> dmadev <> channel (and each channel can have thread-safe capability
>>>> based on how it mapped on HW queues based on the device driver
>>>> capability).
>>>
>>> The key question is how many threads can independently call data-
>> plane dmadev functions (rte_dma_copy() etc.) simultaneously. If I
>> understand your explanation correctly, only one - because you only have
>> one DMA device, and all access to it goes through a single hardware
>> queue.
>>>
>>> I just realized that although you only have one DMA Controller with
>> only one HW queue, your four DMA channels allows four sequentially
>> initiated transactions to be running simultaneously. Does the
>> application have any benefit by knowing that the dmadev can have
>> multiple ongoing transactions, or can the fast-path dmadev API hide
>> that ability?
>>
>> In my view it is better to hide and I have similar proposal at
>> http://mails.dpdk.org/archives/dev/2021-July/213141.html
>> --------------
>>>   7) Because data-plane APIs are not thread-safe, and user could
>> determine
>>>      virt-queue to HW-queue's map (at the queue-setup stage), so it
>> is user's
>>>      duty to ensure thread-safe.
>>
>> +1. But I am not sure how easy for the fast-path application to have
>> this logic,
>> Instead, I think, it is better to tell the capa for queue by driver
>> and in channel configuration,
>> the application can request for requirement (Is multiple producers enq
>> to the same HW queue or not).
>> Based on the request, the implementation can pick the correct function
>> pointer for enq.(lock vs lockless version if HW does not support
>> lockless)
> 
> +1 to that!
> 

add in channel configuration sound good.

>>
>> ------------------------
>>>
> 


  reply	other threads:[~2021-07-06  7:11 UTC|newest]

Thread overview: 79+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-15 13:22 [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library Chengwen Feng
2021-06-15 16:38 ` Bruce Richardson
2021-06-16  7:09   ` Morten Brørup
2021-06-16 10:17     ` fengchengwen
2021-06-16 12:09       ` Morten Brørup
2021-06-16 13:06       ` Bruce Richardson
2021-06-16 14:37       ` Jerin Jacob
2021-06-17  9:15         ` Bruce Richardson
2021-06-18  5:52           ` Jerin Jacob
2021-06-18  9:41             ` fengchengwen
2021-06-22 17:25               ` Jerin Jacob
2021-06-23  3:30                 ` fengchengwen
2021-06-23  7:21                   ` Jerin Jacob
2021-06-23  9:37                     ` Bruce Richardson
2021-06-23 11:40                       ` Jerin Jacob
2021-06-23 14:19                         ` Bruce Richardson
2021-06-24  6:49                           ` Jerin Jacob
2021-06-23  9:41                 ` Bruce Richardson
2021-06-23 10:10                   ` Morten Brørup
2021-06-23 11:46                   ` Jerin Jacob
2021-06-23 14:22                     ` Bruce Richardson
2021-06-18  9:55             ` Bruce Richardson
2021-06-22 17:31               ` Jerin Jacob
2021-06-22 19:17                 ` Bruce Richardson
2021-06-23  7:00                   ` Jerin Jacob
2021-06-16  9:41   ` fengchengwen
2021-06-16 17:31     ` Bruce Richardson
2021-06-16 18:08       ` Jerin Jacob
2021-06-16 19:13         ` Bruce Richardson
2021-06-17  7:42           ` Jerin Jacob
2021-06-17  8:00             ` Bruce Richardson
2021-06-18  5:16               ` Jerin Jacob
2021-06-18 10:03                 ` Bruce Richardson
2021-06-22 17:36                   ` Jerin Jacob
2021-06-17  9:48       ` fengchengwen
2021-06-17 11:02         ` Bruce Richardson
2021-06-17 14:18           ` Bruce Richardson
2021-06-18  8:52             ` fengchengwen
2021-06-18  9:30               ` Bruce Richardson
2021-06-22 17:51               ` Jerin Jacob
2021-06-23  3:50                 ` fengchengwen
2021-06-23 11:00                   ` Jerin Jacob
2021-06-23 14:56                   ` Bruce Richardson
2021-06-24 12:19                     ` fengchengwen
2021-06-26  3:59                       ` [dpdk-dev] dmadev discussion summary fengchengwen
2021-06-28 10:00                         ` Bruce Richardson
2021-06-28 11:14                           ` Ananyev, Konstantin
2021-06-28 12:53                             ` Bruce Richardson
2021-07-02 13:31                           ` fengchengwen
2021-07-01 15:01                         ` Jerin Jacob
2021-07-01 16:33                           ` Bruce Richardson
2021-07-02  7:39                             ` Morten Brørup
2021-07-02 10:05                               ` Bruce Richardson
2021-07-02 13:45                           ` fengchengwen
2021-07-02 14:57                             ` Morten Brørup
2021-07-03  0:32                               ` fengchengwen
2021-07-03  8:53                                 ` Morten Brørup
2021-07-03  9:08                                   ` Jerin Jacob
2021-07-03 12:24                                     ` Morten Brørup
2021-07-04  7:43                                       ` Jerin Jacob
2021-07-05 10:28                                         ` Morten Brørup
2021-07-06  7:11                                           ` fengchengwen [this message]
2021-07-03  9:45                                   ` fengchengwen
2021-07-03 12:00                                     ` Morten Brørup
2021-07-04  7:34                                       ` Jerin Jacob
2021-07-02  7:07                         ` Liang Ma
2021-07-02 13:59                           ` fengchengwen
2021-06-24  7:03                   ` [dpdk-dev] [RFC PATCH] dmadev: introduce DMA device library Jerin Jacob
2021-06-24  7:59                     ` Morten Brørup
2021-06-24  8:05                       ` Jerin Jacob
2021-06-23  5:34       ` Hu, Jiayu
2021-06-23 11:07         ` Jerin Jacob
2021-06-16  2:17 ` Wang, Haiyue
2021-06-16  8:04   ` Bruce Richardson
2021-06-16  8:16     ` Wang, Haiyue
2021-06-16 12:14 ` David Marchand
2021-06-16 13:11   ` Bruce Richardson
2021-06-16 16:48     ` Honnappa Nagarahalli
2021-06-16 19:10       ` Bruce Richardson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1d7e9c47-bbda-0395-c0e4-0de7d7511c3d@huawei.com \
    --to=fengchengwen@huawei.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=jerinj@marvell.com \
    --cc=jerinjacobk@gmail.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=mb@smartsharesystems.com \
    --cc=nipun.gupta@nxp.com \
    --cc=pkapoor@marvell.com \
    --cc=sburla@marvell.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).