RE: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna Attunuru <vattunuru@marvell.com>]

Gentle reminder to share your feedback.

>Hi Bruce, Vladimir, Anatoly
>
>Regarding inter-device or inter-domain DMA capability, could please clarify if
>Intel idxd driver will support this feature.
>I believe the changes Feng has suggested here are in line with the earlier
>"[PATCH v1 0/3] Add support for inter-domain DMA operations" proposal. We
>are planning to implement this feature support in version 25.11.
>
>Your feedback would be appreciated, we are aiming for a more generic
>solution.
>
>Regards
>Vamsi
>
>
>>>On 2025/7/16 18:59, Vamsi Krishna Attunuru wrote:
>>>>
>>>>>
>>>>> Thanks for the explanation.
>>>>>
>>>>> Let me tell you what I understand:
>>>>> 1\ Two dmadev (must belong to the same DMA controller?) each
>>>>> passthrough to diffent domain (VM or container) 2\ The kernel DMA
>>>>> controller driver could config access groups --- there is a secure
>>>>> mechanism
>>>(like Intel IDPTE)
>>>>>   and the two dmadev could communicate if the kernel DMA controller
>>>>> driver has put them in the same access groups.
>>>>> 3\ Application setup access group and get handle (maybe the new
>>>'dev_idx'
>>>>> which you announce in this commit),
>>>>>   and then setup one vchan which config the handle.
>>>>>   and later launch copy request based on this vchan.
>>>>> 4\ The driver will pass the request to dmadev-1 hardware, and
>>>>> dmadev-1 hardware will do some verification,
>>>>>   and maybe use dmadev-2 stream ID for read/write operations?
>>>>>
>>>>> A few question about this:
>>>>> 1\ What the prototype of 'dev_idx', is it uint16_t?
>>>> Yes, it can be uint16_t and use two different dev_idx (src_dev_idx &
>>>> dest_dev_idx) for read & write.
>>>>
>>>>> 2\ How to implement read/write between two dmadev ?  use two
>>>>> different dev_idx? the first for read and the second for write?
>>>> Yes, two different dev_idx will be used.
>>>>
>>>>>
>>>>>
>>>>> I also re-read the patchset "[PATCH v1 0/3] Add support for
>>>>> inter-domain DMA operations", it introduce:
>>>>> 1\ One 'int controller-id' in the rte_dma_info. which maybe used in
>>>>> vendor- specific secure mechanism.
>>>>> 2\ Two new OP_flag and two new datapath API.
>>>>> The reason why this patch didn't continue (I guess) is whether
>>>>> setup one new vchan. Yes, vchan was designed to represents
>>>>> different transfer contexts. But each vchan has its own
>>>>> enqueue/dequeue/ring, it more act like one logic dmadev, some of
>>>>> the hardware can fit this model well, some may not (like Intel in this
>case).
>>>>>
>>>>>
>>>>> So how about the following scheme:
>>>>> 1\ Add inter-domain capability bits, for example:
>>>>> RTE_DMA_CAPA_INTER_PROCESS_DOMAIN,
>>>>> RTE_DMA_CAPA_INTER_OS_DOMAIN 2\ Add one domain_controller_id
>>in
>>>the
>>>>> rte_dma_info which maybe used in vendor-specific secure mechanism.
>>>>> 3\ Add four OP_FLAGs:
>>>>> RTE_DMA_OP_FLAG_SRC_INTER_PROCESS_DOMAIN_HANDLE,
>>>>> RTE_DMA_OP_FLAG_DST_INTER_PROCESS_DOMAIN_HANDLE
>>>>>                      RTE_DMA_OP_FLAG_SRC_INTER_OS_DOMAIN_HANDLE,
>>>>> RTE_DMA_OP_FLAG_DST_INTER_OS_DOMAIN_HANDLE
>>>>> 4\ Reserved 32bit from flag parameter (which all enqueue API both
>>>>> supports) as the src and dst handle.
>>>>>   or only reserved 16bit from flag parameter if we restrict don't
>>>>> support 3rd transfer.
>>>>
>>>> Yes, the above approach seems acceptable to me. I believe src & dst
>>>> handles require 16-bit values. Reserving 32-bits from flag parameter
>>>> would leave 32 flags available, which should be fine.
>>>
>>>Great
>>>tip: there are still 24bit flag reserved after apply this scheme.
>>>
>>>Would like more comments.
>>>
>>
>>If there are no major comments at this time, can we proceed with
>>accepting and merging this notice in this release. Further review can
>>continue once the RFC is available next month.
>>
>>Thanks & Regards
>>Vamsi
>>
>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> On 2025/7/15 13:35, Vamsi Krishna Attunuru wrote:
>>>>>> Hi Feng,
>>>>>>
>>>>>> Thanks for depicting the feature use case.
>>>>>>
>>>>>> From the application’s perspective, inter VM/process communication
>>>>>> is
>>>>> required to exchange the src & dst buffer details, however the
>>>>> specifics of this communication mechanism are outside the scope in
>>>>> this context. Regarding the address translations, these buffer
>>>>> addresses can be either IOVA as PA or IOVA as VA. The DMA hardware
>>>>> must use the appropriate IOMMU stream IDs when initiating the DMA
>>>>> transfers. For example, in the use case shown in the diagram,
>>>>> dmadev-1 and dmadev-2 would join an access group managed by the
>>>>> kernel DMA controller driver. This controller driver will configure
>>>>> the access group on the DMA hardware, enabling the hardware to
>>>>> select the correct stream IDs for read/write operations. New
>>>>> rte_dma APIs could be introduced to join or leave the access group
>>>>> or to query the access group details. Additionally, a secure token
>>>>> mechanism (similar to
>>>vfio-pci token) can be implemented to validate any dmadev attempting
>>>to join the access group.
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> From: fengchengwen <fengchengwen@huawei.com>
>>>>>> Sent: Tuesday, July 15, 2025 6:29 AM
>>>>>> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>;
>dev@dpdk.org;
>>>>>> Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>>>>>> kevin.laatz@intel.com; bruce.richardson@intel.com;
>>>>>> mb@smartsharesystems.com
>>>>>> Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net
>>>>>> Subject: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device
>>>>>> DMA capability support in dmadev
>>>>>>
>>>>>> Hi Vamsi, From the commit log, I guess this commit mainly want to
>>>>>> meet following case: --------------- ---------------- | Container
>>>>>> |
>>>>>> | VirtMachine | | | | | | dmadev-1 | | dmadev2 | ---------------
>>>>>> ---------------- |
>>>>> | ------------------------------ ZjQcmQRYFpfptBannerStart
>>>>> | Prioritize security for
>>>>> external emails:
>>>>>> Confirm sender and content safety before clicking links or opening
>>>>> attachments
>>>>>>     Report Suspicious
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us-
>>2Dphishala
>>>>>> r
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us-
>>2Dphishala
>>>>>> r  >>>>> m-
>>>2D&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=WllrYaumVkxaWjgKto6E
>_r
>>t
>>>DQsh
>>>>>>
>>>hIhik2jkvzFyRhW8&m=3uFGFVHxC4YLkjWHNg9s9rNDHd_ozbhLepOYCAkiZ
>K
>>x
>>>M0sQ0m
>>>>>>
>>>43gqgTQl1cK9koZ&s=3_mvLYuMWu7RbHD3mj21CP65O5JY8L8AK8oVFutdT
>W
>>U
>>>&e=
>>>>>
>>>ewt.proofpoint.com/EWT/v1/CRVmXkqW!tg3ZldV0Yr_wdSwWmT2aDdKMi
>-
>>>>>
>>>4rn2z58vFaxwfOeocS1P19w1BeRdyGs5sjnhV2rU_6m8MOWj4KFbuXKkKJIv
>c
>>q
>>>>> wWD2WEwJW_0$ >
>>>>>> ZjQcmQRYFpfptBannerEnd
>>>>>>
>>>>>> Hi Vamsi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> From the commit log, I guess this commit mainly want to meet
>>>>>> following
>>>>> case:
>>>>>>
>>>>>>
>>>>>>
>>>>>>      ---------------             ----------------
>>>>>>
>>>>>>      |  Container  |             |  VirtMachine |
>>>>>>
>>>>>>      |             |             |              |
>>>>>>
>>>>>>      |  dmadev-1   |             |   dmadev2    |
>>>>>>
>>>>>>      ---------------             ----------------
>>>>>>
>>>>>>            |                            |
>>>>>>
>>>>>>            ------------------------------
>>>>>>
>>>>>>
>>>>>>
>>>>>> App run in the container could launch DMA transfer from local
>>>>>> buffer to the VirtMachine by config
>>>>>>
>>>>>> dmadev-1/2 (the dmadev-1/2 are passthrough to diffent OS domain).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Could you explain how to use it from application perspective (for
>>>>>> example address translation) and
>>>>>>
>>>>>> application & hardware restrictions?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> BTW: In this case, there are two OS domain communication, and I
>>>>>> remember there are also inter-process
>>>>>>
>>>>>> DMA RFC, so maybe we could design more generic solution if you
>>>>>> provide
>>>>> more info.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2025/7/10 16:51, Vamsi Krishna wrote:
>>>>>>
>>>>>>> From: Vamsi Attunuru
>>>>>>> <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>> Modern DMA hardware supports data transfer between multiple
>>>>>>
>>>>>>> DMA devices, enabling data communication across isolated domains
>>>>>>> or
>>>>>>
>>>>>>> containers. To facilitate this, the ``dmadev`` library requires
>>>>>>> changes
>>>>>>
>>>>>>> to allow devices to register with or unregisters from DMA groups
>>>>>>> for
>>>>>>
>>>>>>> inter-device communication. This feature is planned for inclusion
>>>>>>
>>>>>>> in DPDK 25.11.
>>>>>>
>>>>>>>
>>>>>>
>>>>>>> Signed-off-by: Vamsi Attunuru
>>>>>>> <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
>>>>>>
>>>>>>> ---
>>>>>>
>>>>>>>  doc/guides/rel_notes/deprecation.rst | 7 +++++++
>>>>>>
>>>>>>>  1 file changed, 7 insertions(+)
>>>>>>
>>>>>>>
>>>>>>
>>>>>>> diff --git a/doc/guides/rel_notes/deprecation.rst
>>>>>>> b/doc/guides/rel_notes/deprecation.rst
>>>>>>
>>>>>>> index e2d4125308..46836244dd 100644
>>>>>>
>>>>>>> --- a/doc/guides/rel_notes/deprecation.rst
>>>>>>
>>>>>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>>>>>
>>>>>>> @@ -152,3 +152,10 @@ Deprecation Notices
>>>>>>
>>>>>>>  * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
>>>>>>
>>>>>>>    ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to
>>DPDK.
>>>>>>
>>>>>>>    Those API functions are used internally by DPDK core and
>>>>>>> netvsc
>>PMD.
>>>>>>
>>>>>>> +
>>>>>>
>>>>>>> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will
>>>>>>> +be added
>>>>>>
>>>>>>> +  to advertise DMA device's inter-device DMA copy capability. To
>>>>>>> + enable
>>>>>>
>>>>>>> +  this functionality, a few dmadev APIs will be added to
>>>>>>> + configure the DMA
>>>>>>
>>>>>>> +  access groups, facilitating coordinated data communication
>>>>>>> + between
>>>>> devices.
>>>>>>
>>>>>>> +  A new ``dev_idx`` field will be added to the ``struct
>>>>>>> + rte_dma_vchan_conf``
>>>>>>
>>>>>>> +  structure to configure a vchan for data transfers between any
>>>>>>> + two DMA
>>>>> devices.
>>>>>>
>>>>>>
>>>>