RE: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna Attunuru <vattunuru@marvell.com>]

Hi Bruce, Vladimir, Anatoly

Regarding inter-device or inter-domain DMA capability, could please clarify if Intel idxd driver will support this feature.
I believe the changes Feng has suggested here are in line with the earlier "[PATCH v1 0/3] Add support for inter-domain
DMA operations" proposal. We are planning to implement this feature support in version 25.11.

Your feedback would be appreciated, we are aiming for a more generic solution.

Regards
Vamsi


>>On 2025/7/16 18:59, Vamsi Krishna Attunuru wrote:
>>>
>>>>
>>>> Thanks for the explanation.
>>>>
>>>> Let me tell you what I understand:
>>>> 1\ Two dmadev (must belong to the same DMA controller?) each
>>>> passthrough to diffent domain (VM or container) 2\ The kernel DMA
>>>> controller driver could config access groups --- there is a secure
>>>> mechanism
>>(like Intel IDPTE)
>>>>   and the two dmadev could communicate if the kernel DMA controller
>>>> driver has put them in the same access groups.
>>>> 3\ Application setup access group and get handle (maybe the new
>>'dev_idx'
>>>> which you announce in this commit),
>>>>   and then setup one vchan which config the handle.
>>>>   and later launch copy request based on this vchan.
>>>> 4\ The driver will pass the request to dmadev-1 hardware, and
>>>> dmadev-1 hardware will do some verification,
>>>>   and maybe use dmadev-2 stream ID for read/write operations?
>>>>
>>>> A few question about this:
>>>> 1\ What the prototype of 'dev_idx', is it uint16_t?
>>> Yes, it can be uint16_t and use two different dev_idx (src_dev_idx &
>>> dest_dev_idx) for read & write.
>>>
>>>> 2\ How to implement read/write between two dmadev ?  use two
>>>> different dev_idx? the first for read and the second for write?
>>> Yes, two different dev_idx will be used.
>>>
>>>>
>>>>
>>>> I also re-read the patchset "[PATCH v1 0/3] Add support for
>>>> inter-domain DMA operations", it introduce:
>>>> 1\ One 'int controller-id' in the rte_dma_info. which maybe used in
>>>> vendor- specific secure mechanism.
>>>> 2\ Two new OP_flag and two new datapath API.
>>>> The reason why this patch didn't continue (I guess) is whether setup
>>>> one new vchan. Yes, vchan was designed to represents different
>>>> transfer contexts. But each vchan has its own enqueue/dequeue/ring,
>>>> it more act like one logic dmadev, some of the hardware can fit this
>>>> model well, some may not (like Intel in this case).
>>>>
>>>>
>>>> So how about the following scheme:
>>>> 1\ Add inter-domain capability bits, for example:
>>>> RTE_DMA_CAPA_INTER_PROCESS_DOMAIN,
>>>> RTE_DMA_CAPA_INTER_OS_DOMAIN 2\ Add one domain_controller_id
>in
>>the
>>>> rte_dma_info which maybe used in vendor-specific secure mechanism.
>>>> 3\ Add four OP_FLAGs:
>>>> RTE_DMA_OP_FLAG_SRC_INTER_PROCESS_DOMAIN_HANDLE,
>>>> RTE_DMA_OP_FLAG_DST_INTER_PROCESS_DOMAIN_HANDLE
>>>>                      RTE_DMA_OP_FLAG_SRC_INTER_OS_DOMAIN_HANDLE,
>>>> RTE_DMA_OP_FLAG_DST_INTER_OS_DOMAIN_HANDLE
>>>> 4\ Reserved 32bit from flag parameter (which all enqueue API both
>>>> supports) as the src and dst handle.
>>>>   or only reserved 16bit from flag parameter if we restrict don't
>>>> support 3rd transfer.
>>>
>>> Yes, the above approach seems acceptable to me. I believe src & dst
>>> handles require 16-bit values. Reserving 32-bits from flag parameter
>>> would leave 32 flags available, which should be fine.
>>
>>Great
>>tip: there are still 24bit flag reserved after apply this scheme.
>>
>>Would like more comments.
>>
>
>If there are no major comments at this time, can we proceed with accepting
>and merging this notice in this release. Further review can continue once the
>RFC is available next month.
>
>Thanks & Regards
>Vamsi
>
>>>
>>>>
>>>> Thanks
>>>>
>>>> On 2025/7/15 13:35, Vamsi Krishna Attunuru wrote:
>>>>> Hi Feng,
>>>>>
>>>>> Thanks for depicting the feature use case.
>>>>>
>>>>> From the application’s perspective, inter VM/process communication
>>>>> is
>>>> required to exchange the src & dst buffer details, however the
>>>> specifics of this communication mechanism are outside the scope in
>>>> this context. Regarding the address translations, these buffer
>>>> addresses can be either IOVA as PA or IOVA as VA. The DMA hardware
>>>> must use the appropriate IOMMU stream IDs when initiating the DMA
>>>> transfers. For example, in the use case shown in the diagram,
>>>> dmadev-1 and dmadev-2 would join an access group managed by the
>>>> kernel DMA controller driver. This controller driver will configure
>>>> the access group on the DMA hardware, enabling the hardware to
>>>> select the correct stream IDs for read/write operations. New rte_dma
>>>> APIs could be introduced to join or leave the access group or to
>>>> query the access group details. Additionally, a secure token
>>>> mechanism (similar to
>>vfio-pci token) can be implemented to validate any dmadev attempting to
>>join the access group.
>>>>>
>>>>> Regards.
>>>>>
>>>>> From: fengchengwen <fengchengwen@huawei.com>
>>>>> Sent: Tuesday, July 15, 2025 6:29 AM
>>>>> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org;
>>>>> Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>>>>> kevin.laatz@intel.com; bruce.richardson@intel.com;
>>>>> mb@smartsharesystems.com
>>>>> Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net
>>>>> Subject: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device
>>>>> DMA capability support in dmadev
>>>>>
>>>>> Hi Vamsi, From the commit log, I guess this commit mainly want to
>>>>> meet following case: --------------- ---------------- | Container |
>>>>> | VirtMachine | | | | | | dmadev-1 | | dmadev2 | ---------------
>>>>> ---------------- |
>>>> | ------------------------------ ZjQcmQRYFpfptBannerStart Prioritize
>>>> | security for
>>>> external emails:
>>>>> Confirm sender and content safety before clicking links or opening
>>>> attachments
>>>>>     Report Suspicious
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us-
>2Dphishala
>>>>> r
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us-
>2Dphishala
>>>>> r  >>>>> m-
>>2D&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=WllrYaumVkxaWjgKto6E_r
>t
>>DQsh
>>>>>
>>hIhik2jkvzFyRhW8&m=3uFGFVHxC4YLkjWHNg9s9rNDHd_ozbhLepOYCAkiZK
>x
>>M0sQ0m
>>>>>
>>43gqgTQl1cK9koZ&s=3_mvLYuMWu7RbHD3mj21CP65O5JY8L8AK8oVFutdTW
>U
>>&e=
>>>>
>>ewt.proofpoint.com/EWT/v1/CRVmXkqW!tg3ZldV0Yr_wdSwWmT2aDdKMi-
>>>>
>>4rn2z58vFaxwfOeocS1P19w1BeRdyGs5sjnhV2rU_6m8MOWj4KFbuXKkKJIvc
>q
>>>> wWD2WEwJW_0$ >
>>>>> ZjQcmQRYFpfptBannerEnd
>>>>>
>>>>> Hi Vamsi,
>>>>>
>>>>>
>>>>>
>>>>> From the commit log, I guess this commit mainly want to meet
>>>>> following
>>>> case:
>>>>>
>>>>>
>>>>>
>>>>>      ---------------             ----------------
>>>>>
>>>>>      |  Container  |             |  VirtMachine |
>>>>>
>>>>>      |             |             |              |
>>>>>
>>>>>      |  dmadev-1   |             |   dmadev2    |
>>>>>
>>>>>      ---------------             ----------------
>>>>>
>>>>>            |                            |
>>>>>
>>>>>            ------------------------------
>>>>>
>>>>>
>>>>>
>>>>> App run in the container could launch DMA transfer from local
>>>>> buffer to the VirtMachine by config
>>>>>
>>>>> dmadev-1/2 (the dmadev-1/2 are passthrough to diffent OS domain).
>>>>>
>>>>>
>>>>>
>>>>> Could you explain how to use it from application perspective (for
>>>>> example address translation) and
>>>>>
>>>>> application & hardware restrictions?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> BTW: In this case, there are two OS domain communication, and I
>>>>> remember there are also inter-process
>>>>>
>>>>> DMA RFC, so maybe we could design more generic solution if you
>>>>> provide
>>>> more info.
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>> On 2025/7/10 16:51, Vamsi Krishna wrote:
>>>>>
>>>>>> From: Vamsi Attunuru
>>>>>> <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
>>>>>
>>>>>>
>>>>>
>>>>>> Modern DMA hardware supports data transfer between multiple
>>>>>
>>>>>> DMA devices, enabling data communication across isolated domains
>>>>>> or
>>>>>
>>>>>> containers. To facilitate this, the ``dmadev`` library requires
>>>>>> changes
>>>>>
>>>>>> to allow devices to register with or unregisters from DMA groups
>>>>>> for
>>>>>
>>>>>> inter-device communication. This feature is planned for inclusion
>>>>>
>>>>>> in DPDK 25.11.
>>>>>
>>>>>>
>>>>>
>>>>>> Signed-off-by: Vamsi Attunuru
>>>>>> <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
>>>>>
>>>>>> ---
>>>>>
>>>>>>  doc/guides/rel_notes/deprecation.rst | 7 +++++++
>>>>>
>>>>>>  1 file changed, 7 insertions(+)
>>>>>
>>>>>>
>>>>>
>>>>>> diff --git a/doc/guides/rel_notes/deprecation.rst
>>>>>> b/doc/guides/rel_notes/deprecation.rst
>>>>>
>>>>>> index e2d4125308..46836244dd 100644
>>>>>
>>>>>> --- a/doc/guides/rel_notes/deprecation.rst
>>>>>
>>>>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>>>>
>>>>>> @@ -152,3 +152,10 @@ Deprecation Notices
>>>>>
>>>>>>  * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
>>>>>
>>>>>>    ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to
>DPDK.
>>>>>
>>>>>>    Those API functions are used internally by DPDK core and netvsc
>PMD.
>>>>>
>>>>>> +
>>>>>
>>>>>> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will
>>>>>> +be added
>>>>>
>>>>>> +  to advertise DMA device's inter-device DMA copy capability. To
>>>>>> + enable
>>>>>
>>>>>> +  this functionality, a few dmadev APIs will be added to
>>>>>> + configure the DMA
>>>>>
>>>>>> +  access groups, facilitating coordinated data communication
>>>>>> + between
>>>> devices.
>>>>>
>>>>>> +  A new ``dev_idx`` field will be added to the ``struct
>>>>>> + rte_dma_vchan_conf``
>>>>>
>>>>>> +  structure to configure a vchan for data transfers between any
>>>>>> + two DMA
>>>> devices.
>>>>>
>>>>>
>>>