[PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna]

From: Vamsi Attunuru <vattunuru@marvell.com>

Modern DMA hardware supports data transfer between multiple
DMA devices, enabling data communication across isolated domains or
containers. To facilitate this, the ``dmadev`` library requires changes
to allow devices to register with or unregisters from DMA groups for
inter-device communication. This feature is planned for inclusion
in DPDK 25.11.

Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---
 doc/guides/rel_notes/deprecation.rst | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index e2d4125308..46836244dd 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -152,3 +152,10 @@ Deprecation Notices
 * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
   ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
   Those API functions are used internally by DPDK core and netvsc PMD.
+
+* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be added
+  to advertise DMA device's inter-device DMA copy capability. To enable
+  this functionality, a few dmadev APIs will be added to configure the DMA
+  access groups, facilitating coordinated data communication between devices.
+  A new ``dev_idx`` field will be added to the ``struct rte_dma_vchan_conf``
+  structure to configure a vchan for data transfers between any two DMA devices.
-- 
2.34.1

RE: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna Attunuru]

Hi,

Gentle ping for reviews.

>-----Original Message-----
>From: Vamsi Krishna <vattunuru@marvell.com>
>Sent: Thursday, July 10, 2025 2:21 PM
>To: dev@dpdk.org; fengchengwen@huawei.com; Pavan Nikhilesh
>Bhagavatula <pbhagavatula@marvell.com>; kevin.laatz@intel.com;
>bruce.richardson@intel.com; mb@smartsharesystems.com
>Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net; Vamsi Krishna
>Attunuru <vattunuru@marvell.com>
>Subject: [PATCH v0 1/1] doc: announce inter-device DMA capability support in
>dmadev
>
>From: Vamsi Attunuru <vattunuru@marvell.com>
>
>Modern DMA hardware supports data transfer between multiple DMA
>devices, enabling data communication across isolated domains or containers.
>To facilitate this, the ``dmadev`` library requires changes to allow devices to
>register with or unregisters from DMA groups for inter-device communication.
>This feature is planned for inclusion in DPDK 25.11.
>
>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>---
> doc/guides/rel_notes/deprecation.rst | 7 +++++++
> 1 file changed, 7 insertions(+)
>
>diff --git a/doc/guides/rel_notes/deprecation.rst
>b/doc/guides/rel_notes/deprecation.rst
>index e2d4125308..46836244dd 100644
>--- a/doc/guides/rel_notes/deprecation.rst
>+++ b/doc/guides/rel_notes/deprecation.rst
>@@ -152,3 +152,10 @@ Deprecation Notices
> * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
>   ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
>   Those API functions are used internally by DPDK core and netvsc PMD.
>+
>+* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be
>+added
>+  to advertise DMA device's inter-device DMA copy capability. To enable
>+  this functionality, a few dmadev APIs will be added to configure the
>+DMA
>+  access groups, facilitating coordinated data communication between
>devices.
>+  A new ``dev_idx`` field will be added to the ``struct
>+rte_dma_vchan_conf``
>+  structure to configure a vchan for data transfers between any two DMA
>devices.
>--
>2.34.1

RE: [EXTERNAL] [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Anoob Joseph]

> From: Vamsi Attunuru <mailto:vattunuru@marvell.com>
> 
> Modern DMA hardware supports data transfer between multiple
> DMA devices, enabling data communication across isolated domains or
> containers. To facilitate this, the ``dmadev`` library requires changes
> to allow devices to register with or unregisters from DMA groups for
> inter-device communication. This feature is planned for inclusion
> in DPDK 25.11.
> 
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>

Acked-by: Anoob Joseph <anoobj@marvell.com>

Re: [EXTERNAL] [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Pavan Nikhilesh Bhagavatula]

>> From: Vamsi Attunuru <mailto:vattunuru@marvell.com>
>>
>> Modern DMA hardware supports data transfer between multiple
>> DMA devices, enabling data communication across isolated domains or
>> containers. To facilitate this, the ``dmadev`` library requires changes
>> to allow devices to register with or unregisters from DMA groups for
>> inter-device communication. This feature is planned for inclusion
>> in DPDK 25.11.
>>
>> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>
>Acked-by: Anoob Joseph <anoobj@marvell.com>

Acked-by: Pavan Nikhilesh <pbhagavatula@marvell.com>

>
>
>

Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Medvedkin, Vladimir]

Hi Vamsi,

Could you please tell us more about usecases for this feature?

Thanks!

On 7/10/2025 9:51 AM, Vamsi Krishna wrote:
> From: Vamsi Attunuru <vattunuru@marvell.com>
>
> Modern DMA hardware supports data transfer between multiple
> DMA devices, enabling data communication across isolated domains or
> containers. To facilitate this, the ``dmadev`` library requires changes
> to allow devices to register with or unregisters from DMA groups for
> inter-device communication. This feature is planned for inclusion
> in DPDK 25.11.
>
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
>   doc/guides/rel_notes/deprecation.rst | 7 +++++++
>   1 file changed, 7 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index e2d4125308..46836244dd 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -152,3 +152,10 @@ Deprecation Notices
>   * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
>     ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
>     Those API functions are used internally by DPDK core and netvsc PMD.
> +
> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be added
> +  to advertise DMA device's inter-device DMA copy capability. To enable
> +  this functionality, a few dmadev APIs will be added to configure the DMA
> +  access groups, facilitating coordinated data communication between devices.
> +  A new ``dev_idx`` field will be added to the ``struct rte_dma_vchan_conf``
> +  structure to configure a vchan for data transfers between any two DMA devices.

-- 
Regards,
Vladimir

RE: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna Attunuru]

Hi Feng,

Kindly review below.

Regards
Vamsi

>-----Original Message-----
>From: Vamsi Krishna Attunuru <vattunuru@marvell.com>
>Sent: Friday, July 11, 2025 8:35 PM
>To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org;
>fengchengwen@huawei.com; Pavan Nikhilesh Bhagavatula
><pbhagavatula@marvell.com>; kevin.laatz@intel.com;
>bruce.richardson@intel.com; mb@smartsharesystems.com
>Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net
>Subject: RE: [PATCH v0 1/1] doc: announce inter-device DMA capability
>support in dmadev
>
>Hi,
>
>Gentle ping for reviews.
>
>>-----Original Message-----
>>From: Vamsi Krishna <vattunuru@marvell.com>
>>Sent: Thursday, July 10, 2025 2:21 PM
>>To: dev@dpdk.org; fengchengwen@huawei.com; Pavan Nikhilesh
>Bhagavatula
>><pbhagavatula@marvell.com>; kevin.laatz@intel.com;
>>bruce.richardson@intel.com; mb@smartsharesystems.com
>>Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net; Vamsi
>>Krishna Attunuru <vattunuru@marvell.com>
>>Subject: [PATCH v0 1/1] doc: announce inter-device DMA capability
>>support in dmadev
>>
>>From: Vamsi Attunuru <vattunuru@marvell.com>
>>
>>Modern DMA hardware supports data transfer between multiple DMA
>>devices, enabling data communication across isolated domains or containers.
>>To facilitate this, the ``dmadev`` library requires changes to allow
>>devices to register with or unregisters from DMA groups for inter-device
>communication.
>>This feature is planned for inclusion in DPDK 25.11.
>>
>>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>>---
>> doc/guides/rel_notes/deprecation.rst | 7 +++++++
>> 1 file changed, 7 insertions(+)
>>
>>diff --git a/doc/guides/rel_notes/deprecation.rst
>>b/doc/guides/rel_notes/deprecation.rst
>>index e2d4125308..46836244dd 100644
>>--- a/doc/guides/rel_notes/deprecation.rst
>>+++ b/doc/guides/rel_notes/deprecation.rst
>>@@ -152,3 +152,10 @@ Deprecation Notices
>> * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
>>   ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
>>   Those API functions are used internally by DPDK core and netvsc PMD.
>>+
>>+* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be
>>+added
>>+  to advertise DMA device's inter-device DMA copy capability. To
>>+enable
>>+  this functionality, a few dmadev APIs will be added to configure the
>>+DMA
>>+  access groups, facilitating coordinated data communication between
>>devices.
>>+  A new ``dev_idx`` field will be added to the ``struct
>>+rte_dma_vchan_conf``
>>+  structure to configure a vchan for data transfers between any two
>>+DMA
>>devices.
>>--
>>2.34.1

RE: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna Attunuru]

[-- Attachment #1: Type: text/plain, Size: 3146 bytes --]

Hi Vladimir,

Currently, mem-to-mem dma transfers happen within a single process address space. This feature aims to extend support for transfers across multiple process address spaces to meet the requirements of inter-process or inter-VM networking. In environments where share memory is not feasible due to security concerns, this feature allows controlled software to use DMA for packet transfers as needed.

Regards.

From: Medvedkin, Vladimir <vladimir.medvedkin@intel.com>
Sent: Monday, July 14, 2025 3:47 PM
To: dev@dpdk.org
Cc: Bruce Richardson <bruce.richardson@intel.com>
Subject: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

Hi Vamsi, Could you please tell us more about usecases for this feature? Thanks! On 7/10/2025 9: 51 AM, Vamsi Krishna wrote: > From: Vamsi Attunuru <vattunuru@ marvell. com> > > Modern DMA hardware supports data transfer between
ZjQcmQRYFpfptBannerStart
Prioritize security for external emails:
Confirm sender and content safety before clicking links or opening attachments
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/CRVmXkqW!ti3Yn35UY3X61OwcGRf6bRAsmnzalHb1b3IuGKm4KAmFKsaahO5TQ00KAcbC6zX9rZkmmbglmeBA6oy8diWgp0Bz2fptvfWCiA$>   ‌
ZjQcmQRYFpfptBannerEnd

Hi Vamsi,



Could you please tell us more about usecases for this feature?



Thanks!



On 7/10/2025 9:51 AM, Vamsi Krishna wrote:

> From: Vamsi Attunuru <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>

>

> Modern DMA hardware supports data transfer between multiple

> DMA devices, enabling data communication across isolated domains or

> containers. To facilitate this, the ``dmadev`` library requires changes

> to allow devices to register with or unregisters from DMA groups for

> inter-device communication. This feature is planned for inclusion

> in DPDK 25.11.

>

> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>

> ---

>   doc/guides/rel_notes/deprecation.rst | 7 +++++++

>   1 file changed, 7 insertions(+)

>

> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst

> index e2d4125308..46836244dd 100644

> --- a/doc/guides/rel_notes/deprecation.rst

> +++ b/doc/guides/rel_notes/deprecation.rst

> @@ -152,3 +152,10 @@ Deprecation Notices

>   * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in

>     ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.

>     Those API functions are used internally by DPDK core and netvsc PMD.

> +

> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be added

> +  to advertise DMA device's inter-device DMA copy capability. To enable

> +  this functionality, a few dmadev APIs will be added to configure the DMA

> +  access groups, facilitating coordinated data communication between devices.

> +  A new ``dev_idx`` field will be added to the ``struct rte_dma_vchan_conf``

> +  structure to configure a vchan for data transfers between any two DMA devices.



--

Regards,

Vladimir



[-- Attachment #2: Type: text/html, Size: 12230 bytes --]

Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[fengchengwen]

Hi Vamsi,

From the commit log, I guess this commit mainly want to meet following case:

     ---------------             ----------------
     |  Container  |             |  VirtMachine |
     |             |             |              |
     |  dmadev-1   |             |   dmadev2    |
     ---------------             ----------------
           |                            |
           ------------------------------

App run in the container could launch DMA transfer from local buffer to the VirtMachine by config
dmadev-1/2 (the dmadev-1/2 are passthrough to diffent OS domain).

Could you explain how to use it from application perspective (for example address translation) and
application & hardware restrictions?


BTW: In this case, there are two OS domain communication, and I remember there are also inter-process
DMA RFC, so maybe we could design more generic solution if you provide more info.

Thanks

On 2025/7/10 16:51, Vamsi Krishna wrote:
> From: Vamsi Attunuru <vattunuru@marvell.com>
> 
> Modern DMA hardware supports data transfer between multiple
> DMA devices, enabling data communication across isolated domains or
> containers. To facilitate this, the ``dmadev`` library requires changes
> to allow devices to register with or unregisters from DMA groups for
> inter-device communication. This feature is planned for inclusion
> in DPDK 25.11.
> 
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
>  doc/guides/rel_notes/deprecation.rst | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index e2d4125308..46836244dd 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -152,3 +152,10 @@ Deprecation Notices
>  * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
>    ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
>    Those API functions are used internally by DPDK core and netvsc PMD.
> +
> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be added
> +  to advertise DMA device's inter-device DMA copy capability. To enable
> +  this functionality, a few dmadev APIs will be added to configure the DMA
> +  access groups, facilitating coordinated data communication between devices.
> +  A new ``dev_idx`` field will be added to the ``struct rte_dma_vchan_conf``
> +  structure to configure a vchan for data transfers between any two DMA devices.

RE: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna Attunuru]

[-- Attachment #1: Type: text/plain, Size: 4802 bytes --]

Hi Feng,

Thanks for depicting the feature use case.

From the application’s perspective, inter VM/process communication is required to exchange the src & dst buffer details, however the specifics of this communication mechanism are outside the scope in this context. Regarding the address translations, these buffer addresses can be either IOVA as PA or IOVA as VA. The DMA hardware must use the appropriate IOMMU stream IDs when initiating the DMA transfers. For example, in the use case shown in the diagram, dmadev-1 and dmadev-2 would join an access group managed by the kernel DMA controller driver. This controller driver will configure the access group on the DMA hardware, enabling the hardware to select the correct stream IDs for read/write operations. New rte_dma APIs could be introduced to join or leave the access group or to query the access group details. Additionally, a secure token mechanism (similar to vfio-pci token) can be implemented to validate any dmadev attempting to join the access group.

Regards.

From: fengchengwen <fengchengwen@huawei.com>
Sent: Tuesday, July 15, 2025 6:29 AM
To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org; Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; kevin.laatz@intel.com; bruce.richardson@intel.com; mb@smartsharesystems.com
Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net
Subject: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

Hi Vamsi, From the commit log, I guess this commit mainly want to meet following case: --------------- ---------------- | Container | | VirtMachine | | | | | | dmadev-1 | | dmadev2 | --------------- ---------------- | | ------------------------------
ZjQcmQRYFpfptBannerStart
Prioritize security for external emails:
Confirm sender and content safety before clicking links or opening attachments
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/CRVmXkqW!tg3ZldV0Yr_wdSwWmT2aDdKMi-4rn2z58vFaxwfOeocS1P19w1BeRdyGs5sjnhV2rU_6m8MOWj4KFbuXKkKJIvcqwWD2WEwJW_0$>   ‌
ZjQcmQRYFpfptBannerEnd

Hi Vamsi,



From the commit log, I guess this commit mainly want to meet following case:



     ---------------             ----------------

     |  Container  |             |  VirtMachine |

     |             |             |              |

     |  dmadev-1   |             |   dmadev2    |

     ---------------             ----------------

           |                            |

           ------------------------------



App run in the container could launch DMA transfer from local buffer to the VirtMachine by config

dmadev-1/2 (the dmadev-1/2 are passthrough to diffent OS domain).



Could you explain how to use it from application perspective (for example address translation) and

application & hardware restrictions?





BTW: In this case, there are two OS domain communication, and I remember there are also inter-process

DMA RFC, so maybe we could design more generic solution if you provide more info.



Thanks



On 2025/7/10 16:51, Vamsi Krishna wrote:

> From: Vamsi Attunuru <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>

>

> Modern DMA hardware supports data transfer between multiple

> DMA devices, enabling data communication across isolated domains or

> containers. To facilitate this, the ``dmadev`` library requires changes

> to allow devices to register with or unregisters from DMA groups for

> inter-device communication. This feature is planned for inclusion

> in DPDK 25.11.

>

> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>

> ---

>  doc/guides/rel_notes/deprecation.rst | 7 +++++++

>  1 file changed, 7 insertions(+)

>

> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst

> index e2d4125308..46836244dd 100644

> --- a/doc/guides/rel_notes/deprecation.rst

> +++ b/doc/guides/rel_notes/deprecation.rst

> @@ -152,3 +152,10 @@ Deprecation Notices

>  * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in

>    ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.

>    Those API functions are used internally by DPDK core and netvsc PMD.

> +

> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be added

> +  to advertise DMA device's inter-device DMA copy capability. To enable

> +  this functionality, a few dmadev APIs will be added to configure the DMA

> +  access groups, facilitating coordinated data communication between devices.

> +  A new ``dev_idx`` field will be added to the ``struct rte_dma_vchan_conf``

> +  structure to configure a vchan for data transfers between any two DMA devices.



[-- Attachment #2: Type: text/html, Size: 16233 bytes --]

Re: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[fengchengwen]

Hi Vamsi,

Thanks for the explanation.

Let me tell you what I understand:
1\ Two dmadev (must belong to the same DMA controller?) each passthrough to diffent domain (VM or container)
2\ The kernel DMA controller driver could config access groups --- there is a secure mechanism (like Intel IDPTE)
   and the two dmadev could communicate if the kernel DMA controller driver has put them in the same access groups.
3\ Application setup access group and get handle (maybe the new 'dev_idx' which you announce in this commit),
   and then setup one vchan which config the handle.
   and later launch copy request based on this vchan.
4\ The driver will pass the request to dmadev-1 hardware, and dmadev-1 hardware will do some verification,
   and maybe use dmadev-2 stream ID for read/write operations?

A few question about this:
1\ What the prototype of 'dev_idx', is it uint16_t?
2\ How to implement read/write between two dmadev ?  use two different dev_idx? the first for read and the second for write?


I also re-read the patchset "[PATCH v1 0/3] Add support for inter-domain DMA operations", it introduce:
1\ One 'int controller-id' in the rte_dma_info. which maybe used in vendor-specific secure mechanism.
2\ Two new OP_flag and two new datapath API.
The reason why this patch didn't continue (I guess) is whether setup one new vchan. Yes, vchan was designed
to represents different transfer contexts. But each vchan has its own enqueue/dequeue/ring, it more act like
one logic dmadev, some of the hardware can fit this model well, some may not (like Intel in this case).


So how about the following scheme:
1\ Add inter-domain capability bits, for example: RTE_DMA_CAPA_INTER_PROCESS_DOMAIN, RTE_DMA_CAPA_INTER_OS_DOMAIN
2\ Add one domain_controller_id in the rte_dma_info which maybe used in vendor-specific secure mechanism.
3\ Add four OP_FLAGs: RTE_DMA_OP_FLAG_SRC_INTER_PROCESS_DOMAIN_HANDLE, RTE_DMA_OP_FLAG_DST_INTER_PROCESS_DOMAIN_HANDLE
                      RTE_DMA_OP_FLAG_SRC_INTER_OS_DOMAIN_HANDLE, RTE_DMA_OP_FLAG_DST_INTER_OS_DOMAIN_HANDLE
4\ Reserved 32bit from flag parameter (which all enqueue API both supports) as the src and dst handle.
   or only reserved 16bit from flag parameter if we restrict don't support 3rd transfer.

Thanks

On 2025/7/15 13:35, Vamsi Krishna Attunuru wrote:
> Hi Feng,
> 
> Thanks for depicting the feature use case.
> 
> From the application’s perspective, inter VM/process communication is required to exchange the src & dst buffer details, however the specifics of this communication mechanism are outside the scope in this context. Regarding the address translations, these buffer addresses can be either IOVA as PA or IOVA as VA. The DMA hardware must use the appropriate IOMMU stream IDs when initiating the DMA transfers. For example, in the use case shown in the diagram, dmadev-1 and dmadev-2 would join an access group managed by the kernel DMA controller driver. This controller driver will configure the access group on the DMA hardware, enabling the hardware to select the correct stream IDs for read/write operations. New rte_dma APIs could be introduced to join or leave the access group or to query the access group details. Additionally, a secure token mechanism (similar to vfio-pci token) can be implemented to validate any dmadev attempting to join the access group.
> 
> Regards.
> 
> From: fengchengwen <fengchengwen@huawei.com>
> Sent: Tuesday, July 15, 2025 6:29 AM
> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org; Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; kevin.laatz@intel.com; bruce.richardson@intel.com; mb@smartsharesystems.com
> Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net
> Subject: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev
> 
> Hi Vamsi, From the commit log, I guess this commit mainly want to meet following case: --------------- ---------------- | Container | | VirtMachine | | | | | | dmadev-1 | | dmadev2 | --------------- ---------------- | | ------------------------------
> ZjQcmQRYFpfptBannerStart
> Prioritize security for external emails:
> Confirm sender and content safety before clicking links or opening attachments
>     Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/CRVmXkqW!tg3ZldV0Yr_wdSwWmT2aDdKMi-4rn2z58vFaxwfOeocS1P19w1BeRdyGs5sjnhV2rU_6m8MOWj4KFbuXKkKJIvcqwWD2WEwJW_0$>   ‌
> ZjQcmQRYFpfptBannerEnd
> 
> Hi Vamsi,
> 
> 
> 
> From the commit log, I guess this commit mainly want to meet following case:
> 
> 
> 
>      ---------------             ----------------
> 
>      |  Container  |             |  VirtMachine |
> 
>      |             |             |              |
> 
>      |  dmadev-1   |             |   dmadev2    |
> 
>      ---------------             ----------------
> 
>            |                            |
> 
>            ------------------------------
> 
> 
> 
> App run in the container could launch DMA transfer from local buffer to the VirtMachine by config
> 
> dmadev-1/2 (the dmadev-1/2 are passthrough to diffent OS domain).
> 
> 
> 
> Could you explain how to use it from application perspective (for example address translation) and
> 
> application & hardware restrictions?
> 
> 
> 
> 
> 
> BTW: In this case, there are two OS domain communication, and I remember there are also inter-process
> 
> DMA RFC, so maybe we could design more generic solution if you provide more info.
> 
> 
> 
> Thanks
> 
> 
> 
> On 2025/7/10 16:51, Vamsi Krishna wrote:
> 
>> From: Vamsi Attunuru <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
> 
>>
> 
>> Modern DMA hardware supports data transfer between multiple
> 
>> DMA devices, enabling data communication across isolated domains or
> 
>> containers. To facilitate this, the ``dmadev`` library requires changes
> 
>> to allow devices to register with or unregisters from DMA groups for
> 
>> inter-device communication. This feature is planned for inclusion
> 
>> in DPDK 25.11.
> 
>>
> 
>> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
> 
>> ---
> 
>>  doc/guides/rel_notes/deprecation.rst | 7 +++++++
> 
>>  1 file changed, 7 insertions(+)
> 
>>
> 
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> 
>> index e2d4125308..46836244dd 100644
> 
>> --- a/doc/guides/rel_notes/deprecation.rst
> 
>> +++ b/doc/guides/rel_notes/deprecation.rst
> 
>> @@ -152,3 +152,10 @@ Deprecation Notices
> 
>>  * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
> 
>>    ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
> 
>>    Those API functions are used internally by DPDK core and netvsc PMD.
> 
>> +
> 
>> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be added
> 
>> +  to advertise DMA device's inter-device DMA copy capability. To enable
> 
>> +  this functionality, a few dmadev APIs will be added to configure the DMA
> 
>> +  access groups, facilitating coordinated data communication between devices.
> 
>> +  A new ``dev_idx`` field will be added to the ``struct rte_dma_vchan_conf``
> 
>> +  structure to configure a vchan for data transfers between any two DMA devices.
> 
> 

RE: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA capability support in dmadev

[Vamsi Krishna Attunuru]

>
>Thanks for the explanation.
>
>Let me tell you what I understand:
>1\ Two dmadev (must belong to the same DMA controller?) each passthrough
>to diffent domain (VM or container) 2\ The kernel DMA controller driver could
>config access groups --- there is a secure mechanism (like Intel IDPTE)
>   and the two dmadev could communicate if the kernel DMA controller driver
>has put them in the same access groups.
>3\ Application setup access group and get handle (maybe the new 'dev_idx'
>which you announce in this commit),
>   and then setup one vchan which config the handle.
>   and later launch copy request based on this vchan.
>4\ The driver will pass the request to dmadev-1 hardware, and dmadev-1
>hardware will do some verification,
>   and maybe use dmadev-2 stream ID for read/write operations?
>
>A few question about this:
>1\ What the prototype of 'dev_idx', is it uint16_t?
Yes, it can be uint16_t and use two different dev_idx (src_dev_idx & dest_dev_idx)
for read & write.

>2\ How to implement read/write between two dmadev ?  use two different
>dev_idx? the first for read and the second for write?
Yes, two different dev_idx will be used.

>
>
>I also re-read the patchset "[PATCH v1 0/3] Add support for inter-domain
>DMA operations", it introduce:
>1\ One 'int controller-id' in the rte_dma_info. which maybe used in vendor-
>specific secure mechanism.
>2\ Two new OP_flag and two new datapath API.
>The reason why this patch didn't continue (I guess) is whether setup one new
>vchan. Yes, vchan was designed to represents different transfer contexts. But
>each vchan has its own enqueue/dequeue/ring, it more act like one logic
>dmadev, some of the hardware can fit this model well, some may not (like
>Intel in this case).
>
>
>So how about the following scheme:
>1\ Add inter-domain capability bits, for example:
>RTE_DMA_CAPA_INTER_PROCESS_DOMAIN,
>RTE_DMA_CAPA_INTER_OS_DOMAIN 2\ Add one domain_controller_id in
>the rte_dma_info which maybe used in vendor-specific secure mechanism.
>3\ Add four OP_FLAGs:
>RTE_DMA_OP_FLAG_SRC_INTER_PROCESS_DOMAIN_HANDLE,
>RTE_DMA_OP_FLAG_DST_INTER_PROCESS_DOMAIN_HANDLE
>                      RTE_DMA_OP_FLAG_SRC_INTER_OS_DOMAIN_HANDLE,
>RTE_DMA_OP_FLAG_DST_INTER_OS_DOMAIN_HANDLE
>4\ Reserved 32bit from flag parameter (which all enqueue API both supports)
>as the src and dst handle.
>   or only reserved 16bit from flag parameter if we restrict don't support 3rd
>transfer.

Yes, the above approach seems acceptable to me. I believe src & dst handles require
16-bit values. Reserving 32-bits from flag parameter would leave 32 flags available,
which should be fine.

>
>Thanks
>
>On 2025/7/15 13:35, Vamsi Krishna Attunuru wrote:
>> Hi Feng,
>>
>> Thanks for depicting the feature use case.
>>
>> From the application’s perspective, inter VM/process communication is
>required to exchange the src & dst buffer details, however the specifics of this
>communication mechanism are outside the scope in this context. Regarding
>the address translations, these buffer addresses can be either IOVA as PA or
>IOVA as VA. The DMA hardware must use the appropriate IOMMU stream IDs
>when initiating the DMA transfers. For example, in the use case shown in the
>diagram, dmadev-1 and dmadev-2 would join an access group managed by
>the kernel DMA controller driver. This controller driver will configure the
>access group on the DMA hardware, enabling the hardware to select the
>correct stream IDs for read/write operations. New rte_dma APIs could be
>introduced to join or leave the access group or to query the access group
>details. Additionally, a secure token mechanism (similar to vfio-pci token) can
>be implemented to validate any dmadev attempting to join the access group.
>>
>> Regards.
>>
>> From: fengchengwen <fengchengwen@huawei.com>
>> Sent: Tuesday, July 15, 2025 6:29 AM
>> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org;
>> Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>> kevin.laatz@intel.com; bruce.richardson@intel.com;
>> mb@smartsharesystems.com
>> Cc: Jerin Jacob <jerinj@marvell.com>; thomas@monjalon.net
>> Subject: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA
>> capability support in dmadev
>>
>> Hi Vamsi, From the commit log, I guess this commit mainly want to meet
>> following case: --------------- ---------------- | Container | |
>> VirtMachine | | | | | | dmadev-1 | | dmadev2 | --------------- ---------------- |
>| ------------------------------ ZjQcmQRYFpfptBannerStart Prioritize security for
>external emails:
>> Confirm sender and content safety before clicking links or opening
>attachments
>>     Report Suspicious  <https://us-phishalarm-
>ewt.proofpoint.com/EWT/v1/CRVmXkqW!tg3ZldV0Yr_wdSwWmT2aDdKMi-
>4rn2z58vFaxwfOeocS1P19w1BeRdyGs5sjnhV2rU_6m8MOWj4KFbuXKkKJIvcq
>wWD2WEwJW_0$ > 
>> ZjQcmQRYFpfptBannerEnd
>>
>> Hi Vamsi,
>>
>>
>>
>> From the commit log, I guess this commit mainly want to meet following
>case:
>>
>>
>>
>>      ---------------             ----------------
>>
>>      |  Container  |             |  VirtMachine |
>>
>>      |             |             |              |
>>
>>      |  dmadev-1   |             |   dmadev2    |
>>
>>      ---------------             ----------------
>>
>>            |                            |
>>
>>            ------------------------------
>>
>>
>>
>> App run in the container could launch DMA transfer from local buffer
>> to the VirtMachine by config
>>
>> dmadev-1/2 (the dmadev-1/2 are passthrough to diffent OS domain).
>>
>>
>>
>> Could you explain how to use it from application perspective (for
>> example address translation) and
>>
>> application & hardware restrictions?
>>
>>
>>
>>
>>
>> BTW: In this case, there are two OS domain communication, and I
>> remember there are also inter-process
>>
>> DMA RFC, so maybe we could design more generic solution if you provide
>more info.
>>
>>
>>
>> Thanks
>>
>>
>>
>> On 2025/7/10 16:51, Vamsi Krishna wrote:
>>
>>> From: Vamsi Attunuru
>>> <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
>>
>>>
>>
>>> Modern DMA hardware supports data transfer between multiple
>>
>>> DMA devices, enabling data communication across isolated domains or
>>
>>> containers. To facilitate this, the ``dmadev`` library requires
>>> changes
>>
>>> to allow devices to register with or unregisters from DMA groups for
>>
>>> inter-device communication. This feature is planned for inclusion
>>
>>> in DPDK 25.11.
>>
>>>
>>
>>> Signed-off-by: Vamsi Attunuru
>>> <vattunuru@marvell.com<mailto:vattunuru@marvell.com>>
>>
>>> ---
>>
>>>  doc/guides/rel_notes/deprecation.rst | 7 +++++++
>>
>>>  1 file changed, 7 insertions(+)
>>
>>>
>>
>>> diff --git a/doc/guides/rel_notes/deprecation.rst
>>> b/doc/guides/rel_notes/deprecation.rst
>>
>>> index e2d4125308..46836244dd 100644
>>
>>> --- a/doc/guides/rel_notes/deprecation.rst
>>
>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>
>>> @@ -152,3 +152,10 @@ Deprecation Notices
>>
>>>  * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
>>
>>>    ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
>>
>>>    Those API functions are used internally by DPDK core and netvsc PMD.
>>
>>> +
>>
>>> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` will be
>>> +added
>>
>>> +  to advertise DMA device's inter-device DMA copy capability. To
>>> + enable
>>
>>> +  this functionality, a few dmadev APIs will be added to configure
>>> + the DMA
>>
>>> +  access groups, facilitating coordinated data communication between
>devices.
>>
>>> +  A new ``dev_idx`` field will be added to the ``struct
>>> + rte_dma_vchan_conf``
>>
>>> +  structure to configure a vchan for data transfers between any two DMA
>devices.
>>
>>