RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process and inter-OS DMA

[Jerin Jacob <jerinj@marvell.com>]

> -----Original Message-----
> From: Vamsi Krishna Attunuru <vattunuru@marvell.com>
> Sent: Wednesday, September 24, 2025 9:45 AM
> To: fengchengwen <fengchengwen@huawei.com>; dev@dpdk.org;
> anatoly.burakov@intel.com
> Cc: thomas@monjalon.net; bruce.richardson@intel.com;
> vladimir.medvedkin@intel.com; anatoly.burakov@intel.com;
> kevin.laatz@intel.com; Jerin Jacob <jerinj@marvell.com>
> Subject: RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process and inter-OS
> DMA
> 
> Hi Feng, Anatoly,
> 
> Gentle ping for below review.
> 
> >-----Original Message-----
> >From: Vamsi Krishna Attunuru
> >Sent: Monday, September 22, 2025 5:19 PM
> >To: fengchengwen <fengchengwen@huawei.com>; dev@dpdk.org;
> >anatoly.burakov@intel.com
> >Cc: thomas@monjalon.net; bruce.richardson@intel.com;
> >vladimir.medvedkin@intel.com; anatoly.burakov@intel.com;
> >kevin.laatz@intel.com; Jerin Jacob <jerinj@marvell.com>
> >Subject: RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process and
> >inter- OS DMA
> >
> >Hi Feng,
> >
> >>Hi Vamsi, This commit change is more than discussed, it add control
> >>API which for group management. 1. Control API: I check this commit
> >>and Intel commit [1], it seem has a quite difference. I hope Intel
> >>guys can express views
> >>ZjQcmQRYFpfptBannerEnd
> >>Hi Vamsi,
> >>
> >>This commit change is more than discussed, it add control API which
> >>for group management.
> >>
> >>1. Control API: I check this commit and Intel commit [1], it seem has
> >>a quite difference.
> >>   I hope Intel guys can express views. I prefer not add this part if
> >>no
> >response.
> >
> >This new feature needs to be securely managed through control APIs. It
> >would be extremely helpful if the folks at Intel and you as well could
> >provide support or inputs on this.


Beyond adding Intel folks to this thread, I don't see any further steps we can take to drive review at this stage.

That said, the table-based concept used in the current API may not be portable, and we may need improvements here.

Based on my understanding, DMA devices used for inter process copy can be classified into three categories:

Class A: Requires a pair of DMA devices (one on each end of process/domain) for data transfer. Marvell DMA devices fall into this category.
Class B: Requires only a single DMA device (one process/domain has a DMA device, the other process does not). Intel DMA devices fall here.
Class C: Other types of devices that we are not yet aware of.

Abstracting all of these under a single API will be challenging. Linux and other OSes do not provide control-plane APIs for this,
so DPDK must provide control plane mechanisms to support Class A, Class B, and Class C devices.

Proposal: Split development into separate sets:
-----------------------------------------------
Set A: Focus only on the datapath. Assume uint16_t *src_handle and uint16_t *dst_handle come from elsewhere (Class C).
Set B: Introduce capabilities for Class A devices with portable APIs (proposal below, without table concept).
Set C: Introduce capabilities for Class B devices and relevant APIs, to be added when needed.

We can merge Set A in the current release and move Set B to a next release _if_ review or support for Class A devices requires more time.

@fengchengwen Thoughts? 

Class A API Proposal:
---------------------
These APIs are based on a new capability flag for inter-process or inter-OS DMA transfers for Class A devices.


/** Creates an access group for pair-type inter-process or inter-OS DMA transfers. */
int rte_dma_access_pair_group_create(const struct rte_dma_dev *dev,
                                     rte_uuid_t process_id,
                                     rte_uuid_t token,
                                     uint16_t *group_id);

/** Destroys an access group once all participating devices have exited. */
int rte_dma_access_pair_group_destroy(const struct rte_dma_dev *dev,
                                      uint16_t group_id);

/** Allows a device to join an existing access group using a device handle and token. */
int rte_dma_access_pair_group_join(const struct rte_dma_dev *dev,
                                   uint16_t group_id,
                                   rte_uuid_t process_id,
                                   rte_uuid_t token,
                                   rte_dma_access_pair_leave_cb_t leave_cb);

/** Removes a device from an access group. */
int rte_dma_access_pair_group_leave(const struct rte_dma_dev *dev,
                                    uint16_t group_id);

/** Retrieves the source and destination handles for a given device within the group. */
int rte_dma_access_pair_gorup_src_dst_handles_get(const struct rte_dma_dev *dev,
                                            uint16_t group_id,
                                            rte_uuid_t src_process_id,
                                            rte_uuid_t dst_process_id,
                                            uint16_t *src_handle,
                                            uint16_t *dst_handle);


Parameters that need explanation:
--------------------------------
process_id: Unique ID for the process, generated via rte_uuid_ APIs
token: Provided by an administrative actor to grant access, similar to VFIO VF token creation used in VFIO PF driver.
leave_cb: Callback to notify when another side leaves the group


Example Workflow for Class A Inter-Domain DMA Transfer:
-------------------------------------------------------

This example demonstrates how three processes — p0, p1, and p2 — coordinate inter-domain DMA transfers using pair-type(Class A) DMA devices.

Step 1: Group Creation (p0)
Process p0 calls rte_dma_access_pair_group_create() with a unique process handle and token. A group_id is returned.

Step 2: Group Sharing
group_id and token are shared with p1 and p2 via IPC or shared memory.

Step 3: Group Joining (p1 & p2)
Processes p1 and p2 call rte_dma_access_pair_group_join() with their process id  and the shared token from admin

Step 4: Handle Discovery
Each process uses rte_dma_access_pair_gorup_src_dst_handles_get() to retrieve source and destination handles for other processes.

Step 5: Transfer Coordination
Using the handles, each process configures a virtual channel (vchan) and initiates DMA transfers.

Step 6: Group Teardown
When a process no longer needs to participate, it calls rte_dma_access_pair_group_leave(). Other processes are notified via the registered callback with rte_dma_access_pair_group_join().
Once all devices have exited, p0 calls rte_dma_access_pair_group_destroy() to clean up.


For Class B: We can add new capability flag and have new set of APIs rte_dma_access_master_ or so. When such devices comes or when intel wants to add it