RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process and inter-OS DMA

[Vamsi Krishna Attunuru <vattunuru@marvell.com>]

Hi Feng, Bruce, Kevin,

Could you please review the proposal below and share your thoughts.
we are seeking your feedback on this feature to ensure alignment
before the rc deadlines.

Regards
Vamsi

>>> -----Original Message-----
>>> From: Vamsi Krishna Attunuru <vattunuru@marvell.com>
>>> >From: Vamsi Krishna Attunuru
>>> >Sent: Monday, September 22, 2025 5:19 PM
>>> >To: fengchengwen <fengchengwen@huawei.com>; dev@dpdk.org;
>>> >anatoly.burakov@intel.com
>>> >Cc: thomas@monjalon.net; bruce.richardson@intel.com;
>>> >vladimir.medvedkin@intel.com; anatoly.burakov@intel.com;
>>> >kevin.laatz@intel.com; Jerin Jacob <jerinj@marvell.com>
>>> >Subject: RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process
>>> >and
>>> >inter- OS DMA
>>> >
>>> >Hi Feng,
>>> >
>>> >>Hi Vamsi, This commit change is more than discussed, it add control
>>> >>API which for group management. 1. Control API: I check this commit
>>> >>and Intel commit [1], it seem has a quite difference. I hope Intel
>>> >>guys can express views ZjQcmQRYFpfptBannerEnd Hi Vamsi,
>>> >>
>>> >>This commit change is more than discussed, it add control API which
>>> >>for group management.
>>> >>
>>> >>1. Control API: I check this commit and Intel commit [1], it seem
>>> >>has a quite difference.
>>> >>   I hope Intel guys can express views. I prefer not add this part
>>> >>if no
>>> >response.
>>> >
>>> >This new feature needs to be securely managed through control APIs.
>>> >It would be extremely helpful if the folks at Intel and you as well
>>> >could provide support or inputs on this.
>>
>>
>>Beyond adding Intel folks to this thread, I don't see any further steps
>>we can take to drive review at this stage.
>>
>>That said, the table-based concept used in the current API may not be
>>portable, and we may need improvements here.
>>
>>Based on my understanding, DMA devices used for inter process copy can
>>be classified into three categories:
>>
>>Class A: Requires a pair of DMA devices (one on each end of
>>process/domain) for data transfer. Marvell DMA devices fall into this
>category.
>>Class B: Requires only a single DMA device (one process/domain has a
>>DMA device, the other process does not). Intel DMA devices fall here.
>>Class C: Other types of devices that we are not yet aware of.
>>
>>Abstracting all of these under a single API will be challenging. Linux
>>and other OSes do not provide control-plane APIs for this, so DPDK must
>>provide control plane mechanisms to support Class A, Class B, and Class C
>devices.
>>
>>Proposal: Split development into separate sets:
>>-----------------------------------------------
>>Set A: Focus only on the datapath. Assume uint16_t *src_handle and
>>uint16_t *dst_handle come from elsewhere (Class C).
>>Set B: Introduce capabilities for Class A devices with portable APIs
>>(proposal below, without table concept).
>>Set C: Introduce capabilities for Class B devices and relevant APIs, to
>>be added when needed.
>>
>>We can merge Set A in the current release and move Set B to a next
>>release _if_ review or support for Class A devices requires more time.
>>
>>@fengchengwen Thoughts?
>>
>>Class A API Proposal:
>>---------------------
>>These APIs are based on a new capability flag for inter-process or
>>inter-OS DMA transfers for Class A devices.
>>
>>
>>/** Creates an access group for pair-type inter-process or inter-OS DMA
>>transfers. */ int rte_dma_access_pair_group_create(const struct
>>rte_dma_dev *dev,
>>                                     rte_uuid_t process_id,
>>                                     rte_uuid_t token,
>>                                     uint16_t *group_id);
>>
>>/** Destroys an access group once all participating devices have
>>exited. */ int rte_dma_access_pair_group_destroy(const struct
>rte_dma_dev *dev,
>>                                      uint16_t group_id);
>>
>>/** Allows a device to join an existing access group using a device
>>handle and token. */ int rte_dma_access_pair_group_join(const struct
>>rte_dma_dev *dev,
>>                                   uint16_t group_id,
>>                                   rte_uuid_t process_id,
>>                                   rte_uuid_t token,
>>                                   rte_dma_access_pair_leave_cb_t
>>leave_cb);
>>
>>/** Removes a device from an access group. */ int
>>rte_dma_access_pair_group_leave(const struct rte_dma_dev *dev,
>>                                    uint16_t group_id);
>>
>>/** Retrieves the source and destination handles for a given device
>>within the group. */ int
>>rte_dma_access_pair_gorup_src_dst_handles_get(const struct
>rte_dma_dev *dev,
>>                                            uint16_t group_id,
>>                                            rte_uuid_t src_process_id,
>>                                            rte_uuid_t dst_process_id,
>>                                            uint16_t *src_handle,
>>                                            uint16_t *dst_handle);
>>
>>
>>Parameters that need explanation:
>>--------------------------------
>>process_id: Unique ID for the process, generated via rte_uuid_ APIs
>>token: Provided by an administrative actor to grant access, similar to
>>VFIO VF token creation used in VFIO PF driver.
>>leave_cb: Callback to notify when another side leaves the group
>>
>>
>>Example Workflow for Class A Inter-Domain DMA Transfer:
>>-------------------------------------------------------
>>
>>This example demonstrates how three processes — p0, p1, and p2 —
>>coordinate inter-domain DMA transfers using pair-type(Class A) DMA
>devices.
>>
>>Step 1: Group Creation (p0)
>>Process p0 calls rte_dma_access_pair_group_create() with a unique
>>process handle and token. A group_id is returned.
>>
>>Step 2: Group Sharing
>>group_id and token are shared with p1 and p2 via IPC or shared memory.
>>
>>Step 3: Group Joining (p1 & p2)
>>Processes p1 and p2 call rte_dma_access_pair_group_join() with their
>>process id  and the shared token from admin
>>
>>Step 4: Handle Discovery
>>Each process uses rte_dma_access_pair_gorup_src_dst_handles_get() to
>>retrieve source and destination handles for other processes.
>>
>>Step 5: Transfer Coordination
>>Using the handles, each process configures a virtual channel (vchan)
>>and initiates DMA transfers.
>>
>>Step 6: Group Teardown
>>When a process no longer needs to participate, it calls
>>rte_dma_access_pair_group_leave(). Other processes are notified via the
>>registered callback with rte_dma_access_pair_group_join().
>>Once all devices have exited, p0 calls
>>rte_dma_access_pair_group_destroy()
>>to clean up.
>>
>>
>>For Class B: We can add new capability flag and have new set of APIs
>>rte_dma_access_master_ or so. When such devices comes or when intel
>>wants to add it
>>
>>
>>
>>