RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process and inter-OS DMA

[Jerin Jacob <jerinj@marvell.com>]

> -----Original Message-----
> From: fengchengwen <fengchengwen@huawei.com>
> Sent: Thursday, October 9, 2025 4:38 PM
> To: Jerin Jacob <jerinj@marvell.com>; Vamsi Krishna Attunuru
> <vattunuru@marvell.com>; dev@dpdk.org; anatoly.burakov@intel.com
> Cc: thomas@monjalon.net; bruce.richardson@intel.com;
> vladimir.medvedkin@intel.com; kevin.laatz@intel.com
> Subject: Re: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process and inter-OS
> DMA
> 
> On 10/1/2025 1: 57 PM, Jerin Jacob wrote: > > >> -----Original Message----- >>
> From: Vamsi Krishna Attunuru <vattunuru@ marvell. com> >> Sent: Wednesday,
> September 24, 2025 9: 45 AM >> To: fengchengwen
> <fengchengwen@ huawei. com>; ZjQcmQRYFpfptBannerStart Prioritize security
> for external emails:
> Confirm sender and content safety before clicking links or opening attachments
> <https://us-phishalarm-
> ewt.proofpoint.com/EWT/v1/CRVmXkqW!tg3TtN_0At8aXkwdmd36TTsseJBpLJzj
> Bf476S3kPQRmBpOSGmkKThlH1pYubSu5HYYCYrmGWL-
> Qw4sZgCDipBGcUjhqTJuU9g$>
> Report Suspicious
> 
> ZjQcmQRYFpfptBannerEnd
> On 10/1/2025 1:57 PM, Jerin Jacob wrote:
> >
> >
> >> -----Original Message-----
> >> From: Vamsi Krishna Attunuru <vattunuru@marvell.com>
> >> Sent: Wednesday, September 24, 2025 9:45 AM
> >> To: fengchengwen <fengchengwen@huawei.com>; dev@dpdk.org;
> >> anatoly.burakov@intel.com
> >> Cc: thomas@monjalon.net; bruce.richardson@intel.com;
> >> vladimir.medvedkin@intel.com; anatoly.burakov@intel.com;
> >> kevin.laatz@intel.com; Jerin Jacob <jerinj@marvell.com>
> >> Subject: RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process
> >> and inter-OS DMA
> >>
> >> Hi Feng, Anatoly,
> >>
> >> Gentle ping for below review.
> >>
> >>> -----Original Message-----
> >>> From: Vamsi Krishna Attunuru
> >>> Sent: Monday, September 22, 2025 5:19 PM
> >>> To: fengchengwen <fengchengwen@huawei.com>; dev@dpdk.org;
> >>> anatoly.burakov@intel.com
> >>> Cc: thomas@monjalon.net; bruce.richardson@intel.com;
> >>> vladimir.medvedkin@intel.com; anatoly.burakov@intel.com;
> >>> kevin.laatz@intel.com; Jerin Jacob <jerinj@marvell.com>
> >>> Subject: RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process
> >>> and
> >>> inter- OS DMA
> >>>
> >>> Hi Feng,
> >>>
> >>>> Hi Vamsi, This commit change is more than discussed, it add control
> >>>> API which for group management. 1. Control API: I check this commit
> >>>> and Intel commit [1], it seem has a quite difference. I hope Intel
> >>>> guys can express views ZjQcmQRYFpfptBannerEnd Hi Vamsi,
> >>>>
> >>>> This commit change is more than discussed, it add control API which
> >>>> for group management.
> >>>>
> >>>> 1. Control API: I check this commit and Intel commit [1], it seem
> >>>> has a quite difference.
> >>>>   I hope Intel guys can express views. I prefer not add this part
> >>>> if no
> >>> response.
> >>>
> >>> This new feature needs to be securely managed through control APIs.
> >>> It would be extremely helpful if the folks at Intel and you as well
> >>> could provide support or inputs on this.
> >
> >
> > Beyond adding Intel folks to this thread, I don't see any further steps we can
> take to drive review at this stage.
> >
> > That said, the table-based concept used in the current API may not be
> portable, and we may need improvements here.
> >
> > Based on my understanding, DMA devices used for inter process copy can be
> classified into three categories:
> >
> > Class A: Requires a pair of DMA devices (one on each end of process/domain)
> for data transfer. Marvell DMA devices fall into this category.
> > Class B: Requires only a single DMA device (one process/domain has a DMA
> device, the other process does not). Intel DMA devices fall here.
> > Class C: Other types of devices that we are not yet aware of.
> >
> > Abstracting all of these under a single API will be challenging. Linux
> > and other OSes do not provide control-plane APIs for this, so DPDK must
> provide control plane mechanisms to support Class A, Class B, and Class C
> devices.
> >
> > Proposal: Split development into separate sets:
> > -----------------------------------------------
> > Set A: Focus only on the datapath. Assume uint16_t *src_handle and uint16_t
> *dst_handle come from elsewhere (Class C).
> > Set B: Introduce capabilities for Class A devices with portable APIs (proposal
> below, without table concept).
> > Set C: Introduce capabilities for Class B devices and relevant APIs, to be added
> when needed.
> >
> > We can merge Set A in the current release and move Set B to a next release
> _if_ review or support for Class A devices requires more time.
> >
> > @fengchengwen Thoughts?
> 
> okay


Okay.

# We will send Set A and Set B as separate series.
We can review Set A first and merge and then Set B as time permits.

> 
> >
> > Class A API Proposal:
> > ---------------------
> > These APIs are based on a new capability flag for inter-process or inter-OS
> DMA transfers for Class A devices.
> >
> >
> > /** Creates an access group for pair-type inter-process or inter-OS
> > DMA transfers. */ int rte_dma_access_pair_group_create(const struct
> rte_dma_dev *dev,
> >                                      rte_uuid_t process_id,
> >                                      rte_uuid_t token,
> >                                      uint16_t *group_id);
> 
> how about rte_dma_access_group_create(), and add pair-type as one


In Class B case, the peer process does not need to create the group and it may NOT be the same API.
At minimum, arguments will change. In that context, having separate API for class A and Class B will 
be easy. And it will be  rte_dma_access_pair_ and rte_dma_access_single_ respective name space.


> parameter, and also rename process_id as domain_id


We will change to domain_id.


> 
> >
> > /** Destroys an access group once all participating devices have
> > exited. */ int rte_dma_access_pair_group_destroy(const struct rte_dma_dev
> *dev,
> >                                       uint16_t group_id);
> 
> rte_dma_access_group_destroy()
> 
> >
> > /** Allows a device to join an existing access group using a device
> > handle and token. */ int rte_dma_access_pair_group_join(const struct
> rte_dma_dev *dev,
> >                                    uint16_t group_id,
> >                                    rte_uuid_t process_id,
> >                                    rte_uuid_t token,
> >                                    rte_dma_access_pair_leave_cb_t
> > leave_cb);
> 
> rte_dma_access_group_join()

Same as above.

> 
> >
> > /** Removes a device from an access group. */ int
> > rte_dma_access_pair_group_leave(const struct rte_dma_dev *dev,
> >                                     uint16_t group_id);
> >
> > /** Retrieves the source and destination handles for a given device
> > within the group. */ int
> rte_dma_access_pair_gorup_src_dst_handles_get(const struct rte_dma_dev
> *dev,
> >                                             uint16_t group_id,
> >                                             rte_uuid_t src_process_id,
> >                                             rte_uuid_t dst_process_id,
> >                                             uint16_t *src_handle,
> >                                             uint16_t *dst_handle);
> 
> rte_dma_access_group_handle_get(const struct  rte_dma_dev *dev,
> 				uint16_t group_id,
> 				rte_uuid_t domain_id,
> 				uint16_t *handle);
> 
> so user could invoke multiple time if they want to get differ domain_id's handle.

Yes.


> 
> >
> >
> > Parameters that need explanation:
> > --------------------------------
> > process_id: Unique ID for the process, generated via rte_uuid_ APIs
> > token: Provided by an administrative actor to grant access, similar to VFIO VF
> token creation used in VFIO PF driver.
> > leave_cb: Callback to notify when another side leaves the group
> >
> >
> > Example Workflow for Class A Inter-Domain DMA Transfer:
> > -------------------------------------------------------
> >
> > This example demonstrates how three processes — p0, p1, and p2 —
> coordinate inter-domain DMA transfers using pair-type(Class A) DMA devices.
> >
> > Step 1: Group Creation (p0)
> > Process p0 calls rte_dma_access_pair_group_create() with a unique process
> handle and token. A group_id is returned.
> 
> I prefer the group_id is int type so it could hold such like file descriptor.

Will do.



> 
> >
> > Step 2: Group Sharing
> > group_id and token are shared with p1 and p2 via IPC or shared memory.
> >
> > Step 3: Group Joining (p1 & p2)
> > Processes p1 and p2 call rte_dma_access_pair_group_join() with their
> > process id  and the shared token from admin
> >
> > Step 4: Handle Discovery
> > Each process uses rte_dma_access_pair_gorup_src_dst_handles_get() to
> retrieve source and destination handles for other processes.
> >
> > Step 5: Transfer Coordination
> > Using the handles, each process configures a virtual channel (vchan) and
> initiates DMA transfers.
> >
> > Step 6: Group Teardown
> > When a process no longer needs to participate, it calls
> rte_dma_access_pair_group_leave(). Other processes are notified via the
> registered callback with rte_dma_access_pair_group_join().
> > Once all devices have exited, p0 calls rte_dma_access_pair_group_destroy() to
> clean up.
> >
> >
> > For Class B: We can add new capability flag and have new set of APIs
> > rte_dma_access_master_ or so. When such devices comes or when intel
> > wants to add it
> >
> >
> >
> >
> >