RE: [EXT] [PATCH v1 0/3] Add support for inter-domain DMA operations

[Satananda Burla <sburla@marvell.com>]

Hi Anatoly

> -----Original Message-----
> From: Anatoly Burakov <anatoly.burakov@intel.com>
> Sent: Friday, August 11, 2023 9:15 AM
> To: dev@dpdk.org
> Cc: bruce.richardson@intel.com
> Subject: [EXT] [PATCH v1 0/3] Add support for inter-domain DMA
> operations
> 
> External Email
> 
> ----------------------------------------------------------------------
> This patchset adds inter-domain DMA operations, and implements driver
> support
> for them in Intel(R) IDXD driver.
> 
> Inter-domain DMA operations are similar to regular DMA operations,
> except that
> source and/or destination addresses will be in virtual address space of
> another
> process. In this patchset, DMA device is extended to support two new
> data plane
> operations: inter-domain copy, and inter-domain fill. No control plane
> API is
> provided for dmadev to set up inter-domain communication (see below for
> more
> info).
Thanks for posting this.
Do you have usecases where a process from 3rd domain sets up transfer 
between memories from 2 domains? i.e process 1 is src, process 2 is
dest and process 3 executes transfer. The SDXI spec also defines this kind
of a transfer.
Have you considered extending  rte_dma_port_param and rte_dma_vchan_conf
to represent interdomain memory transfer setup as a separate port type like
RTE_DMA_PORT_INTER_DOMAIN ?
And then we could have a separate vchan dedicated for this transfer.
The rte_dma_vchan  can be setup with separate struct rte_dma_port_param
each for source and destination. The union could be extended to provide
the necessary information to pmd, this could be set of fields that
would be needed by different architectures like controller id,
pasid, smmu streamid and substreamid etc, if an opaque handle is needed,
it could also be accommodated in the union.
These transfers could also be initiated between 2 processes each having 2
dmadev VFs from the same PF as well. Marvell hardware supports this mode.
Since control plane for this can differ between PMDs, it is better to
setup the memory sharing outside dmadev and only pass the fields of interest to
the PMD for completing the transfer. For instance, for PCIe EP to Host
DMA transactions (MEM_TO_DEV and DEV_TO_MEM), the process of setting up
shared memory from PCIe host is not part of dmadev.
If we wish to make the memory sharing interface as a part of dmadev, then
preferably the control plane has to be abstracted to work for all the modes
and architectures.

Regards
Satananda