Re: [PATCH v3 1/1] lib/dma: add control-plane APIs for inter-domain DMA transfers

[fengchengwen <fengchengwen@huawei.com>]

[-- Attachment #1: Type: text/plain, Size: 19813 bytes --]

A minor comment below, but it seems could fix later.

Acked-by: Chengwen Feng <fengchengwen@huawei.com>


On 2025/10/17 20:26, Vamsi Krishna wrote:
> From: Vamsi Attunuru<vattunuru@marvell.com>
>
> Secure and controlled inter-domain DMA transfers require dedicated
> control-plane APIs to create and manage access groups.
>
> DMA devices used for inter-process data transfer can be categorized as
> follows:
>
> Class A: Both endpoints require a DMA device for data transfer (e.g.,
> Marvell DMA devices).
> Class B: Only one endpoint requires a DMA device; the other does not.
> Class C: Other device types not currently classified.
>
> Providing a unified API for all these categories is complex, as Linux
> and other operating systems do not offer native control-plane APIs for
> this purpose. Therefore, DPDK can implement its own control-plane
> mechanisms to support Class A, B, and C devices.
>
> This commit introduces the necessary APIs for Class A DMA devices.
> Devices can create or join access groups using token-based
> authentication, ensuring that only authorized devices within the same
> group can perform DMA transfers across processes or OS domains.
>
> API Usage Flow:
>
> Process 1 (Group Creator):
> Invokes rte_dma_access_pair_group_create() to establish a new access
> pair group, then shares the group_id and token with Process 2 via IPC.
>
> Process 2 (Group Joiner):
> Receives the group_id and token from Process 1 and calls
> rte_dma_access_pair_group_join() to join the group.
>
> Both Processes:
> Use rte_dma_access_pair_group_handler_get() to obtain handler information
> for domains in the group.
> Perform DMA transfers as required.
>
> Process 2 (when finished):
> Calls rte_dma_access_pair_group_leave() to exit the group.
>
> Process 1:
> Monitors for group updates and confirms group membership as needed.
>
> Depends-on: patch-36335 (lib/dma: introduce inter-process and inter-OS DMA)
>
> Signed-off-by: Vamsi Attunuru<vattunuru@marvell.com>
> ---
> V3 Changes:
> * Add documentation to dmadev doc.
> * Add common callback for group creator and joiner.
> * Fix V2 review comments
>
>   doc/guides/prog_guide/dmadev.rst |  49 +++++++++++
>   lib/dmadev/rte_dmadev.c          | 138 +++++++++++++++++++++++++++++++
>   lib/dmadev/rte_dmadev.h          | 137 ++++++++++++++++++++++++++++++
>   lib/dmadev/rte_dmadev_pmd.h      |  29 +++++++
>   lib/dmadev/rte_dmadev_trace.h    |  42 ++++++++++
>   5 files changed, 395 insertions(+)
>
> diff --git a/doc/guides/prog_guide/dmadev.rst b/doc/guides/prog_guide/dmadev.rst
> index 9b497ce7bd..c2a1e34c35 100644
> --- a/doc/guides/prog_guide/dmadev.rst
> +++ b/doc/guides/prog_guide/dmadev.rst
> @@ -174,3 +174,52 @@ are shown below.
>   
>   For more information on how to use the Telemetry interface, see
>   the :doc:`../howto/telemetry`.
> +
> +Inter-domain DMA Capabilities
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The inter-domain DMA feature enables DMA devices to perform data transfers
> +across different processes and OS domains. This is achieved by configuring
> +virtual channels (vchans) using `src_handler` and `dst_handler` fields, which
> +represent the source and destination endpoints for inter-domain DMA operations.
> +Handler information is exchanged between devices based on their DMA class type.
> +These handler details will be exchanged based on the type of DMA devices.
> +
> +DMA devices used for inter-process data transfer can be categorized as follows:

inter-process -> inter-domain

> +
> +Class A: Both endpoints require a DMA device for data transfer (e.g., Marvell DMA devices).
> +Class B: Only one endpoint requires a DMA device; the other does not.
> +Class C: Other device types not currently classified.
> +
> +Currently the necessary APIs for Class A DMA devices are avilable for exchanging the handler
> +details. Devices can create or join access groups using token-based authentication, ensuring
> +that only authorized devices within the same group can perform DMA transfers across processes
> +or OS domains.
> +
> +API usage flow for setting up the access pair group for DMA between process#1 & process#2.
> +
> +Process#1 (Group Creator):
> +--------------------------
> +Calls ``rte_dma_access_pair_group_create`` to establish a new access pair group,
> +then shares the ``group_id``, ``token`` and ``domain_id`` with Process#2 via IPC.
> +
> +Process#2 (Group Joiner):
> +-------------------------
> +Receives the ``group_id`` and ``token`` from Process#1 and calls ``rte_dma_access_pair_group_join``
> +to join the group.
> +
> +Both Processes:
> +---------------
> +Use ``rte_dma_access_pair_group_handler_get`` to obtain ``handler`` information for domains
> +in the group.
> +
> +--Perform inter-domain DMA transfers as required.
> +
> +Process#2 (when finished):
> +--------------------------
> +Calls ``rte_dma_access_pair_group_leave`` to exit the group.
> +
> +Process#1 (final cleanup):
> +--------------------------
> +Calls ``rte_dma_access_pair_group_destroy`` to destroy the group.
> +
> diff --git a/lib/dmadev/rte_dmadev.c b/lib/dmadev/rte_dmadev.c
> index 30c3db3a98..cf50d7433c 100644
> --- a/lib/dmadev/rte_dmadev.c
> +++ b/lib/dmadev/rte_dmadev.c
> @@ -816,6 +816,144 @@ rte_dma_vchan_status(int16_t dev_id, uint16_t vchan, enum rte_dma_vchan_status *
>   	return dev->dev_ops->vchan_status(dev, vchan, status);
>   }
>   
> +int
> +rte_dma_access_pair_group_create(int16_t dev_id, rte_uuid_t domain_id, rte_uuid_t token,
> +				 int16_t *group_id, rte_dma_access_pair_group_event_cb_t cb)
> +{
> +	struct rte_dma_info dev_info;
> +	struct rte_dma_dev *dev;
> +
> +	if (!rte_dma_is_valid(dev_id) || group_id == NULL)
> +		return -EINVAL;
> +	dev = &rte_dma_devices[dev_id];
> +
> +	if (rte_dma_info_get(dev_id, &dev_info)) {
> +		RTE_DMA_LOG(ERR, "Device %d get device info fail", dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (!((dev_info.dev_capa & RTE_DMA_CAPA_INTER_PROCESS_DOMAIN) ||
> +	    (dev_info.dev_capa & RTE_DMA_CAPA_INTER_OS_DOMAIN))) {
> +		RTE_DMA_LOG(ERR, "Device %d don't support inter-process or inter-os domain",
> +			    dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (*dev->dev_ops->access_pair_group_create == NULL)
> +		return -ENOTSUP;
> +	return (*dev->dev_ops->access_pair_group_create)(dev, domain_id, token, group_id, cb);
> +}
> +
> +int
> +rte_dma_access_pair_group_destroy(int16_t dev_id, int16_t group_id)
> +{
> +	struct rte_dma_info dev_info;
> +	struct rte_dma_dev *dev;
> +
> +	if (!rte_dma_is_valid(dev_id))
> +		return -EINVAL;
> +	dev = &rte_dma_devices[dev_id];
> +
> +	if (rte_dma_info_get(dev_id, &dev_info)) {
> +		RTE_DMA_LOG(ERR, "Device %d get device info fail", dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (!((dev_info.dev_capa & RTE_DMA_CAPA_INTER_PROCESS_DOMAIN) ||
> +	    (dev_info.dev_capa & RTE_DMA_CAPA_INTER_OS_DOMAIN))) {
> +		RTE_DMA_LOG(ERR, "Device %d don't support inter-process or inter-os domain",
> +			    dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (*dev->dev_ops->access_pair_group_destroy == NULL)
> +		return -ENOTSUP;
> +	return (*dev->dev_ops->access_pair_group_destroy)(dev, group_id);
> +}
> +
> +int
> +rte_dma_access_pair_group_join(int16_t dev_id, int16_t group_id, rte_uuid_t token,
> +			       rte_dma_access_pair_group_event_cb_t cb)
> +{
> +	struct rte_dma_info dev_info;
> +	struct rte_dma_dev *dev;
> +
> +	if (!rte_dma_is_valid(dev_id))
> +		return -EINVAL;
> +	dev = &rte_dma_devices[dev_id];
> +
> +	if (rte_dma_info_get(dev_id, &dev_info)) {
> +		RTE_DMA_LOG(ERR, "Device %d get device info fail", dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (!((dev_info.dev_capa & RTE_DMA_CAPA_INTER_PROCESS_DOMAIN) ||
> +	    (dev_info.dev_capa & RTE_DMA_CAPA_INTER_OS_DOMAIN))) {
> +		RTE_DMA_LOG(ERR, "Device %d don't support inter-process or inter-os domain",
> +			    dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (*dev->dev_ops->access_pair_group_join == NULL)
> +		return -ENOTSUP;
> +	return (*dev->dev_ops->access_pair_group_join)(dev, group_id, token, cb);
> +}
> +
> +int
> +rte_dma_access_pair_group_leave(int16_t dev_id, int16_t group_id)
> +{
> +	struct rte_dma_info dev_info;
> +	struct rte_dma_dev *dev;
> +
> +	if (!rte_dma_is_valid(dev_id))
> +		return -EINVAL;
> +	dev = &rte_dma_devices[dev_id];
> +
> +	if (rte_dma_info_get(dev_id, &dev_info)) {
> +		RTE_DMA_LOG(ERR, "Device %d get device info fail", dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (!((dev_info.dev_capa & RTE_DMA_CAPA_INTER_PROCESS_DOMAIN) ||
> +	    (dev_info.dev_capa & RTE_DMA_CAPA_INTER_OS_DOMAIN))) {
> +		RTE_DMA_LOG(ERR, "Device %d don't support inter-process or inter-os domain",
> +			    dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (*dev->dev_ops->access_pair_group_leave == NULL)
> +		return -ENOTSUP;
> +	return (*dev->dev_ops->access_pair_group_leave)(dev, group_id);
> +}
> +
> +int
> +rte_dma_access_pair_group_handler_get(int16_t dev_id, int16_t group_id, rte_uuid_t domain_id,
> +				      uint16_t *handler)
> +{
> +	struct rte_dma_info dev_info;
> +	struct rte_dma_dev *dev;
> +
> +	if (!rte_dma_is_valid(dev_id) || handler == NULL)
> +		return -EINVAL;
> +	dev = &rte_dma_devices[dev_id];
> +
> +	if (rte_dma_info_get(dev_id, &dev_info)) {
> +		RTE_DMA_LOG(ERR, "Device %d get device info fail", dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (!((dev_info.dev_capa & RTE_DMA_CAPA_INTER_PROCESS_DOMAIN) ||
> +	    (dev_info.dev_capa & RTE_DMA_CAPA_INTER_OS_DOMAIN))) {
> +		RTE_DMA_LOG(ERR, "Device %d don't support inter-process or inter-os domain",
> +			    dev_id);
> +		return -EINVAL;
> +	}
> +
> +	if (*dev->dev_ops->access_pair_group_handler_get == NULL)
> +		return -ENOTSUP;
> +	return (*dev->dev_ops->access_pair_group_handler_get)(dev, group_id, domain_id, handler);
> +}
> +
>   static const char *
>   dma_capability_name(uint64_t capability)
>   {
> diff --git a/lib/dmadev/rte_dmadev.h b/lib/dmadev/rte_dmadev.h
> index 1708311c5b..77c19b2008 100644
> --- a/lib/dmadev/rte_dmadev.h
> +++ b/lib/dmadev/rte_dmadev.h
> @@ -148,6 +148,7 @@
>   
>   #include <rte_bitops.h>
>   #include <rte_common.h>
> +#include <rte_uuid.h>
>   
>   #ifdef __cplusplus
>   extern "C" {
> @@ -797,6 +798,142 @@ rte_dma_vchan_status(int16_t dev_id, uint16_t vchan, enum rte_dma_vchan_status *
>    */
>   int rte_dma_dump(int16_t dev_id, FILE *f);
>   
> +/**
> + * Event types for DMA access pair group notifications.
> + *
> + * When the event type is RTE_DMA_GROUP_EVENT_MEMBER_LEFT, the handler associated
> + * with the departing member's domain is no longer valid. Inter-domain DMA operations
> + * targeting that domain should be avoided.
> + *
> + * When the event type is RTE_DMA_GROUP_EVENT_GROUP_DESTROYED, all handlers associated
> + * with the group become invalid. No further inter-domain DMA operations should be
> + * initiated using those handlers.
> + */
> +enum rte_dma_access_pair_group_event_type {
> +	/** A member left the group (notifies creator and joiners) */
> +	RTE_DMA_GROUP_EVENT_MEMBER_LEFT,
> +	/** Group was destroyed (notifies joiners) */
> +	RTE_DMA_GROUP_EVENT_GROUP_DESTROYED
> +};
> +
> +/**
> + * This callback is used to notify interested parties (either the group creator
> + * or group joiners) about significant events related to the lifecycle of a DMA
> + * access pair group.
> + *
> + * It can be registered by:
> + * - **Group creators or Group joiners** to be notified when a member leaves the group.
> + * - **Group joiners** to be notified when the group is destroyed.
> + *
> + * @param dev_id
> + *   Identifier of the DMA device.
> + * @param group_id
> + *   Identifier of the access pair group where the event occurred.
> + * @param domain_id
> + *   UUID of the domain_id associated with the event. For member leave events,
> + *   this is the domain_id of the member that left. For group destruction events,
> + *   this may refer to the domain_id of the respective member.
> + * @param event
> + *   Type of event that occurred.
> + *   @see rte_dma_access_pair_group_event_type
> + */
> +typedef void (*rte_dma_access_pair_group_event_cb_t)(int16_t dma_id, int16_t group_id,
> +						     rte_uuid_t domain_id,
> +						     enum
> +						     rte_dma_access_pair_group_event_type event);
> +
> +/**
> + * Create an access pair group to enable secure DMA transfers between devices across different
> + * processes or operating system domains.
> + *
> + * @param dev_id
> + *   Identifier of the DMA device initiating the group.
> + * @param domain_id
> + *   Unique identifier representing the process or OS domain.
> + * @param token
> + *   Authentication token used to establish the access group.
> + * @param[out] group_id
> + *   Pointer to store the ID of the newly created access group.
> + * @param cb
> + *   Callback function to be invoked when a member leaves the group.
> + *
> + * @return
> + *   0 on success,
> + *   negative error code on failure.
> + */
> +int rte_dma_access_pair_group_create(int16_t dev_id, rte_uuid_t domain_id, rte_uuid_t token,
> +				     int16_t *group_id, rte_dma_access_pair_group_event_cb_t cb);
> +
> +/**
> + * Destroy an access pair group if all participating devices have exited. This operation is only
> + * permitted by the device that originally created the group; attempts by other devices will result
> + * in failure.
> + *
> + * @param dev_id
> + *   Identifier of the device requesting group destruction.
> + * @param group_id
> + *   ID of the access group to be destroyed.
> + * @return
> + *   0 on success,
> + *   negative value on failure indicating the error code.
> + */
> +int rte_dma_access_pair_group_destroy(int16_t dev_id, int16_t group_id);
> +
> +/**
> + * Join an existing access group to enable secure DMA transfers between devices across different
> + * processes or OS domains.
> + *
> + * @param dev_id
> + *   Identifier of the DMA device attempting to join the group.
> + * @param group_id
> + *   ID of the access group to join.
> + * @param token
> + *   Authentication token used to validate group membership.
> + * @param cb
> + *   Callback function to be invoked when the device leaves the group or when the group
> + *   is destroyed due to some exception or failure.
> + *
> + * @return
> + *   0 on success,
> + *   negative value on failure indicating the error code.
> + */
> +int rte_dma_access_pair_group_join(int16_t dev_id, int16_t group_id, rte_uuid_t token,
> +				   rte_dma_access_pair_group_event_cb_t cb);
> +
> +/**
> + * Leave an access group, removing the device's entry from the group table and disabling
> + * inter-domain DMA transfers to and from this device. This operation is not permitted for the
> + * device that originally created the group.
> + *
> + * @param dev_id
> + *   Identifier of the device requesting to leave the group.
> + * @param group_id
> + *   ID of the access group to leave.
> + * @return
> + *   0 on success,
> + *   negative value on failure indicating the error code.
> + */
> +int rte_dma_access_pair_group_leave(int16_t dev_id, int16_t group_id);
> +
> +/**
> + * Retrieve the handler associated with a specific domain ID, which is used by the application to
> + * query source or destinationin handler to initiate inter-process or inter-OS DMA transfers.
> + *
> + * @param dev_id
> + *   Identifier of the DMA device requesting the handler.
> + * @param group_id
> + *   ID of the access group to query.
> + * @param group_tbl
> + *   Unique identifier of the target process or OS domain.
> + * @param[out] handler
> + *   Pointer to store the retrieved handler value.
> + * @return
> + *   0 on success,
> + *   negative value on failure indicating the error code.
> + */
> +int rte_dma_access_pair_group_handler_get(int16_t dev_id, int16_t group_id, rte_uuid_t domain_id,
> +					  uint16_t *handler);
> +
>   /**
>    * DMA transfer result status code defines.
>    *
> diff --git a/lib/dmadev/rte_dmadev_pmd.h b/lib/dmadev/rte_dmadev_pmd.h
> index 58729088ff..16fd625c2d 100644
> --- a/lib/dmadev/rte_dmadev_pmd.h
> +++ b/lib/dmadev/rte_dmadev_pmd.h
> @@ -64,6 +64,29 @@ typedef int (*rte_dma_vchan_status_t)(const struct rte_dma_dev *dev, uint16_t vc
>   /** @internal Used to dump internal information. */
>   typedef int (*rte_dma_dump_t)(const struct rte_dma_dev *dev, FILE *f);
>   
> +/** @internal Used to create an access pair group for inter-process or inter-OS DMA transfers. */
> +typedef int (*rte_dma_access_pair_group_create_t)(const struct rte_dma_dev *dev,
> +						  rte_uuid_t domain_id, rte_uuid_t token,
> +						  int16_t *group_id,
> +						  rte_dma_access_pair_group_event_cb_t cb);
> +
> +/** @internal Used to destroy an access pair group when all other devices have exited. */
> +typedef int (*rte_dma_access_pair_group_destroy_t)(const struct rte_dma_dev *dev,
> +						   int16_t group_id);
> +
> +/** @internal Used to join an access pair group for inter-process or inter-OS DMA transfers. */
> +typedef int (*rte_dma_access_pair_group_join_t)(const struct rte_dma_dev *dev, int16_t group_id,
> +						rte_uuid_t token,
> +						rte_dma_access_pair_group_event_cb_t cb);
> +
> +/** @internal Used to leave an access pair group, removing the device from the group. */
> +typedef int (*rte_dma_access_pair_group_leave_t)(const struct rte_dma_dev *dev, int16_t group_id);
> +
> +/** @internal Used to retrieve handler information of the domain_id present in the group. */
> +typedef int (*rte_dma_access_pair_group_handler_get_t)(const struct rte_dma_dev *dev,
> +						       int16_t group_id, rte_uuid_t domain_id,
> +						       uint16_t *handler);
> +
>   /**
>    * DMA device operations function pointer table.
>    *
> @@ -83,6 +106,12 @@ struct rte_dma_dev_ops {
>   
>   	rte_dma_vchan_status_t     vchan_status;
>   	rte_dma_dump_t             dev_dump;
> +
> +	rte_dma_access_pair_group_create_t	access_pair_group_create;
> +	rte_dma_access_pair_group_destroy_t	access_pair_group_destroy;
> +	rte_dma_access_pair_group_join_t	access_pair_group_join;
> +	rte_dma_access_pair_group_leave_t	access_pair_group_leave;
> +	rte_dma_access_pair_group_handler_get_t	access_pair_group_handler_get;
>   };
>   
>   /**
> diff --git a/lib/dmadev/rte_dmadev_trace.h b/lib/dmadev/rte_dmadev_trace.h
> index f1a178eeb5..da80a8cb50 100644
> --- a/lib/dmadev/rte_dmadev_trace.h
> +++ b/lib/dmadev/rte_dmadev_trace.h
> @@ -101,6 +101,48 @@ RTE_TRACE_POINT(
>   	rte_trace_point_emit_int(ret);
>   )
>   
> +RTE_TRACE_POINT(
> +	rte_dma_trace_access_pair_group_create,
> +	RTE_TRACE_POINT_ARGS(int16_t dev_id, rte_uuid_t domain_id, rte_uuid_t token,
> +			     int16_t *group_id),
> +	rte_trace_point_emit_i16(dev_id);
> +	rte_trace_point_emit_u8_ptr(&domain_id[0]);
> +	rte_trace_point_emit_u8_ptr(&token[0]);
> +	rte_trace_point_emit_ptr(group_id);
> +)
> +
> +RTE_TRACE_POINT(
> +	rte_dma_trace_access_pair_group_destroy,
> +	RTE_TRACE_POINT_ARGS(int16_t dev_id, int16_t group_id),
> +	rte_trace_point_emit_i16(dev_id);
> +	rte_trace_point_emit_i16(group_id);
> +)
> +
> +RTE_TRACE_POINT(
> +	rte_dma_trace_access_pair_group_join,
> +	RTE_TRACE_POINT_ARGS(int16_t dev_id, int16_t group_id, rte_uuid_t token),
> +	rte_trace_point_emit_i16(dev_id);
> +	rte_trace_point_emit_i16(group_id);
> +	rte_trace_point_emit_u8_ptr(&token[0]);
> +)
> +
> +RTE_TRACE_POINT(
> +	rte_dma_trace_access_pair_group_leave,
> +	RTE_TRACE_POINT_ARGS(int16_t dev_id, int16_t group_id),
> +	rte_trace_point_emit_i16(dev_id);
> +	rte_trace_point_emit_i16(group_id);
> +)
> +
> +RTE_TRACE_POINT(
> +	rte_dma_trace_access_pair_group_handler_get,
> +	RTE_TRACE_POINT_ARGS(int16_t dev_id, int16_t group_id, rte_uuid_t domain_id,
> +			     uint16_t *handler),
> +	rte_trace_point_emit_i16(dev_id);
> +	rte_trace_point_emit_i16(group_id);
> +	rte_trace_point_emit_u8_ptr(&domain_id[0]);
> +	rte_trace_point_emit_ptr(handler);
> +)
> +
>   #ifdef __cplusplus
>   }
>   #endif

[-- Attachment #2: Type: text/html, Size: 20093 bytes --]