[PATCH v3] vfio: combine container_create and group_bind

[PATCH v3] vfio: combine container_create and group_bind

[BillXiang]

From: BillXiang <xiangwencheng@dayudpu.com>

For multi-devices in one group we can only create and bind to one
container. With this new function, device driver does not need to
save the binding info additionally between different devices.

Signed-off-by: BillXiang <xiangwencheng@dayudpu.com>
---
 lib/eal/include/rte_vfio.h | 17 +++++++++++++++++
 lib/eal/linux/eal_vfio.c   | 24 ++++++++++++++++++++++++
 lib/eal/version.map        |  3 +++
 3 files changed, 44 insertions(+)

diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index b774625d9f..f6743b3620 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -407,6 +407,23 @@ int
 rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
 		uint64_t iova, uint64_t len);
 
+/**
+ * Get vfio group fd bound with container fd for certain iommu group.
+ *
+ * @param iommu_group_num
+ *   iommu group num
+ * @param vfio_group_fd
+ *   vfio group fd of the iommu group.
+ * @param vfio_container_fd
+ *   vfio container fd of the iommu group.
+ * @return
+ *    0 if successful
+ *   <0 if failed
+ */
+__rte_experimental
+int
+rte_vfio_get_fd(int iommu_group_num, int *vfio_group_fd, int *vfio_container_fd);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 4e69e72e3b..08510df7e2 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2196,3 +2196,27 @@ rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
 
 	return container_dma_unmap(vfio_cfg, vaddr, iova, len);
 }
+
+int
+rte_vfio_get_fd(int iommu_group_num, int *vfio_group_fd, int *vfio_container_fd)
+{
+	struct vfio_config *vfio_cfg;
+	vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
+	/* do not create new container if the group has bound with one */
+	if (vfio_cfg) {
+		*vfio_container_fd = vfio_cfg->vfio_container_fd;
+	} else {
+		*vfio_container_fd = rte_vfio_container_create();
+		if (*vfio_container_fd < 0)
+			goto err;
+
+		vfio_cfg = get_vfio_cfg_by_container_fd(*vfio_container_fd);
+	}
+	*vfio_group_fd = vfio_get_group_fd(vfio_cfg, iommu_group_num);
+	if (*vfio_group_fd < 0)
+		goto err;
+
+	return 0;
+err:
+	return -1;
+}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 3df50c3fbb..f26b65a504 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -396,6 +396,9 @@ EXPERIMENTAL {
 
 	# added in 24.03
 	rte_vfio_get_device_info; # WINDOWS_NO_EXPORT
+
+	# added in 24.07
+	rte_vfio_get_fd; # WINDOWS_NO_EXPORT
 };
 
 INTERNAL {
-- 
2.30.0

Re: [PATCH v3] vfio: combine container_create and group_bind

[David Marchand]

On Thu, Jul 4, 2024 at 9:48 AM BillXiang <xiangwencheng@dayudpu.com> wrote:
>
> From: BillXiang <xiangwencheng@dayudpu.com>
>
> For multi-devices in one group we can only create and bind to one
> container. With this new function, device driver does not need to
> save the binding info additionally between different devices.
>
> Signed-off-by: BillXiang <xiangwencheng@dayudpu.com>

Thanks for the patch (don't forget to register to the dev@ mailing
list when submitting patches).

Could you please describe which driver benefits from it?


-- 
David Marchand

Re: [PATCH v3] vfio: combine container_create and group_bind

[BillXiang]

> From: "David Marchand"<david.marchand@redhat.com>
> Date:  Tue, Jul 9, 2024, 16:49
> Subject:  Re: [PATCH v3] vfio: combine container_create and group_bind
> To: "BillXiang"<xiangwencheng@dayudpu.com>
> Cc: "Tyler Retzlaff"<roretzla@linux.microsoft.com>, "Anatoly Burakov"<anatoly.burakov@intel.com>, <dev@dpdk.org>
> On Thu, Jul 4, 2024 at 9:48 AM BillXiang <xiangwencheng@dayudpu.com> wrote:
> >
> > From: BillXiang <xiangwencheng@dayudpu.com>
> >
> > For multi-devices in one group we can only create and bind to one
> > container. With this new function, device driver does not need to
> > save the binding info additionally between different devices.
> >
> > Signed-off-by: BillXiang <xiangwencheng@dayudpu.com>
> 
> Thanks for the patch (don't forget to register to the dev@ mailing
> list when submitting patches).
> 
> Could you please describe which driver benefits from it?
> 
> 
> -- 
> David Marchand

Thanks for your replay. I have registered now.

This patch addresses the issue for PCIe devices that have multiple PFs within 
the same IOMMU group, a situation arising from the absence of ACS support.
If we want to use these PFs in vDPA, we have to know if the group has been 
bound to a container by other PF upon each probe.

Re: [PATCH v3] vfio: combine container_create and group_bind

[Stephen Hemminger]

On Wed,  3 Jul 2024 19:59:43 +0800
"BillXiang" <xiangwencheng@dayudpu.com> wrote:

> From: BillXiang <xiangwencheng@dayudpu.com>
> 
> For multi-devices in one group we can only create and bind to one
> container. With this new function, device driver does not need to
> save the binding info additionally between different devices.
> 
> Signed-off-by: BillXiang <xiangwencheng@dayudpu.com>

Ok, but where is the device driver using it?
DPDK upstream doesn't want to keep API's for drivers that are not upstream.

> ---
>  lib/eal/include/rte_vfio.h | 17 +++++++++++++++++
>  lib/eal/linux/eal_vfio.c   | 24 ++++++++++++++++++++++++
>  lib/eal/version.map        |  3 +++
>  3 files changed, 44 insertions(+)
> 
> diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
> index b774625d9f..f6743b3620 100644
> --- a/lib/eal/include/rte_vfio.h
> +++ b/lib/eal/include/rte_vfio.h
> @@ -407,6 +407,23 @@ int
>  rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
>  		uint64_t iova, uint64_t len);
>  
> +/**
> + * Get vfio group fd bound with container fd for certain iommu group.
> + *
> + * @param iommu_group_num
> + *   iommu group num
> + * @param vfio_group_fd
> + *   vfio group fd of the iommu group.
> + * @param vfio_container_fd
> + *   vfio container fd of the iommu group.
> + * @return
> + *    0 if successful
> + *   <0 if failed
> + */
> +__rte_experimental
> +int
> +rte_vfio_get_fd(int iommu_group_num, int *vfio_group_fd, int *vfio_container_fd);
> +
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
> index 4e69e72e3b..08510df7e2 100644
> --- a/lib/eal/linux/eal_vfio.c
> +++ b/lib/eal/linux/eal_vfio.c
> @@ -2196,3 +2196,27 @@ rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
>  
>  	return container_dma_unmap(vfio_cfg, vaddr, iova, len);
>  }
> +
> +int
> +rte_vfio_get_fd(int iommu_group_num, int *vfio_group_fd, int *vfio_container_fd)
> +{
> +	struct vfio_config *vfio_cfg;
> +	vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
> +	/* do not create new container if the group has bound with one */

prefer blank line after declaration

> +	if (vfio_cfg) {
> +		*vfio_container_fd = vfio_cfg->vfio_container_fd;
> +	} else {
> +		*vfio_container_fd = rte_vfio_container_create();
> +		if (*vfio_container_fd < 0)
> +			goto err;

If no unwind necessary, then skip the goto and just return -1;

> +
> +		vfio_cfg = get_vfio_cfg_by_container_fd(*vfio_container_fd);
> +	}
> +	*vfio_group_fd = vfio_get_group_fd(vfio_cfg, iommu_group_num);
> +	if (*vfio_group_fd < 0)
> +		goto err;

ditto

> +
> +	return 0;
> +err:
> +	return -1;
> +}



> diff --git a/lib/eal/version.map b/lib/eal/version.map
> index 3df50c3fbb..f26b65a504 100644
> --- a/lib/eal/version.map
> +++ b/lib/eal/version.map
> @@ -396,6 +396,9 @@ EXPERIMENTAL {
>  
>  	# added in 24.03
>  	rte_vfio_get_device_info; # WINDOWS_NO_EXPORT
> +
> +	# added in 24.07
> +	rte_vfio_get_fd; # WINDOWS_NO_EXPORT
>  };
>  
>  INTERNAL {