[PATCH] vdpa/sfc: handle sync issue between qemu and vhost-user

[PATCH] vdpa/sfc: handle sync issue between qemu and vhost-user

[abhimanyu.saini]

From: Abhimanyu Saini <absaini@amd.com>

When DPDK app is running in the VF, it sometimes rings the doorbell
before dev_config has had a chance to complete and hence it misses
the event. As workaround, ring the doorbell when vDPA reports the
notify_area to QEMU.

Fixes: 5e7596ba7cb3 ("vdpa/sfc: introduce Xilinx vDPA driver")
Cc: stable@dpdk.org

Signed-off-by: Vijay Kumar Srivastava <vsrivast@xilinx.com>
Signed-off-by: Abhimanyu Saini <absaini@amd.com>
---
 drivers/vdpa/sfc/sfc_vdpa_ops.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c
index b3d9b6c..63aa52d 100644
--- a/drivers/vdpa/sfc/sfc_vdpa_ops.c
+++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c
@@ -794,6 +794,8 @@
 	int vfio_dev_fd;
 	efx_rc_t rc;
 	unsigned int bar_offset;
+	volatile void *doorbell;
+	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdpa_dev;
 	struct sfc_vdpa_ops_data *ops_data;
 	struct vfio_region_info reg = { .argsz = sizeof(reg) };
@@ -856,6 +858,18 @@
 	sfc_vdpa_info(dev, "vDPA ops get_notify_area :: offset : 0x%" PRIx64,
 		      *offset);
 
+	pci_dev = sfc_vdpa_adapter_by_dev_handle(dev)->pdev;
+	doorbell = (uint8_t *)pci_dev->mem_resource[reg.index].addr + *offset;
+
+	/*
+	 * virtio-net driver in VM sends queue notifications before
+	 * vDPA has a chance to setup the queues and notification area,
+	 * and hence the HW misses these doorbell notifications.
+	 * Since, it is safe to send duplicate doorbell, send another
+	 * doorbell from vDPA driver as workaround for this timing issue.
+	 */
+	rte_write16(qid, doorbell);
+
 	return 0;
 }
 
-- 
1.8.3.1

Re: [PATCH] vdpa/sfc: handle sync issue between qemu and vhost-user

[Andrew Rybchenko]

When you send a new version, please, don't forget to specify
-v <next-version-number> on part format and use --in-reply-to
with the first mail ID. See contributors guidelines.

Also, new version should make it clear what is changed. See below.

On 6/27/22 11:49, abhimanyu.saini@xilinx.com wrote:
> From: Abhimanyu Saini <absaini@amd.com>
> 
> When DPDK app is running in the VF, it sometimes rings the doorbell
> before dev_config has had a chance to complete and hence it misses
> the event. As workaround, ring the doorbell when vDPA reports the
> notify_area to QEMU.
> 
> Fixes: 5e7596ba7cb3 ("vdpa/sfc: introduce Xilinx vDPA driver")

Above seems to be wrong. Nearby code is added later, so, it should be:

Fixes: 630be406dcbf ("vdpa/sfc: get queue notify area info")


> Cc: stable@dpdk.org
> 
> Signed-off-by: Vijay Kumar Srivastava <vsrivast@xilinx.com>
> Signed-off-by: Abhimanyu Saini <absaini@amd.com>
> ---

Version changelog here.

>   drivers/vdpa/sfc/sfc_vdpa_ops.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c
> index b3d9b6c..63aa52d 100644
> --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c
> +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c
> @@ -794,6 +794,8 @@
>   	int vfio_dev_fd;
>   	efx_rc_t rc;
>   	unsigned int bar_offset;
> +	volatile void *doorbell;
> +	struct rte_pci_device *pci_dev;
>   	struct rte_vdpa_device *vdpa_dev;
>   	struct sfc_vdpa_ops_data *ops_data;
>   	struct vfio_region_info reg = { .argsz = sizeof(reg) };
> @@ -856,6 +858,18 @@
>   	sfc_vdpa_info(dev, "vDPA ops get_notify_area :: offset : 0x%" PRIx64,
>   		      *offset);
>   
> +	pci_dev = sfc_vdpa_adapter_by_dev_handle(dev)->pdev;
> +	doorbell = (uint8_t *)pci_dev->mem_resource[reg.index].addr + *offset;
> +
> +	/*
> +	 * virtio-net driver in VM sends queue notifications before
> +	 * vDPA has a chance to setup the queues and notification area,
> +	 * and hence the HW misses these doorbell notifications.
> +	 * Since, it is safe to send duplicate doorbell, send another
> +	 * doorbell from vDPA driver as workaround for this timing issue.
> +	 */
> +	rte_write16(qid, doorbell);
> +
>   	return 0;
>   }
>   

[PATCH v2] vdpa/sfc: handle sync issue between qemu and vhost-user

[abhimanyu.saini]

From: Abhimanyu Saini <absaini@amd.com>

When DPDK app is running in the VF, it sometimes rings the doorbell
before dev_config has had a chance to complete and hence it misses
the event. As workaround, ring the doorbell when vDPA reports the
notify_area to QEMU.

Fixes: 630be406dcbf ("vdpa/sfc: get queue notify area info")
Cc: stable@dpdk.org

Signed-off-by: Vijay Kumar Srivastava <vsrivast@xilinx.com>
Signed-off-by: Abhimanyu Saini <absaini@amd.com>
---
v1:
* Update the commit id that this patch fixes

 drivers/vdpa/sfc/sfc_vdpa_ops.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c
index b3d9b6c..63aa52d 100644
--- a/drivers/vdpa/sfc/sfc_vdpa_ops.c
+++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c
@@ -794,6 +794,8 @@
 	int vfio_dev_fd;
 	efx_rc_t rc;
 	unsigned int bar_offset;
+	volatile void *doorbell;
+	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdpa_dev;
 	struct sfc_vdpa_ops_data *ops_data;
 	struct vfio_region_info reg = { .argsz = sizeof(reg) };
@@ -856,6 +858,18 @@
 	sfc_vdpa_info(dev, "vDPA ops get_notify_area :: offset : 0x%" PRIx64,
 		      *offset);
 
+	pci_dev = sfc_vdpa_adapter_by_dev_handle(dev)->pdev;
+	doorbell = (uint8_t *)pci_dev->mem_resource[reg.index].addr + *offset;
+
+	/*
+	 * virtio-net driver in VM sends queue notifications before
+	 * vDPA has a chance to setup the queues and notification area,
+	 * and hence the HW misses these doorbell notifications.
+	 * Since, it is safe to send duplicate doorbell, send another
+	 * doorbell from vDPA driver as workaround for this timing issue.
+	 */
+	rte_write16(qid, doorbell);
+
 	return 0;
 }
 
-- 
1.8.3.1

Re: [PATCH v2] vdpa/sfc: handle sync issue between qemu and vhost-user

[Andrew Rybchenko]

On 6/28/22 08:29, abhimanyu.saini@xilinx.com wrote:
> From: Abhimanyu Saini <absaini@amd.com>
> 
> When DPDK app is running in the VF, it sometimes rings the doorbell
> before dev_config has had a chance to complete and hence it misses
> the event. As workaround, ring the doorbell when vDPA reports the
> notify_area to QEMU.
> 
> Fixes: 630be406dcbf ("vdpa/sfc: get queue notify area info")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Vijay Kumar Srivastava <vsrivast@xilinx.com>
> Signed-off-by: Abhimanyu Saini <absaini@amd.com>

Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

Re: [PATCH v2] vdpa/sfc: handle sync issue between qemu and vhost-user

[Maxime Coquelin]

On 6/28/22 07:29, abhimanyu.saini@xilinx.com wrote:
> From: Abhimanyu Saini <absaini@amd.com>
> 
> When DPDK app is running in the VF, it sometimes rings the doorbell
> before dev_config has had a chance to complete and hence it misses
> the event. As workaround, ring the doorbell when vDPA reports the
> notify_area to QEMU.
> 
> Fixes: 630be406dcbf ("vdpa/sfc: get queue notify area info")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Vijay Kumar Srivastava <vsrivast@xilinx.com>
> Signed-off-by: Abhimanyu Saini <absaini@amd.com>
> ---
> v1:
> * Update the commit id that this patch fixes
> 
>   drivers/vdpa/sfc/sfc_vdpa_ops.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH v2] vdpa/sfc: handle sync issue between qemu and vhost-user

[Maxime Coquelin]

On 6/28/22 07:29, abhimanyu.saini@xilinx.com wrote:
> From: Abhimanyu Saini <absaini@amd.com>
> 
> When DPDK app is running in the VF, it sometimes rings the doorbell
> before dev_config has had a chance to complete and hence it misses
> the event. As workaround, ring the doorbell when vDPA reports the
> notify_area to QEMU.
> 
> Fixes: 630be406dcbf ("vdpa/sfc: get queue notify area info")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Vijay Kumar Srivastava <vsrivast@xilinx.com>
> Signed-off-by: Abhimanyu Saini <absaini@amd.com>
> ---
> v1:
> * Update the commit id that this patch fixes
> 
>   drivers/vdpa/sfc/sfc_vdpa_ops.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 

Applied to dpdk-next-virtio/main.

Thanks,
Maxime