[PATCH v2 0/4] Virtio-user queues setup fixes

[PATCH v2 0/4] Virtio-user queues setup fixes

[Maxime Coquelin]

This series aims at fixing several issues found in
Virtio-user PMD related to queues setup and cleanup.

It has been tested with Vhost-vDPA backend using Nvidia
Cx6-Dx vDPA VF.

First patch in the series renames the queues iterator
helper, so it is not a fix. But I would suggest to have it
backported to ease backporting of the fixes.

Changes in v2:
--------------
- Fix regression in patch 4 reported by CI
- Prefix titles with net/virtio-user (David)
- Reword patch 2 title (David)
- Apply David acks on patches 1-3

Maxime Coquelin (4):
  net/virtio-user: rename queue iterator
  net/virtio-user: fix control queue destruction
  net/virtio-user: fix shadow control queue notification init
  net/virtio-user: fix control queue allocation

 .../net/virtio/virtio_user/virtio_user_dev.c  | 133 +++++++++---------
 1 file changed, 68 insertions(+), 65 deletions(-)

-- 
2.44.0

[PATCH v2 1/4] net/virtio-user: rename queue iterator

[Maxime Coquelin]

This is a preliminary rework to prepare for iterating
over queues for non-setup operations.

Also, remove the error log that does not provide much
information given the callbacks already provide one.

Acked-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/net/virtio/virtio_user/virtio_user_dev.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.c b/drivers/net/virtio/virtio_user/virtio_user_dev.c
index 4fdfe70e7c..c3d44880f5 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.c
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.c
@@ -129,7 +129,7 @@ virtio_user_kick_queue(struct virtio_user_dev *dev, uint32_t queue_sel)
 }
 
 static int
-virtio_user_queue_setup(struct virtio_user_dev *dev,
+virtio_user_foreach_queue(struct virtio_user_dev *dev,
 			int (*fn)(struct virtio_user_dev *, uint32_t))
 {
 	uint32_t i, nr_vq;
@@ -138,12 +138,9 @@ virtio_user_queue_setup(struct virtio_user_dev *dev,
 	if (dev->hw_cvq)
 		nr_vq++;
 
-	for (i = 0; i < nr_vq; i++) {
-		if (fn(dev, i) < 0) {
-			PMD_DRV_LOG(ERR, "(%s) setup VQ %u failed", dev->path, i);
+	for (i = 0; i < nr_vq; i++)
+		if (fn(dev, i) < 0)
 			return -1;
-		}
-	}
 
 	return 0;
 }
@@ -157,7 +154,7 @@ virtio_user_dev_set_features(struct virtio_user_dev *dev)
 	pthread_mutex_lock(&dev->mutex);
 
 	/* Step 0: tell vhost to create queues */
-	if (virtio_user_queue_setup(dev, virtio_user_create_queue) < 0)
+	if (virtio_user_foreach_queue(dev, virtio_user_create_queue) < 0)
 		goto error;
 
 	features = dev->features;
@@ -205,7 +202,7 @@ virtio_user_start_device(struct virtio_user_dev *dev)
 		goto error;
 
 	/* Step 3: kick queues */
-	ret = virtio_user_queue_setup(dev, virtio_user_kick_queue);
+	ret = virtio_user_foreach_queue(dev, virtio_user_kick_queue);
 	if (ret < 0)
 		goto error;
 
-- 
2.44.0

[PATCH v2 2/4] net/virtio-user: fix control queue destruction

[Maxime Coquelin]

This patch uses the freshly renamed iterator to destroy
queues at stop time. Doing this, we fix the missing
control queue destruction.

Fixes: 90966e8e5b67 ("net/virtio-user: send shadow virtqueue info to the backend")
Cc: stable@dpdk.org

Acked-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 .../net/virtio/virtio_user/virtio_user_dev.c  | 27 ++++++++++++-------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.c b/drivers/net/virtio/virtio_user/virtio_user_dev.c
index c3d44880f5..0776c54deb 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.c
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.c
@@ -33,6 +33,22 @@ const char * const virtio_user_backend_strings[] = {
 	[VIRTIO_USER_BACKEND_VHOST_VDPA] = "VHOST_VDPA",
 };
 
+static int
+virtio_user_destroy_queue(struct virtio_user_dev *dev, uint32_t queue_sel)
+{
+	struct vhost_vring_state state;
+	int ret;
+
+	state.index = queue_sel;
+	ret = dev->ops->get_vring_base(dev, &state);
+	if (ret < 0) {
+		PMD_DRV_LOG(ERR, "(%s) Failed to destroy queue %u", dev->path, queue_sel);
+		return -1;
+	}
+
+	return 0;
+}
+
 static int
 virtio_user_create_queue(struct virtio_user_dev *dev, uint32_t queue_sel)
 {
@@ -237,7 +253,6 @@ virtio_user_start_device(struct virtio_user_dev *dev)
 
 int virtio_user_stop_device(struct virtio_user_dev *dev)
 {
-	struct vhost_vring_state state;
 	uint32_t i;
 	int ret;
 
@@ -258,14 +273,8 @@ int virtio_user_stop_device(struct virtio_user_dev *dev)
 	}
 
 	/* Stop the backend. */
-	for (i = 0; i < dev->max_queue_pairs * 2; ++i) {
-		state.index = i;
-		ret = dev->ops->get_vring_base(dev, &state);
-		if (ret < 0) {
-			PMD_DRV_LOG(ERR, "(%s) get_vring_base failed, index=%u", dev->path, i);
-			goto err;
-		}
-	}
+	if (virtio_user_foreach_queue(dev, virtio_user_destroy_queue) < 0)
+		goto err;
 
 	dev->started = false;
 
-- 
2.44.0

[PATCH v2 3/4] net/virtio-user: fix shadow control queue notification init

[Maxime Coquelin]

The Virtio-user control queue kick and call FDs were not
uninitialized at device stop time.

This patch fixes this using the queues iterator helper for
both initialization and uninitialization.

Fixes: 90966e8e5b67 ("net/virtio-user: send shadow virtqueue info to the backend")
Cc: stable@dpdk.org

Acked-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 .../net/virtio/virtio_user/virtio_user_dev.c  | 90 +++++++++----------
 1 file changed, 43 insertions(+), 47 deletions(-)

diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.c b/drivers/net/virtio/virtio_user/virtio_user_dev.c
index 0776c54deb..912e87fecf 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.c
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.c
@@ -33,6 +33,45 @@ const char * const virtio_user_backend_strings[] = {
 	[VIRTIO_USER_BACKEND_VHOST_VDPA] = "VHOST_VDPA",
 };
 
+static int
+virtio_user_uninit_notify_queue(struct virtio_user_dev *dev, uint32_t queue_sel)
+{
+	if (dev->kickfds[queue_sel] >= 0) {
+		close(dev->kickfds[queue_sel]);
+		dev->kickfds[queue_sel] = -1;
+	}
+
+	if (dev->callfds[queue_sel] >= 0) {
+		close(dev->callfds[queue_sel]);
+		dev->callfds[queue_sel] = -1;
+	}
+
+	return 0;
+}
+
+static int
+virtio_user_init_notify_queue(struct virtio_user_dev *dev, uint32_t queue_sel)
+{
+	/* May use invalid flag, but some backend uses kickfd and
+	 * callfd as criteria to judge if dev is alive. so finally we
+	 * use real event_fd.
+	 */
+	dev->callfds[queue_sel] = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK);
+	if (dev->callfds[queue_sel] < 0) {
+		PMD_DRV_LOG(ERR, "(%s) Failed to setup callfd for queue %u: %s",
+				dev->path, queue_sel, strerror(errno));
+		return -1;
+	}
+	dev->kickfds[queue_sel] = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK);
+	if (dev->kickfds[queue_sel] < 0) {
+		PMD_DRV_LOG(ERR, "(%s) Failed to setup kickfd for queue %u: %s",
+				dev->path, queue_sel, strerror(errno));
+		return -1;
+	}
+
+	return 0;
+}
+
 static int
 virtio_user_destroy_queue(struct virtio_user_dev *dev, uint32_t queue_sel)
 {
@@ -423,33 +462,9 @@ virtio_user_dev_init_mac(struct virtio_user_dev *dev, const char *mac)
 static int
 virtio_user_dev_init_notify(struct virtio_user_dev *dev)
 {
-	uint32_t i, j, nr_vq;
-	int callfd;
-	int kickfd;
-
-	nr_vq = dev->max_queue_pairs * 2;
-	if (dev->hw_cvq)
-		nr_vq++;
 
-	for (i = 0; i < nr_vq; i++) {
-		/* May use invalid flag, but some backend uses kickfd and
-		 * callfd as criteria to judge if dev is alive. so finally we
-		 * use real event_fd.
-		 */
-		callfd = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK);
-		if (callfd < 0) {
-			PMD_DRV_LOG(ERR, "(%s) callfd error, %s", dev->path, strerror(errno));
-			goto err;
-		}
-		kickfd = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK);
-		if (kickfd < 0) {
-			close(callfd);
-			PMD_DRV_LOG(ERR, "(%s) kickfd error, %s", dev->path, strerror(errno));
-			goto err;
-		}
-		dev->callfds[i] = callfd;
-		dev->kickfds[i] = kickfd;
-	}
+	if (virtio_user_foreach_queue(dev, virtio_user_init_notify_queue) < 0)
+		goto err;
 
 	if (dev->device_features & (1ULL << VIRTIO_F_NOTIFICATION_DATA))
 		if (dev->ops->map_notification_area &&
@@ -458,16 +473,7 @@ virtio_user_dev_init_notify(struct virtio_user_dev *dev)
 
 	return 0;
 err:
-	for (j = 0; j < i; j++) {
-		if (dev->kickfds[j] >= 0) {
-			close(dev->kickfds[j]);
-			dev->kickfds[j] = -1;
-		}
-		if (dev->callfds[j] >= 0) {
-			close(dev->callfds[j]);
-			dev->callfds[j] = -1;
-		}
-	}
+	virtio_user_foreach_queue(dev, virtio_user_uninit_notify_queue);
 
 	return -1;
 }
@@ -475,18 +481,8 @@ virtio_user_dev_init_notify(struct virtio_user_dev *dev)
 static void
 virtio_user_dev_uninit_notify(struct virtio_user_dev *dev)
 {
-	uint32_t i;
+	virtio_user_foreach_queue(dev, virtio_user_uninit_notify_queue);
 
-	for (i = 0; i < dev->max_queue_pairs * 2; ++i) {
-		if (dev->kickfds[i] >= 0) {
-			close(dev->kickfds[i]);
-			dev->kickfds[i] = -1;
-		}
-		if (dev->callfds[i] >= 0) {
-			close(dev->callfds[i]);
-			dev->callfds[i] = -1;
-		}
-	}
 	if (dev->ops->unmap_notification_area && dev->notify_area)
 		dev->ops->unmap_notification_area(dev);
 }
-- 
2.44.0

[PATCH v2 4/4] net/virtio-user: fix control queue allocation

[Maxime Coquelin]

It is possible to have the control queue without the
device advertising VIRTIO_NET_F_MQ.

Rely on the VIRTIO_NET_F_CTRL_VQ feature being advertised
instead.

Fixes: 6fdf32d1e318 ("net/virtio-user: remove max queues limitation")
Cc: stable@dpdk.org

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/net/virtio/virtio_user/virtio_user_dev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.c b/drivers/net/virtio/virtio_user/virtio_user_dev.c
index 912e87fecf..b3aed48452 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.c
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.c
@@ -21,6 +21,7 @@
 #include <rte_io.h>
 
 #include "vhost.h"
+#include "virtio.h"
 #include "virtio_user_dev.h"
 #include "../virtio_ethdev.h"
 
@@ -615,7 +616,7 @@ virtio_user_alloc_vrings(struct virtio_user_dev *dev)
 	bool packed_ring = !!(dev->device_features & (1ull << VIRTIO_F_RING_PACKED));
 
 	nr_vrings = dev->max_queue_pairs * 2;
-	if (dev->device_features & (1ull << VIRTIO_NET_F_MQ))
+	if (dev->device_features & (1ull << VIRTIO_NET_F_CTRL_VQ))
 		nr_vrings++;
 
 	dev->callfds = rte_zmalloc("virtio_user_dev", nr_vrings * sizeof(*dev->callfds), 0);
-- 
2.44.0

Re: [PATCH v2 4/4] net/virtio-user: fix control queue allocation

[David Marchand]

On Thu, Mar 28, 2024 at 2:08 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
> It is possible to have the control queue without the
> device advertising VIRTIO_NET_F_MQ.
>
> Rely on the VIRTIO_NET_F_CTRL_VQ feature being advertised
> instead.
>
> Fixes: 6fdf32d1e318 ("net/virtio-user: remove max queues limitation")
> Cc: stable@dpdk.org
>
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Acked-by: David Marchand <david.marchand@redhat.com>


-- 
David Marchand

Re: [PATCH v2 0/4] Virtio-user queues setup fixes

[Maxime Coquelin]

On 3/28/24 14:08, Maxime Coquelin wrote:
> This series aims at fixing several issues found in
> Virtio-user PMD related to queues setup and cleanup.
> 
> It has been tested with Vhost-vDPA backend using Nvidia
> Cx6-Dx vDPA VF.
> 
> First patch in the series renames the queues iterator
> helper, so it is not a fix. But I would suggest to have it
> backported to ease backporting of the fixes.
> 
> Changes in v2:
> --------------
> - Fix regression in patch 4 reported by CI
> - Prefix titles with net/virtio-user (David)
> - Reword patch 2 title (David)
> - Apply David acks on patches 1-3
> 
> Maxime Coquelin (4):
>    net/virtio-user: rename queue iterator
>    net/virtio-user: fix control queue destruction
>    net/virtio-user: fix shadow control queue notification init
>    net/virtio-user: fix control queue allocation
> 
>   .../net/virtio/virtio_user/virtio_user_dev.c  | 133 +++++++++---------
>   1 file changed, 68 insertions(+), 65 deletions(-)
> 
Applied to next-virtio/for-next-net.

Thanks,
Maxime