[dpdk-dev] [PATCH] virtio_net: kick guest even when virtio queue is full

[dpdk-dev] [PATCH] virtio_net: kick guest even when virtio queue is full

[Agota Benyhe]

From: Patrik Andersson R <patrik.r.andersson@ericsson.com>

Changing the vring call file descriptor during virtio device enqueue
operation may lead to "kick" on a file descriptor that is closed. As
a consequence the guest might not be notified of new packets in the
enqueue.

The suggested change will add some extra load on DPDK and on the
guest if the queue is frequently full. Any application using DPDK
should avoid attempting retransmissions when the zero packets are
enqueued.

To overcome this problem, the kick operation in virtio_dev_merge_rx()
was excluded from the pkt_idx > 0 condition. A similar change was
done in virtio_dev_rx().

Signed-off-by: Patrik Andersson R <patrik.r.andersson@ericsson.com>
Signed-off-by: Agota Benyhe <agota.benyhe@ericsson.com>
---
 lib/librte_vhost/virtio_net.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 6fee16e..f5fc8c0 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -59,6 +59,13 @@ is_valid_virt_queue_idx(uint32_t idx, int is_tx, uint32_t nr_vring)
 }
 
 static __rte_always_inline void
+kick_guest_if_necessary(const struct vhost_virtqueue *vq) {
+	if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
+			&& (vq->callfd >= 0))
+		eventfd_write(vq->callfd, (eventfd_t)1);
+}
+
+static __rte_always_inline void
 do_flush_shadow_used_ring(struct virtio_net *dev, struct vhost_virtqueue *vq,
 			  uint16_t to, uint16_t from, uint16_t size)
 {
@@ -344,8 +351,11 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
 	free_entries = avail_idx - start_idx;
 	count = RTE_MIN(count, free_entries);
 	count = RTE_MIN(count, (uint32_t)MAX_PKT_BURST);
-	if (count == 0)
+
+	if (count == 0) {
+                kick_guest_if_necessary(vq);
 		goto out;
+        }
 
 	LOG_DEBUG(VHOST_DATA, "(%d) start_idx %d | end_idx %d\n",
 		dev->vid, start_idx, start_idx + count);
@@ -411,10 +421,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
 	/* flush used->idx update before we read avail->flags. */
 	rte_mb();
 
-	/* Kick the guest if necessary. */
-	if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
-			&& (vq->callfd >= 0))
-		eventfd_write(vq->callfd, (eventfd_t)1);
+        kick_guest_if_necessary(vq);
 out:
 	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
 		vhost_user_iotlb_rd_unlock(vq);
@@ -704,13 +711,9 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 
 		/* flush used->idx update before we read avail->flags. */
 		rte_mb();
-
-		/* Kick the guest if necessary. */
-		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
-				&& (vq->callfd >= 0))
-			eventfd_write(vq->callfd, (eventfd_t)1);
 	}
 
+        kick_guest_if_necessary(vq);
 out:
 	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
 		vhost_user_iotlb_rd_unlock(vq);
-- 
2.7.4

[dpdk-dev] [PATCH v2] virtio_net: kick guest even when virtio queue is full

[Agota Benyhe]

From: Patrik Andersson R <patrik.r.andersson@ericsson.com>

Changing the vring call file descriptor during virtio device enqueue
operation may lead to "kick" on a file descriptor that is closed. As
a consequence the guest might not be notified of new packets in the
enqueue.

The suggested change will add some extra load on DPDK and on the
guest if the queue is frequently full. Any application using DPDK
should avoid attempting retransmissions when the zero packets are
enqueued.

To overcome this problem, the kick operation in virtio_dev_merge_rx()
was excluded from the pkt_idx > 0 condition. A similar change was
done in virtio_dev_rx().

Signed-off-by: Patrik Andersson R <patrik.r.andersson@ericsson.com>
Signed-off-by: Agota Benyhe <agota.benyhe@ericsson.com>
---
 lib/librte_vhost/virtio_net.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 6fee16e..dcc1879 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -59,6 +59,13 @@ is_valid_virt_queue_idx(uint32_t idx, int is_tx, uint32_t nr_vring)
 }
 
 static __rte_always_inline void
+kick_guest_if_necessary(const struct vhost_virtqueue *vq) {
+	if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
+			&& (vq->callfd >= 0))
+		eventfd_write(vq->callfd, (eventfd_t)1);
+}
+
+static __rte_always_inline void
 do_flush_shadow_used_ring(struct virtio_net *dev, struct vhost_virtqueue *vq,
 			  uint16_t to, uint16_t from, uint16_t size)
 {
@@ -344,8 +351,11 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
 	free_entries = avail_idx - start_idx;
 	count = RTE_MIN(count, free_entries);
 	count = RTE_MIN(count, (uint32_t)MAX_PKT_BURST);
-	if (count == 0)
+
+	if (count == 0) {
+		kick_guest_if_necessary(vq);
 		goto out;
+	}
 
 	LOG_DEBUG(VHOST_DATA, "(%d) start_idx %d | end_idx %d\n",
 		dev->vid, start_idx, start_idx + count);
@@ -411,10 +421,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
 	/* flush used->idx update before we read avail->flags. */
 	rte_mb();
 
-	/* Kick the guest if necessary. */
-	if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
-			&& (vq->callfd >= 0))
-		eventfd_write(vq->callfd, (eventfd_t)1);
+	kick_guest_if_necessary(vq);
 out:
 	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
 		vhost_user_iotlb_rd_unlock(vq);
@@ -704,13 +711,9 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 
 		/* flush used->idx update before we read avail->flags. */
 		rte_mb();
-
-		/* Kick the guest if necessary. */
-		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
-				&& (vq->callfd >= 0))
-			eventfd_write(vq->callfd, (eventfd_t)1);
 	}
 
+	kick_guest_if_necessary(vq);
 out:
 	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
 		vhost_user_iotlb_rd_unlock(vq);
-- 
2.7.4

Re: [dpdk-dev] [PATCH v2] virtio_net: kick guest even when virtio queue is full

[Maxime Coquelin]

Hi Agota, Patrick,

On 12/05/2017 12:30 PM, Agota Benyhe wrote:
> From: Patrik Andersson R<patrik.r.andersson@ericsson.com>
> 
> Changing the vring call file descriptor during virtio device enqueue
> operation may lead to "kick" on a file descriptor that is closed. As
> a consequence the guest might not be notified of new packets in the
> enqueue.
> 
> The suggested change will add some extra load on DPDK and on the
> guest if the queue is frequently full. Any application using DPDK
> should avoid attempting retransmissions when the zero packets are
> enqueued.
> 
> To overcome this problem, the kick operation in virtio_dev_merge_rx()
> was excluded from the pkt_idx > 0 condition. A similar change was
> done in virtio_dev_rx().
> 
> Signed-off-by: Patrik Andersson R<patrik.r.andersson@ericsson.com>
> Signed-off-by: Agota Benyhe<agota.benyhe@ericsson.com>
> ---
>   lib/librte_vhost/virtio_net.c | 23 +++++++++++++----------
>   1 file changed, 13 insertions(+), 10 deletions(-)

We are working on a patch that protects enqueue & dequeue paths from
vhost-user requests handling.
For your case, it would protect call_fd to be changed while vring is
being processed by the PMD thread.

Do you think that would solve the problem you are facing?

Regards,
Maxime

Re: [dpdk-dev] [PATCH v2] virtio_net: kick guest even when virtio queue is full

[Patrik Andersson R]

Hi Maxime,

apologies for the late answer.

Yes I would think that it would solve the problem that we had. But
there is a slight risk that a driver in a VM (not within our area of 
influence) was partly responsible for the severity of the fault and 
that we might again experience some difficulties even with your
change. Though the risk is likely to be very low.

However, if your patch protects the file descriptors when in use I 
would consider that a better solution than doing the extra writes
on the full queue.

I would recommend going with your solution rather than the one
suggested by us.

Regards,

Patrik



Från: Maxime Coquelin <maxime.coquelin@redhat.com>
Skickat: den 8 december 2017 17:11
Till: Ágota Benyhe; dev@dpdk.org
Kopia: Patrik Andersson R; Yuanhan Liu; Jianfeng Tan
Ämne: Re: [dpdk-dev] [PATCH v2] virtio_net: kick guest even when virtio queue is full
  

Hi Agota, Patrick,

On 12/05/2017 12:30 PM, Agota Benyhe wrote:
> From: Patrik Andersson R<patrik.r.andersson@ericsson.com>
> 
> Changing the vring call file descriptor during virtio device enqueue
> operation may lead to "kick" on a file descriptor that is closed. As
> a consequence the guest might not be notified of new packets in the
> enqueue.
> 
> The suggested change will add some extra load on DPDK and on the
> guest if the queue is frequently full. Any application using DPDK
> should avoid attempting retransmissions when the zero packets are
> enqueued.
> 
> To overcome this problem, the kick operation in virtio_dev_merge_rx()
> was excluded from the pkt_idx > 0 condition. A similar change was
> done in virtio_dev_rx().
> 
> Signed-off-by: Patrik Andersson R<patrik.r.andersson@ericsson.com>
> Signed-off-by: Agota Benyhe<agota.benyhe@ericsson.com>
> ---
>   lib/librte_vhost/virtio_net.c | 23 +++++++++++++----------
>   1 file changed, 13 insertions(+), 10 deletions(-)

We are working on a patch that protects enqueue & dequeue paths from
vhost-user requests handling.
For your case, it would protect call_fd to be changed while vring is
being processed by the PMD thread.

Do you think that would solve the problem you are facing?

Regards,
Maxime