[dpdk-stable] [PATCH] vhost: fix interrupt suppression for the split ring

[dpdk-stable] [PATCH] vhost: fix interrupt suppression for the split ring

[Jiayu Hu]

The VIRTIO_RING_F_EVENT_IDX feature of split ring might
be broken, as the value of signalled_used is unpredictable
after live migration or start up. This patch fixes it by
using signalled_used_valid.

In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX
implementation of split ring match kernel backend to suppress
more interrupts.

Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification suppression")
Cc: stable@dpdk.org

Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
---
 lib/librte_vhost/vhost.h | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index fc31796..c4e5e34 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -633,16 +633,20 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
 	if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {
 		uint16_t old = vq->signalled_used;
 		uint16_t new = vq->last_used_idx;
+		bool signalled_used_valid = vq->signalled_used_valid;
+
+		vq->signalled_used = new;
+		vq->signalled_used_valid = true;
 
 		VHOST_LOG_DEBUG(VHOST_DATA, "%s: used_event_idx=%d, old=%d, new=%d\n",
 			__func__,
 			vhost_used_event(vq),
 			old, new);
-		if (vhost_need_event(vhost_used_event(vq), new, old)
-			&& (vq->callfd >= 0)) {
-			vq->signalled_used = vq->last_used_idx;
+
+		if ((vhost_need_event(vhost_used_event(vq), new, old) &&
+					(vq->callfd >= 0)) ||
+				unlikely(!signalled_used_valid))
 			eventfd_write(vq->callfd, (eventfd_t) 1);
-		}
 	} else {
 		/* Kick the guest if necessary. */
 		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
-- 
2.7.4

Re: [dpdk-stable] [PATCH] vhost: fix interrupt suppression for the split ring

[Tiwei Bie]

On Fri, Feb 22, 2019 at 04:13:50PM +0800, Jiayu Hu wrote:
> The VIRTIO_RING_F_EVENT_IDX feature of split ring might
> be broken, as the value of signalled_used is unpredictable
> after live migration or start up. This patch fixes it by
> using signalled_used_valid.

Seems you also need to invalidate the signalled_used in e.g.
set_vring_base, as signalled_used may not match last_used_idx
anymore e.g. when guest driver is reloaded.

> 
> In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX
> implementation of split ring match kernel backend to suppress
> more interrupts.
> 
> Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification suppression")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
> ---
>  lib/librte_vhost/vhost.h | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
> index fc31796..c4e5e34 100644
> --- a/lib/librte_vhost/vhost.h
> +++ b/lib/librte_vhost/vhost.h
> @@ -633,16 +633,20 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
>  	if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {
>  		uint16_t old = vq->signalled_used;
>  		uint16_t new = vq->last_used_idx;
> +		bool signalled_used_valid = vq->signalled_used_valid;
> +
> +		vq->signalled_used = new;
> +		vq->signalled_used_valid = true;
>  
>  		VHOST_LOG_DEBUG(VHOST_DATA, "%s: used_event_idx=%d, old=%d, new=%d\n",
>  			__func__,
>  			vhost_used_event(vq),
>  			old, new);
> -		if (vhost_need_event(vhost_used_event(vq), new, old)
> -			&& (vq->callfd >= 0)) {
> -			vq->signalled_used = vq->last_used_idx;
> +
> +		if ((vhost_need_event(vhost_used_event(vq), new, old) &&
> +					(vq->callfd >= 0)) ||
> +				unlikely(!signalled_used_valid))
>  			eventfd_write(vq->callfd, (eventfd_t) 1);
> -		}
>  	} else {
>  		/* Kick the guest if necessary. */
>  		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
> -- 
> 2.7.4
> 

Re: [dpdk-stable] [PATCH] vhost: fix interrupt suppression for the split ring

[Hu, Jiayu]

> -----Original Message-----
> From: Bie, Tiwei
> Sent: Monday, February 25, 2019 3:07 PM
> To: Hu, Jiayu <jiayu.hu@intel.com>
> Cc: dev@dpdk.org; maxime.coquelin@redhat.com; stable@dpdk.org
> Subject: Re: [PATCH] vhost: fix interrupt suppression for the split ring
> 
> On Fri, Feb 22, 2019 at 04:13:50PM +0800, Jiayu Hu wrote:
> > The VIRTIO_RING_F_EVENT_IDX feature of split ring might
> > be broken, as the value of signalled_used is unpredictable
> > after live migration or start up. This patch fixes it by
> > using signalled_used_valid.
> 
> Seems you also need to invalidate the signalled_used in e.g.
> set_vring_base, as signalled_used may not match last_used_idx
> anymore e.g. when guest driver is reloaded.

To handle the virtio device reload case, how about setting
signalled_used_valid to false in get_vring_base()?

Thanks,
Jiayu

> 
> >
> > In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX
> > implementation of split ring match kernel backend to suppress
> > more interrupts.
> >
> > Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification
> suppression")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
> > ---
> >  lib/librte_vhost/vhost.h | 12 ++++++++----
> >  1 file changed, 8 insertions(+), 4 deletions(-)
> >
> > diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
> > index fc31796..c4e5e34 100644
> > --- a/lib/librte_vhost/vhost.h
> > +++ b/lib/librte_vhost/vhost.h
> > @@ -633,16 +633,20 @@ vhost_vring_call_split(struct virtio_net *dev,
> struct vhost_virtqueue *vq)
> >  	if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {
> >  		uint16_t old = vq->signalled_used;
> >  		uint16_t new = vq->last_used_idx;
> > +		bool signalled_used_valid = vq->signalled_used_valid;
> > +
> > +		vq->signalled_used = new;
> > +		vq->signalled_used_valid = true;
> >
> >  		VHOST_LOG_DEBUG(VHOST_DATA, "%s:
> used_event_idx=%d, old=%d, new=%d\n",
> >  			__func__,
> >  			vhost_used_event(vq),
> >  			old, new);
> > -		if (vhost_need_event(vhost_used_event(vq), new, old)
> > -			&& (vq->callfd >= 0)) {
> > -			vq->signalled_used = vq->last_used_idx;
> > +
> > +		if ((vhost_need_event(vhost_used_event(vq), new, old) &&
> > +					(vq->callfd >= 0)) ||
> > +				unlikely(!signalled_used_valid))
> >  			eventfd_write(vq->callfd, (eventfd_t) 1);
> > -		}
> >  	} else {
> >  		/* Kick the guest if necessary. */
> >  		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
> > --
> > 2.7.4
> >

Re: [dpdk-stable] [PATCH] vhost: fix interrupt suppression for the split ring

[Tiwei Bie]

On Wed, Feb 27, 2019 at 01:38:28PM +0800, Hu, Jiayu wrote:
> > -----Original Message-----
> > From: Bie, Tiwei
> > Sent: Monday, February 25, 2019 3:07 PM
> > To: Hu, Jiayu <jiayu.hu@intel.com>
> > Cc: dev@dpdk.org; maxime.coquelin@redhat.com; stable@dpdk.org
> > Subject: Re: [PATCH] vhost: fix interrupt suppression for the split ring
> > 
> > On Fri, Feb 22, 2019 at 04:13:50PM +0800, Jiayu Hu wrote:
> > > The VIRTIO_RING_F_EVENT_IDX feature of split ring might
> > > be broken, as the value of signalled_used is unpredictable
> > > after live migration or start up. This patch fixes it by
> > > using signalled_used_valid.
> > 
> > Seems you also need to invalidate the signalled_used in e.g.
> > set_vring_base, as signalled_used may not match last_used_idx
> > anymore e.g. when guest driver is reloaded.
> 
> To handle the virtio device reload case, how about setting
> signalled_used_valid to false in get_vring_base()?

Sounds good.

> 
> Thanks,
> Jiayu
> 
> > 
> > >
> > > In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX
> > > implementation of split ring match kernel backend to suppress
> > > more interrupts.
> > >
> > > Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification
> > suppression")
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
> > > ---
> > >  lib/librte_vhost/vhost.h | 12 ++++++++----
> > >  1 file changed, 8 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
> > > index fc31796..c4e5e34 100644
> > > --- a/lib/librte_vhost/vhost.h
> > > +++ b/lib/librte_vhost/vhost.h
> > > @@ -633,16 +633,20 @@ vhost_vring_call_split(struct virtio_net *dev,
> > struct vhost_virtqueue *vq)
> > >  	if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {
> > >  		uint16_t old = vq->signalled_used;
> > >  		uint16_t new = vq->last_used_idx;
> > > +		bool signalled_used_valid = vq->signalled_used_valid;
> > > +
> > > +		vq->signalled_used = new;
> > > +		vq->signalled_used_valid = true;
> > >
> > >  		VHOST_LOG_DEBUG(VHOST_DATA, "%s:
> > used_event_idx=%d, old=%d, new=%d\n",
> > >  			__func__,
> > >  			vhost_used_event(vq),
> > >  			old, new);
> > > -		if (vhost_need_event(vhost_used_event(vq), new, old)
> > > -			&& (vq->callfd >= 0)) {
> > > -			vq->signalled_used = vq->last_used_idx;
> > > +
> > > +		if ((vhost_need_event(vhost_used_event(vq), new, old) &&
> > > +					(vq->callfd >= 0)) ||
> > > +				unlikely(!signalled_used_valid))
> > >  			eventfd_write(vq->callfd, (eventfd_t) 1);
> > > -		}
> > >  	} else {
> > >  		/* Kick the guest if necessary. */
> > >  		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
> > > --
> > > 2.7.4
> > >

[dpdk-stable] [PATCH v2] vhost: fix interrupt suppression for the split ring

[Jiayu Hu]

The VIRTIO_RING_F_EVENT_IDX feature of split ring might
be broken, as the value of signalled_used is invalid
after live migration, start up and virtio driver reload.
This patch fixes it by using signalled_used_valid.

In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX
implementation of split ring match kernel backend to suppress
more interrupts.

Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification suppression")
Cc: stable@dpdk.org

Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
---
change in v2:
- fix virtio-net driver reload

 lib/librte_vhost/vhost.h      | 12 ++++++++----
 lib/librte_vhost/vhost_user.c |  2 ++
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index f008ec4..e9138df 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -633,16 +633,20 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
 	if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {
 		uint16_t old = vq->signalled_used;
 		uint16_t new = vq->last_used_idx;
+		bool signalled_used_valid = vq->signalled_used_valid;
+
+		vq->signalled_used = new;
+		vq->signalled_used_valid = true;
 
 		VHOST_LOG_DEBUG(VHOST_DATA, "%s: used_event_idx=%d, old=%d, new=%d\n",
 			__func__,
 			vhost_used_event(vq),
 			old, new);
-		if (vhost_need_event(vhost_used_event(vq), new, old)
-			&& (vq->callfd >= 0)) {
-			vq->signalled_used = vq->last_used_idx;
+
+		if ((vhost_need_event(vhost_used_event(vq), new, old) &&
+					(vq->callfd >= 0)) ||
+				unlikely(!signalled_used_valid))
 			eventfd_write(vq->callfd, (eventfd_t) 1);
-		}
 	} else {
 		/* Kick the guest if necessary. */
 		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
index 36c0c67..01a8d80 100644
--- a/lib/librte_vhost/vhost_user.c
+++ b/lib/librte_vhost/vhost_user.c
@@ -1298,6 +1298,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
 
 	vq->callfd = VIRTIO_UNINITIALIZED_EVENTFD;
 
+	vq->signalled_used_valid = false;
+
 	if (dev->dequeue_zero_copy)
 		free_zmbufs(vq);
 	if (vq_is_packed(dev)) {
-- 
2.7.4

Re: [dpdk-stable] [PATCH v2] vhost: fix interrupt suppression for the split ring

[Wang, Yinan]

Test-by: Wang, Yinan <yinan.wang@intel.com>

Best Wishes,
Yinan

> -----Original Message-----
> From: Hu, Jiayu
> Sent: 2019年3月17日 14:39
> To: dev@dpdk.org
> Cc: Bie, Tiwei <tiwei.bie@intel.com>; maxime.coquelin@redhat.com; Wang,
> Yinan <yinan.wang@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>;
> stable@dpdk.org
> Subject: [PATCH v2] vhost: fix interrupt suppression for the split ring
> 
> The VIRTIO_RING_F_EVENT_IDX feature of split ring might be broken, as the
> value of signalled_used is invalid after live migration, start up and virtio driver
> reload.
> This patch fixes it by using signalled_used_valid.
> 
> In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX implementation
> of split ring match kernel backend to suppress more interrupts.
> 
> Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification
> suppression")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
> ---
> change in v2:
> - fix virtio-net driver reload
> 
>  lib/librte_vhost/vhost.h      | 12 ++++++++----
>  lib/librte_vhost/vhost_user.c |  2 ++
>  2 files changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h index
> f008ec4..e9138df 100644
> --- a/lib/librte_vhost/vhost.h
> +++ b/lib/librte_vhost/vhost.h
> @@ -633,16 +633,20 @@ vhost_vring_call_split(struct virtio_net *dev, struct
> vhost_virtqueue *vq)
>  	if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {
>  		uint16_t old = vq->signalled_used;
>  		uint16_t new = vq->last_used_idx;
> +		bool signalled_used_valid = vq->signalled_used_valid;
> +
> +		vq->signalled_used = new;
> +		vq->signalled_used_valid = true;
> 
>  		VHOST_LOG_DEBUG(VHOST_DATA, "%s: used_event_idx=%d,
> old=%d, new=%d\n",
>  			__func__,
>  			vhost_used_event(vq),
>  			old, new);
> -		if (vhost_need_event(vhost_used_event(vq), new, old)
> -			&& (vq->callfd >= 0)) {
> -			vq->signalled_used = vq->last_used_idx;
> +
> +		if ((vhost_need_event(vhost_used_event(vq), new, old) &&
> +					(vq->callfd >= 0)) ||
> +				unlikely(!signalled_used_valid))
>  			eventfd_write(vq->callfd, (eventfd_t) 1);
> -		}
>  	} else {
>  		/* Kick the guest if necessary. */
>  		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT) diff --git
> a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c index
> 36c0c67..01a8d80 100644
> --- a/lib/librte_vhost/vhost_user.c
> +++ b/lib/librte_vhost/vhost_user.c
> @@ -1298,6 +1298,8 @@ vhost_user_get_vring_base(struct virtio_net
> **pdev,
> 
>  	vq->callfd = VIRTIO_UNINITIALIZED_EVENTFD;
> 
> +	vq->signalled_used_valid = false;
> +
>  	if (dev->dequeue_zero_copy)
>  		free_zmbufs(vq);
>  	if (vq_is_packed(dev)) {
> --
> 2.7.4

Re: [dpdk-stable] [PATCH v2] vhost: fix interrupt suppression for the split ring

[Tiwei Bie]

On Sun, Mar 17, 2019 at 02:38:32PM +0800, Jiayu Hu wrote:
> The VIRTIO_RING_F_EVENT_IDX feature of split ring might
> be broken, as the value of signalled_used is invalid
> after live migration, start up and virtio driver reload.
> This patch fixes it by using signalled_used_valid.
> 
> In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX
> implementation of split ring match kernel backend to suppress
> more interrupts.
> 
> Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification suppression")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
> ---
> change in v2:
> - fix virtio-net driver reload
> 
>  lib/librte_vhost/vhost.h      | 12 ++++++++----
>  lib/librte_vhost/vhost_user.c |  2 ++
>  2 files changed, 10 insertions(+), 4 deletions(-)

Reviewed-by: Tiwei Bie <tiwei.bie@intel.com>

Re: [dpdk-stable] [PATCH v2] vhost: fix interrupt suppression for the split ring

[Maxime Coquelin]

On 3/17/19 7:38 AM, Jiayu Hu wrote:
> The VIRTIO_RING_F_EVENT_IDX feature of split ring might
> be broken, as the value of signalled_used is invalid
> after live migration, start up and virtio driver reload.
> This patch fixes it by using signalled_used_valid.
> 
> In addition, this patch makes the VIRTIO_RING_F_EVENT_IDX
> implementation of split ring match kernel backend to suppress
> more interrupts.
> 
> Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification suppression")
> Cc:stable@dpdk.org
> 
> Signed-off-by: Jiayu Hu<jiayu.hu@intel.com>
> ---
> change in v2:
> - fix virtio-net driver reload
> 
>   lib/librte_vhost/vhost.h      | 12 ++++++++----
>   lib/librte_vhost/vhost_user.c |  2 ++
>   2 files changed, 10 insertions(+), 4 deletions(-)


Applied to dpdk-next-virtio/master branch.

Thanks,
Maxime