DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd
@ 2024-03-21  9:57 Haoqian He
  2024-04-03  5:08 ` Haoqian He
                   ` (3 more replies)
  0 siblings, 4 replies; 9+ messages in thread
From: Haoqian He @ 2024-03-21  9:57 UTC (permalink / raw)
  To: dev; +Cc: Maxime Coquelin, Chenbo Xia

We should cleanup vq resubmit info when set_inflight_fd
before set_vring_kick which will check if there is any
inflight io waiting for resubmission.

Otherwise, when the vm is rebooting immediately after
reconnecting to the vhost target (inflight io has not
been resubmitted yet), the vhost backend still use the
old resubmit info set when reconnection.

Signed-off-by: Haoqian He <haoqian.he@smartx.com>
---
 lib/vhost/vhost_user.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 414192500e..7c54afc5fb 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
 		if (!vq)
 			continue;
 
+		cleanup_vq_inflight(dev, vq);
 		if (vq_is_packed(dev)) {
 			vq->inflight_packed = addr;
 			vq->inflight_packed->desc_num = queue_size;
-- 
2.41.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd
  2024-03-21  9:57 [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Haoqian He
@ 2024-04-03  5:08 ` Haoqian He
  2024-04-12  8:10 ` Haoqian He
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 9+ messages in thread
From: Haoqian He @ 2024-04-03  5:08 UTC (permalink / raw)
  To: dev; +Cc: Maxime Coquelin, Chenbo Xia, Li Feng, yuhua



> 2024年3月21日 17:57,Haoqian He <haoqian.he@smartx.com> 写道:
> 
> We should cleanup vq resubmit info when set_inflight_fd
> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.
> 
> Otherwise, when the vm is rebooting immediately after
> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the
> old resubmit info set when reconnection.
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> lib/vhost/vhost_user.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
> 		if (!vq)
> 			continue;
> 
> +		cleanup_vq_inflight(dev, vq);
> 		if (vq_is_packed(dev)) {
> 			vq->inflight_packed = addr;
> 			vq->inflight_packed->desc_num = queue_size;
> -- 
> 2.41.0
> 

Ping.

Hi, This issue can be reproduced by restarting vm internally and the vhost live recovery continuously.

Thanks,
Haoqian

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd
  2024-03-21  9:57 [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Haoqian He
  2024-04-03  5:08 ` Haoqian He
@ 2024-04-12  8:10 ` Haoqian He
  2024-04-25 13:12 ` Maxime Coquelin
  2024-06-12  8:30 ` [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Maxime Coquelin
  3 siblings, 0 replies; 9+ messages in thread
From: Haoqian He @ 2024-04-12  8:10 UTC (permalink / raw)
  To: dev, Maxime Coquelin; +Cc: Chenbo Xia, Li Feng

[-- Attachment #1: Type: text/plain, Size: 1815 bytes --]



> 2024年3月21日 17:57,Haoqian He <haoqian.he@smartx.com> 写道:
> 
> We should cleanup vq resubmit info when set_inflight_fd
> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.
> 
> Otherwise, when the vm is rebooting immediately after
> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the
> old resubmit info set when reconnection.
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> lib/vhost/vhost_user.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
> 		if (!vq)
> 			continue;
> 
> +		cleanup_vq_inflight(dev, vq);
> 		if (vq_is_packed(dev)) {
> 			vq->inflight_packed = addr;
> 			vq->inflight_packed->desc_num = queue_size;
> -- 
> 2.41.0
> 

Ping.

Hi, Maxime.

This patch fix the potential error when VM reboot after vhost live recovery which
could lead to the VM hang as missing resubmit info cleanup.

If inflight io that should be resubmitted during the latest vhost reconnection has
not been submitted yet, so GET_VRING_BASE would not wait these inflight io,
at this time the resubmit info has been set and restart the VM immediately.

Currently, we do not cleanup the resubmit info before VM restart, so when VM
restarts, SET_VRING_KICK will resubmit these inflight io (If resubmit info is not
null, function set_vring_kick will return without updating resubmit info).

It’s an error, any stale inflight io should not be resubmitted after the VM restart.

Thanks,
Haoqian



 



[-- Attachment #2: Type: text/html, Size: 3426 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd
  2024-03-21  9:57 [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Haoqian He
  2024-04-03  5:08 ` Haoqian He
  2024-04-12  8:10 ` Haoqian He
@ 2024-04-25 13:12 ` Maxime Coquelin
  2024-04-26 11:09   ` [PATCH v2] vhost: cleanup resubmit info before inflight setup Haoqian He
  2024-06-12  8:30 ` [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Maxime Coquelin
  3 siblings, 1 reply; 9+ messages in thread
From: Maxime Coquelin @ 2024-04-25 13:12 UTC (permalink / raw)
  To: Haoqian He, dev; +Cc: Chenbo Xia

Hi Haoqian,

We try to avoid passing functions or variable names in the commit title.
Maybe something like this would work:
"vhost: cleanup resubmit info before inflight setup"

On 3/21/24 10:57, Haoqian He wrote:
> We should cleanup vq resubmit info when set_inflight_fd

virtqueue*

> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.

IO

> 
> Otherwise, when the vm is rebooting immediately after

VM

> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the

uses

> old resubmit info set when reconnection.

reconnecting

> 

You need to add "Fixes:" tag pointing to the commit introducing the 
issue, so that it is backported in stable releases.

This is documented in the contribution guidelines if you have any doubt
on the formatting.

> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
>   lib/vhost/vhost_user.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
>   		if (!vq)
>   			continue;
>   
> +		cleanup_vq_inflight(dev, vq);
>   		if (vq_is_packed(dev)) {
>   			vq->inflight_packed = addr;
>   			vq->inflight_packed->desc_num = queue_size;


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2] vhost: cleanup resubmit info before inflight setup
  2024-04-25 13:12 ` Maxime Coquelin
@ 2024-04-26 11:09   ` Haoqian He
  2024-04-26 21:28     ` Patrick Robb
                       ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Haoqian He @ 2024-04-26 11:09 UTC (permalink / raw)
  To: maxime.coquelin; +Cc: chenbox, dev, haoqian.he, fengli, stable

This patch fixes a potential VM hang bug when the VM reboots after
vhost live recovery due to missing cleanup virtqueue resubmit info.

Specifically, if inflight IO that should be resubmitted during
the latest vhost reconnection has not been submitted yet while
VM rebooting, so GET_VRING_BASE would not wait for the inflight
IO, at this time the resubmit info has been.  When the VM restarts,
SET_VRING_KICK will resubmit the inflight IO (If resubmit info
is not null, function set_vring_kick will return without updating
resubmit info).

It’s an error, any stale inflight IO should not be resubmitted
after the VM restart.

The solution is to clean up virtqueue resubmit info when function
set_inflight_fd before function set_vring_kick.

Fixes: ad0a4ae491fe ("vhost: checkout resubmit inflight information")
Cc: stable@dpdk.org

Signed-off-by: Haoqian He <haoqian.he@smartx.com>
---
v2: rewrite the commit message.

 lib/vhost/vhost_user.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 414192500e..7c54afc5fb 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
 		if (!vq)
 			continue;
 
+		cleanup_vq_inflight(dev, vq);
 		if (vq_is_packed(dev)) {
 			vq->inflight_packed = addr;
 			vq->inflight_packed->desc_num = queue_size;
-- 
2.44.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] vhost: cleanup resubmit info before inflight setup
  2024-04-26 11:09   ` [PATCH v2] vhost: cleanup resubmit info before inflight setup Haoqian He
@ 2024-04-26 21:28     ` Patrick Robb
  2024-05-21  8:55     ` Haoqian He
  2024-06-07 13:50     ` Maxime Coquelin
  2 siblings, 0 replies; 9+ messages in thread
From: Patrick Robb @ 2024-04-26 21:28 UTC (permalink / raw)
  To: Haoqian He; +Cc: maxime.coquelin, chenbox, dev, fengli, stable

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

Recheck-request: iol-compile-amd64-testing

The DPDK Community Lab updated to the latest Alpine image yesterday, which
resulted in all Alpine builds failing. The failure is unrelated to your
patch, and this recheck should remove the fail on Patchwork, as we have
disabled Alpine testing for now.

[-- Attachment #2: Type: text/html, Size: 361 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] vhost: cleanup resubmit info before inflight setup
  2024-04-26 11:09   ` [PATCH v2] vhost: cleanup resubmit info before inflight setup Haoqian He
  2024-04-26 21:28     ` Patrick Robb
@ 2024-05-21  8:55     ` Haoqian He
  2024-06-07 13:50     ` Maxime Coquelin
  2 siblings, 0 replies; 9+ messages in thread
From: Haoqian He @ 2024-05-21  8:55 UTC (permalink / raw)
  To: Maxime Coquelin, Chenbo Xia, dev; +Cc: Li Feng


> 2024年4月26日 19:09,Haoqian He <haoqian.he@smartx.com> 写道:
> 
> This patch fixes a potential VM hang bug when the VM reboots after
> vhost live recovery due to missing cleanup virtqueue resubmit info.
> 
> Specifically, if inflight IO that should be resubmitted during
> the latest vhost reconnection has not been submitted yet while
> VM rebooting, so GET_VRING_BASE would not wait for the inflight
> IO, at this time the resubmit info has been.  When the VM restarts,
> SET_VRING_KICK will resubmit the inflight IO (If resubmit info
> is not null, function set_vring_kick will return without updating
> resubmit info).
> 
> It’s an error, any stale inflight IO should not be resubmitted
> after the VM restart.
> 
> The solution is to clean up virtqueue resubmit info when function
> set_inflight_fd before function set_vring_kick.
> 
> Fixes: ad0a4ae491fe ("vhost: checkout resubmit inflight information")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> v2: rewrite the commit message.
> 
> lib/vhost/vhost_user.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
> 		if (!vq)
> 			continue;
> 
> +		cleanup_vq_inflight(dev, vq);
> 		if (vq_is_packed(dev)) {
> 			vq->inflight_packed = addr;
> 			vq->inflight_packed->desc_num = queue_size;
> -- 
> 2.44.0
> 

Ping.
Hi Maxime and Chenbo,
Do u have any suggestions for this patch?

Looking forward to hearing from u!

Thanks,
Haoqian

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] vhost: cleanup resubmit info before inflight setup
  2024-04-26 11:09   ` [PATCH v2] vhost: cleanup resubmit info before inflight setup Haoqian He
  2024-04-26 21:28     ` Patrick Robb
  2024-05-21  8:55     ` Haoqian He
@ 2024-06-07 13:50     ` Maxime Coquelin
  2 siblings, 0 replies; 9+ messages in thread
From: Maxime Coquelin @ 2024-06-07 13:50 UTC (permalink / raw)
  To: Haoqian He; +Cc: chenbox, dev, fengli, stable



On 4/26/24 13:09, Haoqian He wrote:
> This patch fixes a potential VM hang bug when the VM reboots after
> vhost live recovery due to missing cleanup virtqueue resubmit info.
> 
> Specifically, if inflight IO that should be resubmitted during
> the latest vhost reconnection has not been submitted yet while
> VM rebooting, so GET_VRING_BASE would not wait for the inflight
> IO, at this time the resubmit info has been.  When the VM restarts,
> SET_VRING_KICK will resubmit the inflight IO (If resubmit info
> is not null, function set_vring_kick will return without updating
> resubmit info).
> 
> It’s an error, any stale inflight IO should not be resubmitted
> after the VM restart.
> 
> The solution is to clean up virtqueue resubmit info when function
> set_inflight_fd before function set_vring_kick.
> 
> Fixes: ad0a4ae491fe ("vhost: checkout resubmit inflight information")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> v2: rewrite the commit message.
> 
>   lib/vhost/vhost_user.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
>   		if (!vq)
>   			continue;
>   
> +		cleanup_vq_inflight(dev, vq);
>   		if (vq_is_packed(dev)) {
>   			vq->inflight_packed = addr;
>   			vq->inflight_packed->desc_num = queue_size;

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd
  2024-03-21  9:57 [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Haoqian He
                   ` (2 preceding siblings ...)
  2024-04-25 13:12 ` Maxime Coquelin
@ 2024-06-12  8:30 ` Maxime Coquelin
  3 siblings, 0 replies; 9+ messages in thread
From: Maxime Coquelin @ 2024-06-12  8:30 UTC (permalink / raw)
  To: Haoqian He, dev; +Cc: Chenbo Xia



On 3/21/24 10:57, Haoqian He wrote:
> We should cleanup vq resubmit info when set_inflight_fd
> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.
> 
> Otherwise, when the vm is rebooting immediately after
> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the
> old resubmit info set when reconnection.
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
>   lib/vhost/vhost_user.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
>   		if (!vq)
>   			continue;
>   
> +		cleanup_vq_inflight(dev, vq);
>   		if (vq_is_packed(dev)) {
>   			vq->inflight_packed = addr;
>   			vq->inflight_packed->desc_num = queue_size;
Applied to next-virtio/for-next-net.

Thanks,
Maxime


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2024-06-12  8:30 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-21  9:57 [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Haoqian He
2024-04-03  5:08 ` Haoqian He
2024-04-12  8:10 ` Haoqian He
2024-04-25 13:12 ` Maxime Coquelin
2024-04-26 11:09   ` [PATCH v2] vhost: cleanup resubmit info before inflight setup Haoqian He
2024-04-26 21:28     ` Patrick Robb
2024-05-21  8:55     ` Haoqian He
2024-06-07 13:50     ` Maxime Coquelin
2024-06-12  8:30 ` [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd Maxime Coquelin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).