[PATCH] vhost: cleanup vq resubmit info when set_inflight_fd

[PATCH] vhost: cleanup vq resubmit info when set_inflight_fd

[Haoqian He]

We should cleanup vq resubmit info when set_inflight_fd
before set_vring_kick which will check if there is any
inflight io waiting for resubmission.

Otherwise, when the vm is rebooting immediately after
reconnecting to the vhost target (inflight io has not
been resubmitted yet), the vhost backend still use the
old resubmit info set when reconnection.

Signed-off-by: Haoqian He <haoqian.he@smartx.com>
---
 lib/vhost/vhost_user.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 414192500e..7c54afc5fb 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
 		if (!vq)
 			continue;
 
+		cleanup_vq_inflight(dev, vq);
 		if (vq_is_packed(dev)) {
 			vq->inflight_packed = addr;
 			vq->inflight_packed->desc_num = queue_size;
-- 
2.41.0

Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd

[Haoqian He]

> 2024年3月21日 17:57，Haoqian He <haoqian.he@smartx.com> 写道：
> 
> We should cleanup vq resubmit info when set_inflight_fd
> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.
> 
> Otherwise, when the vm is rebooting immediately after
> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the
> old resubmit info set when reconnection.
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> lib/vhost/vhost_user.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
> 		if (!vq)
> 			continue;
> 
> +		cleanup_vq_inflight(dev, vq);
> 		if (vq_is_packed(dev)) {
> 			vq->inflight_packed = addr;
> 			vq->inflight_packed->desc_num = queue_size;
> -- 
> 2.41.0
> 

Ping.

Hi, This issue can be reproduced by restarting vm internally and the vhost live recovery continuously.

Thanks,
Haoqian

Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd

[Haoqian He]

[-- Attachment #1: Type: text/plain, Size: 1815 bytes --]



> 2024年3月21日 17:57，Haoqian He <haoqian.he@smartx.com> 写道：
> 
> We should cleanup vq resubmit info when set_inflight_fd
> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.
> 
> Otherwise, when the vm is rebooting immediately after
> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the
> old resubmit info set when reconnection.
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> lib/vhost/vhost_user.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
> 		if (!vq)
> 			continue;
> 
> +		cleanup_vq_inflight(dev, vq);
> 		if (vq_is_packed(dev)) {
> 			vq->inflight_packed = addr;
> 			vq->inflight_packed->desc_num = queue_size;
> -- 
> 2.41.0
> 

Ping.

Hi, Maxime.

This patch fix the potential error when VM reboot after vhost live recovery which
could lead to the VM hang as missing resubmit info cleanup.

If inflight io that should be resubmitted during the latest vhost reconnection has
not been submitted yet, so GET_VRING_BASE would not wait these inflight io,
at this time the resubmit info has been set and restart the VM immediately.

Currently, we do not cleanup the resubmit info before VM restart, so when VM
restarts, SET_VRING_KICK will resubmit these inflight io (If resubmit info is not
null, function set_vring_kick will return without updating resubmit info).

It’s an error, any stale inflight io should not be resubmitted after the VM restart.

Thanks,
Haoqian



 



[-- Attachment #2: Type: text/html, Size: 3426 bytes --]

Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd

[Maxime Coquelin]

Hi Haoqian,

We try to avoid passing functions or variable names in the commit title.
Maybe something like this would work:
"vhost: cleanup resubmit info before inflight setup"

On 3/21/24 10:57, Haoqian He wrote:
> We should cleanup vq resubmit info when set_inflight_fd

virtqueue*

> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.

IO

> 
> Otherwise, when the vm is rebooting immediately after

VM

> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the

uses

> old resubmit info set when reconnection.

reconnecting

> 

You need to add "Fixes:" tag pointing to the commit introducing the 
issue, so that it is backported in stable releases.

This is documented in the contribution guidelines if you have any doubt
on the formatting.

> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
>   lib/vhost/vhost_user.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
>   		if (!vq)
>   			continue;
>   
> +		cleanup_vq_inflight(dev, vq);
>   		if (vq_is_packed(dev)) {
>   			vq->inflight_packed = addr;
>   			vq->inflight_packed->desc_num = queue_size;

[PATCH v2] vhost: cleanup resubmit info before inflight setup

[Haoqian He]

This patch fixes a potential VM hang bug when the VM reboots after
vhost live recovery due to missing cleanup virtqueue resubmit info.

Specifically, if inflight IO that should be resubmitted during
the latest vhost reconnection has not been submitted yet while
VM rebooting, so GET_VRING_BASE would not wait for the inflight
IO, at this time the resubmit info has been.  When the VM restarts,
SET_VRING_KICK will resubmit the inflight IO (If resubmit info
is not null, function set_vring_kick will return without updating
resubmit info).

It’s an error, any stale inflight IO should not be resubmitted
after the VM restart.

The solution is to clean up virtqueue resubmit info when function
set_inflight_fd before function set_vring_kick.

Fixes: ad0a4ae491fe ("vhost: checkout resubmit inflight information")
Cc: stable@dpdk.org

Signed-off-by: Haoqian He <haoqian.he@smartx.com>
---
v2: rewrite the commit message.

 lib/vhost/vhost_user.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 414192500e..7c54afc5fb 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
 		if (!vq)
 			continue;
 
+		cleanup_vq_inflight(dev, vq);
 		if (vq_is_packed(dev)) {
 			vq->inflight_packed = addr;
 			vq->inflight_packed->desc_num = queue_size;
-- 
2.44.0

Re: [PATCH v2] vhost: cleanup resubmit info before inflight setup

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

Recheck-request: iol-compile-amd64-testing

The DPDK Community Lab updated to the latest Alpine image yesterday, which
resulted in all Alpine builds failing. The failure is unrelated to your
patch, and this recheck should remove the fail on Patchwork, as we have
disabled Alpine testing for now.

[-- Attachment #2: Type: text/html, Size: 361 bytes --]

Re: [PATCH v2] vhost: cleanup resubmit info before inflight setup

[Haoqian He]

> 2024年4月26日 19:09，Haoqian He <haoqian.he@smartx.com> 写道：
> 
> This patch fixes a potential VM hang bug when the VM reboots after
> vhost live recovery due to missing cleanup virtqueue resubmit info.
> 
> Specifically, if inflight IO that should be resubmitted during
> the latest vhost reconnection has not been submitted yet while
> VM rebooting, so GET_VRING_BASE would not wait for the inflight
> IO, at this time the resubmit info has been.  When the VM restarts,
> SET_VRING_KICK will resubmit the inflight IO (If resubmit info
> is not null, function set_vring_kick will return without updating
> resubmit info).
> 
> It’s an error, any stale inflight IO should not be resubmitted
> after the VM restart.
> 
> The solution is to clean up virtqueue resubmit info when function
> set_inflight_fd before function set_vring_kick.
> 
> Fixes: ad0a4ae491fe ("vhost: checkout resubmit inflight information")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> v2: rewrite the commit message.
> 
> lib/vhost/vhost_user.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
> 		if (!vq)
> 			continue;
> 
> +		cleanup_vq_inflight(dev, vq);
> 		if (vq_is_packed(dev)) {
> 			vq->inflight_packed = addr;
> 			vq->inflight_packed->desc_num = queue_size;
> -- 
> 2.44.0
> 

Ping.
Hi Maxime and Chenbo,
Do u have any suggestions for this patch?

Looking forward to hearing from u!

Thanks,
Haoqian

Re: [PATCH v2] vhost: cleanup resubmit info before inflight setup

[Maxime Coquelin]

On 4/26/24 13:09, Haoqian He wrote:
> This patch fixes a potential VM hang bug when the VM reboots after
> vhost live recovery due to missing cleanup virtqueue resubmit info.
> 
> Specifically, if inflight IO that should be resubmitted during
> the latest vhost reconnection has not been submitted yet while
> VM rebooting, so GET_VRING_BASE would not wait for the inflight
> IO, at this time the resubmit info has been.  When the VM restarts,
> SET_VRING_KICK will resubmit the inflight IO (If resubmit info
> is not null, function set_vring_kick will return without updating
> resubmit info).
> 
> It’s an error, any stale inflight IO should not be resubmitted
> after the VM restart.
> 
> The solution is to clean up virtqueue resubmit info when function
> set_inflight_fd before function set_vring_kick.
> 
> Fixes: ad0a4ae491fe ("vhost: checkout resubmit inflight information")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
> v2: rewrite the commit message.
> 
>   lib/vhost/vhost_user.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
>   		if (!vq)
>   			continue;
>   
> +		cleanup_vq_inflight(dev, vq);
>   		if (vq_is_packed(dev)) {
>   			vq->inflight_packed = addr;
>   			vq->inflight_packed->desc_num = queue_size;

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH] vhost: cleanup vq resubmit info when set_inflight_fd

[Maxime Coquelin]

On 3/21/24 10:57, Haoqian He wrote:
> We should cleanup vq resubmit info when set_inflight_fd
> before set_vring_kick which will check if there is any
> inflight io waiting for resubmission.
> 
> Otherwise, when the vm is rebooting immediately after
> reconnecting to the vhost target (inflight io has not
> been resubmitted yet), the vhost backend still use the
> old resubmit info set when reconnection.
> 
> Signed-off-by: Haoqian He <haoqian.he@smartx.com>
> ---
>   lib/vhost/vhost_user.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 414192500e..7c54afc5fb 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -1871,6 +1871,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev,
>   		if (!vq)
>   			continue;
>   
> +		cleanup_vq_inflight(dev, vq);
>   		if (vq_is_packed(dev)) {
>   			vq->inflight_packed = addr;
>   			vq->inflight_packed->desc_num = queue_size;
Applied to next-virtio/for-next-net.

Thanks,
Maxime