[PATCH] vdpa/mlx5: fix unregister kick handler order

[PATCH] vdpa/mlx5: fix unregister kick handler order

[Yajun Wu]

The mlx5_vdpa_virtq_kick_handler function may still be running and waiting
on virtq->virtq_lock while mlx5_vdpa_cqe_event_unset function is trying to
re-initialize the virtq->virtq_lock.

This causes mlx5_vdpa_virtq_kick_handler thread can't be wake up and can't
be unregister. Following print may loop forever when calling
rte_vhost_driver_unregister(socket_path):

    mlx5_vdpa: Try again to unregister fd 154 of virtq 11 interrupt
    mlx5_vdpa: Try again to unregister fd 154 of virtq 11 interrupt
    ...

The fix is to move mlx5_vdpa_virtq_unregister_intr_handle before
mlx5_vdpa_cqe_event_unset.

Fixes: 057f7d2084 ("vdpa/mlx5: optimize datapath-control synchronization")
Cc: stable@dpdk.org

Signed-off-by: Yajun Wu <yajunw@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c         | 1 +
 drivers/vdpa/mlx5/mlx5_vdpa_cthread.c | 1 -
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index f1737f82a8..8b1de8bd62 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -282,6 +282,7 @@ _internal_mlx5_vdpa_dev_close(struct mlx5_vdpa_priv *priv,
 	int ret = 0;
 	int vid = priv->vid;
 
+	mlx5_vdpa_virtq_unreg_intr_handle_all(priv);
 	mlx5_vdpa_cqe_event_unset(priv);
 	if (priv->state == MLX5_VDPA_STATE_CONFIGURED) {
 		ret |= mlx5_vdpa_lm_log(priv);
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c b/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c
index 6e6624e5a3..1d84e422d4 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c
@@ -190,7 +190,6 @@ mlx5_vdpa_c_thread_handle(void *arg)
 			pthread_mutex_unlock(&virtq->virtq_lock);
 			break;
 		case MLX5_VDPA_TASK_DEV_CLOSE_NOWAIT:
-			mlx5_vdpa_virtq_unreg_intr_handle_all(priv);
 			pthread_mutex_lock(&priv->steer_update_lock);
 			mlx5_vdpa_steer_unset(priv);
 			pthread_mutex_unlock(&priv->steer_update_lock);
-- 
2.27.0

Re: [PATCH] vdpa/mlx5: fix unregister kick handler order

[Maxime Coquelin]

On 8/8/23 13:32, Yajun Wu wrote:
> The mlx5_vdpa_virtq_kick_handler function may still be running and waiting
> on virtq->virtq_lock while mlx5_vdpa_cqe_event_unset function is trying to
> re-initialize the virtq->virtq_lock.
> 
> This causes mlx5_vdpa_virtq_kick_handler thread can't be wake up and can't
> be unregister. Following print may loop forever when calling
> rte_vhost_driver_unregister(socket_path):
> 
>      mlx5_vdpa: Try again to unregister fd 154 of virtq 11 interrupt
>      mlx5_vdpa: Try again to unregister fd 154 of virtq 11 interrupt
>      ...
> 
> The fix is to move mlx5_vdpa_virtq_unregister_intr_handle before
> mlx5_vdpa_cqe_event_unset.
> 
> Fixes: 057f7d2084 ("vdpa/mlx5: optimize datapath-control synchronization")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Yajun Wu <yajunw@nvidia.com>
> Acked-by: Matan Azrad <matan@nvidia.com>
> ---
>   drivers/vdpa/mlx5/mlx5_vdpa.c         | 1 +
>   drivers/vdpa/mlx5/mlx5_vdpa_cthread.c | 1 -
>   2 files changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
> index f1737f82a8..8b1de8bd62 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> @@ -282,6 +282,7 @@ _internal_mlx5_vdpa_dev_close(struct mlx5_vdpa_priv *priv,
>   	int ret = 0;
>   	int vid = priv->vid;
>   
> +	mlx5_vdpa_virtq_unreg_intr_handle_all(priv);
>   	mlx5_vdpa_cqe_event_unset(priv);
>   	if (priv->state == MLX5_VDPA_STATE_CONFIGURED) {
>   		ret |= mlx5_vdpa_lm_log(priv);
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c b/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c
> index 6e6624e5a3..1d84e422d4 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa_cthread.c
> @@ -190,7 +190,6 @@ mlx5_vdpa_c_thread_handle(void *arg)
>   			pthread_mutex_unlock(&virtq->virtq_lock);
>   			break;
>   		case MLX5_VDPA_TASK_DEV_CLOSE_NOWAIT:
> -			mlx5_vdpa_virtq_unreg_intr_handle_all(priv);
>   			pthread_mutex_lock(&priv->steer_update_lock);
>   			mlx5_vdpa_steer_unset(priv);
>   			pthread_mutex_unlock(&priv->steer_update_lock);

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH] vdpa/mlx5: fix unregister kick handler order

[Maxime Coquelin]

On 8/8/23 13:32, Yajun Wu wrote:
> The mlx5_vdpa_virtq_kick_handler function may still be running and waiting
> on virtq->virtq_lock while mlx5_vdpa_cqe_event_unset function is trying to
> re-initialize the virtq->virtq_lock.
> 
> This causes mlx5_vdpa_virtq_kick_handler thread can't be wake up and can't
> be unregister. Following print may loop forever when calling
> rte_vhost_driver_unregister(socket_path):
> 
>      mlx5_vdpa: Try again to unregister fd 154 of virtq 11 interrupt
>      mlx5_vdpa: Try again to unregister fd 154 of virtq 11 interrupt
>      ...
> 
> The fix is to move mlx5_vdpa_virtq_unregister_intr_handle before
> mlx5_vdpa_cqe_event_unset.
> 
> Fixes: 057f7d2084 ("vdpa/mlx5: optimize datapath-control synchronization")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Yajun Wu <yajunw@nvidia.com>
> Acked-by: Matan Azrad <matan@nvidia.com>
> ---
>   drivers/vdpa/mlx5/mlx5_vdpa.c         | 1 +
>   drivers/vdpa/mlx5/mlx5_vdpa_cthread.c | 1 -
>   2 files changed, 1 insertion(+), 1 deletion(-)


Applied to nex-virtio/for-next-net.

Thanks,
Maxime