From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 377DE438F1; Thu, 18 Jan 2024 11:33:52 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2854140E2D; Thu, 18 Jan 2024 11:33:52 +0100 (CET) Received: from out28-101.mail.aliyun.com (out28-101.mail.aliyun.com [115.124.28.101]) by mails.dpdk.org (Postfix) with ESMTP id 1C53540DCB for ; Thu, 18 Jan 2024 11:33:48 +0100 (CET) X-Alimail-AntiSpam: AC=CONTINUE; BC=0.1933177|-1; CH=green; DM=|CONTINUE|false|; DS=CONTINUE|ham_alarm|0.0140521-0.000425626-0.985522; FP=0|0|0|0|0|-1|-1|-1; HT=ay29a033018047188; MF=chenh@yusur.tech; NM=1; PH=DS; RN=5; RT=5; SR=0; TI=SMTPD_---.W9JvmSa_1705574025; Received: from localhost.localdomain(mailfrom:chenh@yusur.tech fp:SMTPD_---.W9JvmSa_1705574025) by smtp.aliyun-inc.com; Thu, 18 Jan 2024 18:33:46 +0800 From: Hao Chen To: dev@dpdk.org Cc: zy@yusur.tech, huangml@yusur.tech, Maxime Coquelin , Chenbo Xia Subject: [PATCH] vhost: fix deadlock during software live migration of VDPA in a nested virtualization environment Date: Thu, 18 Jan 2024 18:33:44 +0800 Message-Id: <20240118103344.50739-1-chenh@yusur.tech> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In a nested virtualization environment, running dpdk vdpa in QEMU-L1 for software live migration will result in a deadlock between dpdke-vdpa and QEMU-L2 processes. rte_vdpa_relay_vring_used-> __vhost_iova_to_vva-> vhost_user_iotlb_rd_unlock(vq)-> vhost_user_iotlb_miss-> send vhost message VHOST_USER_SLAVE_IOTLB_MSG to QEMU's vdpa socket, then call vhost_user_iotlb_rd_lock(vq) to hold the read lock `iotlb_lock`. But there is no place to release this read lock. QEMU L2 get the VHOST_USER_SLAVE_IOTLB_MSG, then call vhost_user_send_device_iotlb_msg to send VHOST_USER_IOTLB_MSG messages to dpdk-vdpa. Dpdk vdpa will call vhost_user_iotlb_msg-> vhost_user_iotlb_cache_insert, here, will obtain the write lock `iotlb_lock`, but the read lock `iotlb_lock` has not been released and will block here. This patch add lock and unlock function to fix the deadlock. Signed-off-by: Hao Chen --- lib/vhost/vdpa.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c index 9776fc07a9..9132414209 100644 --- a/lib/vhost/vdpa.c +++ b/lib/vhost/vdpa.c @@ -19,6 +19,7 @@ #include "rte_vdpa.h" #include "vdpa_driver.h" #include "vhost.h" +#include "iotlb.h" /** Double linked list of vDPA devices. */ TAILQ_HEAD(vdpa_device_list, rte_vdpa_device); @@ -193,10 +194,12 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) if (unlikely(nr_descs > vq->size)) return -1; + vhost_user_iotlb_rd_lock(vq); desc_ring = (struct vring_desc *)(uintptr_t) vhost_iova_to_vva(dev, vq, vq->desc[desc_id].addr, &dlen, VHOST_ACCESS_RO); + vhost_user_iotlb_rd_unlock(vq); if (unlikely(!desc_ring)) return -1; @@ -220,9 +223,12 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) if (unlikely(nr_descs-- == 0)) goto fail; desc = desc_ring[desc_id]; - if (desc.flags & VRING_DESC_F_WRITE) + if (desc.flags & VRING_DESC_F_WRITE) { + vhost_user_iotlb_rd_lock(vq); vhost_log_write_iova(dev, vq, desc.addr, desc.len); + vhost_user_iotlb_rd_unlock(vq); + } desc_id = desc.next; } while (desc.flags & VRING_DESC_F_NEXT); -- 2.27.0