From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B2D1943922; Mon, 22 Jan 2024 04:27:54 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2999F406A2; Mon, 22 Jan 2024 04:27:53 +0100 (CET) Received: from out28-121.mail.aliyun.com (out28-121.mail.aliyun.com [115.124.28.121]) by mails.dpdk.org (Postfix) with ESMTP id 9691D402EC; Mon, 22 Jan 2024 04:27:51 +0100 (CET) X-Alimail-AntiSpam: AC=CONTINUE; BC=0.1080061|-1; CH=green; DM=|CONTINUE|false|; DS=CONTINUE|ham_alarm|0.00994638-0.000401771-0.989652; FP=0|0|0|0|0|-1|-1|-1; HT=ay29a033018047194; MF=chenh@yusur.tech; NM=1; PH=DS; RN=8; RT=8; SR=0; TI=SMTPD_---.WBqr1pP_1705894067; Received: from localhost.localdomain(mailfrom:chenh@yusur.tech fp:SMTPD_---.WBqr1pP_1705894067) by smtp.aliyun-inc.com; Mon, 22 Jan 2024 11:27:48 +0800 From: Hao Chen To: dev@dpdk.org Cc: david.marchand@redhat.com, zy@yusur.tech, huangml@yusur.tech, stable@dpdk.org, Maxime Coquelin , Chenbo Xia , Xiao Wang Subject: [PATCH v2] vhost: fix deadlock during software live migration of VDPA in a nested virtualization environment Date: Mon, 22 Jan 2024 11:27:44 +0800 Message-Id: <20240122032746.2749586-1-chenh@yusur.tech> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240118103344.50739-1-chenh@yusur.tech> References: <20240118103344.50739-1-chenh@yusur.tech> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In a nested virtualization environment, running dpdk-vdpa in QEMU-L1 for software live migration will result in a deadlock between dpdke-vdpa and QEMU-L2 processes. 'rte_vdpa_relay_vring_used'-> '__vhost_iova_to_vva'-> 'vhost_user_iotlb_rd_unlock(vq)'-> 'vhost_user_iotlb_miss'-> send vhost message 'VHOST_USER_SLAVE_IOTLB_MSG' to QEMU-L2's vdpa socket, then call 'vhost_user_iotlb_rd_lock(vq)' to hold the read lock `iotlb_lock`. But there is no place to release this read lock. QEMU-L2 get the 'VHOST_USER_SLAVE_IOTLB_MSG', then call 'vhost_user_send_device_iotlb_msg' to send 'VHOST_USER_IOTLB_MSG' messages to dpdk-vdpa. Dpdk-vdpa will call vhost_user_iotlb_msg-> vhost_user_iotlb_cache_insert, here, will obtain the write lock `iotlb_lock`, but the read lock `iotlb_lock` has not been released and will block here. This patch add lock and unlock function to fix the deadlock. Fixes: b13ad2decc83 ("vhost: provide helpers for virtio ring relay") Cc: stable@dpdk.org Signed-off-by: Hao Chen --- Changes v1 ... v2: - protect the vhost_alloc_copy_ind_table() call too. lib/vhost/vdpa.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c index 9776fc07a9..a1dd5a753b 100644 --- a/lib/vhost/vdpa.c +++ b/lib/vhost/vdpa.c @@ -19,6 +19,7 @@ #include "rte_vdpa.h" #include "vdpa_driver.h" #include "vhost.h" +#include "iotlb.h" /** Double linked list of vDPA devices. */ TAILQ_HEAD(vdpa_device_list, rte_vdpa_device); @@ -147,7 +148,6 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev) int rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) - __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */ { struct virtio_net *dev = get_device(vid); uint16_t idx, idx_m, desc_id; @@ -193,17 +193,21 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) if (unlikely(nr_descs > vq->size)) return -1; + vhost_user_iotlb_rd_lock(vq); desc_ring = (struct vring_desc *)(uintptr_t) vhost_iova_to_vva(dev, vq, vq->desc[desc_id].addr, &dlen, VHOST_ACCESS_RO); + vhost_user_iotlb_rd_unlock(vq); if (unlikely(!desc_ring)) return -1; if (unlikely(dlen < vq->desc[desc_id].len)) { + vhost_user_iotlb_rd_lock(vq); idesc = vhost_alloc_copy_ind_table(dev, vq, vq->desc[desc_id].addr, vq->desc[desc_id].len); + vhost_user_iotlb_rd_unlock(vq); if (unlikely(!idesc)) return -1; @@ -220,9 +224,12 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) if (unlikely(nr_descs-- == 0)) goto fail; desc = desc_ring[desc_id]; - if (desc.flags & VRING_DESC_F_WRITE) + if (desc.flags & VRING_DESC_F_WRITE) { + vhost_user_iotlb_rd_lock(vq); vhost_log_write_iova(dev, vq, desc.addr, desc.len); + vhost_user_iotlb_rd_unlock(vq); + } desc_id = desc.next; } while (desc.flags & VRING_DESC_F_NEXT); -- 2.27.0