From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2AF2E43829 for ; Tue, 5 Mar 2024 16:35:39 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 254D142D2E; Tue, 5 Mar 2024 16:35:39 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id DA84C42D2E for ; Tue, 5 Mar 2024 16:35:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709652937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fWAxZHje52d7fgCeU/wniHH1PdA1yzq/dRrUvODljdI=; b=etj/S7Ag/j1eDtqN4bxcoQqba7lN3gHVDlLtCfdr8VnudSSA10K+XAdD9p9zmskgiQdwv3 Jc/a3sxRNN9EWR9f/k4US3zZLSO5LdPW4kal23+VXnG2OIW3iGqzQFam1zuHHvj0lIi2Cd Dwe3Vdcoaxc6I7AWcsT4htflGd3Jal0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-231-3Qs32PJRPSOsspm5iSUf-Q-1; Tue, 05 Mar 2024 10:35:36 -0500 X-MC-Unique: 3Qs32PJRPSOsspm5iSUf-Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B881A3806735; Tue, 5 Mar 2024 15:35:35 +0000 (UTC) Received: from rh.redhat.com (unknown [10.39.194.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8D550200AE7F; Tue, 5 Mar 2024 15:35:34 +0000 (UTC) From: Kevin Traynor To: Hao Chen Cc: Maxime Coquelin , dpdk stable Subject: patch 'vhost: fix deadlock during vDPA SW live migration' has been queued to stable release 21.11.7 Date: Tue, 5 Mar 2024 15:34:02 +0000 Message-ID: <20240305153449.263666-29-ktraynor@redhat.com> In-Reply-To: <20240305153449.263666-1-ktraynor@redhat.com> References: <20240305153449.263666-1-ktraynor@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 21.11.7 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 03/11/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/kevintraynor/dpdk-stable This queued commit can be viewed at: https://github.com/kevintraynor/dpdk-stable/commit/15afe0935f9e0e6e216bfee114c0e17855899bea Thanks. Kevin --- >From 15afe0935f9e0e6e216bfee114c0e17855899bea Mon Sep 17 00:00:00 2001 From: Hao Chen Date: Mon, 22 Jan 2024 11:27:44 +0800 Subject: [PATCH] vhost: fix deadlock during vDPA SW live migration [ upstream commit 19639c3b693bcfdc941c56d9d5bd60f65a8eeecb ] In a nested virtualization environment, running dpdk-vdpa in QEMU-L1 for software live migration will result in a deadlock between dpdk-vdpa and QEMU-L2 processes. 'rte_vdpa_relay_vring_used'-> '__vhost_iova_to_vva'-> 'vhost_user_iotlb_rd_unlock(vq)'-> 'vhost_user_iotlb_miss'-> send vhost message 'VHOST_USER_SLAVE_IOTLB_MSG' to QEMU-L2's vdpa socket, then call 'vhost_user_iotlb_rd_lock(vq)' to hold the read lock `iotlb_lock`. But there is no place to release this read lock. QEMU-L2 get the 'VHOST_USER_SLAVE_IOTLB_MSG', then call 'vhost_user_send_device_iotlb_msg' to send 'VHOST_USER_IOTLB_MSG' messages to dpdk-vdpa. dpdk-vdpa will call vhost_user_iotlb_cache_insert and will obtain the write lock `iotlb_lock`, but the read lock `iotlb_lock` has not been released and will block here. This patch add lock and unlock function to fix the deadlock. Fixes: b13ad2decc83 ("vhost: provide helpers for virtio ring relay") Signed-off-by: Hao Chen Reviewed-by: Maxime Coquelin --- lib/vhost/vdpa.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c index 09ad5d866e..bd00c9d2c2 100644 --- a/lib/vhost/vdpa.c +++ b/lib/vhost/vdpa.c @@ -20,4 +20,5 @@ #include "vdpa_driver.h" #include "vhost.h" +#include "iotlb.h" /** Double linked list of vDPA devices. */ @@ -177,15 +178,19 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) return -1; + vhost_user_iotlb_rd_lock(vq); desc_ring = (struct vring_desc *)(uintptr_t) vhost_iova_to_vva(dev, vq, vq->desc[desc_id].addr, &dlen, VHOST_ACCESS_RO); + vhost_user_iotlb_rd_unlock(vq); if (unlikely(!desc_ring)) return -1; if (unlikely(dlen < vq->desc[desc_id].len)) { + vhost_user_iotlb_rd_lock(vq); idesc = vhost_alloc_copy_ind_table(dev, vq, vq->desc[desc_id].addr, vq->desc[desc_id].len); + vhost_user_iotlb_rd_unlock(vq); if (unlikely(!idesc)) return -1; @@ -204,7 +209,10 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) goto fail; desc = desc_ring[desc_id]; - if (desc.flags & VRING_DESC_F_WRITE) + if (desc.flags & VRING_DESC_F_WRITE) { + vhost_user_iotlb_rd_lock(vq); vhost_log_write_iova(dev, vq, desc.addr, desc.len); + vhost_user_iotlb_rd_unlock(vq); + } desc_id = desc.next; } while (desc.flags & VRING_DESC_F_NEXT); -- 2.43.2 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-03-05 14:08:55.383056216 +0000 +++ 0029-vhost-fix-deadlock-during-vDPA-SW-live-migration.patch 2024-03-05 14:08:54.644520749 +0000 @@ -1 +1 @@ -From 19639c3b693bcfdc941c56d9d5bd60f65a8eeecb Mon Sep 17 00:00:00 2001 +From 15afe0935f9e0e6e216bfee114c0e17855899bea Mon Sep 17 00:00:00 2001 @@ -5,0 +6,2 @@ +[ upstream commit 19639c3b693bcfdc941c56d9d5bd60f65a8eeecb ] + @@ -30 +31,0 @@ -Cc: stable@dpdk.org @@ -35,2 +36,2 @@ - lib/vhost/vdpa.c | 11 +++++++++-- - 1 file changed, 9 insertions(+), 2 deletions(-) + lib/vhost/vdpa.c | 10 +++++++++- + 1 file changed, 9 insertions(+), 1 deletion(-) @@ -39 +40 @@ -index 9776fc07a9..a1dd5a753b 100644 +index 09ad5d866e..bd00c9d2c2 100644 @@ -48,7 +49 @@ -@@ -148,5 +149,4 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev) - int - rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) -- __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */ - { - struct virtio_net *dev = get_device(vid); -@@ -194,15 +194,19 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) +@@ -177,15 +178,19 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) @@ -74 +69 @@ -@@ -221,7 +225,10 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m) +@@ -204,7 +209,10 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)