From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B3DFC438F3; Thu, 18 Jan 2024 15:46:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6727A40295; Thu, 18 Jan 2024 15:46:20 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id D98F140285 for ; Thu, 18 Jan 2024 15:46:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705589178; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g7XNAl6j8qHxRHDZ7hnVpdxaeGUGkjSDuAW6YRagbjk=; b=bf9FSRmDhNqicYm9f+nlNjLuYVWXpvg1tdOi7QIta/E882SLfsH/DOMyOzpfVTl7+Xsbp0 j/6YcXpek/CtYyl9qHMTGP8Oap0im8WsqeVTIo82arV/GDvtHu5OQiHvqN7jBv5KAluWeh kvwMoJ2a5ldkhL6OSjfnfUWe6uSz81k= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-360-tmgirpeNMYWLr6IfhwCf5w-1; Thu, 18 Jan 2024 09:46:16 -0500 X-MC-Unique: tmgirpeNMYWLr6IfhwCf5w-1 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-50efc35b708so462016e87.1 for ; Thu, 18 Jan 2024 06:46:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705589175; x=1706193975; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g7XNAl6j8qHxRHDZ7hnVpdxaeGUGkjSDuAW6YRagbjk=; b=NLJ/2PEsSfPfwjuxrBAZlDNc6kvpphrpX1KVQhS7V0gZKXZ3XkiIq2jH2hH2sLqOWF CkShec3nxIsM/KPPVXvQoU7MrB4vRM9ygybX0kJWaQyk2cP6r6CoRkdenZZKTy4cooP7 kuhML7+FJ1rz1ret1rbQ26VNrwbFaCvSb7uKEzUDb42Qvaf4PSYH7ZPd2pw22lt01GUx 8QqQvfdIXBs4XnbSlHxQmdQ3RhSMehgZKz5vSibEK/VMLxznQ96y4LcB2PZt3P+h7xTa iZEWSLBgrpAS+PjXWuy3v6CMkteJs/6y0yHYydr+sLdc5yWO1lJotaQK8KpkOE6ptyKU RuKg== X-Gm-Message-State: AOJu0YwVH4A0HUEpemvfWRu/jk/2Cr9yO2aHUgc6S8LG4GKjv3MQoCMM 5/7hPw1hTbvzGmyLZ45XAjBIYE6z8UhqTCKx7jajvHCtEqkC3wW8gWaUy8MletvYBA4QpIm0q7X FFuJSs2LFCCTid7NejX8Yl4MCODAcC4QD1PKoGjpKacIgI17vsNjT0G19Gt8ZJPZ2HZu2Va3IxT 10DcJVDS1O9YmP+P8= X-Received: by 2002:a05:6512:3135:b0:50e:7780:7028 with SMTP id p21-20020a056512313500b0050e77807028mr1801029lfd.27.1705589174900; Thu, 18 Jan 2024 06:46:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IGF2Wg5/q/UbAdvpMJEY67rnbeCFOaiBxwxxYOcKguNnN6yRuKkafD6PaEUbMtZXaf7rOYwEWJ0XViP1p+f5iE= X-Received: by 2002:a05:6512:3135:b0:50e:7780:7028 with SMTP id p21-20020a056512313500b0050e77807028mr1801026lfd.27.1705589174536; Thu, 18 Jan 2024 06:46:14 -0800 (PST) MIME-Version: 1.0 References: <20240118103344.50739-1-chenh@yusur.tech> In-Reply-To: <20240118103344.50739-1-chenh@yusur.tech> From: David Marchand Date: Thu, 18 Jan 2024 15:46:02 +0100 Message-ID: Subject: Re: [PATCH] vhost: fix deadlock during software live migration of VDPA in a nested virtualization environment To: Hao Chen Cc: dev@dpdk.org, zy@yusur.tech, huangml@yusur.tech, Maxime Coquelin , Chenbo Xia X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hello, On Thu, Jan 18, 2024 at 11:34=E2=80=AFAM Hao Chen wrote: > > In a nested virtualization environment, running dpdk vdpa in QEMU-L1 for > software live migration will result in a deadlock between dpdke-vdpa and > QEMU-L2 processes. > rte_vdpa_relay_vring_used-> > __vhost_iova_to_vva-> > vhost_user_iotlb_rd_unlock(vq)-> > vhost_user_iotlb_miss-> send vhost message VHOST_USER_SLAVE_IOTLB_MSG to > QEMU's vdpa socket, > then call vhost_user_iotlb_rd_lock(vq) to hold the read lock `iotlb_lock`= . > But there is no place to release this read lock. > > QEMU L2 get the VHOST_USER_SLAVE_IOTLB_MSG, > then call vhost_user_send_device_iotlb_msg to send VHOST_USER_IOTLB_MSG > messages to dpdk-vdpa. > Dpdk vdpa will call vhost_user_iotlb_msg-> > vhost_user_iotlb_cache_insert, here, will obtain the write lock > `iotlb_lock`, but the read lock `iotlb_lock` has not been released and > will block here. > > This patch add lock and unlock function to fix the deadlock. Please identify the commit that first had this issue and add a Fixes: tag. > > Signed-off-by: Hao Chen > --- > lib/vhost/vdpa.c | 8 +++++++- > 1 file changed, 7 insertions(+), 1 deletion(-) > > diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c > index 9776fc07a9..9132414209 100644 > --- a/lib/vhost/vdpa.c > +++ b/lib/vhost/vdpa.c > @@ -19,6 +19,7 @@ > #include "rte_vdpa.h" > #include "vdpa_driver.h" > #include "vhost.h" > +#include "iotlb.h" > > /** Double linked list of vDPA devices. */ > TAILQ_HEAD(vdpa_device_list, rte_vdpa_device); > @@ -193,10 +194,12 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, vo= id *vring_m) > if (unlikely(nr_descs > vq->size)) > return -1; > > + vhost_user_iotlb_rd_lock(vq); > desc_ring =3D (struct vring_desc *)(uintptr_t) > vhost_iova_to_vva(dev, vq, > vq->desc[desc_id].addr, &= dlen, > VHOST_ACCESS_RO); > + vhost_user_iotlb_rd_unlock(vq); > if (unlikely(!desc_ring)) > return -1; > > @@ -220,9 +223,12 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, voi= d *vring_m) > if (unlikely(nr_descs-- =3D=3D 0)) > goto fail; > desc =3D desc_ring[desc_id]; > - if (desc.flags & VRING_DESC_F_WRITE) > + if (desc.flags & VRING_DESC_F_WRITE) { > + vhost_user_iotlb_rd_lock(vq); > vhost_log_write_iova(dev, vq, desc.addr, > desc.len); > + vhost_user_iotlb_rd_unlock(vq); > + } > desc_id =3D desc.next; > } while (desc.flags & VRING_DESC_F_NEXT); > Interesting, I suspected a bug in this area as clang was complaining. Please try to remove the __rte_no_thread_safety_analysis annotation and compile with clang. https://git.dpdk.org/dpdk/tree/lib/vhost/vdpa.c#n150 You will get: ccache clang -Ilib/librte_vhost.a.p -Ilib -I../lib -Ilib/vhost -I../lib/vhost -I. -I.. -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry -I../lib/telemetry -Ilib/ethdev -I../lib/ethdev -Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring -Ilib/meter -I../lib/meter -Ilib/cryptodev -I../lib/cryptodev -Ilib/rcu -I../lib/rcu -Ilib/hash -I../lib/hash -Ilib/pci -I../lib/pci -Ilib/dmadev -I../lib/dmadev -fcolor-diagnostics -fsanitize=3Daddress -fno-omit-frame-pointer -D_FILE_OFFSET_BITS=3D64 -Wall -Winvalid-pch -Wextra -Werror -std=3Dc11 -O0 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=3Dnative -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -DVHOST_CLANG_UNROLL_PRAGMA -fno-strict-aliasing -DVHOST_HAS_VDUSE -DRTE_LOG_DEFAULT_LOGTYPE=3Dlib.vhost -DRTE_ANNOTATE_LOCKS -Wthread-safety -MD -MQ lib/librte_vhost.a.p/vhost_vdpa.c.o -MF lib/librte_vhost.a.p/vhost_vdpa.c.o.d -o lib/librte_vhost.a.p/vhost_vdpa.c.o -c ../lib/vhost/vdpa.c ../lib/vhost/vdpa.c:196:5: error: calling function 'vhost_iova_to_vva' requires holding mutex 'vq->iotlb_lock' [-Werror,-Wthread-safety-analysis] vhost_iova_to_vva(dev, vq, ^ ../lib/vhost/vdpa.c:203:13: error: calling function 'vhost_alloc_copy_ind_table' requires holding mutex 'vq->iotlb_lock' [-Werror,-Wthread-safety-analysis] idesc =3D vhost_alloc_copy_ind_table(dev, v= q, ^ ../lib/vhost/vdpa.c:223:5: error: calling function 'vhost_log_write_iova' requires holding mutex 'vq->iotlb_lock' [-Werror,-Wthread-safety-analysis] vhost_log_write_iova(dev, vq, desc.addr, ^ 3 errors generated. We may need to protect the vhost_alloc_copy_ind_table() call too. What do you think? --=20 David Marchand