From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id E7302271; Mon, 27 Nov 2017 09:16:21 +0100 (CET) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 362DB85545; Mon, 27 Nov 2017 08:16:21 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2B03960603; Mon, 27 Nov 2017 08:16:21 +0000 (UTC) Received: from zmail17.collab.prod.int.phx2.redhat.com (zmail17.collab.prod.int.phx2.redhat.com [10.5.83.19]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 19C7E1800BD2; Mon, 27 Nov 2017 08:16:21 +0000 (UTC) Date: Mon, 27 Nov 2017 03:16:20 -0500 (EST) From: Victor Kaplansky To: Maxime Coquelin Cc: dev@dpdk.org, yliu@fridaylinux.org, tiwei bie , jianfeng tan , stable@dpdk.org, jfreiman@redhat.com Message-ID: <1760091245.45753170.1511770580954.JavaMail.zimbra@redhat.com> In-Reply-To: <20171124180826.18439-3-maxime.coquelin@redhat.com> References: <20171124180826.18439-1-maxime.coquelin@redhat.com> <20171124180826.18439-3-maxime.coquelin@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.35.206.59, 10.4.195.16] Thread-Topic: vhost: protect dirty logging against logging base change Thread-Index: xsU56I1omjixvIo5BQKhY/hVNsPEDg== X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 27 Nov 2017 08:16:21 +0000 (UTC) Subject: Re: [dpdk-dev] [PATCH v2 2/3] vhost: protect dirty logging against logging base change X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 08:16:22 -0000 Hi, While I agree that taking full fledged lock by rte_rwlock_read_lock() solves the race condition, I'm afraid that it would be too expensive in case when logging is off, since it introduces acquiring and releasing lock into the main flow of ring updates. It is OK for now, as it fixes the bug, but we need to perform more careful performance measurements, and see whether the performance degradation is not too prohibitive. As alternative, we may consider using more light weighted busy looping. Also, lets fix by this series the __sync_fetch_and_or_8 -> __sync_fetch_and_or, as it may improve the performance slightly. -- Victor ----- Original Message ----- > From: "Maxime Coquelin" > To: dev@dpdk.org, yliu@fridaylinux.org, "tiwei bie" , "jianfeng tan" , > vkaplans@redhat.com > Cc: stable@dpdk.org, jfreiman@redhat.com, "Maxime Coquelin" > Sent: Friday, November 24, 2017 8:08:25 PM > Subject: [PATCH v2 2/3] vhost: protect dirty logging against logging base change > > When performing live-migration with multiple queue pairs, > VHOST_USER_SET_LOG_BASE request is sent multiple times. > > If packets are being processed by the PMD threads, it is > possible that they are setting bits in the dirty log map while > its region is being unmapped by the vhost-user protocol thread. > It results in the following crash: > Thread 3 "lcore-slave-2" received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7f71ca495700 (LWP 32451)] > 0x00000000004bfc8a in vhost_set_bit (addr=0x7f71cbe18432 access memory at address 0x7f71cbe18432>, nr=1) at > /home/max/projects/src/mainline/dpdk/lib/librte_vhost/vhost.h:267 > 267 __sync_fetch_and_or_8(addr, (1U << nr)); > > We can see the vhost-user protocol thread just did the unmap of the > dirty log region when it happens. > > This patch prevents this by introducing a RW lock to protect > the log base. > > Fixes: 54f9e32305d4 ("vhost: handle dirty pages logging request") > Cc: stable@dpdk.org > > Signed-off-by: Maxime Coquelin > --- > lib/librte_vhost/vhost.c | 2 ++ > lib/librte_vhost/vhost.h | 14 +++++++++++--- > lib/librte_vhost/vhost_user.c | 4 ++++ > 3 files changed, 17 insertions(+), 3 deletions(-) > > diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c > index 4f8b73a09..5a7699da0 100644 > --- a/lib/librte_vhost/vhost.c > +++ b/lib/librte_vhost/vhost.c > @@ -311,6 +311,8 @@ vhost_new_device(void) > return -1; > } > > + rte_rwlock_init(&dev->log_lock); > + > vhost_devices[i] = dev; > dev->vid = i; > dev->slave_req_fd = -1; > diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h > index 1cc81c17c..2f36a034e 100644 > --- a/lib/librte_vhost/vhost.h > +++ b/lib/librte_vhost/vhost.h > @@ -243,6 +243,7 @@ struct virtio_net { > uint64_t log_size; > uint64_t log_base; > uint64_t log_addr; > + rte_rwlock_t log_lock; > struct ether_addr mac; > uint16_t mtu; > > @@ -278,12 +279,16 @@ vhost_log_write(struct virtio_net *dev, uint64_t addr, > uint64_t len) > { > uint64_t page; > > + > if (likely(((dev->features & (1ULL << VHOST_F_LOG_ALL)) == 0) || > - !dev->log_base || !len)) > + !len)) > return; > > - if (unlikely(dev->log_size <= ((addr + len - 1) / VHOST_LOG_PAGE / 8))) > - return; > + rte_rwlock_read_lock(&dev->log_lock); > + > + if (unlikely((!dev->log_base) || > + (dev->log_size <= ((addr + len - 1) / VHOST_LOG_PAGE / 8)))) > + goto unlock; > > /* To make sure guest memory updates are committed before logging */ > rte_smp_wmb(); > @@ -293,6 +298,9 @@ vhost_log_write(struct virtio_net *dev, uint64_t addr, > uint64_t len) > vhost_log_page((uint8_t *)(uintptr_t)dev->log_base, page); > page += 1; > } > + > +unlock: > + rte_rwlock_read_unlock(&dev->log_lock); > } > > static __rte_always_inline void > diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c > index f06d9bb65..4b03dbbca 100644 > --- a/lib/librte_vhost/vhost_user.c > +++ b/lib/librte_vhost/vhost_user.c > @@ -929,6 +929,8 @@ vhost_user_set_log_base(struct virtio_net *dev, struct > VhostUserMsg *msg) > goto out; > } > > + rte_rwlock_write_lock(&dev->log_lock); > + > /* > * Free previously mapped log memory on occasionally > * multiple VHOST_USER_SET_LOG_BASE. > @@ -940,6 +942,8 @@ vhost_user_set_log_base(struct virtio_net *dev, struct > VhostUserMsg *msg) > dev->log_base = dev->log_addr + off; > dev->log_size = size; > > + rte_rwlock_write_unlock(&dev->log_lock); > + > out: > close(fd); > > -- > 2.14.3 > >