From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0F68F43B8F; Mon, 4 Mar 2024 16:13:16 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 961A440695; Mon, 4 Mar 2024 16:13:15 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 31A0A4026B for ; Mon, 4 Mar 2024 16:13:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709565193; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rWFvdekP8fTqpiMX/eOKY/9OVF0i9bHDAtmkJzpruP0=; b=WUi0twmcihXmp6mGdKknmDQKgkA7aBrj9gzQhq3BeKC9rWNkm6WmDkGYcB/gBMRNAmem3a WmPewpfm+IHEDYVz3gjMuuj0eR4lF55tdnP79Svf9Ac6BegCVkL3RH05c3rb8OvMPd4fYE 9r9HYpNi5NZZK5jIldL846FGEJmTXEc= Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-647-xToIsVWlMg-VCV6mA6vg_g-1; Mon, 04 Mar 2024 10:13:12 -0500 X-MC-Unique: xToIsVWlMg-VCV6mA6vg_g-1 Received: by mail-lj1-f197.google.com with SMTP id 38308e7fff4ca-2d33e6f838dso31805511fa.0 for ; Mon, 04 Mar 2024 07:13:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709565190; x=1710169990; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=rWFvdekP8fTqpiMX/eOKY/9OVF0i9bHDAtmkJzpruP0=; b=AVaN1aN393BIOYaH5nGeuQQdgQ+GXtr6R0erhSr1vxls4GLykiRAE6A5ULrwHf5qCw M+XGSNPhwdsFqngEPQC4/vDb5fPDBnXF/GphVxwwqUGlHuoRB5Z6XY2gFixP2VP26O5Z i0KUv3tXFoEOLRy+YVogAZl+/4vIWJGfsFmY9vFdI/xdPtETHAymmlJ0j0bWKp3mLZx1 KrVGkz/4UjC7r2+tGJQdaHtuHsvc+It4XppzXixLHYSbgd6EUBiVApznT29Rwz5r18mg GeniESOVdOsOEr0NpqlEbj1GoNp69q4M+wErPX00Y28sG/t+SrwZG0bdBOQSsQ3jqDVB NSyg== X-Gm-Message-State: AOJu0YwhIuSCLz0h4XNUsqLlJKrHgCDpTEQnq0/Boah4mcm9R8tPtHE9 ijMciImAldQL80UIuzLgRmwwqMXVAtnDsmOhJK6nUj4yEGpQEfnSs8xKcBILJkuTkMJD8ymZ4Lg E0NpuupIPwTstraajbeBrM5znw1GiJEcNKUY1XNhMhZPPSmua6aRai5igdZx5mCf/oB7oz4fDcV NHs/pMaakPll88O60= X-Received: by 2002:a2e:9dd3:0:b0:2d2:42ce:3e5b with SMTP id x19-20020a2e9dd3000000b002d242ce3e5bmr6993527ljj.8.1709565190539; Mon, 04 Mar 2024 07:13:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IF9qIgBdbdmXv1GEdcpeyWBz4cDyOgVW2NXT5Ohh9opSkcqNYM25MN+HI7U5+929fErTI8rOgdGg0narIsey80= X-Received: by 2002:a2e:9dd3:0:b0:2d2:42ce:3e5b with SMTP id x19-20020a2e9dd3000000b002d242ce3e5bmr6993482ljj.8.1709565189607; Mon, 04 Mar 2024 07:13:09 -0800 (PST) MIME-Version: 1.0 References: <20240229122502.2572343-2-maxime.coquelin@redhat.com> <20240304103558.1500695-1-david.marchand@redhat.com> In-Reply-To: <20240304103558.1500695-1-david.marchand@redhat.com> From: Maxime Coquelin Date: Mon, 4 Mar 2024 16:12:57 +0100 Message-ID: Subject: Re: [PATCH v2] vhost: fix VDUSE device destruction failure To: David Marchand Cc: dev , Maxime Coquelin , stable@dpdk.org, Chenbo Xia X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/alternative; boundary="000000000000d8eeab0612d72900" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000d8eeab0612d72900 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Le lun. 4 mars 2024, 11:36, David Marchand a =C3=A9crit : > From: Maxime Coquelin > > VDUSE_DESTROY_DEVICE ioctl can fail because the device's > chardev is not released despite close syscall having been > called. It happens because the events handler thread is > still polling the file descriptor. > > fdset_pipe_notify() is not enough because it does not > ensure the notification has been handled by the event > thread, it just returns once the notification is sent. > > To fix this, this patch introduces a synchronization > mechanism based on pthread's condition, so that > fdset_pipe_notify_sync() only returns once the pipe's > read callback has been executed. > > Fixes: 51d018fdac4e ("vhost: add VDUSE events handler") > Cc: stable@dpdk.org > > Signed-off-by: Maxime Coquelin > Signed-off-by: David Marchand > --- > Changes since v1: > - sync'd only when in VDUSE destruction path, > - added explicit init of sync_mutex, > > --- > lib/vhost/fd_man.c | 23 +++++++++++++++++++++-- > lib/vhost/fd_man.h | 6 ++++++ > lib/vhost/socket.c | 1 + > lib/vhost/vduse.c | 3 ++- > 4 files changed, 30 insertions(+), 3 deletions(-) > Reviewed-by: Maxime Coquelin Thanks for improving the patch, Maxime > diff --git a/lib/vhost/fd_man.c b/lib/vhost/fd_man.c > index 79a8d2c006..481e6b900a 100644 > --- a/lib/vhost/fd_man.c > +++ b/lib/vhost/fd_man.c > @@ -309,10 +309,11 @@ fdset_event_dispatch(void *arg) > } > > static void > -fdset_pipe_read_cb(int readfd, void *dat __rte_unused, > +fdset_pipe_read_cb(int readfd, void *dat, > int *remove __rte_unused) > { > char charbuf[16]; > + struct fdset *fdset =3D dat; > int r =3D read(readfd, charbuf, sizeof(charbuf)); > /* > * Just an optimization, we don't care if read() failed > @@ -320,6 +321,11 @@ fdset_pipe_read_cb(int readfd, void *dat __rte_unuse= d, > * compiler happy > */ > RTE_SET_USED(r); > + > + pthread_mutex_lock(&fdset->sync_mutex); > + fdset->sync =3D true; > + pthread_cond_broadcast(&fdset->sync_cond); > + pthread_mutex_unlock(&fdset->sync_mutex); > } > > void > @@ -342,7 +348,7 @@ fdset_pipe_init(struct fdset *fdset) > } > > ret =3D fdset_add(fdset, fdset->u.readfd, > - fdset_pipe_read_cb, NULL, NULL); > + fdset_pipe_read_cb, NULL, fdset); > > if (ret < 0) { > VHOST_FDMAN_LOG(ERR, > @@ -366,5 +372,18 @@ fdset_pipe_notify(struct fdset *fdset) > * compiler happy > */ > RTE_SET_USED(r); > +} > + > +void > +fdset_pipe_notify_sync(struct fdset *fdset) > +{ > + pthread_mutex_lock(&fdset->sync_mutex); > + > + fdset->sync =3D false; > + fdset_pipe_notify(fdset); > + > + while (!fdset->sync) > + pthread_cond_wait(&fdset->sync_cond, &fdset->sync_mutex); > > + pthread_mutex_unlock(&fdset->sync_mutex); > } > diff --git a/lib/vhost/fd_man.h b/lib/vhost/fd_man.h > index 6315904c8e..7816fb11ac 100644 > --- a/lib/vhost/fd_man.h > +++ b/lib/vhost/fd_man.h > @@ -6,6 +6,7 @@ > #define _FD_MAN_H_ > #include > #include > +#include > > #define MAX_FDS 1024 > > @@ -35,6 +36,10 @@ struct fdset { > int writefd; > }; > } u; > + > + pthread_mutex_t sync_mutex; > + pthread_cond_t sync_cond; > + bool sync; > }; > > > @@ -53,5 +58,6 @@ int fdset_pipe_init(struct fdset *fdset); > void fdset_pipe_uninit(struct fdset *fdset); > > void fdset_pipe_notify(struct fdset *fdset); > +void fdset_pipe_notify_sync(struct fdset *fdset); > > #endif > diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c > index a2fdac30a4..96b3ab5595 100644 > --- a/lib/vhost/socket.c > +++ b/lib/vhost/socket.c > @@ -93,6 +93,7 @@ static struct vhost_user vhost_user =3D { > .fd =3D { [0 ... MAX_FDS - 1] =3D {-1, NULL, NULL, NULL, = 0} }, > .fd_mutex =3D PTHREAD_MUTEX_INITIALIZER, > .fd_pooling_mutex =3D PTHREAD_MUTEX_INITIALIZER, > + .sync_mutex =3D PTHREAD_MUTEX_INITIALIZER, > .num =3D 0 > }, > .vsocket_cnt =3D 0, > diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c > index d462428d2c..e0c6991b69 100644 > --- a/lib/vhost/vduse.c > +++ b/lib/vhost/vduse.c > @@ -36,6 +36,7 @@ static struct vduse vduse =3D { > .fd =3D { [0 ... MAX_FDS - 1] =3D {-1, NULL, NULL, NULL, = 0} }, > .fd_mutex =3D PTHREAD_MUTEX_INITIALIZER, > .fd_pooling_mutex =3D PTHREAD_MUTEX_INITIALIZER, > + .sync_mutex =3D PTHREAD_MUTEX_INITIALIZER, > .num =3D 0 > }, > }; > @@ -618,7 +619,7 @@ vduse_device_destroy(const char *path) > vduse_device_stop(dev); > > fdset_del(&vduse.fdset, dev->vduse_dev_fd); > - fdset_pipe_notify(&vduse.fdset); > + fdset_pipe_notify_sync(&vduse.fdset); > > if (dev->vduse_dev_fd >=3D 0) { > close(dev->vduse_dev_fd); > -- > 2.43.0 > > --000000000000d8eeab0612d72900 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


Le lun. 4 mars 2024, 11:36, David Marchand <= david.marchand@redhat.com> a =C3=A9crit=C2=A0:
From: Maxime Coquelin <maxime.coqu= elin@redhat.com>

VDUSE_DESTROY_DEVICE ioctl can fail because the device's
chardev is not released despite close syscall having been
called. It happens because the events handler thread is
still polling the file descriptor.

fdset_pipe_notify() is not enough because it does not
ensure the notification has been handled by the event
thread, it just returns once the notification is sent.

To fix this, this patch introduces a synchronization
mechanism based on pthread's condition, so that
fdset_pipe_notify_sync() only returns once the pipe's
read callback has been executed.

Fixes: 51d018fdac4e ("vhost: add VDUSE events handler")
Cc: stable@dpdk.org

Signed-off-by: Maxime Coquelin <maxime.coquelin@redha= t.com>
Signed-off-by: David Marchand <david.marchand@redhat.c= om>
---
Changes since v1:
- sync'd only when in VDUSE destruction path,
- added explicit init of sync_mutex,

---
=C2=A0lib/vhost/fd_man.c | 23 +++++++++++++++++++++--
=C2=A0lib/vhost/fd_man.h |=C2=A0 6 ++++++
=C2=A0lib/vhost/socket.c |=C2=A0 1 +
=C2=A0lib/vhost/vduse.c=C2=A0 |=C2=A0 3 ++-
=C2=A04 files changed, 30 insertions(+), 3 deletions(-)

Reviewed-by: Maxime = Coquelin <maxime.coquelin@redhat.com>

Thanks for improving the patch,
Maxime


diff --git a/lib/vhost/fd_man.c b/lib/vhost/fd_man.c
index 79a8d2c006..481e6b900a 100644
--- a/lib/vhost/fd_man.c
+++ b/lib/vhost/fd_man.c
@@ -309,10 +309,11 @@ fdset_event_dispatch(void *arg)
=C2=A0}

=C2=A0static void
-fdset_pipe_read_cb(int readfd, void *dat __rte_unused,
+fdset_pipe_read_cb(int readfd, void *dat,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0int *r= emove __rte_unused)
=C2=A0{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 char charbuf[16];
+=C2=A0 =C2=A0 =C2=A0 =C2=A0struct fdset *fdset =3D dat;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 int r =3D read(readfd, charbuf, sizeof(charbuf)= );
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Just an optimization, we don't care= if read() failed
@@ -320,6 +321,11 @@ fdset_pipe_read_cb(int readfd, void *dat __rte_unused,=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* compiler happy
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RTE_SET_USED(r);
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_mutex_lock(&fdset->sync_mutex);<= br> +=C2=A0 =C2=A0 =C2=A0 =C2=A0fdset->sync =3D true;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_cond_broadcast(&fdset->sync_cond= );
+=C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_mutex_unlock(&fdset->sync_mutex)= ;
=C2=A0}

=C2=A0void
@@ -342,7 +348,7 @@ fdset_pipe_init(struct fdset *fdset)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

=C2=A0 =C2=A0 =C2=A0 =C2=A0 ret =3D fdset_add(fdset, fdset->u.readfd, -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0fdset_pipe_read_cb, NULL, NULL);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0fdset_pipe_read_cb, NULL, fdset);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (ret < 0) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VHOST_FDMAN_LOG(ERR= ,
@@ -366,5 +372,18 @@ fdset_pipe_notify(struct fdset *fdset)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* compiler happy
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RTE_SET_USED(r);
+}
+
+void
+fdset_pipe_notify_sync(struct fdset *fdset)
+{
+=C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_mutex_lock(&fdset->sync_mutex);<= br> +
+=C2=A0 =C2=A0 =C2=A0 =C2=A0fdset->sync =3D false;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0fdset_pipe_notify(fdset);
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0while (!fdset->sync)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_cond_wait(&= amp;fdset->sync_cond, &fdset->sync_mutex);

+=C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_mutex_unlock(&fdset->sync_mutex)= ;
=C2=A0}
diff --git a/lib/vhost/fd_man.h b/lib/vhost/fd_man.h
index 6315904c8e..7816fb11ac 100644
--- a/lib/vhost/fd_man.h
+++ b/lib/vhost/fd_man.h
@@ -6,6 +6,7 @@
=C2=A0#define _FD_MAN_H_
=C2=A0#include <pthread.h>
=C2=A0#include <poll.h>
+#include <stdbool.h>

=C2=A0#define MAX_FDS 1024

@@ -35,6 +36,10 @@ struct fdset {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 int writefd;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 };
=C2=A0 =C2=A0 =C2=A0 =C2=A0 } u;
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_mutex_t sync_mutex;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0pthread_cond_t sync_cond;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0bool sync;
=C2=A0};


@@ -53,5 +58,6 @@ int fdset_pipe_init(struct fdset *fdset);
=C2=A0void fdset_pipe_uninit(struct fdset *fdset);

=C2=A0void fdset_pipe_notify(struct fdset *fdset);
+void fdset_pipe_notify_sync(struct fdset *fdset);

=C2=A0#endif
diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
index a2fdac30a4..96b3ab5595 100644
--- a/lib/vhost/socket.c
+++ b/lib/vhost/socket.c
@@ -93,6 +93,7 @@ static struct vhost_user vhost_user =3D {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .fd =3D { [0 ... MA= X_FDS - 1] =3D {-1, NULL, NULL, NULL, 0} },
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .fd_mutex =3D PTHRE= AD_MUTEX_INITIALIZER,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .fd_pooling_mutex = =3D PTHREAD_MUTEX_INITIALIZER,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.sync_mutex =3D PTH= READ_MUTEX_INITIALIZER,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .num =3D 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 },
=C2=A0 =C2=A0 =C2=A0 =C2=A0 .vsocket_cnt =3D 0,
diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c
index d462428d2c..e0c6991b69 100644
--- a/lib/vhost/vduse.c
+++ b/lib/vhost/vduse.c
@@ -36,6 +36,7 @@ static struct vduse vduse =3D {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .fd =3D { [0 ... MA= X_FDS - 1] =3D {-1, NULL, NULL, NULL, 0} },
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .fd_mutex =3D PTHRE= AD_MUTEX_INITIALIZER,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .fd_pooling_mutex = =3D PTHREAD_MUTEX_INITIALIZER,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.sync_mutex =3D PTH= READ_MUTEX_INITIALIZER,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .num =3D 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 },
=C2=A0};
@@ -618,7 +619,7 @@ vduse_device_destroy(const char *path)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vduse_device_stop(dev);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fdset_del(&vduse.fdset, dev->vduse_dev_f= d);
-=C2=A0 =C2=A0 =C2=A0 =C2=A0fdset_pipe_notify(&vduse.fdset);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0fdset_pipe_notify_sync(&vduse.fdset);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (dev->vduse_dev_fd >=3D 0) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 close(dev->vduse= _dev_fd);
--
2.43.0

--000000000000d8eeab0612d72900--