From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 96A1E45A0C; Mon, 23 Sep 2024 09:13:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 788BB402B9; Mon, 23 Sep 2024 09:13:17 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2086.outbound.protection.outlook.com [40.107.223.86]) by mails.dpdk.org (Postfix) with ESMTP id D7D8D402B4 for ; Mon, 23 Sep 2024 09:13:15 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hnVTUOgfbHfoMydrj+QLsDZfHdkSzQdYcVRs4YYTcwpn6CA35OcYTdGSgInfl9LTxNqs5O/b+jClzf7qgFx/R9BmltR6FkT6b6ovmQ/Gur5XRVfe6shoWJn+LwCg6AndjLMPTeDTMg4wRurrxDbCCIBY8PQJ38QcomiWjPpUczkMglz5qnPnh+NPvRbUyO7J4T1Pkc77Za8q7GMTTjBzjcIBA9SdpEoG6TSYcPMuVL8XBhB82eJrcK+VKswPcdI+ADhYgT3+VS/8MKmzjkJTIoAzvhyWI45CdF52J4Ac1WgnxmxO6SJwASU16WIwtU6wBgN1RcSCWTLwoB57pzO5Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=h2hUebXuQlmqGm9TmVAOfPRU6BKeJ2r3xOnI1Y3wiNQ=; b=J9/SQIV4Q/o3HxcZJXW5OArTkz8BocIMXndZq6XUwNcZ2vTQO/d5DvHIs5tn9llZUXpk6Gcnze4R5x7eG/kC5ARJUMD6cXD/ZzFMqBedepI48pBP2nePShA9m2j8SsSkZ32vdIJ8ML/2AvQxHCzqt0MiNu3FmIYYowuTf3clAKf13UnvC9REpmRgeriWphRY+1CbRVa6JIZCc2+FFSTx+2obOJ/Hq1LmAfML4IT+13lSm3KN3bXRWuOyDscSNZsZnJudtCQ7RjOGcFA+/d2KNiC7PUi3dPcXsvSym92rSforXl+ggRDekeB+sPrtkvYZAhRvFW2O+XZodzXmUCvbcw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h2hUebXuQlmqGm9TmVAOfPRU6BKeJ2r3xOnI1Y3wiNQ=; b=ZQcd1Ed9MJ2aZvkQWxfBVP9rkHfvv/ymujSU2hq3/ZNF1fA4C5YBtKjq1riYdOpC4DS2MrvP5+LtvxU7SR167/JueRdKifXTDIBSWGl2k7PlRHfV6FaVBKMuyleJA+iDe5F0kGE3+s9kXw+ETvKcQKIHeijPdZzpPSh1IlXlCCMfZx2LLHTy7UK68g1DDT3Y7NAnP2B3beIEHO44YYNJ3T0Gx889qE+f4ABX1HXCEr5fZnW8PPLlES45G9/QYvPGodCi5m3EOZMgfGXnnt7z1XuZaocWV3cCWN6EP2Tus2FiJaJcAtN9p8Br0aN3gjkV/JVHA+0DAm0ETnBKEci1Fg== Received: from IA0PR12MB8983.namprd12.prod.outlook.com (2603:10b6:208:490::17) by CY8PR12MB7754.namprd12.prod.outlook.com (2603:10b6:930:86::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Mon, 23 Sep 2024 07:13:13 +0000 Received: from IA0PR12MB8983.namprd12.prod.outlook.com ([fe80::68e7:8370:b169:fa0e]) by IA0PR12MB8983.namprd12.prod.outlook.com ([fe80::68e7:8370:b169:fa0e%5]) with mapi id 15.20.7982.022; Mon, 23 Sep 2024 07:13:06 +0000 From: Chenbo Xia To: Maxime Coquelin CC: "dev@dpdk.org" , David Marchand Subject: Re: [PATCH v2 2/2] vhost: add reconnection support to VDUSE Thread-Topic: [PATCH v2 2/2] vhost: add reconnection support to VDUSE Thread-Index: AQHbC28b+mbpoMQ3IUSNbCr3jiEyrrJk+S6A Date: Mon, 23 Sep 2024 07:13:06 +0000 Message-ID: <8DF87CB1-078A-4961-800D-64A617923A3A@nvidia.com> References: <20240920150921.994434-1-maxime.coquelin@redhat.com> <20240920150921.994434-3-maxime.coquelin@redhat.com> In-Reply-To: <20240920150921.994434-3-maxime.coquelin@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA0PR12MB8983:EE_|CY8PR12MB7754:EE_ x-ms-office365-filtering-correlation-id: 51cffd95-018f-4b59-ec6b-08dcdb9f2c34 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?TsK1sGe/dLE0W9NsUgrG2y1Q4B5G7zuOeTiFINX5FOQZgfeQJUlAUB2LKt3J?= =?us-ascii?Q?lS6ubScOmto7BpAaOabEVuF4JS0mmLMgYEKiCczIyDh0qIUtmBnme4qshQxA?= =?us-ascii?Q?MSa1LPl/1l+ByIyqTMC5MG4b86OAybM0fgjN9NSx2XQKBGiifahZzqbTBXBO?= =?us-ascii?Q?r7WbBG3wdE+OiUr5lM/hhMRtDkewiF/iI+13tHQFrUej5SDY7R6u5Kt55BGk?= =?us-ascii?Q?tO1uYre0ri4o8Qp/bNc4Vphq6uocEnLSguoBITxryEqsT8ZnV8OKYd7SwXV0?= =?us-ascii?Q?+dfC9It71hxguBb4j/ntIzCDLZiY1FkAnGZN8v/Q96LukUzj4LtJpCJdbtvc?= =?us-ascii?Q?wyqWkIJrIY9SewnhYi9M7DHvX9G+J2bq6Ic6huZvJUnDNhIpLbVOeWRI00Bg?= =?us-ascii?Q?DXdY4BkZ7qNJrFWg95lmuBrMFnWZv9mJbNyz3kaiqsXoboYyZ7PllXFsjCGz?= =?us-ascii?Q?VQ6tYV1ilTGQL3ulGoYaNqNRlYW1cbBwEitPQgu/nuRuPNWo1E7fbLBsY5lg?= =?us-ascii?Q?rLkAUfsB07RrBJjaGzex7S/9zpYhd+yNWE2IlVZKCTig8MuerMYc/y+Zs5oW?= =?us-ascii?Q?VjeYvKn78zD/vxcLXcaUQYr/NNxvsZDCPuhyaFe25SmH2OXzF+bBRIdM4saH?= =?us-ascii?Q?vFxdUaqZ9YUtWMML5yixDm0iKftj/Z8meqU0J1wCFzS2v7M6BucSASL0qttK?= =?us-ascii?Q?uHwHkzrWGHvWgF7xk1G/QngiCgeXCSUIwzF7RmIHGb4Nfow1/z1prhajCkuQ?= =?us-ascii?Q?wpYpB+gWBTVYrAonRE51Pxq7gp6Hv2gIAKkmmqD9nGZY9PXs1aTWgZRAu4mr?= =?us-ascii?Q?0krQ/Z932DEhGV/XssqbCbbynsK8A26Flzbzt23gWYwC3W7WpRIct5diADV6?= =?us-ascii?Q?wtswvgBHldeaEK8cL7clpX5qiDC8syDug19rCoZslGrCpTaH5SfHHHaL0m9J?= =?us-ascii?Q?Y3GMVs8eFWp2VQrPq4oUuyzHMVmAz7MgnIdiku8kO5erlgo5QSFPLuSc95l+?= =?us-ascii?Q?Xot38xFzYNmRcd8eiKMaBgUQYuE0Iy24qrN0HU05sHYroJW0cYw/wqRek9VG?= =?us-ascii?Q?Zbac/6n/pfpnMf75X7HIVutfPJA9XX8mOibG7sXsR26AxJMat1DPvydHRCO0?= =?us-ascii?Q?WlVh1QyihmuNWj2FZqpb6cZuGUp+SBJIrmREwUBGSPDUdDjG8vv0gXiPdgsb?= =?us-ascii?Q?9QPtEk5WkzvjbzmxOOahtBcW3N+hIWoOP5EMwKbyCsMa4RD406kO9GRN/ElF?= =?us-ascii?Q?ZMQdsVG2vMemlvkSVvRw8Q9EXVGgT8qgNeqXvaNsvG6pNopIU3uX6py6AtEv?= =?us-ascii?Q?VooQp8e/2zn6gTC8RAgxFMrXQ0badV5ukbc++LhiKeArWw=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA0PR12MB8983.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(38070700018); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LqvRWgWVzbPcsG/GdmYxDsb6+ttbrWTUbMyb3tHgZm+GIyxlzmNFv08Lw/M2?= =?us-ascii?Q?BQhj0M+Hje59XYTx/IY+CbXC1DM1L6AG7qrm2E+XayT0YVZOdB4hA5xq8pJd?= =?us-ascii?Q?OaviHCQ8MPFpSqBOpKq+mCfPUevq+J0G8R+9TcEqCZhkCO3b0lOSXGFPqLhc?= =?us-ascii?Q?ABIdkjmZIJZbL1YbLIIv3eL9USoEiumwrZ5+N4tO6B0+GFxhG4vqqWJTRc8m?= =?us-ascii?Q?aa0zcs+4/aZxix3HC/rKvHX+mUbFaplLtEcZ/Esq/yBKYZMAMjfgat64axH8?= =?us-ascii?Q?nZ1lDrTmwKQUJ9BrbWQS1EDE5m0Mp1+bEArhB+lvjgC5ymuq04LpdRnYyILR?= =?us-ascii?Q?ZgI2B2jVEAKQIf9W2gx5meTjJA+dbhhd3vk8X9DX2npE1DQs8AiUmo7SB9Zx?= =?us-ascii?Q?/GvoH3MhICwmSk/IBwhs/+Q+lDWd1yQ+UFPmKLu0nM6AFOq4ijk11sasTATt?= =?us-ascii?Q?ZO2oSYHIcwgOtkCfpAbi9/Apa7NzGy45Nc1BNsmIuZZ+NuFdP6sqVhtk6CD1?= =?us-ascii?Q?o/m/3PgXbvGtTR4uYA4cXiY1J/PK/Nhsw0cPGJlqYaBtZDyWmhFFQ/94ip0k?= =?us-ascii?Q?aR7WGP4fCb6FAX1L+Vioaih1oE/TyHjeV2ewhYALuxYHgWo8Bftgj1eFBLHR?= =?us-ascii?Q?DMCY3+2aXnjpEaSJiy/NeRxKFPKhntmc817rkTRVEVtsVG6azaJ3YUY1+o4a?= =?us-ascii?Q?nwyIruC9j+UmyzRc0h+uQYrHUzS1Buf4fJbJKzi2u/KuPeD7XrGHLK8JFK6x?= =?us-ascii?Q?+EV0obWxDKX3OSG1plNZC27+r/sbAjmYu5GAYjK7+iIfMA39YqW0aDRQnCsa?= =?us-ascii?Q?f4EW5578u6IsUL6XSlGEvIMd64EqSjBpQiaS+mqxwrv+eN3M8iOHTuPh6QNy?= =?us-ascii?Q?0cF7YsYjSrQsrH8FPVoZ6JNS9pD4AGNrzQ/NRHmD50ccavPO4pqXj3fBjYq9?= =?us-ascii?Q?dqa/Qe4Nc+I+yP12T0TYRM9KUInyLTfsfJ0SS+8+6BLgC4EGZdJhGUe5jM0D?= =?us-ascii?Q?BDtHwKxeBLUWEEcpspCGWG5diYsenmq6aOpEMvbX3lif3hVt3v+hqK9WQFvK?= =?us-ascii?Q?zyfQLxflhHPgRWZThN6vKlmGKwx4eA7UtrG8O1AidsccvdO23+U6DkGFL3Vj?= =?us-ascii?Q?a5BFuttkbZ3RPKFL+H0ySfIwX7xmdWZKS4NZY1TekWUMCDGmaFhRMtjEPeZJ?= =?us-ascii?Q?d9vTfmRhqqJyGus5uaM7QcgDGyV5kFw7SDHstIHk73y3HHfwb47Mrs1jxxpk?= =?us-ascii?Q?oBLUbvPZb0jWtAQEgyPIU5BoAqNxIHZczyM3F1EwtJ6zLyYsua0HJd/0GSuD?= =?us-ascii?Q?GsYVgiYyg2GFaUWCrKaVezoiXHaSLp3JvYhoQKnnY6eG9UQeVUDMzkKTKk0s?= =?us-ascii?Q?ugQOqBmioOIZJ/FRB1YBBkNmluM5b1iv16Qlv9vN9lvNgHULBvoOpSl+eZgd?= =?us-ascii?Q?DEwZluy0zgeJY/2yL/5jx5LlOQARBCwoPEmPlsd7j4hidsBcR41HnwawCOh0?= =?us-ascii?Q?7Kl0hzcypD1nNA1ukvgkOldDxAjSJPXUF4F1KixmOwziDEkfJ33D1FTegbcb?= =?us-ascii?Q?voQzZDb4KPlmIMb067n3Cn5UaKxAhp8ftypld1VC?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA0PR12MB8983.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 51cffd95-018f-4b59-ec6b-08dcdb9f2c34 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Sep 2024 07:13:06.7642 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +dGdWqi6WkWtcA8m8n1mw/LcaHx8O1wkTnTw/x1XKkGh7Aje42xdl/1NVogmqFInwEqBqC//7zv1VyMOscs7PQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7754 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > On Sep 20, 2024, at 23:09, Maxime Coquelin w= rote: >=20 > External email: Use caution opening links or attachments >=20 >=20 > This patch enables VDUSE reconnection support making use of > the newly introduced reconnection mechanism in Vhost > library. >=20 > At DPDK VDUSE device creation time, there are two > possibilities: > 1. The Kernel VDUSE device does not exist: > a. A reconnection file named after the VUDSE device name > is created in VDUSE tmpfs. > b. The file is truncated to 'struct vhost_reconnect_data' > size, and mmapped. > c. Negotiated features, Virtio status... are saved for > sanity checks at reconnect time. > 2. The Kernel VDUSE device already exists: > a. Exit with failure if no reconnect file exists for > this device. > b. Open and mmap the reconnect file. > c. Perform sanity check to ensure features are compatible. > d. Restore virtqueues' available indexes at startup time. >=20 > Then at runtime, the virtqueues' available index are logged by > the Vhost reconnection mechanism. >=20 > At DPDK VDUSE device destruction time, there are two > possibilities: > 1. The Kernel VDUSE device destruction succeed, which > means it is no more attached to the vDPA bus. The > reconnection file is unmapped and then removed. > 2. The Kernel VDUSE device destruction failed, meaning it > is no more attached to the vDPA bus. The reconnection > file is unmapped but not removed to make possible later > reconnection. >=20 > Signed-off-by: Maxime Coquelin > --- > lib/vhost/vduse.c | 308 ++++++++++++++++++++++++++++++++++++++++------ > 1 file changed, 268 insertions(+), 40 deletions(-) >=20 > diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c > index c66602905c..f9ac317438 100644 > --- a/lib/vhost/vduse.c > +++ b/lib/vhost/vduse.c > @@ -136,7 +136,7 @@ vduse_control_queue_event(int fd, void *arg, int *rem= ove __rte_unused) > } >=20 > static void > -vduse_vring_setup(struct virtio_net *dev, unsigned int index) > +vduse_vring_setup(struct virtio_net *dev, unsigned int index, bool recon= nect) > { > struct vhost_virtqueue *vq =3D dev->virtqueue[index]; > struct vhost_vring_addr *ra =3D &vq->ring_addrs; > @@ -152,6 +152,19 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > return; > } >=20 > + if (reconnect) { > + vq->last_avail_idx =3D vq->reconnect_log->last_avail_idx; > + vq->last_used_idx =3D vq->reconnect_log->last_avail_idx; > + } else { > + vq->last_avail_idx =3D vq_info.split.avail_index; > + vq->last_used_idx =3D vq_info.split.avail_index; > + } > + vq->size =3D vq_info.num; > + vq->ready =3D true; > + vq->enabled =3D vq_info.ready; > + ra->desc_user_addr =3D vq_info.desc_addr; > + ra->avail_user_addr =3D vq_info.driver_addr; > + ra->used_user_addr =3D vq_info.device_addr; > VHOST_CONFIG_LOG(dev->ifname, INFO, "VQ %u info:", index); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tnum: %u", vq_info.num); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdesc_addr: %llx", > @@ -160,17 +173,9 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > (unsigned long long)vq_info.driver_addr); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdevice_addr: %llx", > (unsigned long long)vq_info.device_addr); > - VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq_info.sp= lit.avail_index); > + VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq->last_a= vail_idx); > + VHOST_CONFIG_LOG(dev->ifname, INFO, "\tused_idx: %u", vq->last_us= ed_idx); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tready: %u", vq_info.ready); > - > - vq->last_avail_idx =3D vq_info.split.avail_index; > - vq->size =3D vq_info.num; > - vq->ready =3D true; > - vq->enabled =3D vq_info.ready; > - ra->desc_user_addr =3D vq_info.desc_addr; > - ra->avail_user_addr =3D vq_info.driver_addr; > - ra->used_user_addr =3D vq_info.device_addr; > - > vq->kickfd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > if (vq->kickfd < 0) { > VHOST_CONFIG_LOG(dev->ifname, ERR, "Failed to init kickfd = for VQ %u: %s", > @@ -267,7 +272,7 @@ vduse_vring_cleanup(struct virtio_net *dev, unsigned = int index) > } >=20 > static void > -vduse_device_start(struct virtio_net *dev) > +vduse_device_start(struct virtio_net *dev, bool reconnect) > { > unsigned int i, ret; >=20 > @@ -287,6 +292,15 @@ vduse_device_start(struct virtio_net *dev) > return; > } >=20 > + if (reconnect && dev->features !=3D dev->reconnect_log->features)= { > + VHOST_CONFIG_LOG(dev->ifname, ERR, > + "Mismatch between reconnect file features= 0x%" PRIx64 " & device features 0x%" PRIx64, > + dev->reconnect_log->features, dev->featur= es); > + return; > + } > + > + dev->reconnect_log->features =3D dev->features; > + > VHOST_CONFIG_LOG(dev->ifname, INFO, "Negotiated Virtio features: 0= x%" PRIx64, > dev->features); >=20 > @@ -300,7 +314,7 @@ vduse_device_start(struct virtio_net *dev) > } >=20 > for (i =3D 0; i < dev->nr_vring; i++) > - vduse_vring_setup(dev, i); > + vduse_vring_setup(dev, i, reconnect); >=20 > dev->flags |=3D VIRTIO_DEV_READY; >=20 > @@ -373,6 +387,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) > req.s.status); > old_status =3D dev->status; > dev->status =3D req.s.status; > + dev->reconnect_log->status =3D dev->status; > resp.result =3D VDUSE_REQ_RESULT_OK; > break; > case VDUSE_UPDATE_IOTLB: > @@ -398,7 +413,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) >=20 > if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) > - vduse_device_start(dev); > + vduse_device_start(dev, false); > else > vduse_device_stop(dev); > } > @@ -407,10 +422,64 @@ vduse_events_handler(int fd, void *arg, int *remove= __rte_unused) > vduse_req_id_to_str(req.type), req.type); > } >=20 > +static char vduse_reconnect_dir[PATH_MAX]; > +static bool vduse_reconnect_path_set; > + > +static int > +vduse_reconnect_path_init(void) > +{ > + const char *directory; > + int ret; > + > + /* from RuntimeDirectory=3D see systemd.exec */ > + directory =3D getenv("RUNTIME_DIRECTORY"); > + if (directory =3D=3D NULL) { > + /* > + * Used standard convention defined in > + * XDG Base Directory Specification and > + * Filesystem Hierarchy Standard. > + */ > + if (getuid() =3D=3D 0) > + directory =3D "/var/run"; > + else > + directory =3D getenv("XDG_RUNTIME_DIR") ? : "/tmp= "; > + } > + > + ret =3D snprintf(vduse_reconnect_dir, sizeof(vduse_reconnect_dir)= , "%s/vduse", > + directory); > + if (ret < 0 || ret =3D=3D sizeof(vduse_reconnect_dir)) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating VDUSE reco= nnect path name"); > + return -1; > + } > + > + ret =3D mkdir(vduse_reconnect_dir, 0700); > + if (ret < 0 && errno !=3D EEXIST) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating '%s': %s", > + vduse_reconnect_dir, strerror(errno)); > + return -1; > + } > + > + VHOST_CONFIG_LOG("vduse", INFO, "Created VDUSE reconnect director= y in %s", > + vduse_reconnect_dir); > + > + return 0; > +} > + > +static void > +vduse_reconnect_handler(int fd, void *arg, int *remove) > +{ > + struct virtio_net *dev =3D arg; > + > + vduse_device_start(dev, true); > + > + close(fd); > + *remove =3D 1; > +} > + > int > vduse_device_create(const char *path, bool compliant_ol_flags) > { > - int control_fd, dev_fd, vid, ret; > + int control_fd, dev_fd, vid, ret, reco_fd; > uint32_t i, max_queue_pairs, total_queues; > struct virtio_net *dev; > struct virtio_net_config vnet_config =3D {{ 0 }}; > @@ -418,6 +487,9 @@ vduse_device_create(const char *path, bool compliant_= ol_flags) > uint64_t features; > struct vduse_dev_config *dev_config =3D NULL; > const char *name =3D path + strlen("/dev/vduse/"); > + char reconnect_file[PATH_MAX]; > + struct vhost_reconnect_data *reconnect_log =3D NULL; > + bool reconnect =3D false; >=20 > if (vduse.fdset =3D=3D NULL) { > vduse.fdset =3D fdset_init("vduse-evt"); > @@ -427,6 +499,20 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > } > } >=20 > + if (vduse_reconnect_path_set =3D=3D false) { > + if (vduse_reconnect_path_init() < 0) { > + VHOST_CONFIG_LOG(path, ERR, "failed to initialize= reconnect path"); > + return -1; > + } > + vduse_reconnect_path_set =3D true; > + } > + > + ret =3D snprintf(reconnect_file, sizeof(reconnect_file), "%s/%s",= vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create vduse recon= nect path name"); > + return -1; > + } > + > control_fd =3D open(VDUSE_CTRL_PATH, O_RDWR); > if (control_fd < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to open %s: %s", > @@ -441,14 +527,6 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_ctrl_close; > } >=20 > - dev_config =3D malloc(offsetof(struct vduse_dev_config, config) + > - sizeof(vnet_config)); > - if (!dev_config) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to allocate VDUSE con= fig"); > - ret =3D -1; > - goto out_ctrl_close; > - } > - > ret =3D rte_vhost_driver_get_features(path, &features); > if (ret < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to get backend feature= s"); > @@ -469,23 +547,118 @@ vduse_device_create(const char *path, bool complia= nt_ol_flags) > else > total_queues +=3D 1; /* Includes ctrl queue */ >=20 > - vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > - memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + if (access(path, F_OK) =3D=3D 0) { > + VHOST_CONFIG_LOG(name, INFO, "Device already exists, reco= nnecting..."); > + reconnect =3D true; > + > + reco_fd =3D open(reconnect_file, O_RDWR, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D ENOENT) > + VHOST_CONFIG_LOG(name, ERR, "Missing reco= nnect file (%s)", > + reconnect_file); > + else > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } >=20 > - strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > - dev_config->device_id =3D VIRTIO_ID_NET; > - dev_config->vendor_id =3D 0; > - dev_config->features =3D features; > - dev_config->vq_num =3D total_queues; > - dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > - dev_config->config_size =3D sizeof(struct virtio_net_config); > - memcpy(dev_config->config, &vnet_config, sizeof(vnet_config)); > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } >=20 > - ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > - if (ret < 0) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to create VDUSE devic= e: %s", > - strerror(errno)); > - goto out_free; > + if (reconnect_log->version !=3D VHOST_RECONNECT_VERSION) = { > + VHOST_CONFIG_LOG(name, ERR, > + "Version mismatch between backend= (0x%x) & reconnection file (0x%x)", > + VHOST_RECONNECT_VERSION, reconnec= t_log->version); > + } > + > + if ((reconnect_log->features & features) !=3D reconnect_l= og->features) { > + VHOST_CONFIG_LOG(name, ERR, > + "Features mismatch between backen= d (0x%" PRIx64 ") & reconnection file (0x%" PRIx64 ")", > + features, reconnect_log->features= ); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + if (reconnect_log->nr_vrings !=3D total_queues) { > + VHOST_CONFIG_LOG(name, ERR, > + "Queues number mismatch between b= ackend (%u) and reconnection file (%u)", > + total_queues, reconnect_log->nr_v= rings); > + ret =3D -1; > + goto out_ctrl_close; > + } > + } else { > + reco_fd =3D open(reconnect_file, O_CREAT | O_EXCL | O_RDW= R, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D EEXIST) { > + VHOST_CONFIG_LOG(name, ERR, "Reconnect fi= le %s exists but not the device", > + reconnect_file); > + } else { > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + } > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + ret =3D ftruncate(reco_fd, sizeof(*reconnect_log)); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to truncate r= econnect file %s (%s)", > + reconnect_file, strerror(errno)); > + close(reco_fd); > + goto out_ctrl_close; > + } > + > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + reconnect_log->version =3D VHOST_RECONNECT_VERSION; > + > + dev_config =3D malloc(offsetof(struct vduse_dev_config, c= onfig) + > + sizeof(vnet_config)); > + if (!dev_config) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to allocate V= DUSE config"); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > + memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + > + rte_strscpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > + dev_config->device_id =3D VIRTIO_ID_NET; > + dev_config->vendor_id =3D 0; > + dev_config->features =3D features; > + dev_config->vq_num =3D total_queues; > + dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > + dev_config->config_size =3D sizeof(struct virtio_net_conf= ig); > + memcpy(dev_config->config, &vnet_config, sizeof(vnet_conf= ig)); > + > + ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create VDU= SE device: %s", > + strerror(errno)); > + goto out_free; > + } > + > + memcpy(&reconnect_log->config, &vnet_config, sizeof(vnet_= config)); > + reconnect_log->nr_vrings =3D total_queues; > + free(dev_config); > + dev_config =3D NULL; > } >=20 > dev_fd =3D open(path, O_RDWR); > @@ -519,10 +692,15 @@ vduse_device_create(const char *path, bool complian= t_ol_flags) > strncpy(dev->ifname, path, IF_NAME_SZ - 1); > dev->vduse_ctrl_fd =3D control_fd; > dev->vduse_dev_fd =3D dev_fd; > + dev->reconnect_log =3D reconnect_log; > + if (reconnect) > + dev->status =3D dev->reconnect_log->status; > + > vhost_setup_virtio_net(dev->vid, true, compliant_ol_flags, true, t= rue); >=20 > for (i =3D 0; i < total_queues; i++) { > struct vduse_vq_config vq_cfg =3D { 0 }; > + struct vhost_virtqueue *vq; >=20 > ret =3D alloc_vring_queue(dev, i); > if (ret) { > @@ -530,6 +708,12 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > + vq =3D dev->virtqueue[i]; > + vq->reconnect_log =3D &reconnect_log->vring[i]; > + > + if (reconnect) > + continue; > + > vq_cfg.index =3D i; > vq_cfg.max_size =3D 1024; >=20 > @@ -549,7 +733,32 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > - free(dev_config); > + if (reconnect && dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > + /* > + * Make vduse_device_start() being executed in the same > + * context for both reconnection and fresh startup. > + */ > + reco_fd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > + if (reco_fd < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create rec= o_fd: %s", > + strerror(errno)); > + ret =3D -1; > + goto out_dev_destroy; > + } > + > + ret =3D fdset_add(vduse.fdset, reco_fd, vduse_reconnect_h= andler, NULL, dev); > + if (ret) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to add reconn= ect fd %d to vduse fdset", > + reco_fd); > + goto out_dev_destroy; > + } > + > + ret =3D eventfd_write(reco_fd, (eventfd_t)1); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to write to r= econnect eventfd"); > + goto out_dev_destroy; > + } > + } >=20 > return 0; >=20 > @@ -587,6 +796,9 @@ vduse_device_destroy(const char *path) > if (vid =3D=3D RTE_MAX_VHOST_DEVICE) > return -1; >=20 > + if (dev->reconnect_log) > + munmap(dev->reconnect_log, sizeof(*dev->reconnect_log)); > + > vduse_device_stop(dev); >=20 > fdset_del(vduse.fdset, dev->vduse_dev_fd); > @@ -597,10 +809,26 @@ vduse_device_destroy(const char *path) > } >=20 > if (dev->vduse_ctrl_fd >=3D 0) { > + char reconnect_file[PATH_MAX]; > + > ret =3D ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name)= ; > - if (ret) > + if (ret) { > VHOST_CONFIG_LOG(name, ERR, "Failed to destroy VDU= SE device: %s", > strerror(errno)); > + } else { > + /* > + * VDUSE device was no more attached to the vDPA = bus, > + * so we can remove the reconnect file. > + */ > + ret =3D snprintf(reconnect_file, sizeof(reconnect= _file), "%s/%s", > + vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) > + VHOST_CONFIG_LOG(name, ERR, > + "Failed to create vduse r= econnect path name"); > + else > + unlink(reconnect_file); > + } > + > close(dev->vduse_ctrl_fd); > dev->vduse_ctrl_fd =3D -1; > } > -- > 2.46.0 >=20 Reviewed-by: Chenbo Xia