From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 032484591A; Fri, 6 Sep 2024 09:14:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E1AB14025D; Fri, 6 Sep 2024 09:14:57 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2081.outbound.protection.outlook.com [40.107.92.81]) by mails.dpdk.org (Postfix) with ESMTP id 100884025C for ; Fri, 6 Sep 2024 09:14:56 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LencbrhzeCmdwA3ax6ieV16EHgDvhv+nE6fPkHIVON5ciMFdZNiKKgOY7N+jnEvAbSpjHgB6dCyppcKO5ShoJL/T0eyksSi6Ti4WSXr5SJzCBaWh7Ed1n3PQNmz03Aoy/gtkb008Hjh+prdfT4PiNFDeSHj1lfZ2tTx5xzn510KHHwUpEQkYADKjynjiwNGWKPEf+DjYyFTXGs5f9hbso2FsYsETDrADQpBhY9ktssfMbt0EmferMDHJECF2Z/Kghm8fOOyTrKo7EsdD25LH+Ma38X1DyKfHubsHg9+ECKLqWUaZ802Z4xoV/2Uc1yQu4Q+Xi8MEL46+fFkc576AVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=; b=DjDl48m5BLv+Yj/g6VHXGzcHPcrn/NTWdxuqJyclpSYr/s0ZZgcsYs+FXMwsH1lJJk7TDUTkd8lP4ev6cK6kwF9lQmtrvZ+w4Xl7pRQOgdnAAe5gqTeLvpnsi869VthWTobvZFUcFKO3nOdQRVIoXza3PcgUdP5NgjkcE4Y0LS7f1VtVKYGbMmwyQpA1oU0D4kGoW3IS6VmJXmEQOQY+QCUFpzjDPExwc9SZmdwg7l+ixWMbdnTV4umdl37UHbz2VSOakFAxwb+LcB77bkJtO/0qiomI/9DHz5dUlgIMPAcR/VjB3tBBshX9ki3qGsjl+RME+Wlutj5/fJrhKS8Bmw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=; b=QrnL6bciDbRnuA7bzYC/SIGMMUSTbglROzxUgq/3tU8iWYyg1/o4AhDG0YqBr4sZfgXGcqcrrGtBkM+rSOUlo4rXI9d8E2Qj7LYFdLa+MlDERRZ0/7MbI3SXysEZhBZQcjJlVbfgggjXnOYW2CwcvCes3JFXJ7TGBTxuBCQgJQg862wXfyAYRYxPrObR55yfUfafJRXs9orNviFbh4LCF6TdtIRKDu/g/fILKu0mf364W6NfweOf4eOzW7qs2wt2PMB4T+tRAG/FMi0kZ1dnFNbtxI3bVfpcTB8PsGwmAnBBEAsBDdM9FeW6AV1g2fOxr9UcgHCI31JRPAbN0y6HcA== Received: from CYYPR12MB8992.namprd12.prod.outlook.com (2603:10b6:930:bc::21) by PH0PR12MB7471.namprd12.prod.outlook.com (2603:10b6:510:1e9::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.28; Fri, 6 Sep 2024 07:14:53 +0000 Received: from CYYPR12MB8992.namprd12.prod.outlook.com ([fe80::a722:c2ca:42c7:aaed]) by CYYPR12MB8992.namprd12.prod.outlook.com ([fe80::a722:c2ca:42c7:aaed%7]) with mapi id 15.20.7918.024; Fri, 6 Sep 2024 07:14:53 +0000 From: Chenbo Xia To: Maxime Coquelin CC: "dev@dpdk.org" , "david.marchand@redhat.com" Subject: Re: [PATCH 2/2] vhost: add reconnection support to VDUSE Thread-Topic: [PATCH 2/2] vhost: add reconnection support to VDUSE Thread-Index: AQHa/5+Tb2hkJpD2pkWfDp9/zoprPbJKWaMA Date: Fri, 6 Sep 2024 07:14:52 +0000 Message-ID: <3543971A-6C8E-4F28-A7D4-F8C222F3C193@nvidia.com> References: <20240905142602.1264345-1-maxime.coquelin@redhat.com> <20240905142602.1264345-3-maxime.coquelin@redhat.com> In-Reply-To: <20240905142602.1264345-3-maxime.coquelin@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CYYPR12MB8992:EE_|PH0PR12MB7471:EE_ x-ms-office365-filtering-correlation-id: 8e70f466-1363-4126-7967-08dcce439a80 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|376014|366016|1800799024|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?zx0DWt7HeeTId2KR7QXZur6JkKYQkBL4h4tN6PYr5QlLwGBh3OM6z+dz8xKU?= =?us-ascii?Q?Pxu/wfFprpDloy/rf8v0UmBlwBair9irkwJeHX0a0LIvNNtl7k2ldpszKmxW?= =?us-ascii?Q?Y50MEFmpZLciFcn03IFHfIDLc1KqaEdfnsP2k1QNVf4vQjuA8yoE1TL8f7A9?= =?us-ascii?Q?d9+UBA9gn3SOiVKb+GqjfdYtjQvrSuoBGY3CEEJVKzghaV8KHZX6IsXaAyiS?= =?us-ascii?Q?g0+ZdhDFR1qBoG+8mIXLd7Q6hkeXHLbs3J7gEHC2IYzcLDk+ESalh8YlyxSJ?= =?us-ascii?Q?9a1t0Rfsw7owdzuO+DtGYzCOGHta6sYyG+dJZzFKOhi8YYaiUXQaVUiya+Gt?= =?us-ascii?Q?qfkevXP6PpKGalJyagzuwQWMG0xSO3D2sKl250giGs8LEin8rCH8F05KtN1O?= =?us-ascii?Q?FA75UW2WUHhibRUZGkbeu5S4FjyilLTUSqOOjLcfrhhuFoNtfSGWF7PcJOVw?= =?us-ascii?Q?oOjlkn124zR0RdkR69h4N2wMevFG1DrbfYsjsO2HPVHLWcxemBAwDx5L2Mis?= =?us-ascii?Q?VIT1Qw+7nh5M9w7UrmZKCOrUT7J6gh5QYxDrLWWz/CFY3GcKMnTNEdzVdIds?= =?us-ascii?Q?APN1C2KKeXyh9NpBYfow1DTPmkRuNrbuWrIcebjKuv3GGvmc+Baa0zY6EzM3?= =?us-ascii?Q?i4aQaZaxZqMvQM8w/N+VWPZYx1OSkOBmIiQD5P3yu0NC423zlHmDkHWjQ1Wr?= =?us-ascii?Q?BxUkrRFAEz2RTNcc6KoPOI1Uqfwy+m3fZvmNum5WXtotJrGXez2IyXzBppzF?= =?us-ascii?Q?ygOq1VgJyGCarKsjNyyobJ6iY2P8v0KSGyDgWqDbWE6+8dajmZJoV3YMZzLx?= =?us-ascii?Q?3Z96LWTGgINOZTanmdJttJF5WudopfHTgTLHOL6diR0pM800tUwDC94FTpZc?= =?us-ascii?Q?83QZxQOTSnTwvTqaG0NtU6kWbYHFsM/kVTVZvVhPxxbDxJqdljHVyuyFaWbk?= =?us-ascii?Q?G8JZCk0zjmsx8OosiVo7LvpjNI4HiFwSHwGIxloGj5Y66hQ0eCxZDBc7+/uh?= =?us-ascii?Q?DCQyJpPR6guI1WaEgYoxDLASBvSjFxbpz6Fvi68C5OaUAk85voeZBP8Qvck2?= =?us-ascii?Q?hZ2yFvng9sSGA9fypKhA2lnOaYybms/s2oA5xJhmmAmPA7QcVLEAwmHKBUcj?= =?us-ascii?Q?JUBrEQarmSvcPFlNSWcvvGcuzgA2Wn7aGS4pTQJuFOzoOiAtImiwrqo4GREm?= =?us-ascii?Q?9unM0HjuQRAsI/b/1teSSa9hKv43PthpKmbm9WmuwNTp6WrjVz1IAVOoifZh?= =?us-ascii?Q?OBu8p4IL5jxN+0d9CWH1Q1FX5JjZ4h55wvay9fLATVjkgm8TW5+w210H7Ynf?= =?us-ascii?Q?9O8der+DaUOCx1IGwUI/TXcEqRNgWiZzvviMFepFmIrPBMduw21W2ndPuY71?= =?us-ascii?Q?48TuG6dG/hzJK5PAOh+yqBnpz/pwX5+nZKm77VXssQ5AXD8Mog=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CYYPR12MB8992.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(366016)(1800799024)(38070700018); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?WTXsGEUCFmmDmgE/bty3qHtCee/ttLJbJ+Bh987Aj4GW7aeF7H882juE15OM?= =?us-ascii?Q?JSxpvZpiddNPE+w8kEr6Vp+shtnix3DjUn4A/N98GmbgmGp3MHsDokYrtL3U?= =?us-ascii?Q?je77hop9Ya9zUP3l9loZuy6HVdV3HvzSYNgvk/+3tiumVA28KXgvLIoLN+yM?= =?us-ascii?Q?uzxqIVS+I9s0lctX1IzNx5Cf57YwXH5N43j3+AMxrTgMxFTwKGSGXVacuIXW?= =?us-ascii?Q?cTdUjyWEzW9LTdVGPlkyZ+eE8xXtE8jnopfMYI/bdalIO2q8MzA9Mjw/NlIs?= =?us-ascii?Q?ZdRSNOyOxgcvk/PDJzfSU+wMrqzEayUp33MPgu8/NSMnM8NBvJMULJwqbjYJ?= =?us-ascii?Q?ixx0oJlmhQW8n1wtDBw4eDm9hcTuMgnVG751q8fVhBsmPwGtSKA4kMwdVxiI?= =?us-ascii?Q?WBVRFt1GnoWBOhCiBZ3rjt/uoDv6XbJhnHIjbFxteabs23QFAWaJ5uHgYFyX?= =?us-ascii?Q?boFMEX/Sl5V2x/CeriQWiJW34KvExYpFUPNdzXID601o3Robrx3l+SphrMjj?= =?us-ascii?Q?jy84YloWmKyLywe+RRsGpNoPGZM86t4bdglpecl4zmGRCdAJdtbvEp1EZ5TX?= =?us-ascii?Q?QUxgxZzemz6PF2Kx5iDqOh28ybkX1cp4VhmQUsYWkMBOxO5AXT0J4yi5Ltgr?= =?us-ascii?Q?FSoAlfzN8QG3Wceme12Lt746d98+2yn4I5yOMINJepN8nnqlpBP8/liRdlg1?= =?us-ascii?Q?W6zGVNizvmYMJY5gYdyPiEy+YAK2YjnSujd3GZ22N0Ta/CTtYxqMYgBqRpEe?= =?us-ascii?Q?oyIGeptWxFYCEWoIxYRKUnL6pNnbDUawX/gGqRoYtE4QQ7/IOXELOuP1L14J?= =?us-ascii?Q?6TBUUpAKd5G7tv8ID1o7rhzZGVd/q2Upg0i69yNBPivTbzpuLZXHbolP/BQW?= =?us-ascii?Q?OwynTmtUiYIaHlkssUsB7yuGINkmxUn53BG5+yKtnS9vXfCTJ5vlTgPtX5td?= =?us-ascii?Q?q+MACo+/Xl7NOp0LjldTbHBDR4xutasfdGdqWJdwYYjeLhP1QiiC0KlvTlwf?= =?us-ascii?Q?csU9zJN9Lvh2FSVUecOynHsRtEkyBL0dY0f/d1qsflNcfzjM08DttUONjAU3?= =?us-ascii?Q?BwW49sH1d53agHC2NzhpXXLOA2ME9QUnEIY27IXlQkC5a/8Vd0xfrMtAHbjL?= =?us-ascii?Q?NKRotZHjQKzkIE0gC4ligGIep0nZzyFjOqjH0RMJtRojNfyuslUWKVxaiEe8?= =?us-ascii?Q?CgaxxCpyA/hz++IT/mEcwhcUbEDC79ttQ0spoS05nIXfxePNIDq/hhozVAVb?= =?us-ascii?Q?oxVLy74lq2WXo9n2UdLulGhTJBvRJVh0mfoNOsb9FY4PNlY5V/0WD+BcUDQO?= =?us-ascii?Q?4F1ORVZ6ASN/BPKmTHCmIcZ1SPVV9aj+aGC8aGNHUQhGC8BenB/kZbWRRBA3?= =?us-ascii?Q?dQvZd0g9NG7n8fzrWEPsFHk4SXHK2Od1lSP7hhDPe7HoiWIEoPF2lEehALya?= =?us-ascii?Q?aJ9PWyLd1pGVUENHTt7K02dsArSIf8+Wez6Pl15UcGZ0aBCzIbWnYJTln36y?= =?us-ascii?Q?uljxX+hvM23a+BmgQeXKxH6K1VlFYTLDxzNpet06efLjBJeAiS7/SvAnsv72?= =?us-ascii?Q?txqP1ck+2E+V3w2lUE+NVkfCSiHhIZwKfbOQmGSCrFmpJKWO61ST27fGcOCT?= =?us-ascii?Q?dg=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CYYPR12MB8992.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8e70f466-1363-4126-7967-08dcce439a80 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2024 07:14:52.9774 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 0LXdZFwi7a//FlOKuaStkDe9GCNGaLJsIxFpJQJV3BW/zfsS7nebnRMhi6zDIy3xvqffBmsCwMIj03FY1uv28A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7471 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Maxime, > On Sep 5, 2024, at 22:26, Maxime Coquelin wr= ote: >=20 > External email: Use caution opening links or attachments >=20 >=20 > This patch enables VDUSE reconnection support making use of > the newly introduced reconnection mechanism in Vhost > library. >=20 > At DPDK VDUSE device creation time, there are two > possibilities: > 1. The Kernel VDUSE device does not exist: > a. A reconnection file named after the VUDSE device name > is created in VDUSE tmpfs. > b. The file is truncated to 'struct vhost_reconnect_data' > size, and mmapped. > c. Negotiated features, Virtio status... are saved for > sanity checks at reconnect time. > 2. The Kernel VDUSE device already exists: > a. Exit with failure if no reconnect file exists for > this device. > b. Open and mmap the reconnect file. > c. Perform sanity check to ensure features are compatible. > d. Restore virtqueues' available indexes at startup time. >=20 > Then at runtime, the virtqueues' available index are logged by > the Vhost reconnection mechanism. >=20 > At DPDK VDUSE device destruction time, there are two > possibilities: > 1. The Kernel VDUSE device destruction succeed, which > means it is no more attached to the vDPA bus. The > reconnection file is unmapped and then removed. > 2. The Kernel VDUSE device destruction failed, meaning it > is no more attached to the vDPA bus. The reconnection > file is unmapped but not removed to make possible later > reconnection. >=20 > Signed-off-by: Maxime Coquelin > --- > lib/vhost/vduse.c | 280 +++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 241 insertions(+), 39 deletions(-) >=20 > diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c > index c66602905c..bd0e492d62 100644 > --- a/lib/vhost/vduse.c > +++ b/lib/vhost/vduse.c > @@ -136,7 +136,7 @@ vduse_control_queue_event(int fd, void *arg, int *rem= ove __rte_unused) > } >=20 > static void > -vduse_vring_setup(struct virtio_net *dev, unsigned int index) > +vduse_vring_setup(struct virtio_net *dev, unsigned int index, bool recon= nect) > { > struct vhost_virtqueue *vq =3D dev->virtqueue[index]; > struct vhost_vring_addr *ra =3D &vq->ring_addrs; > @@ -152,6 +152,19 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > return; > } >=20 > + if (reconnect) { > + vq->last_avail_idx =3D vq->reconnect_log->last_avail_idx; > + vq->last_used_idx =3D vq->reconnect_log->last_avail_idx; > + } else { > + vq->last_avail_idx =3D vq_info.split.avail_index; > + vq->last_used_idx =3D vq_info.split.avail_index; > + } > + vq->size =3D vq_info.num; > + vq->ready =3D true; > + vq->enabled =3D vq_info.ready; > + ra->desc_user_addr =3D vq_info.desc_addr; > + ra->avail_user_addr =3D vq_info.driver_addr; > + ra->used_user_addr =3D vq_info.device_addr; > VHOST_CONFIG_LOG(dev->ifname, INFO, "VQ %u info:", index); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tnum: %u", vq_info.num); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdesc_addr: %llx", > @@ -162,15 +175,6 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > (unsigned long long)vq_info.device_addr); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq_info.spl= it.avail_index); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tready: %u", vq_info.ready); > - > - vq->last_avail_idx =3D vq_info.split.avail_index; > - vq->size =3D vq_info.num; > - vq->ready =3D true; > - vq->enabled =3D vq_info.ready; > - ra->desc_user_addr =3D vq_info.desc_addr; > - ra->avail_user_addr =3D vq_info.driver_addr; > - ra->used_user_addr =3D vq_info.device_addr; > - > vq->kickfd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > if (vq->kickfd < 0) { > VHOST_CONFIG_LOG(dev->ifname, ERR, "Failed to init kickfd = for VQ %u: %s", > @@ -267,7 +271,7 @@ vduse_vring_cleanup(struct virtio_net *dev, unsigned = int index) > } >=20 > static void > -vduse_device_start(struct virtio_net *dev) > +vduse_device_start(struct virtio_net *dev, bool reconnect) > { > unsigned int i, ret; >=20 > @@ -287,6 +291,15 @@ vduse_device_start(struct virtio_net *dev) > return; > } >=20 > + if (reconnect && dev->features !=3D dev->reconnect_log->features)= { > + VHOST_CONFIG_LOG(dev->ifname, ERR, > + "Mismatch between reconnect file features= 0x%" PRIx64 " & device features 0x%" PRIx64, Checkpatch reports long line > + dev->reconnect_log->features, dev->featur= es); > + return; > + } > + > + dev->reconnect_log->features =3D dev->features; > + > VHOST_CONFIG_LOG(dev->ifname, INFO, "Negotiated Virtio features: 0= x%" PRIx64, > dev->features); >=20 > @@ -300,7 +313,7 @@ vduse_device_start(struct virtio_net *dev) > } >=20 > for (i =3D 0; i < dev->nr_vring; i++) > - vduse_vring_setup(dev, i); > + vduse_vring_setup(dev, i, reconnect); >=20 > dev->flags |=3D VIRTIO_DEV_READY; >=20 > @@ -373,6 +386,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) > req.s.status); > old_status =3D dev->status; > dev->status =3D req.s.status; > + dev->reconnect_log->status =3D dev->status; > resp.result =3D VDUSE_REQ_RESULT_OK; > break; > case VDUSE_UPDATE_IOTLB: > @@ -398,7 +412,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) >=20 > if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) > - vduse_device_start(dev); > + vduse_device_start(dev, false); > else > vduse_device_stop(dev); > } > @@ -407,10 +421,64 @@ vduse_events_handler(int fd, void *arg, int *remove= __rte_unused) > vduse_req_id_to_str(req.type), req.type); > } >=20 > +static char vduse_reconnect_dir[PATH_MAX]; > +static bool vduse_reconnect_path_set; > + > +static int > +vduse_reconnect_path_init(void) > +{ > + const char *directory; > + int ret; > + > + /* from RuntimeDirectory=3D see systemd.exec */ > + directory =3D getenv("RUNTIME_DIRECTORY"); > + if (directory =3D=3D NULL) { > + /* > + * Used standard convention defined in > + * XDG Base Directory Specification and > + * Filesystem Hierarchy Standard. > + */ > + if (getuid() =3D=3D 0) > + directory =3D "/var/run"; > + else > + directory =3D getenv("XDG_RUNTIME_DIR") ? : "/tmp= "; > + } > + > + ret =3D snprintf(vduse_reconnect_dir, sizeof(vduse_reconnect_dir)= , "%s/dpdk/vduse", > + directory); > + if (ret < 0 || ret =3D=3D sizeof(vduse_reconnect_dir)) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating VDUSE reco= nnect path name"); > + return -1; > + } > + > + ret =3D mkdir(vduse_reconnect_dir, 0700); > + if (ret < 0 && errno !=3D EEXIST) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating '%s': %s", > + vduse_reconnect_dir, strerror(errno)); > + return -1; > + } > + > + VHOST_CONFIG_LOG("vduse", INFO, "Created VDUSE reconnect director= y in %s", > + vduse_reconnect_dir); > + > + return 0; > +} > + > +static void > +vduse_reconnect_handler(int fd, void *arg, int *remove) > +{ > + struct virtio_net *dev =3D arg; > + > + vduse_device_start(dev, true); > + > + close(fd); > + *remove =3D 1; > +} > + > int > vduse_device_create(const char *path, bool compliant_ol_flags) > { > - int control_fd, dev_fd, vid, ret; > + int control_fd, dev_fd, vid, ret, reco_fd; > uint32_t i, max_queue_pairs, total_queues; > struct virtio_net *dev; > struct virtio_net_config vnet_config =3D {{ 0 }}; > @@ -418,6 +486,9 @@ vduse_device_create(const char *path, bool compliant_= ol_flags) > uint64_t features; > struct vduse_dev_config *dev_config =3D NULL; > const char *name =3D path + strlen("/dev/vduse/"); > + char reconnect_file[PATH_MAX]; > + struct vhost_reconnect_data *reconnect_log =3D NULL; > + bool reconnect =3D false; >=20 > if (vduse.fdset =3D=3D NULL) { > vduse.fdset =3D fdset_init("vduse-evt"); > @@ -427,6 +498,20 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > } > } >=20 > + if (vduse_reconnect_path_set =3D=3D false) { > + if (vduse_reconnect_path_init() < 0) { > + VHOST_CONFIG_LOG(path, ERR, "failed to initialize= reconnect path"); > + return -1; > + } > + vduse_reconnect_path_set =3D true; > + } > + > + ret =3D snprintf(reconnect_file, sizeof(reconnect_file), "%s/%s",= vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create vduse recon= nect path name"); > + return -1; > + } > + > control_fd =3D open(VDUSE_CTRL_PATH, O_RDWR); > if (control_fd < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to open %s: %s", > @@ -441,14 +526,6 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_ctrl_close; > } >=20 > - dev_config =3D malloc(offsetof(struct vduse_dev_config, config) + > - sizeof(vnet_config)); > - if (!dev_config) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to allocate VDUSE con= fig"); > - ret =3D -1; > - goto out_ctrl_close; > - } > - > ret =3D rte_vhost_driver_get_features(path, &features); > if (ret < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to get backend feature= s"); > @@ -469,23 +546,97 @@ vduse_device_create(const char *path, bool complian= t_ol_flags) > else > total_queues +=3D 1; /* Includes ctrl queue */ >=20 > - vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > - memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + if (access(path, F_OK) =3D=3D 0) { > + VHOST_CONFIG_LOG(name, INFO, "Device already exists, reco= nnecting..."); > + reconnect =3D true; > + > + reco_fd =3D open(reconnect_file, O_RDWR, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D ENOENT) { > + VHOST_CONFIG_LOG(name, ERR, "Missing reco= nnect file (%s)", > + reconnect_file); > + } else { > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + } Seems no {} is needed for if-else? > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } > + } else { > + reco_fd =3D open(reconnect_file, O_CREAT | O_EXCL | O_RDW= R, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D EEXIST) { > + VHOST_CONFIG_LOG(name, ERR, "Reconnect fi= le %s exists but not the device", > + reconnect_file); > + } else { > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + } > + ret =3D -1; > + goto out_ctrl_close; > + } >=20 > - strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > - dev_config->device_id =3D VIRTIO_ID_NET; > - dev_config->vendor_id =3D 0; > - dev_config->features =3D features; > - dev_config->vq_num =3D total_queues; > - dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > - dev_config->config_size =3D sizeof(struct virtio_net_config); > - memcpy(dev_config->config, &vnet_config, sizeof(vnet_config)); > + ret =3D ftruncate(reco_fd, sizeof(*reconnect_log)); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to truncate r= econnect file %s (%s)", > + reconnect_file, strerror(errno)); > + close(reco_fd); > + goto out_ctrl_close; > + } >=20 > - ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > - if (ret < 0) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to create VDUSE devic= e: %s", > - strerror(errno)); > - goto out_free; > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + reconnect_log->version =3D 0; > + > + dev_config =3D malloc(offsetof(struct vduse_dev_config, c= onfig) + > + sizeof(vnet_config)); > + if (!dev_config) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to allocate V= DUSE config"); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > + memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + > + rte_strscpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > + dev_config->device_id =3D VIRTIO_ID_NET; > + dev_config->vendor_id =3D 0; > + dev_config->features =3D features; > + dev_config->vq_num =3D total_queues; > + dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > + dev_config->config_size =3D sizeof(struct virtio_net_conf= ig); > + memcpy(dev_config->config, &vnet_config, sizeof(vnet_conf= ig)); > + > + ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create VDU= SE device: %s", > + strerror(errno)); > + goto out_free; > + } > + > + memcpy(&reconnect_log->config, &vnet_config, sizeof(vnet_= config)); > + reconnect_log->nr_vrings =3D total_queues; > + free(dev_config); > + dev_config =3D NULL; > } >=20 > dev_fd =3D open(path, O_RDWR); > @@ -519,10 +670,15 @@ vduse_device_create(const char *path, bool complian= t_ol_flags) > strncpy(dev->ifname, path, IF_NAME_SZ - 1); > dev->vduse_ctrl_fd =3D control_fd; > dev->vduse_dev_fd =3D dev_fd; > + dev->reconnect_log =3D reconnect_log; > + if (reconnect) > + dev->status =3D dev->reconnect_log->status; > + > vhost_setup_virtio_net(dev->vid, true, compliant_ol_flags, true, t= rue); >=20 > for (i =3D 0; i < total_queues; i++) { > struct vduse_vq_config vq_cfg =3D { 0 }; > + struct vhost_virtqueue *vq; >=20 > ret =3D alloc_vring_queue(dev, i); > if (ret) { > @@ -530,6 +686,12 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > + vq =3D dev->virtqueue[i]; > + vq->reconnect_log =3D &reconnect_log->vring[i]; > + > + if (reconnect) > + continue; > + > vq_cfg.index =3D i; > vq_cfg.max_size =3D 1024; >=20 > @@ -549,7 +711,28 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > - free(dev_config); > + if (reconnect && dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > + reco_fd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > + if (reco_fd < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create rec= o_fd: %s", > + strerror(errno)); > + ret =3D -1; > + goto out_dev_destroy; > + } > + > + ret =3D fdset_add(vduse.fdset, reco_fd, vduse_reconnect_h= andler, NULL, dev); > + if (ret) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to add fd %d = to vduse fdset", > + dev->vduse_dev_fd); Should print reco_fd > + goto out_dev_destroy; > + } > + > + ret =3D eventfd_write(reco_fd, (eventfd_t)1); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to write to r= econnect eventfd"); > + goto out_dev_destroy; > + } > + } Maybe I missed something, why we need to implement like this instead of dir= ectly call vduse_device_start? Thanks, Chenbo >=20 > return 0; >=20 > @@ -587,6 +770,9 @@ vduse_device_destroy(const char *path) > if (vid =3D=3D RTE_MAX_VHOST_DEVICE) > return -1; >=20 > + if (dev->reconnect_log) > + munmap(dev->reconnect_log, sizeof(*dev->reconnect_log)); > + > vduse_device_stop(dev); >=20 > fdset_del(vduse.fdset, dev->vduse_dev_fd); > @@ -597,10 +783,26 @@ vduse_device_destroy(const char *path) > } >=20 > if (dev->vduse_ctrl_fd >=3D 0) { > + char reconnect_file[PATH_MAX]; > + > ret =3D ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name)= ; > - if (ret) > + if (ret) { > VHOST_CONFIG_LOG(name, ERR, "Failed to destroy VDU= SE device: %s", > strerror(errno)); > + } else { > + /* > + * VDUSE device was no more attached to the vDPA = bus, > + * so we can remove the reconnect file. > + */ > + ret =3D snprintf(reconnect_file, sizeof(reconnect= _file), "%s/%s", > + vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) > + VHOST_CONFIG_LOG(name, ERR, > + "Failed to create vduse r= econnect path name"); > + else > + unlink(reconnect_file); > + } > + > close(dev->vduse_ctrl_fd); > dev->vduse_ctrl_fd =3D -1; > } > -- > 2.46.0 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C8DBE4591A; Fri, 6 Sep 2024 09:15:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B5F0F42EC6; Fri, 6 Sep 2024 09:15:10 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2064.outbound.protection.outlook.com [40.107.92.64]) by mails.dpdk.org (Postfix) with ESMTP id 9684D42EC4 for ; Fri, 6 Sep 2024 09:15:09 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UwCQ0VKeTlMh7bc/ylDJJgZiRDbkoVSUzvma6gMQe2uM+/2WtnMNqN+IJbmbId3UqXfDlA7HLLyzLnKJ95OzruG1y9t0kmt80d54D+XmOp2xJWRvU52ztJ1v8KgaA9HHdshvy/KwCrgAYp2W8Xid2h4W9e96vCmh8IiGBIzUuv4hz8Aw46sVxm6wq5lYX63F7NwQr741noJFuCzZqungpbEqJ+pb2qGSUEkol0GwM5uXYOpiyH7k9Z552Er8vs5eIiEWmc5gm8/2mHyJYxrFSH5e2sHFqcGwDxjEDJN2rp6lL73QnJVac6Bvgs99KKClr4XVmlMHDss6OydhdInNyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=; b=DLi2Z2Sff4JktiDClsMa8IuZK7LOu2BUEO6TppA0mk4rqZJCqa7q56vguLoBPP8L9QOk688CvI2HtajlclBws1Ezoquu38D/vS2SU6I2DcseRGUqLj85SCOTBzzOdzAAUnnfKUyCG8/Ls/afXxnbcA/GV/A1wUUYcC0Yvs05v47NRADSmLT/ugimfI48TBBKByY/iu5DH/UdE6v2UWemaR9r4zr+MJXJ8hzkUCEZJHpzllqM3B7+U0Oz97qopwPdLkB/0KV4Quxqh3IUinpO/KMTBeuz5xmaeck+HN3vrPK5jyJLBl/tB2PrlwoRrOSwISQ2G/xzgtZC4kWT3E6nkA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=; b=bRVNZ/0CwkW6SR/fG2PpiHfG87Ic9NR3uyQxaFrPUCWnlA+ydqj/eL4XMowLhE0EbjowJ7CgPXXHQ5A15va8D/Qt3q4058llZaxfECbeEgwJx6a6jvOZ4v8KJJm2WqAumijj4HDvzZdONVuHSoSkqH5dzoHYSgXACWy6P/uCJniX5yGEEOxcY4x6Gq82wJ7zFB5OuJQ33CfxYi2PiqrG2oztmlgxD8OriygcDCjr/Iz6YSX/pdbQtWrxFHA2j5LcYsnltxyaDw9XAiM4EsTsHFQYqjRqUQCTSNXLjGUaE8m6r6IWdj+6Le88bpQRR5apxIhx5LDu69NPXz5N7QJpiQ== Received: from CYYPR12MB8992.namprd12.prod.outlook.com (2603:10b6:930:bc::21) by PH0PR12MB7471.namprd12.prod.outlook.com (2603:10b6:510:1e9::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.28; Fri, 6 Sep 2024 07:15:06 +0000 Received: from CYYPR12MB8992.namprd12.prod.outlook.com ([fe80::a722:c2ca:42c7:aaed]) by CYYPR12MB8992.namprd12.prod.outlook.com ([fe80::a722:c2ca:42c7:aaed%7]) with mapi id 15.20.7918.024; Fri, 6 Sep 2024 07:15:06 +0000 From: Chenbo Xia To: Maxime Coquelin CC: "dev@dpdk.org" , "david.marchand@redhat.com" Subject: Re: [PATCH 2/2] vhost: add reconnection support to VDUSE Thread-Topic: [PATCH 2/2] vhost: add reconnection support to VDUSE Thread-Index: AQHa/5+Tb2hkJpD2pkWfDp9/zoprPbJKWaMA Date: Fri, 6 Sep 2024 07:15:06 +0000 Message-ID: <3543971A-6C8E-4F28-A7D4-F8C222F3C193@nvidia.com> References: <20240905142602.1264345-1-maxime.coquelin@redhat.com> <20240905142602.1264345-3-maxime.coquelin@redhat.com> In-Reply-To: <20240905142602.1264345-3-maxime.coquelin@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CYYPR12MB8992:EE_|PH0PR12MB7471:EE_ x-ms-office365-filtering-correlation-id: 62cc0ecb-3afc-4e75-f6ff-08dcce43a295 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|376014|366016|1800799024|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?jOPEweBQUeLeZ3YKN/N99zprZj5fOW5xHJVyUcizMShQkn2dj8aOJWVGpavl?= =?us-ascii?Q?1H8gmn4QRCCvN3wjyu8aJj7vwJ8pJLZctCVk1kvFWsN7RrX6C0DUFcx8dMWh?= =?us-ascii?Q?MnN97fWhXAX4Hbg2MFrAfq2hK6a3Mo1IN3xJN3S5LfHrFU1QL2hfHZIpylJm?= =?us-ascii?Q?kCt5sTfuu5yIhaUxJqKvtqO9iYEUd0H9TBuPi04YqyWxTC0sblnM884OODJw?= =?us-ascii?Q?Q5KEXaKKEtviYj5IBjQS2NZQp28eg7pW4ve3bZhzejIhz6ct63gfVl8hyTAl?= =?us-ascii?Q?CoVgOsqvFdqG3e7NV4gNpPVcz94TmrfzxVI46NxJzb6bsr1ILSdxF6WxJSlF?= =?us-ascii?Q?26In/kv7zud2ZGtihQpqxr2wUovy5cAI2FZQnTEhuGr4tAySBNHSRzD29FwC?= =?us-ascii?Q?qotwsiL3T6ngIen7/1K9p9U6s/k1xRRcAivhx4XMkwR0F8q6P4L7RmDTHkrk?= =?us-ascii?Q?HIEknu7YeGaspIe9qwwgyuhDAPsThnbJljsc5O97iyCBGygcULSuddqgJO7w?= =?us-ascii?Q?nKTQSADL6xj7BkkHr22XRe6emkySvcHJUQ7dpmyZAkuVA30Jbzze7caY4MxH?= =?us-ascii?Q?pbCJ9OhMr1sO7yeW+XY8VfmMlEAlhAC9+mmz1FqfEYFl0sSm2xye7npieJGM?= =?us-ascii?Q?pTiggZGrdM+e/JG8cMssrYLXSz20Q3Go6+B1stOniXqqHG5tGPh5wVGF7DRY?= =?us-ascii?Q?RMPbTEEbENAt+n+afz0CB5xv6RurkfJSu1qdPMDiHHgD4CuFiyf4Bnkudo6y?= =?us-ascii?Q?J0x3ZMCzOfGGneaNjkheXJcmbSn3w4Z9PxrU2m3DVpJ7TnYWyysOHjrCokQU?= =?us-ascii?Q?0+iy9ePQ5jQeiKfT8bE4/8nW8VKKlcw2FZRNr7fWO7c1Kq2FB1T6xn6vSX08?= =?us-ascii?Q?F4oXfIuL1+Ll0g2b78qqhyP0bNGyjS/nf5Tvz+9XeUFi9YPWyN8kbBGoIGc/?= =?us-ascii?Q?4uuqT9610Vkp1OSzVoz803jpFgeHDtjjtQ9c0oeETAnK7rkS01F8LCfqhUX4?= =?us-ascii?Q?gfm8CinEJL+B30Dz2oy5GoMCAX6iyrPH0NjNgCPay7H0z9aajqy/S8ZoIY0L?= =?us-ascii?Q?5KuOekKjkPterJWXSDDQxIA2MSXKmYP2gU4sGsrUcHGZRzmwkNIBEiGBBeYE?= =?us-ascii?Q?cSNxh+y7eXXOggZA1TP6o3MRIVpc7HhHF/2+5Wvs/sLLASsAWMRW5uNIFrX4?= =?us-ascii?Q?R5lFvAmcadqmrGdiDTQwOVVyDw1WL3WGY2WRvCE9HGGgCg51nk8J5S8OgSJ+?= =?us-ascii?Q?erINo/y3kCQNwqE6l3v3Wj4gHfpojcU6dt/eJuvBy8ZYVMYlSxBYtvy2jOEr?= =?us-ascii?Q?o2Iu2wxLyyxjbGQaw6UefYBudfMOrqBwljuz6kIC/bluQVpk8m5vp5k4VnF7?= =?us-ascii?Q?WlmP610MQETmLoA1VvyOXwsVXYg0kuz91zQjLv1JYGynqfxlcw=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CYYPR12MB8992.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(366016)(1800799024)(38070700018); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?im17JPFFvKXies+GaMG0p3ApcU8wSqJNA2N1ybliZAOUdow4FFHYWMdxmCWL?= =?us-ascii?Q?O5mQKn2zZ1qLa6GwyaHPqOFFP1NwFkGSOXGOjhL7wxJXZFfqDAt5JePK+T7e?= =?us-ascii?Q?0x6Zio4gxPn3vRPJ/lVNYXl5CVgiwoh7uQJAWnq74LszKICu+752OAliFgFF?= =?us-ascii?Q?lVWiz1Ts7/NAILshTSsYHXiruuU+YMR+tAt61MuDRVt3UBmY6IFoMxLTkara?= =?us-ascii?Q?2XG9n6d/c1ONdqovzcwkDzzXLm7CeWsHyBmWEMwuOXIWBl/1JkTUJqti6iJ9?= =?us-ascii?Q?QzjtH6JxGWfGPPehXZy+IVm7B9Q/wiF3DRN9cIWQeKSIQJQQx1hLmlE+4A4b?= =?us-ascii?Q?2eXExWOO3Nnk4d2HZNHwgM9TqMpcHq7xO9HRSsj2mHBHSw47aQA0YcesoCjn?= =?us-ascii?Q?HezZIMR/7inXxu2z4qZyu9lt2cZffBRYkSupyjhg//KEitIW/udy9BxwutSW?= =?us-ascii?Q?TTEzYlbinmkYYmAItZ7/2ED23tuCrMV2hapu1m217/wNnwyFfuDO5i3HU4QP?= =?us-ascii?Q?Gilm90Mk+4dIGnVgfYkPd5ZCMUVQGgfSV4m4eHfksd+oHZkW7wQz+GDIfGSD?= =?us-ascii?Q?tg5tduccCJ2QqffjCA0+/+G7W+d8soorl26r1zdQMFX88HT+eojT/jd+3fZn?= =?us-ascii?Q?pYSIZwv6uc1fnSzoorBBfe6d8HyXWZBmIHkaBjldZUMwM48Yf2vxH1Em09Bb?= =?us-ascii?Q?+Yl/tGkt2D75Q3tKKN57yfhAcQrpkqvBpRXf6nxzUHm7A6PCSN5tOjqZCfEM?= =?us-ascii?Q?CXvJuBkcnIgX1Cy0gR1qnOc6+xgiJKyJNWV0ifVk+KQPYj83TYRZZSPl9cS+?= =?us-ascii?Q?nOfvHT6HwG4KK/DUz9EK4IJV484xJvSO720Qs/IIrFJxngi7da1AZIANem5r?= =?us-ascii?Q?lH5XG49D3SDK4QyuB9y8ldUFl5/KbIlBQFRpsT2OhzN7DHSkZX80kz586Hrj?= =?us-ascii?Q?JsVbcMsrMhy88YsB21ZHCLGl9WNdwqMuVPjrYMYYEf0si677SInr9qh4QcW/?= =?us-ascii?Q?PAfvRTK5HnELphhapbXqn7Hlne8ENvQzXyFvRAJr7ZQw5JEw7DiLEAA3HXOK?= =?us-ascii?Q?kw+AC993HeC0nzdreZ9wNziUnNiJHFE2FzYUgN14hzfOmYpcNhyzXDMipsKI?= =?us-ascii?Q?GRCVhFwLfyTaFCoYEHVwvze4u8R0feHrPN6khZ0BziEnwMDLPqyiCAhSjYwp?= =?us-ascii?Q?nvc3oeH/MUcnBSaZFu1tBOQDfp0SW0XJEp4RnfXs46fNLk15VXCQDCcyGu1N?= =?us-ascii?Q?5g9Z4JX2/vjSYxRrWShNsiN8qKLKu9g8/1re9Sr2kKAo6gY4udzn0GsWQ6qc?= =?us-ascii?Q?NGVSEmBj7lYNeraOH1Ff2vibRuiPyQcfj/jmJc/GRdWPFHg7mCkcaDAp4m2x?= =?us-ascii?Q?RbvxhlGlz5DPJ14zbOVaoSmwgiUO2MfaBRhtijRWDKou6F8u8dZbAbkPOcgK?= =?us-ascii?Q?I4+yCJUqh7wOA/0hKCzQmomIJydnMVoEa+cZBltkznUXGRnERqOAVVD3ziRa?= =?us-ascii?Q?v1gOqji+ouj7yayjmQpY2b+C8qUlf56b/k61coGEmGWvgJ7541yhebOlrdMp?= =?us-ascii?Q?5rliWYK3VUgztJS4OCijEjJHX7Za0jY03wBq9AWhwxcUDcDhc2DgzB81zX8O?= =?us-ascii?Q?Og=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CYYPR12MB8992.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 62cc0ecb-3afc-4e75-f6ff-08dcce43a295 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2024 07:15:06.5533 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: bxdSsuYWXbzi3vvTT7Xj+de7dh/eBJZ0gTUB40YDF6oxWSxAB5Ir73qoBr38G4ksUvn0nvvpNxqvyt8V4u91eg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7471 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Message-ID: <20240906071506.8d2Z9tzv5a1g9gKjZjs_i4VvFRq5KCUHgF0Gd_zjpMI@z> Hi Maxime, > On Sep 5, 2024, at 22:26, Maxime Coquelin wr= ote: >=20 > External email: Use caution opening links or attachments >=20 >=20 > This patch enables VDUSE reconnection support making use of > the newly introduced reconnection mechanism in Vhost > library. >=20 > At DPDK VDUSE device creation time, there are two > possibilities: > 1. The Kernel VDUSE device does not exist: > a. A reconnection file named after the VUDSE device name > is created in VDUSE tmpfs. > b. The file is truncated to 'struct vhost_reconnect_data' > size, and mmapped. > c. Negotiated features, Virtio status... are saved for > sanity checks at reconnect time. > 2. The Kernel VDUSE device already exists: > a. Exit with failure if no reconnect file exists for > this device. > b. Open and mmap the reconnect file. > c. Perform sanity check to ensure features are compatible. > d. Restore virtqueues' available indexes at startup time. >=20 > Then at runtime, the virtqueues' available index are logged by > the Vhost reconnection mechanism. >=20 > At DPDK VDUSE device destruction time, there are two > possibilities: > 1. The Kernel VDUSE device destruction succeed, which > means it is no more attached to the vDPA bus. The > reconnection file is unmapped and then removed. > 2. The Kernel VDUSE device destruction failed, meaning it > is no more attached to the vDPA bus. The reconnection > file is unmapped but not removed to make possible later > reconnection. >=20 > Signed-off-by: Maxime Coquelin > --- > lib/vhost/vduse.c | 280 +++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 241 insertions(+), 39 deletions(-) >=20 > diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c > index c66602905c..bd0e492d62 100644 > --- a/lib/vhost/vduse.c > +++ b/lib/vhost/vduse.c > @@ -136,7 +136,7 @@ vduse_control_queue_event(int fd, void *arg, int *rem= ove __rte_unused) > } >=20 > static void > -vduse_vring_setup(struct virtio_net *dev, unsigned int index) > +vduse_vring_setup(struct virtio_net *dev, unsigned int index, bool recon= nect) > { > struct vhost_virtqueue *vq =3D dev->virtqueue[index]; > struct vhost_vring_addr *ra =3D &vq->ring_addrs; > @@ -152,6 +152,19 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > return; > } >=20 > + if (reconnect) { > + vq->last_avail_idx =3D vq->reconnect_log->last_avail_idx; > + vq->last_used_idx =3D vq->reconnect_log->last_avail_idx; > + } else { > + vq->last_avail_idx =3D vq_info.split.avail_index; > + vq->last_used_idx =3D vq_info.split.avail_index; > + } > + vq->size =3D vq_info.num; > + vq->ready =3D true; > + vq->enabled =3D vq_info.ready; > + ra->desc_user_addr =3D vq_info.desc_addr; > + ra->avail_user_addr =3D vq_info.driver_addr; > + ra->used_user_addr =3D vq_info.device_addr; > VHOST_CONFIG_LOG(dev->ifname, INFO, "VQ %u info:", index); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tnum: %u", vq_info.num); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdesc_addr: %llx", > @@ -162,15 +175,6 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > (unsigned long long)vq_info.device_addr); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq_info.spl= it.avail_index); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tready: %u", vq_info.ready); > - > - vq->last_avail_idx =3D vq_info.split.avail_index; > - vq->size =3D vq_info.num; > - vq->ready =3D true; > - vq->enabled =3D vq_info.ready; > - ra->desc_user_addr =3D vq_info.desc_addr; > - ra->avail_user_addr =3D vq_info.driver_addr; > - ra->used_user_addr =3D vq_info.device_addr; > - > vq->kickfd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > if (vq->kickfd < 0) { > VHOST_CONFIG_LOG(dev->ifname, ERR, "Failed to init kickfd = for VQ %u: %s", > @@ -267,7 +271,7 @@ vduse_vring_cleanup(struct virtio_net *dev, unsigned = int index) > } >=20 > static void > -vduse_device_start(struct virtio_net *dev) > +vduse_device_start(struct virtio_net *dev, bool reconnect) > { > unsigned int i, ret; >=20 > @@ -287,6 +291,15 @@ vduse_device_start(struct virtio_net *dev) > return; > } >=20 > + if (reconnect && dev->features !=3D dev->reconnect_log->features)= { > + VHOST_CONFIG_LOG(dev->ifname, ERR, > + "Mismatch between reconnect file features= 0x%" PRIx64 " & device features 0x%" PRIx64, Checkpatch reports long line > + dev->reconnect_log->features, dev->featur= es); > + return; > + } > + > + dev->reconnect_log->features =3D dev->features; > + > VHOST_CONFIG_LOG(dev->ifname, INFO, "Negotiated Virtio features: 0= x%" PRIx64, > dev->features); >=20 > @@ -300,7 +313,7 @@ vduse_device_start(struct virtio_net *dev) > } >=20 > for (i =3D 0; i < dev->nr_vring; i++) > - vduse_vring_setup(dev, i); > + vduse_vring_setup(dev, i, reconnect); >=20 > dev->flags |=3D VIRTIO_DEV_READY; >=20 > @@ -373,6 +386,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) > req.s.status); > old_status =3D dev->status; > dev->status =3D req.s.status; > + dev->reconnect_log->status =3D dev->status; > resp.result =3D VDUSE_REQ_RESULT_OK; > break; > case VDUSE_UPDATE_IOTLB: > @@ -398,7 +412,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) >=20 > if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) > - vduse_device_start(dev); > + vduse_device_start(dev, false); > else > vduse_device_stop(dev); > } > @@ -407,10 +421,64 @@ vduse_events_handler(int fd, void *arg, int *remove= __rte_unused) > vduse_req_id_to_str(req.type), req.type); > } >=20 > +static char vduse_reconnect_dir[PATH_MAX]; > +static bool vduse_reconnect_path_set; > + > +static int > +vduse_reconnect_path_init(void) > +{ > + const char *directory; > + int ret; > + > + /* from RuntimeDirectory=3D see systemd.exec */ > + directory =3D getenv("RUNTIME_DIRECTORY"); > + if (directory =3D=3D NULL) { > + /* > + * Used standard convention defined in > + * XDG Base Directory Specification and > + * Filesystem Hierarchy Standard. > + */ > + if (getuid() =3D=3D 0) > + directory =3D "/var/run"; > + else > + directory =3D getenv("XDG_RUNTIME_DIR") ? : "/tmp= "; > + } > + > + ret =3D snprintf(vduse_reconnect_dir, sizeof(vduse_reconnect_dir)= , "%s/dpdk/vduse", > + directory); > + if (ret < 0 || ret =3D=3D sizeof(vduse_reconnect_dir)) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating VDUSE reco= nnect path name"); > + return -1; > + } > + > + ret =3D mkdir(vduse_reconnect_dir, 0700); > + if (ret < 0 && errno !=3D EEXIST) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating '%s': %s", > + vduse_reconnect_dir, strerror(errno)); > + return -1; > + } > + > + VHOST_CONFIG_LOG("vduse", INFO, "Created VDUSE reconnect director= y in %s", > + vduse_reconnect_dir); > + > + return 0; > +} > + > +static void > +vduse_reconnect_handler(int fd, void *arg, int *remove) > +{ > + struct virtio_net *dev =3D arg; > + > + vduse_device_start(dev, true); > + > + close(fd); > + *remove =3D 1; > +} > + > int > vduse_device_create(const char *path, bool compliant_ol_flags) > { > - int control_fd, dev_fd, vid, ret; > + int control_fd, dev_fd, vid, ret, reco_fd; > uint32_t i, max_queue_pairs, total_queues; > struct virtio_net *dev; > struct virtio_net_config vnet_config =3D {{ 0 }}; > @@ -418,6 +486,9 @@ vduse_device_create(const char *path, bool compliant_= ol_flags) > uint64_t features; > struct vduse_dev_config *dev_config =3D NULL; > const char *name =3D path + strlen("/dev/vduse/"); > + char reconnect_file[PATH_MAX]; > + struct vhost_reconnect_data *reconnect_log =3D NULL; > + bool reconnect =3D false; >=20 > if (vduse.fdset =3D=3D NULL) { > vduse.fdset =3D fdset_init("vduse-evt"); > @@ -427,6 +498,20 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > } > } >=20 > + if (vduse_reconnect_path_set =3D=3D false) { > + if (vduse_reconnect_path_init() < 0) { > + VHOST_CONFIG_LOG(path, ERR, "failed to initialize= reconnect path"); > + return -1; > + } > + vduse_reconnect_path_set =3D true; > + } > + > + ret =3D snprintf(reconnect_file, sizeof(reconnect_file), "%s/%s",= vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create vduse recon= nect path name"); > + return -1; > + } > + > control_fd =3D open(VDUSE_CTRL_PATH, O_RDWR); > if (control_fd < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to open %s: %s", > @@ -441,14 +526,6 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_ctrl_close; > } >=20 > - dev_config =3D malloc(offsetof(struct vduse_dev_config, config) + > - sizeof(vnet_config)); > - if (!dev_config) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to allocate VDUSE con= fig"); > - ret =3D -1; > - goto out_ctrl_close; > - } > - > ret =3D rte_vhost_driver_get_features(path, &features); > if (ret < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to get backend feature= s"); > @@ -469,23 +546,97 @@ vduse_device_create(const char *path, bool complian= t_ol_flags) > else > total_queues +=3D 1; /* Includes ctrl queue */ >=20 > - vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > - memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + if (access(path, F_OK) =3D=3D 0) { > + VHOST_CONFIG_LOG(name, INFO, "Device already exists, reco= nnecting..."); > + reconnect =3D true; > + > + reco_fd =3D open(reconnect_file, O_RDWR, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D ENOENT) { > + VHOST_CONFIG_LOG(name, ERR, "Missing reco= nnect file (%s)", > + reconnect_file); > + } else { > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + } Seems no {} is needed for if-else? > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } > + } else { > + reco_fd =3D open(reconnect_file, O_CREAT | O_EXCL | O_RDW= R, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D EEXIST) { > + VHOST_CONFIG_LOG(name, ERR, "Reconnect fi= le %s exists but not the device", > + reconnect_file); > + } else { > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + } > + ret =3D -1; > + goto out_ctrl_close; > + } >=20 > - strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > - dev_config->device_id =3D VIRTIO_ID_NET; > - dev_config->vendor_id =3D 0; > - dev_config->features =3D features; > - dev_config->vq_num =3D total_queues; > - dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > - dev_config->config_size =3D sizeof(struct virtio_net_config); > - memcpy(dev_config->config, &vnet_config, sizeof(vnet_config)); > + ret =3D ftruncate(reco_fd, sizeof(*reconnect_log)); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to truncate r= econnect file %s (%s)", > + reconnect_file, strerror(errno)); > + close(reco_fd); > + goto out_ctrl_close; > + } >=20 > - ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > - if (ret < 0) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to create VDUSE devic= e: %s", > - strerror(errno)); > - goto out_free; > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + reconnect_log->version =3D 0; > + > + dev_config =3D malloc(offsetof(struct vduse_dev_config, c= onfig) + > + sizeof(vnet_config)); > + if (!dev_config) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to allocate V= DUSE config"); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > + memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + > + rte_strscpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > + dev_config->device_id =3D VIRTIO_ID_NET; > + dev_config->vendor_id =3D 0; > + dev_config->features =3D features; > + dev_config->vq_num =3D total_queues; > + dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > + dev_config->config_size =3D sizeof(struct virtio_net_conf= ig); > + memcpy(dev_config->config, &vnet_config, sizeof(vnet_conf= ig)); > + > + ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create VDU= SE device: %s", > + strerror(errno)); > + goto out_free; > + } > + > + memcpy(&reconnect_log->config, &vnet_config, sizeof(vnet_= config)); > + reconnect_log->nr_vrings =3D total_queues; > + free(dev_config); > + dev_config =3D NULL; > } >=20 > dev_fd =3D open(path, O_RDWR); > @@ -519,10 +670,15 @@ vduse_device_create(const char *path, bool complian= t_ol_flags) > strncpy(dev->ifname, path, IF_NAME_SZ - 1); > dev->vduse_ctrl_fd =3D control_fd; > dev->vduse_dev_fd =3D dev_fd; > + dev->reconnect_log =3D reconnect_log; > + if (reconnect) > + dev->status =3D dev->reconnect_log->status; > + > vhost_setup_virtio_net(dev->vid, true, compliant_ol_flags, true, t= rue); >=20 > for (i =3D 0; i < total_queues; i++) { > struct vduse_vq_config vq_cfg =3D { 0 }; > + struct vhost_virtqueue *vq; >=20 > ret =3D alloc_vring_queue(dev, i); > if (ret) { > @@ -530,6 +686,12 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > + vq =3D dev->virtqueue[i]; > + vq->reconnect_log =3D &reconnect_log->vring[i]; > + > + if (reconnect) > + continue; > + > vq_cfg.index =3D i; > vq_cfg.max_size =3D 1024; >=20 > @@ -549,7 +711,28 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > - free(dev_config); > + if (reconnect && dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > + reco_fd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > + if (reco_fd < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create rec= o_fd: %s", > + strerror(errno)); > + ret =3D -1; > + goto out_dev_destroy; > + } > + > + ret =3D fdset_add(vduse.fdset, reco_fd, vduse_reconnect_h= andler, NULL, dev); > + if (ret) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to add fd %d = to vduse fdset", > + dev->vduse_dev_fd); Should print reco_fd > + goto out_dev_destroy; > + } > + > + ret =3D eventfd_write(reco_fd, (eventfd_t)1); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to write to r= econnect eventfd"); > + goto out_dev_destroy; > + } > + } Maybe I missed something, why we need to implement like this instead of dir= ectly call vduse_device_start? Thanks, Chenbo >=20 > return 0; >=20 > @@ -587,6 +770,9 @@ vduse_device_destroy(const char *path) > if (vid =3D=3D RTE_MAX_VHOST_DEVICE) > return -1; >=20 > + if (dev->reconnect_log) > + munmap(dev->reconnect_log, sizeof(*dev->reconnect_log)); > + > vduse_device_stop(dev); >=20 > fdset_del(vduse.fdset, dev->vduse_dev_fd); > @@ -597,10 +783,26 @@ vduse_device_destroy(const char *path) > } >=20 > if (dev->vduse_ctrl_fd >=3D 0) { > + char reconnect_file[PATH_MAX]; > + > ret =3D ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name)= ; > - if (ret) > + if (ret) { > VHOST_CONFIG_LOG(name, ERR, "Failed to destroy VDU= SE device: %s", > strerror(errno)); > + } else { > + /* > + * VDUSE device was no more attached to the vDPA = bus, > + * so we can remove the reconnect file. > + */ > + ret =3D snprintf(reconnect_file, sizeof(reconnect= _file), "%s/%s", > + vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) > + VHOST_CONFIG_LOG(name, ERR, > + "Failed to create vduse r= econnect path name"); > + else > + unlink(reconnect_file); > + } > + > close(dev->vduse_ctrl_fd); > dev->vduse_ctrl_fd =3D -1; > } > -- > 2.46.0 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A4E3E4591C; Fri, 6 Sep 2024 11:20:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 54D8142E12; Fri, 6 Sep 2024 11:20:29 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2043.outbound.protection.outlook.com [40.107.223.43]) by mails.dpdk.org (Postfix) with ESMTP id 5E5C64029E for ; Fri, 6 Sep 2024 11:20:27 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=F1Dy58HVifXb9l4uvT1f6R1e61WZVYaOdVoOBlcEeHVFf6vP3VCliXsC+dCPUDr+E/Y9Pq+xmi/89p5go+9CG3R4iVnRjOHuBRJFi1cXbgkXARtoP90EjuAfplijqXtrNqyFC8+XBIvdqSgouLEJfMDG2IwEFje+JTXzdZaTaIHb2VO+lWzWY8g8xjSOnGbiCGVYbizfgIImglTsg4kMNu5xbr5p8q8xwvrYQdCb6f6O1V/AWk+L/Z5D9jsGAHvvllJnTmHLRdenBV44B8MIWGYO9lIDHbQUxAmUSOksvNyVk0K04WN4q3oZNe0VzuQwhdFlUGOMTXKW9urWwX4yuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=; b=N8hnwPzDJKmm59Y0DAn8hQ+zeMotGiCptwbVTsUHGulq05UeHFM9Go3MM7dAakMJcj9aotDW3DTh1mjuWctdMWaVHkf4lAnH+xOCP3+NC+QybHjHTx4MHhzB/NoCdeY8JndI1CA4xK+DX8R4thKvCF0CphRN1zBfaRXXPXSbxj0078+M1SSSxyrbmX/XFeFdmeseBkwO358X2AyFyWFKKXF/cJ97AajViMZnDoqEPZCSLKBLd4GirW4ySHgCd5NPhgeqiYVI/jRZARwEnNJkNICFWz8ffSAXDwmo7ypexRwhWvR8gE+6kq5N8vCmGV2ocSpxULwb8EYwWFUR40Txuw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=; b=E4paWSStWcYvZT6cxq5s4LFZR1vJhCwnkDDjuFm2bFDZyYVl0c4i0KlZQ6JdcN9yGI+GQPf1GLr+SaBU/t8ru80VBDjUlYMV4iGgXtsPqAHXQ2JXuLHQs9B28olJW/lrxxWw5x6T/M2fMx0a8g8cxwCJBsrXTeyLEpxk8c7tiddT1k511diF7dfVX2WfsNCx+t79wmxT6xNaa0FU7GKbbi/ruYzZ4dbWVH81052QELbL4VfIHe0M/sxvKf6sVXzEMbR0rk+RxLAZDwLkBiuHdXMa9uldXgeN/UzVS1FQO+wjDaTClOiiuEVtMtr5GiKJTN4R1magbAq/ZX+KFXC24A== Received: from IA0PR12MB8983.namprd12.prod.outlook.com (2603:10b6:208:490::17) by MW4PR12MB6974.namprd12.prod.outlook.com (2603:10b6:303:207::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.17; Fri, 6 Sep 2024 09:20:24 +0000 Received: from IA0PR12MB8983.namprd12.prod.outlook.com ([fe80::68e7:8370:b169:fa0e]) by IA0PR12MB8983.namprd12.prod.outlook.com ([fe80::68e7:8370:b169:fa0e%6]) with mapi id 15.20.7939.017; Fri, 6 Sep 2024 09:20:22 +0000 From: Chenbo Xia To: Maxime Coquelin CC: "dev@dpdk.org" , "david.marchand@redhat.com" Subject: Re: [PATCH 2/2] vhost: add reconnection support to VDUSE Thread-Topic: [PATCH 2/2] vhost: add reconnection support to VDUSE Thread-Index: AQHa/5+Tb2hkJpD2pkWfDp9/zoprPbJKWaMA Date: Fri, 6 Sep 2024 09:20:22 +0000 Message-ID: <3543971A-6C8E-4F28-A7D4-F8C222F3C193@nvidia.com> References: <20240905142602.1264345-1-maxime.coquelin@redhat.com> <20240905142602.1264345-3-maxime.coquelin@redhat.com> In-Reply-To: <20240905142602.1264345-3-maxime.coquelin@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA0PR12MB8983:EE_|MW4PR12MB6974:EE_ x-ms-office365-filtering-correlation-id: 09b5c1d9-9a7f-4eed-0827-08dcce552246 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|1800799024|376014|366016|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?cvDLqQJ9mQPn8WMw5Mb+73qUM2ts9o+aGGp/3344vS+6tgo85PjB4xLscVr6?= =?us-ascii?Q?L1jH7tRmr0Srz7UenYd8G+PuZydzRMH5XG2WDpifc6zP0w5/7ex9lH3HOAHt?= =?us-ascii?Q?jOANCL3ocvawamBtijrPvhanzU+mAIUMt3BIc0/MiY6JHdaAmCycdNmYF86X?= =?us-ascii?Q?4y6pwWPjtCT9/bDFE6v0VLe9+yoEPtV0enNUYY53ZICVv7uhvkxqQ5LGC0nO?= =?us-ascii?Q?svVyEdT8T+S4bitlSl7lLKGShqLobfAteAkjI6CQ4ARp7muImnMwW8dIjiIu?= =?us-ascii?Q?SiKJZCaER31lFdNlmUUslF2CFFY/c3Bmu4OS6urnSv6nqvPYbJu0ZkU6shlw?= =?us-ascii?Q?eCnzKG3rpN95Hv2FndRu+yl/0kXcDPR4qCTyFh/mr6a1JnCilRiboQIs6RjR?= =?us-ascii?Q?2RsQeMccAGuZQSr5ApYelhWeUjqLHSZYNQcL9dRwMLmSxadigaTNjPewIiA9?= =?us-ascii?Q?dUJamyUoa17RfiaBUnIaUp2f+uO2Cs2wuHSUh7lxkNCI3f/XPczhfa4tiWsd?= =?us-ascii?Q?UZPF4LSO2A9SwI2J60WVsmJ+Bbi5335qpVlJbJXsEyT7hlR6nPNh0Ed+6wJX?= =?us-ascii?Q?O9ybcTcbfcoiHbTNpe8fadW3ld/QugtV+q4fajuIf/tpAmo8uCUTKf0klJMd?= =?us-ascii?Q?bmzhl7I3CzGy8Stu7azOJHJ6oCSSEQ/O8Mr4H+K7YpCo0sRfDHFFIv3YFLxO?= =?us-ascii?Q?JQN4r3lRBli6i+09rllRNLZ6tHmZNkgvp/Hp7pxTp2rNq1+vztNpFjK6M6eB?= =?us-ascii?Q?eCQKbkz6G5WoQz0kgVGSbEbk9DkNShnb/fzoRpcl+SampHoue+FVG+K41rIk?= =?us-ascii?Q?J8u0FaFGK82sdzghN/uMLthah+Gb0RspkXjsHGVgk1Ms4W3+qz5jjr2vPbWa?= =?us-ascii?Q?XBQ/9pDTbRyts5qlwiErzHnGX5LSclwLV7pyGVhT5sjgYwr+3CNaa7kyR0EA?= =?us-ascii?Q?KqbzJg+SBCmiNfL/zEYLr4z6qMP1z1SnYGE2ciEauh/v0BYUTEzStXVeTjAV?= =?us-ascii?Q?ayC9CiOuvf6c509+vQkDIq71lXK0mIApV5SRTgPPQ8rxMG9FeeRRrSdmsRmG?= =?us-ascii?Q?O+sHlkFBK2SgUCCBbzzMX4boonlJmxU4cFh2S1I3BOF5+f7ZakIEV9QsPskH?= =?us-ascii?Q?3n+jMZNkIKEGZHObp+MLqP4KeBeY2JcOpu1QMzT0/FtX/JYevjUTJ9o4Dzck?= =?us-ascii?Q?im618EbMWp7O3LGy/ixrfBASkHVVde4Jnv3zaWmfkfZ10YXxIdq/O0josv+1?= =?us-ascii?Q?yBDn81UUnBshgwRG1tw7Bw7RHkBh6cd0+5hHnRAZX+paViurBX8ISA8ADD25?= =?us-ascii?Q?O6VhVxMn9aPVX8L/i8au6+GDkL7/FVcWZvKUXVnb9uOF0/KCzEt8Y12wYzu9?= =?us-ascii?Q?xW0lwYGxhge9vUDBjDSBRgAbHzV9OWPTA0rqSGfExua+hFx2Ng=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA0PR12MB8983.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(376014)(366016)(38070700018); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LR+XRBUxjUbu0dWkQ7uognYYojvloSJy0xmbWX4M4uwuYLIHxOerfkEuMsi5?= =?us-ascii?Q?0UH6RFbFRF7XMRpma6YuRaBlmhFSGH5TJ/0kcTICw4cOX6Qq1hSwBk4Cm/20?= =?us-ascii?Q?CjUkqxAW9KrwK6SeYDdXv8tHAS3eRCc5yKXsZ02Fhty7vqJ7EM/IQCzGb/Bs?= =?us-ascii?Q?XC/8j5+0S9NA4gvBc8kPs76FEyOIVAm5dggh7N9n1LsUfcpWSBSH4zW0YDgU?= =?us-ascii?Q?YSCBH0onqcHAwe4jYpvQh5Qlv5PTwfDVZ7VVNAhqkjkUlDuZESSLMiCtEV7o?= =?us-ascii?Q?0Qf2bzbm1PE+VsaYc1GYsPmji4qLbUfSVkk4rz0S6uFeEppihYg+PzjLqMFF?= =?us-ascii?Q?f+Vb8UoBWyg3qyJ/ouKt/n4eRqUp0eQ9RWr6Hs2OsA+N8P1xwjnUxzfByRdh?= =?us-ascii?Q?CWx50a/xMg3suaa7/gLDiSRTJTsTTVbXAYRadX8cmGWk6+NAZMFdnYSjaHMG?= =?us-ascii?Q?7iSkLFuKoQKKqirc98cFwgmVnj6BHBHccjjx/2aErGt2wHhFpokutX5zrDwb?= =?us-ascii?Q?FhVH6JGXvR+wIjxkZJP1yEBQz6bRMGvkgCHgyhwRK8N3RbVckqr5l/d9N6kp?= =?us-ascii?Q?UZoQLoQbz7rW2XmMFzuw/g7NjdB+lnlCx8SFROYDe0ULXV39wW8nFLaGopaV?= =?us-ascii?Q?Vc0wX6NN/ByQLKLoSDijbBXwezP5D14QY4E+f/rcaFPF0MkfxSddZjyh1VEY?= =?us-ascii?Q?OSTRpuSxcOx7fja95sRjpCbpVpLxuDhX9fQsHViS6OcOGjZsXIU0cXBB/nO+?= =?us-ascii?Q?OfWb5hWvC+OaoDWq03IOcm/FGkDWkrTwKTy/ckpN8JFn82wCdQDSxiHyA4tt?= =?us-ascii?Q?LJLTOJyUV2sPOyTYAcTTWSa+k0FcPGXOdgq7SBlUV7oXGWhp+ZpzyIjFdof9?= =?us-ascii?Q?Vp1qFvLD3stDCw9Hlo8ftcuafkzX+bQ5sShPUQ2SrMxXCGVNpBwNaxN9MVNC?= =?us-ascii?Q?35Kbndg+i4Y4bHv4EkNlvY23ILlM64tSx3OmknLm8k5BQxc2P7a43H2BIt0n?= =?us-ascii?Q?TEdMiV+r+BoZf2PGyt6u7KZrvAc/3fhOjiyqXsKygY5lF9SgB9cykemXvdQa?= =?us-ascii?Q?FrJcaaju70TigjITRGaXf5QNHmwjlCq3ORSY9yx+XmAFJRrf2obvYsPkJdlD?= =?us-ascii?Q?axvXLW00z8GaltmQUn6AjvynZ9S2flm3KqdGZ6YcxLftZUIeXXdYG9nckuq0?= =?us-ascii?Q?2SfSnJw4RC2shn+EXJ6VEqNQQjdq+c0lFtwTSbgOG8adeNuBTPa0LPDKiiX5?= =?us-ascii?Q?NDJhfiI2ATvskx/8ujIt+GOf0Wc3VxUsHKHH6ooUi67ik9sSWJhw/lDHCSNc?= =?us-ascii?Q?DZMd9PiqTklrn5hq2C9L72lZkS0xVBIEOnLixs09bliTbCHxtA/lnyDUkf9G?= =?us-ascii?Q?i9p9TR/dCjyulY7XZlpUz9TACVed1yci4fp4UNqHL+E23sgecFPCQujI/kkK?= =?us-ascii?Q?JACZCmCaKzTVoV1VMx1ZnZf/U1Li+3BZQI7YfS+Sm4fbBCjW8ZNRrmZ4Sqm0?= =?us-ascii?Q?ESbKzMVrENhDW/XS8/4dzjXzBpnLszRbqV7KNOPBm2bOBGpHbu46uR34R3RQ?= =?us-ascii?Q?J4F6aldm3MwgzA+B4Uruv6+XE+Z9WEPQ4N2ojmzY?= Content-Type: text/plain; charset="us-ascii" Content-ID: <653EA2332AD28C47B1129265C455B270@namprd12.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA0PR12MB8983.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 09b5c1d9-9a7f-4eed-0827-08dcce552246 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2024 09:20:22.2026 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: xEtwcwEagKYllpZPd0GhD2t/22Lgq9Cgvxfzo4tAYTXZYEWBMlwscIoczPTjOrWkrREzTB3LM6cz3xzvimlisA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6974 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Message-ID: <20240906092022.huyWoLEZ6iBOwKo3R_Z8BK-wR98RkulzEY2Mz8kSPXw@z> Hi Maxime, > On Sep 5, 2024, at 22:26, Maxime Coquelin wr= ote: >=20 > External email: Use caution opening links or attachments >=20 >=20 > This patch enables VDUSE reconnection support making use of > the newly introduced reconnection mechanism in Vhost > library. >=20 > At DPDK VDUSE device creation time, there are two > possibilities: > 1. The Kernel VDUSE device does not exist: > a. A reconnection file named after the VUDSE device name > is created in VDUSE tmpfs. > b. The file is truncated to 'struct vhost_reconnect_data' > size, and mmapped. > c. Negotiated features, Virtio status... are saved for > sanity checks at reconnect time. > 2. The Kernel VDUSE device already exists: > a. Exit with failure if no reconnect file exists for > this device. > b. Open and mmap the reconnect file. > c. Perform sanity check to ensure features are compatible. > d. Restore virtqueues' available indexes at startup time. >=20 > Then at runtime, the virtqueues' available index are logged by > the Vhost reconnection mechanism. >=20 > At DPDK VDUSE device destruction time, there are two > possibilities: > 1. The Kernel VDUSE device destruction succeed, which > means it is no more attached to the vDPA bus. The > reconnection file is unmapped and then removed. > 2. The Kernel VDUSE device destruction failed, meaning it > is no more attached to the vDPA bus. The reconnection > file is unmapped but not removed to make possible later > reconnection. >=20 > Signed-off-by: Maxime Coquelin > --- > lib/vhost/vduse.c | 280 +++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 241 insertions(+), 39 deletions(-) >=20 > diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c > index c66602905c..bd0e492d62 100644 > --- a/lib/vhost/vduse.c > +++ b/lib/vhost/vduse.c > @@ -136,7 +136,7 @@ vduse_control_queue_event(int fd, void *arg, int *rem= ove __rte_unused) > } >=20 > static void > -vduse_vring_setup(struct virtio_net *dev, unsigned int index) > +vduse_vring_setup(struct virtio_net *dev, unsigned int index, bool recon= nect) > { > struct vhost_virtqueue *vq =3D dev->virtqueue[index]; > struct vhost_vring_addr *ra =3D &vq->ring_addrs; > @@ -152,6 +152,19 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > return; > } >=20 > + if (reconnect) { > + vq->last_avail_idx =3D vq->reconnect_log->last_avail_idx; > + vq->last_used_idx =3D vq->reconnect_log->last_avail_idx; > + } else { > + vq->last_avail_idx =3D vq_info.split.avail_index; > + vq->last_used_idx =3D vq_info.split.avail_index; > + } > + vq->size =3D vq_info.num; > + vq->ready =3D true; > + vq->enabled =3D vq_info.ready; > + ra->desc_user_addr =3D vq_info.desc_addr; > + ra->avail_user_addr =3D vq_info.driver_addr; > + ra->used_user_addr =3D vq_info.device_addr; > VHOST_CONFIG_LOG(dev->ifname, INFO, "VQ %u info:", index); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tnum: %u", vq_info.num); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdesc_addr: %llx", > @@ -162,15 +175,6 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i= nt index) > (unsigned long long)vq_info.device_addr); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq_info.spl= it.avail_index); > VHOST_CONFIG_LOG(dev->ifname, INFO, "\tready: %u", vq_info.ready); > - > - vq->last_avail_idx =3D vq_info.split.avail_index; > - vq->size =3D vq_info.num; > - vq->ready =3D true; > - vq->enabled =3D vq_info.ready; > - ra->desc_user_addr =3D vq_info.desc_addr; > - ra->avail_user_addr =3D vq_info.driver_addr; > - ra->used_user_addr =3D vq_info.device_addr; > - > vq->kickfd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > if (vq->kickfd < 0) { > VHOST_CONFIG_LOG(dev->ifname, ERR, "Failed to init kickfd = for VQ %u: %s", > @@ -267,7 +271,7 @@ vduse_vring_cleanup(struct virtio_net *dev, unsigned = int index) > } >=20 > static void > -vduse_device_start(struct virtio_net *dev) > +vduse_device_start(struct virtio_net *dev, bool reconnect) > { > unsigned int i, ret; >=20 > @@ -287,6 +291,15 @@ vduse_device_start(struct virtio_net *dev) > return; > } >=20 > + if (reconnect && dev->features !=3D dev->reconnect_log->features)= { > + VHOST_CONFIG_LOG(dev->ifname, ERR, > + "Mismatch between reconnect file features= 0x%" PRIx64 " & device features 0x%" PRIx64, Checkpatch reports long line > + dev->reconnect_log->features, dev->featur= es); > + return; > + } > + > + dev->reconnect_log->features =3D dev->features; > + > VHOST_CONFIG_LOG(dev->ifname, INFO, "Negotiated Virtio features: 0= x%" PRIx64, > dev->features); >=20 > @@ -300,7 +313,7 @@ vduse_device_start(struct virtio_net *dev) > } >=20 > for (i =3D 0; i < dev->nr_vring; i++) > - vduse_vring_setup(dev, i); > + vduse_vring_setup(dev, i, reconnect); >=20 > dev->flags |=3D VIRTIO_DEV_READY; >=20 > @@ -373,6 +386,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) > req.s.status); > old_status =3D dev->status; > dev->status =3D req.s.status; > + dev->reconnect_log->status =3D dev->status; > resp.result =3D VDUSE_REQ_RESULT_OK; > break; > case VDUSE_UPDATE_IOTLB: > @@ -398,7 +412,7 @@ vduse_events_handler(int fd, void *arg, int *remove _= _rte_unused) >=20 > if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) > - vduse_device_start(dev); > + vduse_device_start(dev, false); > else > vduse_device_stop(dev); > } > @@ -407,10 +421,64 @@ vduse_events_handler(int fd, void *arg, int *remove= __rte_unused) > vduse_req_id_to_str(req.type), req.type); > } >=20 > +static char vduse_reconnect_dir[PATH_MAX]; > +static bool vduse_reconnect_path_set; > + > +static int > +vduse_reconnect_path_init(void) > +{ > + const char *directory; > + int ret; > + > + /* from RuntimeDirectory=3D see systemd.exec */ > + directory =3D getenv("RUNTIME_DIRECTORY"); > + if (directory =3D=3D NULL) { > + /* > + * Used standard convention defined in > + * XDG Base Directory Specification and > + * Filesystem Hierarchy Standard. > + */ > + if (getuid() =3D=3D 0) > + directory =3D "/var/run"; > + else > + directory =3D getenv("XDG_RUNTIME_DIR") ? : "/tmp= "; > + } > + > + ret =3D snprintf(vduse_reconnect_dir, sizeof(vduse_reconnect_dir)= , "%s/dpdk/vduse", > + directory); > + if (ret < 0 || ret =3D=3D sizeof(vduse_reconnect_dir)) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating VDUSE reco= nnect path name"); > + return -1; > + } > + > + ret =3D mkdir(vduse_reconnect_dir, 0700); > + if (ret < 0 && errno !=3D EEXIST) { > + VHOST_CONFIG_LOG("vduse", ERR, "Error creating '%s': %s", > + vduse_reconnect_dir, strerror(errno)); > + return -1; > + } > + > + VHOST_CONFIG_LOG("vduse", INFO, "Created VDUSE reconnect director= y in %s", > + vduse_reconnect_dir); > + > + return 0; > +} > + > +static void > +vduse_reconnect_handler(int fd, void *arg, int *remove) > +{ > + struct virtio_net *dev =3D arg; > + > + vduse_device_start(dev, true); > + > + close(fd); > + *remove =3D 1; > +} > + > int > vduse_device_create(const char *path, bool compliant_ol_flags) > { > - int control_fd, dev_fd, vid, ret; > + int control_fd, dev_fd, vid, ret, reco_fd; > uint32_t i, max_queue_pairs, total_queues; > struct virtio_net *dev; > struct virtio_net_config vnet_config =3D {{ 0 }}; > @@ -418,6 +486,9 @@ vduse_device_create(const char *path, bool compliant_= ol_flags) > uint64_t features; > struct vduse_dev_config *dev_config =3D NULL; > const char *name =3D path + strlen("/dev/vduse/"); > + char reconnect_file[PATH_MAX]; > + struct vhost_reconnect_data *reconnect_log =3D NULL; > + bool reconnect =3D false; >=20 > if (vduse.fdset =3D=3D NULL) { > vduse.fdset =3D fdset_init("vduse-evt"); > @@ -427,6 +498,20 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > } > } >=20 > + if (vduse_reconnect_path_set =3D=3D false) { > + if (vduse_reconnect_path_init() < 0) { > + VHOST_CONFIG_LOG(path, ERR, "failed to initialize= reconnect path"); > + return -1; > + } > + vduse_reconnect_path_set =3D true; > + } > + > + ret =3D snprintf(reconnect_file, sizeof(reconnect_file), "%s/%s",= vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create vduse recon= nect path name"); > + return -1; > + } > + > control_fd =3D open(VDUSE_CTRL_PATH, O_RDWR); > if (control_fd < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to open %s: %s", > @@ -441,14 +526,6 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_ctrl_close; > } >=20 > - dev_config =3D malloc(offsetof(struct vduse_dev_config, config) + > - sizeof(vnet_config)); > - if (!dev_config) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to allocate VDUSE con= fig"); > - ret =3D -1; > - goto out_ctrl_close; > - } > - > ret =3D rte_vhost_driver_get_features(path, &features); > if (ret < 0) { > VHOST_CONFIG_LOG(name, ERR, "Failed to get backend feature= s"); > @@ -469,23 +546,97 @@ vduse_device_create(const char *path, bool complian= t_ol_flags) > else > total_queues +=3D 1; /* Includes ctrl queue */ >=20 > - vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > - memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + if (access(path, F_OK) =3D=3D 0) { > + VHOST_CONFIG_LOG(name, INFO, "Device already exists, reco= nnecting..."); > + reconnect =3D true; > + > + reco_fd =3D open(reconnect_file, O_RDWR, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D ENOENT) { > + VHOST_CONFIG_LOG(name, ERR, "Missing reco= nnect file (%s)", > + reconnect_file); > + } else { > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + } Seems no {} is needed for if-else? > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } > + } else { > + reco_fd =3D open(reconnect_file, O_CREAT | O_EXCL | O_RDW= R, 0600); > + if (reco_fd < 0) { > + if (errno =3D=3D EEXIST) { > + VHOST_CONFIG_LOG(name, ERR, "Reconnect fi= le %s exists but not the device", > + reconnect_file); > + } else { > + VHOST_CONFIG_LOG(name, ERR, "Failed to op= en reconnect file %s (%s)", > + reconnect_file, strerror(= errno)); > + } > + ret =3D -1; > + goto out_ctrl_close; > + } >=20 > - strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > - dev_config->device_id =3D VIRTIO_ID_NET; > - dev_config->vendor_id =3D 0; > - dev_config->features =3D features; > - dev_config->vq_num =3D total_queues; > - dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > - dev_config->config_size =3D sizeof(struct virtio_net_config); > - memcpy(dev_config->config, &vnet_config, sizeof(vnet_config)); > + ret =3D ftruncate(reco_fd, sizeof(*reconnect_log)); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to truncate r= econnect file %s (%s)", > + reconnect_file, strerror(errno)); > + close(reco_fd); > + goto out_ctrl_close; > + } >=20 > - ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > - if (ret < 0) { > - VHOST_CONFIG_LOG(name, ERR, "Failed to create VDUSE devic= e: %s", > - strerror(errno)); > - goto out_free; > + reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT= _READ | PROT_WRITE, > + MAP_SHARED, reco_fd, 0); > + close(reco_fd); > + if (reconnect_log =3D=3D MAP_FAILED) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon= nect file %s (%s)", > + reconnect_file, strerror(errno)); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + reconnect_log->version =3D 0; > + > + dev_config =3D malloc(offsetof(struct vduse_dev_config, c= onfig) + > + sizeof(vnet_config)); > + if (!dev_config) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to allocate V= DUSE config"); > + ret =3D -1; > + goto out_ctrl_close; > + } > + > + vnet_config.max_virtqueue_pairs =3D max_queue_pairs; > + memset(dev_config, 0, sizeof(struct vduse_dev_config)); > + > + rte_strscpy(dev_config->name, name, VDUSE_NAME_MAX - 1); > + dev_config->device_id =3D VIRTIO_ID_NET; > + dev_config->vendor_id =3D 0; > + dev_config->features =3D features; > + dev_config->vq_num =3D total_queues; > + dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE); > + dev_config->config_size =3D sizeof(struct virtio_net_conf= ig); > + memcpy(dev_config->config, &vnet_config, sizeof(vnet_conf= ig)); > + > + ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create VDU= SE device: %s", > + strerror(errno)); > + goto out_free; > + } > + > + memcpy(&reconnect_log->config, &vnet_config, sizeof(vnet_= config)); > + reconnect_log->nr_vrings =3D total_queues; > + free(dev_config); > + dev_config =3D NULL; > } >=20 > dev_fd =3D open(path, O_RDWR); > @@ -519,10 +670,15 @@ vduse_device_create(const char *path, bool complian= t_ol_flags) > strncpy(dev->ifname, path, IF_NAME_SZ - 1); > dev->vduse_ctrl_fd =3D control_fd; > dev->vduse_dev_fd =3D dev_fd; > + dev->reconnect_log =3D reconnect_log; > + if (reconnect) > + dev->status =3D dev->reconnect_log->status; > + > vhost_setup_virtio_net(dev->vid, true, compliant_ol_flags, true, t= rue); >=20 > for (i =3D 0; i < total_queues; i++) { > struct vduse_vq_config vq_cfg =3D { 0 }; > + struct vhost_virtqueue *vq; >=20 > ret =3D alloc_vring_queue(dev, i); > if (ret) { > @@ -530,6 +686,12 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > + vq =3D dev->virtqueue[i]; > + vq->reconnect_log =3D &reconnect_log->vring[i]; > + > + if (reconnect) > + continue; > + > vq_cfg.index =3D i; > vq_cfg.max_size =3D 1024; >=20 > @@ -549,7 +711,28 @@ vduse_device_create(const char *path, bool compliant= _ol_flags) > goto out_dev_destroy; > } >=20 > - free(dev_config); > + if (reconnect && dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) { > + reco_fd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); > + if (reco_fd < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to create rec= o_fd: %s", > + strerror(errno)); > + ret =3D -1; > + goto out_dev_destroy; > + } > + > + ret =3D fdset_add(vduse.fdset, reco_fd, vduse_reconnect_h= andler, NULL, dev); > + if (ret) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to add fd %d = to vduse fdset", > + dev->vduse_dev_fd); Should print reco_fd > + goto out_dev_destroy; > + } > + > + ret =3D eventfd_write(reco_fd, (eventfd_t)1); > + if (ret < 0) { > + VHOST_CONFIG_LOG(name, ERR, "Failed to write to r= econnect eventfd"); > + goto out_dev_destroy; > + } > + } Maybe I missed something, why we need to implement like this instead of dir= ectly call vduse_device_start? Thanks, Chenbo >=20 > return 0; >=20 > @@ -587,6 +770,9 @@ vduse_device_destroy(const char *path) > if (vid =3D=3D RTE_MAX_VHOST_DEVICE) > return -1; >=20 > + if (dev->reconnect_log) > + munmap(dev->reconnect_log, sizeof(*dev->reconnect_log)); > + > vduse_device_stop(dev); >=20 > fdset_del(vduse.fdset, dev->vduse_dev_fd); > @@ -597,10 +783,26 @@ vduse_device_destroy(const char *path) > } >=20 > if (dev->vduse_ctrl_fd >=3D 0) { > + char reconnect_file[PATH_MAX]; > + > ret =3D ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name)= ; > - if (ret) > + if (ret) { > VHOST_CONFIG_LOG(name, ERR, "Failed to destroy VDU= SE device: %s", > strerror(errno)); > + } else { > + /* > + * VDUSE device was no more attached to the vDPA = bus, > + * so we can remove the reconnect file. > + */ > + ret =3D snprintf(reconnect_file, sizeof(reconnect= _file), "%s/%s", > + vduse_reconnect_dir, name); > + if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) > + VHOST_CONFIG_LOG(name, ERR, > + "Failed to create vduse r= econnect path name"); > + else > + unlink(reconnect_file); > + } > + > close(dev->vduse_ctrl_fd); > dev->vduse_ctrl_fd =3D -1; > } > -- > 2.46.0 >=20