From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 032484591A;
	Fri,  6 Sep 2024 09:14:58 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id E1AB14025D;
	Fri,  6 Sep 2024 09:14:57 +0200 (CEST)
Received: from NAM10-BN7-obe.outbound.protection.outlook.com
 (mail-bn7nam10on2081.outbound.protection.outlook.com [40.107.92.81])
 by mails.dpdk.org (Postfix) with ESMTP id 100884025C
 for <dev@dpdk.org>; Fri,  6 Sep 2024 09:14:56 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=LencbrhzeCmdwA3ax6ieV16EHgDvhv+nE6fPkHIVON5ciMFdZNiKKgOY7N+jnEvAbSpjHgB6dCyppcKO5ShoJL/T0eyksSi6Ti4WSXr5SJzCBaWh7Ed1n3PQNmz03Aoy/gtkb008Hjh+prdfT4PiNFDeSHj1lfZ2tTx5xzn510KHHwUpEQkYADKjynjiwNGWKPEf+DjYyFTXGs5f9hbso2FsYsETDrADQpBhY9ktssfMbt0EmferMDHJECF2Z/Kghm8fOOyTrKo7EsdD25LH+Ma38X1DyKfHubsHg9+ECKLqWUaZ802Z4xoV/2Uc1yQu4Q+Xi8MEL46+fFkc576AVA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=;
 b=DjDl48m5BLv+Yj/g6VHXGzcHPcrn/NTWdxuqJyclpSYr/s0ZZgcsYs+FXMwsH1lJJk7TDUTkd8lP4ev6cK6kwF9lQmtrvZ+w4Xl7pRQOgdnAAe5gqTeLvpnsi869VthWTobvZFUcFKO3nOdQRVIoXza3PcgUdP5NgjkcE4Y0LS7f1VtVKYGbMmwyQpA1oU0D4kGoW3IS6VmJXmEQOQY+QCUFpzjDPExwc9SZmdwg7l+ixWMbdnTV4umdl37UHbz2VSOakFAxwb+LcB77bkJtO/0qiomI/9DHz5dUlgIMPAcR/VjB3tBBshX9ki3qGsjl+RME+Wlutj5/fJrhKS8Bmw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=;
 b=QrnL6bciDbRnuA7bzYC/SIGMMUSTbglROzxUgq/3tU8iWYyg1/o4AhDG0YqBr4sZfgXGcqcrrGtBkM+rSOUlo4rXI9d8E2Qj7LYFdLa+MlDERRZ0/7MbI3SXysEZhBZQcjJlVbfgggjXnOYW2CwcvCes3JFXJ7TGBTxuBCQgJQg862wXfyAYRYxPrObR55yfUfafJRXs9orNviFbh4LCF6TdtIRKDu/g/fILKu0mf364W6NfweOf4eOzW7qs2wt2PMB4T+tRAG/FMi0kZ1dnFNbtxI3bVfpcTB8PsGwmAnBBEAsBDdM9FeW6AV1g2fOxr9UcgHCI31JRPAbN0y6HcA==
Received: from CYYPR12MB8992.namprd12.prod.outlook.com (2603:10b6:930:bc::21)
 by PH0PR12MB7471.namprd12.prod.outlook.com (2603:10b6:510:1e9::18)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.28; Fri, 6 Sep
 2024 07:14:53 +0000
Received: from CYYPR12MB8992.namprd12.prod.outlook.com
 ([fe80::a722:c2ca:42c7:aaed]) by CYYPR12MB8992.namprd12.prod.outlook.com
 ([fe80::a722:c2ca:42c7:aaed%7]) with mapi id 15.20.7918.024; Fri, 6 Sep 2024
 07:14:53 +0000
From: Chenbo Xia <chenbox@nvidia.com>
To: Maxime Coquelin <maxime.coquelin@redhat.com>
CC: "dev@dpdk.org" <dev@dpdk.org>, "david.marchand@redhat.com"
 <david.marchand@redhat.com>
Subject: Re: [PATCH 2/2] vhost: add reconnection support to VDUSE
Thread-Topic: [PATCH 2/2] vhost: add reconnection support to VDUSE
Thread-Index: AQHa/5+Tb2hkJpD2pkWfDp9/zoprPbJKWaMA
Date: Fri, 6 Sep 2024 07:14:52 +0000
Message-ID: <3543971A-6C8E-4F28-A7D4-F8C222F3C193@nvidia.com>
References: <20240905142602.1264345-1-maxime.coquelin@redhat.com>
 <20240905142602.1264345-3-maxime.coquelin@redhat.com>
In-Reply-To: <20240905142602.1264345-3-maxime.coquelin@redhat.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: CYYPR12MB8992:EE_|PH0PR12MB7471:EE_
x-ms-office365-filtering-correlation-id: 8e70f466-1363-4126-7967-08dcce439a80
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0; ARA:13230040|376014|366016|1800799024|38070700018;
x-microsoft-antispam-message-info: =?us-ascii?Q?zx0DWt7HeeTId2KR7QXZur6JkKYQkBL4h4tN6PYr5QlLwGBh3OM6z+dz8xKU?=
 =?us-ascii?Q?Pxu/wfFprpDloy/rf8v0UmBlwBair9irkwJeHX0a0LIvNNtl7k2ldpszKmxW?=
 =?us-ascii?Q?Y50MEFmpZLciFcn03IFHfIDLc1KqaEdfnsP2k1QNVf4vQjuA8yoE1TL8f7A9?=
 =?us-ascii?Q?d9+UBA9gn3SOiVKb+GqjfdYtjQvrSuoBGY3CEEJVKzghaV8KHZX6IsXaAyiS?=
 =?us-ascii?Q?g0+ZdhDFR1qBoG+8mIXLd7Q6hkeXHLbs3J7gEHC2IYzcLDk+ESalh8YlyxSJ?=
 =?us-ascii?Q?9a1t0Rfsw7owdzuO+DtGYzCOGHta6sYyG+dJZzFKOhi8YYaiUXQaVUiya+Gt?=
 =?us-ascii?Q?qfkevXP6PpKGalJyagzuwQWMG0xSO3D2sKl250giGs8LEin8rCH8F05KtN1O?=
 =?us-ascii?Q?FA75UW2WUHhibRUZGkbeu5S4FjyilLTUSqOOjLcfrhhuFoNtfSGWF7PcJOVw?=
 =?us-ascii?Q?oOjlkn124zR0RdkR69h4N2wMevFG1DrbfYsjsO2HPVHLWcxemBAwDx5L2Mis?=
 =?us-ascii?Q?VIT1Qw+7nh5M9w7UrmZKCOrUT7J6gh5QYxDrLWWz/CFY3GcKMnTNEdzVdIds?=
 =?us-ascii?Q?APN1C2KKeXyh9NpBYfow1DTPmkRuNrbuWrIcebjKuv3GGvmc+Baa0zY6EzM3?=
 =?us-ascii?Q?i4aQaZaxZqMvQM8w/N+VWPZYx1OSkOBmIiQD5P3yu0NC423zlHmDkHWjQ1Wr?=
 =?us-ascii?Q?BxUkrRFAEz2RTNcc6KoPOI1Uqfwy+m3fZvmNum5WXtotJrGXez2IyXzBppzF?=
 =?us-ascii?Q?ygOq1VgJyGCarKsjNyyobJ6iY2P8v0KSGyDgWqDbWE6+8dajmZJoV3YMZzLx?=
 =?us-ascii?Q?3Z96LWTGgINOZTanmdJttJF5WudopfHTgTLHOL6diR0pM800tUwDC94FTpZc?=
 =?us-ascii?Q?83QZxQOTSnTwvTqaG0NtU6kWbYHFsM/kVTVZvVhPxxbDxJqdljHVyuyFaWbk?=
 =?us-ascii?Q?G8JZCk0zjmsx8OosiVo7LvpjNI4HiFwSHwGIxloGj5Y66hQ0eCxZDBc7+/uh?=
 =?us-ascii?Q?DCQyJpPR6guI1WaEgYoxDLASBvSjFxbpz6Fvi68C5OaUAk85voeZBP8Qvck2?=
 =?us-ascii?Q?hZ2yFvng9sSGA9fypKhA2lnOaYybms/s2oA5xJhmmAmPA7QcVLEAwmHKBUcj?=
 =?us-ascii?Q?JUBrEQarmSvcPFlNSWcvvGcuzgA2Wn7aGS4pTQJuFOzoOiAtImiwrqo4GREm?=
 =?us-ascii?Q?9unM0HjuQRAsI/b/1teSSa9hKv43PthpKmbm9WmuwNTp6WrjVz1IAVOoifZh?=
 =?us-ascii?Q?OBu8p4IL5jxN+0d9CWH1Q1FX5JjZ4h55wvay9fLATVjkgm8TW5+w210H7Ynf?=
 =?us-ascii?Q?9O8der+DaUOCx1IGwUI/TXcEqRNgWiZzvviMFepFmIrPBMduw21W2ndPuY71?=
 =?us-ascii?Q?48TuG6dG/hzJK5PAOh+yqBnpz/pwX5+nZKm77VXssQ5AXD8Mog=3D=3D?=
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:CYYPR12MB8992.namprd12.prod.outlook.com; PTR:; CAT:NONE;
 SFS:(13230040)(376014)(366016)(1800799024)(38070700018); DIR:OUT; SFP:1101; 
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?WTXsGEUCFmmDmgE/bty3qHtCee/ttLJbJ+Bh987Aj4GW7aeF7H882juE15OM?=
 =?us-ascii?Q?JSxpvZpiddNPE+w8kEr6Vp+shtnix3DjUn4A/N98GmbgmGp3MHsDokYrtL3U?=
 =?us-ascii?Q?je77hop9Ya9zUP3l9loZuy6HVdV3HvzSYNgvk/+3tiumVA28KXgvLIoLN+yM?=
 =?us-ascii?Q?uzxqIVS+I9s0lctX1IzNx5Cf57YwXH5N43j3+AMxrTgMxFTwKGSGXVacuIXW?=
 =?us-ascii?Q?cTdUjyWEzW9LTdVGPlkyZ+eE8xXtE8jnopfMYI/bdalIO2q8MzA9Mjw/NlIs?=
 =?us-ascii?Q?ZdRSNOyOxgcvk/PDJzfSU+wMrqzEayUp33MPgu8/NSMnM8NBvJMULJwqbjYJ?=
 =?us-ascii?Q?ixx0oJlmhQW8n1wtDBw4eDm9hcTuMgnVG751q8fVhBsmPwGtSKA4kMwdVxiI?=
 =?us-ascii?Q?WBVRFt1GnoWBOhCiBZ3rjt/uoDv6XbJhnHIjbFxteabs23QFAWaJ5uHgYFyX?=
 =?us-ascii?Q?boFMEX/Sl5V2x/CeriQWiJW34KvExYpFUPNdzXID601o3Robrx3l+SphrMjj?=
 =?us-ascii?Q?jy84YloWmKyLywe+RRsGpNoPGZM86t4bdglpecl4zmGRCdAJdtbvEp1EZ5TX?=
 =?us-ascii?Q?QUxgxZzemz6PF2Kx5iDqOh28ybkX1cp4VhmQUsYWkMBOxO5AXT0J4yi5Ltgr?=
 =?us-ascii?Q?FSoAlfzN8QG3Wceme12Lt746d98+2yn4I5yOMINJepN8nnqlpBP8/liRdlg1?=
 =?us-ascii?Q?W6zGVNizvmYMJY5gYdyPiEy+YAK2YjnSujd3GZ22N0Ta/CTtYxqMYgBqRpEe?=
 =?us-ascii?Q?oyIGeptWxFYCEWoIxYRKUnL6pNnbDUawX/gGqRoYtE4QQ7/IOXELOuP1L14J?=
 =?us-ascii?Q?6TBUUpAKd5G7tv8ID1o7rhzZGVd/q2Upg0i69yNBPivTbzpuLZXHbolP/BQW?=
 =?us-ascii?Q?OwynTmtUiYIaHlkssUsB7yuGINkmxUn53BG5+yKtnS9vXfCTJ5vlTgPtX5td?=
 =?us-ascii?Q?q+MACo+/Xl7NOp0LjldTbHBDR4xutasfdGdqWJdwYYjeLhP1QiiC0KlvTlwf?=
 =?us-ascii?Q?csU9zJN9Lvh2FSVUecOynHsRtEkyBL0dY0f/d1qsflNcfzjM08DttUONjAU3?=
 =?us-ascii?Q?BwW49sH1d53agHC2NzhpXXLOA2ME9QUnEIY27IXlQkC5a/8Vd0xfrMtAHbjL?=
 =?us-ascii?Q?NKRotZHjQKzkIE0gC4ligGIep0nZzyFjOqjH0RMJtRojNfyuslUWKVxaiEe8?=
 =?us-ascii?Q?CgaxxCpyA/hz++IT/mEcwhcUbEDC79ttQ0spoS05nIXfxePNIDq/hhozVAVb?=
 =?us-ascii?Q?oxVLy74lq2WXo9n2UdLulGhTJBvRJVh0mfoNOsb9FY4PNlY5V/0WD+BcUDQO?=
 =?us-ascii?Q?4F1ORVZ6ASN/BPKmTHCmIcZ1SPVV9aj+aGC8aGNHUQhGC8BenB/kZbWRRBA3?=
 =?us-ascii?Q?dQvZd0g9NG7n8fzrWEPsFHk4SXHK2Od1lSP7hhDPe7HoiWIEoPF2lEehALya?=
 =?us-ascii?Q?aJ9PWyLd1pGVUENHTt7K02dsArSIf8+Wez6Pl15UcGZ0aBCzIbWnYJTln36y?=
 =?us-ascii?Q?uljxX+hvM23a+BmgQeXKxH6K1VlFYTLDxzNpet06efLjBJeAiS7/SvAnsv72?=
 =?us-ascii?Q?txqP1ck+2E+V3w2lUE+NVkfCSiHhIZwKfbOQmGSCrFmpJKWO61ST27fGcOCT?=
 =?us-ascii?Q?dg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D467F3B0238D814A93D1EFF4F2C1B147@namprd12.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CYYPR12MB8992.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8e70f466-1363-4126-7967-08dcce439a80
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2024 07:14:52.9774 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 0LXdZFwi7a//FlOKuaStkDe9GCNGaLJsIxFpJQJV3BW/zfsS7nebnRMhi6zDIy3xvqffBmsCwMIj03FY1uv28A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7471
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

Hi Maxime,

> On Sep 5, 2024, at 22:26, Maxime Coquelin <maxime.coquelin@redhat.com> wr=
ote:
>=20
> External email: Use caution opening links or attachments
>=20
>=20
> This patch enables VDUSE reconnection support making use of
> the newly introduced reconnection mechanism in Vhost
> library.
>=20
> At DPDK VDUSE device creation time, there are two
> possibilities:
> 1. The Kernel VDUSE device does not exist:
>  a. A reconnection file named after the VUDSE device name
>     is created in VDUSE tmpfs.
>  b. The file is truncated to 'struct vhost_reconnect_data'
>     size, and mmapped.
>  c. Negotiated features, Virtio status... are saved for
>     sanity checks at reconnect time.
> 2. The Kernel VDUSE device already exists:
>  a. Exit with failure if no reconnect file exists for
>     this device.
>  b. Open and mmap the reconnect file.
>  c. Perform sanity check to ensure features are compatible.
>  d. Restore virtqueues' available indexes at startup time.
>=20
> Then at runtime, the virtqueues' available index are logged by
> the Vhost reconnection mechanism.
>=20
> At DPDK VDUSE device destruction time, there are two
> possibilities:
> 1. The Kernel VDUSE device destruction succeed, which
>    means it is no more attached to the vDPA bus. The
>    reconnection file is unmapped and then removed.
> 2. The Kernel VDUSE device destruction failed, meaning it
>    is no more attached to the vDPA bus. The reconnection
>    file is unmapped but not removed to make possible later
>    reconnection.
>=20
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/vhost/vduse.c | 280 +++++++++++++++++++++++++++++++++++++++-------
> 1 file changed, 241 insertions(+), 39 deletions(-)
>=20
> diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c
> index c66602905c..bd0e492d62 100644
> --- a/lib/vhost/vduse.c
> +++ b/lib/vhost/vduse.c
> @@ -136,7 +136,7 @@ vduse_control_queue_event(int fd, void *arg, int *rem=
ove __rte_unused)
> }
>=20
> static void
> -vduse_vring_setup(struct virtio_net *dev, unsigned int index)
> +vduse_vring_setup(struct virtio_net *dev, unsigned int index, bool recon=
nect)
> {
>        struct vhost_virtqueue *vq =3D dev->virtqueue[index];
>        struct vhost_vring_addr *ra =3D &vq->ring_addrs;
> @@ -152,6 +152,19 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i=
nt index)
>                return;
>        }
>=20
> +       if (reconnect) {
> +               vq->last_avail_idx =3D vq->reconnect_log->last_avail_idx;
> +               vq->last_used_idx =3D vq->reconnect_log->last_avail_idx;
> +       } else {
> +               vq->last_avail_idx =3D vq_info.split.avail_index;
> +               vq->last_used_idx =3D vq_info.split.avail_index;
> +       }
> +       vq->size =3D vq_info.num;
> +       vq->ready =3D true;
> +       vq->enabled =3D vq_info.ready;
> +       ra->desc_user_addr =3D vq_info.desc_addr;
> +       ra->avail_user_addr =3D vq_info.driver_addr;
> +       ra->used_user_addr =3D vq_info.device_addr;
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "VQ %u info:", index);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tnum: %u", vq_info.num);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdesc_addr: %llx",
> @@ -162,15 +175,6 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i=
nt index)
>                        (unsigned long long)vq_info.device_addr);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq_info.spl=
it.avail_index);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tready: %u", vq_info.ready);
> -
> -       vq->last_avail_idx =3D vq_info.split.avail_index;
> -       vq->size =3D vq_info.num;
> -       vq->ready =3D true;
> -       vq->enabled =3D vq_info.ready;
> -       ra->desc_user_addr =3D vq_info.desc_addr;
> -       ra->avail_user_addr =3D vq_info.driver_addr;
> -       ra->used_user_addr =3D vq_info.device_addr;
> -
>        vq->kickfd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
>        if (vq->kickfd < 0) {
>                VHOST_CONFIG_LOG(dev->ifname, ERR, "Failed to init kickfd =
for VQ %u: %s",
> @@ -267,7 +271,7 @@ vduse_vring_cleanup(struct virtio_net *dev, unsigned =
int index)
> }
>=20
> static void
> -vduse_device_start(struct virtio_net *dev)
> +vduse_device_start(struct virtio_net *dev, bool reconnect)
> {
>        unsigned int i, ret;
>=20
> @@ -287,6 +291,15 @@ vduse_device_start(struct virtio_net *dev)
>                return;
>        }
>=20
> +       if (reconnect && dev->features !=3D dev->reconnect_log->features)=
 {
> +               VHOST_CONFIG_LOG(dev->ifname, ERR,
> +                               "Mismatch between reconnect file features=
 0x%" PRIx64 " & device features 0x%" PRIx64,

Checkpatch reports long line

> +                               dev->reconnect_log->features, dev->featur=
es);
> +               return;
> +       }
> +
> +       dev->reconnect_log->features =3D dev->features;
> +
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "Negotiated Virtio features: 0=
x%" PRIx64,
>                dev->features);
>=20
> @@ -300,7 +313,7 @@ vduse_device_start(struct virtio_net *dev)
>        }
>=20
>        for (i =3D 0; i < dev->nr_vring; i++)
> -               vduse_vring_setup(dev, i);
> +               vduse_vring_setup(dev, i, reconnect);
>=20
>        dev->flags |=3D VIRTIO_DEV_READY;
>=20
> @@ -373,6 +386,7 @@ vduse_events_handler(int fd, void *arg, int *remove _=
_rte_unused)
>                                req.s.status);
>                old_status =3D dev->status;
>                dev->status =3D req.s.status;
> +               dev->reconnect_log->status =3D dev->status;
>                resp.result =3D VDUSE_REQ_RESULT_OK;
>                break;
>        case VDUSE_UPDATE_IOTLB:
> @@ -398,7 +412,7 @@ vduse_events_handler(int fd, void *arg, int *remove _=
_rte_unused)
>=20
>        if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) {
>                if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK)
> -                       vduse_device_start(dev);
> +                       vduse_device_start(dev, false);
>                else
>                        vduse_device_stop(dev);
>        }
> @@ -407,10 +421,64 @@ vduse_events_handler(int fd, void *arg, int *remove=
 __rte_unused)
>                        vduse_req_id_to_str(req.type), req.type);
> }
>=20
> +static char vduse_reconnect_dir[PATH_MAX];
> +static bool vduse_reconnect_path_set;
> +
> +static int
> +vduse_reconnect_path_init(void)
> +{
> +       const char *directory;
> +       int ret;
> +
> +       /* from RuntimeDirectory=3D see systemd.exec */
> +       directory =3D getenv("RUNTIME_DIRECTORY");
> +       if (directory =3D=3D NULL) {
> +               /*
> +                * Used standard convention defined in
> +                * XDG Base Directory Specification and
> +                * Filesystem Hierarchy Standard.
> +                */
> +               if (getuid() =3D=3D 0)
> +                       directory =3D "/var/run";
> +               else
> +                       directory =3D getenv("XDG_RUNTIME_DIR") ? : "/tmp=
";
> +       }
> +
> +       ret =3D snprintf(vduse_reconnect_dir, sizeof(vduse_reconnect_dir)=
, "%s/dpdk/vduse",
> +                       directory);
> +       if (ret < 0 || ret =3D=3D sizeof(vduse_reconnect_dir)) {
> +               VHOST_CONFIG_LOG("vduse", ERR, "Error creating VDUSE reco=
nnect path name");
> +               return -1;
> +       }
> +
> +       ret =3D mkdir(vduse_reconnect_dir, 0700);
> +       if (ret < 0 && errno !=3D EEXIST) {
> +               VHOST_CONFIG_LOG("vduse", ERR, "Error creating '%s': %s",
> +                               vduse_reconnect_dir, strerror(errno));
> +               return -1;
> +       }
> +
> +       VHOST_CONFIG_LOG("vduse", INFO, "Created VDUSE reconnect director=
y in %s",
> +                       vduse_reconnect_dir);
> +
> +       return 0;
> +}
> +
> +static void
> +vduse_reconnect_handler(int fd, void *arg, int *remove)
> +{
> +       struct virtio_net *dev =3D arg;
> +
> +       vduse_device_start(dev, true);
> +
> +       close(fd);
> +       *remove =3D 1;
> +}
> +
> int
> vduse_device_create(const char *path, bool compliant_ol_flags)
> {
> -       int control_fd, dev_fd, vid, ret;
> +       int control_fd, dev_fd, vid, ret, reco_fd;
>        uint32_t i, max_queue_pairs, total_queues;
>        struct virtio_net *dev;
>        struct virtio_net_config vnet_config =3D {{ 0 }};
> @@ -418,6 +486,9 @@ vduse_device_create(const char *path, bool compliant_=
ol_flags)
>        uint64_t features;
>        struct vduse_dev_config *dev_config =3D NULL;
>        const char *name =3D path + strlen("/dev/vduse/");
> +       char reconnect_file[PATH_MAX];
> +       struct vhost_reconnect_data *reconnect_log =3D NULL;
> +       bool reconnect =3D false;
>=20
>        if (vduse.fdset =3D=3D NULL) {
>                vduse.fdset =3D fdset_init("vduse-evt");
> @@ -427,6 +498,20 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                }
>        }
>=20
> +       if (vduse_reconnect_path_set =3D=3D false) {
> +               if (vduse_reconnect_path_init() < 0) {
> +                       VHOST_CONFIG_LOG(path, ERR, "failed to initialize=
 reconnect path");
> +                       return -1;
> +               }
> +               vduse_reconnect_path_set =3D true;
> +       }
> +
> +       ret =3D snprintf(reconnect_file, sizeof(reconnect_file), "%s/%s",=
 vduse_reconnect_dir, name);
> +       if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) {
> +               VHOST_CONFIG_LOG(name, ERR, "Failed to create vduse recon=
nect path name");
> +               return -1;
> +       }
> +
>        control_fd =3D open(VDUSE_CTRL_PATH, O_RDWR);
>        if (control_fd < 0) {
>                VHOST_CONFIG_LOG(name, ERR, "Failed to open %s: %s",
> @@ -441,14 +526,6 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                goto out_ctrl_close;
>        }
>=20
> -       dev_config =3D malloc(offsetof(struct vduse_dev_config, config) +
> -                       sizeof(vnet_config));
> -       if (!dev_config) {
> -               VHOST_CONFIG_LOG(name, ERR, "Failed to allocate VDUSE con=
fig");
> -               ret =3D -1;
> -               goto out_ctrl_close;
> -       }
> -
>        ret =3D rte_vhost_driver_get_features(path, &features);
>        if (ret < 0) {
>                VHOST_CONFIG_LOG(name, ERR, "Failed to get backend feature=
s");
> @@ -469,23 +546,97 @@ vduse_device_create(const char *path, bool complian=
t_ol_flags)
>        else
>                total_queues +=3D 1; /* Includes ctrl queue */
>=20
> -       vnet_config.max_virtqueue_pairs =3D max_queue_pairs;
> -       memset(dev_config, 0, sizeof(struct vduse_dev_config));
> +       if (access(path, F_OK) =3D=3D 0) {
> +               VHOST_CONFIG_LOG(name, INFO, "Device already exists, reco=
nnecting...");
> +               reconnect =3D true;
> +
> +               reco_fd =3D open(reconnect_file, O_RDWR, 0600);
> +               if (reco_fd < 0) {
> +                       if (errno =3D=3D ENOENT) {
> +                               VHOST_CONFIG_LOG(name, ERR, "Missing reco=
nnect file (%s)",
> +                                               reconnect_file);
> +                       } else {
> +                               VHOST_CONFIG_LOG(name, ERR, "Failed to op=
en reconnect file %s (%s)",
> +                                               reconnect_file, strerror(=
errno));
> +                       }

Seems no {} is needed for if-else?

> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT=
_READ | PROT_WRITE,
> +                               MAP_SHARED, reco_fd, 0);
> +               close(reco_fd);
> +               if (reconnect_log =3D=3D MAP_FAILED) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon=
nect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +       } else {
> +               reco_fd =3D open(reconnect_file, O_CREAT | O_EXCL | O_RDW=
R, 0600);
> +               if (reco_fd < 0) {
> +                       if (errno =3D=3D EEXIST) {
> +                               VHOST_CONFIG_LOG(name, ERR, "Reconnect fi=
le %s exists but not the device",
> +                                               reconnect_file);
> +                       } else {
> +                               VHOST_CONFIG_LOG(name, ERR, "Failed to op=
en reconnect file %s (%s)",
> +                                               reconnect_file, strerror(=
errno));
> +                       }
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
>=20
> -       strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1);
> -       dev_config->device_id =3D VIRTIO_ID_NET;
> -       dev_config->vendor_id =3D 0;
> -       dev_config->features =3D features;
> -       dev_config->vq_num =3D total_queues;
> -       dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE);
> -       dev_config->config_size =3D sizeof(struct virtio_net_config);
> -       memcpy(dev_config->config, &vnet_config, sizeof(vnet_config));
> +               ret =3D ftruncate(reco_fd, sizeof(*reconnect_log));
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to truncate r=
econnect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       close(reco_fd);
> +                       goto out_ctrl_close;
> +               }
>=20
> -       ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config);
> -       if (ret < 0) {
> -               VHOST_CONFIG_LOG(name, ERR, "Failed to create VDUSE devic=
e: %s",
> -                               strerror(errno));
> -               goto out_free;
> +               reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT=
_READ | PROT_WRITE,
> +                                       MAP_SHARED, reco_fd, 0);
> +               close(reco_fd);
> +               if (reconnect_log =3D=3D MAP_FAILED) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon=
nect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               reconnect_log->version =3D 0;
> +
> +               dev_config =3D malloc(offsetof(struct vduse_dev_config, c=
onfig) +
> +                               sizeof(vnet_config));
> +               if (!dev_config) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to allocate V=
DUSE config");
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               vnet_config.max_virtqueue_pairs =3D max_queue_pairs;
> +               memset(dev_config, 0, sizeof(struct vduse_dev_config));
> +
> +               rte_strscpy(dev_config->name, name, VDUSE_NAME_MAX - 1);
> +               dev_config->device_id =3D VIRTIO_ID_NET;
> +               dev_config->vendor_id =3D 0;
> +               dev_config->features =3D features;
> +               dev_config->vq_num =3D total_queues;
> +               dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE);
> +               dev_config->config_size =3D sizeof(struct virtio_net_conf=
ig);
> +               memcpy(dev_config->config, &vnet_config, sizeof(vnet_conf=
ig));
> +
> +               ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config);
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to create VDU=
SE device: %s",
> +                                       strerror(errno));
> +                       goto out_free;
> +               }
> +
> +               memcpy(&reconnect_log->config, &vnet_config, sizeof(vnet_=
config));
> +               reconnect_log->nr_vrings =3D total_queues;
> +               free(dev_config);
> +               dev_config =3D NULL;
>        }
>=20
>        dev_fd =3D open(path, O_RDWR);
> @@ -519,10 +670,15 @@ vduse_device_create(const char *path, bool complian=
t_ol_flags)
>        strncpy(dev->ifname, path, IF_NAME_SZ - 1);
>        dev->vduse_ctrl_fd =3D control_fd;
>        dev->vduse_dev_fd =3D dev_fd;
> +       dev->reconnect_log =3D reconnect_log;
> +       if (reconnect)
> +               dev->status =3D dev->reconnect_log->status;
> +
>        vhost_setup_virtio_net(dev->vid, true, compliant_ol_flags, true, t=
rue);
>=20
>        for (i =3D 0; i < total_queues; i++) {
>                struct vduse_vq_config vq_cfg =3D { 0 };
> +               struct vhost_virtqueue *vq;
>=20
>                ret =3D alloc_vring_queue(dev, i);
>                if (ret) {
> @@ -530,6 +686,12 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                        goto out_dev_destroy;
>                }
>=20
> +               vq =3D dev->virtqueue[i];
> +               vq->reconnect_log =3D &reconnect_log->vring[i];
> +
> +               if (reconnect)
> +                       continue;
> +
>                vq_cfg.index =3D i;
>                vq_cfg.max_size =3D 1024;
>=20
> @@ -549,7 +711,28 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                goto out_dev_destroy;
>        }
>=20
> -       free(dev_config);
> +       if (reconnect && dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK)  {
> +               reco_fd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
> +               if (reco_fd < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to create rec=
o_fd: %s",
> +                                       strerror(errno));
> +                       ret =3D -1;
> +                       goto out_dev_destroy;
> +               }
> +
> +               ret =3D fdset_add(vduse.fdset, reco_fd, vduse_reconnect_h=
andler, NULL, dev);
> +               if (ret) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to add fd %d =
to vduse fdset",
> +                                       dev->vduse_dev_fd);

Should print reco_fd

> +                       goto out_dev_destroy;
> +               }
> +
> +               ret =3D eventfd_write(reco_fd, (eventfd_t)1);
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to write to r=
econnect eventfd");
> +                       goto out_dev_destroy;
> +               }
> +       }

Maybe I missed something, why we need to implement like this instead of dir=
ectly call vduse_device_start?

Thanks,
Chenbo

>=20
>        return 0;
>=20
> @@ -587,6 +770,9 @@ vduse_device_destroy(const char *path)
>        if (vid =3D=3D RTE_MAX_VHOST_DEVICE)
>                return -1;
>=20
> +       if (dev->reconnect_log)
> +               munmap(dev->reconnect_log, sizeof(*dev->reconnect_log));
> +
>        vduse_device_stop(dev);
>=20
>        fdset_del(vduse.fdset, dev->vduse_dev_fd);
> @@ -597,10 +783,26 @@ vduse_device_destroy(const char *path)
>        }
>=20
>        if (dev->vduse_ctrl_fd >=3D 0) {
> +               char reconnect_file[PATH_MAX];
> +
>                ret =3D ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name)=
;
> -               if (ret)
> +               if (ret) {
>                        VHOST_CONFIG_LOG(name, ERR, "Failed to destroy VDU=
SE device: %s",
>                                        strerror(errno));
> +               } else {
> +                       /*
> +                        * VDUSE device was no more attached to the vDPA =
bus,
> +                        * so we can remove the reconnect file.
> +                        */
> +                       ret =3D snprintf(reconnect_file, sizeof(reconnect=
_file), "%s/%s",
> +                                       vduse_reconnect_dir, name);
> +                       if (ret < 0 || ret =3D=3D sizeof(reconnect_file))
> +                               VHOST_CONFIG_LOG(name, ERR,
> +                                               "Failed to create vduse r=
econnect path name");
> +                       else
> +                               unlink(reconnect_file);
> +               }
> +
>                close(dev->vduse_ctrl_fd);
>                dev->vduse_ctrl_fd =3D -1;
>        }
> --
> 2.46.0
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id C8DBE4591A;
	Fri,  6 Sep 2024 09:15:10 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id B5F0F42EC6;
	Fri,  6 Sep 2024 09:15:10 +0200 (CEST)
Received: from NAM10-BN7-obe.outbound.protection.outlook.com
 (mail-bn7nam10on2064.outbound.protection.outlook.com [40.107.92.64])
 by mails.dpdk.org (Postfix) with ESMTP id 9684D42EC4
 for <dev@dpdk.org>; Fri,  6 Sep 2024 09:15:09 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=UwCQ0VKeTlMh7bc/ylDJJgZiRDbkoVSUzvma6gMQe2uM+/2WtnMNqN+IJbmbId3UqXfDlA7HLLyzLnKJ95OzruG1y9t0kmt80d54D+XmOp2xJWRvU52ztJ1v8KgaA9HHdshvy/KwCrgAYp2W8Xid2h4W9e96vCmh8IiGBIzUuv4hz8Aw46sVxm6wq5lYX63F7NwQr741noJFuCzZqungpbEqJ+pb2qGSUEkol0GwM5uXYOpiyH7k9Z552Er8vs5eIiEWmc5gm8/2mHyJYxrFSH5e2sHFqcGwDxjEDJN2rp6lL73QnJVac6Bvgs99KKClr4XVmlMHDss6OydhdInNyQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=;
 b=DLi2Z2Sff4JktiDClsMa8IuZK7LOu2BUEO6TppA0mk4rqZJCqa7q56vguLoBPP8L9QOk688CvI2HtajlclBws1Ezoquu38D/vS2SU6I2DcseRGUqLj85SCOTBzzOdzAAUnnfKUyCG8/Ls/afXxnbcA/GV/A1wUUYcC0Yvs05v47NRADSmLT/ugimfI48TBBKByY/iu5DH/UdE6v2UWemaR9r4zr+MJXJ8hzkUCEZJHpzllqM3B7+U0Oz97qopwPdLkB/0KV4Quxqh3IUinpO/KMTBeuz5xmaeck+HN3vrPK5jyJLBl/tB2PrlwoRrOSwISQ2G/xzgtZC4kWT3E6nkA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=;
 b=bRVNZ/0CwkW6SR/fG2PpiHfG87Ic9NR3uyQxaFrPUCWnlA+ydqj/eL4XMowLhE0EbjowJ7CgPXXHQ5A15va8D/Qt3q4058llZaxfECbeEgwJx6a6jvOZ4v8KJJm2WqAumijj4HDvzZdONVuHSoSkqH5dzoHYSgXACWy6P/uCJniX5yGEEOxcY4x6Gq82wJ7zFB5OuJQ33CfxYi2PiqrG2oztmlgxD8OriygcDCjr/Iz6YSX/pdbQtWrxFHA2j5LcYsnltxyaDw9XAiM4EsTsHFQYqjRqUQCTSNXLjGUaE8m6r6IWdj+6Le88bpQRR5apxIhx5LDu69NPXz5N7QJpiQ==
Received: from CYYPR12MB8992.namprd12.prod.outlook.com (2603:10b6:930:bc::21)
 by PH0PR12MB7471.namprd12.prod.outlook.com (2603:10b6:510:1e9::18)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.28; Fri, 6 Sep
 2024 07:15:06 +0000
Received: from CYYPR12MB8992.namprd12.prod.outlook.com
 ([fe80::a722:c2ca:42c7:aaed]) by CYYPR12MB8992.namprd12.prod.outlook.com
 ([fe80::a722:c2ca:42c7:aaed%7]) with mapi id 15.20.7918.024; Fri, 6 Sep 2024
 07:15:06 +0000
From: Chenbo Xia <chenbox@nvidia.com>
To: Maxime Coquelin <maxime.coquelin@redhat.com>
CC: "dev@dpdk.org" <dev@dpdk.org>, "david.marchand@redhat.com"
 <david.marchand@redhat.com>
Subject: Re: [PATCH 2/2] vhost: add reconnection support to VDUSE
Thread-Topic: [PATCH 2/2] vhost: add reconnection support to VDUSE
Thread-Index: AQHa/5+Tb2hkJpD2pkWfDp9/zoprPbJKWaMA
Date: Fri, 6 Sep 2024 07:15:06 +0000
Message-ID: <3543971A-6C8E-4F28-A7D4-F8C222F3C193@nvidia.com>
References: <20240905142602.1264345-1-maxime.coquelin@redhat.com>
 <20240905142602.1264345-3-maxime.coquelin@redhat.com>
In-Reply-To: <20240905142602.1264345-3-maxime.coquelin@redhat.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: CYYPR12MB8992:EE_|PH0PR12MB7471:EE_
x-ms-office365-filtering-correlation-id: 62cc0ecb-3afc-4e75-f6ff-08dcce43a295
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0; ARA:13230040|376014|366016|1800799024|38070700018;
x-microsoft-antispam-message-info: =?us-ascii?Q?jOPEweBQUeLeZ3YKN/N99zprZj5fOW5xHJVyUcizMShQkn2dj8aOJWVGpavl?=
 =?us-ascii?Q?1H8gmn4QRCCvN3wjyu8aJj7vwJ8pJLZctCVk1kvFWsN7RrX6C0DUFcx8dMWh?=
 =?us-ascii?Q?MnN97fWhXAX4Hbg2MFrAfq2hK6a3Mo1IN3xJN3S5LfHrFU1QL2hfHZIpylJm?=
 =?us-ascii?Q?kCt5sTfuu5yIhaUxJqKvtqO9iYEUd0H9TBuPi04YqyWxTC0sblnM884OODJw?=
 =?us-ascii?Q?Q5KEXaKKEtviYj5IBjQS2NZQp28eg7pW4ve3bZhzejIhz6ct63gfVl8hyTAl?=
 =?us-ascii?Q?CoVgOsqvFdqG3e7NV4gNpPVcz94TmrfzxVI46NxJzb6bsr1ILSdxF6WxJSlF?=
 =?us-ascii?Q?26In/kv7zud2ZGtihQpqxr2wUovy5cAI2FZQnTEhuGr4tAySBNHSRzD29FwC?=
 =?us-ascii?Q?qotwsiL3T6ngIen7/1K9p9U6s/k1xRRcAivhx4XMkwR0F8q6P4L7RmDTHkrk?=
 =?us-ascii?Q?HIEknu7YeGaspIe9qwwgyuhDAPsThnbJljsc5O97iyCBGygcULSuddqgJO7w?=
 =?us-ascii?Q?nKTQSADL6xj7BkkHr22XRe6emkySvcHJUQ7dpmyZAkuVA30Jbzze7caY4MxH?=
 =?us-ascii?Q?pbCJ9OhMr1sO7yeW+XY8VfmMlEAlhAC9+mmz1FqfEYFl0sSm2xye7npieJGM?=
 =?us-ascii?Q?pTiggZGrdM+e/JG8cMssrYLXSz20Q3Go6+B1stOniXqqHG5tGPh5wVGF7DRY?=
 =?us-ascii?Q?RMPbTEEbENAt+n+afz0CB5xv6RurkfJSu1qdPMDiHHgD4CuFiyf4Bnkudo6y?=
 =?us-ascii?Q?J0x3ZMCzOfGGneaNjkheXJcmbSn3w4Z9PxrU2m3DVpJ7TnYWyysOHjrCokQU?=
 =?us-ascii?Q?0+iy9ePQ5jQeiKfT8bE4/8nW8VKKlcw2FZRNr7fWO7c1Kq2FB1T6xn6vSX08?=
 =?us-ascii?Q?F4oXfIuL1+Ll0g2b78qqhyP0bNGyjS/nf5Tvz+9XeUFi9YPWyN8kbBGoIGc/?=
 =?us-ascii?Q?4uuqT9610Vkp1OSzVoz803jpFgeHDtjjtQ9c0oeETAnK7rkS01F8LCfqhUX4?=
 =?us-ascii?Q?gfm8CinEJL+B30Dz2oy5GoMCAX6iyrPH0NjNgCPay7H0z9aajqy/S8ZoIY0L?=
 =?us-ascii?Q?5KuOekKjkPterJWXSDDQxIA2MSXKmYP2gU4sGsrUcHGZRzmwkNIBEiGBBeYE?=
 =?us-ascii?Q?cSNxh+y7eXXOggZA1TP6o3MRIVpc7HhHF/2+5Wvs/sLLASsAWMRW5uNIFrX4?=
 =?us-ascii?Q?R5lFvAmcadqmrGdiDTQwOVVyDw1WL3WGY2WRvCE9HGGgCg51nk8J5S8OgSJ+?=
 =?us-ascii?Q?erINo/y3kCQNwqE6l3v3Wj4gHfpojcU6dt/eJuvBy8ZYVMYlSxBYtvy2jOEr?=
 =?us-ascii?Q?o2Iu2wxLyyxjbGQaw6UefYBudfMOrqBwljuz6kIC/bluQVpk8m5vp5k4VnF7?=
 =?us-ascii?Q?WlmP610MQETmLoA1VvyOXwsVXYg0kuz91zQjLv1JYGynqfxlcw=3D=3D?=
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:CYYPR12MB8992.namprd12.prod.outlook.com; PTR:; CAT:NONE;
 SFS:(13230040)(376014)(366016)(1800799024)(38070700018); DIR:OUT; SFP:1101; 
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?im17JPFFvKXies+GaMG0p3ApcU8wSqJNA2N1ybliZAOUdow4FFHYWMdxmCWL?=
 =?us-ascii?Q?O5mQKn2zZ1qLa6GwyaHPqOFFP1NwFkGSOXGOjhL7wxJXZFfqDAt5JePK+T7e?=
 =?us-ascii?Q?0x6Zio4gxPn3vRPJ/lVNYXl5CVgiwoh7uQJAWnq74LszKICu+752OAliFgFF?=
 =?us-ascii?Q?lVWiz1Ts7/NAILshTSsYHXiruuU+YMR+tAt61MuDRVt3UBmY6IFoMxLTkara?=
 =?us-ascii?Q?2XG9n6d/c1ONdqovzcwkDzzXLm7CeWsHyBmWEMwuOXIWBl/1JkTUJqti6iJ9?=
 =?us-ascii?Q?QzjtH6JxGWfGPPehXZy+IVm7B9Q/wiF3DRN9cIWQeKSIQJQQx1hLmlE+4A4b?=
 =?us-ascii?Q?2eXExWOO3Nnk4d2HZNHwgM9TqMpcHq7xO9HRSsj2mHBHSw47aQA0YcesoCjn?=
 =?us-ascii?Q?HezZIMR/7inXxu2z4qZyu9lt2cZffBRYkSupyjhg//KEitIW/udy9BxwutSW?=
 =?us-ascii?Q?TTEzYlbinmkYYmAItZ7/2ED23tuCrMV2hapu1m217/wNnwyFfuDO5i3HU4QP?=
 =?us-ascii?Q?Gilm90Mk+4dIGnVgfYkPd5ZCMUVQGgfSV4m4eHfksd+oHZkW7wQz+GDIfGSD?=
 =?us-ascii?Q?tg5tduccCJ2QqffjCA0+/+G7W+d8soorl26r1zdQMFX88HT+eojT/jd+3fZn?=
 =?us-ascii?Q?pYSIZwv6uc1fnSzoorBBfe6d8HyXWZBmIHkaBjldZUMwM48Yf2vxH1Em09Bb?=
 =?us-ascii?Q?+Yl/tGkt2D75Q3tKKN57yfhAcQrpkqvBpRXf6nxzUHm7A6PCSN5tOjqZCfEM?=
 =?us-ascii?Q?CXvJuBkcnIgX1Cy0gR1qnOc6+xgiJKyJNWV0ifVk+KQPYj83TYRZZSPl9cS+?=
 =?us-ascii?Q?nOfvHT6HwG4KK/DUz9EK4IJV484xJvSO720Qs/IIrFJxngi7da1AZIANem5r?=
 =?us-ascii?Q?lH5XG49D3SDK4QyuB9y8ldUFl5/KbIlBQFRpsT2OhzN7DHSkZX80kz586Hrj?=
 =?us-ascii?Q?JsVbcMsrMhy88YsB21ZHCLGl9WNdwqMuVPjrYMYYEf0si677SInr9qh4QcW/?=
 =?us-ascii?Q?PAfvRTK5HnELphhapbXqn7Hlne8ENvQzXyFvRAJr7ZQw5JEw7DiLEAA3HXOK?=
 =?us-ascii?Q?kw+AC993HeC0nzdreZ9wNziUnNiJHFE2FzYUgN14hzfOmYpcNhyzXDMipsKI?=
 =?us-ascii?Q?GRCVhFwLfyTaFCoYEHVwvze4u8R0feHrPN6khZ0BziEnwMDLPqyiCAhSjYwp?=
 =?us-ascii?Q?nvc3oeH/MUcnBSaZFu1tBOQDfp0SW0XJEp4RnfXs46fNLk15VXCQDCcyGu1N?=
 =?us-ascii?Q?5g9Z4JX2/vjSYxRrWShNsiN8qKLKu9g8/1re9Sr2kKAo6gY4udzn0GsWQ6qc?=
 =?us-ascii?Q?NGVSEmBj7lYNeraOH1Ff2vibRuiPyQcfj/jmJc/GRdWPFHg7mCkcaDAp4m2x?=
 =?us-ascii?Q?RbvxhlGlz5DPJ14zbOVaoSmwgiUO2MfaBRhtijRWDKou6F8u8dZbAbkPOcgK?=
 =?us-ascii?Q?I4+yCJUqh7wOA/0hKCzQmomIJydnMVoEa+cZBltkznUXGRnERqOAVVD3ziRa?=
 =?us-ascii?Q?v1gOqji+ouj7yayjmQpY2b+C8qUlf56b/k61coGEmGWvgJ7541yhebOlrdMp?=
 =?us-ascii?Q?5rliWYK3VUgztJS4OCijEjJHX7Za0jY03wBq9AWhwxcUDcDhc2DgzB81zX8O?=
 =?us-ascii?Q?Og=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A8C79D6374493347919E8E3C5432F2A5@namprd12.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CYYPR12MB8992.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 62cc0ecb-3afc-4e75-f6ff-08dcce43a295
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2024 07:15:06.5533 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: bxdSsuYWXbzi3vvTT7Xj+de7dh/eBJZ0gTUB40YDF6oxWSxAB5Ir73qoBr38G4ksUvn0nvvpNxqvyt8V4u91eg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7471
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Message-ID: <20240906071506.8d2Z9tzv5a1g9gKjZjs_i4VvFRq5KCUHgF0Gd_zjpMI@z>

Hi Maxime,

> On Sep 5, 2024, at 22:26, Maxime Coquelin <maxime.coquelin@redhat.com> wr=
ote:
>=20
> External email: Use caution opening links or attachments
>=20
>=20
> This patch enables VDUSE reconnection support making use of
> the newly introduced reconnection mechanism in Vhost
> library.
>=20
> At DPDK VDUSE device creation time, there are two
> possibilities:
> 1. The Kernel VDUSE device does not exist:
>  a. A reconnection file named after the VUDSE device name
>     is created in VDUSE tmpfs.
>  b. The file is truncated to 'struct vhost_reconnect_data'
>     size, and mmapped.
>  c. Negotiated features, Virtio status... are saved for
>     sanity checks at reconnect time.
> 2. The Kernel VDUSE device already exists:
>  a. Exit with failure if no reconnect file exists for
>     this device.
>  b. Open and mmap the reconnect file.
>  c. Perform sanity check to ensure features are compatible.
>  d. Restore virtqueues' available indexes at startup time.
>=20
> Then at runtime, the virtqueues' available index are logged by
> the Vhost reconnection mechanism.
>=20
> At DPDK VDUSE device destruction time, there are two
> possibilities:
> 1. The Kernel VDUSE device destruction succeed, which
>    means it is no more attached to the vDPA bus. The
>    reconnection file is unmapped and then removed.
> 2. The Kernel VDUSE device destruction failed, meaning it
>    is no more attached to the vDPA bus. The reconnection
>    file is unmapped but not removed to make possible later
>    reconnection.
>=20
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/vhost/vduse.c | 280 +++++++++++++++++++++++++++++++++++++++-------
> 1 file changed, 241 insertions(+), 39 deletions(-)
>=20
> diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c
> index c66602905c..bd0e492d62 100644
> --- a/lib/vhost/vduse.c
> +++ b/lib/vhost/vduse.c
> @@ -136,7 +136,7 @@ vduse_control_queue_event(int fd, void *arg, int *rem=
ove __rte_unused)
> }
>=20
> static void
> -vduse_vring_setup(struct virtio_net *dev, unsigned int index)
> +vduse_vring_setup(struct virtio_net *dev, unsigned int index, bool recon=
nect)
> {
>        struct vhost_virtqueue *vq =3D dev->virtqueue[index];
>        struct vhost_vring_addr *ra =3D &vq->ring_addrs;
> @@ -152,6 +152,19 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i=
nt index)
>                return;
>        }
>=20
> +       if (reconnect) {
> +               vq->last_avail_idx =3D vq->reconnect_log->last_avail_idx;
> +               vq->last_used_idx =3D vq->reconnect_log->last_avail_idx;
> +       } else {
> +               vq->last_avail_idx =3D vq_info.split.avail_index;
> +               vq->last_used_idx =3D vq_info.split.avail_index;
> +       }
> +       vq->size =3D vq_info.num;
> +       vq->ready =3D true;
> +       vq->enabled =3D vq_info.ready;
> +       ra->desc_user_addr =3D vq_info.desc_addr;
> +       ra->avail_user_addr =3D vq_info.driver_addr;
> +       ra->used_user_addr =3D vq_info.device_addr;
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "VQ %u info:", index);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tnum: %u", vq_info.num);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdesc_addr: %llx",
> @@ -162,15 +175,6 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i=
nt index)
>                        (unsigned long long)vq_info.device_addr);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq_info.spl=
it.avail_index);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tready: %u", vq_info.ready);
> -
> -       vq->last_avail_idx =3D vq_info.split.avail_index;
> -       vq->size =3D vq_info.num;
> -       vq->ready =3D true;
> -       vq->enabled =3D vq_info.ready;
> -       ra->desc_user_addr =3D vq_info.desc_addr;
> -       ra->avail_user_addr =3D vq_info.driver_addr;
> -       ra->used_user_addr =3D vq_info.device_addr;
> -
>        vq->kickfd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
>        if (vq->kickfd < 0) {
>                VHOST_CONFIG_LOG(dev->ifname, ERR, "Failed to init kickfd =
for VQ %u: %s",
> @@ -267,7 +271,7 @@ vduse_vring_cleanup(struct virtio_net *dev, unsigned =
int index)
> }
>=20
> static void
> -vduse_device_start(struct virtio_net *dev)
> +vduse_device_start(struct virtio_net *dev, bool reconnect)
> {
>        unsigned int i, ret;
>=20
> @@ -287,6 +291,15 @@ vduse_device_start(struct virtio_net *dev)
>                return;
>        }
>=20
> +       if (reconnect && dev->features !=3D dev->reconnect_log->features)=
 {
> +               VHOST_CONFIG_LOG(dev->ifname, ERR,
> +                               "Mismatch between reconnect file features=
 0x%" PRIx64 " & device features 0x%" PRIx64,

Checkpatch reports long line

> +                               dev->reconnect_log->features, dev->featur=
es);
> +               return;
> +       }
> +
> +       dev->reconnect_log->features =3D dev->features;
> +
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "Negotiated Virtio features: 0=
x%" PRIx64,
>                dev->features);
>=20
> @@ -300,7 +313,7 @@ vduse_device_start(struct virtio_net *dev)
>        }
>=20
>        for (i =3D 0; i < dev->nr_vring; i++)
> -               vduse_vring_setup(dev, i);
> +               vduse_vring_setup(dev, i, reconnect);
>=20
>        dev->flags |=3D VIRTIO_DEV_READY;
>=20
> @@ -373,6 +386,7 @@ vduse_events_handler(int fd, void *arg, int *remove _=
_rte_unused)
>                                req.s.status);
>                old_status =3D dev->status;
>                dev->status =3D req.s.status;
> +               dev->reconnect_log->status =3D dev->status;
>                resp.result =3D VDUSE_REQ_RESULT_OK;
>                break;
>        case VDUSE_UPDATE_IOTLB:
> @@ -398,7 +412,7 @@ vduse_events_handler(int fd, void *arg, int *remove _=
_rte_unused)
>=20
>        if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) {
>                if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK)
> -                       vduse_device_start(dev);
> +                       vduse_device_start(dev, false);
>                else
>                        vduse_device_stop(dev);
>        }
> @@ -407,10 +421,64 @@ vduse_events_handler(int fd, void *arg, int *remove=
 __rte_unused)
>                        vduse_req_id_to_str(req.type), req.type);
> }
>=20
> +static char vduse_reconnect_dir[PATH_MAX];
> +static bool vduse_reconnect_path_set;
> +
> +static int
> +vduse_reconnect_path_init(void)
> +{
> +       const char *directory;
> +       int ret;
> +
> +       /* from RuntimeDirectory=3D see systemd.exec */
> +       directory =3D getenv("RUNTIME_DIRECTORY");
> +       if (directory =3D=3D NULL) {
> +               /*
> +                * Used standard convention defined in
> +                * XDG Base Directory Specification and
> +                * Filesystem Hierarchy Standard.
> +                */
> +               if (getuid() =3D=3D 0)
> +                       directory =3D "/var/run";
> +               else
> +                       directory =3D getenv("XDG_RUNTIME_DIR") ? : "/tmp=
";
> +       }
> +
> +       ret =3D snprintf(vduse_reconnect_dir, sizeof(vduse_reconnect_dir)=
, "%s/dpdk/vduse",
> +                       directory);
> +       if (ret < 0 || ret =3D=3D sizeof(vduse_reconnect_dir)) {
> +               VHOST_CONFIG_LOG("vduse", ERR, "Error creating VDUSE reco=
nnect path name");
> +               return -1;
> +       }
> +
> +       ret =3D mkdir(vduse_reconnect_dir, 0700);
> +       if (ret < 0 && errno !=3D EEXIST) {
> +               VHOST_CONFIG_LOG("vduse", ERR, "Error creating '%s': %s",
> +                               vduse_reconnect_dir, strerror(errno));
> +               return -1;
> +       }
> +
> +       VHOST_CONFIG_LOG("vduse", INFO, "Created VDUSE reconnect director=
y in %s",
> +                       vduse_reconnect_dir);
> +
> +       return 0;
> +}
> +
> +static void
> +vduse_reconnect_handler(int fd, void *arg, int *remove)
> +{
> +       struct virtio_net *dev =3D arg;
> +
> +       vduse_device_start(dev, true);
> +
> +       close(fd);
> +       *remove =3D 1;
> +}
> +
> int
> vduse_device_create(const char *path, bool compliant_ol_flags)
> {
> -       int control_fd, dev_fd, vid, ret;
> +       int control_fd, dev_fd, vid, ret, reco_fd;
>        uint32_t i, max_queue_pairs, total_queues;
>        struct virtio_net *dev;
>        struct virtio_net_config vnet_config =3D {{ 0 }};
> @@ -418,6 +486,9 @@ vduse_device_create(const char *path, bool compliant_=
ol_flags)
>        uint64_t features;
>        struct vduse_dev_config *dev_config =3D NULL;
>        const char *name =3D path + strlen("/dev/vduse/");
> +       char reconnect_file[PATH_MAX];
> +       struct vhost_reconnect_data *reconnect_log =3D NULL;
> +       bool reconnect =3D false;
>=20
>        if (vduse.fdset =3D=3D NULL) {
>                vduse.fdset =3D fdset_init("vduse-evt");
> @@ -427,6 +498,20 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                }
>        }
>=20
> +       if (vduse_reconnect_path_set =3D=3D false) {
> +               if (vduse_reconnect_path_init() < 0) {
> +                       VHOST_CONFIG_LOG(path, ERR, "failed to initialize=
 reconnect path");
> +                       return -1;
> +               }
> +               vduse_reconnect_path_set =3D true;
> +       }
> +
> +       ret =3D snprintf(reconnect_file, sizeof(reconnect_file), "%s/%s",=
 vduse_reconnect_dir, name);
> +       if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) {
> +               VHOST_CONFIG_LOG(name, ERR, "Failed to create vduse recon=
nect path name");
> +               return -1;
> +       }
> +
>        control_fd =3D open(VDUSE_CTRL_PATH, O_RDWR);
>        if (control_fd < 0) {
>                VHOST_CONFIG_LOG(name, ERR, "Failed to open %s: %s",
> @@ -441,14 +526,6 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                goto out_ctrl_close;
>        }
>=20
> -       dev_config =3D malloc(offsetof(struct vduse_dev_config, config) +
> -                       sizeof(vnet_config));
> -       if (!dev_config) {
> -               VHOST_CONFIG_LOG(name, ERR, "Failed to allocate VDUSE con=
fig");
> -               ret =3D -1;
> -               goto out_ctrl_close;
> -       }
> -
>        ret =3D rte_vhost_driver_get_features(path, &features);
>        if (ret < 0) {
>                VHOST_CONFIG_LOG(name, ERR, "Failed to get backend feature=
s");
> @@ -469,23 +546,97 @@ vduse_device_create(const char *path, bool complian=
t_ol_flags)
>        else
>                total_queues +=3D 1; /* Includes ctrl queue */
>=20
> -       vnet_config.max_virtqueue_pairs =3D max_queue_pairs;
> -       memset(dev_config, 0, sizeof(struct vduse_dev_config));
> +       if (access(path, F_OK) =3D=3D 0) {
> +               VHOST_CONFIG_LOG(name, INFO, "Device already exists, reco=
nnecting...");
> +               reconnect =3D true;
> +
> +               reco_fd =3D open(reconnect_file, O_RDWR, 0600);
> +               if (reco_fd < 0) {
> +                       if (errno =3D=3D ENOENT) {
> +                               VHOST_CONFIG_LOG(name, ERR, "Missing reco=
nnect file (%s)",
> +                                               reconnect_file);
> +                       } else {
> +                               VHOST_CONFIG_LOG(name, ERR, "Failed to op=
en reconnect file %s (%s)",
> +                                               reconnect_file, strerror(=
errno));
> +                       }

Seems no {} is needed for if-else?

> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT=
_READ | PROT_WRITE,
> +                               MAP_SHARED, reco_fd, 0);
> +               close(reco_fd);
> +               if (reconnect_log =3D=3D MAP_FAILED) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon=
nect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +       } else {
> +               reco_fd =3D open(reconnect_file, O_CREAT | O_EXCL | O_RDW=
R, 0600);
> +               if (reco_fd < 0) {
> +                       if (errno =3D=3D EEXIST) {
> +                               VHOST_CONFIG_LOG(name, ERR, "Reconnect fi=
le %s exists but not the device",
> +                                               reconnect_file);
> +                       } else {
> +                               VHOST_CONFIG_LOG(name, ERR, "Failed to op=
en reconnect file %s (%s)",
> +                                               reconnect_file, strerror(=
errno));
> +                       }
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
>=20
> -       strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1);
> -       dev_config->device_id =3D VIRTIO_ID_NET;
> -       dev_config->vendor_id =3D 0;
> -       dev_config->features =3D features;
> -       dev_config->vq_num =3D total_queues;
> -       dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE);
> -       dev_config->config_size =3D sizeof(struct virtio_net_config);
> -       memcpy(dev_config->config, &vnet_config, sizeof(vnet_config));
> +               ret =3D ftruncate(reco_fd, sizeof(*reconnect_log));
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to truncate r=
econnect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       close(reco_fd);
> +                       goto out_ctrl_close;
> +               }
>=20
> -       ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config);
> -       if (ret < 0) {
> -               VHOST_CONFIG_LOG(name, ERR, "Failed to create VDUSE devic=
e: %s",
> -                               strerror(errno));
> -               goto out_free;
> +               reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT=
_READ | PROT_WRITE,
> +                                       MAP_SHARED, reco_fd, 0);
> +               close(reco_fd);
> +               if (reconnect_log =3D=3D MAP_FAILED) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon=
nect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               reconnect_log->version =3D 0;
> +
> +               dev_config =3D malloc(offsetof(struct vduse_dev_config, c=
onfig) +
> +                               sizeof(vnet_config));
> +               if (!dev_config) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to allocate V=
DUSE config");
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               vnet_config.max_virtqueue_pairs =3D max_queue_pairs;
> +               memset(dev_config, 0, sizeof(struct vduse_dev_config));
> +
> +               rte_strscpy(dev_config->name, name, VDUSE_NAME_MAX - 1);
> +               dev_config->device_id =3D VIRTIO_ID_NET;
> +               dev_config->vendor_id =3D 0;
> +               dev_config->features =3D features;
> +               dev_config->vq_num =3D total_queues;
> +               dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE);
> +               dev_config->config_size =3D sizeof(struct virtio_net_conf=
ig);
> +               memcpy(dev_config->config, &vnet_config, sizeof(vnet_conf=
ig));
> +
> +               ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config);
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to create VDU=
SE device: %s",
> +                                       strerror(errno));
> +                       goto out_free;
> +               }
> +
> +               memcpy(&reconnect_log->config, &vnet_config, sizeof(vnet_=
config));
> +               reconnect_log->nr_vrings =3D total_queues;
> +               free(dev_config);
> +               dev_config =3D NULL;
>        }
>=20
>        dev_fd =3D open(path, O_RDWR);
> @@ -519,10 +670,15 @@ vduse_device_create(const char *path, bool complian=
t_ol_flags)
>        strncpy(dev->ifname, path, IF_NAME_SZ - 1);
>        dev->vduse_ctrl_fd =3D control_fd;
>        dev->vduse_dev_fd =3D dev_fd;
> +       dev->reconnect_log =3D reconnect_log;
> +       if (reconnect)
> +               dev->status =3D dev->reconnect_log->status;
> +
>        vhost_setup_virtio_net(dev->vid, true, compliant_ol_flags, true, t=
rue);
>=20
>        for (i =3D 0; i < total_queues; i++) {
>                struct vduse_vq_config vq_cfg =3D { 0 };
> +               struct vhost_virtqueue *vq;
>=20
>                ret =3D alloc_vring_queue(dev, i);
>                if (ret) {
> @@ -530,6 +686,12 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                        goto out_dev_destroy;
>                }
>=20
> +               vq =3D dev->virtqueue[i];
> +               vq->reconnect_log =3D &reconnect_log->vring[i];
> +
> +               if (reconnect)
> +                       continue;
> +
>                vq_cfg.index =3D i;
>                vq_cfg.max_size =3D 1024;
>=20
> @@ -549,7 +711,28 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                goto out_dev_destroy;
>        }
>=20
> -       free(dev_config);
> +       if (reconnect && dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK)  {
> +               reco_fd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
> +               if (reco_fd < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to create rec=
o_fd: %s",
> +                                       strerror(errno));
> +                       ret =3D -1;
> +                       goto out_dev_destroy;
> +               }
> +
> +               ret =3D fdset_add(vduse.fdset, reco_fd, vduse_reconnect_h=
andler, NULL, dev);
> +               if (ret) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to add fd %d =
to vduse fdset",
> +                                       dev->vduse_dev_fd);

Should print reco_fd

> +                       goto out_dev_destroy;
> +               }
> +
> +               ret =3D eventfd_write(reco_fd, (eventfd_t)1);
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to write to r=
econnect eventfd");
> +                       goto out_dev_destroy;
> +               }
> +       }

Maybe I missed something, why we need to implement like this instead of dir=
ectly call vduse_device_start?

Thanks,
Chenbo

>=20
>        return 0;
>=20
> @@ -587,6 +770,9 @@ vduse_device_destroy(const char *path)
>        if (vid =3D=3D RTE_MAX_VHOST_DEVICE)
>                return -1;
>=20
> +       if (dev->reconnect_log)
> +               munmap(dev->reconnect_log, sizeof(*dev->reconnect_log));
> +
>        vduse_device_stop(dev);
>=20
>        fdset_del(vduse.fdset, dev->vduse_dev_fd);
> @@ -597,10 +783,26 @@ vduse_device_destroy(const char *path)
>        }
>=20
>        if (dev->vduse_ctrl_fd >=3D 0) {
> +               char reconnect_file[PATH_MAX];
> +
>                ret =3D ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name)=
;
> -               if (ret)
> +               if (ret) {
>                        VHOST_CONFIG_LOG(name, ERR, "Failed to destroy VDU=
SE device: %s",
>                                        strerror(errno));
> +               } else {
> +                       /*
> +                        * VDUSE device was no more attached to the vDPA =
bus,
> +                        * so we can remove the reconnect file.
> +                        */
> +                       ret =3D snprintf(reconnect_file, sizeof(reconnect=
_file), "%s/%s",
> +                                       vduse_reconnect_dir, name);
> +                       if (ret < 0 || ret =3D=3D sizeof(reconnect_file))
> +                               VHOST_CONFIG_LOG(name, ERR,
> +                                               "Failed to create vduse r=
econnect path name");
> +                       else
> +                               unlink(reconnect_file);
> +               }
> +
>                close(dev->vduse_ctrl_fd);
>                dev->vduse_ctrl_fd =3D -1;
>        }
> --
> 2.46.0
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id A4E3E4591C;
	Fri,  6 Sep 2024 11:20:29 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 54D8142E12;
	Fri,  6 Sep 2024 11:20:29 +0200 (CEST)
Received: from NAM11-DM6-obe.outbound.protection.outlook.com
 (mail-dm6nam11on2043.outbound.protection.outlook.com [40.107.223.43])
 by mails.dpdk.org (Postfix) with ESMTP id 5E5C64029E
 for <dev@dpdk.org>; Fri,  6 Sep 2024 11:20:27 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=F1Dy58HVifXb9l4uvT1f6R1e61WZVYaOdVoOBlcEeHVFf6vP3VCliXsC+dCPUDr+E/Y9Pq+xmi/89p5go+9CG3R4iVnRjOHuBRJFi1cXbgkXARtoP90EjuAfplijqXtrNqyFC8+XBIvdqSgouLEJfMDG2IwEFje+JTXzdZaTaIHb2VO+lWzWY8g8xjSOnGbiCGVYbizfgIImglTsg4kMNu5xbr5p8q8xwvrYQdCb6f6O1V/AWk+L/Z5D9jsGAHvvllJnTmHLRdenBV44B8MIWGYO9lIDHbQUxAmUSOksvNyVk0K04WN4q3oZNe0VzuQwhdFlUGOMTXKW9urWwX4yuw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=;
 b=N8hnwPzDJKmm59Y0DAn8hQ+zeMotGiCptwbVTsUHGulq05UeHFM9Go3MM7dAakMJcj9aotDW3DTh1mjuWctdMWaVHkf4lAnH+xOCP3+NC+QybHjHTx4MHhzB/NoCdeY8JndI1CA4xK+DX8R4thKvCF0CphRN1zBfaRXXPXSbxj0078+M1SSSxyrbmX/XFeFdmeseBkwO358X2AyFyWFKKXF/cJ97AajViMZnDoqEPZCSLKBLd4GirW4ySHgCd5NPhgeqiYVI/jRZARwEnNJkNICFWz8ffSAXDwmo7ypexRwhWvR8gE+6kq5N8vCmGV2ocSpxULwb8EYwWFUR40Txuw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=q508NGhTkBF5aMmuyM4PAlvLQ4Ucdec0RYDrRNqckXE=;
 b=E4paWSStWcYvZT6cxq5s4LFZR1vJhCwnkDDjuFm2bFDZyYVl0c4i0KlZQ6JdcN9yGI+GQPf1GLr+SaBU/t8ru80VBDjUlYMV4iGgXtsPqAHXQ2JXuLHQs9B28olJW/lrxxWw5x6T/M2fMx0a8g8cxwCJBsrXTeyLEpxk8c7tiddT1k511diF7dfVX2WfsNCx+t79wmxT6xNaa0FU7GKbbi/ruYzZ4dbWVH81052QELbL4VfIHe0M/sxvKf6sVXzEMbR0rk+RxLAZDwLkBiuHdXMa9uldXgeN/UzVS1FQO+wjDaTClOiiuEVtMtr5GiKJTN4R1magbAq/ZX+KFXC24A==
Received: from IA0PR12MB8983.namprd12.prod.outlook.com (2603:10b6:208:490::17)
 by MW4PR12MB6974.namprd12.prod.outlook.com (2603:10b6:303:207::18)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.17; Fri, 6 Sep
 2024 09:20:24 +0000
Received: from IA0PR12MB8983.namprd12.prod.outlook.com
 ([fe80::68e7:8370:b169:fa0e]) by IA0PR12MB8983.namprd12.prod.outlook.com
 ([fe80::68e7:8370:b169:fa0e%6]) with mapi id 15.20.7939.017; Fri, 6 Sep 2024
 09:20:22 +0000
From: Chenbo Xia <chenbox@nvidia.com>
To: Maxime Coquelin <maxime.coquelin@redhat.com>
CC: "dev@dpdk.org" <dev@dpdk.org>, "david.marchand@redhat.com"
 <david.marchand@redhat.com>
Subject: Re: [PATCH 2/2] vhost: add reconnection support to VDUSE
Thread-Topic: [PATCH 2/2] vhost: add reconnection support to VDUSE
Thread-Index: AQHa/5+Tb2hkJpD2pkWfDp9/zoprPbJKWaMA
Date: Fri, 6 Sep 2024 09:20:22 +0000
Message-ID: <3543971A-6C8E-4F28-A7D4-F8C222F3C193@nvidia.com>
References: <20240905142602.1264345-1-maxime.coquelin@redhat.com>
 <20240905142602.1264345-3-maxime.coquelin@redhat.com>
In-Reply-To: <20240905142602.1264345-3-maxime.coquelin@redhat.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: IA0PR12MB8983:EE_|MW4PR12MB6974:EE_
x-ms-office365-filtering-correlation-id: 09b5c1d9-9a7f-4eed-0827-08dcce552246
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0; ARA:13230040|1800799024|376014|366016|38070700018;
x-microsoft-antispam-message-info: =?us-ascii?Q?cvDLqQJ9mQPn8WMw5Mb+73qUM2ts9o+aGGp/3344vS+6tgo85PjB4xLscVr6?=
 =?us-ascii?Q?L1jH7tRmr0Srz7UenYd8G+PuZydzRMH5XG2WDpifc6zP0w5/7ex9lH3HOAHt?=
 =?us-ascii?Q?jOANCL3ocvawamBtijrPvhanzU+mAIUMt3BIc0/MiY6JHdaAmCycdNmYF86X?=
 =?us-ascii?Q?4y6pwWPjtCT9/bDFE6v0VLe9+yoEPtV0enNUYY53ZICVv7uhvkxqQ5LGC0nO?=
 =?us-ascii?Q?svVyEdT8T+S4bitlSl7lLKGShqLobfAteAkjI6CQ4ARp7muImnMwW8dIjiIu?=
 =?us-ascii?Q?SiKJZCaER31lFdNlmUUslF2CFFY/c3Bmu4OS6urnSv6nqvPYbJu0ZkU6shlw?=
 =?us-ascii?Q?eCnzKG3rpN95Hv2FndRu+yl/0kXcDPR4qCTyFh/mr6a1JnCilRiboQIs6RjR?=
 =?us-ascii?Q?2RsQeMccAGuZQSr5ApYelhWeUjqLHSZYNQcL9dRwMLmSxadigaTNjPewIiA9?=
 =?us-ascii?Q?dUJamyUoa17RfiaBUnIaUp2f+uO2Cs2wuHSUh7lxkNCI3f/XPczhfa4tiWsd?=
 =?us-ascii?Q?UZPF4LSO2A9SwI2J60WVsmJ+Bbi5335qpVlJbJXsEyT7hlR6nPNh0Ed+6wJX?=
 =?us-ascii?Q?O9ybcTcbfcoiHbTNpe8fadW3ld/QugtV+q4fajuIf/tpAmo8uCUTKf0klJMd?=
 =?us-ascii?Q?bmzhl7I3CzGy8Stu7azOJHJ6oCSSEQ/O8Mr4H+K7YpCo0sRfDHFFIv3YFLxO?=
 =?us-ascii?Q?JQN4r3lRBli6i+09rllRNLZ6tHmZNkgvp/Hp7pxTp2rNq1+vztNpFjK6M6eB?=
 =?us-ascii?Q?eCQKbkz6G5WoQz0kgVGSbEbk9DkNShnb/fzoRpcl+SampHoue+FVG+K41rIk?=
 =?us-ascii?Q?J8u0FaFGK82sdzghN/uMLthah+Gb0RspkXjsHGVgk1Ms4W3+qz5jjr2vPbWa?=
 =?us-ascii?Q?XBQ/9pDTbRyts5qlwiErzHnGX5LSclwLV7pyGVhT5sjgYwr+3CNaa7kyR0EA?=
 =?us-ascii?Q?KqbzJg+SBCmiNfL/zEYLr4z6qMP1z1SnYGE2ciEauh/v0BYUTEzStXVeTjAV?=
 =?us-ascii?Q?ayC9CiOuvf6c509+vQkDIq71lXK0mIApV5SRTgPPQ8rxMG9FeeRRrSdmsRmG?=
 =?us-ascii?Q?O+sHlkFBK2SgUCCBbzzMX4boonlJmxU4cFh2S1I3BOF5+f7ZakIEV9QsPskH?=
 =?us-ascii?Q?3n+jMZNkIKEGZHObp+MLqP4KeBeY2JcOpu1QMzT0/FtX/JYevjUTJ9o4Dzck?=
 =?us-ascii?Q?im618EbMWp7O3LGy/ixrfBASkHVVde4Jnv3zaWmfkfZ10YXxIdq/O0josv+1?=
 =?us-ascii?Q?yBDn81UUnBshgwRG1tw7Bw7RHkBh6cd0+5hHnRAZX+paViurBX8ISA8ADD25?=
 =?us-ascii?Q?O6VhVxMn9aPVX8L/i8au6+GDkL7/FVcWZvKUXVnb9uOF0/KCzEt8Y12wYzu9?=
 =?us-ascii?Q?xW0lwYGxhge9vUDBjDSBRgAbHzV9OWPTA0rqSGfExua+hFx2Ng=3D=3D?=
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:IA0PR12MB8983.namprd12.prod.outlook.com; PTR:; CAT:NONE;
 SFS:(13230040)(1800799024)(376014)(366016)(38070700018); DIR:OUT; SFP:1101; 
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LR+XRBUxjUbu0dWkQ7uognYYojvloSJy0xmbWX4M4uwuYLIHxOerfkEuMsi5?=
 =?us-ascii?Q?0UH6RFbFRF7XMRpma6YuRaBlmhFSGH5TJ/0kcTICw4cOX6Qq1hSwBk4Cm/20?=
 =?us-ascii?Q?CjUkqxAW9KrwK6SeYDdXv8tHAS3eRCc5yKXsZ02Fhty7vqJ7EM/IQCzGb/Bs?=
 =?us-ascii?Q?XC/8j5+0S9NA4gvBc8kPs76FEyOIVAm5dggh7N9n1LsUfcpWSBSH4zW0YDgU?=
 =?us-ascii?Q?YSCBH0onqcHAwe4jYpvQh5Qlv5PTwfDVZ7VVNAhqkjkUlDuZESSLMiCtEV7o?=
 =?us-ascii?Q?0Qf2bzbm1PE+VsaYc1GYsPmji4qLbUfSVkk4rz0S6uFeEppihYg+PzjLqMFF?=
 =?us-ascii?Q?f+Vb8UoBWyg3qyJ/ouKt/n4eRqUp0eQ9RWr6Hs2OsA+N8P1xwjnUxzfByRdh?=
 =?us-ascii?Q?CWx50a/xMg3suaa7/gLDiSRTJTsTTVbXAYRadX8cmGWk6+NAZMFdnYSjaHMG?=
 =?us-ascii?Q?7iSkLFuKoQKKqirc98cFwgmVnj6BHBHccjjx/2aErGt2wHhFpokutX5zrDwb?=
 =?us-ascii?Q?FhVH6JGXvR+wIjxkZJP1yEBQz6bRMGvkgCHgyhwRK8N3RbVckqr5l/d9N6kp?=
 =?us-ascii?Q?UZoQLoQbz7rW2XmMFzuw/g7NjdB+lnlCx8SFROYDe0ULXV39wW8nFLaGopaV?=
 =?us-ascii?Q?Vc0wX6NN/ByQLKLoSDijbBXwezP5D14QY4E+f/rcaFPF0MkfxSddZjyh1VEY?=
 =?us-ascii?Q?OSTRpuSxcOx7fja95sRjpCbpVpLxuDhX9fQsHViS6OcOGjZsXIU0cXBB/nO+?=
 =?us-ascii?Q?OfWb5hWvC+OaoDWq03IOcm/FGkDWkrTwKTy/ckpN8JFn82wCdQDSxiHyA4tt?=
 =?us-ascii?Q?LJLTOJyUV2sPOyTYAcTTWSa+k0FcPGXOdgq7SBlUV7oXGWhp+ZpzyIjFdof9?=
 =?us-ascii?Q?Vp1qFvLD3stDCw9Hlo8ftcuafkzX+bQ5sShPUQ2SrMxXCGVNpBwNaxN9MVNC?=
 =?us-ascii?Q?35Kbndg+i4Y4bHv4EkNlvY23ILlM64tSx3OmknLm8k5BQxc2P7a43H2BIt0n?=
 =?us-ascii?Q?TEdMiV+r+BoZf2PGyt6u7KZrvAc/3fhOjiyqXsKygY5lF9SgB9cykemXvdQa?=
 =?us-ascii?Q?FrJcaaju70TigjITRGaXf5QNHmwjlCq3ORSY9yx+XmAFJRrf2obvYsPkJdlD?=
 =?us-ascii?Q?axvXLW00z8GaltmQUn6AjvynZ9S2flm3KqdGZ6YcxLftZUIeXXdYG9nckuq0?=
 =?us-ascii?Q?2SfSnJw4RC2shn+EXJ6VEqNQQjdq+c0lFtwTSbgOG8adeNuBTPa0LPDKiiX5?=
 =?us-ascii?Q?NDJhfiI2ATvskx/8ujIt+GOf0Wc3VxUsHKHH6ooUi67ik9sSWJhw/lDHCSNc?=
 =?us-ascii?Q?DZMd9PiqTklrn5hq2C9L72lZkS0xVBIEOnLixs09bliTbCHxtA/lnyDUkf9G?=
 =?us-ascii?Q?i9p9TR/dCjyulY7XZlpUz9TACVed1yci4fp4UNqHL+E23sgecFPCQujI/kkK?=
 =?us-ascii?Q?JACZCmCaKzTVoV1VMx1ZnZf/U1Li+3BZQI7YfS+Sm4fbBCjW8ZNRrmZ4Sqm0?=
 =?us-ascii?Q?ESbKzMVrENhDW/XS8/4dzjXzBpnLszRbqV7KNOPBm2bOBGpHbu46uR34R3RQ?=
 =?us-ascii?Q?J4F6aldm3MwgzA+B4Uruv6+XE+Z9WEPQ4N2ojmzY?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <653EA2332AD28C47B1129265C455B270@namprd12.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: IA0PR12MB8983.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 09b5c1d9-9a7f-4eed-0827-08dcce552246
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2024 09:20:22.2026 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: xEtwcwEagKYllpZPd0GhD2t/22Lgq9Cgvxfzo4tAYTXZYEWBMlwscIoczPTjOrWkrREzTB3LM6cz3xzvimlisA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6974
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Message-ID: <20240906092022.huyWoLEZ6iBOwKo3R_Z8BK-wR98RkulzEY2Mz8kSPXw@z>

Hi Maxime,

> On Sep 5, 2024, at 22:26, Maxime Coquelin <maxime.coquelin@redhat.com> wr=
ote:
>=20
> External email: Use caution opening links or attachments
>=20
>=20
> This patch enables VDUSE reconnection support making use of
> the newly introduced reconnection mechanism in Vhost
> library.
>=20
> At DPDK VDUSE device creation time, there are two
> possibilities:
> 1. The Kernel VDUSE device does not exist:
>  a. A reconnection file named after the VUDSE device name
>     is created in VDUSE tmpfs.
>  b. The file is truncated to 'struct vhost_reconnect_data'
>     size, and mmapped.
>  c. Negotiated features, Virtio status... are saved for
>     sanity checks at reconnect time.
> 2. The Kernel VDUSE device already exists:
>  a. Exit with failure if no reconnect file exists for
>     this device.
>  b. Open and mmap the reconnect file.
>  c. Perform sanity check to ensure features are compatible.
>  d. Restore virtqueues' available indexes at startup time.
>=20
> Then at runtime, the virtqueues' available index are logged by
> the Vhost reconnection mechanism.
>=20
> At DPDK VDUSE device destruction time, there are two
> possibilities:
> 1. The Kernel VDUSE device destruction succeed, which
>    means it is no more attached to the vDPA bus. The
>    reconnection file is unmapped and then removed.
> 2. The Kernel VDUSE device destruction failed, meaning it
>    is no more attached to the vDPA bus. The reconnection
>    file is unmapped but not removed to make possible later
>    reconnection.
>=20
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/vhost/vduse.c | 280 +++++++++++++++++++++++++++++++++++++++-------
> 1 file changed, 241 insertions(+), 39 deletions(-)
>=20
> diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c
> index c66602905c..bd0e492d62 100644
> --- a/lib/vhost/vduse.c
> +++ b/lib/vhost/vduse.c
> @@ -136,7 +136,7 @@ vduse_control_queue_event(int fd, void *arg, int *rem=
ove __rte_unused)
> }
>=20
> static void
> -vduse_vring_setup(struct virtio_net *dev, unsigned int index)
> +vduse_vring_setup(struct virtio_net *dev, unsigned int index, bool recon=
nect)
> {
>        struct vhost_virtqueue *vq =3D dev->virtqueue[index];
>        struct vhost_vring_addr *ra =3D &vq->ring_addrs;
> @@ -152,6 +152,19 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i=
nt index)
>                return;
>        }
>=20
> +       if (reconnect) {
> +               vq->last_avail_idx =3D vq->reconnect_log->last_avail_idx;
> +               vq->last_used_idx =3D vq->reconnect_log->last_avail_idx;
> +       } else {
> +               vq->last_avail_idx =3D vq_info.split.avail_index;
> +               vq->last_used_idx =3D vq_info.split.avail_index;
> +       }
> +       vq->size =3D vq_info.num;
> +       vq->ready =3D true;
> +       vq->enabled =3D vq_info.ready;
> +       ra->desc_user_addr =3D vq_info.desc_addr;
> +       ra->avail_user_addr =3D vq_info.driver_addr;
> +       ra->used_user_addr =3D vq_info.device_addr;
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "VQ %u info:", index);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tnum: %u", vq_info.num);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tdesc_addr: %llx",
> @@ -162,15 +175,6 @@ vduse_vring_setup(struct virtio_net *dev, unsigned i=
nt index)
>                        (unsigned long long)vq_info.device_addr);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tavail_idx: %u", vq_info.spl=
it.avail_index);
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "\tready: %u", vq_info.ready);
> -
> -       vq->last_avail_idx =3D vq_info.split.avail_index;
> -       vq->size =3D vq_info.num;
> -       vq->ready =3D true;
> -       vq->enabled =3D vq_info.ready;
> -       ra->desc_user_addr =3D vq_info.desc_addr;
> -       ra->avail_user_addr =3D vq_info.driver_addr;
> -       ra->used_user_addr =3D vq_info.device_addr;
> -
>        vq->kickfd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
>        if (vq->kickfd < 0) {
>                VHOST_CONFIG_LOG(dev->ifname, ERR, "Failed to init kickfd =
for VQ %u: %s",
> @@ -267,7 +271,7 @@ vduse_vring_cleanup(struct virtio_net *dev, unsigned =
int index)
> }
>=20
> static void
> -vduse_device_start(struct virtio_net *dev)
> +vduse_device_start(struct virtio_net *dev, bool reconnect)
> {
>        unsigned int i, ret;
>=20
> @@ -287,6 +291,15 @@ vduse_device_start(struct virtio_net *dev)
>                return;
>        }
>=20
> +       if (reconnect && dev->features !=3D dev->reconnect_log->features)=
 {
> +               VHOST_CONFIG_LOG(dev->ifname, ERR,
> +                               "Mismatch between reconnect file features=
 0x%" PRIx64 " & device features 0x%" PRIx64,

Checkpatch reports long line

> +                               dev->reconnect_log->features, dev->featur=
es);
> +               return;
> +       }
> +
> +       dev->reconnect_log->features =3D dev->features;
> +
>        VHOST_CONFIG_LOG(dev->ifname, INFO, "Negotiated Virtio features: 0=
x%" PRIx64,
>                dev->features);
>=20
> @@ -300,7 +313,7 @@ vduse_device_start(struct virtio_net *dev)
>        }
>=20
>        for (i =3D 0; i < dev->nr_vring; i++)
> -               vduse_vring_setup(dev, i);
> +               vduse_vring_setup(dev, i, reconnect);
>=20
>        dev->flags |=3D VIRTIO_DEV_READY;
>=20
> @@ -373,6 +386,7 @@ vduse_events_handler(int fd, void *arg, int *remove _=
_rte_unused)
>                                req.s.status);
>                old_status =3D dev->status;
>                dev->status =3D req.s.status;
> +               dev->reconnect_log->status =3D dev->status;
>                resp.result =3D VDUSE_REQ_RESULT_OK;
>                break;
>        case VDUSE_UPDATE_IOTLB:
> @@ -398,7 +412,7 @@ vduse_events_handler(int fd, void *arg, int *remove _=
_rte_unused)
>=20
>        if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) {
>                if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK)
> -                       vduse_device_start(dev);
> +                       vduse_device_start(dev, false);
>                else
>                        vduse_device_stop(dev);
>        }
> @@ -407,10 +421,64 @@ vduse_events_handler(int fd, void *arg, int *remove=
 __rte_unused)
>                        vduse_req_id_to_str(req.type), req.type);
> }
>=20
> +static char vduse_reconnect_dir[PATH_MAX];
> +static bool vduse_reconnect_path_set;
> +
> +static int
> +vduse_reconnect_path_init(void)
> +{
> +       const char *directory;
> +       int ret;
> +
> +       /* from RuntimeDirectory=3D see systemd.exec */
> +       directory =3D getenv("RUNTIME_DIRECTORY");
> +       if (directory =3D=3D NULL) {
> +               /*
> +                * Used standard convention defined in
> +                * XDG Base Directory Specification and
> +                * Filesystem Hierarchy Standard.
> +                */
> +               if (getuid() =3D=3D 0)
> +                       directory =3D "/var/run";
> +               else
> +                       directory =3D getenv("XDG_RUNTIME_DIR") ? : "/tmp=
";
> +       }
> +
> +       ret =3D snprintf(vduse_reconnect_dir, sizeof(vduse_reconnect_dir)=
, "%s/dpdk/vduse",
> +                       directory);
> +       if (ret < 0 || ret =3D=3D sizeof(vduse_reconnect_dir)) {
> +               VHOST_CONFIG_LOG("vduse", ERR, "Error creating VDUSE reco=
nnect path name");
> +               return -1;
> +       }
> +
> +       ret =3D mkdir(vduse_reconnect_dir, 0700);
> +       if (ret < 0 && errno !=3D EEXIST) {
> +               VHOST_CONFIG_LOG("vduse", ERR, "Error creating '%s': %s",
> +                               vduse_reconnect_dir, strerror(errno));
> +               return -1;
> +       }
> +
> +       VHOST_CONFIG_LOG("vduse", INFO, "Created VDUSE reconnect director=
y in %s",
> +                       vduse_reconnect_dir);
> +
> +       return 0;
> +}
> +
> +static void
> +vduse_reconnect_handler(int fd, void *arg, int *remove)
> +{
> +       struct virtio_net *dev =3D arg;
> +
> +       vduse_device_start(dev, true);
> +
> +       close(fd);
> +       *remove =3D 1;
> +}
> +
> int
> vduse_device_create(const char *path, bool compliant_ol_flags)
> {
> -       int control_fd, dev_fd, vid, ret;
> +       int control_fd, dev_fd, vid, ret, reco_fd;
>        uint32_t i, max_queue_pairs, total_queues;
>        struct virtio_net *dev;
>        struct virtio_net_config vnet_config =3D {{ 0 }};
> @@ -418,6 +486,9 @@ vduse_device_create(const char *path, bool compliant_=
ol_flags)
>        uint64_t features;
>        struct vduse_dev_config *dev_config =3D NULL;
>        const char *name =3D path + strlen("/dev/vduse/");
> +       char reconnect_file[PATH_MAX];
> +       struct vhost_reconnect_data *reconnect_log =3D NULL;
> +       bool reconnect =3D false;
>=20
>        if (vduse.fdset =3D=3D NULL) {
>                vduse.fdset =3D fdset_init("vduse-evt");
> @@ -427,6 +498,20 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                }
>        }
>=20
> +       if (vduse_reconnect_path_set =3D=3D false) {
> +               if (vduse_reconnect_path_init() < 0) {
> +                       VHOST_CONFIG_LOG(path, ERR, "failed to initialize=
 reconnect path");
> +                       return -1;
> +               }
> +               vduse_reconnect_path_set =3D true;
> +       }
> +
> +       ret =3D snprintf(reconnect_file, sizeof(reconnect_file), "%s/%s",=
 vduse_reconnect_dir, name);
> +       if (ret < 0 || ret =3D=3D sizeof(reconnect_file)) {
> +               VHOST_CONFIG_LOG(name, ERR, "Failed to create vduse recon=
nect path name");
> +               return -1;
> +       }
> +
>        control_fd =3D open(VDUSE_CTRL_PATH, O_RDWR);
>        if (control_fd < 0) {
>                VHOST_CONFIG_LOG(name, ERR, "Failed to open %s: %s",
> @@ -441,14 +526,6 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                goto out_ctrl_close;
>        }
>=20
> -       dev_config =3D malloc(offsetof(struct vduse_dev_config, config) +
> -                       sizeof(vnet_config));
> -       if (!dev_config) {
> -               VHOST_CONFIG_LOG(name, ERR, "Failed to allocate VDUSE con=
fig");
> -               ret =3D -1;
> -               goto out_ctrl_close;
> -       }
> -
>        ret =3D rte_vhost_driver_get_features(path, &features);
>        if (ret < 0) {
>                VHOST_CONFIG_LOG(name, ERR, "Failed to get backend feature=
s");
> @@ -469,23 +546,97 @@ vduse_device_create(const char *path, bool complian=
t_ol_flags)
>        else
>                total_queues +=3D 1; /* Includes ctrl queue */
>=20
> -       vnet_config.max_virtqueue_pairs =3D max_queue_pairs;
> -       memset(dev_config, 0, sizeof(struct vduse_dev_config));
> +       if (access(path, F_OK) =3D=3D 0) {
> +               VHOST_CONFIG_LOG(name, INFO, "Device already exists, reco=
nnecting...");
> +               reconnect =3D true;
> +
> +               reco_fd =3D open(reconnect_file, O_RDWR, 0600);
> +               if (reco_fd < 0) {
> +                       if (errno =3D=3D ENOENT) {
> +                               VHOST_CONFIG_LOG(name, ERR, "Missing reco=
nnect file (%s)",
> +                                               reconnect_file);
> +                       } else {
> +                               VHOST_CONFIG_LOG(name, ERR, "Failed to op=
en reconnect file %s (%s)",
> +                                               reconnect_file, strerror(=
errno));
> +                       }

Seems no {} is needed for if-else?

> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT=
_READ | PROT_WRITE,
> +                               MAP_SHARED, reco_fd, 0);
> +               close(reco_fd);
> +               if (reconnect_log =3D=3D MAP_FAILED) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon=
nect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +       } else {
> +               reco_fd =3D open(reconnect_file, O_CREAT | O_EXCL | O_RDW=
R, 0600);
> +               if (reco_fd < 0) {
> +                       if (errno =3D=3D EEXIST) {
> +                               VHOST_CONFIG_LOG(name, ERR, "Reconnect fi=
le %s exists but not the device",
> +                                               reconnect_file);
> +                       } else {
> +                               VHOST_CONFIG_LOG(name, ERR, "Failed to op=
en reconnect file %s (%s)",
> +                                               reconnect_file, strerror(=
errno));
> +                       }
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
>=20
> -       strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1);
> -       dev_config->device_id =3D VIRTIO_ID_NET;
> -       dev_config->vendor_id =3D 0;
> -       dev_config->features =3D features;
> -       dev_config->vq_num =3D total_queues;
> -       dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE);
> -       dev_config->config_size =3D sizeof(struct virtio_net_config);
> -       memcpy(dev_config->config, &vnet_config, sizeof(vnet_config));
> +               ret =3D ftruncate(reco_fd, sizeof(*reconnect_log));
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to truncate r=
econnect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       close(reco_fd);
> +                       goto out_ctrl_close;
> +               }
>=20
> -       ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config);
> -       if (ret < 0) {
> -               VHOST_CONFIG_LOG(name, ERR, "Failed to create VDUSE devic=
e: %s",
> -                               strerror(errno));
> -               goto out_free;
> +               reconnect_log =3D mmap(NULL, sizeof(*reconnect_log), PROT=
_READ | PROT_WRITE,
> +                                       MAP_SHARED, reco_fd, 0);
> +               close(reco_fd);
> +               if (reconnect_log =3D=3D MAP_FAILED) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to mmap recon=
nect file %s (%s)",
> +                                       reconnect_file, strerror(errno));
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               reconnect_log->version =3D 0;
> +
> +               dev_config =3D malloc(offsetof(struct vduse_dev_config, c=
onfig) +
> +                               sizeof(vnet_config));
> +               if (!dev_config) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to allocate V=
DUSE config");
> +                       ret =3D -1;
> +                       goto out_ctrl_close;
> +               }
> +
> +               vnet_config.max_virtqueue_pairs =3D max_queue_pairs;
> +               memset(dev_config, 0, sizeof(struct vduse_dev_config));
> +
> +               rte_strscpy(dev_config->name, name, VDUSE_NAME_MAX - 1);
> +               dev_config->device_id =3D VIRTIO_ID_NET;
> +               dev_config->vendor_id =3D 0;
> +               dev_config->features =3D features;
> +               dev_config->vq_num =3D total_queues;
> +               dev_config->vq_align =3D sysconf(_SC_PAGE_SIZE);
> +               dev_config->config_size =3D sizeof(struct virtio_net_conf=
ig);
> +               memcpy(dev_config->config, &vnet_config, sizeof(vnet_conf=
ig));
> +
> +               ret =3D ioctl(control_fd, VDUSE_CREATE_DEV, dev_config);
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to create VDU=
SE device: %s",
> +                                       strerror(errno));
> +                       goto out_free;
> +               }
> +
> +               memcpy(&reconnect_log->config, &vnet_config, sizeof(vnet_=
config));
> +               reconnect_log->nr_vrings =3D total_queues;
> +               free(dev_config);
> +               dev_config =3D NULL;
>        }
>=20
>        dev_fd =3D open(path, O_RDWR);
> @@ -519,10 +670,15 @@ vduse_device_create(const char *path, bool complian=
t_ol_flags)
>        strncpy(dev->ifname, path, IF_NAME_SZ - 1);
>        dev->vduse_ctrl_fd =3D control_fd;
>        dev->vduse_dev_fd =3D dev_fd;
> +       dev->reconnect_log =3D reconnect_log;
> +       if (reconnect)
> +               dev->status =3D dev->reconnect_log->status;
> +
>        vhost_setup_virtio_net(dev->vid, true, compliant_ol_flags, true, t=
rue);
>=20
>        for (i =3D 0; i < total_queues; i++) {
>                struct vduse_vq_config vq_cfg =3D { 0 };
> +               struct vhost_virtqueue *vq;
>=20
>                ret =3D alloc_vring_queue(dev, i);
>                if (ret) {
> @@ -530,6 +686,12 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                        goto out_dev_destroy;
>                }
>=20
> +               vq =3D dev->virtqueue[i];
> +               vq->reconnect_log =3D &reconnect_log->vring[i];
> +
> +               if (reconnect)
> +                       continue;
> +
>                vq_cfg.index =3D i;
>                vq_cfg.max_size =3D 1024;
>=20
> @@ -549,7 +711,28 @@ vduse_device_create(const char *path, bool compliant=
_ol_flags)
>                goto out_dev_destroy;
>        }
>=20
> -       free(dev_config);
> +       if (reconnect && dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK)  {
> +               reco_fd =3D eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
> +               if (reco_fd < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to create rec=
o_fd: %s",
> +                                       strerror(errno));
> +                       ret =3D -1;
> +                       goto out_dev_destroy;
> +               }
> +
> +               ret =3D fdset_add(vduse.fdset, reco_fd, vduse_reconnect_h=
andler, NULL, dev);
> +               if (ret) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to add fd %d =
to vduse fdset",
> +                                       dev->vduse_dev_fd);

Should print reco_fd

> +                       goto out_dev_destroy;
> +               }
> +
> +               ret =3D eventfd_write(reco_fd, (eventfd_t)1);
> +               if (ret < 0) {
> +                       VHOST_CONFIG_LOG(name, ERR, "Failed to write to r=
econnect eventfd");
> +                       goto out_dev_destroy;
> +               }
> +       }

Maybe I missed something, why we need to implement like this instead of dir=
ectly call vduse_device_start?

Thanks,
Chenbo

>=20
>        return 0;
>=20
> @@ -587,6 +770,9 @@ vduse_device_destroy(const char *path)
>        if (vid =3D=3D RTE_MAX_VHOST_DEVICE)
>                return -1;
>=20
> +       if (dev->reconnect_log)
> +               munmap(dev->reconnect_log, sizeof(*dev->reconnect_log));
> +
>        vduse_device_stop(dev);
>=20
>        fdset_del(vduse.fdset, dev->vduse_dev_fd);
> @@ -597,10 +783,26 @@ vduse_device_destroy(const char *path)
>        }
>=20
>        if (dev->vduse_ctrl_fd >=3D 0) {
> +               char reconnect_file[PATH_MAX];
> +
>                ret =3D ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name)=
;
> -               if (ret)
> +               if (ret) {
>                        VHOST_CONFIG_LOG(name, ERR, "Failed to destroy VDU=
SE device: %s",
>                                        strerror(errno));
> +               } else {
> +                       /*
> +                        * VDUSE device was no more attached to the vDPA =
bus,
> +                        * so we can remove the reconnect file.
> +                        */
> +                       ret =3D snprintf(reconnect_file, sizeof(reconnect=
_file), "%s/%s",
> +                                       vduse_reconnect_dir, name);
> +                       if (ret < 0 || ret =3D=3D sizeof(reconnect_file))
> +                               VHOST_CONFIG_LOG(name, ERR,
> +                                               "Failed to create vduse r=
econnect path name");
> +                       else
> +                               unlink(reconnect_file);
> +               }
> +
>                close(dev->vduse_ctrl_fd);
>                dev->vduse_ctrl_fd =3D -1;
>        }
> --
> 2.46.0
>=20