From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C5921A04FD; Mon, 23 May 2022 07:31:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7B8914014F; Mon, 23 May 2022 07:31:36 +0200 (CEST) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 3CBEE40041 for ; Mon, 23 May 2022 07:31:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653283894; x=1684819894; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=YPSOxlrvr6DFBpNSsDmFNuY9M8GGAJodjDLNgXJmVjw=; b=h7YLF/p4rUimm4OVdrf1reWB/6z1JnpbfMHQ24oDmIKV88EZCtjVWUBl 6Yx788aYBDkhWv2/SizQmto7vvS23Qy3PPeEyQJXb/Z+XuIHV80XU54jh mhtl7ci7vsBhPYunhyHcNSRVo8NBIf2Gj+nWPWjIyWv46suIIqYGD9jeH BXxTmGLFQZDn8yUtoGZDaBsNtRiOJ0hcdgD2gBwDFpPL4d2KiRF++fXtO wqTKBUg9qnInLbX2nC5ZbOQ+EpAblRZBYrkfMIENYdXHKTnDbjA1325B0 LDi9wTEp2Ni6PY6HKlSDNuJHO9da3xGCUmXaoEx2B/h87Y9hjJZdzzQgE w==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="333750838" X-IronPort-AV: E=Sophos;i="5.91,245,1647327600"; d="scan'208";a="333750838" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 May 2022 22:31:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,245,1647327600"; d="scan'208";a="702825189" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga004.jf.intel.com with ESMTP; 22 May 2022 22:31:33 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Sun, 22 May 2022 22:31:32 -0700 Received: from orsmsx602.amr.corp.intel.com (10.22.229.15) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Sun, 22 May 2022 22:31:32 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Sun, 22 May 2022 22:31:32 -0700 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (104.47.74.43) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Sun, 22 May 2022 22:31:32 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WLDwQFtJHbZtu8ce3Hf4OF88yFJGIppseEpwMB5de3xp6zoxB3WzdpJff+h6AWyxUR3wl8vKfksWTGWQb5MGmZY6vp2nIezTZfpVrc7bI+UPcVdE/l6mYDq8r738ActKfKhjcLiVfjcuLD6FSU/xtYtL0sTB1ciRMORaF9+0D1r1KOkzIFOZDOI5USNY2OSm/3Q4HN7Gzm5ATKZPId+LOm7cLbl//mdTirSjx3xyiIfAwzLliGq6Bmq5UqeUh4MrNVxuJZMalxf4b8eIUSXjYAzEgSlJiIcyfZ4UKvyPPWw2gmNzo5pmJqm0Fk82gDySXaZqGbJd488DNiQwrbPXkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pLko//qSQacb6FvaOVJaBWkOwZnyIOElm4YKNM+ZU54=; b=WW9EDsHeAQh+Bp5lpdq3rSMXQO9xRvAFmYv1+7aGAORjY6FjgeccY1Jb7iIQZrYzRBeo6ZI1yrnN9qCVRtEXO8JVmdQoQ91icoZrnaxtL0WZ4K/6YRVXWX7U5wN1lbgevK94Mc65NHHlveKXeLCb1u4kT2SEzDN4jGWM0M1IXxEcey5DVvcIkGPlrJhTkg376tyLxdzku6I/4h/M/KBGsT7ayQrDzTpYu6jtvC0xvGjy2mgJWQkPPBgPQjXvZByIwgjhJcvfjs9NjSM1J7LpbY8EFMJgWPYZm0jQ1Et+KDa7xTS3RlL7YtFKhGSaKKO2gtLgqYl3nlaqhqJwmKQoig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from DM5PR11MB1739.namprd11.prod.outlook.com (2603:10b6:3:111::22) by BYAPR11MB3752.namprd11.prod.outlook.com (2603:10b6:a03:fc::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.16; Mon, 23 May 2022 05:31:30 +0000 Received: from DM5PR11MB1739.namprd11.prod.outlook.com ([fe80::8c3b:8a54:d186:b555]) by DM5PR11MB1739.namprd11.prod.outlook.com ([fe80::8c3b:8a54:d186:b555%11]) with mapi id 15.20.5273.022; Mon, 23 May 2022 05:31:29 +0000 From: "Pei, Andy" To: "Xia, Chenbo" , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "Cao, Gang" , "Liu, Changpeng" , "Xu, Rosen" , "Xiao, QimaiX" Subject: RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration Thread-Topic: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration Thread-Index: AQHYardZX94NucTIS0a9h7ZDJKv6SK0r9TRQgAAB3UA= Date: Mon, 23 May 2022 05:31:29 +0000 Message-ID: References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> <1652876035-70513-7-git-send-email-andy.pei@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.6.401.20 dlp-reaction: no-action authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6b3efeb9-61de-40b1-d450-08da3c7d7d28 x-ms-traffictypediagnostic: BYAPR11MB3752:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: tfaFVP6wWiqVJlmwxAx0tMbuogtx3UuD+zcNANlEnkO6IMxhNk14JOYepQMzinlUi1lxE4Sv8X9jOcRwIHXtsYScNb3c78AjY+6cFR5HkNxmAHK22KhESr4p2RwTEZ36REUNHyvBjrhzoAMScWFJCUZwCh4xphUFufVjjGShlEQrqxyKg/ZDGFL0r+5HVj2XWOhJFPjyfIzCfxdndrs6P8PAemgxxFslfW2Tgozk5k4kh4TvuUYbi9FWcRIb7jv5ztuLFI2ZDwImD/SC73yCsYemNZvIWNBtOLn0A5yNh5oq2RKDgOSSs3Lw3zIdUnS4nrsZZJqTrbfBYlaOWoN34rrzFvNidqyXKCp24h6zV2zAhv327NKFFsDkCMzyAR9JyNEt2xswdH8JuOM6bCL+ghdFRKsThLB9SGTU0Zjg3bGySEowsuxyq+u4UuF2wPjXczqYi2qQD0H8KbwEo1FQQzJZnZ7JyT96hAeXT9aOWpyUIXyT5N2PnWWNWXhJWOB7EUFmlP8Ii2E9T9/STwitqxrcEBzIUa+3Hr5mHx2HFJ3GdrLhtenHT7+tZknXdW6s6WhxfQSSH6Ay07xDIqNoTcOIeBwIlxFAZbAoZlo/pi15QqpjBRu4rLBvW15frpqL3QU9+1kPVquhNRtjvGoCey4TSdqOihvpozuVN9q3Woz+T9Z82pogIyWXQsbnYqT5kamWsm0ZV1CtSFnypUlk2Q== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR11MB1739.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(186003)(66946007)(33656002)(66556008)(76116006)(66476007)(66446008)(64756008)(83380400001)(8676002)(316002)(4326008)(54906003)(6506007)(7696005)(53546011)(508600001)(71200400001)(107886003)(26005)(9686003)(110136005)(86362001)(55016003)(38100700002)(5660300002)(38070700005)(82960400001)(122000001)(8936002)(52536014)(2906002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?SVktiEo5WZUSYiFFSHy6h1DWumXQTUH+k3PDjoh7gdgtNwm2X7RBWlrGYcmw?= =?us-ascii?Q?4OgCiKPuMWzFa/hpm/KyG2/+T/3KEbOkZCSDSprtyMazPbB8tEbffw6rQxij?= =?us-ascii?Q?Ro9QjxqWbmK0cs/d/oX3t6EuZpbxoVQc5ta8V0q7ClCmDui2yKRyCZZ9RgtU?= =?us-ascii?Q?lkbWymbrFjFPn+3scThvPJz135rZRtHVpekUmvMh+ZnNGWFOPi3hKSzBaWpx?= =?us-ascii?Q?DYsiZIgHElj8oHKXMmTlLLJB24rfd2ABAIrDtkNAIKkR0ZwmzxsG9/DuWSXo?= =?us-ascii?Q?5SFpt2/DcDsF+2G2U1LbiOqaHO8PVISYOUlhIuln5LWojBUsFNzXqDjBNvFZ?= =?us-ascii?Q?ZyGW4m4ngjLYyrPlz673fLockSjVaauq6s6N1iMXs5jmrAQF8uBAsWI3YiHl?= =?us-ascii?Q?ONKx5ScpjsOHgv9wN7pY/vYLYGHZOlRAPNxAdHWE1n80Pc46M17qpmy4zhXE?= =?us-ascii?Q?XufNr0vMoaQAMFP59wBcC0LTdyEF6uPO4RRZIkGnxAvvhqrik1uOSZB1e+Pa?= =?us-ascii?Q?5aM+AoADXY9etb48kENm89H3TvcQfE2TRkLw+kHOV30BfgrslYiRAYbz563+?= =?us-ascii?Q?AT2OxlEi8ITCT8VIP+Mu5WoBvhFiE0Tu0aYTGgUWtlamd9KPteOrCr+nSPgy?= =?us-ascii?Q?Kj1O1/arDiJTpYVuUjujwpGA6V2AAxZlA6x72NVOOkP7zvMKQIIPLYUbvlZW?= =?us-ascii?Q?f1UWHTQ+kB+uj/z1dhQA5goTlL2jPVd+mMfKXmBWbqXdBSORCm54ZTu9LHQQ?= =?us-ascii?Q?iL+WDf+7JqgnPet6erD0KLyZ2ay+K0RV5e1GikQmT3cnFKtclIpMGk5x+3dA?= =?us-ascii?Q?RPnpZO+nJHmzG1vguZUGzGTDWJgN9jBfOBPEWWYlCBTOK8CZxZExfxOU355S?= =?us-ascii?Q?CZqtI6IFV6OGFmHyHa8d4zrjj9pWG12lz0vUfGpKxF+OTYVeTIqVAUYhwGh8?= =?us-ascii?Q?VmXcwYvEQBRYr1M4hBLwvCFeS+K5QRGmeEuZCbhl54M/Kdb8WniVD+XTXOXj?= =?us-ascii?Q?CKugCsd5roo0ft1uw5Y5J4EQ982QISAJdzWmhX4D1GWLNzsQfglpwvhi3H2x?= =?us-ascii?Q?9ex6P138VwtAWLFp441sVqPBxMMoexUBJPUTf/MjFVJnzuqyBDFpD3BLAuG1?= =?us-ascii?Q?Fh+eQqsnTPm0tPDvZpnzWOkheWL0GF16jRM9PADkAGgX51NcshMWPXMsbJHo?= =?us-ascii?Q?IMOrBSLCc2zZbunFb/s8sj6VzVZB/6EyB9EiinSfzPnvwpU/K6NzxFR+fpTL?= =?us-ascii?Q?bxi+8xX3g8kFvEqzOzAx1Nc7WggbH+W6EHOn3gzWw+r17gN2/+O7P0uKsJNi?= =?us-ascii?Q?iHiA4epko1mIs2B6kCFP35jFj7lJ2Tq+n/vYbV0OmB1ZM/Zk7MOefjoaOjkk?= =?us-ascii?Q?he+w73PLW6B5aYvC3i4FhITio9OI7d8RtzbFJXOz0egiuTDUjpI788US6zCv?= =?us-ascii?Q?PWlY7hY0pUt3XkKWhLgMr9NsT0rQUmD4m8J9FVJ5TFx+ktYVViIH1DCWSZ8B?= =?us-ascii?Q?FY3uDnAxAxrhhd/0ILOrwFs77qmnH9OGzLXVFl3cQBUti+gQ7Pik9YESf2D2?= =?us-ascii?Q?tZXksNC5+vcmeW6e/55M7qRQ/7sHzpBSUV9dt0cly/ikt3DjASEGKZN7QAH4?= =?us-ascii?Q?4A9HGZ0JbEcHSLyK5Zypj2wcd6Q+Ed79ANMC33VKMKAG7N6i+IhpKHlQ+hfx?= =?us-ascii?Q?h9qOE6KeTJ42fCGYKO02raTCjR4EVtC+2XWcKoo86QpqIZWP64D9GjbuD5nH?= =?us-ascii?Q?Wwu+nuymhw=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM5PR11MB1739.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6b3efeb9-61de-40b1-d450-08da3c7d7d28 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 May 2022 05:31:29.4284 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: P8o0yAkAzfBXwRzpymJuZQL8DBin7XzoyquYZcWv/82itdHm8ul5zfsMCEoW/AvhwmWNQWZOmqnlDVeoYcas7g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3752 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Chenbo, Thanks for your reply, I will fix them in next version. > -----Original Message----- > From: Xia, Chenbo > Sent: Monday, May 23, 2022 1:25 PM > To: Pei, Andy ; dev@dpdk.org > Cc: maxime.coquelin@redhat.com; Cao, Gang ; Liu, > Changpeng ; Xu, Rosen ; > Xiao, QimaiX > Subject: RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migratio= n >=20 > > -----Original Message----- > > From: Pei, Andy > > Sent: Wednesday, May 18, 2022 8:14 PM > > To: dev@dpdk.org > > Cc: Xia, Chenbo ; maxime.coquelin@redhat.com; > > Cao, Gang ; Liu, Changpeng > > ; Xu, Rosen ; Xiao, > > QimaiX > > Subject: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration > > > > Add SW live-migration support to block device. > > For block device, it is critical that no packet should be dropped. So > > when virtio blk device is pause, make sure hardware last_avail_idx and >=20 > paused >=20 > > last_used_idx are the same. This indicates all requests have recieved > > acks, and no inflight IO. >=20 > received >=20 > With above fixed: >=20 > Reviewed-by: Chenbo Xia >=20 > > > > Signed-off-by: Andy Pei > > --- > > drivers/vdpa/ifc/base/ifcvf.h | 1 + > > drivers/vdpa/ifc/ifcvf_vdpa.c | 42 > > ++++++++++++++++++++++++++++++++++++++---- > > 2 files changed, 39 insertions(+), 4 deletions(-) > > > > diff --git a/drivers/vdpa/ifc/base/ifcvf.h > > b/drivers/vdpa/ifc/base/ifcvf.h index 244de46..4fb1736 100644 > > --- a/drivers/vdpa/ifc/base/ifcvf.h > > +++ b/drivers/vdpa/ifc/base/ifcvf.h > > @@ -65,6 +65,7 @@ > > #define IFCVF_MEDIATED_VRING 0x200000000000 > > > > #define IFCVF_32_BIT_MASK 0xffffffff > > +#define IFCVF_16_BIT_MASK 0xffff > > > > > > #ifndef VHOST_USER_PROTOCOL_F_CONFIG > > diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c > > b/drivers/vdpa/ifc/ifcvf_vdpa.c index 509a1ed..3e78c7d 100644 > > --- a/drivers/vdpa/ifc/ifcvf_vdpa.c > > +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c > > @@ -316,8 +316,34 @@ struct rte_vdpa_dev_info { > > uint64_t features =3D 0; > > uint64_t log_base =3D 0, log_size =3D 0; > > uint64_t len; > > + u32 ring_state =3D 0; > > > > vid =3D internal->vid; > > + > > + /* to make sure no packet is lost for blk device > > + * do not stop until last_avail_idx =3D=3D last_used_idx > > + */ > > + if (internal->hw.device_type =3D=3D IFCVF_BLK) { > > + for (i =3D 0; i < hw->nr_vring; i++) { > > + do { > > + if (hw->lm_cfg !=3D NULL) > > + ring_state =3D *(u32 *)(hw->lm_cfg + > > + > IFCVF_LM_RING_STATE_OFFSET + > > + i * IFCVF_LM_CFG_SIZE); > > + hw->vring[i].last_avail_idx =3D > > + (u16)(ring_state & > IFCVF_16_BIT_MASK); > > + hw->vring[i].last_used_idx =3D > > + (u16)(ring_state >> 16); > > + if (hw->vring[i].last_avail_idx !=3D > > + hw->vring[i].last_used_idx) { > > + ifcvf_notify_queue(hw, i); > > + usleep(10); > > + } > > + } while (hw->vring[i].last_avail_idx !=3D > > + hw->vring[i].last_used_idx); > > + } > > + } > > + > > ifcvf_stop_hw(hw); > > > > for (i =3D 0; i < hw->nr_vring; i++) > > @@ -641,8 +667,10 @@ struct rte_vdpa_dev_info { > > } > > hw->vring[i].avail =3D gpa; > > > > - /* Direct I/O for Tx queue, relay for Rx queue */ > > - if (i & 1) { > > + /* NET: Direct I/O for Tx queue, relay for Rx queue > > + * BLK: relay every queue > > + */ > > + if ((internal->hw.device_type =3D=3D IFCVF_NET) && (i & 1)) { > > gpa =3D hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); > > if (gpa =3D=3D 0) { > > DRV_LOG(ERR, "Fail to get GPA for used > ring."); @@ -692,8 +720,12 > > @@ struct rte_vdpa_dev_info { > > > > for (i =3D 0; i < hw->nr_vring; i++) { > > /* synchronize remaining new used entries if any */ > > - if ((i & 1) =3D=3D 0) > > + if (internal->hw.device_type =3D=3D IFCVF_NET) { > > + if ((i & 1) =3D=3D 0) > > + update_used_ring(internal, i); > > + } else if (internal->hw.device_type =3D=3D IFCVF_BLK) { > > update_used_ring(internal, i); > > + } > > > > rte_vhost_get_vhost_vring(vid, i, &vq); > > len =3D IFCVF_USED_RING_LEN(vq.size); @@ -755,7 +787,9 > @@ struct > > rte_vdpa_dev_info { > > } > > } > > > > - for (qid =3D 0; qid < q_num; qid +=3D 2) { > > + for (qid =3D 0; qid < q_num; qid +=3D 1) { > > + if ((internal->hw.device_type =3D=3D IFCVF_NET) && (qid & 1)) > > + continue; > > ev.events =3D EPOLLIN | EPOLLPRI; > > /* leave a flag to mark it's for interrupt */ > > ev.data.u64 =3D 1 | qid << 1 | > > -- > > 1.8.3.1