From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 84A8DA04FD; Mon, 23 May 2022 07:25:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 308D54014F; Mon, 23 May 2022 07:25:32 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id C8B9040041 for ; Mon, 23 May 2022 07:25:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653283531; x=1684819531; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=5fZH6bHbejssFhT7ZBGN7Df3uW1n5buji/FaJNqUc+8=; b=AzWoym0lSj2U5V5k5cM5G/GVaOhdfoKLsROG+C2XU8suUx1BQ0UmsTXB FbXIesBTgq0P4SXZ4+tEEzCBxyGUE9/JdsmHRSQcUtZ8AV/mapdbaTg9w t8qKTmETCPqPbX/5GiLvPxAEnfKtRb9HezlQRuHESXr4tJ/mWHgvMOF+K ACAsNBUJhEwwC5JYkMtindzLXe8pJ1jvuclZ7dXrXqhH2vaixzetGQ3r0 o6r27kqkqz/S4bBr8L+74H7GFRqYGx5rBNJicGP61UmiMPomKYczGntuS Y47RUu2Nw4DZD1VmXNDJFXjIWWpu/tCD28wjDzhWJ5pVlMVwBuYRGCBde w==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="359500258" X-IronPort-AV: E=Sophos;i="5.91,245,1647327600"; d="scan'208";a="359500258" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 May 2022 22:25:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,245,1647327600"; d="scan'208";a="558479399" Received: from fmsmsx604.amr.corp.intel.com ([10.18.126.84]) by orsmga002.jf.intel.com with ESMTP; 22 May 2022 22:25:29 -0700 Received: from fmsmsx608.amr.corp.intel.com (10.18.126.88) by fmsmsx604.amr.corp.intel.com (10.18.126.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Sun, 22 May 2022 22:25:29 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx608.amr.corp.intel.com (10.18.126.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Sun, 22 May 2022 22:25:29 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.106) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Sun, 22 May 2022 22:25:28 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KM7PMIZS9Zfip3HtG+ErAXAJ18cwoVN+Tl53/xiuQL3qtmUuJD/kw9rpWlhuTTJFanEgd+Ipxz2pAW0sWpeTQdBjXw3jXr/A06JLGUhlrE35S36AJWZu1Te3JXah8t+v7b/Liy28XIuDhuP5CfzfXQKSO5iAMKMe/QIMp6uaNKvXbuUZdxm481BMcHFf4dhnM/zBKj4scdAuv0aMyYHs045m/T2r67Fm7rBY5IAVg2loyP0GeZn2Fb0+PX+yjUAoBqBTG6qJmrMG/d4CRlC0ASX4abkn3kYp7/P45FEyv06OcDflNIlIzIuCTI1xlsaVY170bJkmHqIdzZgG0P9hDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pyY2QcxxE6dali5+VDv2wWhIqV70He8XqHC7dDMM23Y=; b=iuE6llnDAoHDPFiV7KjbvkhoboQ4cH4I8+0eISo1DB3BtKtDfwONBP162EWVlUAPGb+8yXYUbp/5RlqSIoUzNCFXXhyf6pkaNbw9KyPCNu/HJ5OqdEg3Xyj3cMghehGAxFvNnzo09WWxATlF2lDmLA+2qcTOcj73WNAn9tMwBalUX4aA4vu7irwWzgTN11Gt3ENuM2Qsk6K3zSODwHdw7/cM0+AQRgAwH6M2eRv2Mk0fpVQ8EURBsJCOHuCNEJf0tGEw+mExFWd1CQD2YQs9HPymYnKOX1CMPRcjNV3P4ow/jsGL+t1DHD4CEnqBXK4fWRM1AbrUdDCiAvoYNwISCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17) by CH0PR11MB5457.namprd11.prod.outlook.com (2603:10b6:610:d0::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.13; Mon, 23 May 2022 05:25:27 +0000 Received: from SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::54e1:e802:17c8:93f]) by SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::54e1:e802:17c8:93f%7]) with mapi id 15.20.5273.022; Mon, 23 May 2022 05:25:26 +0000 From: "Xia, Chenbo" To: "Pei, Andy" , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "Cao, Gang" , "Liu, Changpeng" , "Xu, Rosen" , "Xiao, QimaiX" Subject: RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration Thread-Topic: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration Thread-Index: AQHYardZX94NucTIS0a9h7ZDJKv6SK0r9TRQ Date: Mon, 23 May 2022 05:25:26 +0000 Message-ID: References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> <1652876035-70513-7-git-send-email-andy.pei@intel.com> In-Reply-To: <1652876035-70513-7-git-send-email-andy.pei@intel.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 49c58209-badc-4620-66eb-08da3c7ca4d0 x-ms-traffictypediagnostic: CH0PR11MB5457:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: IeP6F76KPAjmKHBo2PiGRB03rIrRorlXM6j+VaOVnsY2ngp5J88Z7UpKzHaTg2uxlPKw0s6dvvvLgAN5RPXGgt70yZipoOyNx8uhIGi5Anzh2I/TsDGKEa3wvxkWYczWGHS+7u4bSxkxMQQDflGMGE5pD3eDuoH5/g+KtXQBtx0Yg7R8fYKXl9LEzI5NYa2FEAf6mP/hLy6WRK7JfscZl9mdQsoCgFHMC3pVEqLVSTQtM0/cU4usQHsjQtcQ1UHcc5jHdjfbTERIUVXTUjlFLgEv3V9VFlXsWJhi4phmry9YEIO/Ocn14zp2ddimh+IeVFBqUYlaP1S8/IqhoRFKf4KRvNMIGACqGEOO/Myl/gX1nJxrRlkKC8p+1L+wizHmxjNV3n/A5IXelHLVYHpcOFLxT7DtXefZCufIddbJUspcNz3k41zB/lDCCT2Yu+rL70IeDh89FFxfxjhJL0vQD7qpLctLqlpDrmwLq2xmpVepJNaIBGiJGMXxpm0p2LBohhwL3rRNNxK6fu/YBELgWvoOgaMBpSPfGtNcZdsYEHP3e9ZUahUQ2DfGhfYuFzZQJereci1omkdkobmMWdna/EhdLyo2vmE0Yy9BVbpVoYCRPNiiRqNYdmRXUxg1uCpX1cCR8GYYmHCdG0aMkZlJDawyxSmOReC0/UGNrhrF5irlMVcsJlta0vtoxVNvYaF4a2T4TQhiTIf5+uzD8x84bg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(122000001)(508600001)(26005)(38070700005)(9686003)(83380400001)(107886003)(186003)(52536014)(5660300002)(33656002)(7696005)(6506007)(82960400001)(53546011)(8936002)(2906002)(316002)(66476007)(54906003)(110136005)(71200400001)(55016003)(66446008)(8676002)(66946007)(4326008)(76116006)(64756008)(66556008)(86362001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?VJjZriKJP0ATDov01rYcOutJw0bVv2OECUdlh9t45DyZiVio2AlGa4qFk9ar?= =?us-ascii?Q?9+Mo8NR/GOR+2HoXVjo77TV+btgckp35GNXQ+G/YD+i3col2oga5hUdhU1dZ?= =?us-ascii?Q?jLuv8VCbmU2xHmBq8KxEGEZoH7zgWj0drOAYNPw8kleKjvAsFgrW1fG9IwQj?= =?us-ascii?Q?w87xD2x3DtD/YAP3WhbhnSJKQZfLUUaEdDYLUBg21tUKMZgGJCHszBPT7+6N?= =?us-ascii?Q?l9nJsn2mO8aa5BZq8FYpoGU5FIR76sxwx6Y9BbC7LY2p3DO4H6U0nXX5VRbs?= =?us-ascii?Q?3seed9VWb24rl/r/++Yz7df8s4YUg9X7yj0UMJXoO26owRpmVIs58ZJAZsq7?= =?us-ascii?Q?SY6zf0OQXl48KfaZ1J+V9WP64HTXzvWMN0cqaopd5QSH47Mo76YO8+uq+Lyh?= =?us-ascii?Q?3DnXZQT+KsGLaWssKL0LSrHMB0TsNnOY1j2QrCccfD30KGdamKTM2L+XmkOt?= =?us-ascii?Q?xSXUXHYj1RxdpCvvdS7O2dn9G6nqJAm/n1dZ4s+R7xD1DDhosE5jyooXpa11?= =?us-ascii?Q?FKobx+5pWhfMTCRxhgSHyBc0ZtCSDEa76pgsLf+tArl5rBeaV0qXHeaJBCPY?= =?us-ascii?Q?KWhx9e9nIYE6LVNMO6W6krEy1SgViN3943rVqQVxvps6FbPNabqZeaa79xHy?= =?us-ascii?Q?ndNOzadOOtb24ZjYemkddrzQrxLFX5bAtMuVK70dr+SbYZ9AZSLkJMrxm82M?= =?us-ascii?Q?izLr1UPzwOCKN7EnWFubwyqq07U2d5XfUW7czkDhNC6w3pRxE7aiMWd0ASoM?= =?us-ascii?Q?twD0IuFNyBKPb0Ygj0m5ocpyqiJ05r6Xu5S4lUNiJ9y/yThO+23FksbyzoBB?= =?us-ascii?Q?bSOPOHqZ1ZG417fMMqlaKJXDr/W9t70P2Ivrq9/q7soDJ26hUzbIen+/iOI7?= =?us-ascii?Q?6m0L87Hx4yABtDnnNfzLQ8/WR5We6lE/FlviLTs2wuP5GXbrbpxcZLe0Q+tW?= =?us-ascii?Q?vlX6Kh43xjMNcNQnErcreR4AHiJKUE5iUOSkUFZSQI/mYoLfgVHNTO/gNdGn?= =?us-ascii?Q?6Wrp/yIbskdC5ggVrnF98vK+nSIx0rHYLi0E/ICrJJf8ixg4b2kxLNqJGOhy?= =?us-ascii?Q?i6oy03MOqNCybxJZQOIkRZErbtsePBMduJZ5xOkh/mJkkO5mrQB2pxq321cb?= =?us-ascii?Q?nnS+2txT6XomChV8eFfVF9xfG2UFPHs9uI5VE1U9AzTSpYzqMnIWrDWZyonu?= =?us-ascii?Q?GaTRIKOb6XZeuHTjZmd3bV8XwylS9bzEgv3GPsctRfgUDuiAnmoRIkzFCDKW?= =?us-ascii?Q?YUQnZG20hdGIxomBZO4Hipws55+UO7twaWiPtyd4iVAWKTffsJt1m9vnfAcE?= =?us-ascii?Q?jbWcaMFSSjpX723CwuqpCXAODQuVzHrjQsG6E3/WNjXtjM06D+wPFrYfBsnT?= =?us-ascii?Q?rTwirYdcMwRwkAV5k5dvAi+lZGq/UBuDiJXnuwzh8phVvsedkMVqhdmwwaXf?= =?us-ascii?Q?NzfdO8HI9uIBs04SlzFVy0J+ihqO2AO53YXks6TSCH3qwMfGOIZ6u2QBtGKp?= =?us-ascii?Q?CeeiaDyLBebyZrTQoPbs2xO8x4YYhsKCu0ho9yBKDai4t9Rl9JIrQwL10KwG?= =?us-ascii?Q?JSl+RlTTLSrq3J8dCGlAH4xHtTNfOC5WqTQommcHdcK3mi7tHMFwecBycjaz?= =?us-ascii?Q?QPkWWHKwrXIl6mzDEBA4YeWmli2Us5p/0oiANBlVHiHFi7WzSYEfLKsF5Izi?= =?us-ascii?Q?8oVoPR/MwZtYZiz7QGZgMJE2nk8/cyUF0FgynI1g0ncMG2j03ejhb+4qLlMh?= =?us-ascii?Q?KthGuYUMaA=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 49c58209-badc-4620-66eb-08da3c7ca4d0 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 May 2022 05:25:26.4766 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: vF4HR/EGV7sTMwbupJmulfwc8GRZ+VQKMV+vKduMDI0MVr3tG6YjZLZPqaVZmSJKy+AT/6zTDjccu03dw+40zA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5457 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Pei, Andy > Sent: Wednesday, May 18, 2022 8:14 PM > To: dev@dpdk.org > Cc: Xia, Chenbo ; maxime.coquelin@redhat.com; Cao, > Gang ; Liu, Changpeng ; Xu, > Rosen ; Xiao, QimaiX > Subject: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration >=20 > Add SW live-migration support to block device. > For block device, it is critical that no packet > should be dropped. So when virtio blk device is > pause, make sure hardware last_avail_idx and paused > last_used_idx are the same. This indicates all > requests have recieved acks, and no inflight IO. received With above fixed: Reviewed-by: Chenbo Xia >=20 > Signed-off-by: Andy Pei > --- > drivers/vdpa/ifc/base/ifcvf.h | 1 + > drivers/vdpa/ifc/ifcvf_vdpa.c | 42 > ++++++++++++++++++++++++++++++++++++++---- > 2 files changed, 39 insertions(+), 4 deletions(-) >=20 > diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.= h > index 244de46..4fb1736 100644 > --- a/drivers/vdpa/ifc/base/ifcvf.h > +++ b/drivers/vdpa/ifc/base/ifcvf.h > @@ -65,6 +65,7 @@ > #define IFCVF_MEDIATED_VRING 0x200000000000 >=20 > #define IFCVF_32_BIT_MASK 0xffffffff > +#define IFCVF_16_BIT_MASK 0xffff >=20 >=20 > #ifndef VHOST_USER_PROTOCOL_F_CONFIG > diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.= c > index 509a1ed..3e78c7d 100644 > --- a/drivers/vdpa/ifc/ifcvf_vdpa.c > +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c > @@ -316,8 +316,34 @@ struct rte_vdpa_dev_info { > uint64_t features =3D 0; > uint64_t log_base =3D 0, log_size =3D 0; > uint64_t len; > + u32 ring_state =3D 0; >=20 > vid =3D internal->vid; > + > + /* to make sure no packet is lost for blk device > + * do not stop until last_avail_idx =3D=3D last_used_idx > + */ > + if (internal->hw.device_type =3D=3D IFCVF_BLK) { > + for (i =3D 0; i < hw->nr_vring; i++) { > + do { > + if (hw->lm_cfg !=3D NULL) > + ring_state =3D *(u32 *)(hw->lm_cfg + > + IFCVF_LM_RING_STATE_OFFSET + > + i * IFCVF_LM_CFG_SIZE); > + hw->vring[i].last_avail_idx =3D > + (u16)(ring_state & IFCVF_16_BIT_MASK); > + hw->vring[i].last_used_idx =3D > + (u16)(ring_state >> 16); > + if (hw->vring[i].last_avail_idx !=3D > + hw->vring[i].last_used_idx) { > + ifcvf_notify_queue(hw, i); > + usleep(10); > + } > + } while (hw->vring[i].last_avail_idx !=3D > + hw->vring[i].last_used_idx); > + } > + } > + > ifcvf_stop_hw(hw); >=20 > for (i =3D 0; i < hw->nr_vring; i++) > @@ -641,8 +667,10 @@ struct rte_vdpa_dev_info { > } > hw->vring[i].avail =3D gpa; >=20 > - /* Direct I/O for Tx queue, relay for Rx queue */ > - if (i & 1) { > + /* NET: Direct I/O for Tx queue, relay for Rx queue > + * BLK: relay every queue > + */ > + if ((internal->hw.device_type =3D=3D IFCVF_NET) && (i & 1)) { > gpa =3D hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); > if (gpa =3D=3D 0) { > DRV_LOG(ERR, "Fail to get GPA for used ring."); > @@ -692,8 +720,12 @@ struct rte_vdpa_dev_info { >=20 > for (i =3D 0; i < hw->nr_vring; i++) { > /* synchronize remaining new used entries if any */ > - if ((i & 1) =3D=3D 0) > + if (internal->hw.device_type =3D=3D IFCVF_NET) { > + if ((i & 1) =3D=3D 0) > + update_used_ring(internal, i); > + } else if (internal->hw.device_type =3D=3D IFCVF_BLK) { > update_used_ring(internal, i); > + } >=20 > rte_vhost_get_vhost_vring(vid, i, &vq); > len =3D IFCVF_USED_RING_LEN(vq.size); > @@ -755,7 +787,9 @@ struct rte_vdpa_dev_info { > } > } >=20 > - for (qid =3D 0; qid < q_num; qid +=3D 2) { > + for (qid =3D 0; qid < q_num; qid +=3D 1) { > + if ((internal->hw.device_type =3D=3D IFCVF_NET) && (qid & 1)) > + continue; > ev.events =3D EPOLLIN | EPOLLPRI; > /* leave a flag to mark it's for interrupt */ > ev.data.u64 =3D 1 | qid << 1 | > -- > 1.8.3.1