From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C537CA00BE; Thu, 12 May 2022 14:55:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F8E9410EF; Thu, 12 May 2022 14:55:11 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 852FC40E64 for ; Thu, 12 May 2022 14:55:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652360109; x=1683896109; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=jVXTUaNTliGZ5B3e/olrgYiljVlPfglzJlpOxGPAbOw=; b=FhHeMn/Dmz9r3vNtIFETtA8orJV4D8iehQiaKozrrJtneMXG212k0tbJ 8c9LuG//LpCRMdIkS3zJkw7aax3bhrTjNmBCI0hVQEXt6joaVQtDvRl3P a7orFYEV7LzwUFs0+5vsJNidXhdN4wDklBXbGS3ttx0GYcruB6Nm+ZR+M O6OXhm4uv5c1uoU/XwcXdB4a/4JLbWtAYwTHbsuJUGHhX60dr1JrHGCfp lzDe6mpwpIySseln/C5XwGkBmtnLDk28UKOcV04odkuPUdJJUa0WW20dZ /2xw5dduIBNeOZCUOUwJBMG30PdUxSh6LkQ+NHYx2/duPpZzFUmzPm10x w==; X-IronPort-AV: E=McAfee;i="6400,9594,10344"; a="257537528" X-IronPort-AV: E=Sophos;i="5.91,219,1647327600"; d="scan'208";a="257537528" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 May 2022 05:55:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,219,1647327600"; d="scan'208";a="566670062" Received: from orsmsx606.amr.corp.intel.com ([10.22.229.19]) by orsmga007.jf.intel.com with ESMTP; 12 May 2022 05:55:07 -0700 Received: from orsmsx606.amr.corp.intel.com (10.22.229.19) by ORSMSX606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Thu, 12 May 2022 05:55:06 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Thu, 12 May 2022 05:55:06 -0700 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (104.47.74.45) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Thu, 12 May 2022 05:55:05 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oVD+W2h1IA1fRtzuIR7SnHatXNh/F51XxIxR4nQS8M4k6F93AssHKiIQ/+Zz+QWDCGZw8CieA5KoBsM6aNC6BRteRwMvFBzWOSCAUXy2XhaKEma++6f7cpEeCNME6UQ9HCraWfZMSfy4i9HVEgIR1EM7AbGS8/ZQDCKI+3c20iMQoTTfNd2aEexRIUvo3VREeYl/9AtmXfaDfv8e5kGIh7Ec8JK5IJ6r5xh178XGBRCJ3QxbOw4IVjXIhq9h4GP3dUwOYjZyNB43bZUNigBsX0RKpDzPbicAYfvsnHqNwCZq8VdPQzP/Fccqpad+doqCRc0E4PgkPefZqmM+zslKQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=baR+kYyRgt8VJvtUR7RtP+w/s1aZ50XfORsCVXDNW5M=; b=L0qtgDj1qRsbk7qZjy+V/scIzRuK4JIguXdSuHw1jUUUs6wYpbeW1QhgQaPMmjG0ss1oMYVAvmMHe1YLgqEL/voJpcsmiTNX4xifqkrAbx2z3FkK5djhQHkR8PsOFsQCqeuAcr0b1thiSh8hPPGPEK2UknzzYMjpp9Ioof3c5iBu4r9uDewRdYB7Ykv7zPGxLgvhRsiPxpDB0XILpSgm5JJExziLj97J6a2zQRyzu6K/aHv0ji0qqSyxAU4xe9UNbHfji0MzMORW5v4IWpBK/QTwUSFRNUVhAjtgnPEp3pE9As0Dxm3/51d9VAjFwHY4jXlRe0Xjd0EkUR3GgyISnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17) by DM6PR11MB2841.namprd11.prod.outlook.com (2603:10b6:5:c8::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23; Thu, 12 May 2022 12:55:03 +0000 Received: from SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::2c8f:42df:ed85:99e5]) by SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::2c8f:42df:ed85:99e5%4]) with mapi id 15.20.5227.023; Thu, 12 May 2022 12:55:02 +0000 From: "Xia, Chenbo" To: "Pei, Andy" , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "Cao, Gang" , "Liu, Changpeng" Subject: RE: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migration Thread-Topic: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migration Thread-Index: AQHYWheDsJei7826WUGDCpro8PYkba0bSdbQ Date: Thu, 12 May 2022 12:55:02 +0000 Message-ID: References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> <1651048206-282372-7-git-send-email-andy.pei@intel.com> In-Reply-To: <1651048206-282372-7-git-send-email-andy.pei@intel.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3b902e1a-8288-4f1c-74b2-08da3416a169 x-ms-traffictypediagnostic: DM6PR11MB2841:EE_ x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Lt3CBQaBCqk47X36ZEe1w9iiiqT6si5Yf6so4SHCR0JNgtBljKaTBCtXA3qMEKwv1g3bL45ErHYtaf07nvKOcYyiyxtZoJwPVtEiiFApzorwgleuUm3JAO+8uwo5E6Ppsix4J4m103HOIorgIsm8DSk0PPPfl6RDqGt6h2KKA/VdL4vAmHPN3RMhgNn7Blx60Xzyf2BhVGDwDDDovpcSOWOiArmPjnUcXy08kKPpv6glifMq9xrKMcW72MP4sfjX9SNQP1c6qIsaC5o+ljlzufaKprnVnQj2/C6uRzl8q3e2FookpG7rizixeaht78yi4Om452yHz1rlzuKGhJ4IftKN7NO+bie6Wbl9LrV0d/jVCDuwnswRMcdEqjezc8qrnUFbnoXQL4t+3F7gKq1akDBbNHA93yRdQnT1viHdTBE/2yeDicMig9tU/skRJwbAMKRhQjZbQsR40AZyD20MfLeL6YUs86wswhHXRSXDPmBI37fP698RwxlRDylXCaRsEPrPdyHm5ZZu+YrwCxront0KS4ixnAxBLQU6UDH1hd4yPIZNmENEH5ufj5hwGJ2Aa7x3Y05JIQaviarCASWuJ+c4MGcqZB8WdyVc0qOLa4WiDkh+Mk8eU6MkQ3Ri60rXJ+ujEdAXFyEjD8E7PMIbmd4xP2rmsfrlKEZ0vMQSxHTEBGBXmihM991vepKwwrJ2d2ZJkVeDbDwtl0F7tnlXJA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(5660300002)(55016003)(186003)(86362001)(64756008)(82960400001)(38100700002)(38070700005)(122000001)(71200400001)(83380400001)(66946007)(508600001)(4326008)(8676002)(66476007)(66446008)(66556008)(316002)(2906002)(76116006)(52536014)(6506007)(7696005)(26005)(53546011)(54906003)(33656002)(9686003)(110136005)(8936002)(107886003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?8lAZ3L2+vvn5zUWboyAFacPQmSxd8Qn4P7p8ZNjpg+QMgZOh7N6pjNRNCnk4?= =?us-ascii?Q?ptJKzjxD6ThhyaMwU4YycskJfElESu5GuT2t8MTJec6F1VkxglKvHvEv8G0A?= =?us-ascii?Q?UzAn3qMb6shwhLJ1gLo5PJRRHTKw4ICa0PCgfcHtAdV4ahS/0Ui+NFHa2oCf?= =?us-ascii?Q?mcaHAt+m+YWj40Ebt4WAVc98DYrHdia03ubmjxo05XrS9PmE8KFygoivJYsU?= =?us-ascii?Q?c7OwN+haZUp9z7ysgoMEwvU1iKQdu6knlXURUlIsc9znn5n+q8eat3yZZNXZ?= =?us-ascii?Q?HlVcc9I+EHf31fcEfDZV2QBjPnkhv8Jl2rcNlU+qK/g6YcPgDqqPmJVFV9tO?= =?us-ascii?Q?+Mkl5iuVwhh2NInDFTKLvJi87BQmMv1+a8TyqLxrJcHIyvobgsca9p2L0v4t?= =?us-ascii?Q?t85JdHEfAMLmMjb/twZK4UT/I7pQ6rQ47mqbmu7JQHkm4yJ9IPzuoEvXXZBE?= =?us-ascii?Q?lHuV+fI3+OT/2m6ynOa26o8H36I1YU//r4BRkbxYTRh21Im+3WTO3/A3IwId?= =?us-ascii?Q?a6eMt/v05pqDkv72RJKqKpAkv71+FjPowwH8kuosSxBlg+YG+nTyEEv6xYtJ?= =?us-ascii?Q?yQAAVj7YobhUHacrV+Pde/JBN9HuzzYZ6Iybiww8ahvpH+qIcCJUOIXfviZJ?= =?us-ascii?Q?QNes8iLGeYu6RZrg3UWOPBpubPyQZno1IeIp26WHbafu/lZaD+I8tJPm5LUf?= =?us-ascii?Q?JzMBswLJkXFz5oqABk614Nj+pJPv3bbZSB+prgndWEkhqdFxnAq4Jp4H19B+?= =?us-ascii?Q?Z+XhhHv/xdS3Hj49c6gbLtppNUqyJEHrhRt0JUqvg+bqdG8Lr6hb4ErqwNkO?= =?us-ascii?Q?K0yNY9mgVlRehLyK/dMaXnPpoLgXlI68cjL6d2GpvG9u3dL8yGbEaJ3T8EIu?= =?us-ascii?Q?s756W1lCMXoeYb1CBGAqtkPlmqPlr9qzzXUwqq2GV1xylxDCcRhU13X1ngeI?= =?us-ascii?Q?iY8b/ce62Ft2YVHLKwoPorX9fT0tN1RWfAyEmlJtClkG75INuUGVSIiTEgAK?= =?us-ascii?Q?QDmm+43o8aHC9VZ73wCX+zGT1UnBPDkC/xvjtNfFvzjEa5GE9v4CKuecw2yt?= =?us-ascii?Q?uHVcJGViG1WU30vHCBNbb736qf06t5UvRcSq50PjsQKfFcGmWS0QA9+xB6+p?= =?us-ascii?Q?5WIk1+zCs0npKnm2b8xMlvW+TLnF97zVKgfTFnoJA4R8qGYq67fAm7SHbmRF?= =?us-ascii?Q?5HStjZB+9ib+fsbPGHW4XJDVnEPOHwCz2jUopGoAonUUzl++e4pfiy50HfDV?= =?us-ascii?Q?e84ftKdIGymCfR1LaltGB0AjLsdKvr6I+GYeihUnD9mehFNixL1QqQREpsMQ?= =?us-ascii?Q?bWYdBEE1FP4zoRTus/7sbEH7bFfHWeexvusllp8oNrI1konwPRCbDGn1xR2t?= =?us-ascii?Q?NefPUMsH4cJllnwrps16HUsQzRaQ3esOU+SxvF3L5nWl+np5I2AQzT+jkLIn?= =?us-ascii?Q?wTHILbmJ6aoaMiBPjmgF9ZTp+2ZJt0pJFBXMSFnLrlswwXvFjt3ggUtjU/cz?= =?us-ascii?Q?QST9vOoSF6+twQU3mMsgCdr730RfBxDGo0OoS/srKBxImORQpHM2s4Q2sUuT?= =?us-ascii?Q?1/nOwjfkrOVmM+djeiV0ayvy2GncO+aW3Oe6p9DSZGgu+eZNNl3KHLIRNCn9?= =?us-ascii?Q?l+wMK3QgLTVIYTiTGjZ20YtQ6jsTpg5dc7kvhk4H341GoyOCU0HBjUmLX8Pq?= =?us-ascii?Q?6xUtkHZ+auZPYcQoO7sGuN2vaFmB42TocUyppFmjnRFN3wVG1iIy8cit/IuI?= =?us-ascii?Q?eNMHYKA9rA=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3b902e1a-8288-4f1c-74b2-08da3416a169 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 May 2022 12:55:02.8158 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: sV+SIgMrUuNxcfx3Cm/tD8qYASV/c+smMCQqwLZtLT+VL674+iSb545nbbOPVUa/UTA5Sdeqz/PE/fStNY8qMA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB2841 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Pei, Andy > Sent: Wednesday, April 27, 2022 4:30 PM > To: dev@dpdk.org > Cc: Xia, Chenbo ; maxime.coquelin@redhat.com; Cao, > Gang ; Liu, Changpeng > Subject: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migration >=20 > Add SW live-migration support to block device. >=20 > Signed-off-by: Andy Pei > --- > drivers/vdpa/ifc/ifcvf_vdpa.c | 33 +++++++++++++++++++++++++++++---- > 1 file changed, 29 insertions(+), 4 deletions(-) >=20 > diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.= c > index 07fc3ca..8a260b7 100644 > --- a/drivers/vdpa/ifc/ifcvf_vdpa.c > +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c > @@ -312,6 +312,7 @@ struct rte_vdpa_dev_info { > vdpa_ifcvf_stop(struct ifcvf_internal *internal) > { > struct ifcvf_hw *hw =3D &internal->hw; > + struct rte_vhost_vring vq; > uint32_t i; > int vid; > uint64_t features =3D 0; > @@ -319,6 +320,22 @@ struct rte_vdpa_dev_info { > uint64_t len; >=20 > vid =3D internal->vid; > + > + /* to make sure no packet is lost for blk device > + * do not stop until last_avail_idx =3D=3D last_used_idx > + */ > + if (internal->device_type =3D=3D IFCVF_BLK) { > + for (i =3D 0; i < hw->nr_vring; i++) { > + rte_vhost_get_vhost_vring(internal->vid, i, &vq); > + while (vq.avail->idx !=3D vq.used->idx) { > + ifcvf_notify_queue(hw, i); > + usleep(10); > + } > + hw->vring[i].last_avail_idx =3D vq.avail->idx; > + hw->vring[i].last_used_idx =3D vq.used->idx; > + } > + } > + This seems not match with the above comment about avoiding in-flight packet= s. But the change in patch 17 seems good. Why not just using the implementatio= n in patch 17? Thanks, Chenbo > ifcvf_stop_hw(hw); >=20 > for (i =3D 0; i < hw->nr_vring; i++) > @@ -642,8 +659,10 @@ struct rte_vdpa_dev_info { > } > hw->vring[i].avail =3D gpa; >=20 > - /* Direct I/O for Tx queue, relay for Rx queue */ > - if (i & 1) { > + /* NET: Direct I/O for Tx queue, relay for Rx queue > + * BLK: relay every queue > + */ > + if ((internal->device_type =3D=3D IFCVF_NET) && (i & 1)) { > gpa =3D hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); > if (gpa =3D=3D 0) { > DRV_LOG(ERR, "Fail to get GPA for used ring."); > @@ -693,8 +712,12 @@ struct rte_vdpa_dev_info { >=20 > for (i =3D 0; i < hw->nr_vring; i++) { > /* synchronize remaining new used entries if any */ > - if ((i & 1) =3D=3D 0) > + if (internal->device_type =3D=3D IFCVF_NET) { > + if ((i & 1) =3D=3D 0) > + update_used_ring(internal, i); > + } else if (internal->device_type =3D=3D IFCVF_BLK) { > update_used_ring(internal, i); > + } >=20 > rte_vhost_get_vhost_vring(vid, i, &vq); > len =3D IFCVF_USED_RING_LEN(vq.size); > @@ -756,7 +779,9 @@ struct rte_vdpa_dev_info { > } > } >=20 > - for (qid =3D 0; qid < q_num; qid +=3D 2) { > + for (qid =3D 0; qid < q_num; qid +=3D 1) { > + if ((internal->device_type =3D=3D IFCVF_NET) && (qid & 1)) > + continue; > ev.events =3D EPOLLIN | EPOLLPRI; > /* leave a flag to mark it's for interrupt */ > ev.data.u64 =3D 1 | qid << 1 | > -- > 1.8.3.1