From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E8DDFA00BE; Mon, 25 Apr 2022 15:10:28 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8BFCC41109; Mon, 25 Apr 2022 15:10:28 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 7043F40E78 for ; Mon, 25 Apr 2022 15:10:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1650892226; x=1682428226; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=djpKQUqid5zP0w4Ra+1ydvL2Xt1KzrkJLWJEEJlcuL4=; b=hmbzN1fKnGcX60LXfvgHqRF/0mgf5g8kq6aCXFTwKQ/8TLQiohsWLkL2 PGc1SUuHMU/4trXZpNCV09H2yAwgWFnFqkx+G79SPhFBcCehWRvWesOZr Dc6LsPxlJHKRwKtQrsHfGisrMxNeXvAm8A7WWMaqsFVkSBAxRqArh23wu 9VXpo/pNgxbL3T6pL6KpOirnmh1RrJIcQzVAGMUYLKVnlV4tEpmvGRYOp CtzMKpILjiailCP3amtclgF1tjNKsokUFHUJiVmDnqPozPnDWlzchCu6/ wrMtmJVxVX7JMnmoQcywEzD8dc9JZvxPTuOeenarX9i200P1Df7cX2rlj g==; X-IronPort-AV: E=McAfee;i="6400,9594,10327"; a="245167158" X-IronPort-AV: E=Sophos;i="5.90,288,1643702400"; d="scan'208";a="245167158" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Apr 2022 06:10:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,288,1643702400"; d="scan'208";a="616510322" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga008.fm.intel.com with ESMTP; 25 Apr 2022 06:10:09 -0700 Received: from fmsmsx606.amr.corp.intel.com (10.18.126.86) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Mon, 25 Apr 2022 06:10:08 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx606.amr.corp.intel.com (10.18.126.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Mon, 25 Apr 2022 06:10:08 -0700 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (104.47.57.49) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Mon, 25 Apr 2022 06:10:08 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VtsrpwwotEA/4H2lWBXGr9STZMwRpypkETCccgCPQIfaShmpcrrDQKWy0I9Kz/yT+G5FN1mZ+tKzgWqEbcuD7pUISnoj3RPTPeoacFJJCcsYNMUoY3+ST32fdXIWR4ktkhiv7P33oA8EGl/zuQ3XFfaDM1cpbopY3G3nlQ/jSiQNWUibd+96rSPiXsidzfIXTTc2EZA4IykaCskcgY2HABRRyx8cK5ZovdD/+zG8RhLLrFH0PcAH2gayNMILCfi5eeqmolXvjDtxUmZ4cdOBWOcjm/d+U0cej+0sa3ymCWzn3/C6PfoeaJ66h+y06ysGJRjXInmFVXERoC7r0OSSsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G9it+tFH8D6UeCsrpijrOefVzSNurhxqux8iQ3U3/1A=; b=Vlzd/7RZndoqPCOs6h3h1hj2kWjA3UqweJ9JhLN93xmSvhXQqHZLRQOu6bUUK9scDzBSMVYcj7Q5Yje6M/t71RWo1rY/ZR218hdQlm51cK7JMcEiEi12sWtMy3zgf1r8m2itNEMsBBDOBAUVVYcyOH92ct97zjicDDDaosjNGDopuZ+5fvQ9K0mM7u358fauIz+GFf3nQNJxSengLzsUCNpxQRHdtXyG8Ncuhr3aB561/S2TAC+mRcOH0LDiC4UAvezxSax6GOV/1ysIJjAlvDZVP9bAIDlynE/7LFP/vT68GtX07KW+C9HxCKMKLhhAubqYBwQskw32oeoRLrHMMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17) by DM4PR11MB5565.namprd11.prod.outlook.com (2603:10b6:5:39e::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5186.15; Mon, 25 Apr 2022 13:10:06 +0000 Received: from SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::4d1d:6453:f00c:cf8a]) by SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::4d1d:6453:f00c:cf8a%6]) with mapi id 15.20.5186.020; Mon, 25 Apr 2022 13:10:06 +0000 From: "Xia, Chenbo" To: "Pei, Andy" , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "Cao, Gang" , "Liu, Changpeng" Subject: RE: [PATCH v6 06/16] vdpa/ifc: add block device SW live-migration Thread-Topic: [PATCH v6 06/16] vdpa/ifc: add block device SW live-migration Thread-Index: AQHYVWD65JXsuzZrC0WNxheAnlGfoa0Antdg Date: Mon, 25 Apr 2022 13:10:06 +0000 Message-ID: References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1650530034-59744-1-git-send-email-andy.pei@intel.com> <1650530034-59744-7-git-send-email-andy.pei@intel.com> In-Reply-To: <1650530034-59744-7-git-send-email-andy.pei@intel.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 47f403b0-938e-4539-e2e4-08da26bceb09 x-ms-traffictypediagnostic: DM4PR11MB5565:EE_ x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: X1V7KMHzYvW8mBeYMeluo1YC0v/bFetxkfKAiBPX/+ceR2RPab2CRUDjng5vxO5BHxg2TpAhOb+7g3neH/5EnHYvrFDgaWt01SezPlYLvxb+94DSBCulYCkc4xlaL4pkjQKXFSceeOnAC/hAIs0+OTvsRd82LzhGpnFIEOP33Tz2ruPE+96gIste5yh51GIurMhLPOPrqjcsguGupqW/0eSSfan5LS+9NaPiiBiVqVkC8XzQc109PjWUAWrw1HJBGVHouvilquPRFdUQtGa5u4zEIEcPM5ev75GAY82uoSsByktz1hz1789WHVtET038YfCJYwd0BZnKLw77eyGhsVinB/NgDkoao+dkF4ivH0QRWYcfOwxE0daDtZCCUt0UNQs8S94PhA+DE9OcVFiCjweEBvkX5U8NVwtOS9ctwhuXPiOGMl3Ok2NbnEWTFq+NLjliW1zJE2OKxZ6No8KVCqZMHp7gyKgJ490XLv8nlTOQvkfb38Kto2u5NcacENG2zqRfL/vrojg4e2u1o04CfC7BY+3aYW24VQYuHMcYL93vG3nAJv3RnG5UBnsoEI1CVkO0pzzvNtbbDr3vwKvED8CnkGs9R0zizle0YSujO1KdWp5x4cgdvLJYAnP+wbW3c4vbCizGejTIdSCW6hAX+ISsCwEkgM/gewC/KXvFWKGqDmm0x9x/5lMxEqLtZtvizmhw90RswB8k+6F1IdvRjQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(83380400001)(110136005)(54906003)(8936002)(53546011)(55016003)(316002)(107886003)(186003)(71200400001)(4326008)(33656002)(66556008)(52536014)(8676002)(64756008)(76116006)(66446008)(66476007)(26005)(66946007)(82960400001)(122000001)(38100700002)(38070700005)(508600001)(2906002)(5660300002)(9686003)(6506007)(86362001)(7696005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?1Lvdu+uD9ZiLpXVw/ERsMpbSR7TTb0fMAJArA6aZfFaaZGFwVVVSZupnqvDL?= =?us-ascii?Q?3YfHtMfJWIQYL7+xGdg5I47KOOD1uj0el02f043bOLWiLuUcI85+dhIGrPmk?= =?us-ascii?Q?awnnvKNlpK5CV05QyskpKDepIYt2OYuk3Rm60X8XEnZVz67dyv/ob42qvhoC?= =?us-ascii?Q?l+CCEoUpuuwapvMRsBgtELH6o/FJFidLPDAooxdLw36fla7jpww9yygu+lx+?= =?us-ascii?Q?ivI0CLAcJNaLw9ejHysb7JZvIiW2NjJSlH5ntsj7AZCGHNzu9OxNvdFItEqt?= =?us-ascii?Q?j5XqTXtmALcHmRYXKirkRICHa8zZikuOQfmhGfLHotfehS+gLfpVnYWQpSg4?= =?us-ascii?Q?Jcl7dHM1vT1oWNvyKFTYvrdfMj23fiyfhhlGb+Y3H5AcDKEx04EvcF1uGDQk?= =?us-ascii?Q?KIRGnDOH2PTlYvo5K4A15JjBKVS6qmZhhZl5pqtrnVATZHYkSnQ//WKfRY5Q?= =?us-ascii?Q?lfG8790uyOO9Url9coi9l5apKX+qwHD9GpEuFRzhw+HmjZgFvSy5y5l3+122?= =?us-ascii?Q?56D1rwJ3OtiEqv69M+qLJz6+UcYpMcXvB856NjvWokszvHeLTL5yYY9EJGor?= =?us-ascii?Q?zyYbWEx1/EjNeRvg2zO2kh5FKIb1qwDk/ePGXgSmIH7I4GvdM4pJHLENws8o?= =?us-ascii?Q?gfU5Zw6/2/vTginkKeNO+V5cvxH1+hONYI9Ar6txOqgcpJigQ7Yj1pXYrtVq?= =?us-ascii?Q?fIqM5SPex45L3mwiWhxMOkdhYxF0yh7M4cP/sZwT5tzBI1A45Ajn9BPEnA3Q?= =?us-ascii?Q?3TCAyFY0fwaDLZtJndB1AIF6jgQRBZM/Ny9aqnWXBDlsgwUiXZ3BYXUE7Q49?= =?us-ascii?Q?ZCK5ZQIYfBu5/dVPygwe0znvMIdbRqxgCXlxLvBevDEj5k7aBepxBGynZ5i2?= =?us-ascii?Q?pIqxuA+paGJyFUjvYOiyN8wiJN2kP2geHyzqXSHRochxjCHmXKMoZ3ZQ8XCu?= =?us-ascii?Q?Yc0PdgPsitVOLvmkIrfUa/kg/8AFai9bDSI84p74InTaVU/myr5aWVLE74ao?= =?us-ascii?Q?ayYY9P/EePK0Kh+hBTq4KVNPVuoBQEroVwHUaoV1vIf0DuQWa3LTkZQ0qzgl?= =?us-ascii?Q?/Dc2pYF4VtajlWktyvQY/w5k1F02FOovA+YBaJsaHLh0c2lLL1702HlY5Se2?= =?us-ascii?Q?1z1HFWVMKMXrR1uOhLcnQcpteKMbjJ+0YJnCYD4or9w5fNreEVqMo4a2HtpM?= =?us-ascii?Q?BH9Ot3gESR1i0kCFSNnCQHMTZRmCgIqRgfVNdEuL/H0R5+GOwd15icJ2cPZ4?= =?us-ascii?Q?jE4x3yAWTo5DZ7F67TJI28bi2iA55o4JTPasRd+mqMfCvyOAc/lPC9K+Ppwg?= =?us-ascii?Q?pE6pM5Fl3ybUKphOUQS3/c8VmKn36DhhSiKAboctC8Y6Fg6SfHeXMGfklIl0?= =?us-ascii?Q?zh0zs2/wmCNHek2/fp1ziDL337I8tEbcsfQVHZ8l8yq9PWIWtcyO94fQGsHx?= =?us-ascii?Q?t/qBozv+eErO2EwbWKMY511lns2cCzWqy8Of18cMOcDHqZY0pPkAuBYq8SzU?= =?us-ascii?Q?R3E/CtgJfK/fa3lMea/PjX5WVxb4a5sV6F2+3aeZjK81K9MEtrnv9a0PXEMr?= =?us-ascii?Q?5JkkGkYlgiX6nOFb7lOETokjKqwZUXyZuHfu+DwGaSVG8V29viVQq6suJXqq?= =?us-ascii?Q?k5R79Keno+AjWAjnX81O6mhWVDfc01QFDrMTljl6BIhl4yBYKbFX2CFaqrk8?= =?us-ascii?Q?BMgEJz9MtjpcHxTE2qNMo6B+sqZ6hWufi1RwuI0PvdBmfyagrmJS0UmxocCZ?= =?us-ascii?Q?f8NTIRHEYQ=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 47f403b0-938e-4539-e2e4-08da26bceb09 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2022 13:10:06.4878 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 27eExhCdhZK80uPPBHzs+Q+ynqwxNphTxQdwliKnvw1ThUdcwjNtSTub5CPJhfy5q+tUCTS18M2pr7qThUgu3g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5565 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Pei, Andy > Sent: Thursday, April 21, 2022 4:34 PM > To: dev@dpdk.org > Cc: Xia, Chenbo ; maxime.coquelin@redhat.com; Cao, > Gang ; Liu, Changpeng > Subject: [PATCH v6 06/16] vdpa/ifc: add block device SW live-migration >=20 > Add SW live-migration support to block device. > Add dirty page logging to block device. Add SW live-migration support including dirty page logging for block device= . >=20 > Signed-off-by: Andy Pei > --- > drivers/vdpa/ifc/base/ifcvf.c | 4 +- > drivers/vdpa/ifc/base/ifcvf.h | 6 ++ > drivers/vdpa/ifc/ifcvf_vdpa.c | 128 +++++++++++++++++++++++++++++++++++-= - > ----- > 3 files changed, 115 insertions(+), 23 deletions(-) >=20 > diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.= c > index d10c1fd..e417c50 100644 > --- a/drivers/vdpa/ifc/base/ifcvf.c > +++ b/drivers/vdpa/ifc/base/ifcvf.c > @@ -191,7 +191,7 @@ > IFCVF_WRITE_REG32(val >> 32, hi); > } >=20 > -STATIC int > +int > ifcvf_hw_enable(struct ifcvf_hw *hw) > { > struct ifcvf_pci_common_cfg *cfg; > @@ -240,7 +240,7 @@ > return 0; > } >=20 > -STATIC void > +void > ifcvf_hw_disable(struct ifcvf_hw *hw) > { > u32 i; > diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.= h > index 769c603..6dd7925 100644 > --- a/drivers/vdpa/ifc/base/ifcvf.h > +++ b/drivers/vdpa/ifc/base/ifcvf.h > @@ -179,4 +179,10 @@ struct ifcvf_hw { > u64 > ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid); >=20 > +int > +ifcvf_hw_enable(struct ifcvf_hw *hw); > + > +void > +ifcvf_hw_disable(struct ifcvf_hw *hw); > + > #endif /* _IFCVF_H_ */ > diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.= c > index 8d104b7..a23dc2d 100644 > --- a/drivers/vdpa/ifc/ifcvf_vdpa.c > +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c > @@ -345,6 +345,56 @@ struct rte_vdpa_dev_info { > } > } >=20 > +static void > +vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) > +{ > + struct ifcvf_hw *hw =3D &internal->hw; > + struct rte_vhost_vring vq; > + int i, vid; > + uint64_t features =3D 0; > + uint64_t log_base =3D 0, log_size =3D 0; > + uint64_t len; > + > + vid =3D internal->vid; > + > + if (internal->device_type =3D=3D IFCVF_BLK) { > + for (i =3D 0; i < hw->nr_vring; i++) { > + rte_vhost_get_vhost_vring(internal->vid, i, &vq); > + while (vq.avail->idx !=3D vq.used->idx) { > + ifcvf_notify_queue(hw, i); > + usleep(10); > + } > + hw->vring[i].last_avail_idx =3D vq.avail->idx; > + hw->vring[i].last_used_idx =3D vq.used->idx; > + } > + } > + > + ifcvf_hw_disable(hw); > + > + for (i =3D 0; i < hw->nr_vring; i++) > + rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, > + hw->vring[i].last_used_idx); > + > + if (internal->sw_lm) > + return; > + > + rte_vhost_get_negotiated_features(vid, &features); > + if (RTE_VHOST_NEED_LOG(features)) { > + ifcvf_disable_logging(hw); > + rte_vhost_get_log_base(internal->vid, &log_base, &log_size); > + rte_vfio_container_dma_unmap(internal->vfio_container_fd, > + log_base, IFCVF_LOG_BASE, log_size); > + /* > + * IFCVF marks dirty memory pages for only packet buffer, > + * SW helps to mark the used ring as dirty after device stops. > + */ > + for (i =3D 0; i < hw->nr_vring; i++) { > + len =3D IFCVF_USED_RING_LEN(hw->vring[i].size); > + rte_vhost_log_used_vring(vid, i, 0, len); > + } > + } > +} Can we consider combining vdpa_ifcvf_blk_pause and vdpa_ifcvf_stop to one f= unction and check device type internally to do different things? Because as I see, most= logic is=20 the same. > + > #define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \ > sizeof(int) * (IFCVF_MAX_QUEUES * 2 + 1)) > static int > @@ -659,15 +709,22 @@ struct rte_vdpa_dev_info { > } > hw->vring[i].avail =3D gpa; >=20 > - /* Direct I/O for Tx queue, relay for Rx queue */ > - if (i & 1) { > - gpa =3D hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); > - if (gpa =3D=3D 0) { > - DRV_LOG(ERR, "Fail to get GPA for used ring."); > - return -1; > + if (internal->device_type =3D=3D IFCVF_NET) { > + /* Direct I/O for Tx queue, relay for Rx queue */ > + if (i & 1) { > + gpa =3D hva_to_gpa(vid, > (uint64_t)(uintptr_t)vq.used); > + if (gpa =3D=3D 0) { > + DRV_LOG(ERR, "Fail to get GPA for used > ring."); > + return -1; > + } > + hw->vring[i].used =3D gpa; > + } else { > + hw->vring[i].used =3D m_vring_iova + > + (char *)internal->m_vring[i].used - > + (char *)internal->m_vring[i].desc; > } > - hw->vring[i].used =3D gpa; > - } else { > + } else if (internal->device_type =3D=3D IFCVF_BLK) { > + /* BLK: relay every queue */ > hw->vring[i].used =3D m_vring_iova + > (char *)internal->m_vring[i].used - > (char *)internal->m_vring[i].desc; > @@ -686,7 +743,10 @@ struct rte_vdpa_dev_info { > } > hw->nr_vring =3D nr_vring; >=20 > - return ifcvf_start_hw(&internal->hw); > + if (internal->device_type =3D=3D IFCVF_NET) > + return ifcvf_start_hw(&internal->hw); > + else if (internal->device_type =3D=3D IFCVF_BLK) > + return ifcvf_hw_enable(&internal->hw); >=20 > error: > for (i =3D 0; i < nr_vring; i++) > @@ -710,8 +770,12 @@ struct rte_vdpa_dev_info { >=20 > for (i =3D 0; i < hw->nr_vring; i++) { > /* synchronize remaining new used entries if any */ > - if ((i & 1) =3D=3D 0) > + if (internal->device_type =3D=3D IFCVF_NET) { > + if ((i & 1) =3D=3D 0) > + update_used_ring(internal, i); > + } else if (internal->device_type =3D=3D IFCVF_BLK) { > update_used_ring(internal, i); > + } >=20 > rte_vhost_get_vhost_vring(vid, i, &vq); > len =3D IFCVF_USED_RING_LEN(vq.size); > @@ -773,17 +837,36 @@ struct rte_vdpa_dev_info { > } > } >=20 > - for (qid =3D 0; qid < q_num; qid +=3D 2) { > - ev.events =3D EPOLLIN | EPOLLPRI; > - /* leave a flag to mark it's for interrupt */ > - ev.data.u64 =3D 1 | qid << 1 | > - (uint64_t)internal->intr_fd[qid] << 32; > - if (epoll_ctl(epfd, EPOLL_CTL_ADD, internal->intr_fd[qid], &ev) > - < 0) { > - DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); > - return NULL; > + if (internal->device_type =3D=3D IFCVF_NET) { > + for (qid =3D 0; qid < q_num; qid +=3D 2) { > + ev.events =3D EPOLLIN | EPOLLPRI; > + /* leave a flag to mark it's for interrupt */ > + ev.data.u64 =3D 1 | qid << 1 | > + (uint64_t)internal->intr_fd[qid] << 32; > + if (epoll_ctl(epfd, EPOLL_CTL_ADD, > + internal->intr_fd[qid], &ev) > + < 0) { > + DRV_LOG(ERR, "epoll add error: %s", > + strerror(errno)); > + return NULL; > + } > + update_used_ring(internal, qid); > + } > + } else if (internal->device_type =3D=3D IFCVF_BLK) { > + for (qid =3D 0; qid < q_num; qid +=3D 1) { > + ev.events =3D EPOLLIN | EPOLLPRI; > + /* leave a flag to mark it's for interrupt */ > + ev.data.u64 =3D 1 | qid << 1 | > + (uint64_t)internal->intr_fd[qid] << 32; > + if (epoll_ctl(epfd, EPOLL_CTL_ADD, > + internal->intr_fd[qid], &ev) > + < 0) { > + DRV_LOG(ERR, "epoll add error: %s", > + strerror(errno)); > + return NULL; > + } > + update_used_ring(internal, qid); It seems we can also reduce duplicate code for above case. And for other ch= ecks, if we can use only one combined condition to check, I prefer to just use on= e. Thanks, Chenbo > } > - update_used_ring(internal, qid); > } >=20 > /* start relay with a first kick */ > @@ -871,7 +954,10 @@ struct rte_vdpa_dev_info { >=20 > /* stop the direct IO data path */ > unset_notify_relay(internal); > - vdpa_ifcvf_stop(internal); > + if (internal->device_type =3D=3D IFCVF_NET) > + vdpa_ifcvf_stop(internal); > + else if (internal->device_type =3D=3D IFCVF_BLK) > + vdpa_ifcvf_blk_pause(internal); > vdpa_disable_vfio_intr(internal); >=20 > ret =3D rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); > -- > 1.8.3.1