From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 4CB24A00C3;
Fri, 13 May 2022 05:32:37 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 09D2240DDE;
Fri, 13 May 2022 05:32:37 +0200 (CEST)
Received: from mga09.intel.com (mga09.intel.com [134.134.136.24])
by mails.dpdk.org (Postfix) with ESMTP id 979B040698
for <dev@dpdk.org>; Fri, 13 May 2022 05:32:35 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;
d=intel.com; i=@intel.com; q=dns/txt; s=Intel;
t=1652412755; x=1683948755;
h=from:to:cc:subject:date:message-id:references:
in-reply-to:content-transfer-encoding:mime-version;
bh=MGMo3/zSZvxN6soSmBuiPzAtkgaFh5HEAeTqIJ35rVo=;
b=cl6l9y5ok6KUgZFz0dObX5Y4JEUEcCeM1EydQczrSEjpe4o501cOE6u0
5H6Hjkn2R8AQbSV9VpjDCKWszvCEvNT3CBYV8z6xmMgtnkQaqe9QDxCKW
Cg607GcDs/5lhIDbcSvUjz1eLYzIRTLU0kaK+8+8jJiVgmvSYmxgi55mi
UYtx3kNxSPmu6rR6GdH9ECwjXrMge913MY5ro1twAx6Thv0oJHbWvS6pC
FBjrfDRxXz5X7kflRTKv5vXmhc9XTQCMLP1H9UnIi+kYrT8Ql0kMTsI29
DYnB/Vyv6RqeRJ4/KdEnmvKkCaz9xUi6izPVx+fsTwkJ9wEpVDJKdUPEu A==;
X-IronPort-AV: E=McAfee;i="6400,9594,10345"; a="269874224"
X-IronPort-AV: E=Sophos;i="5.91,221,1647327600"; d="scan'208";a="269874224"
Received: from fmsmga007.fm.intel.com ([10.253.24.52])
by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
12 May 2022 20:32:34 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.91,221,1647327600"; d="scan'208";a="572801138"
Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83])
by fmsmga007.fm.intel.com with ESMTP; 12 May 2022 20:32:34 -0700
Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by
fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.1.2308.27; Thu, 12 May 2022 20:32:33 -0700
Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by
fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.1.2308.27; Thu, 12 May 2022 20:32:33 -0700
Received: from fmsmsx612.amr.corp.intel.com ([10.18.126.92]) by
fmsmsx612.amr.corp.intel.com ([10.18.126.92]) with mapi id 15.01.2308.027;
Thu, 12 May 2022 20:32:33 -0700
From: "Pei, Andy" <andy.pei@intel.com>
To: "Xia, Chenbo" <chenbo.xia@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
CC: "maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>, "Cao, Gang"
<gang.cao@intel.com>, "Liu, Changpeng" <changpeng.liu@intel.com>
Subject: RE: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migration
Thread-Topic: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migration
Thread-Index: AQHYWheDsJei7826WUGDCpro8PYkba0bSdbQgAD13CA=
Date: Fri, 13 May 2022 03:32:33 +0000
Message-ID: <235853b3989949ccab972d28d217e7c7@intel.com>
References: <1643093258-47258-2-git-send-email-andy.pei@intel.com>
<1651048206-282372-1-git-send-email-andy.pei@intel.com>
<1651048206-282372-7-git-send-email-andy.pei@intel.com>
<SN6PR11MB3504769DC6182A50DC2FFB139CCB9@SN6PR11MB3504.namprd11.prod.outlook.com>
In-Reply-To: <SN6PR11MB3504769DC6182A50DC2FFB139CCB9@SN6PR11MB3504.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
dlp-product: dlpe-windows
dlp-version: 11.6.401.20
dlp-reaction: no-action
x-originating-ip: [10.239.127.36]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Hi Chenbo,
Thanks for your reply.
Your suggestion is good. I will fix in next version.
> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Thursday, May 12, 2022 8:55 PM
> To: Pei, Andy <andy.pei@intel.com>; dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; Cao, Gang <gang.cao@intel.com>; Liu,
> Changpeng <changpeng.liu@intel.com>
> Subject: RE: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migratio=
n
>=20
> > -----Original Message-----
> > From: Pei, Andy <andy.pei@intel.com>
> > Sent: Wednesday, April 27, 2022 4:30 PM
> > To: dev@dpdk.org
> > Cc: Xia, Chenbo <chenbo.xia@intel.com>; maxime.coquelin@redhat.com;
> > Cao, Gang <gang.cao@intel.com>; Liu, Changpeng
> > <changpeng.liu@intel.com>
> > Subject: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migration
> >
> > Add SW live-migration support to block device.
> >
> > Signed-off-by: Andy Pei <andy.pei@intel.com>
> > ---
> > drivers/vdpa/ifc/ifcvf_vdpa.c | 33 +++++++++++++++++++++++++++++----
> > 1 file changed, 29 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c
> > b/drivers/vdpa/ifc/ifcvf_vdpa.c index 07fc3ca..8a260b7 100644
> > --- a/drivers/vdpa/ifc/ifcvf_vdpa.c
> > +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
> > @@ -312,6 +312,7 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(struct
> > ifcvf_internal *internal) {
> > struct ifcvf_hw *hw =3D &internal->hw;
> > + struct rte_vhost_vring vq;
> > uint32_t i;
> > int vid;
> > uint64_t features =3D 0;
> > @@ -319,6 +320,22 @@ struct rte_vdpa_dev_info {
> > uint64_t len;
> >
> > vid =3D internal->vid;
> > +
> > + /* to make sure no packet is lost for blk device
> > + * do not stop until last_avail_idx =3D=3D last_used_idx
> > + */
> > + if (internal->device_type =3D=3D IFCVF_BLK) {
> > + for (i =3D 0; i < hw->nr_vring; i++) {
> > + rte_vhost_get_vhost_vring(internal->vid, i, &vq);
> > + while (vq.avail->idx !=3D vq.used->idx) {
> > + ifcvf_notify_queue(hw, i);
> > + usleep(10);
> > + }
> > + hw->vring[i].last_avail_idx =3D vq.avail->idx;
> > + hw->vring[i].last_used_idx =3D vq.used->idx;
> > + }
> > + }
> > +
>=20
> This seems not match with the above comment about avoiding in-flight
> packets.
> But the change in patch 17 seems good. Why not just using the
> implementation in patch 17?
>=20
> Thanks,
> Chenbo
>=20
> > ifcvf_stop_hw(hw);
> >
> > for (i =3D 0; i < hw->nr_vring; i++)
> > @@ -642,8 +659,10 @@ struct rte_vdpa_dev_info {
> > }
> > hw->vring[i].avail =3D gpa;
> >
> > - /* Direct I/O for Tx queue, relay for Rx queue */
> > - if (i & 1) {
> > + /* NET: Direct I/O for Tx queue, relay for Rx queue
> > + * BLK: relay every queue
> > + */
> > + if ((internal->device_type =3D=3D IFCVF_NET) && (i & 1)) {
> > gpa =3D hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);
> > if (gpa =3D=3D 0) {
> > DRV_LOG(ERR, "Fail to get GPA for used
> ring."); @@ -693,8 +712,12
> > @@ struct rte_vdpa_dev_info {
> >
> > for (i =3D 0; i < hw->nr_vring; i++) {
> > /* synchronize remaining new used entries if any */
> > - if ((i & 1) =3D=3D 0)
> > + if (internal->device_type =3D=3D IFCVF_NET) {
> > + if ((i & 1) =3D=3D 0)
> > + update_used_ring(internal, i);
> > + } else if (internal->device_type =3D=3D IFCVF_BLK) {
> > update_used_ring(internal, i);
> > + }
> >
> > rte_vhost_get_vhost_vring(vid, i, &vq);
> > len =3D IFCVF_USED_RING_LEN(vq.size); @@ -756,7 +779,9
> @@ struct
> > rte_vdpa_dev_info {
> > }
> > }
> >
> > - for (qid =3D 0; qid < q_num; qid +=3D 2) {
> > + for (qid =3D 0; qid < q_num; qid +=3D 1) {
> > + if ((internal->device_type =3D=3D IFCVF_NET) && (qid & 1))
> > + continue;
> > ev.events =3D EPOLLIN | EPOLLPRI;
> > /* leave a flag to mark it's for interrupt */
> > ev.data.u64 =3D 1 | qid << 1 |
> > --
> > 1.8.3.1
>=20