From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 72679A04A5; Thu, 18 Jun 2020 13:36:52 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8AF4F1BEDF; Thu, 18 Jun 2020 13:36:51 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 194001B6B4 for ; Thu, 18 Jun 2020 13:36:48 +0200 (CEST) IronPort-SDR: iFqp3qluMW7L6UC/HG8n6N8eMSYbAGcnrxjXSsy819Tw95AT31+Lggi8HyoRCXYqwYsD42cjnp CxLT3v1jISww== X-IronPort-AV: E=McAfee;i="6000,8403,9655"; a="130946499" X-IronPort-AV: E=Sophos;i="5.73,526,1583222400"; d="scan'208";a="130946499" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Jun 2020 04:36:48 -0700 IronPort-SDR: nWc5XTE3JtEVM2zNx33a7+xD8Wfnc1QIXv8N7lDsj7a/p/2CsBOJbD/NkMBuETFLtm2YGBU3Vw tFQZSUVY/bpA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,526,1583222400"; d="scan'208";a="352387589" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga001.jf.intel.com with ESMTP; 18 Jun 2020 04:36:47 -0700 Received: from fmsmsx607.amr.corp.intel.com (10.18.126.87) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 18 Jun 2020 04:36:47 -0700 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx607.amr.corp.intel.com (10.18.126.87) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 18 Jun 2020 04:36:47 -0700 Received: from FMSEDG001.ED.cps.intel.com (10.1.192.133) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 18 Jun 2020 04:36:47 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.108) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 18 Jun 2020 04:36:47 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JL/ss2x7SCd7xnAf0PT51IN+JD7tjFfk7FLQVnphcb557ECmrULWbH/n282hQkB8s1//M7+ZhaYl+NCqfaBSftARxa2e/A34TQ62OTjxgrcrCIr9gZ+Od9eL0u1CRouShh2T7rzT9bDwGmvpEYEievTvQ+uZ0aPW/HFmcXYDggCPY4KYthl8P+Rjm6rustz/DS4P9pdLgnRMH5kNu+pfc/HXBU1ZX/QKbYKfWQgF9F2YnZrA9zGacwl5hvNejfUY+IIjP7WbWsOZ2Dl4IOaCjWwKVpYwrrHvK03ZBKcaXoQOsFIzM2Ti7tQRvL+LSAXvzpQhKYkySo940e5Rjr+iXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iiyA39vwxsBQROVtwGXVi3CO+o4s1HoNhOLwEvXieRc=; b=WZ/RGfp3Gv1tgFnPak9snP6sjvn5q+HyOaidDKSakCWGLfRqQtvXTsJwF6TbG/EbIX4a9wd+x6JjB3HsKkM5J/xMyGPQZAYHPniRChGa2pCNfGeCk6DU/xGiKwp7gg3ZsSaCJJRnAzvJQk1+Lqw+ZCP41FiShhdp1KrH5OlDOFdtd889+xkTIeh7AEKNvupshPcPHLszfEa1umbucErCHc7jR7ci9mxXme20c9kA3eZArREm3ABCdBUeYpGEraoMIrj3KiVFPkStyJrhwgrY1DtPgMWDK0y6EKLYQc4heC6ol1v/TTtBowyA4CnnYsIThj0BequXgB1UUVZlAF/Fjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iiyA39vwxsBQROVtwGXVi3CO+o4s1HoNhOLwEvXieRc=; b=Zm+3CY6nBYcgeyj56OgMcyBiA5q+1eyQx8AHrLzNUtQeRtyfiA6Vb3TTlhmB687MlPubve7bxv2/WUuSv1RRpF7SXbn6EEtrPJfKWV86ie713jGhbJCdZ971dkE3kku1E001Xm3bzO5v5IqAsHVbxZQenAi8j0BSpXdkGgeVFm0= Received: from BYAPR11MB3735.namprd11.prod.outlook.com (2603:10b6:a03:b4::31) by BY5PR11MB3944.namprd11.prod.outlook.com (2603:10b6:a03:18e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.18; Thu, 18 Jun 2020 11:36:43 +0000 Received: from BYAPR11MB3735.namprd11.prod.outlook.com ([fe80::4d57:a18:a46e:4a27]) by BYAPR11MB3735.namprd11.prod.outlook.com ([fe80::4d57:a18:a46e:4a27%7]) with mapi id 15.20.3109.021; Thu, 18 Jun 2020 11:36:43 +0000 From: "Fu, Patrick" To: "Liu, Yong" CC: "Jiang, Cheng1" , "Liang, Cunming" , "dev@dpdk.org" , "maxime.coquelin@redhat.com" , "Xia, Chenbo" , "Wang, Zhihong" , "Ye, Xiaolong" Thread-Topic: [dpdk-dev] [PATCH v1 2/2] vhost: introduce async enqueue for split ring Thread-Index: AQHWRT2rhfkGvKUh602OYd01E0ZbrKjeFbfA Date: Thu, 18 Jun 2020 11:36:43 +0000 Message-ID: References: <1591869725-13331-1-git-send-email-patrick.fu@intel.com> <1591869725-13331-3-git-send-email-patrick.fu@intel.com> <86228AFD5BCD8E4EBFD2B90117B5E81E635F613A@SHSMSX103.ccr.corp.intel.com> In-Reply-To: <86228AFD5BCD8E4EBFD2B90117B5E81E635F613A@SHSMSX103.ccr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.2.0.6 dlp-product: dlpe-windows dlp-reaction: no-action authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.198.147.197] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8b89d519-39e0-44a4-5d09-08d8137bdffd x-ms-traffictypediagnostic: BY5PR11MB3944: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0438F90F17 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: HynLQoFxMjgx5xFN4HhTughqpNq8Hpd2beQQ6nyEPUF1JN+7pcuewuYP5LrdG4KMTPf+GD4oHQMkCWCUbFdhrN8j+ZlfcNOjxiWxPzpH/Qf578bUip6wjEYeWZnc2n+oJGzgWSUCyeheC1eTyTbGHW4Ro8w+/yykaDkk07FUDZhMyhdJCkKp4idDBc/FZapLypGkLfTsut/LKvAK90pTnQ3N+/57tO23kPxMyOINg8tCnPcNYPKzIXsP7EhWsWzk6NQyh9zOTd17CAEAqIF8g5gr0lVvn8BKupdMxRPhj7KzUksXc/MsSpQfia15dHihPmm32fi57Op5YJnxGA8nTw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3735.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(396003)(136003)(346002)(39860400002)(376002)(366004)(86362001)(9686003)(5660300002)(52536014)(107886003)(55016002)(6862004)(83380400001)(4326008)(6636002)(33656002)(316002)(186003)(2906002)(53546011)(6506007)(7696005)(26005)(76116006)(66946007)(66476007)(66556008)(64756008)(71200400001)(54906003)(66446008)(478600001)(8936002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: drQqUon8q94PcZixei67xSZuCPV21Z5oA1gOZUIkoyiyh1QiBITDQuO5qTHTzn07bDXnRsbffNVSGAaxo4WblEp36gsi60mfP75PNKfsxv8jCrIQ60emQCGsUAhaalOnOT/yihVwgKUu56paDqOmsKUlpp6M2HvxUs1f7Fx9QtRGC+xMGMyCryOyoES+xjo9zTRIBek2DxF6e9fmn5Xwlrv+I+XxQ8aQ45Q+wU9fzZVe/AEeYbietoqOOXrJYmMYRgqL9xbQIKap/tYNc2e5J7nQDiEU5vsbTLiSQEPa+H5A1zN8LP+txrIM2uJRtflMtG3/csaPes6RHJGGqWTkCUNHnVWY5znZGDAqmg4iqjEIxwxCpm6dFVv0MAXtP42JTYIadO1vSHuaqsrnblvoKQIRe0DpJMrhGvJn1UTAES7slnSWgW/8lgMI6bSlwSnW9XDy2ejxz1FSaJyvFVCDswhTO+QOLZm/VNh4G/v0dcH13XPjWEQV9x8CUSkLbKYS Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 8b89d519-39e0-44a4-5d09-08d8137bdffd X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jun 2020 11:36:43.1154 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: zvwiAp1VXNtLHqJQAzrVAvOwE6O3XwELHMwW6vYTBAhsjOEIeNFOShEigOcmMsk6Mjj2p8EwOB48RVxbS+PiVw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB3944 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v1 2/2] vhost: introduce async enqueue for split ring X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, > -----Original Message----- > From: Liu, Yong > Sent: Thursday, June 18, 2020 2:57 PM > To: Fu, Patrick > Cc: Fu, Patrick ; Jiang, Cheng1 > ; Liang, Cunming ; > dev@dpdk.org; maxime.coquelin@redhat.com; Xia, Chenbo > ; Wang, Zhihong ; Ye, > Xiaolong > Subject: RE: [dpdk-dev] [PATCH v1 2/2] vhost: introduce async enqueue for > split ring >=20 > Thanks, Patrick. Some comments are inline. >=20 > > > > From: Patrick > > > > This patch implement async enqueue data path for split ring. > > > > Signed-off-by: Patrick > > --- > > lib/librte_vhost/rte_vhost_async.h | 38 +++ > > lib/librte_vhost/virtio_net.c | 538 > > ++++++++++++++++++++++++++++++++++++- > > 2 files changed, 574 insertions(+), 2 deletions(-) > > > > diff --git a/lib/librte_vhost/rte_vhost_async.h > > b/lib/librte_vhost/rte_vhost_async.h > > index 82f2ebe..efcba0a 100644 > > --- a/lib/librte_vhost/rte_vhost_async.h > > +++ b/lib/librte_vhost/rte_vhost_async.h > > + > > +static __rte_always_inline int > > +async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, > > + struct rte_mbuf *m, struct buf_vector *buf_vec, > > + uint16_t nr_vec, uint16_t num_buffers, > > + struct iovec *src_iovec, struct iovec *dst_iovec, > > + struct iov_it *src_it, struct iov_it *dst_it) { >=20 > There're too much arguments in this function, please check whether it wil= l > impact performance. >=20 I guess src_iovec & dst_iovec could be removed from the parameter list.=20 > > + uint32_t vec_idx =3D 0; > > + uint32_t mbuf_offset, mbuf_avail; > > + uint32_t buf_offset, buf_avail; > > + uint64_t buf_addr, buf_iova, buf_len; > > + uint32_t cpy_len, cpy_threshold; > > + uint64_t hdr_addr; > > + struct rte_mbuf *hdr_mbuf; > > + struct batch_copy_elem *batch_copy =3D vq->batch_copy_elems; > > + struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =3D NULL; > > + int error =3D 0; > > + > > + uint32_t tlen =3D 0; > > + int tvec_idx =3D 0; > > + void *hpa; > > + > > + if (unlikely(m =3D=3D NULL)) { > > + error =3D -1; > > + goto out; > > + } > > + > > + cpy_threshold =3D vq->async_threshold; > > + > > + buf_addr =3D buf_vec[vec_idx].buf_addr; > > + buf_iova =3D buf_vec[vec_idx].buf_iova; > > + buf_len =3D buf_vec[vec_idx].buf_len; > > + > > + if (unlikely(buf_len < dev->vhost_hlen && nr_vec <=3D 1)) { > > + error =3D -1; > > + goto out; > > + } > > + > > + hdr_mbuf =3D m; > > + hdr_addr =3D buf_addr; > > + if (unlikely(buf_len < dev->vhost_hlen)) > > + hdr =3D &tmp_hdr; > > + else > > + hdr =3D (struct virtio_net_hdr_mrg_rxbuf > > *)(uintptr_t)hdr_addr; > > + > > + VHOST_LOG_DATA(DEBUG, "(%d) RX: num merge buffers %d\n", > > + dev->vid, num_buffers); > > + > > + if (unlikely(buf_len < dev->vhost_hlen)) { > > + buf_offset =3D dev->vhost_hlen - buf_len; > > + vec_idx++; > > + buf_addr =3D buf_vec[vec_idx].buf_addr; > > + buf_iova =3D buf_vec[vec_idx].buf_iova; > > + buf_len =3D buf_vec[vec_idx].buf_len; > > + buf_avail =3D buf_len - buf_offset; > > + } else { > > + buf_offset =3D dev->vhost_hlen; > > + buf_avail =3D buf_len - dev->vhost_hlen; > > + } > > + > > + mbuf_avail =3D rte_pktmbuf_data_len(m); > > + mbuf_offset =3D 0; > > + > > + while (mbuf_avail !=3D 0 || m->next !=3D NULL) { > > + /* done with current buf, get the next one */ > > + if (buf_avail =3D=3D 0) { > > + vec_idx++; > > + if (unlikely(vec_idx >=3D nr_vec)) { > > + error =3D -1; > > + goto out; > > + } > > + > > + buf_addr =3D buf_vec[vec_idx].buf_addr; > > + buf_iova =3D buf_vec[vec_idx].buf_iova; > > + buf_len =3D buf_vec[vec_idx].buf_len; > > + > > + buf_offset =3D 0; > > + buf_avail =3D buf_len; > > + } > > + > > + /* done with current mbuf, get the next one */ > > + if (mbuf_avail =3D=3D 0) { > > + m =3D m->next; > > + > > + mbuf_offset =3D 0; > > + mbuf_avail =3D rte_pktmbuf_data_len(m); > > + } > > + > > + if (hdr_addr) { > > + virtio_enqueue_offload(hdr_mbuf, &hdr->hdr); > > + if (rxvq_is_mergeable(dev)) > > + ASSIGN_UNLESS_EQUAL(hdr->num_buffers, > > + num_buffers); > > + > > + if (unlikely(hdr =3D=3D &tmp_hdr)) { > > + copy_vnet_hdr_to_desc(dev, vq, buf_vec, > > hdr); > > + } else { > > + PRINT_PACKET(dev, (uintptr_t)hdr_addr, > > + dev->vhost_hlen, 0); > > + vhost_log_cache_write_iova(dev, vq, > > + buf_vec[0].buf_iova, > > + dev->vhost_hlen); > > + } > > + > > + hdr_addr =3D 0; > > + } > > + > > + cpy_len =3D RTE_MIN(buf_avail, mbuf_avail); > > + > > + if (unlikely(cpy_len >=3D cpy_threshold)) { > > + hpa =3D (void *)(uintptr_t)gpa_to_hpa(dev, > > + buf_iova + buf_offset, cpy_len); >=20 > I have one question here. If user has called async copy directly, should = vhost > library still check copy threshold for software fallback? > If need software fallback, IMHO it will be more suitable to handle it in = copy > device driver. >=20 Technically, we can delegate the threshold judgement from vhost to applicat= ion callbacks.=20 This will significantly increase the complexity of the callback implementat= ions, which have to maintain correct ordering between dma copied data and CPU copies data. Meanwhile, it= actually doesn't help to=20 boost performance comparing with current vhost implementation. Considering = this threshold is a=20 generic design, I would still prefer to keep it in vhost. > IMHO, the cost will be too high for checking and fix virtio header in asy= nc > copy function. > Since this is async copy datapath, could it possible that eliminate the c= ost in > calculation of segmented addresses? >=20 Yes, I believe async data path certainly brings new opportunity to apply mo= re optimizations.=20 However, at current time frame, settling down the overall async framework m= ight be the priority.=20 Thanks, Patrick