From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4F95EA0562; Thu, 2 Apr 2020 06:45:50 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E577C2C15; Thu, 2 Apr 2020 06:45:49 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 1D60A2B8B for ; Thu, 2 Apr 2020 06:45:47 +0200 (CEST) IronPort-SDR: ZK3rGD8b7uEmyZ3QBHDTgz0jJ7aNUV3YyEB+Vre/sQNbEOzYzAsWMQC6OeVde82WKtVWBsTAv5 kvOGXMy5b77Q== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2020 21:45:47 -0700 IronPort-SDR: eQUa3QYJ3IADTctUIkf480Cqz6V2Kj1IY0N2SXq9VAfFcEYWlw1qGN/geI0XiYJ1BwexjscQQF zOv/uVLwqkUQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,334,1580803200"; d="scan'208";a="252873695" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2020 21:45:46 -0700 Received: from fmsmsx155.amr.corp.intel.com (10.18.116.71) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 1 Apr 2020 21:45:46 -0700 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by FMSMSX155.amr.corp.intel.com (10.18.116.71) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 1 Apr 2020 21:45:45 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.146]) by shsmsx102.ccr.corp.intel.com ([169.254.2.138]) with mapi id 14.03.0439.000; Thu, 2 Apr 2020 12:45:43 +0800 From: "Liu, Yong" To: Gavin Hu , "maxime.coquelin@redhat.com" , "Ye, Xiaolong" , "Wang, Zhihong" CC: "dev@dpdk.org" , nd , nd Thread-Topic: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translation Thread-Index: AQHWB/Ux3HhOESg2LUy+M7Zh4d4Ys6hjhJyAgAC0/oCAAGdDAIAAn+rA Date: Thu, 2 Apr 2020 04:45:42 +0000 Message-ID: <86228AFD5BCD8E4EBFD2B90117B5E81E635260EB@SHSMSX103.ccr.corp.intel.com> References: <20200316153353.112897-1-yong.liu@intel.com> <20200401145011.67357-1-yong.liu@intel.com> <20200401145011.67357-2-yong.liu@intel.com> <86228AFD5BCD8E4EBFD2B90117B5E81E63524453@SHSMSX103.ccr.corp.intel.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Gavin Hu > Sent: Thursday, April 2, 2020 11:05 AM > To: Liu, Yong ; maxime.coquelin@redhat.com; Ye, > Xiaolong ; Wang, Zhihong > > Cc: dev@dpdk.org; nd ; nd > Subject: RE: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translatio= n >=20 > Hi Marvin, >=20 > > -----Original Message----- > > From: Liu, Yong > > Sent: Wednesday, April 1, 2020 9:01 PM > > To: Gavin Hu ; maxime.coquelin@redhat.com; Ye, > > Xiaolong ; Wang, Zhihong > > > > Cc: dev@dpdk.org; nd > > Subject: RE: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translat= ion > > > > > > > > > -----Original Message----- > > > From: Gavin Hu > > > Sent: Wednesday, April 1, 2020 6:07 PM > > > To: Liu, Yong ; maxime.coquelin@redhat.com; Ye, > > > Xiaolong ; Wang, Zhihong > > > > > > Cc: dev@dpdk.org; nd > > > Subject: RE: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa > translation > > > > > > Hi Marvin, > > > > > > > -----Original Message----- > > > > From: dev On Behalf Of Marvin Liu > > > > Sent: Wednesday, April 1, 2020 10:50 PM > > > > To: maxime.coquelin@redhat.com; xiaolong.ye@intel.com; > > > > zhihong.wang@intel.com > > > > Cc: dev@dpdk.org; Marvin Liu > > > > Subject: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translat= ion > > > > > > > > If Tx zero copy enabled, gpa to hpa mapping table is updated one by > > > > one. This will harm performance when guest memory backend using > 2M > > > > hugepages. Now add cached mapping table which will sorted by using > > > > sequence. Address translation will first check cached mapping table= , > > > > then check unsorted mapping table if no match found. > > > > > > > > Signed-off-by: Marvin Liu > > > > > > > > diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h > > > > index 2087d1400..5cb0e83dd 100644 > > > > --- a/lib/librte_vhost/vhost.h > > > > +++ b/lib/librte_vhost/vhost.h > > > > @@ -368,7 +368,9 @@ struct virtio_net { > > > > struct vhost_device_ops const *notify_ops; > > > > > > > > uint32_t nr_guest_pages; > > > > + uint32_t nr_cached_guest_pages; > > > > uint32_t max_guest_pages; > > > > + struct guest_page *cached_guest_pages; > > > > struct guest_page *guest_pages; > > > > > > > > int slave_req_fd; > > > > @@ -553,12 +555,25 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t > > gpa, > > > > uint64_t size) > > > > { > > > > uint32_t i; > > > > struct guest_page *page; > > > > + uint32_t cached_pages =3D dev->nr_cached_guest_pages; > > > > + >=20 > Add a comment here, something like "Firstly look up in the cached pages"? >=20 > > > > + for (i =3D 0; i < cached_pages; i++) { >=20 > Should the searching order reversed here to search the most recent entri= es? >=20 > > > > + page =3D &dev->cached_guest_pages[i]; > > > > + if (gpa >=3D page->guest_phys_addr && > > > > + gpa + size < page->guest_phys_addr + page->size) { > > > > + return gpa - page->guest_phys_addr + > > > > + page->host_phys_addr; > > > > + } > > > > + } > > > Sorry, I did not see any speedup with cached guest pages in compariso= n > to > > > the old code below. > > > Is it not a simple copy? > > > Is it a better idea to use hash instead to speed up the translation? > > > /Gavin > > > > Hi Gavin, > > Here just resort the overall mapping table according to using sequence. > > Most likely virtio driver will reuse recently cycled buffers, thus sear= ch will > > find match in beginning. > > That is simple fix for performance enhancement. If use hash for index, = it > will > > take much more cost in normal case. > > > > Regards, > > Marvin >=20 > There are issues here, the cached table is growing over time, will it bec= ome > less efficient when growing too big and even bigger than the original tab= le > and overflow happen? > Is it a good idea to limit the cached entries to small therefore make the > search quicker? > /Gavin > > Gavin, Cached table size is the same as mapping table, it only recorded entries fr= om original table which have been used. At worst case like every access of guest memory is random, cached table wil= l be same size of original size. Search cost is same as before.=20 It will be hard to predict which size is more suitable for caching, that is= depend on how guest driver allocate buffer. Maybe less than ten when usin= g 2M page and thousands when using 4K page. So here just add resorted table, which cost is much less in normal case and= same as before at worst case.=20 Thanks, Marvin > > > > > > > Add a comment here, something like "Fall back to normal lookup if failed = to > get from the cached table"? >=20 > > > > for (i =3D 0; i < dev->nr_guest_pages; i++) { > > > > page =3D &dev->guest_pages[i]; > > > > > > > > if (gpa >=3D page->guest_phys_addr && > > > > gpa + size < page->guest_phys_addr + page->size) { > > > > + rte_memcpy(&dev- > > > > >cached_guest_pages[cached_pages], > > > > + page, sizeof(struct guest_page)); > > > > + dev->nr_cached_guest_pages++; >=20 > Will overflow happen over time when there are many translations? >=20 > > > > return gpa - page->guest_phys_addr + > > > > page->host_phys_addr; > > > > } > > > > diff --git a/lib/librte_vhost/vhost_user.c > b/lib/librte_vhost/vhost_user.c > > > > index 79fcb9d19..1bae1fddc 100644 > > > > --- a/lib/librte_vhost/vhost_user.c > > > > +++ b/lib/librte_vhost/vhost_user.c > > > > @@ -192,7 +192,9 @@ vhost_backend_cleanup(struct virtio_net *dev) > > > > } > > > > > > > > rte_free(dev->guest_pages); > > > > + rte_free(dev->cached_guest_pages); > > > > dev->guest_pages =3D NULL; > > > > + dev->cached_guest_pages =3D NULL; > > > > > > > > if (dev->log_addr) { > > > > munmap((void *)(uintptr_t)dev->log_addr, dev->log_size); > > > > @@ -898,7 +900,7 @@ add_one_guest_page(struct virtio_net *dev, > > > > uint64_t guest_phys_addr, > > > > uint64_t host_phys_addr, uint64_t size) > > > > { > > > > struct guest_page *page, *last_page; > > > > - struct guest_page *old_pages; > > > > + struct guest_page *old_pages, *old_cached_pages; > > > > > > > > if (dev->nr_guest_pages =3D=3D dev->max_guest_pages) { > > > > dev->max_guest_pages *=3D 2; > > > > @@ -906,9 +908,19 @@ add_one_guest_page(struct virtio_net *dev, > > > > uint64_t guest_phys_addr, > > > > dev->guest_pages =3D rte_realloc(dev->guest_pages, > > > > dev->max_guest_pages * > > > > sizeof(*page), > > > > RTE_CACHE_LINE_SIZE); > > > > - if (dev->guest_pages =3D=3D NULL) { > > > > + old_cached_pages =3D dev->cached_guest_pages; > > > > + dev->cached_guest_pages =3D rte_realloc(dev- > > > > >cached_guest_pages, > > > > + dev->max_guest_pages * > > > > + sizeof(*page), > > > > + RTE_CACHE_LINE_SIZE); > > > > + dev->nr_cached_guest_pages =3D 0; > > > > + if (dev->guest_pages =3D=3D NULL || > > > > + dev->cached_guest_pages =3D=3D NULL) { > > > > VHOST_LOG_CONFIG(ERR, "cannot realloc > > > > guest_pages\n"); > > > > rte_free(old_pages); > > > > + rte_free(old_cached_pages); > > > > + dev->guest_pages =3D NULL; > > > > + dev->cached_guest_pages =3D NULL; > > > > return -1; > > > > } > > > > } > > > > @@ -1078,6 +1090,20 @@ vhost_user_set_mem_table(struct > virtio_net > > > > **pdev, struct VhostUserMsg *msg, > > > > } > > > > } > > > > > > > > + if (dev->cached_guest_pages =3D=3D NULL) { > > > > + dev->cached_guest_pages =3D rte_zmalloc(NULL, > > > > + dev->max_guest_pages * > > > > + sizeof(struct guest_page), > > > > + RTE_CACHE_LINE_SIZE); > > > > + if (dev->cached_guest_pages =3D=3D NULL) { > > > > + VHOST_LOG_CONFIG(ERR, > > > > + "(%d) failed to allocate memory " > > > > + "for dev->cached_guest_pages\n", > > > > + dev->vid); > > > > + return RTE_VHOST_MSG_RESULT_ERR; > > > > + } > > > > + } > > > > + > > > > dev->mem =3D rte_zmalloc("vhost-mem-table", sizeof(struct > > > > rte_vhost_memory) + > > > > sizeof(struct rte_vhost_mem_region) * memory->nregions, > > > > 0); > > > > if (dev->mem =3D=3D NULL) { > > > > -- > > > > 2.17.1