From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 05C81A0562; Thu, 2 Apr 2020 05:04:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 676021BEAC; Thu, 2 Apr 2020 05:04:49 +0200 (CEST) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2071.outbound.protection.outlook.com [40.107.20.71]) by dpdk.org (Postfix) with ESMTP id 2B6631BEA7 for ; Thu, 2 Apr 2020 05:04:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q17gxGKLufDfXzU88VQEDzgMw6ZYErjh3WLE7TPVvPY=; b=mBHfDVZ5OtgyHzoQ5+Ki58IFOBTh2EnPo7tdLo6H69lIhC44urdBJK6Yt42dSXyqP/dzK2lC9RGNZ1gAGMehV/sVc/Gne2KStXdxhnGT74nnAW5lUwZdOSqZXqg1wqgvUx26bGapj1YkugavtNVOZD2UOnOacpeljOIAFkl7UD0= Received: from AM0PR01CA0038.eurprd01.prod.exchangelabs.com (2603:10a6:208:e6::15) by VE1SPR01MB0008.eurprd08.prod.outlook.com (2603:10a6:802:90::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.16; Thu, 2 Apr 2020 03:04:47 +0000 Received: from AM5EUR03FT054.eop-EUR03.prod.protection.outlook.com (2603:10a6:208:e6:cafe::12) by AM0PR01CA0038.outlook.office365.com (2603:10a6:208:e6::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.15 via Frontend Transport; Thu, 2 Apr 2020 03:04:47 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT054.mail.protection.outlook.com (10.152.16.212) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2856.17 via Frontend Transport; Thu, 2 Apr 2020 03:04:46 +0000 Received: ("Tessian outbound 1425309d4c0b:v50"); Thu, 02 Apr 2020 03:04:46 +0000 X-CR-MTA-TID: 64aa7808 Received: from 1695abce908e.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id FD34B048-6542-4E6C-8153-6DF3ABFB447B.1; Thu, 02 Apr 2020 03:04:41 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 1695abce908e.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 02 Apr 2020 03:04:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nc0wkxEaRwu5jyNR7xGcQHXhWNHsc37REFvIzs6Gi/yH3DfFN3EZ6jbWQLuT4XBx/h68QJSgGD5BP3L8mpwKwJG+QFr5C9rr5joixgPFPTKDxKjP9TDqZZrdL9e1TdA5WOl0mpfBydlZYKDNBd07dGI/edUXlaWKI8qNTJnnnLXpsxgAknuuUb6nYEAJTjurz63CI8cqL2R5BlFdjAqNrR2mqzOfdTXzKWbWKYwkZ6KOUQgSfJBiXuifnVBvTSlagq11k+O+xJCphPtcMOehxud6MompWAcshXj0tgLpR+WVYt4jIrSrP8WqN2lKAuz64VbvQIyPUqDbmde1FVT/cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q17gxGKLufDfXzU88VQEDzgMw6ZYErjh3WLE7TPVvPY=; b=AF3cEwvDu7qye2DyGqAptpVEGFNb1/gWUBCOBDY7bTc3YayjpycJItV5ylCRNbxCGROGXRcd6U+abWOBeGuYrIYd5M/ybJEWj1N94sJU/T0C8GVdcg+14XixtbWEU3HvC28IdEIne2nqJwasiURmZsnvdWXr7v5BiNFt0ysozy3CmKnKV8QBNkB5h8NhHYjz06VAgcN3IY7bAA/F+8iBMra5JrQLpq0dGvuD82edzWMPx24s1ZvfzSVPDle/oE6tEN8whqpi30xmYRAYXZg0NWI3rJUCKdygRGyId+0EI96zq6csAz12wyDIAj1RB3A9pTzhYfs4YfrG0IkRxUPXuw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q17gxGKLufDfXzU88VQEDzgMw6ZYErjh3WLE7TPVvPY=; b=mBHfDVZ5OtgyHzoQ5+Ki58IFOBTh2EnPo7tdLo6H69lIhC44urdBJK6Yt42dSXyqP/dzK2lC9RGNZ1gAGMehV/sVc/Gne2KStXdxhnGT74nnAW5lUwZdOSqZXqg1wqgvUx26bGapj1YkugavtNVOZD2UOnOacpeljOIAFkl7UD0= Received: from VI1PR08MB5376.eurprd08.prod.outlook.com (10.255.196.79) by VI1PR08MB4000.eurprd08.prod.outlook.com (20.178.125.94) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.16; Thu, 2 Apr 2020 03:04:39 +0000 Received: from VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::a0e2:2a9f:be7b:4b15]) by VI1PR08MB5376.eurprd08.prod.outlook.com ([fe80::a0e2:2a9f:be7b:4b15%3]) with mapi id 15.20.2856.019; Thu, 2 Apr 2020 03:04:39 +0000 From: Gavin Hu To: "Liu, Yong" , "maxime.coquelin@redhat.com" , "Ye, Xiaolong" , "Wang, Zhihong" CC: "dev@dpdk.org" , nd , nd Thread-Topic: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translation Thread-Index: AQHWB/UyN8J3fCX6WkKxI4Ky3yKHxKhkCX5QgAAx2ICAAOcNcA== Date: Thu, 2 Apr 2020 03:04:38 +0000 Message-ID: References: <20200316153353.112897-1-yong.liu@intel.com> <20200401145011.67357-1-yong.liu@intel.com> <20200401145011.67357-2-yong.liu@intel.com> <86228AFD5BCD8E4EBFD2B90117B5E81E63524453@SHSMSX103.ccr.corp.intel.com> In-Reply-To: <86228AFD5BCD8E4EBFD2B90117B5E81E63524453@SHSMSX103.ccr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: d8215ea2-7690-48d1-9bda-c2dee60998ac.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; x-originating-ip: [221.239.170.106] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 76822edb-ffab-4684-5eb1-08d7d6b299da x-ms-traffictypediagnostic: VI1PR08MB4000:|VE1SPR01MB0008: X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:6430;OLM:6430; x-forefront-prvs: 0361212EA8 X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR08MB5376.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(136003)(396003)(366004)(376002)(39860400002)(346002)(316002)(64756008)(66446008)(66556008)(66476007)(76116006)(66946007)(71200400001)(4326008)(55016002)(6506007)(7696005)(53546011)(8936002)(54906003)(110136005)(81156014)(9686003)(26005)(8676002)(81166006)(5660300002)(186003)(86362001)(33656002)(478600001)(2906002)(52536014); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: oUnn0TUjvoXoOQ0hPln2rSqbBcx0+4mL9/UkaZC0250WAyQsjSCEVcyq+Jb4rNueY8h+WfiYQJKsu1yvj67IwrXZiijz9xSW5Lt/T3tdv1JWK1Aa735bYrmWxLeu2WaxBCqLb8KC+76NQ0fTjM7QYgTJCmzkRPie7TJhSlz8uVy0g5lbB2oyewa4RCJDruRRFJyXCzmjWT/RDakcFbFEFJAyBP/6dGzChBNyrzKaCtC2bX5Tfg/z7TTRucUvXDTCKRtfIX+hPvy8D/jEYIC9u1q0JEylKZUN5OaBAL1ni4gQ+76i0ww/hRFVQ8C8jcBpZ8nUqjEagy4+Qv7WvTlBROhjB741bGtgftevFZz8N15RKIwN31Ei5lJId2JuFCPE4BnTJdOPSfNi9zcOQxNG7CpLY0sybY9grbx5njoG3X0KvjZDSinFzmqnsuTo2aHT x-ms-exchange-antispam-messagedata: 4wRdd+jfsCe63C0Rw3l7m4jd+u9zqCbPQjZq4R+DOuqrggVFKHvEdRam6qnbDoUKmVzunDWMrXPxBFr7R+JDieEEg/Bwg7ZHL8sicDeREQDaT9MJ8pIDMssKk5awtI02vFZZHYGqr2NJs+bktzmncQ== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB4000 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT054.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(376002)(396003)(136003)(346002)(39860400002)(46966005)(110136005)(33656002)(82740400003)(54906003)(2906002)(356004)(5660300002)(4326008)(26005)(316002)(36906005)(478600001)(52536014)(6506007)(70586007)(9686003)(336012)(186003)(47076004)(53546011)(8936002)(70206006)(86362001)(26826003)(7696005)(81166006)(55016002)(8676002)(81156014); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: 854af89e-bcd2-4f45-75fd-08d7d6b29517 X-Forefront-PRVS: 0361212EA8 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3pjdpflMvW35ikqezWdhwOW6k7eiYO6FDtKUfsLkR8+2vdWaSrNSHFQ2iFHeNCVl4yMKj8McDeO89stiYG1IY8sp20ctl+su2ai0pr//uTsQNc8z9zG2MPxAz7hSWoUVFfUlnd+gq101cTlOkLwI767yki6hr7lRQJwyFR3fX+ITqC2CsZgMzdxpP56KfZx1taUNGPDPb96FHxYsRgY0AUana2NIxStQ+K/Y2ipA9q8/2KfV2Ga5RC9+0pXd7VE+jP57Io5cum19x6AZ2igmznDe9OT19RFFv9fy3e+2z5v7WTfSdSI2Xk80NJPshwNho2tQWLqWBWxidF4F39mduvlRewbP7XCuGl5r7ZIjHDjh3Weq/vnVU3CtNMsYDFEOUO23GFGn6Opew8hml++xMDw6lXlhkMWksC08S/KgZFE4RBXQwyvLdSNgC+X5yzGQZyo49jrM/S3qCyH9INt/vefnM70sIEXvPLDUtzTKvyoAcNPiyXDBl0AgUtxk9q+fsk7HNQ4p+Hat+PSAakuwcw== X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Apr 2020 03:04:46.9321 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 76822edb-ffab-4684-5eb1-08d7d6b299da X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1SPR01MB0008 Subject: Re: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Marvin, > -----Original Message----- > From: Liu, Yong > Sent: Wednesday, April 1, 2020 9:01 PM > To: Gavin Hu ; maxime.coquelin@redhat.com; Ye, > Xiaolong ; Wang, Zhihong > > Cc: dev@dpdk.org; nd > Subject: RE: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translatio= n >=20 >=20 >=20 > > -----Original Message----- > > From: Gavin Hu > > Sent: Wednesday, April 1, 2020 6:07 PM > > To: Liu, Yong ; maxime.coquelin@redhat.com; Ye, > > Xiaolong ; Wang, Zhihong > > > > Cc: dev@dpdk.org; nd > > Subject: RE: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translat= ion > > > > Hi Marvin, > > > > > -----Original Message----- > > > From: dev On Behalf Of Marvin Liu > > > Sent: Wednesday, April 1, 2020 10:50 PM > > > To: maxime.coquelin@redhat.com; xiaolong.ye@intel.com; > > > zhihong.wang@intel.com > > > Cc: dev@dpdk.org; Marvin Liu > > > Subject: [dpdk-dev] [PATCH v2 2/2] vhost: cache gpa to hpa translatio= n > > > > > > If Tx zero copy enabled, gpa to hpa mapping table is updated one by > > > one. This will harm performance when guest memory backend using 2M > > > hugepages. Now add cached mapping table which will sorted by using > > > sequence. Address translation will first check cached mapping table, > > > then check unsorted mapping table if no match found. > > > > > > Signed-off-by: Marvin Liu > > > > > > diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h > > > index 2087d1400..5cb0e83dd 100644 > > > --- a/lib/librte_vhost/vhost.h > > > +++ b/lib/librte_vhost/vhost.h > > > @@ -368,7 +368,9 @@ struct virtio_net { > > > struct vhost_device_ops const *notify_ops; > > > > > > uint32_t nr_guest_pages; > > > + uint32_t nr_cached_guest_pages; > > > uint32_t max_guest_pages; > > > + struct guest_page *cached_guest_pages; > > > struct guest_page *guest_pages; > > > > > > int slave_req_fd; > > > @@ -553,12 +555,25 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t > gpa, > > > uint64_t size) > > > { > > > uint32_t i; > > > struct guest_page *page; > > > + uint32_t cached_pages =3D dev->nr_cached_guest_pages; > > > + Add a comment here, something like "Firstly look up in the cached pages"?=20 > > > + for (i =3D 0; i < cached_pages; i++) { Should the searching order reversed here to search the most recent entries= ?=20 > > > + page =3D &dev->cached_guest_pages[i]; > > > + if (gpa >=3D page->guest_phys_addr && > > > + gpa + size < page->guest_phys_addr + page->size) { > > > + return gpa - page->guest_phys_addr + > > > + page->host_phys_addr; > > > + } > > > + } > > Sorry, I did not see any speedup with cached guest pages in comparison = to > > the old code below. > > Is it not a simple copy? > > Is it a better idea to use hash instead to speed up the translation? > > /Gavin >=20 > Hi Gavin, > Here just resort the overall mapping table according to using sequence. > Most likely virtio driver will reuse recently cycled buffers, thus search= will > find match in beginning. > That is simple fix for performance enhancement. If use hash for index, it= will > take much more cost in normal case. >=20 > Regards, > Marvin There are issues here, the cached table is growing over time, will it becom= e less efficient when growing too big and even bigger than the original tab= le and overflow happen?=20 Is it a good idea to limit the cached entries to small therefore make the s= earch quicker?=20 /Gavin >=20 >=20 > > > Add a comment here, something like "Fall back to normal lookup if failed to= get from the cached table"?=20 > > > for (i =3D 0; i < dev->nr_guest_pages; i++) { > > > page =3D &dev->guest_pages[i]; > > > > > > if (gpa >=3D page->guest_phys_addr && > > > gpa + size < page->guest_phys_addr + page->size) { > > > + rte_memcpy(&dev- > > > >cached_guest_pages[cached_pages], > > > + page, sizeof(struct guest_page)); > > > + dev->nr_cached_guest_pages++; Will overflow happen over time when there are many translations?=20 > > > return gpa - page->guest_phys_addr + > > > page->host_phys_addr; > > > } > > > diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_u= ser.c > > > index 79fcb9d19..1bae1fddc 100644 > > > --- a/lib/librte_vhost/vhost_user.c > > > +++ b/lib/librte_vhost/vhost_user.c > > > @@ -192,7 +192,9 @@ vhost_backend_cleanup(struct virtio_net *dev) > > > } > > > > > > rte_free(dev->guest_pages); > > > + rte_free(dev->cached_guest_pages); > > > dev->guest_pages =3D NULL; > > > + dev->cached_guest_pages =3D NULL; > > > > > > if (dev->log_addr) { > > > munmap((void *)(uintptr_t)dev->log_addr, dev->log_size); > > > @@ -898,7 +900,7 @@ add_one_guest_page(struct virtio_net *dev, > > > uint64_t guest_phys_addr, > > > uint64_t host_phys_addr, uint64_t size) > > > { > > > struct guest_page *page, *last_page; > > > - struct guest_page *old_pages; > > > + struct guest_page *old_pages, *old_cached_pages; > > > > > > if (dev->nr_guest_pages =3D=3D dev->max_guest_pages) { > > > dev->max_guest_pages *=3D 2; > > > @@ -906,9 +908,19 @@ add_one_guest_page(struct virtio_net *dev, > > > uint64_t guest_phys_addr, > > > dev->guest_pages =3D rte_realloc(dev->guest_pages, > > > dev->max_guest_pages * > > > sizeof(*page), > > > RTE_CACHE_LINE_SIZE); > > > - if (dev->guest_pages =3D=3D NULL) { > > > + old_cached_pages =3D dev->cached_guest_pages; > > > + dev->cached_guest_pages =3D rte_realloc(dev- > > > >cached_guest_pages, > > > + dev->max_guest_pages * > > > + sizeof(*page), > > > + RTE_CACHE_LINE_SIZE); > > > + dev->nr_cached_guest_pages =3D 0; > > > + if (dev->guest_pages =3D=3D NULL || > > > + dev->cached_guest_pages =3D=3D NULL) { > > > VHOST_LOG_CONFIG(ERR, "cannot realloc > > > guest_pages\n"); > > > rte_free(old_pages); > > > + rte_free(old_cached_pages); > > > + dev->guest_pages =3D NULL; > > > + dev->cached_guest_pages =3D NULL; > > > return -1; > > > } > > > } > > > @@ -1078,6 +1090,20 @@ vhost_user_set_mem_table(struct virtio_net > > > **pdev, struct VhostUserMsg *msg, > > > } > > > } > > > > > > + if (dev->cached_guest_pages =3D=3D NULL) { > > > + dev->cached_guest_pages =3D rte_zmalloc(NULL, > > > + dev->max_guest_pages * > > > + sizeof(struct guest_page), > > > + RTE_CACHE_LINE_SIZE); > > > + if (dev->cached_guest_pages =3D=3D NULL) { > > > + VHOST_LOG_CONFIG(ERR, > > > + "(%d) failed to allocate memory " > > > + "for dev->cached_guest_pages\n", > > > + dev->vid); > > > + return RTE_VHOST_MSG_RESULT_ERR; > > > + } > > > + } > > > + > > > dev->mem =3D rte_zmalloc("vhost-mem-table", sizeof(struct > > > rte_vhost_memory) + > > > sizeof(struct rte_vhost_mem_region) * memory->nregions, > > > 0); > > > if (dev->mem =3D=3D NULL) { > > > -- > > > 2.17.1