From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sessmg23.ericsson.net (sessmg23.ericsson.net [193.180.251.45]) by dpdk.org (Postfix) with ESMTP id C9BCD5689 for ; Thu, 1 Jun 2017 09:55:35 +0200 (CEST) X-AuditID: c1b4fb2d-5a49e9a000000d37-6d-592fc877b0c2 Received: from ESESSHC002.ericsson.se (Unknown_Domain [153.88.183.24]) by sessmg23.ericsson.net (Symantec Mail Security) with SMTP id 9B.DD.03383.778CF295; Thu, 1 Jun 2017 09:55:35 +0200 (CEST) Received: from EUR02-VE1-obe.outbound.protection.outlook.com (153.88.183.145) by oa.msg.ericsson.com (153.88.183.24) with Microsoft SMTP Server (TLS) id 14.3.339.0; Thu, 1 Jun 2017 09:55:25 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.onmicrosoft.com; s=selector1-ericsson-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=YQlwNC6PvmybQFKb7vrm9tX6KDpz7A87NM5c4akItT8=; b=RaVoqsRc4up6v+KtGvF9D2ZQfH4/UTNfFjGhTGFucKIO1oyJhP0UL19+ofTI2VY4Jtl0mm3z3N4RQBlLhcMLJP2Z+iafHYmOWhwTTOqw01aMkcWkXAzK005qSXxUVjqF3+zT8YwN5f6+1XmNhtoDzJtjBoizKJnEdOOb18HfGmw= Received: from VI1PR07MB1357.eurprd07.prod.outlook.com (10.164.92.147) by VI1PR07MB1247.eurprd07.prod.outlook.com (10.164.92.13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1143.6; Thu, 1 Jun 2017 07:55:21 +0000 Received: from VI1PR07MB1357.eurprd07.prod.outlook.com ([fe80::6102:14bc:3cbb:b0a4]) by VI1PR07MB1357.eurprd07.prod.outlook.com ([fe80::6102:14bc:3cbb:b0a4%14]) with mapi id 15.01.1143.009; Thu, 1 Jun 2017 07:55:21 +0000 From: Imre Pinter To: "users@dpdk.org" CC: =?iso-8859-1?Q?Gabor_Hal=E1sz?= , =?iso-8859-1?Q?P=E9ter_Suskovics?= Thread-Topic: Slow DPDK startup with many 1G hugepages Thread-Index: AdLYf50vhjMfOET4SySM24szocOCwABljQHw Date: Thu, 1 Jun 2017 07:55:20 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=ericsson.com; x-originating-ip: [91.82.100.59] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR07MB1247; 7:cESJFvhJVronKJCyyqUh+EEWELnPZ/4zsk/gZf54pTxMTuozO45KZgO4u+RtXZW8o87V+pWc7fpcOpQDkBBILNrdnmHBZHTBgYJXRoi9YxmolzfKBFUQY2LDESReUEyCh/TT1IK9f2zi7IgAlP6Cyl6K3FAlBUZfeLFRRtVoNv8upNC0hntPchv45SJHemGs9rG0LmZw96+r8f6j6Ho6oDHfyGQVRURfmhQkobixR17zioiaQO0y+MjwcLNrUvhXbOpTaiV3xgEAiy9CThb7I98qQxm2luqKSnikUlVZi98GFxeYduUEdeoG81fa3OrpcujoJUVtIdzqXCmBje2RVw== x-forefront-antispam-report: SFV:SKI; SCL:-1SFV:NSPM; SFS:(10009020)(6009001)(39860400002)(39840400002)(39400400002)(39450400003)(39850400002)(39410400002)(38730400002)(33656002)(14454004)(54906002)(99286003)(54896002)(6436002)(81166006)(6506006)(2900100001)(110136004)(107886003)(53936002)(55016002)(5640700003)(8936002)(6306002)(8676002)(9326002)(9686003)(4326008)(5660300001)(45080400002)(86362001)(66066001)(25786009)(478600001)(189998001)(76176999)(54356999)(2906002)(50986999)(7696004)(3660700001)(2950100002)(6916009)(2501003)(3280700002)(74316002)(3846002)(5250100002)(7736002)(2351001)(790700001)(5630700001)(6116002)(102836003)(1730700003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR07MB1247; H:VI1PR07MB1357.eurprd07.prod.outlook.com; FPR:; SPF:None; MLV:ovrnspm; PTR:InfoNoRecords; LANG:en; x-ms-traffictypediagnostic: VI1PR07MB1247: x-ms-office365-filtering-correlation-id: 7b36af3a-bb73-45e3-cc48-08d4a8c38d67 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081); SRVR:VI1PR07MB1247; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(21748063052155); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3002001)(6041248)(20161123558100)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123560025)(20161123564025)(6072148); SRVR:VI1PR07MB1247; BCL:0; PCL:0; RULEID:; SRVR:VI1PR07MB1247; x-forefront-prvs: 0325F6C77B spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jun 2017 07:55:20.9624 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR07MB1247 X-OriginatorOrg: ericsson.com X-Brightmail-Tracker: H4sIAAAAAAAAA02SbUhTURjHObt38zqaXNfUB9OooWimM7XsvfSL9EIglKGh1MqLinPK7tIm WIpJuak5nS6d5RYZuLYKtbQyzSUsdVHZFwl7QUdkA6OyLCVp21ngt9/zf37nPM+BQxFCEzeU KpArGYVcKhPz+GRbZj/ElT2Pz9rS1ha8o3rkLpmCDiwZu7jp6AR/Ty4jKyhlFPH7TvHzTfM1 nBLDA3TuqdqAKtGACamRPwX0VrCMaTlqxKeE9CgC7bUXBC7sCPqNHTyPRdL1BDR15+CGjgOf e/7ycDGDoPtTM+GxeHQs/Kmu4XhYREfCQvsT0iMR9EUEdQtNXmktnQCjtdPu4ZRbSobXDXLs J8L88CXftAhw3rN4dQGdDYt9k14WurlP2+nnYX86Bz44rKSHER0Mi+MW71yCDoG3zk4OfhsN NwdfEpiDYG52hYt9DYImUwbON0CP8asf5nCY7NQgz85A1xFgm3hP4kYjD5Y0KswxMG7QcDEf gYXpG74B56H5i8OXF8KPn498Z3fBZa2diy/9zgFrf63vQBg469+gRhTTvmpxzMXgal/2soAO hLE2J4lzCUy16HiYN8Mtk4vAHAdXV2zk6tyI/MwoiGVYtigvMUnCKArOsGyxXCJnlD3I/XFG +pbjBtBtV6oN0RQSrxEEDsdnCbnSUlZVZENAEWKRIEDljgS5UlU5oyg+qTgrY1gbWkeR4hBB ytCrTCGdJ1UyhQxTwij+dzmUf2gl6kgy17Atgw3Rc9b7CRH8CsdsxVBEpD6td5Ndrz/edSVW Ov0rI7ZKVFiuy06bOzRvdihLxGWt1juP979rSZE1bMy7EHk9SpB+9KBaF/RRTDx81hWe7Krd K9q92DsjnTktX1+Vun1qp31CYtk2q6fM335HcQ4rUgOiw5IJ17FWg5hk86UJMYSClf4DE3YW RTQDAAA= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Slow DPDK startup with many 1G hugepages X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jun 2017 07:55:36 -0000 Hi, We experience slow startup time in DPDK-OVS, when backing memory with 1G hu= gepages instead of 2M hugepages. Currently we're mapping 2M hugepages as memory backend for DPDK OVS. In the= future we would like to allocate this memory from the 1G hugepage pool. Cu= rrently in our deployments we have significant amount of 1G hugepages alloc= ated (min. 54G) for VMs and only 2G memory on 2M hugepages. Typical setup for 2M hugepages: GRUB: hugepagesz=3D2M hugepages=3D1024 hugepagesz=3D1G hugepages=3D54 default_hug= epagesz=3D1G $ grep hugetlbfs /proc/mounts nodev /mnt/huge_ovs_2M hugetlbfs rw,relatime,pagesize=3D2M 0 0 nodev /mnt/huge_qemu_1G hugetlbfs rw,relatime,pagesize=3D1G 0 0 Typical setup for 1GB hugepages: GRUB: hugepagesz=3D1G hugepages=3D56 default_hugepagesz=3D1G $ grep hugetlbfs /proc/mounts nodev /mnt/huge_qemu_1G hugetlbfs rw,relatime,pagesize=3D1G 0 0 DPDK OVS startup times based on the ovs-vswitchd.log logs: * 2M (2G memory allocated) - startup time ~3 sec: 2017-05-03T08:13:50.177Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x1 --hu= ge-dir /mnt/huge_ovs_2M --socket-mem 1024,1024 2017-05-03T08:13:50.708Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: Datapat= h supports recirculation * 1G (56G memory allocated) - startup time ~13 sec: 2017-05-03T08:09:22.114Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x1 --hu= ge-dir /mnt/huge_qemu_1G --socket-mem 1024,1024 2017-05-03T08:09:32.706Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: Datapat= h supports recirculation I used DPDK 16.11 for OVS and testpmd and tested on Ubuntu 14.04 with kerne= l 3.13.0-117-generic and 4.4.0-78-generic. We had a discussion with Mark Gray (from Intel), and he come up with the fo= llowing items: =B7 The ~10 sec time difference is there with testpmd as well =B7 They believe it is a kernel overhead (mmap is slow, perhaps it = is zeroing pages). The following code from eal_memory.c does the above ment= ioned printout in EAL startup: 469 /* map the segment, and populate page tables, 470 * the kernel fills this segment with zeros */ 468 uint64_t start =3D rte_rdtsc(); 471 virtaddr =3D mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE, 472 MAP_SHARED | MAP_POPULATE, fd, 0); 473 if (virtaddr =3D=3D MAP_FAILED) { 474 RTE_LOG(DEBUG, EAL, "%s(): mmap failed: %s\n", __func__, 475 strerror(errno)); 476 close(fd); 477 return i; 478 } 479 480 if (orig) { 481 hugepg_tbl[i].orig_va =3D virtaddr; 482 printf("Original mapping of page %u took: %"PRIu64" ticks, %= "PRIu64" ms\n ", 483 i, rte_rdtsc() - start, 484 (rte_rdtsc() - start) * 1000 / 485 rte_get_timer_hz()); 486 } A solution could be to mount 1G hugepages to 2 separate directory: 2G for O= VS and the remaining for the VMs, but the NUMA location for these hugepages= is non-deterministic. Since mount cannot handle NUMA related parameters du= ring mounting hugetlbfs, and fstab forks the mounts during boot. Do you have a solution on how to use 1G hugepages for VMs and have reasonab= le DPDK EAL startup time? Thanks, Imre