From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sessmg23.ericsson.net (sessmg23.ericsson.net [193.180.251.45]) by dpdk.org (Postfix) with ESMTP id 45C9A2BB9 for ; Thu, 8 Jun 2017 16:30:53 +0200 (CEST) X-AuditID: c1b4fb2d-ef7ff7000000080d-79-59395f9c70dc Received: from ESESSHC018.ericsson.se (Unknown_Domain [153.88.183.72]) by sessmg23.ericsson.net (Symantec Mail Security) with SMTP id C5.39.02061.C9F59395; Thu, 8 Jun 2017 16:30:52 +0200 (CEST) Received: from EUR02-HE1-obe.outbound.protection.outlook.com (153.88.183.145) by oa.msg.ericsson.com (153.88.183.72) with Microsoft SMTP Server (TLS) id 14.3.339.0; Thu, 8 Jun 2017 16:30:56 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.onmicrosoft.com; s=selector1-ericsson-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=gtFdLY4H6s4C0QfltaCckfs3a6QImH8wd7TK36/704k=; b=eKp4j3Cu1VP1WhtUTaMDwIhw8gsMF08U8vykM5ORPcWfObytR+4ksCxo3UFm6nK/uPXsxqfPohGKaJehbSixWxkLZT2j27Em9JEOge7ig0bo1A40rGd+2QFVil0zyH3u5GUfXAiqEa+7DLgz1Hdzwo2pP7Ls6tdBW4WH8tvLhAo= Received: from VI1PR07MB1357.eurprd07.prod.outlook.com (10.164.92.147) by VI1PR07MB3517.eurprd07.prod.outlook.com (10.175.244.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.5; Thu, 8 Jun 2017 14:30:43 +0000 Received: from VI1PR07MB1357.eurprd07.prod.outlook.com ([fe80::4cb5:fcd7:e9ce:fee4]) by VI1PR07MB1357.eurprd07.prod.outlook.com ([fe80::4cb5:fcd7:e9ce:fee4%14]) with mapi id 15.01.1178.003; Thu, 8 Jun 2017 14:30:42 +0000 From: Imre Pinter To: Sergio Gonzalez Monroy , "users@dpdk.org" CC: =?iso-8859-1?Q?Gabor_Hal=E1sz?= , =?iso-8859-1?Q?P=E9ter_Suskovics?= Thread-Topic: [dpdk-users] Slow DPDK startup with many 1G hugepages Thread-Index: AdLYf50vhjMfOET4SySM24szocOCwABljQHwACgC0YABafIVMA== Date: Thu, 8 Jun 2017 14:30:42 +0000 Message-ID: References: <2addf963-8e23-f7fa-038a-da23a9dbcde2@intel.com> In-Reply-To: <2addf963-8e23-f7fa-038a-da23a9dbcde2@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=ericsson.com; x-originating-ip: [91.82.100.59] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR07MB3517; 7:20C+JJ3+1ZfzO6p4lU7kWpNrNm+947YyPKXudRlz5OT5y/Qw97JY/E0JmOBILKHaXdznv59xo6PbpSDMvhF8WN4TrCjGdIZ4jL28Tz0I62gHO/11wK9QdeK4NpBTZXg9Idj2HoDH9eGtXAYxq+zeYNinI3B6DpANEY4SWZxzdJFTANOlUqR64GGAQUNqTiMl0mMF8/bv4dnXnrPQkQjQ/EKsZfu3aGJ2kTQ8JHAZN+AMnWjiCCjuLnXlPN+hwmlB2RPkZAfFzLYc5PFdedmWo4ulZANpdJpU0U0GbckHSvtt3rH9nUiMSQ4eSry4plNyS3SAuL0mHWfDKSTqxV7OpA== x-forefront-antispam-report: SFV:SKI; SCL:-1SFV:NSPM; SFS:(10009020)(6009001)(39860400002)(39840400002)(39450400003)(39400400002)(39850400002)(39410400002)(13464003)(51914003)(24454002)(229853002)(81166006)(6306002)(53936002)(66066001)(2906002)(6436002)(3660700001)(3280700002)(2501003)(55016002)(107886003)(8676002)(5250100002)(53376002)(38730400002)(6506006)(33656002)(50986999)(54356999)(478600001)(2950100002)(76176999)(45080400002)(4326008)(86362001)(6246003)(14454004)(189998001)(2900100001)(5660300001)(53546009)(54906002)(25786009)(99286003)(9686003)(3846002)(102836003)(74316002)(7736002)(7696004)(305945005)(966005)(8936002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR07MB3517; H:VI1PR07MB1357.eurprd07.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; x-ms-office365-filtering-correlation-id: a92b814b-7d79-4026-cb32-08d4ae7af163 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081); SRVR:VI1PR07MB3517; x-ms-traffictypediagnostic: VI1PR07MB3517: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(37575265505322)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(3002001)(100000703101)(100105400095)(10201501046)(6041248)(20161123558100)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123555025)(20161123562025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:VI1PR07MB3517; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:VI1PR07MB3517; x-forefront-prvs: 0332AACBC3 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jun 2017 14:30:42.4081 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR07MB3517 X-OriginatorOrg: ericsson.com X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpjleLIzCtJLcpLzFFi42KZGbHdQ3dOvGWkwaoma4sVE44wWjQfXM/i wOTxa8FSVo/Fe14yBTBFcdmkpOZklqUW6dslcGUcv3mSveCuSsWSw9eYGxiXy3QxcnJICJhI TFtwkamLkYtDSOAIo8TKY0dZIZzjjBL3Jq1kBnFYBHqZJfbNngSVmcYk0TblNRuE84hRYunC iawgw9gEdCR+NrcygdgiAkkSrbN6wDqYBVoYJXq+TGIGSQgLOElcfr8VqIgDqMhZYuOzcoh6 J4nfFw+zgoRZBFQkPm4OBQnzCsRIXDrcxAK36+DcG2C7OAVsJS7dPwlmMwqISXw/tQZsL7OA uMStJ/OZIJ4TkFiy5zwzhC0q8fLxP7B7GAV6GSXW3N3LApFQkNi04D07hC0rcWl+NyNIkYRA D7PEj80/WSCc16wS7TMPs4GcJyHgKzFnSjlEQ53E3HPzoQZpSSxvWc8IYWdLzPz3EuoKb4mp fXOhhh5ilbjVfpMRYo6MxKWPMRMYtWchORzC1pO4MXUKG4StLbFs4WvmWeDQEJQ4OfMJywJG llWMosWpxcW56UbGeqlFmcnFxfl5enmpJZsYganj4JbfujsYV792PMQowMGoxMO7XswyUog1 say4MvcQowQHs5II71EDoBBvSmJlVWpRfnxRaU5q8SFGaQ4WJXFeh30XIoQE0hNLUrNTUwtS i2CyTBycUg2MUtZd/8ubph97cCwrMsbu/pIeJSeFqVFPvGX3tSV5qqtbr47mvOXlcflkwFGl mTOKUj4wrQ82vMFuItTNw33WtV2dUcfXjeGoyiuBhLzQDU971iyquqn28N3zJVLPLj64mrg6 RzHkrrZDyOEe+S7xZcaLXpw4u3ZnN3uIzul8/lcvpO16u5iUWIozEg21mIuKEwHdF4nvGQMA AA== Subject: Re: [dpdk-users] Slow DPDK startup with many 1G hugepages X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 14:30:53 -0000 > -----Original Message----- > From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com] > Sent: 2017. j=FAnius 1. 11:03 > To: Imre Pinter ; users@dpdk.org > Cc: Gabor Hal=E1sz ; P=E9ter Suskovics > > Subject: Re: [dpdk-users] Slow DPDK startup with many 1G hugepages >=20 > On 01/06/2017 08:55, Imre Pinter wrote: > > Hi, > > > > We experience slow startup time in DPDK-OVS, when backing memory > with 1G hugepages instead of 2M hugepages. > > Currently we're mapping 2M hugepages as memory backend for DPDK > OVS. In the future we would like to allocate this memory from the 1G > hugepage pool. Currently in our deployments we have significant amount of > 1G hugepages allocated (min. 54G) for VMs and only 2G memory on 2M > hugepages. > > > > Typical setup for 2M hugepages: > > GRUB: > > hugepagesz=3D2M hugepages=3D1024 hugepagesz=3D1G hugepages=3D54 > > default_hugepagesz=3D1G > > > > $ grep hugetlbfs /proc/mounts > > nodev /mnt/huge_ovs_2M hugetlbfs rw,relatime,pagesize=3D2M 0 0 nodev > > /mnt/huge_qemu_1G hugetlbfs rw,relatime,pagesize=3D1G 0 0 > > > > Typical setup for 1GB hugepages: > > GRUB: > > hugepagesz=3D1G hugepages=3D56 default_hugepagesz=3D1G > > > > $ grep hugetlbfs /proc/mounts > > nodev /mnt/huge_qemu_1G hugetlbfs rw,relatime,pagesize=3D1G 0 0 > > > > DPDK OVS startup times based on the ovs-vswitchd.log logs: > > > > * 2M (2G memory allocated) - startup time ~3 sec: > > > > 2017-05-03T08:13:50.177Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c > 0x1 > > --huge-dir /mnt/huge_ovs_2M --socket-mem 1024,1024 > > > > 2017-05-03T08:13:50.708Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: > > Datapath supports recirculation > > > > * 1G (56G memory allocated) - startup time ~13 sec: > > 2017-05-03T08:09:22.114Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c > 0x1 > > --huge-dir /mnt/huge_qemu_1G --socket-mem 1024,1024 > > 2017-05-03T08:09:32.706Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: > > Datapath supports recirculation I used DPDK 16.11 for OVS and testpmd > and tested on Ubuntu 14.04 with kernel 3.13.0-117-generic and 4.4.0-78- > generic. > > > > We had a discussion with Mark Gray (from Intel), and he come up with th= e > following items: > > > > =B7 The ~10 sec time difference is there with testpmd as well > > > > =B7 They believe it is a kernel overhead (mmap is slow, perhaps= it is > zeroing pages). The following code from eal_memory.c does the above > mentioned printout in EAL startup: > > 469 /* map the segment, and populate page tables, > > 470 * the kernel fills this segment with zeros */ > > 468 uint64_t start =3D rte_rdtsc(); > > 471 virtaddr =3D mmap(vma_addr, hugepage_sz, PROT_READ | > PROT_WRITE, > > 472 MAP_SHARED | MAP_POPULATE, fd, 0); > > 473 if (virtaddr =3D=3D MAP_FAILED) { > > 474 RTE_LOG(DEBUG, EAL, "%s(): mmap failed: %s\n", __func__, > > 475 strerror(errno)); > > 476 close(fd); > > 477 return i; > > 478 } > > 479 > > 480 if (orig) { > > 481 hugepg_tbl[i].orig_va =3D virtaddr; > > 482 printf("Original mapping of page %u took: %"PRIu64" tick= s, > %"PRIu64" ms\n ", > > 483 i, rte_rdtsc() - start, > > 484 (rte_rdtsc() - start) * 1000 / > > 485 rte_get_timer_hz()); > > 486 } > > > > > > A solution could be to mount 1G hugepages to 2 separate directory: 2G f= or > OVS and the remaining for the VMs, but the NUMA location for these > hugepages is non-deterministic. Since mount cannot handle NUMA related > parameters during mounting hugetlbfs, and fstab forks the mounts during > boot. > > > > Do you have a solution on how to use 1G hugepages for VMs and have > reasonable DPDK EAL startup time? >=20 > In theory, one solution would be to use cgroup , as described here: > http://dpdk.org/ml/archives/dev/2017-February/057742.html > http://dpdk.org/ml/archives/dev/2017-April/063442.html >=20 > Then use 'numactl --interleave' policy. >=20 > I said in theory because it does not seem to work as one would expect, so > the proposed patch in above threads would be a solution by forcing > allocation from specific numa node for each page. >=20 > Thanks, > Sergio >=20 Thanks for the reply Sergio! The following patch (v5) at the end of the mentioned mail thread seems to b= e solving the issue. http://dpdk.org/dev/patchwork/patch/25069/ Thanks, Imre > > Thanks, > > Imre > >