From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7AF72A052B; Wed, 29 Jul 2020 10:50:58 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 205FB10A3; Wed, 29 Jul 2020 10:50:57 +0200 (CEST) Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-eopbgr40075.outbound.protection.outlook.com [40.107.4.75]) by dpdk.org (Postfix) with ESMTP id 73F441C025 for ; Tue, 28 Jul 2020 17:56:12 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mJ3cYTWbTLBr01cdMnl7uUu+/C23kCaBrPxuiJ9h1hmeKn+P82/JViYKfq4fkhzORYm/ogPtBX3SSD9EaCzIbSw8gVy5gbcgByMEqtVO4Ax+tSOXPNQYLBKZmHiCKoBi5KZyzAqdJIdCJLn15KECLWEY6mOlZJCpGJONK1i0dWwTkvmJN31N3fxIwwuoIQeqvkTU0I4weNiyvBN9jXENkqUQWd2ABD80mMMiEyXkpeMxZrmIpvitPyG9zzfgUdqf+LniX9l6nifs60vrtjLyI9T3KXcM8wlBgyDWH5miyHmNwQ+6JSm9tEVSQDTwVqMPSz/9tjcqJvh2pqt6uq0Z/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=amysiFwGCUgi2Vei4L6MvKC0l0rnMrn8Z8pneVBnPM4=; b=c6AkooT3rXReI7PDKOIeFytLDZSbO67tU+wyn2qJuTQm4WCs4zI7kPV4J0GbUsONmOCVvreQhmzsheVpIT7Bsmm0zG+XhFDvty/xiANqIYp7k0YCpfGJcozrbMBVMaugUFoYB45uisb1Cc+TSP+wbK3XCj6VJWGJcTA5SUJCdUOLU8grGe88NMkSqq1t6yZYMUkmVVRpVTV/SQ2a2kRcjol80x5e/fnawkqDIOuaFF+m8f24QZMpwobz6o3rcMeBXaH1GBs6xlmeA9Wq3JODQDLqgqAhTyNKBZlZvq7BuccSHSXnwSWeJjuDAFo9Fen9F0JA1kC5SRmsJg3BiUJAuw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ericsson.com; dmarc=pass action=none header.from=ericsson.com; dkim=pass header.d=ericsson.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=amysiFwGCUgi2Vei4L6MvKC0l0rnMrn8Z8pneVBnPM4=; b=YYHfnupZTBxKBcAhErjJR1H2xuAFdmyXpCB8mY8huowYNJO5+kGMZFHq0kqZ2OE+PcxSmM/2cb0aMy6uGXenh0VlkC9GwvDXW8X8DWzfFLmFjB8v10JbQivfi1iDBC7Zv6o2uOL7mWfrNT1l2psqfTgHSkUxM3Jm4eQQQ6GLt9o= Received: from AM0PR07MB6436.eurprd07.prod.outlook.com (2603:10a6:20b:15d::15) by AM0PR07MB6274.eurprd07.prod.outlook.com (2603:10a6:20b:152::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3239.12; Tue, 28 Jul 2020 15:56:11 +0000 Received: from AM0PR07MB6436.eurprd07.prod.outlook.com ([fe80::ed04:ef5b:cd0:5b90]) by AM0PR07MB6436.eurprd07.prod.outlook.com ([fe80::ed04:ef5b:cd0:5b90%3]) with mapi id 15.20.3239.013; Tue, 28 Jul 2020 15:56:11 +0000 From: Balaji K N To: "dev@dpdk.org" Thread-Topic: long initialization time for EAL init Thread-Index: AdZk92DSbpHzHW9QTMuDuuG9athRuw== Date: Tue, 28 Jul 2020 15:56:11 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=ericsson.com; x-originating-ip: [2402:3a80:d3b:5c0e:49cb:d48c:34d8:d5be] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1201d47a-ee15-4d7f-3635-08d8330ebfd3 x-ms-traffictypediagnostic: AM0PR07MB6274: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: zyeInTwSvpMqty3BmFCjxsKx1yGqIvbVUP/Q3vtKnzJU4UpeKntrm5mYXAtsqMEm7iUYyTZDgbdsZypjh69AdCnztte16dwATRdthOjJocJl2PhWJYiuVJ0b7gFfQ1uPU0dnoFAzCH50h+sEZBn56oXsKGgf2pMojc4CrtrDRRjdSs6By7SamnVMSBSmpurm6kkDVqdAAV+Yql6v/02XU5TWF2XB21DlfgRfsvePw6X+J5+PcOkXrY9nYYYqRV3y7sweVy2KhYkAQFnXmMjFo3Iaf0GDYxeqD7mW52M0QHvhb0cMfipq5s/UYDLS4woNoMxsUKFqPmlUd8YKhnDRgA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR07MB6436.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(396003)(136003)(366004)(376002)(346002)(39860400002)(8676002)(71200400001)(7696005)(66446008)(2906002)(9686003)(6916009)(52536014)(66946007)(66476007)(66556008)(64756008)(6506007)(5660300002)(76116006)(83380400001)(8936002)(33656002)(86362001)(478600001)(55016002)(186003)(316002)(9326002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: APEk6hYoa7lgQk9cY08iIf5Q+qlWIGOycAyi+XMhTu6yC2iBL1Har72OpoPAZ1D+PNx37qmAv+1A8bstQ0H8AVn/fV0TUKHbCX1cGVQAs2JF776XP+tc98AMU49rk9eG04Nng8jxrw8M/2t1JXulXH4G6R9NBKqz14k7jkZyHvzq63mW4am1JmHFHh2cACeUDe5fUNM1LdomWgSKViAPomtxKEE2DlFri+bA2uUuhDC7M4DRdH3FsuWWCEG8PdfzfLEht8dzenjNvIPo32LJTdFlkTqsyCnD487H2b/gSvFNrBibvc9WRUcE8PBpyKI8RuKGR6WSj2gITjGOlCfwVg5k8L+R0rBl9CkYcYiKVz5UoWZLwU08VOBoUrjSDjDqwCYmMgTTXQDvQIbCHch468T8G//KyAszEJ+gidQ0m9XwlTD3/zdscOlQFFMZrIcrMKDNemRTkE/k7Xr3k051TZbZtVHqDvN+VATh06ECdcrbVnNNTl2k8GFVZFtwAdxp2JjElQSvT0N99RE4BmiiHP8mhxCT2qEK7uooAB2C8TX/JV6tWPQX05bGGwKJ4miQtmD3RplMq+N75BI7VwiCXQKTHwSvhNPL/5ujtlc0Sg7Qw7plj04jYMzqCnmy0Bny06BVHGXX4zNo4SnFNVcYGvCX4jyxkQwqt6BHEmMcJMLLYsYInbBNRSPdMHc0bG8FkxrKlWnRoN8y9zBdb07OlA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM0PR07MB6436.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1201d47a-ee15-4d7f-3635-08d8330ebfd3 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jul 2020 15:56:11.3286 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: lTI57GIx+SxP1SX+SDQaVP4ljQcOjPqkKgNH4pePWLY4dnSUygr5/+9n6fvuJS6ZOLMuXzGcIGV9+TH+QAQHLw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR07MB6274 X-Mailman-Approved-At: Wed, 29 Jul 2020 10:50:55 +0200 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] long initialization time for EAL init X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hello, We are using OVS 2.12.0 with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We = do observe long initialization time for EAL init ( ~10 seconds for EAL memo= ry initialization). Below is our hardware and memory configuration Hardware : Dell Server Sockets(NUMA) : 2 CPUs : Each NUMA with 10 CPUs (Intel(R) Xeon(R= ) CPU E5-2660 v2 @ 2.20GHz) RAM : 128 GB Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (rese= rved during boot time). We are using only 2MB huge pages for OVS process. NIC : Ethernet 10G 2P X520 ( ixgbe driver = ) Passing below EAL arguments with OVS process. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:= 00.0,support-multi-driver=3D1 -w 0000:42:00.0,support-multi-driver=3D1 -c 0= x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2= 048. We debugged further in DPDK and found out that mmap called in eal_get_virtu= al_area is taking more time ( rte_eal_init-> rte_eal_memory_init -> rte_ea= l_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_a= rea ->mmap ). Long initialization time is noticeable difference compared to= prior release. Any pointers would be helpful. EAL Init logs ( rte_eal_memseg_init function taking ~8 seconds ) 2020-07-23T09:46:58.878Z|00008|dpdk|INFO|Using DPDK 18.11.2 2020-07-23T09:46:58.878Z|00009|dpdk|INFO|DPDK Enabled - initializing... 2020-07-23T09:46:58.878Z|00010|dpdk|INFO|No vhost-sock-dir provided - defau= lting to /var/run/openvswitch 2020-07-23T09:46:58.878Z|00011|dpdk|INFO|IOMMU support for vhost-user-clien= t disabled. 2020-07-23T09:46:58.878Z|00012|dpdk|INFO|POSTCOPY support for vhost-user-cl= ient disabled. 2020-07-23T09:46:58.878Z|00013|dpdk|INFO|Per port memory for DPDK devices d= isabled. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:= 00.0,support-multi-driver=3D1 -w 0000:42:00.0,support-multi-driver=3D1 -c 0= x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2= 048. 2020-07-23T09:46:58.882Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s) 2020-07-23T09:46:58.882Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-07-23T09:46:58.885Z|00017|dpdk|INFO|EAL: Multi-process socket /var/run= /dpdk/rte/mp_socket 2020-07-23T09:46:58.907Z|00018|dpdk|INFO|EAL: 100 hugepages of size 1073741= 824 reserved, but no mounted hugetlbfs found for that size 2020-07-23T09:46:58.907Z|00019|dpdk|INFO|EAL: Probing VFIO support... 2020-07-23T09:46:58.907Z|00020|dpdk|INFO|EAL: VFIO support initialized 2020-07-23T09:46:58.907Z|00021|dpdk|INFO|EAL: Ask a virtual area of 0x2e000= bytes 2020-07-23T09:46:58.907Z|00023|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00024|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00026|dpdk|INFO|EAL: Virtual area found at 0x10000= 0000 (size =3D 0x2e000) 2020-07-23T09:46:58.907Z|00029|dpdk|INFO|EAL: Setting up physically contigu= ous memory... 2020-07-23T09:46:58.907Z|00032|dpdk|INFO|EAL: Calling rte_eal_memseg_init..= . 2020-07-23T09:46:58.907Z|00034|dpdk|INFO|EAL: Setting maximum number of ope= n files to 65535 2020-07-23T09:46:58.907Z|00039|dpdk|INFO|EAL: Creating 4 segment lists: n_s= egs:8192 socket_id:0 hugepage_sz:2097152 2020-07-23T09:46:58.907Z|00042|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:46:58.907Z|00044|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00045|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00047|dpdk|INFO|EAL: Virtual area found at 0x10002= e000 (size =3D 0x61000) 2020-07-23T09:46:58.907Z|00053|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:46:58.907Z|00055|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.887Z|00056|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.887Z|00058|dpdk|INFO|EAL: Virtual area found at 0x10020= 0000 (size =3D 0x400000000) 2020-07-23T09:46:59.888Z|00063|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:46:59.888Z|00065|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.888Z|00066|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.888Z|00068|dpdk|INFO|EAL: Virtual area found at 0x50020= 0000 (size =3D 0x61000) 2020-07-23T09:46:59.888Z|00074|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:46:59.888Z|00076|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00077|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00079|dpdk|INFO|EAL: Virtual area found at 0x50040= 0000 (size =3D 0x400000000) 2020-07-23T09:47:00.824Z|00084|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:47:00.824Z|00086|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00087|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00089|dpdk|INFO|EAL: Virtual area found at 0x90040= 0000 (size =3D 0x61000) 2020-07-23T09:47:00.824Z|00095|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:47:00.824Z|00097|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00098|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00100|dpdk|INFO|EAL: Virtual area found at 0x90060= 0000 (size =3D 0x400000000) 2020-07-23T09:47:01.754Z|00105|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:47:01.754Z|00107|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00108|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00110|dpdk|INFO|EAL: Virtual area found at 0xd0060= 0000 (size =3D 0x61000) 2020-07-23T09:47:01.754Z|00116|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:47:01.754Z|00118|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00119|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00121|dpdk|INFO|EAL: Virtual area found at 0xd0080= 0000 (size =3D 0x400000000) 2020-07-23T09:47:02.711Z|00124|dpdk|INFO|EAL: Creating 4 segment lists: n_s= egs:8192 socket_id:1 hugepage_sz:2097152 2020-07-23T09:47:02.711Z|00127|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:47:02.711Z|00129|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00130|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00132|dpdk|INFO|EAL: Virtual area found at 0x11008= 00000 (size =3D 0x61000) 2020-07-23T09:47:02.711Z|00138|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:47:02.711Z|00140|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00141|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00143|dpdk|INFO|EAL: Virtual area found at 0x1100a= 00000 (size =3D 0x400000000) 2020-07-23T09:47:03.641Z|00148|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:47:03.641Z|00150|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00151|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00153|dpdk|INFO|EAL: Virtual area found at 0x1500a= 00000 (size =3D 0x61000) 2020-07-23T09:47:03.642Z|00159|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:47:03.642Z|00161|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.628Z|00162|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.628Z|00164|dpdk|INFO|EAL: Virtual area found at 0x1500c= 00000 (size =3D 0x400000000) 2020-07-23T09:47:04.629Z|00169|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:47:04.629Z|00171|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.629Z|00172|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.629Z|00174|dpdk|INFO|EAL: Virtual area found at 0x1900c= 00000 (size =3D 0x61000) 2020-07-23T09:47:04.629Z|00180|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:47:04.629Z|00182|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00183|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00185|dpdk|INFO|EAL: Virtual area found at 0x1900e= 00000 (size =3D 0x400000000) 2020-07-23T09:47:05.733Z|00190|dpdk|INFO|EAL: Ask a virtual area of 0x61000= bytes 2020-07-23T09:47:05.733Z|00192|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00193|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00195|dpdk|INFO|EAL: Virtual area found at 0x1d00e= 00000 (size =3D 0x61000) 2020-07-23T09:47:05.734Z|00201|dpdk|INFO|EAL: Ask a virtual area of 0x40000= 0000 bytes 2020-07-23T09:47:05.734Z|00203|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:06.728Z|00204|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:06.728Z|00206|dpdk|INFO|EAL: Virtual area found at 0x1d010= 00000 (size =3D 0x400000000) 2020-07-23T09:47:06.728Z|00210|dpdk|INFO|EAL: Done rte_eal_memseg_init... 2020-07-23T09:47:06.728Z|00213|dpdk|INFO|EAL: Calling rte_eal_hugepage_init= /rte_eal_hugepage_attach... 2020-07-23T09:47:08.115Z|00214|dpdk|INFO|EAL: Done rte_eal_hugepage_init/rt= e_eal_hugepage_attach... 2020-07-23T09:47:08.237Z|00215|dpdk|INFO|EAL: PCI device 0000:04:00.0 on NU= MA socket 0 2020-07-23T09:47:08.237Z|00216|dpdk|INFO|EAL: probe driver: 8086:154d net= _ixgbe 2020-07-23T09:47:08.237Z|00217|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) 2020-07-23T09:47:08.481Z|00218|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:08.831Z|00219|dpdk|INFO|EAL: PCI device 0000:42:00.0 on NU= MA socket 1 2020-07-23T09:47:08.831Z|00220|dpdk|INFO|EAL: probe driver: 8086:154d net= _ixgbe 2020-07-23T09:47:08.957Z|00221|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:09.308Z|00222|dpdk|INFO|DPDK Enabled - initialized Regards, Balaji