From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5143BA052B; Tue, 28 Jul 2020 19:58:30 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A3DE010A3; Tue, 28 Jul 2020 19:58:29 +0200 (CEST) Received: from mail.altencalsoftlabs.com (mail.altencalsoftlabs.com [182.73.72.41]) by dpdk.org (Postfix) with ESMTP id 3F431A69 for ; Tue, 28 Jul 2020 19:58:27 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail.altencalsoftlabs.com (Postfix) with ESMTP id 756694420074; Tue, 28 Jul 2020 23:28:26 +0530 (IST) Received: from mail.altencalsoftlabs.com ([127.0.0.1]) by localhost (mail.altencalsoftlabs.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id vmGp2Y0f0pld; Tue, 28 Jul 2020 23:28:25 +0530 (IST) Received: from localhost (localhost [127.0.0.1]) by mail.altencalsoftlabs.com (Postfix) with ESMTP id 6F11F44200AE; Tue, 28 Jul 2020 23:28:25 +0530 (IST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.altencalsoftlabs.com 6F11F44200AE DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=altencalsoftlabs.com; s=selector; t=1595959105; bh=WV522PvxgPAmy1NwxpYXYdHOOKzjWCZc+gdhaDm/yGI=; h=From:To:Date:Message-ID:MIME-Version; b=5tHdm0/rLXwKLeRzwSnOIEH2edvH8bjcW2hua2MJIREIkmDkcECHkw/31eSO9Jmms yUFt1PnlBVXCHw93kaL0HckRlJus1hv3/YgfjvyLIYMK5lJR+6YNzTPyWGuaBa//KQ RZuJsofyXNfm358Kp/ZWtDTy7yBqsVFFb3gvMlZs= X-Virus-Scanned: amavisd-new at altencalsoftlabs.com Received: from mail.altencalsoftlabs.com ([127.0.0.1]) by localhost (mail.altencalsoftlabs.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Eapcuj0QJoh1; Tue, 28 Jul 2020 23:28:25 +0530 (IST) Received: from ACLBLRFC865 (unknown [10.1.0.4]) by mail.altencalsoftlabs.com (Postfix) with ESMTPSA id 17A9F4420074; Tue, 28 Jul 2020 23:28:24 +0530 (IST) From: "Anil Kumar Koli" To: Cc: "'Balaji K N'" Date: Tue, 28 Jul 2020 23:28:24 +0530 Message-ID: <00d501d66508$b0b6c4c0$12244e40$@altencalsoftlabs.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: AdZlCCtuR9KpxtfzRdKXxAtPzsqz3A== Content-Language: en-us Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] long initialization time for EAL init X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Team, Posting this query on behalf of Balaji as he was not able to send mail to dpdk-dev. We are using OVS 2.12.0 with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization). Below is our hardware and memory configuration Hardware : Dell Server Sockets(NUMA) : 2 CPUs : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz) RAM : 128 GB Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process. NIC : Ethernet 10G 2P X520 ( ixgbe driver ) Passing below EAL arguments with OVS process. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048. We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init-> rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful. EAL Init logs ( rte_eal_memseg_init function taking ~8 seconds ) 2020-07-23T09:46:58.878Z|00008|dpdk|INFO|Using DPDK 18.11.2 2020-07-23T09:46:58.878Z|00009|dpdk|INFO|DPDK Enabled - initializing... 2020-07-23T09:46:58.878Z|00010|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2020-07-23T09:46:58.878Z|00011|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2020-07-23T09:46:58.878Z|00012|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2020-07-23T09:46:58.878Z|00013|dpdk|INFO|Per port memory for DPDK devices disabled. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048. 2020-07-23T09:46:58.882Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s) 2020-07-23T09:46:58.882Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-07-23T09:46:58.885Z|00017|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket 2020-07-23T09:46:58.907Z|00018|dpdk|INFO|EAL: 100 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size 2020-07-23T09:46:58.907Z|00019|dpdk|INFO|EAL: Probing VFIO support... 2020-07-23T09:46:58.907Z|00020|dpdk|INFO|EAL: VFIO support initialized 2020-07-23T09:46:58.907Z|00021|dpdk|INFO|EAL: Ask a virtual area of 0x2e000 bytes 2020-07-23T09:46:58.907Z|00023|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00024|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00026|dpdk|INFO|EAL: Virtual area found at 0x100000000 (size = 0x2e000) 2020-07-23T09:46:58.907Z|00029|dpdk|INFO|EAL: Setting up physically contiguous memory... 2020-07-23T09:46:58.907Z|00032|dpdk|INFO|EAL: Calling rte_eal_memseg_init... 2020-07-23T09:46:58.907Z|00034|dpdk|INFO|EAL: Setting maximum number of open files to 65535 2020-07-23T09:46:58.907Z|00039|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 2020-07-23T09:46:58.907Z|00042|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:46:58.907Z|00044|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00045|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00047|dpdk|INFO|EAL: Virtual area found at 0x10002e000 (size = 0x61000) 2020-07-23T09:46:58.907Z|00053|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:46:58.907Z|00055|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.887Z|00056|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.887Z|00058|dpdk|INFO|EAL: Virtual area found at 0x100200000 (size = 0x400000000) 2020-07-23T09:46:59.888Z|00063|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:46:59.888Z|00065|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.888Z|00066|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.888Z|00068|dpdk|INFO|EAL: Virtual area found at 0x500200000 (size = 0x61000) 2020-07-23T09:46:59.888Z|00074|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:46:59.888Z|00076|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00077|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00079|dpdk|INFO|EAL: Virtual area found at 0x500400000 (size = 0x400000000) 2020-07-23T09:47:00.824Z|00084|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:00.824Z|00086|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00087|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00089|dpdk|INFO|EAL: Virtual area found at 0x900400000 (size = 0x61000) 2020-07-23T09:47:00.824Z|00095|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:00.824Z|00097|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00098|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00100|dpdk|INFO|EAL: Virtual area found at 0x900600000 (size = 0x400000000) 2020-07-23T09:47:01.754Z|00105|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:01.754Z|00107|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00108|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00110|dpdk|INFO|EAL: Virtual area found at 0xd00600000 (size = 0x61000) 2020-07-23T09:47:01.754Z|00116|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:01.754Z|00118|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00119|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00121|dpdk|INFO|EAL: Virtual area found at 0xd00800000 (size = 0x400000000) 2020-07-23T09:47:02.711Z|00124|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152 2020-07-23T09:47:02.711Z|00127|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:02.711Z|00129|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00130|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00132|dpdk|INFO|EAL: Virtual area found at 0x1100800000 (size = 0x61000) 2020-07-23T09:47:02.711Z|00138|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:02.711Z|00140|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00141|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00143|dpdk|INFO|EAL: Virtual area found at 0x1100a00000 (size = 0x400000000) 2020-07-23T09:47:03.641Z|00148|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:03.641Z|00150|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00151|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00153|dpdk|INFO|EAL: Virtual area found at 0x1500a00000 (size = 0x61000) 2020-07-23T09:47:03.642Z|00159|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:03.642Z|00161|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.628Z|00162|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.628Z|00164|dpdk|INFO|EAL: Virtual area found at 0x1500c00000 (size = 0x400000000) 2020-07-23T09:47:04.629Z|00169|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:04.629Z|00171|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.629Z|00172|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.629Z|00174|dpdk|INFO|EAL: Virtual area found at 0x1900c00000 (size = 0x61000) 2020-07-23T09:47:04.629Z|00180|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:04.629Z|00182|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00183|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00185|dpdk|INFO|EAL: Virtual area found at 0x1900e00000 (size = 0x400000000) 2020-07-23T09:47:05.733Z|00190|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:05.733Z|00192|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00193|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00195|dpdk|INFO|EAL: Virtual area found at 0x1d00e00000 (size = 0x61000) 2020-07-23T09:47:05.734Z|00201|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:05.734Z|00203|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:06.728Z|00204|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:06.728Z|00206|dpdk|INFO|EAL: Virtual area found at 0x1d01000000 (size = 0x400000000) 2020-07-23T09:47:06.728Z|00210|dpdk|INFO|EAL: Done rte_eal_memseg_init... 2020-07-23T09:47:06.728Z|00213|dpdk|INFO|EAL: Calling rte_eal_hugepage_init/rte_eal_hugepage_attach... 2020-07-23T09:47:08.115Z|00214|dpdk|INFO|EAL: Done rte_eal_hugepage_init/rte_eal_hugepage_attach... 2020-07-23T09:47:08.237Z|00215|dpdk|INFO|EAL: PCI device 0000:04:00.0 on NUMA socket 0 2020-07-23T09:47:08.237Z|00216|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe 2020-07-23T09:47:08.237Z|00217|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) 2020-07-23T09:47:08.481Z|00218|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:08.831Z|00219|dpdk|INFO|EAL: PCI device 0000:42:00.0 on NUMA socket 1 2020-07-23T09:47:08.831Z|00220|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe 2020-07-23T09:47:08.957Z|00221|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:09.308Z|00222|dpdk|INFO|DPDK Enabled - initialized Thanks & Regards, Anil Kumar.