Hello, We are using OVS 2.12.0 with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization). Below is our hardware and memory configuration Hardware : Dell Server Sockets(NUMA) : 2 CPUs : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz) RAM : 128 GB Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process. NIC : Ethernet 10G 2P X520 ( ixgbe driver ) Passing below EAL arguments with OVS process. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048. We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init-> rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful. EAL Init logs ( rte_eal_memseg_init function taking ~8 seconds ) 2020-07-23T09:46:58.878Z|00008|dpdk|INFO|Using DPDK 18.11.2 2020-07-23T09:46:58.878Z|00009|dpdk|INFO|DPDK Enabled - initializing... 2020-07-23T09:46:58.878Z|00010|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2020-07-23T09:46:58.878Z|00011|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2020-07-23T09:46:58.878Z|00012|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2020-07-23T09:46:58.878Z|00013|dpdk|INFO|Per port memory for DPDK devices disabled. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048. 2020-07-23T09:46:58.882Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s) 2020-07-23T09:46:58.882Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-07-23T09:46:58.885Z|00017|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket 2020-07-23T09:46:58.907Z|00018|dpdk|INFO|EAL: 100 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size 2020-07-23T09:46:58.907Z|00019|dpdk|INFO|EAL: Probing VFIO support... 2020-07-23T09:46:58.907Z|00020|dpdk|INFO|EAL: VFIO support initialized 2020-07-23T09:46:58.907Z|00021|dpdk|INFO|EAL: Ask a virtual area of 0x2e000 bytes 2020-07-23T09:46:58.907Z|00023|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00024|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00026|dpdk|INFO|EAL: Virtual area found at 0x100000000 (size = 0x2e000) 2020-07-23T09:46:58.907Z|00029|dpdk|INFO|EAL: Setting up physically contiguous memory... 2020-07-23T09:46:58.907Z|00032|dpdk|INFO|EAL: Calling rte_eal_memseg_init... 2020-07-23T09:46:58.907Z|00034|dpdk|INFO|EAL: Setting maximum number of open files to 65535 2020-07-23T09:46:58.907Z|00039|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 2020-07-23T09:46:58.907Z|00042|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:46:58.907Z|00044|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00045|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00047|dpdk|INFO|EAL: Virtual area found at 0x10002e000 (size = 0x61000) 2020-07-23T09:46:58.907Z|00053|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:46:58.907Z|00055|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.887Z|00056|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.887Z|00058|dpdk|INFO|EAL: Virtual area found at 0x100200000 (size = 0x400000000) 2020-07-23T09:46:59.888Z|00063|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:46:59.888Z|00065|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.888Z|00066|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.888Z|00068|dpdk|INFO|EAL: Virtual area found at 0x500200000 (size = 0x61000) 2020-07-23T09:46:59.888Z|00074|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:46:59.888Z|00076|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00077|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00079|dpdk|INFO|EAL: Virtual area found at 0x500400000 (size = 0x400000000) 2020-07-23T09:47:00.824Z|00084|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:00.824Z|00086|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00087|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00089|dpdk|INFO|EAL: Virtual area found at 0x900400000 (size = 0x61000) 2020-07-23T09:47:00.824Z|00095|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:00.824Z|00097|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00098|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00100|dpdk|INFO|EAL: Virtual area found at 0x900600000 (size = 0x400000000) 2020-07-23T09:47:01.754Z|00105|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:01.754Z|00107|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00108|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00110|dpdk|INFO|EAL: Virtual area found at 0xd00600000 (size = 0x61000) 2020-07-23T09:47:01.754Z|00116|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:01.754Z|00118|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00119|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00121|dpdk|INFO|EAL: Virtual area found at 0xd00800000 (size = 0x400000000) 2020-07-23T09:47:02.711Z|00124|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152 2020-07-23T09:47:02.711Z|00127|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:02.711Z|00129|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00130|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00132|dpdk|INFO|EAL: Virtual area found at 0x1100800000 (size = 0x61000) 2020-07-23T09:47:02.711Z|00138|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:02.711Z|00140|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00141|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00143|dpdk|INFO|EAL: Virtual area found at 0x1100a00000 (size = 0x400000000) 2020-07-23T09:47:03.641Z|00148|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:03.641Z|00150|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00151|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00153|dpdk|INFO|EAL: Virtual area found at 0x1500a00000 (size = 0x61000) 2020-07-23T09:47:03.642Z|00159|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:03.642Z|00161|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.628Z|00162|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.628Z|00164|dpdk|INFO|EAL: Virtual area found at 0x1500c00000 (size = 0x400000000) 2020-07-23T09:47:04.629Z|00169|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:04.629Z|00171|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.629Z|00172|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.629Z|00174|dpdk|INFO|EAL: Virtual area found at 0x1900c00000 (size = 0x61000) 2020-07-23T09:47:04.629Z|00180|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:04.629Z|00182|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00183|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00185|dpdk|INFO|EAL: Virtual area found at 0x1900e00000 (size = 0x400000000) 2020-07-23T09:47:05.733Z|00190|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:05.733Z|00192|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00193|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00195|dpdk|INFO|EAL: Virtual area found at 0x1d00e00000 (size = 0x61000) 2020-07-23T09:47:05.734Z|00201|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:05.734Z|00203|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:06.728Z|00204|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:06.728Z|00206|dpdk|INFO|EAL: Virtual area found at 0x1d01000000 (size = 0x400000000) 2020-07-23T09:47:06.728Z|00210|dpdk|INFO|EAL: Done rte_eal_memseg_init... 2020-07-23T09:47:06.728Z|00213|dpdk|INFO|EAL: Calling rte_eal_hugepage_init/rte_eal_hugepage_attach... 2020-07-23T09:47:08.115Z|00214|dpdk|INFO|EAL: Done rte_eal_hugepage_init/rte_eal_hugepage_attach... 2020-07-23T09:47:08.237Z|00215|dpdk|INFO|EAL: PCI device 0000:04:00.0 on NUMA socket 0 2020-07-23T09:47:08.237Z|00216|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe 2020-07-23T09:47:08.237Z|00217|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) 2020-07-23T09:47:08.481Z|00218|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:08.831Z|00219|dpdk|INFO|EAL: PCI device 0000:42:00.0 on NUMA socket 1 2020-07-23T09:47:08.831Z|00220|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe 2020-07-23T09:47:08.957Z|00221|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:09.308Z|00222|dpdk|INFO|DPDK Enabled - initialized Regards, Balaji
Hi Team, Posting this query on behalf of Balaji as he was not able to send mail to dpdk-dev. We are using OVS 2.12.0 with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization). Below is our hardware and memory configuration Hardware : Dell Server Sockets(NUMA) : 2 CPUs : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz) RAM : 128 GB Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process. NIC : Ethernet 10G 2P X520 ( ixgbe driver ) Passing below EAL arguments with OVS process. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048. We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init-> rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful. EAL Init logs ( rte_eal_memseg_init function taking ~8 seconds ) 2020-07-23T09:46:58.878Z|00008|dpdk|INFO|Using DPDK 18.11.2 2020-07-23T09:46:58.878Z|00009|dpdk|INFO|DPDK Enabled - initializing... 2020-07-23T09:46:58.878Z|00010|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2020-07-23T09:46:58.878Z|00011|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2020-07-23T09:46:58.878Z|00012|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2020-07-23T09:46:58.878Z|00013|dpdk|INFO|Per port memory for DPDK devices disabled. 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048. 2020-07-23T09:46:58.882Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s) 2020-07-23T09:46:58.882Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-07-23T09:46:58.885Z|00017|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket 2020-07-23T09:46:58.907Z|00018|dpdk|INFO|EAL: 100 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size 2020-07-23T09:46:58.907Z|00019|dpdk|INFO|EAL: Probing VFIO support... 2020-07-23T09:46:58.907Z|00020|dpdk|INFO|EAL: VFIO support initialized 2020-07-23T09:46:58.907Z|00021|dpdk|INFO|EAL: Ask a virtual area of 0x2e000 bytes 2020-07-23T09:46:58.907Z|00023|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00024|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00026|dpdk|INFO|EAL: Virtual area found at 0x100000000 (size = 0x2e000) 2020-07-23T09:46:58.907Z|00029|dpdk|INFO|EAL: Setting up physically contiguous memory... 2020-07-23T09:46:58.907Z|00032|dpdk|INFO|EAL: Calling rte_eal_memseg_init... 2020-07-23T09:46:58.907Z|00034|dpdk|INFO|EAL: Setting maximum number of open files to 65535 2020-07-23T09:46:58.907Z|00039|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 2020-07-23T09:46:58.907Z|00042|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:46:58.907Z|00044|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:58.907Z|00045|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:58.907Z|00047|dpdk|INFO|EAL: Virtual area found at 0x10002e000 (size = 0x61000) 2020-07-23T09:46:58.907Z|00053|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:46:58.907Z|00055|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.887Z|00056|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.887Z|00058|dpdk|INFO|EAL: Virtual area found at 0x100200000 (size = 0x400000000) 2020-07-23T09:46:59.888Z|00063|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:46:59.888Z|00065|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:46:59.888Z|00066|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:46:59.888Z|00068|dpdk|INFO|EAL: Virtual area found at 0x500200000 (size = 0x61000) 2020-07-23T09:46:59.888Z|00074|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:46:59.888Z|00076|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00077|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00079|dpdk|INFO|EAL: Virtual area found at 0x500400000 (size = 0x400000000) 2020-07-23T09:47:00.824Z|00084|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:00.824Z|00086|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:00.824Z|00087|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:00.824Z|00089|dpdk|INFO|EAL: Virtual area found at 0x900400000 (size = 0x61000) 2020-07-23T09:47:00.824Z|00095|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:00.824Z|00097|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00098|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00100|dpdk|INFO|EAL: Virtual area found at 0x900600000 (size = 0x400000000) 2020-07-23T09:47:01.754Z|00105|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:01.754Z|00107|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:01.754Z|00108|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:01.754Z|00110|dpdk|INFO|EAL: Virtual area found at 0xd00600000 (size = 0x61000) 2020-07-23T09:47:01.754Z|00116|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:01.754Z|00118|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00119|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00121|dpdk|INFO|EAL: Virtual area found at 0xd00800000 (size = 0x400000000) 2020-07-23T09:47:02.711Z|00124|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152 2020-07-23T09:47:02.711Z|00127|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:02.711Z|00129|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:02.711Z|00130|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:02.711Z|00132|dpdk|INFO|EAL: Virtual area found at 0x1100800000 (size = 0x61000) 2020-07-23T09:47:02.711Z|00138|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:02.711Z|00140|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00141|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00143|dpdk|INFO|EAL: Virtual area found at 0x1100a00000 (size = 0x400000000) 2020-07-23T09:47:03.641Z|00148|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:03.641Z|00150|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:03.641Z|00151|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:03.641Z|00153|dpdk|INFO|EAL: Virtual area found at 0x1500a00000 (size = 0x61000) 2020-07-23T09:47:03.642Z|00159|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:03.642Z|00161|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.628Z|00162|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.628Z|00164|dpdk|INFO|EAL: Virtual area found at 0x1500c00000 (size = 0x400000000) 2020-07-23T09:47:04.629Z|00169|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:04.629Z|00171|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:04.629Z|00172|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:04.629Z|00174|dpdk|INFO|EAL: Virtual area found at 0x1900c00000 (size = 0x61000) 2020-07-23T09:47:04.629Z|00180|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:04.629Z|00182|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00183|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00185|dpdk|INFO|EAL: Virtual area found at 0x1900e00000 (size = 0x400000000) 2020-07-23T09:47:05.733Z|00190|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes 2020-07-23T09:47:05.733Z|00192|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:05.733Z|00193|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:05.733Z|00195|dpdk|INFO|EAL: Virtual area found at 0x1d00e00000 (size = 0x61000) 2020-07-23T09:47:05.734Z|00201|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes 2020-07-23T09:47:05.734Z|00203|dpdk|INFO|EAL: Before mmap ... 2020-07-23T09:47:06.728Z|00204|dpdk|INFO|EAL: After mmap ... 2020-07-23T09:47:06.728Z|00206|dpdk|INFO|EAL: Virtual area found at 0x1d01000000 (size = 0x400000000) 2020-07-23T09:47:06.728Z|00210|dpdk|INFO|EAL: Done rte_eal_memseg_init... 2020-07-23T09:47:06.728Z|00213|dpdk|INFO|EAL: Calling rte_eal_hugepage_init/rte_eal_hugepage_attach... 2020-07-23T09:47:08.115Z|00214|dpdk|INFO|EAL: Done rte_eal_hugepage_init/rte_eal_hugepage_attach... 2020-07-23T09:47:08.237Z|00215|dpdk|INFO|EAL: PCI device 0000:04:00.0 on NUMA socket 0 2020-07-23T09:47:08.237Z|00216|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe 2020-07-23T09:47:08.237Z|00217|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) 2020-07-23T09:47:08.481Z|00218|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:08.831Z|00219|dpdk|INFO|EAL: PCI device 0000:42:00.0 on NUMA socket 1 2020-07-23T09:47:08.831Z|00220|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe 2020-07-23T09:47:08.957Z|00221|dpdk|INFO|EAL: Ignore mapping IO port bar(2) 2020-07-23T09:47:09.308Z|00222|dpdk|INFO|DPDK Enabled - initialized Thanks & Regards, Anil Kumar.
On Wed, Jul 29, 2020 at 10:51 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
> We are using OVS 2.12.0 with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization).
>
> Below is our hardware and memory configuration
> Hardware : Dell Server
> Sockets(NUMA) : 2
> CPUs : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz)
> RAM : 128 GB
> Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process.
> NIC : Ethernet 10G 2P X520 ( ixgbe driver )
>
> Passing below EAL arguments with OVS process.
> 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
>
> We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init-> rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful.
Did you try to start ovs with the --no-mlockall option?
--
David Marchand
On 29-Jul-20 9:52 AM, David Marchand wrote:
> On Wed, Jul 29, 2020 at 10:51 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
>> We are using OVS 2.12.0 with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization).
>>
>> Below is our hardware and memory configuration
>> Hardware : Dell Server
>> Sockets(NUMA) : 2
>> CPUs : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz)
>> RAM : 128 GB
>> Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process.
>> NIC : Ethernet 10G 2P X520 ( ixgbe driver )
>>
>> Passing below EAL arguments with OVS process.
>> 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
>>
>> We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init-> rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful.
>
> Did you try to start ovs with the --no-mlockall option?
>
>
That's most likely the culprit. I believe the recent mlock/MPROT changes
are going to be backported to 18.11?
--
Thanks,
Anatoly
Hello,
I tried by disabling mlockall and not do see long time for EAL init and is done quickly. Thanks for help.
ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
OVS logs
2020-07-29T09:48:00.233Z|00008|dpdk|INFO|Using DPDK 18.11.6
2020-07-29T09:48:00.233Z|00009|dpdk|INFO|DPDK Enabled - initializing...
2020-07-29T09:48:00.233Z|00010|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2020-07-29T09:48:00.233Z|00011|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2020-07-29T09:48:00.233Z|00012|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2020-07-29T09:48:00.233Z|00013|dpdk|INFO|Per port memory for DPDK devices disabled.
2020-07-29T09:48:00.233Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
2020-07-29T09:48:00.237Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s)
2020-07-29T09:48:00.237Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes
2020-07-29T09:48:00.237Z|00017|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
2020-07-29T09:48:00.280Z|00018|dpdk|INFO|EAL: 99 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size
2020-07-29T09:48:00.281Z|00019|dpdk|INFO|EAL: Probing VFIO support...
2020-07-29T09:48:00.281Z|00020|dpdk|INFO|EAL: VFIO support initialized
2020-07-29T09:48:01.696Z|00021|dpdk|INFO|EAL: PCI device 0000:04:00.0 on NUMA socket 0
2020-07-29T09:48:01.696Z|00022|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe
2020-07-29T09:48:01.696Z|00023|dpdk|INFO|EAL: using IOMMU type 1 (Type 1)
2020-07-29T09:48:01.940Z|00024|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2020-07-29T09:48:02.285Z|00025|dpdk|INFO|EAL: PCI device 0000:42:00.0 on NUMA socket 1
2020-07-29T09:48:02.285Z|00026|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe
2020-07-29T09:48:02.412Z|00027|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2020-07-29T09:48:02.757Z|00028|dpdk|INFO|DPDK Enabled - initialized
Regards,
Balaji
-----Original Message-----
From: Burakov, Anatoly <anatoly.burakov@intel.com>
Sent: Wednesday, July 29, 2020 2:38 PM
To: David Marchand <david.marchand@redhat.com>; Balaji K N <balaji.k.n@ericsson.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] long initialization time for EAL init
On 29-Jul-20 9:52 AM, David Marchand wrote:
> On Wed, Jul 29, 2020 at 10:51 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
>> We are using OVS 2.12.0 with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization).
>>
>> Below is our hardware and memory configuration
>> Hardware : Dell Server
>> Sockets(NUMA) : 2
>> CPUs : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz)
>> RAM : 128 GB
>> Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process.
>> NIC : Ethernet 10G 2P X520 ( ixgbe driver )
>>
>> Passing below EAL arguments with OVS process.
>> 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
>>
>> We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init-> rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful.
>
> Did you try to start ovs with the --no-mlockall option?
>
>
That's most likely the culprit. I believe the recent mlock/MPROT changes are going to be backported to 18.11?
--
Thanks,
Anatoly
On Wed, Jul 29, 2020 at 11:56 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
>
> Hello,
>
> I tried by disabling mlockall and not do see long time for EAL init and is done quickly. Thanks for help.
>
> ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
Ok, just keep in mind that pages from the process are likely not
prefaulted and you could get random packet drops at the start of
packet handling.
You _might_ also get pages swapped out during the life of the process,
and again random packet drops.
We tried to enhance the DPDK init in this regard as Anatoly
mentionned, but this is not backported to 18.11 atm.
--
David Marchand