DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Anil Kumar Koli" <anilkumar.k@altencalsoftlabs.com>
To: <dev@dpdk.org>
Cc: "'Balaji K N'" <balaji.k.n@ericsson.com>
Subject: [dpdk-dev] long initialization time for EAL init
Date: Tue, 28 Jul 2020 23:28:24 +0530
Message-ID: <00d501d66508$b0b6c4c0$12244e40$@altencalsoftlabs.com> (raw)

Hi Team,


Posting this query on behalf of Balaji as he was not able to send mail to


We are using OVS 2.12.0  with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We
do observe long initialization time for EAL init ( ~10 seconds for EAL
memory initialization).


Below is our hardware and memory configuration

Hardware                      : Dell Server

Sockets(NUMA)           : 2

CPUs                              : Each NUMA with 10 CPUs (Intel(R) Xeon(R)
CPU E5-2660 v2 @ 2.20GHz)

RAM                              : 128 GB

Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G
(reserved during boot time). We are using only 2MB huge pages for OVS

NIC                                  : Ethernet 10G 2P X520 ( ixgbe driver )


Passing below EAL arguments with OVS process.

2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w
0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1
-c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit


We debugged further in DPDK and found out that mmap called in
eal_get_virtual_area is taking more time ( rte_eal_init->
rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init ->
alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is
noticeable difference compared to prior release. Any pointers would be


EAL Init logs ( rte_eal_memseg_init function taking ~8 seconds )

2020-07-23T09:46:58.878Z|00008|dpdk|INFO|Using DPDK 18.11.2

2020-07-23T09:46:58.878Z|00009|dpdk|INFO|DPDK Enabled - initializing...

2020-07-23T09:46:58.878Z|00010|dpdk|INFO|No vhost-sock-dir provided -
defaulting to /var/run/openvswitch

2020-07-23T09:46:58.878Z|00011|dpdk|INFO|IOMMU support for vhost-user-client

2020-07-23T09:46:58.878Z|00012|dpdk|INFO|POSTCOPY support for
vhost-user-client disabled.

2020-07-23T09:46:58.878Z|00013|dpdk|INFO|Per port memory for DPDK devices

2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w
0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1
-c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit

2020-07-23T09:46:58.882Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s)

2020-07-23T09:46:58.882Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes

2020-07-23T09:46:58.885Z|00017|dpdk|INFO|EAL: Multi-process socket

2020-07-23T09:46:58.907Z|00018|dpdk|INFO|EAL: 100 hugepages of size
1073741824 reserved, but no mounted hugetlbfs found for that size

2020-07-23T09:46:58.907Z|00019|dpdk|INFO|EAL: Probing VFIO support...

2020-07-23T09:46:58.907Z|00020|dpdk|INFO|EAL: VFIO support initialized

2020-07-23T09:46:58.907Z|00021|dpdk|INFO|EAL: Ask a virtual area of 0x2e000

2020-07-23T09:46:58.907Z|00023|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:58.907Z|00024|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:58.907Z|00026|dpdk|INFO|EAL: Virtual area found at
0x100000000 (size = 0x2e000)

2020-07-23T09:46:58.907Z|00029|dpdk|INFO|EAL: Setting up physically
contiguous memory...

2020-07-23T09:46:58.907Z|00032|dpdk|INFO|EAL: Calling rte_eal_memseg_init...

2020-07-23T09:46:58.907Z|00034|dpdk|INFO|EAL: Setting maximum number of open
files to 65535

2020-07-23T09:46:58.907Z|00039|dpdk|INFO|EAL: Creating 4 segment lists:
n_segs:8192 socket_id:0 hugepage_sz:2097152

2020-07-23T09:46:58.907Z|00042|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:46:58.907Z|00044|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:58.907Z|00045|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:58.907Z|00047|dpdk|INFO|EAL: Virtual area found at
0x10002e000 (size = 0x61000)

2020-07-23T09:46:58.907Z|00053|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:46:58.907Z|00055|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:59.887Z|00056|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:59.887Z|00058|dpdk|INFO|EAL: Virtual area found at
0x100200000 (size = 0x400000000)

2020-07-23T09:46:59.888Z|00063|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:46:59.888Z|00065|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:59.888Z|00066|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:59.888Z|00068|dpdk|INFO|EAL: Virtual area found at
0x500200000 (size = 0x61000)

2020-07-23T09:46:59.888Z|00074|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:46:59.888Z|00076|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:00.824Z|00077|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:00.824Z|00079|dpdk|INFO|EAL: Virtual area found at
0x500400000 (size = 0x400000000)

2020-07-23T09:47:00.824Z|00084|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:47:00.824Z|00086|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:00.824Z|00087|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:00.824Z|00089|dpdk|INFO|EAL: Virtual area found at
0x900400000 (size = 0x61000)

2020-07-23T09:47:00.824Z|00095|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:00.824Z|00097|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:01.754Z|00098|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:01.754Z|00100|dpdk|INFO|EAL: Virtual area found at
0x900600000 (size = 0x400000000)

2020-07-23T09:47:01.754Z|00105|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:47:01.754Z|00107|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:01.754Z|00108|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:01.754Z|00110|dpdk|INFO|EAL: Virtual area found at
0xd00600000 (size = 0x61000)

2020-07-23T09:47:01.754Z|00116|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:01.754Z|00118|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:02.711Z|00119|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:02.711Z|00121|dpdk|INFO|EAL: Virtual area found at
0xd00800000 (size = 0x400000000)

2020-07-23T09:47:02.711Z|00124|dpdk|INFO|EAL: Creating 4 segment lists:
n_segs:8192 socket_id:1 hugepage_sz:2097152

2020-07-23T09:47:02.711Z|00127|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:47:02.711Z|00129|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:02.711Z|00130|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:02.711Z|00132|dpdk|INFO|EAL: Virtual area found at
0x1100800000 (size = 0x61000)

2020-07-23T09:47:02.711Z|00138|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:02.711Z|00140|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:03.641Z|00141|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:03.641Z|00143|dpdk|INFO|EAL: Virtual area found at
0x1100a00000 (size = 0x400000000)

2020-07-23T09:47:03.641Z|00148|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:47:03.641Z|00150|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:03.641Z|00151|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:03.641Z|00153|dpdk|INFO|EAL: Virtual area found at
0x1500a00000 (size = 0x61000)

2020-07-23T09:47:03.642Z|00159|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:03.642Z|00161|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:04.628Z|00162|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:04.628Z|00164|dpdk|INFO|EAL: Virtual area found at
0x1500c00000 (size = 0x400000000)

2020-07-23T09:47:04.629Z|00169|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:47:04.629Z|00171|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:04.629Z|00172|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:04.629Z|00174|dpdk|INFO|EAL: Virtual area found at
0x1900c00000 (size = 0x61000)

2020-07-23T09:47:04.629Z|00180|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:04.629Z|00182|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:05.733Z|00183|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:05.733Z|00185|dpdk|INFO|EAL: Virtual area found at
0x1900e00000 (size = 0x400000000)

2020-07-23T09:47:05.733Z|00190|dpdk|INFO|EAL: Ask a virtual area of 0x61000

2020-07-23T09:47:05.733Z|00192|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:05.733Z|00193|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:05.733Z|00195|dpdk|INFO|EAL: Virtual area found at
0x1d00e00000 (size = 0x61000)

2020-07-23T09:47:05.734Z|00201|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:05.734Z|00203|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:06.728Z|00204|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:06.728Z|00206|dpdk|INFO|EAL: Virtual area found at
0x1d01000000 (size = 0x400000000)

2020-07-23T09:47:06.728Z|00210|dpdk|INFO|EAL: Done rte_eal_memseg_init...

2020-07-23T09:47:06.728Z|00213|dpdk|INFO|EAL: Calling

2020-07-23T09:47:08.115Z|00214|dpdk|INFO|EAL: Done

2020-07-23T09:47:08.237Z|00215|dpdk|INFO|EAL: PCI device 0000:04:00.0 on
NUMA socket 0

2020-07-23T09:47:08.237Z|00216|dpdk|INFO|EAL:   probe driver: 8086:154d

2020-07-23T09:47:08.237Z|00217|dpdk|INFO|EAL:   using IOMMU type 1 (Type 1)

2020-07-23T09:47:08.481Z|00218|dpdk|INFO|EAL: Ignore mapping IO port bar(2)

2020-07-23T09:47:08.831Z|00219|dpdk|INFO|EAL: PCI device 0000:42:00.0 on
NUMA socket 1

2020-07-23T09:47:08.831Z|00220|dpdk|INFO|EAL:   probe driver: 8086:154d

2020-07-23T09:47:08.957Z|00221|dpdk|INFO|EAL: Ignore mapping IO port bar(2)

2020-07-23T09:47:09.308Z|00222|dpdk|INFO|DPDK Enabled - initialized


Thanks & Regards,

Anil Kumar.

             reply	other threads:[~2020-07-28 17:58 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-28 17:58 Anil Kumar Koli [this message]
  -- strict thread matches above, loose matches on Subject: below --
2020-07-28 15:56 Balaji K N
2020-07-29  8:52 ` David Marchand
2020-07-29  9:07   ` Burakov, Anatoly
2020-07-29  9:56     ` Balaji K N
2020-07-29 10:02       ` David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='00d501d66508$b0b6c4c0$12244e40$@altencalsoftlabs.com' \
    --to=anilkumar.k@altencalsoftlabs.com \
    --cc=balaji.k.n@ericsson.com \
    --cc=dev@dpdk.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git