DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] long initialization time for EAL init
@ 2020-07-28 17:58 Anil Kumar Koli
  0 siblings, 0 replies; 6+ messages in thread
From: Anil Kumar Koli @ 2020-07-28 17:58 UTC (permalink / raw)
  To: dev; +Cc: Balaji K N

Hi Team,

 

Posting this query on behalf of Balaji as he was not able to send mail to
dpdk-dev.

 

We are using OVS 2.12.0  with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We
do observe long initialization time for EAL init ( ~10 seconds for EAL
memory initialization).

 

Below is our hardware and memory configuration

Hardware                      : Dell Server

Sockets(NUMA)           : 2

CPUs                              : Each NUMA with 10 CPUs (Intel(R) Xeon(R)
CPU E5-2660 v2 @ 2.20GHz)

RAM                              : 128 GB

Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G
(reserved during boot time). We are using only 2MB huge pages for OVS
process.

NIC                                  : Ethernet 10G 2P X520 ( ixgbe driver )

 

Passing below EAL arguments with OVS process.

2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w
0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1
-c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit
2048,2048.

 

We debugged further in DPDK and found out that mmap called in
eal_get_virtual_area is taking more time ( rte_eal_init->
rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init ->
alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is
noticeable difference compared to prior release. Any pointers would be
helpful.

 

EAL Init logs ( rte_eal_memseg_init function taking ~8 seconds )

2020-07-23T09:46:58.878Z|00008|dpdk|INFO|Using DPDK 18.11.2

2020-07-23T09:46:58.878Z|00009|dpdk|INFO|DPDK Enabled - initializing...

2020-07-23T09:46:58.878Z|00010|dpdk|INFO|No vhost-sock-dir provided -
defaulting to /var/run/openvswitch

2020-07-23T09:46:58.878Z|00011|dpdk|INFO|IOMMU support for vhost-user-client
disabled.

2020-07-23T09:46:58.878Z|00012|dpdk|INFO|POSTCOPY support for
vhost-user-client disabled.

2020-07-23T09:46:58.878Z|00013|dpdk|INFO|Per port memory for DPDK devices
disabled.

2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w
0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1
-c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit
2048,2048.

2020-07-23T09:46:58.882Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s)

2020-07-23T09:46:58.882Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes

2020-07-23T09:46:58.885Z|00017|dpdk|INFO|EAL: Multi-process socket
/var/run/dpdk/rte/mp_socket

2020-07-23T09:46:58.907Z|00018|dpdk|INFO|EAL: 100 hugepages of size
1073741824 reserved, but no mounted hugetlbfs found for that size

2020-07-23T09:46:58.907Z|00019|dpdk|INFO|EAL: Probing VFIO support...

2020-07-23T09:46:58.907Z|00020|dpdk|INFO|EAL: VFIO support initialized

2020-07-23T09:46:58.907Z|00021|dpdk|INFO|EAL: Ask a virtual area of 0x2e000
bytes

2020-07-23T09:46:58.907Z|00023|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:58.907Z|00024|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:58.907Z|00026|dpdk|INFO|EAL: Virtual area found at
0x100000000 (size = 0x2e000)

2020-07-23T09:46:58.907Z|00029|dpdk|INFO|EAL: Setting up physically
contiguous memory...

2020-07-23T09:46:58.907Z|00032|dpdk|INFO|EAL: Calling rte_eal_memseg_init...

2020-07-23T09:46:58.907Z|00034|dpdk|INFO|EAL: Setting maximum number of open
files to 65535

2020-07-23T09:46:58.907Z|00039|dpdk|INFO|EAL: Creating 4 segment lists:
n_segs:8192 socket_id:0 hugepage_sz:2097152

2020-07-23T09:46:58.907Z|00042|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:46:58.907Z|00044|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:58.907Z|00045|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:58.907Z|00047|dpdk|INFO|EAL: Virtual area found at
0x10002e000 (size = 0x61000)

2020-07-23T09:46:58.907Z|00053|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:46:58.907Z|00055|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:59.887Z|00056|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:59.887Z|00058|dpdk|INFO|EAL: Virtual area found at
0x100200000 (size = 0x400000000)

2020-07-23T09:46:59.888Z|00063|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:46:59.888Z|00065|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:46:59.888Z|00066|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:46:59.888Z|00068|dpdk|INFO|EAL: Virtual area found at
0x500200000 (size = 0x61000)

2020-07-23T09:46:59.888Z|00074|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:46:59.888Z|00076|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:00.824Z|00077|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:00.824Z|00079|dpdk|INFO|EAL: Virtual area found at
0x500400000 (size = 0x400000000)

2020-07-23T09:47:00.824Z|00084|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:47:00.824Z|00086|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:00.824Z|00087|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:00.824Z|00089|dpdk|INFO|EAL: Virtual area found at
0x900400000 (size = 0x61000)

2020-07-23T09:47:00.824Z|00095|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:00.824Z|00097|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:01.754Z|00098|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:01.754Z|00100|dpdk|INFO|EAL: Virtual area found at
0x900600000 (size = 0x400000000)

2020-07-23T09:47:01.754Z|00105|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:47:01.754Z|00107|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:01.754Z|00108|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:01.754Z|00110|dpdk|INFO|EAL: Virtual area found at
0xd00600000 (size = 0x61000)

2020-07-23T09:47:01.754Z|00116|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:01.754Z|00118|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:02.711Z|00119|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:02.711Z|00121|dpdk|INFO|EAL: Virtual area found at
0xd00800000 (size = 0x400000000)

2020-07-23T09:47:02.711Z|00124|dpdk|INFO|EAL: Creating 4 segment lists:
n_segs:8192 socket_id:1 hugepage_sz:2097152

2020-07-23T09:47:02.711Z|00127|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:47:02.711Z|00129|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:02.711Z|00130|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:02.711Z|00132|dpdk|INFO|EAL: Virtual area found at
0x1100800000 (size = 0x61000)

2020-07-23T09:47:02.711Z|00138|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:02.711Z|00140|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:03.641Z|00141|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:03.641Z|00143|dpdk|INFO|EAL: Virtual area found at
0x1100a00000 (size = 0x400000000)

2020-07-23T09:47:03.641Z|00148|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:47:03.641Z|00150|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:03.641Z|00151|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:03.641Z|00153|dpdk|INFO|EAL: Virtual area found at
0x1500a00000 (size = 0x61000)

2020-07-23T09:47:03.642Z|00159|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:03.642Z|00161|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:04.628Z|00162|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:04.628Z|00164|dpdk|INFO|EAL: Virtual area found at
0x1500c00000 (size = 0x400000000)

2020-07-23T09:47:04.629Z|00169|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:47:04.629Z|00171|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:04.629Z|00172|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:04.629Z|00174|dpdk|INFO|EAL: Virtual area found at
0x1900c00000 (size = 0x61000)

2020-07-23T09:47:04.629Z|00180|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:04.629Z|00182|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:05.733Z|00183|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:05.733Z|00185|dpdk|INFO|EAL: Virtual area found at
0x1900e00000 (size = 0x400000000)

2020-07-23T09:47:05.733Z|00190|dpdk|INFO|EAL: Ask a virtual area of 0x61000
bytes

2020-07-23T09:47:05.733Z|00192|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:05.733Z|00193|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:05.733Z|00195|dpdk|INFO|EAL: Virtual area found at
0x1d00e00000 (size = 0x61000)

2020-07-23T09:47:05.734Z|00201|dpdk|INFO|EAL: Ask a virtual area of
0x400000000 bytes

2020-07-23T09:47:05.734Z|00203|dpdk|INFO|EAL: Before mmap ...

2020-07-23T09:47:06.728Z|00204|dpdk|INFO|EAL: After mmap ...

2020-07-23T09:47:06.728Z|00206|dpdk|INFO|EAL: Virtual area found at
0x1d01000000 (size = 0x400000000)

2020-07-23T09:47:06.728Z|00210|dpdk|INFO|EAL: Done rte_eal_memseg_init...

2020-07-23T09:47:06.728Z|00213|dpdk|INFO|EAL: Calling
rte_eal_hugepage_init/rte_eal_hugepage_attach...

2020-07-23T09:47:08.115Z|00214|dpdk|INFO|EAL: Done
rte_eal_hugepage_init/rte_eal_hugepage_attach...

2020-07-23T09:47:08.237Z|00215|dpdk|INFO|EAL: PCI device 0000:04:00.0 on
NUMA socket 0

2020-07-23T09:47:08.237Z|00216|dpdk|INFO|EAL:   probe driver: 8086:154d
net_ixgbe

2020-07-23T09:47:08.237Z|00217|dpdk|INFO|EAL:   using IOMMU type 1 (Type 1)

2020-07-23T09:47:08.481Z|00218|dpdk|INFO|EAL: Ignore mapping IO port bar(2)

2020-07-23T09:47:08.831Z|00219|dpdk|INFO|EAL: PCI device 0000:42:00.0 on
NUMA socket 1

2020-07-23T09:47:08.831Z|00220|dpdk|INFO|EAL:   probe driver: 8086:154d
net_ixgbe

2020-07-23T09:47:08.957Z|00221|dpdk|INFO|EAL: Ignore mapping IO port bar(2)

2020-07-23T09:47:09.308Z|00222|dpdk|INFO|DPDK Enabled - initialized

 

Thanks & Regards,

Anil Kumar.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-dev] long initialization time for EAL init
  2020-07-29  9:56     ` Balaji K N
@ 2020-07-29 10:02       ` David Marchand
  0 siblings, 0 replies; 6+ messages in thread
From: David Marchand @ 2020-07-29 10:02 UTC (permalink / raw)
  To: Balaji K N; +Cc: Burakov, Anatoly, dev

On Wed, Jul 29, 2020 at 11:56 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
>
> Hello,
>
> I tried by disabling mlockall and not do see long time for EAL init and is done quickly. Thanks for help.
>
> ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach

Ok, just keep in mind that pages from the process are likely not
prefaulted and you could get random packet drops at the start of
packet handling.
You _might_ also get pages swapped out during the life of the process,
and again random packet drops.

We tried to enhance the DPDK init in this regard as Anatoly
mentionned, but this is not backported to 18.11 atm.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-dev] long initialization time for EAL init
  2020-07-29  9:07   ` Burakov, Anatoly
@ 2020-07-29  9:56     ` Balaji K N
  2020-07-29 10:02       ` David Marchand
  0 siblings, 1 reply; 6+ messages in thread
From: Balaji K N @ 2020-07-29  9:56 UTC (permalink / raw)
  To: Burakov, Anatoly, David Marchand; +Cc: dev

Hello,

I tried by disabling mlockall and not do see long time for EAL init and is done quickly. Thanks for help.

ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach

OVS logs
2020-07-29T09:48:00.233Z|00008|dpdk|INFO|Using DPDK 18.11.6
2020-07-29T09:48:00.233Z|00009|dpdk|INFO|DPDK Enabled - initializing...
2020-07-29T09:48:00.233Z|00010|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2020-07-29T09:48:00.233Z|00011|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2020-07-29T09:48:00.233Z|00012|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2020-07-29T09:48:00.233Z|00013|dpdk|INFO|Per port memory for DPDK devices disabled.
2020-07-29T09:48:00.233Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
2020-07-29T09:48:00.237Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s)
2020-07-29T09:48:00.237Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes
2020-07-29T09:48:00.237Z|00017|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
2020-07-29T09:48:00.280Z|00018|dpdk|INFO|EAL: 99 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size
2020-07-29T09:48:00.281Z|00019|dpdk|INFO|EAL: Probing VFIO support...
2020-07-29T09:48:00.281Z|00020|dpdk|INFO|EAL: VFIO support initialized
2020-07-29T09:48:01.696Z|00021|dpdk|INFO|EAL: PCI device 0000:04:00.0 on NUMA socket 0
2020-07-29T09:48:01.696Z|00022|dpdk|INFO|EAL:   probe driver: 8086:154d net_ixgbe
2020-07-29T09:48:01.696Z|00023|dpdk|INFO|EAL:   using IOMMU type 1 (Type 1)
2020-07-29T09:48:01.940Z|00024|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2020-07-29T09:48:02.285Z|00025|dpdk|INFO|EAL: PCI device 0000:42:00.0 on NUMA socket 1
2020-07-29T09:48:02.285Z|00026|dpdk|INFO|EAL:   probe driver: 8086:154d net_ixgbe
2020-07-29T09:48:02.412Z|00027|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2020-07-29T09:48:02.757Z|00028|dpdk|INFO|DPDK Enabled - initialized

Regards,
Balaji

-----Original Message-----
From: Burakov, Anatoly <anatoly.burakov@intel.com> 
Sent: Wednesday, July 29, 2020 2:38 PM
To: David Marchand <david.marchand@redhat.com>; Balaji K N <balaji.k.n@ericsson.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] long initialization time for EAL init

On 29-Jul-20 9:52 AM, David Marchand wrote:
> On Wed, Jul 29, 2020 at 10:51 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
>> We are using OVS 2.12.0  with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization).
>>
>> Below is our hardware and memory configuration
>> Hardware                      : Dell Server
>> Sockets(NUMA)           : 2
>> CPUs                              : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz)
>> RAM                              : 128 GB
>> Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process.
>> NIC                                  : Ethernet 10G 2P X520 ( ixgbe driver )
>>
>> Passing below EAL arguments with OVS process.
>> 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
>>
>> We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init->  rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful.
> 
> Did you try to start ovs with the --no-mlockall option?
> 
> 

That's most likely the culprit. I believe the recent mlock/MPROT changes are going to be backported to 18.11?

--
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-dev] long initialization time for EAL init
  2020-07-29  8:52 ` David Marchand
@ 2020-07-29  9:07   ` Burakov, Anatoly
  2020-07-29  9:56     ` Balaji K N
  0 siblings, 1 reply; 6+ messages in thread
From: Burakov, Anatoly @ 2020-07-29  9:07 UTC (permalink / raw)
  To: David Marchand, Balaji K N; +Cc: dev

On 29-Jul-20 9:52 AM, David Marchand wrote:
> On Wed, Jul 29, 2020 at 10:51 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
>> We are using OVS 2.12.0  with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization).
>>
>> Below is our hardware and memory configuration
>> Hardware                      : Dell Server
>> Sockets(NUMA)           : 2
>> CPUs                              : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz)
>> RAM                              : 128 GB
>> Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process.
>> NIC                                  : Ethernet 10G 2P X520 ( ixgbe driver )
>>
>> Passing below EAL arguments with OVS process.
>> 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
>>
>> We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init->  rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful.
> 
> Did you try to start ovs with the --no-mlockall option?
> 
> 

That's most likely the culprit. I believe the recent mlock/MPROT changes 
are going to be backported to 18.11?

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-dev] long initialization time for EAL init
  2020-07-28 15:56 Balaji K N
@ 2020-07-29  8:52 ` David Marchand
  2020-07-29  9:07   ` Burakov, Anatoly
  0 siblings, 1 reply; 6+ messages in thread
From: David Marchand @ 2020-07-29  8:52 UTC (permalink / raw)
  To: Balaji K N; +Cc: dev

On Wed, Jul 29, 2020 at 10:51 AM Balaji K N <balaji.k.n@ericsson.com> wrote:
> We are using OVS 2.12.0  with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization).
>
> Below is our hardware and memory configuration
> Hardware                      : Dell Server
> Sockets(NUMA)           : 2
> CPUs                              : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz)
> RAM                              : 128 GB
> Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process.
> NIC                                  : Ethernet 10G 2P X520 ( ixgbe driver )
>
> Passing below EAL arguments with OVS process.
> 2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
>
> We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init->  rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful.

Did you try to start ovs with the --no-mlockall option?


-- 
David Marchand


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [dpdk-dev] long initialization time for EAL init
@ 2020-07-28 15:56 Balaji K N
  2020-07-29  8:52 ` David Marchand
  0 siblings, 1 reply; 6+ messages in thread
From: Balaji K N @ 2020-07-28 15:56 UTC (permalink / raw)
  To: dev

Hello,

We are using OVS 2.12.0  with DPDK 18.11.2 release on Ubuntu 16.04 LTS. We do observe long initialization time for EAL init ( ~10 seconds for EAL memory initialization).

Below is our hardware and memory configuration
Hardware                      : Dell Server
Sockets(NUMA)           : 2
CPUs                              : Each NUMA with 10 CPUs (Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz)
RAM                              : 128 GB
Hugepages reserverd : 2048 huge pages of 2MB and 100 Huge pages of 1G (reserved during boot time). We are using only 2MB huge pages for OVS process.
NIC                                  : Ethernet 10G 2P X520 ( ixgbe driver )

Passing below EAL arguments with OVS process.
2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.

We debugged further in DPDK and found out that mmap called in eal_get_virtual_area is taking more time ( rte_eal_init->  rte_eal_memory_init -> rte_eal_memseg_init -> memseg_primary_init -> alloc_va_space -> eal_get_virtual_area ->mmap ). Long initialization time is noticeable difference compared to prior release. Any pointers would be helpful.

EAL Init logs ( rte_eal_memseg_init function taking ~8 seconds )
2020-07-23T09:46:58.878Z|00008|dpdk|INFO|Using DPDK 18.11.2
2020-07-23T09:46:58.878Z|00009|dpdk|INFO|DPDK Enabled - initializing...
2020-07-23T09:46:58.878Z|00010|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2020-07-23T09:46:58.878Z|00011|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2020-07-23T09:46:58.878Z|00012|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2020-07-23T09:46:58.878Z|00013|dpdk|INFO|Per port memory for DPDK devices disabled.
2020-07-23T09:46:58.878Z|00014|dpdk|INFO|EAL ARGS: ovs-vswitchd -w 0000:04:00.0,support-multi-driver=1 -w 0000:42:00.0,support-multi-driver=1 -c 0x1 --huge-dir /mnt/huge_ovs_2M --socket-mem 2048,2048 --socket-limit 2048,2048.
2020-07-23T09:46:58.882Z|00015|dpdk|INFO|EAL: Detected 40 lcore(s)
2020-07-23T09:46:58.882Z|00016|dpdk|INFO|EAL: Detected 2 NUMA nodes
2020-07-23T09:46:58.885Z|00017|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
2020-07-23T09:46:58.907Z|00018|dpdk|INFO|EAL: 100 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size
2020-07-23T09:46:58.907Z|00019|dpdk|INFO|EAL: Probing VFIO support...
2020-07-23T09:46:58.907Z|00020|dpdk|INFO|EAL: VFIO support initialized
2020-07-23T09:46:58.907Z|00021|dpdk|INFO|EAL: Ask a virtual area of 0x2e000 bytes
2020-07-23T09:46:58.907Z|00023|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:46:58.907Z|00024|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:46:58.907Z|00026|dpdk|INFO|EAL: Virtual area found at 0x100000000 (size = 0x2e000)
2020-07-23T09:46:58.907Z|00029|dpdk|INFO|EAL: Setting up physically contiguous memory...
2020-07-23T09:46:58.907Z|00032|dpdk|INFO|EAL: Calling rte_eal_memseg_init...
2020-07-23T09:46:58.907Z|00034|dpdk|INFO|EAL: Setting maximum number of open files to 65535
2020-07-23T09:46:58.907Z|00039|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
2020-07-23T09:46:58.907Z|00042|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:46:58.907Z|00044|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:46:58.907Z|00045|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:46:58.907Z|00047|dpdk|INFO|EAL: Virtual area found at 0x10002e000 (size = 0x61000)
2020-07-23T09:46:58.907Z|00053|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:46:58.907Z|00055|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:46:59.887Z|00056|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:46:59.887Z|00058|dpdk|INFO|EAL: Virtual area found at 0x100200000 (size = 0x400000000)
2020-07-23T09:46:59.888Z|00063|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:46:59.888Z|00065|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:46:59.888Z|00066|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:46:59.888Z|00068|dpdk|INFO|EAL: Virtual area found at 0x500200000 (size = 0x61000)
2020-07-23T09:46:59.888Z|00074|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:46:59.888Z|00076|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:00.824Z|00077|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:00.824Z|00079|dpdk|INFO|EAL: Virtual area found at 0x500400000 (size = 0x400000000)
2020-07-23T09:47:00.824Z|00084|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:47:00.824Z|00086|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:00.824Z|00087|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:00.824Z|00089|dpdk|INFO|EAL: Virtual area found at 0x900400000 (size = 0x61000)
2020-07-23T09:47:00.824Z|00095|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:47:00.824Z|00097|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:01.754Z|00098|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:01.754Z|00100|dpdk|INFO|EAL: Virtual area found at 0x900600000 (size = 0x400000000)
2020-07-23T09:47:01.754Z|00105|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:47:01.754Z|00107|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:01.754Z|00108|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:01.754Z|00110|dpdk|INFO|EAL: Virtual area found at 0xd00600000 (size = 0x61000)
2020-07-23T09:47:01.754Z|00116|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:47:01.754Z|00118|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:02.711Z|00119|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:02.711Z|00121|dpdk|INFO|EAL: Virtual area found at 0xd00800000 (size = 0x400000000)
2020-07-23T09:47:02.711Z|00124|dpdk|INFO|EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152
2020-07-23T09:47:02.711Z|00127|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:47:02.711Z|00129|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:02.711Z|00130|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:02.711Z|00132|dpdk|INFO|EAL: Virtual area found at 0x1100800000 (size = 0x61000)
2020-07-23T09:47:02.711Z|00138|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:47:02.711Z|00140|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:03.641Z|00141|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:03.641Z|00143|dpdk|INFO|EAL: Virtual area found at 0x1100a00000 (size = 0x400000000)
2020-07-23T09:47:03.641Z|00148|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:47:03.641Z|00150|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:03.641Z|00151|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:03.641Z|00153|dpdk|INFO|EAL: Virtual area found at 0x1500a00000 (size = 0x61000)
2020-07-23T09:47:03.642Z|00159|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:47:03.642Z|00161|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:04.628Z|00162|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:04.628Z|00164|dpdk|INFO|EAL: Virtual area found at 0x1500c00000 (size = 0x400000000)
2020-07-23T09:47:04.629Z|00169|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:47:04.629Z|00171|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:04.629Z|00172|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:04.629Z|00174|dpdk|INFO|EAL: Virtual area found at 0x1900c00000 (size = 0x61000)
2020-07-23T09:47:04.629Z|00180|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:47:04.629Z|00182|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:05.733Z|00183|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:05.733Z|00185|dpdk|INFO|EAL: Virtual area found at 0x1900e00000 (size = 0x400000000)
2020-07-23T09:47:05.733Z|00190|dpdk|INFO|EAL: Ask a virtual area of 0x61000 bytes
2020-07-23T09:47:05.733Z|00192|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:05.733Z|00193|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:05.733Z|00195|dpdk|INFO|EAL: Virtual area found at 0x1d00e00000 (size = 0x61000)
2020-07-23T09:47:05.734Z|00201|dpdk|INFO|EAL: Ask a virtual area of 0x400000000 bytes
2020-07-23T09:47:05.734Z|00203|dpdk|INFO|EAL: Before mmap ...
2020-07-23T09:47:06.728Z|00204|dpdk|INFO|EAL: After mmap ...
2020-07-23T09:47:06.728Z|00206|dpdk|INFO|EAL: Virtual area found at 0x1d01000000 (size = 0x400000000)
2020-07-23T09:47:06.728Z|00210|dpdk|INFO|EAL: Done rte_eal_memseg_init...
2020-07-23T09:47:06.728Z|00213|dpdk|INFO|EAL: Calling rte_eal_hugepage_init/rte_eal_hugepage_attach...
2020-07-23T09:47:08.115Z|00214|dpdk|INFO|EAL: Done rte_eal_hugepage_init/rte_eal_hugepage_attach...
2020-07-23T09:47:08.237Z|00215|dpdk|INFO|EAL: PCI device 0000:04:00.0 on NUMA socket 0
2020-07-23T09:47:08.237Z|00216|dpdk|INFO|EAL:   probe driver: 8086:154d net_ixgbe
2020-07-23T09:47:08.237Z|00217|dpdk|INFO|EAL:   using IOMMU type 1 (Type 1)
2020-07-23T09:47:08.481Z|00218|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2020-07-23T09:47:08.831Z|00219|dpdk|INFO|EAL: PCI device 0000:42:00.0 on NUMA socket 1
2020-07-23T09:47:08.831Z|00220|dpdk|INFO|EAL:   probe driver: 8086:154d net_ixgbe
2020-07-23T09:47:08.957Z|00221|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2020-07-23T09:47:09.308Z|00222|dpdk|INFO|DPDK Enabled - initialized

Regards,
Balaji


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-07-29 14:56 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-28 17:58 [dpdk-dev] long initialization time for EAL init Anil Kumar Koli
  -- strict thread matches above, loose matches on Subject: below --
2020-07-28 15:56 Balaji K N
2020-07-29  8:52 ` David Marchand
2020-07-29  9:07   ` Burakov, Anatoly
2020-07-29  9:56     ` Balaji K N
2020-07-29 10:02       ` David Marchand

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git