DPDK 22.11.2 requires too much VIRT memory, how to reduce

DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Lombardo, Ed]

[-- Attachment #1: Type: text/plain, Size: 2877 bytes --]

Hi,
I upgraded from DPDK 17.11 to DPDK 22.11.
VM host is running Oracle 9 with kernel 5.14.0-162 in VMWARE 7.03 and PCI Passthrough.

Was able to detect the NIC ethernet port with DPDK API (rte_eth_dev_count_avail()), however I see problems when do mmap in our application just after the DPDK initialization.

[root@vstreamOR9-160 bin]# /opt/dpdk/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:0b:00.0 'Ethernet Controller E810-XXV for SFP 159b' drv=igb_uio unused=ice

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio *Active*


Found errors when application is started with gdb:
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
[New Thread 0x7f60c8cb0640 (LWP 3872)]
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
[New Thread 0x7f60c84af640 (LWP 3873)]
EAL: Selected IOVA mode 'PA'
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
EAL: FATAL: Cannot init memory
EAL: Cannot init memory

This may be a hint why mmap fails?
Tried the following:

  *   increase the VM memory from 16 GB to 32 GB,
  *   increase the amount of hugepages from 2GB to 4GB,
  *   ulimit -n shows open files = 1024, had to increase to 2048 to eliminate the rte_mem_virt2phy() errors above when start app with gdb.
  *   free shows 62579808 free and available
  *   htop shows our application VIRT value is 66 GB.  What a surprise to see this value so high.  With DPDK 17.11 this value is around 5.5 GB.
  *   pmap -d <app PID> shows there are 4 [ anon ] of 16777216K each, which is what I believe is bumping the VIRT memory.

Was able to get our application to process packets from E810 once I changed the VM total memory to 80GB.
In HTOP the Mem value shows 5.28G/78.5G and VIRT is 69.0 GB.

When the main application executes rte_eal_init() is when the VIRT memory jumps to 66 GB, this is before the application configures the DPDK resources (worker rings, port configuration, etc).

Is there a setting in meson to reduce the DPDK resource in our application?

See the same VIRT memory issue on bare metal with same Oracle 9 and Intel I226 MAC.  I have systems that have only 16 GB of memory and a VM requiring 80 GB is not acceptable for our customers.


Any help will be appreciated.

Thanks,
Ed


[-- Attachment #2: Type: text/html, Size: 9191 bytes --]

Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Dmitry Kozlyuk]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

Hi Ed,

DPDK indeed reserves a large amount of virtual address space at startup.
This does not prevent DPDK from running on low-memory systems, since this
reservation does not consume physical memory. So the large figure in VIRT
is not an issue. Hugepages are mapped on demand. Each mapped hugepage
consunes a file descriptor. If you use 2MB hugepages, the number of open
files can exhaust the limit easily. If you don't want to increase it,
please see:
https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#segment-file-descriptors

>

[-- Attachment #2: Type: text/html, Size: 1040 bytes --]

RE: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Lombardo, Ed]

[-- Attachment #1: Type: text/plain, Size: 3301 bytes --]

Hi Demitry,
Thank you for your reply.

Tried many things with no success.
I have tried 1G hugepage size x 2 pages and the VIRT value went up from 66.0 G to 128G.
I used DPDK 17.11 with 1024 Hugepages of size 2MB and did not see VIRT more than 5.5G.

With 1G hugepage size:
[root@vstreamOR9-160 ~]# ./dpdk-hugepages.py --show
Node Pages Size Total
0    2     1Gb    2Gb

Hugepages mounted on /dev/hugepages /mnt/huge

You can see the 16 G and 32 G mappings with 1G hugepage size.  Any additional mmap we do in our application fails for “OUT OF MEMORY (12)”
pmap -d <appPID>
0000000100035000       4 rw-s- 0000000000000000 000:00019 fbarray_memseg-1048576k-0-0
0000000140000000 1048576 rw-s- 0000000000000000 000:00027 rtemap_0
0000000180000000 1048576 rw-s- 0000000000000000 000:00027 rtemap_1
00000001c0000000 31457280 ----- 0000000000000000 000:00000   [ anon ]
0000000940000000       4 rw-s- 0000000000000000 000:00019 fbarray_memseg-1048576k-0-1
0000000980000000 33554432 ----- 0000000000000000 000:00000   [ anon ]
0000001180000000     388 rw-s- 0000000000000000 000:00019 fbarray_memseg-2048k-0-0
0000001180200000 16777216 ----- 0000000000000000 000:00000   [ anon ]
0000001580200000     388 rw-s- 0000000000000000 000:00019 fbarray_memseg-2048k-0-1
0000001580400000 16777216 ----- 0000000000000000 000:00000   [ anon ]
0000001980400000     388 rw-s- 0000000000000000 000:00019 fbarray_memseg-2048k-0-2
0000001980600000 16777216 ----- 0000000000000000 000:00000   [ anon ]
0000001d80600000     388 rw-s- 0000000000000000 000:00019 fbarray_memseg-2048k-0-3
0000001d80800000 16777216 ----- 0000000000000000 000:00000   [ anon ]

I tried dpdk_testpmd (# ./dpdk-testpmd -l 0-3  -n 1 -- -i --portmask=0x01) on my VM and the VIRT memory jumps to 66 GB with 2MB hugepage size and 1024 huge pages.

I also tried adding –single-file-segments in the EAL init argument.

When you say VIRT memory is not important then what would you suggest I focus on to determine the resource hog that DPDK 22.11 injects in our application?
Is there too many mmaps done by DPDK in 22.11?

Thanks,
Ed

From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Sent: Sunday, November 5, 2023 1:23 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: users <users@dpdk.org>
Subject: Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Ed,

DPDK indeed reserves a large amount of virtual address space at startup. This does not prevent DPDK from running on low-memory systems, since this reservation does not consume physical memory. So the large figure in VIRT is not an issue. Hugepages are mapped on demand. Each mapped hugepage consunes a file descriptor. If you use 2MB hugepages, the number of open files can exhaust the limit easily. If you don't want to increase it, please see: https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#segment-file-descriptors<https://urldefense.com/v3/__https:/doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html*segment-file-descriptors__;Iw!!Nzg7nt7_!C6krMpuvUwO2CGGJ-X-hjU7gIsnPFVpoBW1CyXcNJrZAWfOP_oODYN5YQ1v0y7yy2Ac2ej2uicsBFMNaZYgBTmXOiOM$>

[-- Attachment #2: Type: text/html, Size: 7746 bytes --]

Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Dmitry Kozlyuk]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

My suggestions to use 1G hugepages and/or --single-file-segments were to
counter "Too many open files" error, which you should observe no longer.
"Out of memory" is a new error. What are the failing mmap() call arguments?
For example, if it uses fixed addresses, it may clash with the area
reserved by DPDK (also affected by --base-virtaddr). DPDK 23.11 should
consume less RES memory at startup since it doesn't map it all but on
demand. Does --legacy-mem option (which mimics the old babevior) affect the
crash?

>

[-- Attachment #2: Type: text/html, Size: 843 bytes --]

RE: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Lombardo, Ed]

[-- Attachment #1.1: Type: text/plain, Size: 1078 bytes --]

Hi Dmitry,
I went to the dpdk.org and I don’t see DPDK 23.11,

[cid:image001.png@01DA1268.AA1AC1F0]

Do you have a pre-release version?

Thanks,
Ed

From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Sent: Wednesday, November 8, 2023 1:12 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: users <users@dpdk.org>
Subject: Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.
My suggestions to use 1G hugepages and/or --single-file-segments were to counter "Too many open files" error, which you should observe no longer. "Out of memory" is a new error. What are the failing mmap() call arguments? For example, if it uses fixed addresses, it may clash with the area reserved by DPDK (also affected by --base-virtaddr). DPDK 23.11 should consume less RES memory at startup since it doesn't map it all but on demand. Does --legacy-mem option (which mimics the old babevior) affect the crash?

[-- Attachment #1.2: Type: text/html, Size: 4086 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 25080 bytes --]

Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Dmitry Kozlyuk]

2023-11-08 22:26 (UTC+0000), Lombardo, Ed:
> Hi Dmitry,
> I went to the dpdk.org and I don’t see DPDK 23.11,
> 
> [cid:image001.png@01DA1268.AA1AC1F0]
> 
> Do you have a pre-release version?

Sorry for the typo, I meant v22.11.2 that you're testing.

RE: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Lombardo, Ed]

Hi Dmitry,
I finally finished testing all the options and found the VIRT value can be reduced from 66 GB to 16 GB with --legacy-mem setting in EAL init argument.  So I therefore had to increase the VM memory from 16 GB to 24 GB (instead of 80 GB without this setting).

I wonder what do we give up with this setting?

All the other settings I tried and combinations of these had no impact (socket-limit=2048, single-file-segments, no-shconf, and no-telemetry) on VIRT memory.

Thanks,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Wednesday, November 8, 2023 6:06 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: users <users@dpdk.org>
Subject: Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

2023-11-08 22:26 (UTC+0000), Lombardo, Ed:
> Hi Dmitry,
> I went to the dpdk.org and I don’t see DPDK 23.11,
> 
> [cid:image001.png@01DA1268.AA1AC1F0]
> 
> Do you have a pre-release version?

Sorry for the typo, I meant v22.11.2 that you're testing.

Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Dmitry Kozlyuk]

Hi Ed,

2023-11-10 00:16 (UTC+0000), Lombardo, Ed:
> I finally finished testing all the options and found the VIRT value can be
> reduced from 66 GB to 16 GB with --legacy-mem setting in EAL init argument.

Right.
By default, DPDK can use up to 64 GB of hugepage memory,
so it reserves 64 GB of VIRT (but does not map most of it),
RES should be low until the app actually allocates something.
In legacy mode, DPDK maps all available hugepage memory at startup,
in your case 16 GB, VIRT and RES should be close.

>  So I therefore had to increase the VM memory from 16 GB to 24 GB (instead
> of 80 GB without this setting).

I don't understand why you have to do that.
Possible VIRT is not limited by available RAM.
DPDK should be able to reserve 64 GB of VIRT on a machine with 16 GB of RAM,
it will just be unable to map more than 16 GB (obviously).

> I wonder what do we give up with this setting?
> 
> All the other settings I tried and combinations of these had no impact (socket-limit=2048, single-file-segments, no-shconf, and no-telemetry) on VIRT memory.
> 
> Thanks,
> Ed
> 
> -----Original Message-----
> From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
> Sent: Wednesday, November 8, 2023 6:06 PM
> To: Lombardo, Ed <Ed.Lombardo@netscout.com>
> Cc: users <users@dpdk.org>
> Subject: Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce
> 
> External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 2023-11-08 22:26 (UTC+0000), Lombardo, Ed:
> > Hi Dmitry,
> > I went to the dpdk.org and I don’t see DPDK 23.11,
> > 
> > [cid:image001.png@01DA1268.AA1AC1F0]
> > 
> > Do you have a pre-release version?  
> 
> Sorry for the typo, I meant v22.11.2 that you're testing.
> 

Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Dmitry Kozlyuk]

2023-11-10 12:31 (UTC+0300), Dmitry Kozlyuk:
> Hi Ed,
> 
> 2023-11-10 00:16 (UTC+0000), Lombardo, Ed:
> > I finally finished testing all the options and found the VIRT value can be
> > reduced from 66 GB to 16 GB with --legacy-mem setting in EAL init argument.  
> 
> Right.
> By default, DPDK can use up to 64 GB of hugepage memory,
> so it reserves 64 GB of VIRT (but does not map most of it),
> RES should be low until the app actually allocates something.
> In legacy mode, DPDK maps all available hugepage memory at startup,
> in your case 16 GB, VIRT and RES should be close.
> 
> >  So I therefore had to increase the VM memory from 16 GB to 24 GB (instead
> > of 80 GB without this setting).  
> 
> I don't understand why you have to do that.
> Possible VIRT is not limited by available RAM.
> DPDK should be able to reserve 64 GB of VIRT on a machine with 16 GB of RAM,
> it will just be unable to map more than 16 GB (obviously).

Sorry, I've sent the message early by mistake.

> > I wonder what do we give up with this setting?

Most importantly, in legacy mode DPDK will consume all available hugepages at
startup and will not free them back to the system until the all is terminated.
The default dynamic mode allocates and frees physical RAM on demand.
Some advanced DPDK memory API don't work in legacy mode.

> > 
> > All the other settings I tried and combinations of these had no impact
> > (socket-limit=2048, single-file-segments, no-shconf, and no-telemetry) on
> > VIRT memory.

Right, they should not.
DPDK assumes that VIRT reservation is almost free and unlimited.
May it be that your system somehow limits it?

RE: DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Lombardo, Ed]

Hi,
I am revisiting this issue I have been living with.  The workaround I am using is to not memlock memory in our application.  
Error is "VIRT memory is too high and mmap fails, Cannot allocate memory (12)."

I tried DPDK 23.11 and I see the same issue I see with DPDK 22.11.  
In meson I tried setting -Db_lto=true -Dbuildtype=minsize but did not help my issue.

We have only 16 GB of memory and I setup 2x1GB=2GB hugepage size, (legacy mode EAL setting).
Also running Oracle91 kernel 5.14.0-284.

I see same the issue on virtual VM on Intel host, and on hardware bare metal with Atom processor.  (Both running Oracle91)

Can I reduce VIRT memory if I switch from static libraries to shared libraries?

Any help will be greatly appreciated.

Thanks,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Friday, November 10, 2023 4:38 AM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: users <users@dpdk.org>
Subject: Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

2023-11-10 12:31 (UTC+0300), Dmitry Kozlyuk:
> Hi Ed,
> 
> 2023-11-10 00:16 (UTC+0000), Lombardo, Ed:
> > I finally finished testing all the options and found the VIRT value 
> > can be reduced from 66 GB to 16 GB with --legacy-mem setting in EAL init argument.
> 
> Right.
> By default, DPDK can use up to 64 GB of hugepage memory, so it 
> reserves 64 GB of VIRT (but does not map most of it), RES should be 
> low until the app actually allocates something.
> In legacy mode, DPDK maps all available hugepage memory at startup, in 
> your case 16 GB, VIRT and RES should be close.
> 
> >  So I therefore had to increase the VM memory from 16 GB to 24 GB 
> > (instead of 80 GB without this setting).
> 
> I don't understand why you have to do that.
> Possible VIRT is not limited by available RAM.
> DPDK should be able to reserve 64 GB of VIRT on a machine with 16 GB 
> of RAM, it will just be unable to map more than 16 GB (obviously).

Sorry, I've sent the message early by mistake.

> > I wonder what do we give up with this setting?

Most importantly, in legacy mode DPDK will consume all available hugepages at startup and will not free them back to the system until the all is terminated.
The default dynamic mode allocates and frees physical RAM on demand.
Some advanced DPDK memory API don't work in legacy mode.

> > 
> > All the other settings I tried and combinations of these had no 
> > impact (socket-limit=2048, single-file-segments, no-shconf, and 
> > no-telemetry) on VIRT memory.

Right, they should not.
DPDK assumes that VIRT reservation is almost free and unlimited.
May it be that your system somehow limits it?