DPDK 22.11.2 requires too much VIRT memory, how to reduce

["Lombardo, Ed" <Ed.Lombardo@netscout.com>]

[-- Attachment #1: Type: text/plain, Size: 2877 bytes --]

Hi,
I upgraded from DPDK 17.11 to DPDK 22.11.
VM host is running Oracle 9 with kernel 5.14.0-162 in VMWARE 7.03 and PCI Passthrough.

Was able to detect the NIC ethernet port with DPDK API (rte_eth_dev_count_avail()), however I see problems when do mmap in our application just after the DPDK initialization.

[root@vstreamOR9-160 bin]# /opt/dpdk/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:0b:00.0 'Ethernet Controller E810-XXV for SFP 159b' drv=igb_uio unused=ice

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio *Active*


Found errors when application is started with gdb:
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
[New Thread 0x7f60c8cb0640 (LWP 3872)]
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
[New Thread 0x7f60c84af640 (LWP 3873)]
EAL: Selected IOVA mode 'PA'
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
EAL: FATAL: Cannot init memory
EAL: Cannot init memory

This may be a hint why mmap fails?
Tried the following:

  *   increase the VM memory from 16 GB to 32 GB,
  *   increase the amount of hugepages from 2GB to 4GB,
  *   ulimit -n shows open files = 1024, had to increase to 2048 to eliminate the rte_mem_virt2phy() errors above when start app with gdb.
  *   free shows 62579808 free and available
  *   htop shows our application VIRT value is 66 GB.  What a surprise to see this value so high.  With DPDK 17.11 this value is around 5.5 GB.
  *   pmap -d <app PID> shows there are 4 [ anon ] of 16777216K each, which is what I believe is bumping the VIRT memory.

Was able to get our application to process packets from E810 once I changed the VM total memory to 80GB.
In HTOP the Mem value shows 5.28G/78.5G and VIRT is 69.0 GB.

When the main application executes rte_eal_init() is when the VIRT memory jumps to 66 GB, this is before the application configures the DPDK resources (worker rings, port configuration, etc).

Is there a setting in meson to reduce the DPDK resource in our application?

See the same VIRT memory issue on bare metal with same Oracle 9 and Intel I226 MAC.  I have systems that have only 16 GB of memory and a VM requiring 80 GB is not acceptable for our customers.


Any help will be appreciated.

Thanks,
Ed


[-- Attachment #2: Type: text/html, Size: 9191 bytes --]