Issue setting up the DPDK development with non-privileged user

[Boris Ouretskey <borisusun@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 9579 bytes --]

Hi,

We are trying to set up a development environment for DPDK with non
privileged user. We are facing two issues which seem to be related (but
maybe not). The platform information appears at the end of the mail
(briefly: linux guest VM, on virtual box windows).

1) First issue is that after following the DPDK guide on running the
application with non privileged user and adding the capabilities as
advised, the ./dpdk-helloworld application still fails with EPERM.

-----------------
[user@dredd examples]$ sudo setcap cap_ipc_lock,cap_sys_admin+ep
./dpdk-helloworld
[user@dredd examples]$ ./dpdk-helloworld
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /run/user/1000/dpdk/rte/mp_socket
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not
available
EAL: Cannot use IOVA as 'PA' since physical addresses are not availableas
PANIC in main():
Cannot init EAL
5: [./dpdk-helloworld(_start+0x2e) [0x58ce4e]]
4: [/lib64/libc.so.6(__libc_start_main+0xf3) [0x7fe4265fbcf3]]
3: [./dpdk-helloworld(main+0x42) [0x58cf87]]
2: [./dpdk-helloworld(__rte_panic+0xdb) [0xa01c59]]
1: [./dpdk-helloworld(rte_dump_stack+0x27) [0xa2fcac]]
Aborted
-----------------

After adding all capabilities the application seems to be working when run
from command line, but of course it defies the goal of using the minimal
set of permission needed.

-----------------
[user@dredd examples]$ sudo setcap all+ep ./dpdk-helloworld
[user@dredd examples]$ ./dpdk-helloworld
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /run/user/1000/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_e1000_em (8086:100e) device: 0000:00:08.0
(socket 0)
-----------------

First question would be if DPDK guide is not updated and there's some
capabilities that need to be added in addition to what was mentioned in the
guide. Or something that we are doing wrong.

2) Second issue is a more weird one. When I am trying to debug the
application with gdb (or with strace) the application behaves differently
It seems to pass this time, the permission check (regardless if that's
"all" or specific one) but it fails within rte_mem_virt2phy function while
doing the following check.


/*
* the pfn (page frame number) are bits 0-54 (see
* pagemap.txt in linux Documentation)
*/
if ((page & 0x7fffffffffffffULL) == 0)
return RTE_BAD_IOVA;


---------------------------------
(gdb) r
Starting program:
/home/user/packages/dpdk/dpdk-22.03/build/examples/dpdk-helloworld
--huge-dir /mnt/huge/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
[New Thread 0x7fb88d7a9400 (LWP 12217)]
EAL: Multi-process socket /run/user/1000/dpdk/rte/mp_socket
[New Thread 0x7fb88cfa8400 (LWP 12218)]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not
available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
PANIC in main():
Cannot init EAL
---------------------------------

The question is why the behaviour is different between running from command
line and gdb, and what can cause the EAL to fail the check above on one
side but pass the permission check that does not pass when running from
command line ???

Thanks
===================================
The platform information
====================================
Linux Guest On Virtual Box (Nested Vt-x enabled)
----
DPDK Version = 22.03
---
[user@dredd examples]$ ../../usertools/dpdk-devbind.py  --status

Network devices using DPDK-compatible driver
============================================
0000:00:08.0 '82540EM Gigabit Ethernet Controller 100e' drv=vfio-pci
unused=e1000
0000:00:09.0 '82540EM Gigabit Ethernet Controller 100e' drv=vfio-pci
unused=e1000

Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller 100e' if=enp0s3 drv=e1000
unused=vfio-pci *Active*
---
[user@dredd examples]$ ../../usertools/dpdk-hugepages.py --show
Node Pages Size Total
0    10    2Mb    20Mb

Hugepages mounted on /dev/hugepages /mnt/huge
----
[user@dredd examples]$ uname -a
Linux dredd 4.18.0-348.el8.x86_64 #1 SMP Tue Nov 9 06:28:28 EST 2021 x86_64
x86_64 x86_64 GNU/Linux
---
[user@dredd examples]$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.5 (Arctic Sphynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.5 (Arctic Sphynx)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.5"
-------
[user@dredd examples]$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 165
model name      : Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz
stepping        : 2
cpu MHz         : 2303.998
cache size      : 16384 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni
pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave
avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single tpr_shadow
vnmi flexpriority vpid fsgsbase avx2 invpcid rdseed clflushopt md_clear
flush_l1d arch_capabilities
bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs
itlb_multihit
bogomips        : 4607.99
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 165
model name      : Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz
stepping        : 2
cpu MHz         : 2303.998
cache size      : 16384 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni
pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave
avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single tpr_shadow
vnmi flexpriority vpid fsgsbase avx2 invpcid rdseed clflushopt md_clear
flush_l1d arch_capabilities
bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs
itlb_multihit
bogomips        : 4607.99
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 165
model name      : Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz
stepping        : 2
cpu MHz         : 2303.998
cache size      : 16384 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni
pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave
avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single tpr_shadow
vnmi flexpriority vpid fsgsbase avx2 invpcid rdseed clflushopt md_clear
flush_l1d arch_capabilities
bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs
itlb_multihit
bogomips        : 4607.99
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 165
model name      : Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz
stepping        : 2
cpu MHz         : 2303.998
cache size      : 16384 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni
pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave
avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single tpr_shadow
vnmi flexpriority vpid fsgsbase avx2 invpcid rdseed clflushopt md_clear
flush_l1d arch_capabilities
bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs
itlb_multihit
bogomips        : 4607.99
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

[-- Attachment #2: Type: text/html, Size: 11001 bytes --]