* [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0
@ 2018-10-22 6:34 Kenvish Butani
2018-10-22 9:15 ` Burakov, Anatoly
0 siblings, 1 reply; 6+ messages in thread
From: Kenvish Butani @ 2018-10-22 6:34 UTC (permalink / raw)
To: dev
Hi All,
I am new to dpdk, and trying it to integrate DPDK with one of my server
having intel nic's and 24 core cpu.
I am trying to run testpmd app and it gets crashed with below logs
++++++++++++++++++++++++++++++++++++++
# ./testpmd-cc-on-local -c 0xF -n 1 -w 01:00.0 -- -i
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: PCI device igb_uio 0000:01:00.0: uio device registered with irq 44
0000:01:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10a7 net_e1000_igb
EAL: Error - exiting with code: 1
Cause: rte_zmalloc(32 struct rte_port) failed
++++++++++++++++++++++++++++++++++++++
I also tried by changing n=2 in testpmd commandline option but same crash
found.
Before running testpmd application here is the *script which setups the
environment*
++++++++++++++++++++++++++++++++++++++
/sbin/insmod
/lib/modules/4.14.30-ws-symbol/kernel/drivers/net/ethernet/intel/igb/igb.ko.lzma
/sbin/insmod /lib/modules/4.14.30-ws-symbol/kernel/drivers/uio/uio.ko.lzma
/sbin/insmod
/lib/modules/4.14.30-ws-symbol/kernel/drivers/uio/uio_pci_generic.ko.lzma
/sbin/insmod /lib/modules/4.14.30-ws-symbol/extra/dpdk/igb_uio.ko.lzma
/share/dpdk/usertools/dpdk-devbind.py --status
/share/dpdk/usertools/dpdk-devbind.py --bind=igb_uio eth0
/share/dpdk/usertools/dpdk-devbind.py --bind=igb_uio eth1
/share/dpdk/usertools/dpdk-devbind.py --status
echo 64 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
/bin/mknod /dev/uio0 c 247 0
/bin/chmod 777 /dev/uio0
+++++++++++++++++++++++++++++++++++++
Note : /dev/uio0 entry was not created after loading the driver and so i am
creating it manually with mknod.
*System config:*
++++++++++++++++++++++++++++++++++++
# /share/dpdk/usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'Device 10a7' drv=igb_uio unused=uio_pci_generic
0000:01:00.1 'Device 10a7' drv=igb_uio unused=uio_pci_generic
Network devices using kernel driver
===================================
<none>
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
#cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
stepping : 2
microcode : 0x15
cpu MHz : 1596.160
cache size : 12288 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts ret
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 4788.86
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
# cat /proc/meminfo
MemTotal: 36951260 kB
MemFree: 33182740 kB
MemAvailable: 33238132 kB
Buffers: 25632 kB
Cached: 263308 kB
SwapCached: 0 kB
Active: 1514816 kB
Inactive: 208784 kB
Active(anon): 1439404 kB
Inactive(anon): 1124 kB
Active(file): 75412 kB
Inactive(file): 207660 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 7823616 kB
SwapFree: 7823616 kB
Dirty: 160 kB
Writeback: 0 kB
AnonPages: 1434828 kB
Mapped: 70988 kB
Shmem: 5904 kB
Slab: 53732 kB
SReclaimable: 22184 kB
SUnreclaim: 31548 kB
KernelStack: 14304 kB
PageTables: 35768 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 26233708 kB
Committed_AS: 7762260 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HugePages_Total: 64
HugePages_Free: 63
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 79828 kB
DirectMap2M: 4048896 kB
DirectMap1G: 33554432 kB
# uname -a
Linux nx9000-6C883E 4.14.30-ws-symbol #1 SMP PREEMPT Thu Sep 20 15:04:33
IST 2018 x86_64 GNU/Linux
# lsmod
Module Size Used by
igb_uio 16384 0
uio_pci_generic 16384 0
uio 16384 2 igb_uio,uio_pci_generic
igb 143360 0
ipi_hsl 139264 7
dataplane 29323264 151 ipi_hsl
ipv6 303104 88 [permanent]
ftdi_sio 45056 0
usbserial 28672 1 ftdi_sio
xt_tcpudp 16384 0
xt_mark 16384 0
iptable_nat 16384 0
nf_nat_ipv4 16384 1 iptable_nat
nf_conntrack_ipv4 16384 1
nf_defrag_ipv4 16384 1 nf_conntrack_ipv4
nf_nat 24576 1 nf_nat_ipv4
xt_connlimit 16384 0
nf_conntrack 69632 4
nf_nat_ipv4,nf_conntrack_ipv4,nf_nat,xt_connlimit
iptable_filter 16384 0
ip_tables 20480 2 iptable_nat,iptable_filter
x_tables 20480 5
xt_tcpudp,xt_mark,xt_connlimit,iptable_filter,ip_tables
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
Can anyone help me out here to fix this. Please let me know if any more
information is required.
Thanks,
--Kenvish
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0
2018-10-22 6:34 [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0 Kenvish Butani
@ 2018-10-22 9:15 ` Burakov, Anatoly
2018-10-22 9:28 ` Kenvish Butani
0 siblings, 1 reply; 6+ messages in thread
From: Burakov, Anatoly @ 2018-10-22 9:15 UTC (permalink / raw)
To: Kenvish Butani, dev
On 22-Oct-18 7:34 AM, Kenvish Butani wrote:
> Hi All,
>
> I am new to dpdk, and trying it to integrate DPDK with one of my server
> having intel nic's and 24 core cpu.
>
> I am trying to run testpmd app and it gets crashed with below logs
>
> ++++++++++++++++++++++++++++++++++++++
> # ./testpmd-cc-on-local -c 0xF -n 1 -w 01:00.0 -- -i
> EAL: Detected 24 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Probing VFIO support...
> EAL: PCI device igb_uio 0000:01:00.0: uio device registered with irq 44
> 0000:01:00.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:10a7 net_e1000_igb
> EAL: Error - exiting with code: 1
> Cause: rte_zmalloc(32 struct rte_port) failed
> ++++++++++++++++++++++++++++++++++++++
Hi Kenvish,
The error looks like you don't have any memory available. Adding
--log-level=eal,8 will reveal more info on what is happening during
attempt at allocation time.
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0
2018-10-22 9:15 ` Burakov, Anatoly
@ 2018-10-22 9:28 ` Kenvish Butani
2018-10-22 9:46 ` Burakov, Anatoly
0 siblings, 1 reply; 6+ messages in thread
From: Kenvish Butani @ 2018-10-22 9:28 UTC (permalink / raw)
To: anatoly.burakov; +Cc: dev
Thanks a lot for your prompt reply !
JFYI, The invalid NUma socket was resolved by manually changing the entry
to 0
echo 0 > /sys/bus/pci/devices/0000\:01\:00.0/uio/uio0/device/numa_node
echo 0 > /sys/bus/pci/devices/0000\:01\:00.1/uio/uio1/device/numa_node
# ./testpmd-cc-on-local -c 0xF -n 1 --log-level=eal,8
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 1
EAL: Detected lcore 2 as core 1 on socket 0
EAL: Detected lcore 3 as core 1 on socket 1
EAL: Detected lcore 4 as core 2 on socket 0
EAL: Detected lcore 5 as core 2 on socket 1
EAigb_uio 0000:01:00.0: uio device registered with irq 44
L: Detected lcore 6 as core 8 on socket 0
EAL: Detected lcore 7 as core 8 on socket 1
EAL: Detected lcore 8 as core 9 on socket 0
EAL: Detected lcore 9 as core 9 on socket 1
EAL: Detected lcore 10 as core 10 on socket 0
EAL: Detected lcore 11 as core 10 on socket 1
EAL: Detected lcore 12 as core 0 on socket 0
EAL: Detected lcore 13 as core 0 on socket 1
EAL: Detected lcore 14 as core 1 on socket 0
EAL: Detected lcore 15 as core 1 on socket 1
EAL: Detected lcore 16 as core 2 on socket 0
EAL: Detected lcore 17 as core 2 on socket 1
EAL: Detected lcore 18 as core 8 on socket 0
EAL: Detected lcore 19 as core 8 on socket 1
EAL: Detected lcore 20 as core 9 on socket 0
EAL: Detected lcore 21 as core 9 on socket 1
EAL: Detected lcore 22 as core 10 on socket 0
EAL: Detected lcore 23 as core 10 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
directory)
EAL: VFIO PCI modules not loaded
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x7fdf72c49000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf72be8000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fdb70c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf72b87000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fd770a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf72b26000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fd370800000 (size = 0x400000000)
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf72ac5000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fcf70600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf70d30000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 1
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fcb70400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf70ccf000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 1
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fc770200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf70c6e000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 1
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fc370000000 (size = 0x400000000)
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x7fdf70c0d000 (size = 0x61000)
EAL: Memseg list allocated: 0x800kB at socket 1
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7fbf6fe00000 (size = 0x400000000)
EAL: TSC frequency is ~2394274 KHz
EAL: Master lcore 0 is ready (tid=72c7ec00;cpuset=[0])
EAL: lcore 1 is ready (tid=6f5fe700;cpuset=[1])
EAL: lcore 2 is ready (tid=6edfd700;cpuset=[2])
EAL: lcore 3 is ready (tid=6e5fc700;cpuset=[3])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 2MB
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:10a7 net_e1000_igb
EAL: PCI memory mapped at 0x7fdf72aa5000
EAL: PCI memory mapped at 0x7fdf72aa1igb_uio 0000:01:00.1: uio device
registered with irq 45
000
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:10a7 net_e1000_igb
EAL: PCI memory mapped at 0x7fdb70be0000
EAL: PCI memory mapped at 0x7fdf72a9d000
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: attempted to allocate 45 segments, but only 31 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 1
EAL: alloc_seg(): allocation happened on wrong socket (wanted 1, got 0)
EAL: attempted to allocate 45 segments, but only 32 were allocated
EAL: Restoring previous memory policy: 0
EAL: Error - exiting with code: 1
Cause: rte_zmalloc(32 struct rte_port) failed
Is vfio driver mandatory ? i see few error messages related to it.
Also, I see few error messages related to memory allocation, but I am not
sure how to fix this. Can you please help me out here.
Thanks,
--Kenvish
On Mon, Oct 22, 2018 at 2:45 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
> On 22-Oct-18 7:34 AM, Kenvish Butani wrote:
> > Hi All,
> >
> > I am new to dpdk, and trying it to integrate DPDK with one of my server
> > having intel nic's and 24 core cpu.
> >
> > I am trying to run testpmd app and it gets crashed with below logs
> >
> > ++++++++++++++++++++++++++++++++++++++
> > # ./testpmd-cc-on-local -c 0xF -n 1 -w 01:00.0 -- -i
> > EAL: Detected 24 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Probing VFIO support...
> > EAL: PCI device igb_uio 0000:01:00.0: uio device registered with irq 44
> > 0000:01:00.0 on NUMA socket -1
> > EAL: Invalid NUMA socket, default to 0
> > EAL: probe driver: 8086:10a7 net_e1000_igb
> > EAL: Error - exiting with code: 1
> > Cause: rte_zmalloc(32 struct rte_port) failed
> > ++++++++++++++++++++++++++++++++++++++
>
> Hi Kenvish,
>
> The error looks like you don't have any memory available. Adding
> --log-level=eal,8 will reveal more info on what is happening during
> attempt at allocation time.
>
> --
> Thanks,
> Anatoly
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0
2018-10-22 9:28 ` Kenvish Butani
@ 2018-10-22 9:46 ` Burakov, Anatoly
2018-10-22 11:08 ` Kenvish Butani
0 siblings, 1 reply; 6+ messages in thread
From: Burakov, Anatoly @ 2018-10-22 9:46 UTC (permalink / raw)
To: Kenvish Butani; +Cc: dev
On 22-Oct-18 10:28 AM, Kenvish Butani wrote:
> Thanks a lot for your prompt reply !
>
> JFYI, The invalid NUma socket was resolved by manually changing the
> entry to 0
>
> echo 0 > /sys/bus/pci/devices/0000\:01\:00.0/uio/uio0/device/numa_node
> echo 0 > /sys/bus/pci/devices/0000\:01\:00.1/uio/uio1/device/numa_node
>
> # ./testpmd-cc-on-local -c 0xF -n 1 --log-level=eal,8
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 0 on socket 1
> EAL: Detected lcore 2 as core 1 on socket 0
> EAL: Detected lcore 3 as core 1 on socket 1
> EAL: Detected lcore 4 as core 2 on socket 0
> EAL: Detected lcore 5 as core 2 on socket 1
> EAigb_uio 0000:01:00.0: uio device registered with irq 44
> L: Detected lcore 6 as core 8 on socket 0
> EAL: Detected lcore 7 as core 8 on socket 1
> EAL: Detected lcore 8 as core 9 on socket 0
> EAL: Detected lcore 9 as core 9 on socket 1
> EAL: Detected lcore 10 as core 10 on socket 0
> EAL: Detected lcore 11 as core 10 on socket 1
> EAL: Detected lcore 12 as core 0 on socket 0
> EAL: Detected lcore 13 as core 0 on socket 1
> EAL: Detected lcore 14 as core 1 on socket 0
> EAL: Detected lcore 15 as core 1 on socket 1
> EAL: Detected lcore 16 as core 2 on socket 0
> EAL: Detected lcore 17 as core 2 on socket 1
> EAL: Detected lcore 18 as core 8 on socket 0
> EAL: Detected lcore 19 as core 8 on socket 1
> EAL: Detected lcore 20 as core 9 on socket 0
> EAL: Detected lcore 21 as core 9 on socket 1
> EAL: Detected lcore 22 as core 10 on socket 0
> EAL: Detected lcore 23 as core 10 on socket 1
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 24 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: DPAA Bus not present. Skipping.
> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
> directory)
> EAL: VFIO PCI modules not loaded
> EAL: Probing VFIO support...
> EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
> EAL: VFIO modules not loaded, skipping VFIO support...
> EAL: Ask a virtual area of 0x2e000 bytes
> EAL: Virtual area found at 0x7fdf72c49000 (size = 0x2e000)
> EAL: Setting up physically contiguous memory...
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf72be8000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fdb70c00000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf72b87000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fd770a00000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf72b26000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fd370800000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf72ac5000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fcf70600000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf70d30000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 1
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fcb70400000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf70ccf000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 1
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fc770200000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf70c6e000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 1
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fc370000000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x61000 bytes
> EAL: Virtual area found at 0x7fdf70c0d000 (size = 0x61000)
> EAL: Memseg list allocated: 0x800kB at socket 1
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7fbf6fe00000 (size = 0x400000000)
> EAL: TSC frequency is ~2394274 KHz
> EAL: Master lcore 0 is ready (tid=72c7ec00;cpuset=[0])
> EAL: lcore 1 is ready (tid=6f5fe700;cpuset=[1])
> EAL: lcore 2 is ready (tid=6edfd700;cpuset=[2])
> EAL: lcore 3 is ready (tid=6e5fc700;cpuset=[3])
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 0
> EAL: Restoring previous memory policy: 0
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 2MB
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL: probe driver: 8086:10a7 net_e1000_igb
> EAL: PCI memory mapped at 0x7fdf72aa5000
> EAL: PCI memory mapped at 0x7fdf72aa1igb_uio 0000:01:00.1: uio device
> registered with irq 45
> 000
> EAL: PCI device 0000:01:00.1 on NUMA socket 0
> EAL: probe driver: 8086:10a7 net_e1000_igb
> EAL: PCI memory mapped at 0x7fdb70be0000
> EAL: PCI memory mapped at 0x7fdf72a9d000
> EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 0
> EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
> EAL: attempted to allocate 45 segments, but only 31 were allocated
> EAL: Restoring previous memory policy: 0
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1
> EAL: alloc_seg(): allocation happened on wrong socket (wanted 1, got 0)
> EAL: attempted to allocate 45 segments, but only 32 were allocated
> EAL: Restoring previous memory policy: 0
> EAL: Error - exiting with code: 1
> Cause: rte_zmalloc(32 struct rte_port) failed
This looks like you don't have enough memory. What probably happened is,
you've reserved some huge pages but they were spread across sockets, so
you don't have enough on any given socket, even though technically you
may have enough memory as far as total amount is concerned.
My advice would be to increase the total number of reserved hugepages.
>
>
> Is vfio driver mandatory ? i see few error messages related to it.
These errors are harmless.
>
> Also, I see few error messages related to memory allocation, but I am
> not sure how to fix this. Can you please help me out here.
>
> Thanks,
> --Kenvish
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0
2018-10-22 9:46 ` Burakov, Anatoly
@ 2018-10-22 11:08 ` Kenvish Butani
2018-10-22 11:19 ` Burakov, Anatoly
0 siblings, 1 reply; 6+ messages in thread
From: Kenvish Butani @ 2018-10-22 11:08 UTC (permalink / raw)
To: anatoly.burakov; +Cc: dev
Thanks a lot Anatoly for your inputs !
*This got fixed by reserving total of "512" hugepages.*
Thanks a lot :)
Appreciate.
Thanks,
--Kenvish
On Mon, Oct 22, 2018 at 3:16 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
> On 22-Oct-18 10:28 AM, Kenvish Butani wrote:
> > Thanks a lot for your prompt reply !
> >
> > JFYI, The invalid NUma socket was resolved by manually changing the
> > entry to 0
> >
> > echo 0 > /sys/bus/pci/devices/0000\:01\:00.0/uio/uio0/device/numa_node
> > echo 0 > /sys/bus/pci/devices/0000\:01\:00.1/uio/uio1/device/numa_node
> >
> > # ./testpmd-cc-on-local -c 0xF -n 1 --log-level=eal,8
> > EAL: Detected lcore 0 as core 0 on socket 0
> > EAL: Detected lcore 1 as core 0 on socket 1
> > EAL: Detected lcore 2 as core 1 on socket 0
> > EAL: Detected lcore 3 as core 1 on socket 1
> > EAL: Detected lcore 4 as core 2 on socket 0
> > EAL: Detected lcore 5 as core 2 on socket 1
> > EAigb_uio 0000:01:00.0: uio device registered with irq 44
> > L: Detected lcore 6 as core 8 on socket 0
> > EAL: Detected lcore 7 as core 8 on socket 1
> > EAL: Detected lcore 8 as core 9 on socket 0
> > EAL: Detected lcore 9 as core 9 on socket 1
> > EAL: Detected lcore 10 as core 10 on socket 0
> > EAL: Detected lcore 11 as core 10 on socket 1
> > EAL: Detected lcore 12 as core 0 on socket 0
> > EAL: Detected lcore 13 as core 0 on socket 1
> > EAL: Detected lcore 14 as core 1 on socket 0
> > EAL: Detected lcore 15 as core 1 on socket 1
> > EAL: Detected lcore 16 as core 2 on socket 0
> > EAL: Detected lcore 17 as core 2 on socket 1
> > EAL: Detected lcore 18 as core 8 on socket 0
> > EAL: Detected lcore 19 as core 8 on socket 1
> > EAL: Detected lcore 20 as core 9 on socket 0
> > EAL: Detected lcore 21 as core 9 on socket 1
> > EAL: Detected lcore 22 as core 10 on socket 0
> > EAL: Detected lcore 23 as core 10 on socket 1
> > EAL: Support maximum 128 logical core(s) by configuration.
> > EAL: Detected 24 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: DPAA Bus not present. Skipping.
> > EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
> > directory)
> > EAL: VFIO PCI modules not loaded
> > EAL: Probing VFIO support...
> > EAL: Module /sys/module/vfio not found! error 2 (No such file or
> directory)
> > EAL: VFIO modules not loaded, skipping VFIO support...
> > EAL: Ask a virtual area of 0x2e000 bytes
> > EAL: Virtual area found at 0x7fdf72c49000 (size = 0x2e000)
> > EAL: Setting up physically contiguous memory...
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72be8000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fdb70c00000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72b87000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fd770a00000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72b26000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fd370800000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72ac5000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fcf70600000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70d30000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fcb70400000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70ccf000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fc770200000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70c6e000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fc370000000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70c0d000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fbf6fe00000 (size = 0x400000000)
> > EAL: TSC frequency is ~2394274 KHz
> > EAL: Master lcore 0 is ready (tid=72c7ec00;cpuset=[0])
> > EAL: lcore 1 is ready (tid=6f5fe700;cpuset=[1])
> > EAL: lcore 2 is ready (tid=6edfd700;cpuset=[2])
> > EAL: lcore 3 is ready (tid=6e5fc700;cpuset=[3])
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 0
> > EAL: Restoring previous memory policy: 0
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 2MB
> > EAL: PCI device 0000:01:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:10a7 net_e1000_igb
> > EAL: PCI memory mapped at 0x7fdf72aa5000
> > EAL: PCI memory mapped at 0x7fdf72aa1igb_uio 0000:01:00.1: uio device
> > registered with irq 45
> > 000
> > EAL: PCI device 0000:01:00.1 on NUMA socket 0
> > EAL: probe driver: 8086:10a7 net_e1000_igb
> > EAL: PCI memory mapped at 0x7fdb70be0000
> > EAL: PCI memory mapped at 0x7fdf72a9d000
> > EAL: Module /sys/module/vfio not found! error 2 (No such file or
> directory)
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 0
> > EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
> > EAL: attempted to allocate 45 segments, but only 31 were allocated
> > EAL: Restoring previous memory policy: 0
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 1
> > EAL: alloc_seg(): allocation happened on wrong socket (wanted 1, got 0)
> > EAL: attempted to allocate 45 segments, but only 32 were allocated
> > EAL: Restoring previous memory policy: 0
> > EAL: Error - exiting with code: 1
> > Cause: rte_zmalloc(32 struct rte_port) failed
>
> This looks like you don't have enough memory. What probably happened is,
> you've reserved some huge pages but they were spread across sockets, so
> you don't have enough on any given socket, even though technically you
> may have enough memory as far as total amount is concerned.
>
> My advice would be to increase the total number of reserved hugepages.
>
> >
> >
> > Is vfio driver mandatory ? i see few error messages related to it.
>
> These errors are harmless.
>
> >
> > Also, I see few error messages related to memory allocation, but I am
> > not sure how to fix this. Can you please help me out here.
> >
> > Thanks,
> > --Kenvish
>
> --
> Thanks,
> Anatoly
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0
2018-10-22 11:08 ` Kenvish Butani
@ 2018-10-22 11:19 ` Burakov, Anatoly
0 siblings, 0 replies; 6+ messages in thread
From: Burakov, Anatoly @ 2018-10-22 11:19 UTC (permalink / raw)
To: Kenvish Butani; +Cc: dev
On 22-Oct-18 12:08 PM, Kenvish Butani wrote:
> Thanks a lot Anatoly for your inputs !
>
> *This got fixed by reserving total of "512" hugepages.*
>
> Thanks a lot :)
> Appreciate.
>
You're welcome :)
> Thanks,
> --Kenvish
>
> On Mon, Oct 22, 2018 at 3:16 PM Burakov, Anatoly
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>
> On 22-Oct-18 10:28 AM, Kenvish Butani wrote:
> > Thanks a lot for your prompt reply !
> >
> > JFYI, The invalid NUma socket was resolved by manually changing the
> > entry to 0
> >
> > echo 0 >
> /sys/bus/pci/devices/0000\:01\:00.0/uio/uio0/device/numa_node
> > echo 0 >
> /sys/bus/pci/devices/0000\:01\:00.1/uio/uio1/device/numa_node
> >
> > # ./testpmd-cc-on-local -c 0xF -n 1 --log-level=eal,8
> > EAL: Detected lcore 0 as core 0 on socket 0
> > EAL: Detected lcore 1 as core 0 on socket 1
> > EAL: Detected lcore 2 as core 1 on socket 0
> > EAL: Detected lcore 3 as core 1 on socket 1
> > EAL: Detected lcore 4 as core 2 on socket 0
> > EAL: Detected lcore 5 as core 2 on socket 1
> > EAigb_uio 0000:01:00.0: uio device registered with irq 44
> > L: Detected lcore 6 as core 8 on socket 0
> > EAL: Detected lcore 7 as core 8 on socket 1
> > EAL: Detected lcore 8 as core 9 on socket 0
> > EAL: Detected lcore 9 as core 9 on socket 1
> > EAL: Detected lcore 10 as core 10 on socket 0
> > EAL: Detected lcore 11 as core 10 on socket 1
> > EAL: Detected lcore 12 as core 0 on socket 0
> > EAL: Detected lcore 13 as core 0 on socket 1
> > EAL: Detected lcore 14 as core 1 on socket 0
> > EAL: Detected lcore 15 as core 1 on socket 1
> > EAL: Detected lcore 16 as core 2 on socket 0
> > EAL: Detected lcore 17 as core 2 on socket 1
> > EAL: Detected lcore 18 as core 8 on socket 0
> > EAL: Detected lcore 19 as core 8 on socket 1
> > EAL: Detected lcore 20 as core 9 on socket 0
> > EAL: Detected lcore 21 as core 9 on socket 1
> > EAL: Detected lcore 22 as core 10 on socket 0
> > EAL: Detected lcore 23 as core 10 on socket 1
> > EAL: Support maximum 128 logical core(s) by configuration.
> > EAL: Detected 24 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: DPAA Bus not present. Skipping.
> > EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
> > directory)
> > EAL: VFIO PCI modules not loaded
> > EAL: Probing VFIO support...
> > EAL: Module /sys/module/vfio not found! error 2 (No such file or
> directory)
> > EAL: VFIO modules not loaded, skipping VFIO support...
> > EAL: Ask a virtual area of 0x2e000 bytes
> > EAL: Virtual area found at 0x7fdf72c49000 (size = 0x2e000)
> > EAL: Setting up physically contiguous memory...
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72be8000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fdb70c00000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72b87000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fd770a00000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72b26000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fd370800000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf72ac5000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fcf70600000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70d30000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fcb70400000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70ccf000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fc770200000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70c6e000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fc370000000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x61000 bytes
> > EAL: Virtual area found at 0x7fdf70c0d000 (size = 0x61000)
> > EAL: Memseg list allocated: 0x800kB at socket 1
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0x7fbf6fe00000 (size = 0x400000000)
> > EAL: TSC frequency is ~2394274 KHz
> > EAL: Master lcore 0 is ready (tid=72c7ec00;cpuset=[0])
> > EAL: lcore 1 is ready (tid=6f5fe700;cpuset=[1])
> > EAL: lcore 2 is ready (tid=6edfd700;cpuset=[2])
> > EAL: lcore 3 is ready (tid=6e5fc700;cpuset=[3])
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 0
> > EAL: Restoring previous memory policy: 0
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 2MB
> > EAL: PCI device 0000:01:00.0 on NUMA socket 0
> > EAL: probe driver: 8086:10a7 net_e1000_igb
> > EAL: PCI memory mapped at 0x7fdf72aa5000
> > EAL: PCI memory mapped at 0x7fdf72aa1igb_uio 0000:01:00.1: uio
> device
> > registered with irq 45
> > 000
> > EAL: PCI device 0000:01:00.1 on NUMA socket 0
> > EAL: probe driver: 8086:10a7 net_e1000_igb
> > EAL: PCI memory mapped at 0x7fdb70be0000
> > EAL: PCI memory mapped at 0x7fdf72a9d000
> > EAL: Module /sys/module/vfio not found! error 2 (No such file or
> directory)
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 0
> > EAL: alloc_seg(): allocation happened on wrong socket (wanted 0,
> got 1)
> > EAL: attempted to allocate 45 segments, but only 31 were allocated
> > EAL: Restoring previous memory policy: 0
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 1
> > EAL: alloc_seg(): allocation happened on wrong socket (wanted 1,
> got 0)
> > EAL: attempted to allocate 45 segments, but only 32 were allocated
> > EAL: Restoring previous memory policy: 0
> > EAL: Error - exiting with code: 1
> > Cause: rte_zmalloc(32 struct rte_port) failed
>
> This looks like you don't have enough memory. What probably happened
> is,
> you've reserved some huge pages but they were spread across sockets, so
> you don't have enough on any given socket, even though technically you
> may have enough memory as far as total amount is concerned.
>
> My advice would be to increase the total number of reserved hugepages.
>
> >
> >
> > Is vfio driver mandatory ? i see few error messages related to it.
>
> These errors are harmless.
>
> >
> > Also, I see few error messages related to memory allocation, but
> I am
> > not sure how to fix this. Can you please help me out here.
> >
> > Thanks,
> > --Kenvish
>
> --
> Thanks,
> Anatoly
>
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-10-22 11:19 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-22 6:34 [dpdk-dev] testpmd crashes with Invalid NUMA socket, default to 0 Kenvish Butani
2018-10-22 9:15 ` Burakov, Anatoly
2018-10-22 9:28 ` Kenvish Butani
2018-10-22 9:46 ` Burakov, Anatoly
2018-10-22 11:08 ` Kenvish Butani
2018-10-22 11:19 ` Burakov, Anatoly
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).