[dpdk-dev] testpmd crashes on startup

[dpdk-dev] testpmd crashes on startup

[Ophir Munk]

Hi Jianfeng,

Following commit 3d09a6e26d ("eal: fix threads block on barrier") I am seeing testpmd crashes on startups. 
The crashes occur 10% of the time with a regular compilation: 

make install T=x86_64-native-linuxapp-gcc 

and 70% of the time when using "-g" and "-o0" flags:

make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS+=-g EXTRA_CFLAGS+=-O0

config/common_base:
CONFIG_RTE_LIBRTE_MLX4_PMD=y

I am running with an mlx4 Mellanox NIC on an Azure VM.

Please note the crash back trace in [1].

Have you witnessed similar reports?
I can provide more details if needed.

[1]
Thread 1 "testpmd" received signal SIGABRT, Aborted.
0x00007ffff6ae4428 in GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007ffff6ae4428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff6ae602a in __GI_abort () at abort.c:89
#2 0x00007ffff6b267ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff6c3fed8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff6b2d6ed in malloc_printerr (ar_ptr=0x7ffff6e73b20 <main_arena>, ptr=<optimized out>, str=0x7ffff6c3cc92 "corrupted double-linked list", action=<optimized out>) at malloc.c:5006
#4 malloc_consolidate (av=av@entry=0x7ffff6e73b20 <main_arena>) at malloc.c:4183
#5 0x00007ffff6b30cde in _int_malloc (av=av@entry=0x7ffff6e73b20 <main_arena>, bytes=bytes@entry=1968) at malloc.c:3450
#6 0x00007ffff6b33184 in __GI_libc_malloc (bytes=1968) at malloc.c:2913
#7 0x000000000057a939 in pci_scan_one (dirname=0x7fffffffd120 "/sys/bus/pci/devices/0002:00:02.0", addr=0x7fffffffd100)
at ***/dpdk-branches/drivers/bus/pci/linux/pci.c:226
#8 0x000000000057b3d8 in rte_pci_scan () at ***/dpdk-branches/drivers/bus/pci/linux/pci.c:463
#9 0x0000000000554bce in rte_bus_scan () at ***/dpdk-branches/lib/librte_eal/common/eal_common_bus.c:77
#10 0x000000000053e3e8 in rte_eal_init (argc=17, argv=0x7fffffffe428) at ***/dpdk-branches/lib/librte_eal/linuxapp/eal/eal.c:783
#11 0x0000000000470279 in main (argc=17, argv=0x7fffffffe428) at ***/dpdk-branches/app/test-pmd/testpmd.c:2597

Re: [dpdk-dev] testpmd crashes on startup

[Ophir Munk]

The error printed with the crash:

*** Error in `/download/dpdk/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd': corrupted double-linked list: 0x00000000010f1200 ***

> -----Original Message-----
> From: Ophir Munk
> Sent: Friday, May 04, 2018 12:38 PM
> To: dev@dpdk.org; jianfeng.tan@intel.com; shreyansh.jain@nxp.com
> Cc: Ferruh Yigit <ferruh.yigit@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Olga Shern <olgas@mellanox.com>; Shahaf
> Shuler <shahafs@mellanox.com>
> Subject: testpmd crashes on startup
> 
> Hi Jianfeng,
> 
> Following commit 3d09a6e26d ("eal: fix threads block on barrier") I am
> seeing testpmd crashes on startups.
> The crashes occur 10% of the time with a regular compilation:
> 
> make install T=x86_64-native-linuxapp-gcc
> 
> and 70% of the time when using "-g" and "-o0" flags:
> 
> make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS+=-g
> EXTRA_CFLAGS+=-O0
> 
> config/common_base:
> CONFIG_RTE_LIBRTE_MLX4_PMD=y
> 
> I am running with an mlx4 Mellanox NIC on an Azure VM.
> 
> Please note the crash back trace in [1].
> 
> Have you witnessed similar reports?
> I can provide more details if needed.
> 
> [1]
> Thread 1 "testpmd" received signal SIGABRT, Aborted.
> 0x00007ffff6ae4428 in GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0 0x00007ffff6ae4428 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1 0x00007ffff6ae602a in __GI_abort () at abort.c:89
> #2 0x00007ffff6b267ea in __libc_message (do_abort=do_abort@entry=2,
> fmt=fmt@entry=0x7ffff6c3fed8 "*** Error in `%s': %s: 0x%s ***\n") at
> ../sysdeps/posix/libc_fatal.c:175
> #3 0x00007ffff6b2d6ed in malloc_printerr (ar_ptr=0x7ffff6e73b20
> <main_arena>, ptr=<optimized out>, str=0x7ffff6c3cc92 "corrupted double-
> linked list", action=<optimized out>) at malloc.c:5006
> #4 malloc_consolidate (av=av@entry=0x7ffff6e73b20 <main_arena>) at
> malloc.c:4183
> #5 0x00007ffff6b30cde in _int_malloc (av=av@entry=0x7ffff6e73b20
> <main_arena>, bytes=bytes@entry=1968) at malloc.c:3450
> #6 0x00007ffff6b33184 in __GI_libc_malloc (bytes=1968) at malloc.c:2913
> #7 0x000000000057a939 in pci_scan_one (dirname=0x7fffffffd120
> "/sys/bus/pci/devices/0002:00:02.0", addr=0x7fffffffd100) at ***/dpdk-
> branches/drivers/bus/pci/linux/pci.c:226
> #8 0x000000000057b3d8 in rte_pci_scan () at ***/dpdk-
> branches/drivers/bus/pci/linux/pci.c:463
> #9 0x0000000000554bce in rte_bus_scan () at ***/dpdk-
> branches/lib/librte_eal/common/eal_common_bus.c:77
> #10 0x000000000053e3e8 in rte_eal_init (argc=17, argv=0x7fffffffe428) at
> ***/dpdk-branches/lib/librte_eal/linuxapp/eal/eal.c:783
> #11 0x0000000000470279 in main (argc=17, argv=0x7fffffffe428) at
> ***/dpdk-branches/app/test-pmd/testpmd.c:2597