[dpdk-users] Segfault while running on older CPU

[dpdk-users] Segfault while running on older CPU

[Filip Janiszewski]

Hi Everybody,

We have one 'slightly' older machine (well, very old CPU.) in our Lab
that seems to crash DPDK on every execution attempt, I was wondering if
anybody encountered a similar issue and if there's a change in the DPDK
config that might remedy the problem, this is the stack trace of the fault:

.
#0 0x000000000057bd19 in rte_cpu_get_flag_enabled ()
#1 0x000000000046067e in rte_acl_init ()
#2 0x000000000088359d in __libc_csu_init ()
#3 0x00007ffff6642cf5 in __libc_start_main () from /lib64/libc.so.6
#4 0x0000000000464fee in _start ()
.

The CPU on this machine is: "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
(fam: 06, model: 3a, stepping: 09)" running with "3.10.0-862.el7.x86_64".

Our builds are running fine on newest hardware, nevertheless the
segfault seems a bit weird even for an unsupported CPU (some error
prompt would be more friendly), any suggestion on what the problem might be?

Thanks

-- 
BR, Filip
+48 666 369 823

Re: [dpdk-users] Segfault while running on older CPU

[Filip Janiszewski]

Hi All,

We've just encountered the same issue on a new server with a couple of
Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, stack trace the same:

.
Program received signal SIGILL, Illegal instruction.

0x0000000000557ce9 in rte_cpu_get_flag_enabled ()

Missing separate debuginfos, use: debuginfo-install
glibc-2.17-260.el7.x86_64 libaio-0.3.109-13.el7.x86_64
libgcc-4.8.5-36.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64
numactl-libs-2.0.9-7.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-18.el7.x86_64

(gdb) bt

#0  0x0000000000557ce9 in rte_cpu_get_flag_enabled ()
#1  0x000000000046076e in rte_acl_init ()
#2  0x000000000085f56d in __libc_csu_init ()
#3  0x00007ffff6636365 in __libc_start_main () from /lib64/libc.so.6
#4  0x00000000004650de in _start ()
.

I'm building DPDK with x86_64-native-linuxapp-gcc, nothing else, plain
config from DPDK. I've attempted to recompile with CONFIG_RTE_EXEC_ENV
set to 'native' instead of 'linuxapp' but nothing changes.

Did anybody had a similar issue? Any suggestion?

Thanks

Il 06/02/19 11:47, Filip Janiszewski ha scritto:
> Hi Everybody,
> 
> We have one 'slightly' older machine (well, very old CPU.) in our Lab
> that seems to crash DPDK on every execution attempt, I was wondering if
> anybody encountered a similar issue and if there's a change in the DPDK
> config that might remedy the problem, this is the stack trace of the fault:
> 
> .
> #0 0x000000000057bd19 in rte_cpu_get_flag_enabled ()
> #1 0x000000000046067e in rte_acl_init ()
> #2 0x000000000088359d in __libc_csu_init ()
> #3 0x00007ffff6642cf5 in __libc_start_main () from /lib64/libc.so.6
> #4 0x0000000000464fee in _start ()
> .
> 
> The CPU on this machine is: "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
> (fam: 06, model: 3a, stepping: 09)" running with "3.10.0-862.el7.x86_64".
> 
> Our builds are running fine on newest hardware, nevertheless the
> segfault seems a bit weird even for an unsupported CPU (some error
> prompt would be more friendly), any suggestion on what the problem might be?
> 
> Thanks
> 

-- 
BR, Filip
+48 666 369 823

Re: [dpdk-users] Segfault while running on older CPU

[Wiles, Keith]

> On Feb 14, 2019, at 7:04 AM, Filip Janiszewski <contact@filipjaniszewski.com> wrote:
> 
> Hi All,
> 
> We've just encountered the same issue on a new server with a couple of
> Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, stack trace the same:
> 

What version of DPDK?
> .
> Program received signal SIGILL, Illegal instruction.
> 
> 0x0000000000557ce9 in rte_cpu_get_flag_enabled ()
> 
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-260.el7.x86_64 libaio-0.3.109-13.el7.x86_64
> libgcc-4.8.5-36.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64
> numactl-libs-2.0.9-7.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
> zlib-1.2.7-18.el7.x86_64
> 
> (gdb) bt
> 
> #0  0x0000000000557ce9 in rte_cpu_get_flag_enabled ()
> #1  0x000000000046076e in rte_acl_init ()
> #2  0x000000000085f56d in __libc_csu_init ()
> #3  0x00007ffff6636365 in __libc_start_main () from /lib64/libc.so.6
> #4  0x00000000004650de in _start ()
> .
> 
> I'm building DPDK with x86_64-native-linuxapp-gcc, nothing else, plain
> config from DPDK. I've attempted to recompile with CONFIG_RTE_EXEC_ENV
> set to 'native' instead of 'linuxapp' but nothing changes.

Not sure what this means it looks like you already compiled it as native.
> 
> Did anybody had a similar issue? Any suggestion?

maybe compile with  -g for debug symbols. i use ‘make install T=$RTE_TARGET EXTRA_CFLAGS=“-g”’  you can also try reducing the optimization, but I do not think that is the problem.

Does this happen every time, what OS version, what is the application yours or one of the examples?
If testpmd works then it maybe your application. I can not tell for sure but could this be the first time this routine is called?
Are you calling some thing in DPDK before rte_eal_init() is called?

With -g you should them be able to location the line in DPDK where it is failing.
> 
> Thanks
> 
> Il 06/02/19 11:47, Filip Janiszewski ha scritto:
>> Hi Everybody,
>> 
>> We have one 'slightly' older machine (well, very old CPU.) in our Lab
>> that seems to crash DPDK on every execution attempt, I was wondering if
>> anybody encountered a similar issue and if there's a change in the DPDK
>> config that might remedy the problem, this is the stack trace of the fault:
>> 
>> .
>> #0 0x000000000057bd19 in rte_cpu_get_flag_enabled ()
>> #1 0x000000000046067e in rte_acl_init ()
>> #2 0x000000000088359d in __libc_csu_init ()
>> #3 0x00007ffff6642cf5 in __libc_start_main () from /lib64/libc.so.6
>> #4 0x0000000000464fee in _start ()
>> .
>> 
>> The CPU on this machine is: "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
>> (fam: 06, model: 3a, stepping: 09)" running with "3.10.0-862.el7.x86_64".
>> 
>> Our builds are running fine on newest hardware, nevertheless the
>> segfault seems a bit weird even for an unsupported CPU (some error
>> prompt would be more friendly), any suggestion on what the problem might be?
>> 
>> Thanks
>> 
> 
> -- 
> BR, Filip
> +48 666 369 823

Regards,
Keith

Re: [dpdk-users] Segfault while running on older CPU

[Filip Janiszewski]

Hi,

I prepared a simple test application with only a call to rte_eal_init
and it crashes, DPDK version is 18.05 and apparently the problem is with
the BMI2 instruction set, the faulty line is:

.
 │0x57fec9 <rte_cpu_get_flag_enabled+89>  shrx   %eax,%edx,%eax
.

I'm attempting to disable those instructions by setting march to corei7
which according to gcc does not include BMI2, but still when opening the
asm list of the binary can see that instruction.

If I understand correctly how the DPDK building process works then I
should be able to set this 'corei7' switch by just changing
mk/machine/default/rte.vars.mk and then setting
CONFIG_RTE_MACHINE="default" in the x86_64-native-linuxapp-gcc/.config
file, but I'm not sure whether it's picking it up while building (make
T=x86_64-native-linuxapp-gcc DESTDIR=. -j28).

Il 14/02/19 14:34, Wiles, Keith ha scritto:
> 
> 
>> On Feb 14, 2019, at 7:04 AM, Filip Janiszewski <contact@filipjaniszewski.com> wrote:
>>
>> Hi All,
>>
>> We've just encountered the same issue on a new server with a couple of
>> Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, stack trace the same:
>>
> 
> What version of DPDK?
>> .
>> Program received signal SIGILL, Illegal instruction.
>>
>> 0x0000000000557ce9 in rte_cpu_get_flag_enabled ()
>>
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.17-260.el7.x86_64 libaio-0.3.109-13.el7.x86_64
>> libgcc-4.8.5-36.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64
>> numactl-libs-2.0.9-7.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
>> zlib-1.2.7-18.el7.x86_64
>>
>> (gdb) bt
>>
>> #0  0x0000000000557ce9 in rte_cpu_get_flag_enabled ()
>> #1  0x000000000046076e in rte_acl_init ()
>> #2  0x000000000085f56d in __libc_csu_init ()
>> #3  0x00007ffff6636365 in __libc_start_main () from /lib64/libc.so.6
>> #4  0x00000000004650de in _start ()
>> .
>>
>> I'm building DPDK with x86_64-native-linuxapp-gcc, nothing else, plain
>> config from DPDK. I've attempted to recompile with CONFIG_RTE_EXEC_ENV
>> set to 'native' instead of 'linuxapp' but nothing changes.
> 
> Not sure what this means it looks like you already compiled it as native.
>>
>> Did anybody had a similar issue? Any suggestion?
> 
> maybe compile with  -g for debug symbols. i use ‘make install T=$RTE_TARGET EXTRA_CFLAGS=“-g”’  you can also try reducing the optimization, but I do not think that is the problem.
> 
> Does this happen every time, what OS version, what is the application yours or one of the examples?
> If testpmd works then it maybe your application. I can not tell for sure but could this be the first time this routine is called?
> Are you calling some thing in DPDK before rte_eal_init() is called?
> 
> With -g you should them be able to location the line in DPDK where it is failing.
>>
>> Thanks
>>
>> Il 06/02/19 11:47, Filip Janiszewski ha scritto:
>>> Hi Everybody,
>>>
>>> We have one 'slightly' older machine (well, very old CPU.) in our Lab
>>> that seems to crash DPDK on every execution attempt, I was wondering if
>>> anybody encountered a similar issue and if there's a change in the DPDK
>>> config that might remedy the problem, this is the stack trace of the fault:
>>>
>>> .
>>> #0 0x000000000057bd19 in rte_cpu_get_flag_enabled ()
>>> #1 0x000000000046067e in rte_acl_init ()
>>> #2 0x000000000088359d in __libc_csu_init ()
>>> #3 0x00007ffff6642cf5 in __libc_start_main () from /lib64/libc.so.6
>>> #4 0x0000000000464fee in _start ()
>>> .
>>>
>>> The CPU on this machine is: "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
>>> (fam: 06, model: 3a, stepping: 09)" running with "3.10.0-862.el7.x86_64".
>>>
>>> Our builds are running fine on newest hardware, nevertheless the
>>> segfault seems a bit weird even for an unsupported CPU (some error
>>> prompt would be more friendly), any suggestion on what the problem might be?
>>>
>>> Thanks
>>>
>>
>> -- 
>> BR, Filip
>> +48 666 369 823
> 
> Regards,
> Keith
> 

-- 
BR, Filip
+48 666 369 823

Re: [dpdk-users] Segfault while running on older CPU

[Van Haaren, Harry]

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Filip Janiszewski
> Sent: Thursday, February 14, 2019 2:15 PM
> To: Wiles, Keith <keith.wiles@intel.com>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] Segfault while running on older CPU
> 
> Hi,
> 
> I prepared a simple test application with only a call to rte_eal_init
> and it crashes, DPDK version is 18.05 and apparently the problem is with
> the BMI2 instruction set, the faulty line is:
> 
> .
>  │0x57fec9 <rte_cpu_get_flag_enabled+89>  shrx   %eax,%edx,%eax
> .
> 
> I'm attempting to disable those instructions by setting march to corei7
> which according to gcc does not include BMI2, but still when opening the
> asm list of the binary can see that instruction.
> 
> If I understand correctly how the DPDK building process works then I
> should be able to set this 'corei7' switch by just changing
> mk/machine/default/rte.vars.mk and then setting
> CONFIG_RTE_MACHINE="default" in the x86_64-native-linuxapp-gcc/.config
> file, but I'm not sure whether it's picking it up while building (make
> T=x86_64-native-linuxapp-gcc DESTDIR=. -j28).


Possibly related crashing bugs;

DPDK rte memcpy / binutils 2.30:
https://bugs.dpdk.org/show_bug.cgi?id=97#c48

Gcc bug report;
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88096
And a related/duplicate: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86735


Please check if you're using binutils 2.30, and if so try latest DPDK code which has
merged a fix for disabling certain features as a workaround.

As Keith mentioned, what OS / Distro / Release, and what version of gcc / binutils is in use?


Regards, -Harry


<snip previous discussion>

Re: [dpdk-users] Segfault while running on older CPU

[Filip Janiszewski]

Il 14/02/19 15:46, Van Haaren, Harry ha scritto:
>> -----Original Message-----
>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Filip Janiszewski
>> Sent: Thursday, February 14, 2019 2:15 PM
>> To: Wiles, Keith <keith.wiles@intel.com>
>> Cc: users@dpdk.org
>> Subject: Re: [dpdk-users] Segfault while running on older CPU
>>
>> Hi,
>>
>> I prepared a simple test application with only a call to rte_eal_init
>> and it crashes, DPDK version is 18.05 and apparently the problem is with
>> the BMI2 instruction set, the faulty line is:
>>
>> .
>>  │0x57fec9 <rte_cpu_get_flag_enabled+89>  shrx   %eax,%edx,%eax
>> .
>>
>> I'm attempting to disable those instructions by setting march to corei7
>> which according to gcc does not include BMI2, but still when opening the
>> asm list of the binary can see that instruction.
>>
>> If I understand correctly how the DPDK building process works then I
>> should be able to set this 'corei7' switch by just changing
>> mk/machine/default/rte.vars.mk and then setting
>> CONFIG_RTE_MACHINE="default" in the x86_64-native-linuxapp-gcc/.config
>> file, but I'm not sure whether it's picking it up while building (make
>> T=x86_64-native-linuxapp-gcc DESTDIR=. -j28).
> 
> 
> Possibly related crashing bugs;
> 
> DPDK rte memcpy / binutils 2.30:
> https://bugs.dpdk.org/show_bug.cgi?id=97#c48
> 
> Gcc bug report;
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88096
> And a related/duplicate: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86735
> 
> 
> Please check if you're using binutils 2.30, and if so try latest DPDK code which has
> merged a fix for disabling certain features as a workaround.
> 

I'm running version 2.27, will try to upgrade to the latest DPDK code
and check if things are getting fixed.

> As Keith mentioned, what OS / Distro / Release, and what version of gcc / binutils is in use?
> 

OS CentOS 7.5.1804 with 3.10.0-862.14.4.el7.x86_64, gcc 4.8.5 (20150623)
and binutils 2.27.

Thanks.

> 
> Regards, -Harry
> 
> 
> <snip previous discussion>
> 

-- 
BR, Filip
+48 666 369 823