[dpdk-users] Dell PowerEdge CPU IERR caused by DPDK 16.07

[dpdk-users] Dell PowerEdge CPU IERR caused by DPDK 16.07

[Norbert Nyakó]

Hi,

We have two Dell R520 machines with the same configuration but
different Linux kernel versions [1][2]. I've compiled DPDK 16.07,
loaded the igb_uio and rte_kni kernel modules and configured the
machine to use the pmd for the 82598EB ethernet card.
As soon as I ran the testpipeline sample application on one of the
servers it stops responding  and on the LCD info screen and in the
logs CPU IERR error messages appear. After rebooting the error message
disappears. Does anyone have the similar problem? What could be the
root cause of this?

Thank you in advance!

Best regards,
Norbert

[1] Configuration of both machine

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Stepping:              4
CPU MHz:               2999.023
BogoMIPS:              5000.95
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31

02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720
Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720
Gigabit Ethernet PCIe
09:00.0 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AT
CX4 Network Connection (rev 01)
09:00.1 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AT
CX4 Network Connection (rev 01)
41:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719
Gigabit Ethernet PCIe (rev 01)
41:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719
Gigabit Ethernet PCIe (rev 01)
41:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719
Gigabit Ethernet PCIe (rev 01)
41:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719
Gigabit Ethernet PCIe (rev 01)

BIOS Information
        Vendor: Dell Inc.
        Version: 2.1.2
        Release Date: 01/20/2014
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 8192 kB
        BIOS Revision: 2.1

[2] Linux versions

Machine1:
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
Kernel version: 3.19.0-71-generic

Machine2:
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
Kernel version: 3.13.0-65-generic

Re: [dpdk-users] Dell PowerEdge CPU IERR caused by DPDK 16.07

[Anupam Kapoor]

On Mon, Nov 21, 2016 at 3:45 PM, Norbert Nyakó <norbert.nyako@gmail.com>
wrote:

> As soon as I ran the testpipeline sample application on one of the
> servers it stops responding  and on the LCD info screen and in the
> logs CPU IERR error messages appear
>

​difficult to *conclusively* conclude ​that this might be _caused_ by dpdk.

what is the error that you see (not just the fact that you saw a CPU-IERR)
? for some more information have a look here:
http://www.dell.com/support/article/in/en/indhs1/SLN292634. maybe reseating
memory modules (or some such) on the machine where you see this might fix
the issue ?

one laborious but possible approach to conclude that this is caused (rather
than just _triggered_) by dpdk would be to invert machine configuration and
then retest ?

​--
thanks
anupam
​

In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

Re: [dpdk-users] Dell PowerEdge CPU IERR caused by DPDK 16.07

[Norbert Nyakó]

Hi,

Thank you for your help!
There was also a ”bus fatal error" in the iDRAC / OMSA log and I also
looked at /var/log/* but there was no any suspicious log entry, I
think it because the machine rebooted as soon as the error triggered.
In my opinion it is not hardware fault because we have two machines
with the same configs and both behaves the same, but we tried with the
same network card with a different machine and there was no error.
Also we will try it with a different network card soon.
Since my first mail we updated the BIOS of the machines but it did not
help. Maybe it can be some firmware bug in the server which
incorrectly identifies CPU fault.

Best regards,
Norbert

2016-11-22 7:48 GMT+01:00 Anupam Kapoor <anupam.kapoor@gmail.com>:
>
> On Mon, Nov 21, 2016 at 3:45 PM, Norbert Nyakó <norbert.nyako@gmail.com>
> wrote:
>>
>> As soon as I ran the testpipeline sample application on one of the
>> servers it stops responding  and on the LCD info screen and in the
>> logs CPU IERR error messages appear
>
>
> difficult to *conclusively* conclude that this might be _caused_ by dpdk.
>
> what is the error that you see (not just the fact that you saw a CPU-IERR) ?
> for some more information have a look here:
> http://www.dell.com/support/article/in/en/indhs1/SLN292634. maybe reseating
> memory modules (or some such) on the machine where you see this might fix
> the issue ?
>
> one laborious but possible approach to conclude that this is caused (rather
> than just _triggered_) by dpdk would be to invert machine configuration and
> then retest ?
>
> --
> thanks
> anupam
>
> In the beginning was the lambda, and the lambda was with Emacs, and Emacs
> was the lambda.