[dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()

[dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()

[Kamraan Nasim]

Greetings,

I have been hitting this issue fairly consistently for the ixgbe driver

MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0, cookie=badbadbadadd2e55
PANIC in __mempool_check_cookies():
MEMPOOL: bad header cookie (get)

It seems to be a corruption in the mempool bound to my ixgbe port. What I
have observed is that this ONLY happens if I initialize dpdk(i.e. start
dpdk application) AFTER traffic is already flowing in through the port. If
I initialize dpdk and bind BEFORE I start traffic then things seem to work
fine.

Any clues on why this might be happening?

A bit stumped, so would really appreciate all the help I can get on this
one.

Thanks,
Kam


(bt for your reference)

#2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
"__mempool_check_cookies", format=
    0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
#3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
#4  rte_mempool_get_bulk (rxq=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
#5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
#6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
rx_pkts=0x900410,
    nb_pkts=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
#7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0, rx_pkts=0x900410,
nb_pkts=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
#8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
bsz_rd=<value optimized out>, bsz_wr=
    144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368

Re: [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()

[Bruce Richardson]

On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> Greetings,
> 
> I have been hitting this issue fairly consistently for the ixgbe driver
> 
> MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0, cookie=badbadbadadd2e55
> PANIC in __mempool_check_cookies():
> MEMPOOL: bad header cookie (get)
> 
> It seems to be a corruption in the mempool bound to my ixgbe port. What I
> have observed is that this ONLY happens if I initialize dpdk(i.e. start
> dpdk application) AFTER traffic is already flowing in through the port. If
> I initialize dpdk and bind BEFORE I start traffic then things seem to work
> fine.
> 
> Any clues on why this might be happening?
> 
> A bit stumped, so would really appreciate all the help I can get on this
> one.
> 

Hi.
Can you confirm if your application works ok with the mempool debug options
disabled? The mempool debug options are not often used because they slow things
down so it's possible the bug could be in the cookie checking itself.

/Bruce

> Thanks,
> Kam
> 
> 
> (bt for your reference)
> 
> #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> "__mempool_check_cookies", format=
>     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> #4  rte_mempool_get_bulk (rxq=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> rx_pkts=0x900410,
>     nb_pkts=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0, rx_pkts=0x900410,
> nb_pkts=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> bsz_rd=<value optimized out>, bsz_wr=
>     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368

Re: [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()

[Kamraan Nasim]

Hi Bruce,

The issue seems to be triggered by a combination of using the
CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG  option and contention in the mempool(small
mempool size).

I disabled that and the header panics went away.

Thank you for your help!

--Kam

On Fri, Nov 7, 2014 at 4:48 AM, Bruce Richardson <bruce.richardson@intel.com
> wrote:

> On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> > Greetings,
> >
> > I have been hitting this issue fairly consistently for the ixgbe driver
> >
> > MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0,
> cookie=badbadbadadd2e55
> > PANIC in __mempool_check_cookies():
> > MEMPOOL: bad header cookie (get)
> >
> > It seems to be a corruption in the mempool bound to my ixgbe port. What I
> > have observed is that this ONLY happens if I initialize dpdk(i.e. start
> > dpdk application) AFTER traffic is already flowing in through the port.
> If
> > I initialize dpdk and bind BEFORE I start traffic then things seem to
> work
> > fine.
> >
> > Any clues on why this might be happening?
> >
> > A bit stumped, so would really appreciate all the help I can get on this
> > one.
> >
>
> Hi.
> Can you confirm if your application works ok with the mempool debug options
> disabled? The mempool debug options are not often used because they slow
> things
> down so it's possible the bug could be in the cookie checking itself.
>
> /Bruce
>
> > Thanks,
> > Kam
> >
> >
> > (bt for your reference)
> >
> > #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> > "__mempool_check_cookies", format=
> >     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> > #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> > out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> > #4  rte_mempool_get_bulk (rxq=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> > #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> > #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> > rx_pkts=0x900410,
> >     nb_pkts=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> > #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0,
> rx_pkts=0x900410,
> > nb_pkts=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> > #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> > bsz_rd=<value optimized out>, bsz_wr=
> >     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368
>

Re: [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()

[Bruce Richardson]

On Wed, Nov 12, 2014 at 09:50:50PM -0500, Kamraan Nasim wrote:
> Hi Bruce,
> 
> The issue seems to be triggered by a combination of using the
> CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG  option and contention in the mempool(small
> mempool size).
> 
> I disabled that and the header panics went away.
> 
> Thank you for your help!

No problem, though it does mean that we have a bug in the mempool debug functions
that will have to be investigated and fixed in the future. :-(

> 
> --Kam
> 
> On Fri, Nov 7, 2014 at 4:48 AM, Bruce Richardson <bruce.richardson@intel.com
> > wrote:
> 
> > On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> > > Greetings,
> > >
> > > I have been hitting this issue fairly consistently for the ixgbe driver
> > >
> > > MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0,
> > cookie=badbadbadadd2e55
> > > PANIC in __mempool_check_cookies():
> > > MEMPOOL: bad header cookie (get)
> > >
> > > It seems to be a corruption in the mempool bound to my ixgbe port. What I
> > > have observed is that this ONLY happens if I initialize dpdk(i.e. start
> > > dpdk application) AFTER traffic is already flowing in through the port.
> > If
> > > I initialize dpdk and bind BEFORE I start traffic then things seem to
> > work
> > > fine.
> > >
> > > Any clues on why this might be happening?
> > >
> > > A bit stumped, so would really appreciate all the help I can get on this
> > > one.
> > >
> >
> > Hi.
> > Can you confirm if your application works ok with the mempool debug options
> > disabled? The mempool debug options are not often used because they slow
> > things
> > down so it's possible the bug could be in the cookie checking itself.
> >
> > /Bruce
> >
> > > Thanks,
> > > Kam
> > >
> > >
> > > (bt for your reference)
> > >
> > > #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> > > "__mempool_check_cookies", format=
> > >     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> > > #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> > > out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> > > #4  rte_mempool_get_bulk (rxq=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> > > #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> > > #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> > > rx_pkts=0x900410,
> > >     nb_pkts=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> > > #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0,
> > rx_pkts=0x900410,
> > > nb_pkts=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> > > #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> > > bsz_rd=<value optimized out>, bsz_wr=
> > >     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368
> >