DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()
@ 2014-11-06 21:28 Kamraan Nasim
  2014-11-07  9:48 ` Bruce Richardson
  0 siblings, 1 reply; 4+ messages in thread
From: Kamraan Nasim @ 2014-11-06 21:28 UTC (permalink / raw)
  To: dev; +Cc: Steve Noble

Greetings,

I have been hitting this issue fairly consistently for the ixgbe driver

MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0, cookie=badbadbadadd2e55
PANIC in __mempool_check_cookies():
MEMPOOL: bad header cookie (get)

It seems to be a corruption in the mempool bound to my ixgbe port. What I
have observed is that this ONLY happens if I initialize dpdk(i.e. start
dpdk application) AFTER traffic is already flowing in through the port. If
I initialize dpdk and bind BEFORE I start traffic then things seem to work
fine.

Any clues on why this might be happening?

A bit stumped, so would really appreciate all the help I can get on this
one.

Thanks,
Kam


(bt for your reference)

#2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
"__mempool_check_cookies", format=
    0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
#3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
#4  rte_mempool_get_bulk (rxq=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
#5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
#6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
rx_pkts=0x900410,
    nb_pkts=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
#7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0, rx_pkts=0x900410,
nb_pkts=<value optimized out>)
    at
/b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
#8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
bsz_rd=<value optimized out>, bsz_wr=
    144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()
  2014-11-06 21:28 [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies() Kamraan Nasim
@ 2014-11-07  9:48 ` Bruce Richardson
  2014-11-13  2:50   ` Kamraan Nasim
  0 siblings, 1 reply; 4+ messages in thread
From: Bruce Richardson @ 2014-11-07  9:48 UTC (permalink / raw)
  To: Kamraan Nasim; +Cc: dev, Steve Noble

On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> Greetings,
> 
> I have been hitting this issue fairly consistently for the ixgbe driver
> 
> MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0, cookie=badbadbadadd2e55
> PANIC in __mempool_check_cookies():
> MEMPOOL: bad header cookie (get)
> 
> It seems to be a corruption in the mempool bound to my ixgbe port. What I
> have observed is that this ONLY happens if I initialize dpdk(i.e. start
> dpdk application) AFTER traffic is already flowing in through the port. If
> I initialize dpdk and bind BEFORE I start traffic then things seem to work
> fine.
> 
> Any clues on why this might be happening?
> 
> A bit stumped, so would really appreciate all the help I can get on this
> one.
> 

Hi.
Can you confirm if your application works ok with the mempool debug options
disabled? The mempool debug options are not often used because they slow things
down so it's possible the bug could be in the cookie checking itself.

/Bruce

> Thanks,
> Kam
> 
> 
> (bt for your reference)
> 
> #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> "__mempool_check_cookies", format=
>     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> #4  rte_mempool_get_bulk (rxq=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> rx_pkts=0x900410,
>     nb_pkts=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0, rx_pkts=0x900410,
> nb_pkts=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> bsz_rd=<value optimized out>, bsz_wr=
>     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()
  2014-11-07  9:48 ` Bruce Richardson
@ 2014-11-13  2:50   ` Kamraan Nasim
  2014-11-13 12:59     ` Bruce Richardson
  0 siblings, 1 reply; 4+ messages in thread
From: Kamraan Nasim @ 2014-11-13  2:50 UTC (permalink / raw)
  To: Bruce Richardson; +Cc: dev

Hi Bruce,

The issue seems to be triggered by a combination of using the
CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG  option and contention in the mempool(small
mempool size).

I disabled that and the header panics went away.

Thank you for your help!

--Kam

On Fri, Nov 7, 2014 at 4:48 AM, Bruce Richardson <bruce.richardson@intel.com
> wrote:

> On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> > Greetings,
> >
> > I have been hitting this issue fairly consistently for the ixgbe driver
> >
> > MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0,
> cookie=badbadbadadd2e55
> > PANIC in __mempool_check_cookies():
> > MEMPOOL: bad header cookie (get)
> >
> > It seems to be a corruption in the mempool bound to my ixgbe port. What I
> > have observed is that this ONLY happens if I initialize dpdk(i.e. start
> > dpdk application) AFTER traffic is already flowing in through the port.
> If
> > I initialize dpdk and bind BEFORE I start traffic then things seem to
> work
> > fine.
> >
> > Any clues on why this might be happening?
> >
> > A bit stumped, so would really appreciate all the help I can get on this
> > one.
> >
>
> Hi.
> Can you confirm if your application works ok with the mempool debug options
> disabled? The mempool debug options are not often used because they slow
> things
> down so it's possible the bug could be in the cookie checking itself.
>
> /Bruce
>
> > Thanks,
> > Kam
> >
> >
> > (bt for your reference)
> >
> > #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> > "__mempool_check_cookies", format=
> >     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> > #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> > out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> > #4  rte_mempool_get_bulk (rxq=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> > #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> > #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> > rx_pkts=0x900410,
> >     nb_pkts=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> > #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0,
> rx_pkts=0x900410,
> > nb_pkts=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> > #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> > bsz_rd=<value optimized out>, bsz_wr=
> >     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368
>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()
  2014-11-13  2:50   ` Kamraan Nasim
@ 2014-11-13 12:59     ` Bruce Richardson
  0 siblings, 0 replies; 4+ messages in thread
From: Bruce Richardson @ 2014-11-13 12:59 UTC (permalink / raw)
  To: Kamraan Nasim; +Cc: dev

On Wed, Nov 12, 2014 at 09:50:50PM -0500, Kamraan Nasim wrote:
> Hi Bruce,
> 
> The issue seems to be triggered by a combination of using the
> CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG  option and contention in the mempool(small
> mempool size).
> 
> I disabled that and the header panics went away.
> 
> Thank you for your help!

No problem, though it does mean that we have a bug in the mempool debug functions
that will have to be investigated and fixed in the future. :-(

> 
> --Kam
> 
> On Fri, Nov 7, 2014 at 4:48 AM, Bruce Richardson <bruce.richardson@intel.com
> > wrote:
> 
> > On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> > > Greetings,
> > >
> > > I have been hitting this issue fairly consistently for the ixgbe driver
> > >
> > > MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0,
> > cookie=badbadbadadd2e55
> > > PANIC in __mempool_check_cookies():
> > > MEMPOOL: bad header cookie (get)
> > >
> > > It seems to be a corruption in the mempool bound to my ixgbe port. What I
> > > have observed is that this ONLY happens if I initialize dpdk(i.e. start
> > > dpdk application) AFTER traffic is already flowing in through the port.
> > If
> > > I initialize dpdk and bind BEFORE I start traffic then things seem to
> > work
> > > fine.
> > >
> > > Any clues on why this might be happening?
> > >
> > > A bit stumped, so would really appreciate all the help I can get on this
> > > one.
> > >
> >
> > Hi.
> > Can you confirm if your application works ok with the mempool debug options
> > disabled? The mempool debug options are not often used because they slow
> > things
> > down so it's possible the bug could be in the cookie checking itself.
> >
> > /Bruce
> >
> > > Thanks,
> > > Kam
> > >
> > >
> > > (bt for your reference)
> > >
> > > #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> > > "__mempool_check_cookies", format=
> > >     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> > > #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> > > out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> > > #4  rte_mempool_get_bulk (rxq=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> > > #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> > > #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> > > rx_pkts=0x900410,
> > >     nb_pkts=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> > > #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0,
> > rx_pkts=0x900410,
> > > nb_pkts=<value optimized out>)
> > >     at
> > >
> > /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> > > #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> > > bsz_rd=<value optimized out>, bsz_wr=
> > >     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368
> >

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2014-11-13 12:49 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-11-06 21:28 [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies() Kamraan Nasim
2014-11-07  9:48 ` Bruce Richardson
2014-11-13  2:50   ` Kamraan Nasim
2014-11-13 12:59     ` Bruce Richardson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).