From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <bruce.richardson@intel.com>
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20])
 by dpdk.org (Postfix) with ESMTP id 11EFC7F29
 for <dev@dpdk.org>; Fri,  7 Nov 2014 10:39:00 +0100 (CET)
Received: from orsmga002.jf.intel.com ([10.7.209.21])
 by orsmga101.jf.intel.com with ESMTP; 07 Nov 2014 01:48:31 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,331,1413270000"; d="scan'208";a="633201597"
Received: from bricha3-mobl3.ger.corp.intel.com ([10.243.20.32])
 by orsmga002.jf.intel.com with SMTP; 07 Nov 2014 01:48:29 -0800
Received: by  (sSMTP sendmail emulation); Fri, 07 Nov 2014 09:48:28 +0025
Date: Fri, 7 Nov 2014 09:48:28 +0000
From: Bruce Richardson <bruce.richardson@intel.com>
To: Kamraan Nasim <knasim@sidebandnetworks.com>
Message-ID: <20141107094828.GC4628@bricha3-MOBL3>
References: <CAPrTskijLw42woTkBvt_cYcaNKcyDsYVhj_zEpQ2n7DPMw1_Kg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAPrTskijLw42woTkBvt_cYcaNKcyDsYVhj_zEpQ2n7DPMw1_Kg@mail.gmail.com>
Organization: Intel Shannon Ltd.
User-Agent: Mutt/1.5.23 (2014-03-12)
Cc: dev@dpdk.org, Steve Noble <snoble@sidebandnetworks.com>
Subject: Re: [dpdk-dev] Panic in rte MEMPOOL__mempool_check_cookies()
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Nov 2014 09:39:01 -0000

On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> Greetings,
> 
> I have been hitting this issue fairly consistently for the ixgbe driver
> 
> MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0, cookie=badbadbadadd2e55
> PANIC in __mempool_check_cookies():
> MEMPOOL: bad header cookie (get)
> 
> It seems to be a corruption in the mempool bound to my ixgbe port. What I
> have observed is that this ONLY happens if I initialize dpdk(i.e. start
> dpdk application) AFTER traffic is already flowing in through the port. If
> I initialize dpdk and bind BEFORE I start traffic then things seem to work
> fine.
> 
> Any clues on why this might be happening?
> 
> A bit stumped, so would really appreciate all the help I can get on this
> one.
> 

Hi.
Can you confirm if your application works ok with the mempool debug options
disabled? The mempool debug options are not often used because they slow things
down so it's possible the bug could be in the cookie checking itself.

/Bruce

> Thanks,
> Kam
> 
> 
> (bt for your reference)
> 
> #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> "__mempool_check_cookies", format=
>     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> #4  rte_mempool_get_bulk (rxq=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> rx_pkts=0x900410,
>     nb_pkts=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0, rx_pkts=0x900410,
> nb_pkts=<value optimized out>)
>     at
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> bsz_rd=<value optimized out>, bsz_wr=
>     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368