[dpdk-users] DPDK 16.7 with Intel i350 igb, panic, rte_mbuf_raw_alloc

[dpdk-users] DPDK 16.7 with Intel i350 igb, panic, rte_mbuf_raw_alloc

[martin_curran-gray]

Hi,

Having got my X710/i40e working to some extent in 16.7 (moving from 2.2.0), I was doing some testing with i350, igb 

I seem to get a panic if I bring the link up while dpdk is running

I don't get this with my X710
I don't get this with the i350 with 2.2.0

It seems to happen both 
	if the link is up, I take it down, then bring it up  ( by issuing ifconfig eth# down etc on the other (non dpdk) end of the link)
or
	the link is down when dpdk start up, then I bring it up


had a look in the release notes for 16.04 and 16.07 but nothing is really springing out at me as something I have to change?


Run time shell output from my app

dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
PANIC in rte_mbuf_raw_alloc():
line 1153       assert "rte_mbuf_refcnt_read(m) == 0" failed


gdb looking at the core

Program terminated with signal 6, Aborted.
#0  0x0000003488c328a5 in raise () from /lib64/libc.so.6
 (gdb) bt
#0  0x0000003488c328a5 in raise () from /lib64/libc.so.6
#1  0x0000003488c34085 in abort () from /lib64/libc.so.6
#2  0x00007fe927e9f527 in __rte_panic (funcname=<value optimized out>,
    format=0x7fe927ffe690 "line %d\tassert \"rte_mbuf_refcnt_read(m) == 0\" failed\n%.0s")
    at /root/xxxxxx/dpdk-16.07/lib/librte_eal/linuxapp/eal/eal_debug.c:86
#3  0x00007fe927f56e76 in rte_mbuf_raw_alloc (rx_queue=0x7fe8fa6caa00, rx_pkts=0x7fe8d65fc190,
    nb_pkts=32) at /root/xxxxxx/dpdk-16.07/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1153
#4  eth_igb_recv_pkts (rx_queue=0x7fe8fa6caa00, rx_pkts=0x7fe8d65fc190, nb_pkts=32)
    at /root/xxxxxx/dpdk-16.07/drivers/net/e1000/igb_rxtx.c:832
#5  0x00007fe927f1ebef in rte_eth_rx_burst () from /opt/EMRFeed/lib/libIssDPDKDriverInterface.so
#6  0x00007fe927f2e684 in dpdk_receive () from /opt/EMRFeed/lib/libIssDPDKDriverInterface.so
#7  0x00007fe927f3d023 in eal_thread_loop (arg=<value optimized out>)
    at /root/xxxxxx/dpdk-16.07/lib/librte_eal/linuxapp/eal/eal_thread.c:184
#8  0x0000003489007851 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003488ce890d in clone () from /lib64/libc.so.6
(gdb)


Anyone come across this or know what I'm missing doing?

thanks

Martin Curran-Gray
HW/FPGA/SW Engineer

Re: [dpdk-users] DPDK 16.7 with Intel i350 igb, panic, rte_mbuf_raw_alloc

[Olivier Matz]

Hi Martin,

On 08/26/2016 10:24 AM, martin_curran-gray at keysight.com
(martin_curran-gray at keysight.com) wrote:
> Hi,
> 
> Having got my X710/i40e working to some extent in 16.7 (moving from 2.2.0), I was doing some testing with i350, igb 
> 
> I seem to get a panic if I bring the link up while dpdk is running
> 
> I don't get this with my X710
> I don't get this with the i350 with 2.2.0
> 
> It seems to happen both 
> 	if the link is up, I take it down, then bring it up  ( by issuing ifconfig eth# down etc on the other (non dpdk) end of the link)
> or
> 	the link is down when dpdk start up, then I bring it up
> 
> 
> had a look in the release notes for 16.04 and 16.07 but nothing is really springing out at me as something I have to change?
> 
> 
> Run time shell output from my app
> 
> dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
> dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
> dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
> dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
> dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
> dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
> dpdk: Get Link Status port 0,  status is 0, speed is 0, duplex is 0 mode is 1
> dpdk: Get Link Status port 0,  status is 1, speed is 1000, duplex is 1 mode is 1
> PANIC in rte_mbuf_raw_alloc():
> line 1153       assert "rte_mbuf_refcnt_read(m) == 0" failed
> 
> 
> gdb looking at the core
> 
> Program terminated with signal 6, Aborted.
> #0  0x0000003488c328a5 in raise () from /lib64/libc.so.6
>  (gdb) bt
> #0  0x0000003488c328a5 in raise () from /lib64/libc.so.6
> #1  0x0000003488c34085 in abort () from /lib64/libc.so.6
> #2  0x00007fe927e9f527 in __rte_panic (funcname=<value optimized out>,
>     format=0x7fe927ffe690 "line %d\tassert \"rte_mbuf_refcnt_read(m) == 0\" failed\n%.0s")
>     at /root/xxxxxx/dpdk-16.07/lib/librte_eal/linuxapp/eal/eal_debug.c:86
> #3  0x00007fe927f56e76 in rte_mbuf_raw_alloc (rx_queue=0x7fe8fa6caa00, rx_pkts=0x7fe8d65fc190,
>     nb_pkts=32) at /root/xxxxxx/dpdk-16.07/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1153
> #4  eth_igb_recv_pkts (rx_queue=0x7fe8fa6caa00, rx_pkts=0x7fe8d65fc190, nb_pkts=32)
>     at /root/xxxxxx/dpdk-16.07/drivers/net/e1000/igb_rxtx.c:832
> #5  0x00007fe927f1ebef in rte_eth_rx_burst () from /opt/EMRFeed/lib/libIssDPDKDriverInterface.so
> #6  0x00007fe927f2e684 in dpdk_receive () from /opt/EMRFeed/lib/libIssDPDKDriverInterface.so
> #7  0x00007fe927f3d023 in eal_thread_loop (arg=<value optimized out>)
>     at /root/xxxxxx/dpdk-16.07/lib/librte_eal/linuxapp/eal/eal_thread.c:184
> #8  0x0000003489007851 in start_thread () from /lib64/libpthread.so.0
> #9  0x0000003488ce890d in clone () from /lib64/libc.so.6
> (gdb)
> 
> 
> Anyone come across this or know what I'm missing doing?
> 

It looks you are running your application with mbuf debugging enabled.
This checks that the mbuf refcount is 0 when allocating it from the
pool. This refcount is normally set to 0 before putting it back into the
pool in rte_pktmbuf_free() or rte_pktmbuf_free_seg().

The functions __rte_mbuf_raw_free() or rte_mempool_put*() don't set it,
so maybe your application calls one of these functions to free a mbuf,
without setting back the refcount to 0. If that's the case, prefer using
rte_pktmbuf_free() or rte_pktmbuf_free_seg().

This has been discussed here:
http://dpdk.org/ml/archives/dev/2016-June/040558.html

This (not so well documented) behavior may change in 16.11.

Regards,
Olivier