[dpdk-dev] Debug crash in bucket_enqueue ()

[dpdk-dev] Debug crash in bucket_enqueue ()

[Ravi Kerur]

DPDK team,

I have a simple multi process setup with OVS/DPDK as primary and another
process as secondary. OVS/DPDK sets up mempool and rings with MP/MC flags
and secondary process attaches to it. I have used
example/multi-process/simple_mp as an example. I have disabled 'ASLR' and
made sure primary and secondary use distinct EAL lcores.

test case is that for every 10/12 packets OVS/DPDK does
rte_mempool_get() and rte_ring_enqueue()
with information about the packets

secondary process does
rte_ring_dequeue() and rte_mempool_put()

Couple of minutes into the test secondary process crashes with following
baktrace.

Any other system level settings need to be set or inputs on debugging
bucket_enqueue?

#0  0x0000555555734160 in bucket_enqueue ()
#1  0x00005555555edf06 in rte_mempool_ops_enqueue_bulk (mp=0x17fa9ac80,
obj_table=0x17fac8210, n=16) at
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:704
#2  0x00005555555f072a in __mempool_generic_put (cache=0x17fac8100, n=1,
obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1249
#3  rte_mempool_generic_put (cache=0x17fac8100, n=1,
obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1285
#4  rte_mempool_put_bulk (n=1, obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1308
#5  rte_mempool_put (obj=0x17f994300, mp=0x17fa9ac80) at
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1326
#6  lcore_recv (arg=0x7fffffffd5b3) at /home/rkerur/simple_mp_ovs/main.c:369
#7  0x00005555556df8f1 in eal_thread_loop ()
#8  0x00007ffff72086db in start_thread (arg=0x7ffff4e0b700) at
pthread_create.c:463
#9  0x00007ffff6f3188f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thanks,
Ravi

Re: [dpdk-dev] Debug crash in bucket_enqueue ()

[Andrew Rybchenko]

Hello,

please, see note on the following page:
https://doc.dpdk.org/guides/prog_guide/mempool_lib.html#mempool-handlers

Andrew.

On 10/24/19 7:42 PM, Ravi Kerur wrote:
> DPDK team,
>
> I have a simple multi process setup with OVS/DPDK as primary and another
> process as secondary. OVS/DPDK sets up mempool and rings with MP/MC flags
> and secondary process attaches to it. I have used
> example/multi-process/simple_mp as an example. I have disabled 'ASLR' and
> made sure primary and secondary use distinct EAL lcores.
>
> test case is that for every 10/12 packets OVS/DPDK does
> rte_mempool_get() and rte_ring_enqueue()
> with information about the packets
>
> secondary process does
> rte_ring_dequeue() and rte_mempool_put()
>
> Couple of minutes into the test secondary process crashes with following
> baktrace.
>
> Any other system level settings need to be set or inputs on debugging
> bucket_enqueue?
>
> #0  0x0000555555734160 in bucket_enqueue ()
> #1  0x00005555555edf06 in rte_mempool_ops_enqueue_bulk (mp=0x17fa9ac80,
> obj_table=0x17fac8210, n=16) at
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:704
> #2  0x00005555555f072a in __mempool_generic_put (cache=0x17fac8100, n=1,
> obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1249
> #3  rte_mempool_generic_put (cache=0x17fac8100, n=1,
> obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1285
> #4  rte_mempool_put_bulk (n=1, obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1308
> #5  rte_mempool_put (obj=0x17f994300, mp=0x17fa9ac80) at
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1326
> #6  lcore_recv (arg=0x7fffffffd5b3) at /home/rkerur/simple_mp_ovs/main.c:369
> #7  0x00005555556df8f1 in eal_thread_loop ()
> #8  0x00007ffff72086db in start_thread (arg=0x7ffff4e0b700) at
> pthread_create.c:463
> #9  0x00007ffff6f3188f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
> Thanks,
> Ravi

Re: [dpdk-dev] Debug crash in bucket_enqueue ()

[Ravi Kerur]

On Thu, Oct 24, 2019 at 9:58 AM Andrew Rybchenko <arybchenko@solarflare.com>
wrote:

> Hello,
>
> please, see note on the following page:
> https://doc.dpdk.org/guides/prog_guide/mempool_lib.html#mempool-handlers


Thanks for your inputs. Note refers to '-d' for shared objects. In my case
both OVS/DPDK and multi_process/simple_mp use static linked libraries. I am
pretty use OVS/DPDK uses libdpdk.a which is a group of all archives. I see
following from sample_mp_ovs.map file

Archive member included to satisfy reference by file (symbol)

/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/lib/librte_pipeline.a(rte_pipeline.o)
                              (--whole-archive)
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/lib/librte_pipeline.a(rte_port_in_action.o)
                              (--whole-archive)
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/lib/librte_pipeline.a(rte_table_action.o)
                              (--whole-archive)
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/lib/librte_table.a(rte_table_lpm.o)
                              (--whole-archive)
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/lib/librte_table.a(rte_table_lpm_ipv6.o)
                              (--whole-archive)
/home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/lib/librte_table.a(rte_table_acl.o)
                              (--whole-archive)
...

Thanks.

>
>
> Andrew.
>
> On 10/24/19 7:42 PM, Ravi Kerur wrote:
> > DPDK team,
> >
> > I have a simple multi process setup with OVS/DPDK as primary and another
> > process as secondary. OVS/DPDK sets up mempool and rings with MP/MC flags
> > and secondary process attaches to it. I have used
> > example/multi-process/simple_mp as an example. I have disabled 'ASLR' and
> > made sure primary and secondary use distinct EAL lcores.
> >
> > test case is that for every 10/12 packets OVS/DPDK does
> > rte_mempool_get() and rte_ring_enqueue()
> > with information about the packets
> >
> > secondary process does
> > rte_ring_dequeue() and rte_mempool_put()
> >
> > Couple of minutes into the test secondary process crashes with following
> > baktrace.
> >
> > Any other system level settings need to be set or inputs on debugging
> > bucket_enqueue?
> >
> > #0  0x0000555555734160 in bucket_enqueue ()
> > #1  0x00005555555edf06 in rte_mempool_ops_enqueue_bulk (mp=0x17fa9ac80,
> > obj_table=0x17fac8210, n=16) at
> >
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:704
> > #2  0x00005555555f072a in __mempool_generic_put (cache=0x17fac8100, n=1,
> > obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
> >
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1249
> > #3  rte_mempool_generic_put (cache=0x17fac8100, n=1,
> > obj_table=0x7ffff4e09050, mp=0x17fa9ac80) at
> >
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1285
> > #4  rte_mempool_put_bulk (n=1, obj_table=0x7ffff4e09050, mp=0x17fa9ac80)
> at
> >
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1308
> > #5  rte_mempool_put (obj=0x17f994300, mp=0x17fa9ac80) at
> >
> /home/rkerur/dpdk-stable-18.11.2//x86_64-native-linuxapp-gcc/include/rte_mempool.h:1326
> > #6  lcore_recv (arg=0x7fffffffd5b3) at
> /home/rkerur/simple_mp_ovs/main.c:369
> > #7  0x00005555556df8f1 in eal_thread_loop ()
> > #8  0x00007ffff72086db in start_thread (arg=0x7ffff4e0b700) at
> > pthread_create.c:463
> > #9  0x00007ffff6f3188f in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> >
> > Thanks,
> > Ravi
>
>