From: Manish Jangid <manish.jangid08@gmail.com>
To: users@dpdk.org, dev@dpdk.org
Cc: malkitvicky@gmail.com
Subject: [dpdk-users] Facing issue as data processing packet got stuck in rte_pause
Date: Thu, 6 Jul 2017 18:34:19 +0530 [thread overview]
Message-ID: <CAB107XBDJQvQPko_mMWvxtPOthX0FZV+_HJXq43xU1QqA-uPfQ@mail.gmail.com> (raw)
Hello all,
I am facing an issue while processing packets in dpdk in which we reach a
situation where packet processing gets stopped. When I attach gdb to
process, I see the following back trace. The data processing thread is
stuck in rte_pause() call inside common_ring_mp_enqueue because of a check
on freepool state. The following is the analysis which I found while
debugging it.
ring <MP_FEDP_MBUF_POOL>@0x7f33b0f0f0c0
flags=0
size=131072
ct=3736608500
ch=3736608500
pt=3736611582
ph=3736612099
used=3082
avail=127989
watermark=0
no statistics available
gdb) bt
#0 0x00007f9fb36ff425 in *common_ring_mp_enqueue (*) from
/usr/lib64/librte_mempool.so.2.1
#1 0x000000000040ad68 in rte_mempool_ops_enqueue_bulk (n=<optimized out>,
obj_table=<optimized out>, mp=<optimized out>)
at
/home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:495
#2 __mempool_generic_put (flags=<optimized out>, cache=0x7f33c3844e40, n=1,
obj_table=0x7f9fab0a9b88, mp=<optimized out>)
at
/home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1071
#3 rte_mempool_generic_put (flags=<optimized out>, cache=0x7f33c3844e40,
n=1, obj_table=0x7f9fab0a9b88, mp=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1110
#4 rte_mempool_put_bulk (n=1, obj_table=0x7f9fab0a9b88, mp=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1173
#5 rte_mempool_put (obj=0x7f33c1634780, mp=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1225
#6 __rte_mbuf_raw_free (m=0x7f33c1634780)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:783
#7 rte_pktmbuf_free_seg (m=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1224
#8 rte_pktmbuf_free (m=0x7f33c1634780)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1245
#9 lcore_main (args=args@entry=0x0) at dp_app.c:565
When we try to use debug APIs given by Dpdk I see that MEMPOOL is giving
error of invalid cache len as shown below.
Auditing mbuf_pool using 'rte_mempool_audit'
MEMPOOL: badness on cache[27]
PANIC in mempool_audit_cache():
*MEMPOOL: invalid cache len-*
When I further dump rte_mempool cache , I saw the following
stqh_last = 0x7f33b0f0f040}}
(gdb) p ((struct rte_mempool *)0x7f33c380e900)->local_cache
$22 = (struct rte_mempool_cache *) 0x7f33c380e9c0
(gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+26)
$49 = {size = 250, flushthresh = 375, len = 0, objs = {0x0 <repeats 1536
times>}}
*(gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+27)*
$50 = {size = 250, flushthresh = 375, len = 377, objs = {0x7f33b7f50780,
0x7f33b7f510c0, 0x7f33b7f51a00, 0x7f33b7f52340, 0x7f33b7f52c80,
0x7f33b7f535c0, 0x7f33b7f53f00,
I am really looking forward for help as under what circumstances we can
reach to such condition. Thank you in advance for your help.
-
Thanks
Manish Kumar
next reply other threads:[~2017-07-06 13:04 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-06 13:04 Manish Jangid [this message]
2017-07-12 10:30 ` Manish Jangid
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAB107XBDJQvQPko_mMWvxtPOthX0FZV+_HJXq43xU1QqA-uPfQ@mail.gmail.com \
--to=manish.jangid08@gmail.com \
--cc=dev@dpdk.org \
--cc=malkitvicky@gmail.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).