From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f43.google.com (mail-oi0-f43.google.com [209.85.218.43]) by dpdk.org (Postfix) with ESMTP id 530F7374C for ; Wed, 12 Jul 2017 12:30:12 +0200 (CEST) Received: by mail-oi0-f43.google.com with SMTP id x187so15704971oig.3 for ; Wed, 12 Jul 2017 03:30:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Y5ftWsaUye7wYkB8lbGK5qE/RO29LDzru6x79V9CAHU=; b=LMB0g5+/pUVY03leizL68+ddm7GWHdOYKR+IcJELvw6Xj9HGLKQDLzHQ2QsJK4K6Tp 7NTNc5jtg8m4AdlPlxwxt4EJJ/IpF4idU0xPZ8KV8vUtCiC+nfc71FtTNJRIPH3MpkR2 73ohkrt53XBk4BbmwZLEI5xCef4BhDJM0nPXFpF2O6ksg9+UmUc2ZtaKfmFNbMr8KbO0 CmYrj1eEXfPHMIxOgYXYdcO00rxuI46hx4bDxtSru+0gtm7RiRzBeVv4rGpFJafY2WqP GPBDDtI5cVqOUelQQIkDTZ7/4HK3y7OVqbacHiAqKxnzd1nNwy8vHpGiRx/xjtdkybVd GzUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Y5ftWsaUye7wYkB8lbGK5qE/RO29LDzru6x79V9CAHU=; b=bOnSCeiNhJEjOEE0bZrGwJ15oXDjWlY7HyNRrD647H1Y4u+W2h+R9uhDnwpqD8s0gH RjmQDv4frcERgPZfiVa04qmusMXurFC5MnLH5kX78ZIpCcun2Wm0ce9F0779dj4I5v+b 796zFPv5o/J0sL/nFpSWVJUECepudOSF91xHsR3IHMuMUTHfzwz9EfgSt9fpT1fJuDdA uabiv0GUx3eax3SH9F7wZOUW/+RUBUnQ71QKyT36u3PdTrKsNGXjfE5uOLa6+BM9ZzD0 e5YQYkkfMj1UvWvjuDblcfc1jOKEJiNUREeSy9f67ALxh4lgShqZxG/W8DaPeHvMpktU iPrA== X-Gm-Message-State: AIVw111fBsFuiTId6lF4s+BhwkCDGeZitCec26ryAF7b528laMaQXOcU 014f9HHqe4kwXLYNIa6wmtWQ9EqzNPDP X-Received: by 10.202.169.76 with SMTP id s73mr3248376oie.87.1499855411275; Wed, 12 Jul 2017 03:30:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.39.166 with HTTP; Wed, 12 Jul 2017 03:30:10 -0700 (PDT) Received: by 10.182.39.166 with HTTP; Wed, 12 Jul 2017 03:30:10 -0700 (PDT) In-Reply-To: References: From: Manish Jangid Date: Wed, 12 Jul 2017 16:00:10 +0530 Message-ID: To: users@dpdk.org Cc: Malkit Singh Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Facing issue as data processing packet got stuck in rte_pause X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jul 2017 10:30:12 -0000 Second Retry: Could anyone please help in this issue ? Thanks in advance. Regards Manish On Jul 6, 2017 6:34 PM, "Manish Jangid" wrote: > Hello all, > > I am facing an issue while processing packets in dpdk in which we reach a > situation where packet processing gets stopped. When I attach gdb to > process, I see the following back trace. The data processing thread is > stuck in rte_pause() call inside common_ring_mp_enqueue because of a check > on freepool state. The following is the analysis which I found while > debugging it. > > ring @0x7f33b0f0f0c0 > flags=0 > size=131072 > ct=3736608500 > ch=3736608500 > pt=3736611582 > ph=3736612099 > used=3082 > avail=127989 > watermark=0 > no statistics available > > > gdb) bt > #0 0x00007f9fb36ff425 in *common_ring_mp_enqueue (*) from > /usr/lib64/librte_mempool.so.2.1 > #1 0x000000000040ad68 in rte_mempool_ops_enqueue_bulk (n=, > obj_table=, mp=) > at /home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mempool.h:495 > #2 __mempool_generic_put (flags=, cache=0x7f33c3844e40, > n=1, obj_table=0x7f9fab0a9b88, mp=) > at /home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mempool.h:1071 > #3 rte_mempool_generic_put (flags=, cache=0x7f33c3844e40, > n=1, obj_table=0x7f9fab0a9b88, mp=) > at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mempool.h:1110 > #4 rte_mempool_put_bulk (n=1, obj_table=0x7f9fab0a9b88, mp=) > at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mempool.h:1173 > #5 rte_mempool_put (obj=0x7f33c1634780, mp=) > at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mempool.h:1225 > #6 __rte_mbuf_raw_free (m=0x7f33c1634780) > at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mbuf.h:783 > #7 rte_pktmbuf_free_seg (m=) > at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mbuf.h:1224 > #8 rte_pktmbuf_free (m=0x7f33c1634780) > at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native- > linuxapp-gcc/include/rte_mbuf.h:1245 > #9 lcore_main (args=args@entry=0x0) at dp_app.c:565 > > When we try to use debug APIs given by Dpdk I see that MEMPOOL is giving > error of invalid cache len as shown below. > > Auditing mbuf_pool using 'rte_mempool_audit' > MEMPOOL: badness on cache[27] > PANIC in mempool_audit_cache(): > > *MEMPOOL: invalid cache len-* > When I further dump rte_mempool cache , I saw the following > > stqh_last = 0x7f33b0f0f040}} > (gdb) p ((struct rte_mempool *)0x7f33c380e900)->local_cache > $22 = (struct rte_mempool_cache *) 0x7f33c380e9c0 > (gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+26) > $49 = {size = 250, flushthresh = 375, len = 0, objs = {0x0 times>}} > *(gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+27)* > $50 = {size = 250, flushthresh = 375, len = 377, objs = {0x7f33b7f50780, > 0x7f33b7f510c0, 0x7f33b7f51a00, 0x7f33b7f52340, 0x7f33b7f52c80, > 0x7f33b7f535c0, 0x7f33b7f53f00, > > I am really looking forward for help as under what circumstances we can > reach to such condition. Thank you in advance for your help. > > - > Thanks > Manish Kumar > > > >