From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f51.google.com (mail-oi0-f51.google.com [209.85.218.51]) by dpdk.org (Postfix) with ESMTP id 53FC31094; Thu, 6 Jul 2017 15:04:41 +0200 (CEST) Received: by mail-oi0-f51.google.com with SMTP id x187so1210007oig.3; Thu, 06 Jul 2017 06:04:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=0MhO1+J8PRBXMhOaux5wpQkOoLjmrWGItDveoYCjzJM=; b=bWtKlixAkjXCA00LljUFEC9lEmU6Tor8w7oSkiPNSLRAeRcdwFBELxneYqL+LFqvtI BrniEsX3iWmUkiAY00lS9tk7MVGgyF/1G8GyPwMWv11VtMtQuMzkGuJ7qKfDPBhMCuEN cF0kuOM6n66EmH9plrA73aBV3isJ9sg7Zza72ITXmaSzGDFjRUpRqyGvBOHV9Zku87rN /YOUVBeaDTzcaDzph6Z96C8HJKy4qTJxs7YfcDTSPFDsYV/TXwodfZqujrq9CxYTeaDg PjIcD2SGv63uvay60sKWaE0PoNIz6o+Oo5yhOZgmJXEwAyuIXVeYn7Vz28HJwnMfQ4kZ VuTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=0MhO1+J8PRBXMhOaux5wpQkOoLjmrWGItDveoYCjzJM=; b=lbWo1huw62Bqq8E5OrgD3c8LjFmSKtx9GkqQq68PrrXMm9fb1EcFutcDxNew8QeWDo oBX/ubSYkQjy6NBJr91eiXjOgvaFG7nCkpxg0UrcSWaOtImd/ZFNrt5QS/FlPZ311CQv RJ/8f3JrH/Q5Vy3EWZZBxlwc3vsr2M+H2V2LITRcW4WSojjGsPr2CrXVfp3iP6Xic7Xd mO3n3NRxBVrgUx92ibkOI1lic5l9kRt0uuoNKeP+fgxI7J8KuSPRMKiI6y4cggAnwaeI 1312xEuEWgHZ4pTf8VnaB35csIdaWmLOUZvsnry1dajT+wKOwKiF7mFeov6BLSxpvauW c4IA== X-Gm-Message-State: AKS2vOybCHJY6Mgv+0RuCU4STCt8otkv38XRDVDqIL5B9B1CCIZyTpPk Ih9cqfIyQl0HXmiYDFvqmQX7G3pDrmRyVtE= X-Received: by 10.202.78.15 with SMTP id c15mr32006233oib.203.1499346280112; Thu, 06 Jul 2017 06:04:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.39.166 with HTTP; Thu, 6 Jul 2017 06:04:19 -0700 (PDT) From: Manish Jangid Date: Thu, 6 Jul 2017 18:34:19 +0530 Message-ID: To: users@dpdk.org, dev@dpdk.org Cc: malkitvicky@gmail.com Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Facing issue as data processing packet got stuck in rte_pause X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jul 2017 13:04:41 -0000 Hello all, I am facing an issue while processing packets in dpdk in which we reach a situation where packet processing gets stopped. When I attach gdb to process, I see the following back trace. The data processing thread is stuck in rte_pause() call inside common_ring_mp_enqueue because of a check on freepool state. The following is the analysis which I found while debugging it. ring @0x7f33b0f0f0c0 flags=0 size=131072 ct=3736608500 ch=3736608500 pt=3736611582 ph=3736612099 used=3082 avail=127989 watermark=0 no statistics available gdb) bt #0 0x00007f9fb36ff425 in *common_ring_mp_enqueue (*) from /usr/lib64/librte_mempool.so.2.1 #1 0x000000000040ad68 in rte_mempool_ops_enqueue_bulk (n=, obj_table=, mp=) at /home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:495 #2 __mempool_generic_put (flags=, cache=0x7f33c3844e40, n=1, obj_table=0x7f9fab0a9b88, mp=) at /home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1071 #3 rte_mempool_generic_put (flags=, cache=0x7f33c3844e40, n=1, obj_table=0x7f9fab0a9b88, mp=) at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1110 #4 rte_mempool_put_bulk (n=1, obj_table=0x7f9fab0a9b88, mp=) at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1173 #5 rte_mempool_put (obj=0x7f33c1634780, mp=) at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1225 #6 __rte_mbuf_raw_free (m=0x7f33c1634780) at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:783 #7 rte_pktmbuf_free_seg (m=) at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1224 #8 rte_pktmbuf_free (m=0x7f33c1634780) at /home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1245 #9 lcore_main (args=args@entry=0x0) at dp_app.c:565 When we try to use debug APIs given by Dpdk I see that MEMPOOL is giving error of invalid cache len as shown below. Auditing mbuf_pool using 'rte_mempool_audit' MEMPOOL: badness on cache[27] PANIC in mempool_audit_cache(): *MEMPOOL: invalid cache len-* When I further dump rte_mempool cache , I saw the following stqh_last = 0x7f33b0f0f040}} (gdb) p ((struct rte_mempool *)0x7f33c380e900)->local_cache $22 = (struct rte_mempool_cache *) 0x7f33c380e9c0 (gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+26) $49 = {size = 250, flushthresh = 375, len = 0, objs = {0x0 }} *(gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+27)* $50 = {size = 250, flushthresh = 375, len = 377, objs = {0x7f33b7f50780, 0x7f33b7f510c0, 0x7f33b7f51a00, 0x7f33b7f52340, 0x7f33b7f52c80, 0x7f33b7f535c0, 0x7f33b7f53f00, I am really looking forward for help as under what circumstances we can reach to such condition. Thank you in advance for your help. - Thanks Manish Kumar