[dpdk-users] Segfault while freeing mbuf in the primary process

[dpdk-users] Segfault while freeing mbuf in the primary process

[Farbod]

Hi,

I am using DPDK multi-processor mode for sending packets from one 
application (secondary) to another application (primary) using rings.
The primary applications role is to just free the packet with 
`rte_pktmbuf_free()`.

I encounter a SEGFAULT error in the line of `rte_pktmbuf_free()`.

My DPDK version is 19.11.1.

```

Signal: 11 (Segmentation fault), si_code: 1 (SEGV_MAPERR: address not 
mapped to object)
Backtrace (recent calls first) ---
(0): (+0x81a1ee) [0x562d35df51ee]
     bucket_enqueue_single at 
---/dpdk-19.11.1/drivers/mempool/bucket/rte_mempool_bucket.c:111 
(discriminator 3)
          108:   addr &= bd->bucket_page_mask;
          109:   hdr = (struct bucket_header *)addr;
110:
       -> 111:   if (likely(hdr->lcore_id == lcore_id)) {
          112:           if (hdr->fill_cnt < bd->obj_per_bucket - 1) {
          113: hdr->fill_cnt++;
          114:           } else {
      (inlined by) bucket_enqueue at 
---/dpdk-19.11.1/drivers/mempool/bucket/rte_mempool_bucket.c:148 
(discriminator 3)
          145:   int rc = 0;
146:
          147:   for (i = 0; i < n; i++) {
       -> 148:           rc = bucket_enqueue_single(bd, obj_table[i]);
          149:           RTE_ASSERT(rc == 0);
          150: }
[0x562d35cd2f8c]
     rte_mempool_ops_enqueue_bulk at 
---/dpdk-19.11.1/build/include/rte_mempool.h:786
       -> 786:   return ops->enqueue(mp, obj_table, n);
      (inlined by) __mempool_generic_put at 
---/dpdk-19.11.1/build/include/rte_mempool.h:1329
       -> 1329:          rte_mempool_ops_enqueue_bulk(mp, 
&cache->objs[cache->size],
      (inlined by) rte_mempool_generic_put at 
---/dpdk-19.11.1/build/include/rte_mempool.h:1365
       -> 1365:  __mempool_generic_put(mp, obj_table, n, cache);
      (inlined by) rte_mempool_put_bulk at 
---/dpdk-19.11.1/build/include/rte_mempool.h:1388
       -> 1388:  rte_mempool_generic_put(mp, obj_table, n, cache);
      (inlined by) rte_mempool_put at 
---/dpdk-19.11.1/build/include/rte_mempool.h:1406
       -> 1406:  rte_mempool_put_bulk(mp, &obj, 1);
      (inlined by) rte_mbuf_raw_free at 
---/dpdk-19.11.1/build/include/rte_mbuf.h:579
       -> 579:   rte_mempool_put(m->pool, m);
      (inlined by) rte_pktmbuf_free_seg at 
---/dpdk-19.11.1/build/include/rte_mbuf.h:1223
       -> 1223:          rte_mbuf_raw_free(m);         151:   if 
(local_stack->top > bd->bucket_stack_thresh) {
        (inlined by) rte_pktmbuf_free at ---/rte_mbuf.h:1244
       -> 1244: rte_pktmbuf_free_seg(m);
      (inlined by) ?? at --- /packet.h:199
       -> 199:     rte_pktmbuf_free(reinterpret_cast<struct rte_mbuf 
*>(pkt));

```

I have made sure that primary and secondary process does not share any 
cpu in common. The packets received in the primary application are valid 
and the information inside them is readable. It only runs in to SEGFAULT 
when I am trying to free the mubf structures.

I would like to mention that the applications are in two different 
projects and are built separately.

Thank you

Re: [dpdk-users] Segfault while freeing mbuf in the primary process

[Farbod]

Hi,

Regarding to the SEGFAULT question I asked yesterday, while I was 
searching on the internet I stumbled upon an email sent on DPDK user 
list which explained the solution to a similar problem. (Link to the 
email: 
https://inbox.dpdk.org/users/8AC28827-5B04-4392-AFB3-AD259DFBECA9@fireeye.com/t/#m3f14d7e92c0a0b3c530717632ca5047420083112 
)

As Andrew Rybchenko mentioned in the  email, the way two applications 
are built can result in issues with mempool operations. I do not 
understand how the mempool handlers work and how the order of shared 
libraries affect the DPDK system but I can confirm that by changing one 
of the applications' Makefile I could get through this problem and thing 
are working properly now.

It looks like there is a note on DPDK guide about this subject. (Link to 
DPDK guide: https://doc.dpdk.org/guides/prog_guide/mempool_lib.html )
excerpt from DPDK guide:
```

When running a multi-process application with shared libraries, the -d 
arguments for mempool handlers /must be specified in the same order for 
all processes/ to ensure correct operation.

```

I am not using `-d` while I am running my applications maybe I should. I 
just wanted to share my findings with you and thank you all for creating 
such a community.

Thank you
~ Farbod


On 10/23/20 6:08 PM, Farbod wrote:
> Hi,
>
> I am using DPDK multi-processor mode for sending packets from one 
> application (secondary) to another application (primary) using rings.
> The primary applications role is to just free the packet with 
> `rte_pktmbuf_free()`.
>
> I encounter a SEGFAULT error in the line of `rte_pktmbuf_free()`.
>
> My DPDK version is 19.11.1.
>
> ```
>
> Signal: 11 (Segmentation fault), si_code: 1 (SEGV_MAPERR: address not 
> mapped to object)
> Backtrace (recent calls first) ---
> (0): (+0x81a1ee) [0x562d35df51ee]
>     bucket_enqueue_single at 
> ---/dpdk-19.11.1/drivers/mempool/bucket/rte_mempool_bucket.c:111 
> (discriminator 3)
>          108:   addr &= bd->bucket_page_mask;
>          109:   hdr = (struct bucket_header *)addr;
> 110:
>       -> 111:   if (likely(hdr->lcore_id == lcore_id)) {
>          112:           if (hdr->fill_cnt < bd->obj_per_bucket - 1) {
>          113: hdr->fill_cnt++;
>          114:           } else {
>      (inlined by) bucket_enqueue at 
> ---/dpdk-19.11.1/drivers/mempool/bucket/rte_mempool_bucket.c:148 
> (discriminator 3)
>          145:   int rc = 0;
> 146:
>          147:   for (i = 0; i < n; i++) {
>       -> 148:           rc = bucket_enqueue_single(bd, obj_table[i]);
>          149:           RTE_ASSERT(rc == 0);
>          150: }
> [0x562d35cd2f8c]
>     rte_mempool_ops_enqueue_bulk at 
> ---/dpdk-19.11.1/build/include/rte_mempool.h:786
>       -> 786:   return ops->enqueue(mp, obj_table, n);
>      (inlined by) __mempool_generic_put at 
> ---/dpdk-19.11.1/build/include/rte_mempool.h:1329
>       -> 1329:          rte_mempool_ops_enqueue_bulk(mp, 
> &cache->objs[cache->size],
>      (inlined by) rte_mempool_generic_put at 
> ---/dpdk-19.11.1/build/include/rte_mempool.h:1365
>       -> 1365:  __mempool_generic_put(mp, obj_table, n, cache);
>      (inlined by) rte_mempool_put_bulk at 
> ---/dpdk-19.11.1/build/include/rte_mempool.h:1388
>       -> 1388:  rte_mempool_generic_put(mp, obj_table, n, cache);
>      (inlined by) rte_mempool_put at 
> ---/dpdk-19.11.1/build/include/rte_mempool.h:1406
>       -> 1406:  rte_mempool_put_bulk(mp, &obj, 1);
>      (inlined by) rte_mbuf_raw_free at 
> ---/dpdk-19.11.1/build/include/rte_mbuf.h:579
>       -> 579:   rte_mempool_put(m->pool, m);
>      (inlined by) rte_pktmbuf_free_seg at 
> ---/dpdk-19.11.1/build/include/rte_mbuf.h:1223
>       -> 1223:          rte_mbuf_raw_free(m);         151:   if 
> (local_stack->top > bd->bucket_stack_thresh) {
>        (inlined by) rte_pktmbuf_free at ---/rte_mbuf.h:1244
>       -> 1244: rte_pktmbuf_free_seg(m);
>      (inlined by) ?? at --- /packet.h:199
>       -> 199:     rte_pktmbuf_free(reinterpret_cast<struct rte_mbuf 
> *>(pkt));
>
> ```
>
> I have made sure that primary and secondary process does not share any 
> cpu in common. The packets received in the primary application are 
> valid and the information inside them is readable. It only runs in to 
> SEGFAULT when I am trying to free the mubf structures.
>
> I would like to mention that the applications are in two different 
> projects and are built separately.
>
> Thank you
>