RE: tailqs issue

["Lombardo, Ed" <Ed.Lombardo@netscout.com>]

Hi Sephen,
Thank you for your help.  I made good progress up to now.

When I try to use the dpdk_simple_mp application to send a message to my application I get a Segmentation fault.

First, I re-verified the dpdk_simple_mp process=primary and dpdk_simple_mp process-secondary does pass messages successfully.  So, my hugepages are created and DPDK initializes successfully on both at startup.

In my application I created the send and recv rings and message_pool as the primary process.  The logs I added do not show any errors.  
Once my application starts and settles I started the dpdk_simple_mp application: # ./dpdk-simple_mp_dbg -l 30-31 -n 4 --legacy-mem --proc-type secondary --

However, on the dpdk_simple_mp side do "send hello" and I then get a segmentation fault.
The debugger takes me deep within the dpdk libraries which I am not too familiar with.
The rte_ring_elem.h file function: rte_ring_dequeue_build_elem() is where I end up with segmentation fault.  I notice that the variables are optimized out, not sure why since I built the dpdk libraries with debug flag.

Here is the back trace and could you point me in the direction to look.  

# gdb dpdk-simple_mp /core/core.dpdk-simple_mp.241269
warning: Unexpected size of section `.reg-xstate/241269' in core file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./dpdk-simple_mp -l 30-31 -n 4 --legacy-mem --proc-type secondary --'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/241269' in core file.
#0  0x0000000000cf446a in bucket_dequeue ()
[Current thread is 1 (Thread 0x7f946f835c00 (LWP 241269))]
Missing separate debuginfos, use: dnf debuginfo-install elfutils-libelf-0.189-3.el9.x86_64 glibc-2.34-83.0.1.el9_3.7.x86_64 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libpcap-1.10.0-4.el9.x86_64 libzstd-1.5.1-2.el9.x86_64 numactl-libs-2.0.16-1.el9.x86_64 openssl-libs-3.0.7-25.0.1.el9_3.x86_64 zlib-1.2.11-40.el9.x86_64
(gdb) bt
#0  0x0000000000cf446a in bucket_dequeue ()
#1  0x00000000007ce77d in cmd_send_parsed ()
#2  0x0000000000aa5d96 in __cmdline_parse ()
#3  0x0000000000aa4d70 in cmdline_valid_buffer ()
#4  0x0000000000aa826b in rdline_char_in ()
#5  0x0000000000aa4e41 in cmdline_in ()
#6  0x0000000000aa4f60 in cmdline_interact ()
#7  0x00000000004fe47a in main.cold ()
#8  0x00007f946f03feb0 in __libc_start_call_main () from /lib64/libc.so.6
#9  0x00007f946f03ff60 in __libc_start_main_impl () from /lib64/libc.so.6
#10 0x00000000007ce605 in _start ()

Gdb - stepping through the code, gdb attached to dpdk_simple_mp_debug
(gdb)
0x0000000000cf42c5 in rte_ring_dequeue_bulk_elem (available=<optimized out>, n=<optimized out>, esize=<optimized out>,
    obj_table=<optimized out>, r=<optimized out>) at ../lib/ring/rte_ring_elem.h:375
375     ../lib/ring/rte_ring_elem.h: No such file or directory.
(gdb) p r
$17 = <optimized out>
(gdb) p obj_table
$18 = <optimized out>
(gdb) p available
$19 = <optimized out>
(gdb) n

Thread 1 "dpdk-simple_mp_" received signal SIGSEGV, Segmentation fault.
bucket_dequeue_orphans (n_orphans=33, obj_table=0x14f09b5c0, bd=0x14f05aa80)
    at ../drivers/mempool/bucket/rte_mempool_bucket.c:191
191     ../drivers/mempool/bucket/rte_mempool_bucket.c: No such file or directory.
(gdb) bt
#0  bucket_dequeue_orphans (n_orphans=33, obj_table=0x14f09b5c0, bd=0x14f05aa80)
    at ../drivers/mempool/bucket/rte_mempool_bucket.c:191
#1  bucket_dequeue (mp=<optimized out>, obj_table=0x14f09b5c0, n=33) at ../drivers/mempool/bucket/rte_mempool_bucket.c:289
#2  0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=<optimized out>, obj_table=0x14f09b5c0, mp=0x14f05ed40)
    at ../lib/mempool/rte_mempool.h:793
#3  rte_mempool_do_generic_get (cache=0x14f09b580, n=1, obj_table=0x7fff8df066f0, mp=0x14f05ed40)
    at ../lib/mempool/rte_mempool.h:1570
#4  rte_mempool_generic_get (cache=0x14f09b580, n=1, obj_table=0x7fff8df066f0, mp=0x14f05ed40)
    at ../lib/mempool/rte_mempool.h:1649
#5  rte_mempool_get_bulk (n=1, obj_table=0x7fff8df066f0, mp=0x14f05ed40) at ../lib/mempool/rte_mempool.h:1684
#6  rte_mempool_get (obj_p=0x7fff8df066f0, mp=0x14f05ed40) at ../lib/mempool/rte_mempool.h:1710
#7  cmd_send_parsed (parsed_result=parsed_result@entry=0x7fff8df06790, cl=cl@entry=0x2f73220, data=data@entry=0x0)
    at ../examples/multi_process/simple_mp/mp_commands.c:18
#8  0x0000000000aa5d96 in __cmdline_parse (cl=cl@entry=0x2f73220, buf=0x2f73268 "send hello\n",
    call_fn=call_fn@entry=true) at ../lib/cmdline/cmdline_parse.c:294
#9  0x0000000000aa5f1a in cmdline_parse (cl=cl@entry=0x2f73220, buf=<optimized out>) at ../lib/cmdline/cmdline_parse.c:302
#10 0x0000000000aa4d70 in cmdline_valid_buffer (rdl=<optimized out>, buf=<optimized out>, size=<optimized out>)
    at ../lib/cmdline/cmdline.c:24
#11 0x0000000000aa826b in rdline_char_in (rdl=rdl@entry=0x2f73230, c=<optimized out>)
    at ../lib/cmdline/cmdline_rdline.c:444
#12 0x0000000000aa4e41 in cmdline_in (size=<optimized out>, buf=<optimized out>, cl=<optimized out>)
    at ../lib/cmdline/cmdline.c:146
#13 cmdline_in (cl=0x2f73220, buf=0x7fff8df0c89f "\n\200", size=<optimized out>) at ../lib/cmdline/cmdline.c:135
#14 0x0000000000aa4f60 in cmdline_interact (cl=cl@entry=0x2f73220) at ../lib/cmdline/cmdline.c:192
#15 0x00000000004fe47a in main (argc=<optimized out>, argv=<optimized out>)
    at ../examples/multi_process/simple_mp/main.c:122

Appreciate if you can help.

Thanks,
Ed

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Wednesday, March 19, 2025 7:17 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: users@dpdk.org
Subject: Re: tailqs issue

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Wed, 19 Mar 2025 21:52:39 +0000
"Lombardo, Ed" <Ed.Lombardo@netscout.com> wrote:

> Hi Stephen,
> I added the fib library, but I now see there are many more dpdk libraries I need to add.  Is this typically the case with the example files working with primary DPDK application?
> 
> I am using meson and ninja to build the examples, but I don't know how to know the library dependencies.
> 
> How do I learn ahead of building my Application as to what extra libraries I need to include for the DPDK example to work?
> 
> I am doing incremental build-test-find_missing_library.
> 
> So far, I needed to add these: -lrte_fib -lrte_rib -lrte_stack -lrte_member -lrte_efd
> 
> Thanks,
> Ed

The typical case is to make sure that primary and secondary are built with the same libraries.