RE: tailqs issue

["Lombardo, Ed" <Ed.Lombardo@netscout.com>]

Hi,
Further debugging has left me clueless as to the problem with getting the first Obj from the mempool on the secondary process to send to the primary process (my Application).

PRIMARY:
My application side (primary process) EAL Arguments:
"--proc-type=primary", "--file-prefix=dpdk_shared", "-l 25,26,27,28", "-n4" , "--socket-mem=2048", "--legacy-mem",

Create the _MSG_POOL
t_message_pool = rte_mempool_create(_MSG_POOL,       // Pool name
                                             t_pool_size,     // Number of elements in the pool
                                             STR_TOKEN_SIZE,  // Size of each message
                                             t_pool_cache,    // Cache size
                                             t_priv_data_sz,  // Private data size
                                             NULL,            // mp_init
                                             NULL,            // mp_init_arg
                                             NULL,            // obj_init
                                             NULL,            // obj_init_arg
                                             0,                   // rte_socket_id(), // socket_id
                                             t_flags          // flags
                                             );

The t_message_pool pointer value matches the secondary process when execute " message_pool = rte_mempool_lookup(_MSG_POOL);"
NTSMON: send ring (0x1bfb43480), recv ring (0x1bfb45a00) and message pool (0x14f570ac0) are created successfully.


SECONDARY:
Secondary process I execute: " # ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file-prefix=dpdk_shared --proc-type=secondary --"


Notes: 
* hugepages are created 2x1G on NUMA Node 0
      [root@localhost ~]# /opt/dpdk/dpdk-hugepages.py -s
      Node Pages Size Total
      0    2     1Gb    2Gb
* --file-prefix=dpdk_shared is provided to both primary and secondary
* --proc-type is correctly defined for both primary and secondary process.
* The secondary process reports correct socket=0 (See dump command output below)
* The secondary process showed Available mempool count is 0 and in-use count is 1024 (which looks incorrect).
* Secondary process reports mempool is empty
* Secondary audit passed (rte_mempool_audit()), no panic occurred.
* Tried disabling ASLR
* Tried turning off legacy-mem

EXECUTE Secondary process application "dpdk-simple_mp_dbg3"
[root@localhost ~]# ./dpdk-simple_mp_dbg3 -l 30-31 -n 4 --file-prefix=dpdk_shared --proc-type=secondary --legacy-mem --
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/dpdk_shared/mp_socket_221890_1ba0562a6ae95
EAL: Selected IOVA mode 'PA'
APP: Finished Process Init.
APP: Availble mempool count is 0, in-use count is 1024, (mempool ptr=0x14f570ac0)
APP: is mempool empty (1) or full (0)?
APP: check audit on message_pool
APP: Secondary mempool dump file write
APP: NO Objs in message pool (_MSG_POOL), exit the app

simple_mp > Starting core 31


simple_mp > send hello
message_pool pointer is 0x14f570ac0
Failed to get msg obj from mem pool: Success (ret=-2)
EAL: PANIC in cmd_send_parsed():
Failed to get message buffer
0: ./dpdk-simple_mp_dbg3 (rte_dump_stack+0x2b) [b1ee1b]
1: ./dpdk-simple_mp_dbg3 (__rte_panic+0xbd) [525ede]
2: ./dpdk-simple_mp_dbg3 (cmd_send_parsed+0x2d8) [7ceb68]
3: ./dpdk-simple_mp_dbg3 (400000+0x6a5f46) [aa5f46]
4: ./dpdk-simple_mp_dbg3 (400000+0x6a4f20) [aa4f20]
5: ./dpdk-simple_mp_dbg3 (rdline_char_in+0x34b) [aa841b]
6: ./dpdk-simple_mp_dbg3 (cmdline_in+0x71) [aa4ff1]
7: ./dpdk-simple_mp_dbg3 (cmdline_interact+0x30) [aa5110]
8: ./dpdk-simple_mp_dbg3 (400000+0xfe5cb) [4fe5cb]
9: /lib64/libc.so.6 (7f2d76600000+0x3feb0) [7f2d7663feb0]
10: /lib64/libc.so.6 (__libc_start_main+0x80) [7f2d7663ff60]
11: ./dpdk-simple_mp_dbg3 (_start+0x25) [7ce795]
Aborted (core dumped)

The rte_mempool_dump() is:
mempool <MSG_POOL>@0x14f570ac0
  flags=1c
  socket_id=0
  pool=0x14f56c8c0
  iova=0x1cf570ac0
  nrte_mempool_getb_mem_chunks=1
  size=1024
  populated_size=1024
  header_size=64
  elt_size=64
  trailer_size=64
  total_obj_size=192
  private_data_size=0
  ops_index=1
  ops_name: <cn9k_mempool_ops>
  memory chunk at 0x1bfb43180, addr=0x14f53c780, iova=0x1cf53c780, len=196800
  avg bytes/object=192.187500
  internal cache infos:
    cache_size=32    cache_count[0]=0
    .....
    cache_count[127]=0
    total_cache_count=0
  common_pool_count=1024
  no statistics available

Any guidance is appreciated.

Thanks,
Ed

-----Original Message-----
From: Lombardo, Ed 
Sent: Friday, March 21, 2025 2:19 PM
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: users@dpdk.org
Subject: RE: tailqs issue

Hi Sephen,
Thank you for your help.  I made good progress up to now.

When I try to use the dpdk_simple_mp application to send a message to my application I get a Segmentation fault.

First, I re-verified the dpdk_simple_mp process=primary and dpdk_simple_mp process-secondary does pass messages successfully.  So, my hugepages are created and DPDK initializes successfully on both at startup.

In my application I created the send and recv rings and message_pool as the primary process.  The logs I added do not show any errors.  
Once my application starts and settles I started the dpdk_simple_mp application: # ./dpdk-simple_mp_dbg -l 30-31 -n 4 --legacy-mem --proc-type secondary --

However, on the dpdk_simple_mp side do "send hello" and I then get a segmentation fault.
The debugger takes me deep within the dpdk libraries which I am not too familiar with.
The rte_ring_elem.h file function: rte_ring_dequeue_build_elem() is where I end up with segmentation fault.  I notice that the variables are optimized out, not sure why since I built the dpdk libraries with debug flag.

Here is the back trace and could you point me in the direction to look.  

# gdb dpdk-simple_mp /core/core.dpdk-simple_mp.241269
warning: Unexpected size of section `.reg-xstate/241269' in core file.
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./dpdk-simple_mp -l 30-31 -n 4 --legacy-mem --proc-type secondary --'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/241269' in core file.
#0  0x0000000000cf446a in bucket_dequeue () [Current thread is 1 (Thread 0x7f946f835c00 (LWP 241269))] Missing separate debuginfos, use: dnf debuginfo-install elfutils-libelf-0.189-3.el9.x86_64 glibc-2.34-83.0.1.el9_3.7.x86_64 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libpcap-1.10.0-4.el9.x86_64 libzstd-1.5.1-2.el9.x86_64 numactl-libs-2.0.16-1.el9.x86_64 openssl-libs-3.0.7-25.0.1.el9_3.x86_64 zlib-1.2.11-40.el9.x86_64
(gdb) bt
#0  0x0000000000cf446a in bucket_dequeue ()
#1  0x00000000007ce77d in cmd_send_parsed ()
#2  0x0000000000aa5d96 in __cmdline_parse ()
#3  0x0000000000aa4d70 in cmdline_valid_buffer ()
#4  0x0000000000aa826b in rdline_char_in ()
#5  0x0000000000aa4e41 in cmdline_in ()
#6  0x0000000000aa4f60 in cmdline_interact ()
#7  0x00000000004fe47a in main.cold ()
#8  0x00007f946f03feb0 in __libc_start_call_main () from /lib64/libc.so.6
#9  0x00007f946f03ff60 in __libc_start_main_impl () from /lib64/libc.so.6
#10 0x00000000007ce605 in _start ()

Gdb - stepping through the code, gdb attached to dpdk_simple_mp_debug
(gdb)
0x0000000000cf42c5 in rte_ring_dequeue_bulk_elem (available=<optimized out>, n=<optimized out>, esize=<optimized out>,
    obj_table=<optimized out>, r=<optimized out>) at ../lib/ring/rte_ring_elem.h:375
375     ../lib/ring/rte_ring_elem.h: No such file or directory.
(gdb) p r
$17 = <optimized out>
(gdb) p obj_table
$18 = <optimized out>
(gdb) p available
$19 = <optimized out>
(gdb) n

Thread 1 "dpdk-simple_mp_" received signal SIGSEGV, Segmentation fault.
bucket_dequeue_orphans (n_orphans=33, obj_table=0x14f09b5c0, bd=0x14f05aa80)
    at ../drivers/mempool/bucket/rte_mempool_bucket.c:191
191     ../drivers/mempool/bucket/rte_mempool_bucket.c: No such file or directory.
(gdb) bt
#0  bucket_dequeue_orphans (n_orphans=33, obj_table=0x14f09b5c0, bd=0x14f05aa80)
    at ../drivers/mempool/bucket/rte_mempool_bucket.c:191
#1  bucket_dequeue (mp=<optimized out>, obj_table=0x14f09b5c0, n=33) at ../drivers/mempool/bucket/rte_mempool_bucket.c:289
#2  0x00000000007ce77d in rte_mempool_ops_dequeue_bulk (n=<optimized out>, obj_table=0x14f09b5c0, mp=0x14f05ed40)
    at ../lib/mempool/rte_mempool.h:793
#3  rte_mempool_do_generic_get (cache=0x14f09b580, n=1, obj_table=0x7fff8df066f0, mp=0x14f05ed40)
    at ../lib/mempool/rte_mempool.h:1570
#4  rte_mempool_generic_get (cache=0x14f09b580, n=1, obj_table=0x7fff8df066f0, mp=0x14f05ed40)
    at ../lib/mempool/rte_mempool.h:1649
#5  rte_mempool_get_bulk (n=1, obj_table=0x7fff8df066f0, mp=0x14f05ed40) at ../lib/mempool/rte_mempool.h:1684
#6  rte_mempool_get (obj_p=0x7fff8df066f0, mp=0x14f05ed40) at ../lib/mempool/rte_mempool.h:1710
#7  cmd_send_parsed (parsed_result=parsed_result@entry=0x7fff8df06790, cl=cl@entry=0x2f73220, data=data@entry=0x0)
    at ../examples/multi_process/simple_mp/mp_commands.c:18
#8  0x0000000000aa5d96 in __cmdline_parse (cl=cl@entry=0x2f73220, buf=0x2f73268 "send hello\n",
    call_fn=call_fn@entry=true) at ../lib/cmdline/cmdline_parse.c:294
#9  0x0000000000aa5f1a in cmdline_parse (cl=cl@entry=0x2f73220, buf=<optimized out>) at ../lib/cmdline/cmdline_parse.c:302
#10 0x0000000000aa4d70 in cmdline_valid_buffer (rdl=<optimized out>, buf=<optimized out>, size=<optimized out>)
    at ../lib/cmdline/cmdline.c:24
#11 0x0000000000aa826b in rdline_char_in (rdl=rdl@entry=0x2f73230, c=<optimized out>)
    at ../lib/cmdline/cmdline_rdline.c:444
#12 0x0000000000aa4e41 in cmdline_in (size=<optimized out>, buf=<optimized out>, cl=<optimized out>)
    at ../lib/cmdline/cmdline.c:146
#13 cmdline_in (cl=0x2f73220, buf=0x7fff8df0c89f "\n\200", size=<optimized out>) at ../lib/cmdline/cmdline.c:135
#14 0x0000000000aa4f60 in cmdline_interact (cl=cl@entry=0x2f73220) at ../lib/cmdline/cmdline.c:192
#15 0x00000000004fe47a in main (argc=<optimized out>, argv=<optimized out>)
    at ../examples/multi_process/simple_mp/main.c:122

Appreciate if you can help.

Thanks,
Ed

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Wednesday, March 19, 2025 7:17 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: users@dpdk.org
Subject: Re: tailqs issue

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Wed, 19 Mar 2025 21:52:39 +0000
"Lombardo, Ed" <Ed.Lombardo@netscout.com> wrote:

> Hi Stephen,
> I added the fib library, but I now see there are many more dpdk libraries I need to add.  Is this typically the case with the example files working with primary DPDK application?
> 
> I am using meson and ninja to build the examples, but I don't know how to know the library dependencies.
> 
> How do I learn ahead of building my Application as to what extra libraries I need to include for the DPDK example to work?
> 
> I am doing incremental build-test-find_missing_library.
> 
> So far, I needed to add these: -lrte_fib -lrte_rib -lrte_stack 
> -lrte_member -lrte_efd
> 
> Thanks,
> Ed

The typical case is to make sure that primary and secondary are built with the same libraries.