Re: [dpdk-dev] [PATCH v7 1/3] rcu: add RCU library supporting QSBR mechanism

[Jerin Jacob Kollanukkaran <jerinj@marvell.com>]

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Honnappa Nagarahalli
> Sent: Wednesday, April 24, 2019 2:53 AM
> To: paulmck@linux.ibm.com
> Cc: konstantin.ananyev@intel.com; stephen@networkplumber.org;
> marko.kovacevic@intel.com; dev@dpdk.org; Gavin Hu (Arm Technology China)
> <Gavin.Hu@arm.com>; Dharmik Thakkar <Dharmik.Thakkar@arm.com>;
> Malvika Gupta <Malvika.Gupta@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; bruce.richardson@intel.com; nd
> <nd@arm.com>; thomas@monjalon.net; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v7 1/3] rcu: add RCU library supporting QSBR
> mechanism
> 
> >
> > On Mon, Apr 22, 2019 at 11:31:28PM -0500, Honnappa Nagarahalli wrote:
> > > Add RCU library supporting quiescent state based memory reclamation
> > method.
> > > This library helps identify the quiescent state of the reader
> > > threads so that the writers can free the memory associated with the
> > > lock less data structures.
> > >
> > > Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > > Reviewed-by: Steve Capper <steve.capper@arm.com>
> > > Reviewed-by: Gavin Hu <gavin.hu@arm.com>
> > > Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com>
> > > Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >
> > Much better!
> >
> > Acked-by: Paul E. McKenney <paulmck@linux.ibm.com>
> >
> Thanks a lot, appreciate your feedback.
> 
> Any views from maintainers on including this library into RC3? IMO, this library is
> independent and should not affect existing code.

Tested rcu_qsbr_autotest and rcu_qsbr_perf_autotest UT on a armv8.2 machine(octeontx2).
Found rcu_qsbr_perf_autotest() runs successfully on 24 cores.
There is come issue with rcu_qsbr_autotest on 24 cores. It works fine upto 20 cores.

Please find below the success log, failure log and core dump.

[master][dpdk.org] $ echo "rcu_qsbr_autotest" | sudo ./build/app/test -c 0xfffff0
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
APP: HPET is not enabled, using TSC as default timer
RTE>>rcu_qsbr_autotest

Test rte_rcu_qsbr_thread_register()
rte_rcu_qsbr_get_memsize(): Invalid max_threads 0

Test rte_rcu_qsbr_init()
rte_rcu_qsbr_init(): Invalid input parameter

Test rte_rcu_qsbr_thread_register()
rte_rcu_qsbr_thread_register(): Invalid input parameter
rte_rcu_qsbr_thread_register(): Invalid input parameter
rte_rcu_qsbr_thread_register(): Invalid input parameter

Test rte_rcu_qsbr_thread_unregister()
rte_rcu_qsbr_thread_unregister(): Invalid input parameter
rte_rcu_qsbr_thread_unregister(): Invalid input parameter
rte_rcu_qsbr_thread_unregister(): Invalid input parameter

Test rte_rcu_qsbr_start()

Test rte_rcu_qsbr_check()
Test rte_rcu_qsbr_synchronize()

Test rte_rcu_qsbr_dump()
rte_rcu_qsbr_dump(): Invalid input parameter
rte_rcu_qsbr_dump(): Invalid input parameter
rte_rcu_qsbr_dump(): Invalid input parameter

Quiescent State Variable @0x13ff94100
  QS variable memory size = 16768
  Given # max threads = 128
  Current # threads = 0
  Registered thread ID mask = 0x00
  Token = 1
Quiescent State Counts for readers:

Quiescent State Variable @0x13ff94100
  QS variable memory size = 16768
  Given # max threads = 128
  Current # threads = 1
  Registered thread ID mask = 0x200
  Token = 1
Quiescent State Counts for readers:
thread ID = 5, count = 0, lock count = 0

Quiescent State Variable @0x13ff8ff00
  QS variable memory size = 16768
  Given # max threads = 128
  Current # threads = 2
  Registered thread ID mask = 0xc00
  Token = 1
Quiescent State Counts for readers:
thread ID = 6, count = 0, lock count = 0
thread ID = 7, count = 0, lock count = 0

Test rte_rcu_qsbr_thread_online()

Test rte_rcu_qsbr_thread_offline()

Functional tests
Test: 1 writer, 1 QSBR variable, simultaneous QSBR queries
Test: 8 writers, 4 QSBR variable, simultaneous QSBR queries

Test OK

[master] [dpdk.org] $ echo "rcu_qsbr_autotest" | sudo ./build/app/test -c 0xffffff
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
APP: HPET is not enabled, using TSC as default timer
RTE>>rcu_qsbr_autotest

Test rte_rcu_qsbr_thread_register()
rte_rcu_qsbr_get_memsize(): Invalid max_threads 0

Test rte_rcu_qsbr_init()
rte_rcu_qsbr_init(): Invalid input parameter

Test rte_rcu_qsbr_thread_register()
rte_rcu_qsbr_thread_register(): Invalid input parameter
rte_rcu_qsbr_thread_register(): Invalid input parameter
rte_rcu_qsbr_thread_register(): Invalid input parameter

Test rte_rcu_qsbr_thread_unregister()
rte_rcu_qsbr_thread_unregister(): Invalid input parameter
rte_rcu_qsbr_thread_unregister(): Invalid input parameter
rte_rcu_qsbr_thread_unregister(): Invalid input parameter

Test rte_rcu_qsbr_start()

Test rte_rcu_qsbr_check()

Test rte_rcu_qsbr_synchronize()

Test rte_rcu_qsbr_dump()
rte_rcu_qsbr_dump(): Invalid input parameter
rte_rcu_qsbr_dump(): Invalid input parameter
rte_rcu_qsbr_dump(): Invalid input parameter

Quiescent State Variable @0x13ff94100
  QS variable memory size = 16768
  Given # max threads = 128
  Current # threads = 0
  Registered thread ID mask = 0x00
  Token = 1
Quiescent State Counts for readers:

Quiescent State Variable @0x13ff94100
  QS variable memory size = 16768
  Given # max threads = 128
  Current # threads = 1
  Registered thread ID mask = 0x20
  Token = 1
Quiescent State Counts for readers:
thread ID = 1, count = 0, lock count = 0

Quiescent State Variable @0x13ff8ff00
  QS variable memory size = 16768
  Given # max threads = 128
  Current # threads = 2
  Registered thread ID mask = 0xc0
  Token = 1
Quiescent State Counts for readers:
thread ID = 2, count = 0, lock count = 0
thread ID = 3, count = 0, lock count = 0

Test rte_rcu_qsbr_thread_online()

Test rte_rcu_qsbr_thread_offline()

Functional tests
Test: 1 writer, 1 QSBR variable, simultaneous QSBR queries
Test: 10 writers, 5 QSBR variable, simultaneous QSBR queries
rte_rcu_qsbr_init(): Invalid input parameter
rte_rcu_qsbr_thread_register(): Invalid input parameter
rte_rcu_qsbr_thread_register(): Invalid input parameter
Segmentation fault


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `./build/app/test -c 0xffffff'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  rte_rcu_qsbr_thread_online (thread_id=<optimized out>, v=0x0) at /home/jerin/dpdk.org/build/include/rte_rcu_qsbr.h:238
/home/jerin/dpdk.org/lib/librte_rcu/rte_rcu_qsbr.h:238:7712:beg:0x564df4
[Current thread is 1 (Thread 0xffff9c06d900 (LWP 1938))]
(gdb) bt
#0  rte_rcu_qsbr_thread_online (thread_id=<optimized out>, v=0x0) at /home/jerin/dpdk.org/build/include/rte_rcu_qsbr.h:238
#1  test_rcu_qsbr_reader (arg=<optimized out>) at /home/jerin/dpdk.org/app/test/test_rcu_qsbr.c:641
#2  0x0000000000652430 in eal_thread_loop (arg=<optimized out>) at /home/jerin/dpdk.org/lib/librte_eal/linux/eal/eal_thread.c:153
#3  0x0000ffffa13a756c in start_thread () from /usr/lib/libpthread.so.0
#4  0x0000ffffa11e301c in thread_start () from /usr/lib/libc.so.6