RE: [PATCH v13 00/16] graph enhancement for multi-core dispatch

["Yan, Zhirun" <zhirun.yan@intel.com>]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Tuesday, June 13, 2023 7:13 PM
> To: Yan, Zhirun <zhirun.yan@intel.com>
> Cc: dev@dpdk.org; jerinj@marvell.com; kirankumark@marvell.com;
> ndabilpuram@marvell.com; stephen@networkplumber.org;
> pbhagavatula@marvell.com; david.marchand@redhat.com; Liang, Cunming
> <cunming.liang@intel.com>; Wang, Haiyue <haiyue.wang@intel.com>;
> mattias.ronnblom <mattias.ronnblom@ericsson.com>
> Subject: Re: [PATCH v13 00/16] graph enhancement for multi-core dispatch
> 
> On Tue, Jun 13, 2023 at 3:53 PM Zhirun Yan <zhirun.yan@intel.com> wrote:
> >
> > V13:
> > Add sub header into meson indirect_headers list to export.(change
> > meson.build in patch02, 05)
> 
> Please rebase with dpdk.org/main.  There are build issues now also make sure
> monitor CI after pushing the patches.
> 
Got it. I will rebase and fix the issues.

> [569/2998] Compiling C object lib/librte_node.a.p/node_ethdev_ctrl.c.o
> [570/2998] Compiling C object lib/librte_graph.a.p/graph_graph.c.o
> [571/2998] Compiling C object lib/librte_node.a.p/node_ip6_rewrite.c.o
> FAILED: lib/librte_node.a.p/node_ip6_rewrite.c.o
> ccache gcc -Ilib/librte_node.a.p -Ilib -I../lib -Ilib/node -I../lib/node -I. -I.. -Iconfig
> -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -
> I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -
> Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs -
> I../lib/kvargs -Ilib/metrics -I../lib/metrics -Ilib/telemetry -I../lib/telemetry -
> Ilib/graph -I../lib/graph -Ilib/pcapng -I../lib/pcapng -Ilib/ethdev -I../lib/ethdev -
> Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -
> Ilib/ring -I../lib/ring -Ilib/meter -I../lib/meter -Ilib/lpm -I../lib/lpm -Ilib/hash -
> I../lib/hash -Ilib/rcu -I../lib/rcu -Ilib/cryptodev -I../lib/cryptodev -fd iagnostics-
> color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -
> O2 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat-
> nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -
> Wnested-ext erns -Wold-style-definition -Wpointer-arith -Wsign-compare -
> Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member
> -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-zero-length-
> bounds -D_GNU_SOURCE -fPIC -march= native -DALLOW_EXPERIMENTAL_API -
> DALLOW_INTERNAL_API -Wno-format-truncation -fno-strict-aliasing -
> DRTE_LOG_DEFAULT_LOGTYPE=lib.node -MD -MQ
> lib/librte_node.a.p/node_ip6_rewrite.c.o -MF
> lib/librte_node.a.p/node_ip6_rewrite.c.o.d -o lib/librt
> e_node.a.p/node_ip6_rewrite.c.o -c ../lib/node/ip6_rewrite.c
> ../lib/node/ip6_rewrite.c:8:10: fatal error: rte_graph_worker.h: No such file or
> directory
>     8 | #include <rte_graph_worker.h>
>       |          ^~~~~~~~~~~~~~~~~~~~
> compilation terminated.
> [572/2998] Compiling C object
> drivers/libtmp_rte_common_cnxk.a.p/common_cnxk_roc_idev.c.o
> 
> 
> >
> > V12:
> > Fix compilation broken at patch 1.(keep renamed header align with
> > patch 1,2)
> >
> > V11:
> > Update comments and fix to add experimental flags for
> rte_graph_model_is_valid() in patch 04.
> > Update added symbols in alphabetical order in version.map with patch
> 04,05,06,08,10.
> > Update commit message in patch 16.
> >
> > V10:
> > Add rte_graph_worker_model_no_check_get() for fast path, extract
> > rte_graph_model_is_valid() in patch 04.
> > Change RTE_ASSERT to return in patch 06.
> > Change to treat not defined RTE_GRAPH_MODEL_SELECT as runtime pick in
> patch 13.
> > Move stats into dispatch union in patch 14.
> > Change example to align with RTE_GRAPH_MODEL_SELECT scheme in patch
> 16.
> > Squash patch 17(doc) into patch 13(prog_guide), 16(example guide).
> >
> >
> > V9:
> > Fix CI build issues for doc building(move TAILQ next pointer out of union) in
> patch 09,10.
> > Fix graph model check in rte_graph_worker_model_set() in patch 04.
> > Fix typo in doc.
> >
> > V8:
> > No performance dorp for original l3fwd-graph and graph_perf_autotest.
> >
> > Update graph model set/get functions and add graph_model_is_valid() in
> patch 04.
> > Update doc for new scheme usage(choose model in runtime or compile time).
> > Update dispatch schedule struct into union.
> > Change enum rte_graph_worker_model to macro define in
> rte_graph_worker_common.h.
> > Add model clone in graph_clone() in patch 08.
> > Remove unnecessary inline for slow path func graph_src_node_avail() in patch
> 06.
> >
> >
> > V7:
> > Revert rte_rdtsc_precise() in fastpath to fix performance issues in patch 03.
> > Introduce new scheme for model choosing. Use RTE_GRAPH_MODEL_SELECT
> to choose in
> >   compile-time in patch 13, 15.(must have rte_graph_worker_model_set() to
> help
> >   other config func to do model specific things like alloc wq, collect
> > stats) Extract the common func clone_name() into graph_private.h for
> graph/node clone in
> >   patch 07.(new patch)
> > Use rte_graph->model in rte_graph_worker_model_set() instead of
> RTE_PER_LCORE_*.
> > Add test case for all new APIs in patch 16(new patch).
> > Remove *_END line in enum rte_graph_worker_model in patch 04.
> > Add model check for graph lcore binding.
> > Rename workqueue as graph_mcore_dispatch_wq_node in patch 09.
> > Change all new model files/APIs with prefix _mcore_dispatch_.
> > Change description of new API, comments of func/structure to explicitly
> mention for
> >   mcore dispatch model only. Add Doxygen comments.
> > Update l3fwd-graph with new scheme, Update doc.
> > Update MAINTAINERS.
> > Fix typo and format issues.
> >
> > V6:
> > Change rte_rdtsc() to rte_rdtsc_precise().
> > Add union in rte_graph_param to configure models.
> > Remove memset in fastpath, add RTE_ASSERT for cloned graph.
> > Update copyright in patch 02.
> > Update l3fwd-graph node affinity, start from rx core successively.
> >
> > V5:
> > Fix CI build issues about dynamically update doc.
> >
> > V4:
> > Fix CI build issues about undefined reference of sched apis.
> > Remove inline for model setting.
> >
> > V3:
> > Fix CI build issues about TLS and typo.
> >
> > V2:
> > Use git mv to keep git history.
> > Use TLS for per-thread local storage.
> > Change model name to mcore dispatch.
> > Change API with specific mode name.
> > Split big patch.
> > Fix CI issues.
> > Rebase l3fwd-graph example.
> > Update doc and maintainers files.
> >
> >
> > Currently, rte_graph supports RTC (Run-To-Completion) model within
> > each of a single core.
> > RTC is one of the typical model of packet processing. Others like
> > Pipeline or Hybrid are lack of support.
> >
> > The patch set introduces a 'multicore dispatch' model selection which
> > is a self-reacting scheme according to the core affinity.
> > The new model enables a cross-core dispatching mechanism which employs
> > a scheduling work-queue to dispatch streams to other worker cores
> > which being associated with the destination node. When core flavor of
> > the destination node is a default 'current', the stream can be
> > continue executed as normal.
> >
> > Example:
> > 3-node graph targets 3-core budget
> >
> > RTC:
> > Graph: node-0 -> node-1 -> node-2 @Core0.
> >
> > + - - - - - - - - - - - - - - - - - - - - - +
> > '                Core #0/1/2                '
> > '                                           '
> > ' +--------+     +---------+     +--------+ '
> > ' | Node-0 | --> | Node-1  | --> | Node-2 | '
> > ' +--------+     +---------+     +--------+ '
> > '                                           '
> > + - - - - - - - - - - - - - - - - - - - - - +
> >
> > Dispatch:
> >
> > Graph topo: node-0 -> Core1; node-1 -> node-2; node-2 -> node-3.
> > Config graph: node-0 @Core0; node-1/3 @Core1; node-2 @Core2.
> >
> > .. code-block:: diff
> >
> >     + - - - - - -+     +- - - - - - - - - - - - - +     + - - - - - -+
> >     '  Core #0   '     '          Core #1         '     '  Core #2   '
> >     '            '     '                          '     '            '
> >     ' +--------+ '     ' +--------+    +--------+ '     ' +--------+ '
> >     ' | Node-0 | - - - ->| Node-1 |    | Node-3 |<- - - - | Node-2 | '
> >     ' +--------+ '     ' +--------+    +--------+ '     ' +--------+ '
> >     '            '     '     |                    '     '      ^     '
> >     + - - - - - -+     +- - -|- - - - - - - - - - +     + - - -|- - -+
> >                              |                                 |
> >                              + - - - - - - - - - - - - - - - - +
> >
> >
> > The patch set has been break down as below:
> >
> > 1. Split graph worker into common and default model part.
> > 2. Inline graph node processing to make it reusable.
> > 3. Add set/get APIs to choose worker model.
> > 4. Introduce core affinity API to set the node run on specific worker core.
> >   (only use in new model)
> > 5. Introduce graph affinity API to bind one graph with specific worker
> >   core.
> > 6. Introduce graph clone API.
> > 7. Introduce stream moving with scheduler work-queue in patch 8~12.
> > 8. Add stats for new models.
> > 9. Abstract default graph config process and integrate new model into
> >   example/l3fwd-graph. Add new parameters for model choosing.
> >
> > We could run with new worker model by this:
> > ./dpdk-l3fwd-graph -l 8,9,10,11 -n 4 -- -p 0x1 --config="(0,0,9)" -P
> > --model="dispatch"
> >
> > References:
> > https://static.sched.com/hosted_files/dpdkuserspace22/a6/graph%20intro
> > duce%20remote%20dispatch%20for%20mult-core%20scaling.pdf
> >
> >
> >
> >
> > Zhirun Yan (16):
> >   graph: rename rte_graph_work as common
> >   graph: split graph worker into common and default model
> >   graph: move node process into inline function
> >   graph: add get/set graph worker model APIs
> >   graph: introduce graph node core affinity API
> >   graph: introduce graph bind unbind API
> >   graph: move node clone name func into private as common
> >   graph: introduce graph clone API for other worker core
> >   graph: add structure for stream moving between cores
> >   graph: introduce stream moving cross cores
> >   graph: enable create and destroy graph scheduling workqueue
> >   graph: introduce graph walk by cross-core dispatch
> >   graph: enable graph multicore dispatch scheduler model
> >   graph: add stats for mcore dispatch model
> >   test/graph: add functional tests for mcore dispatch model
> >   examples/l3fwd-graph: introduce mcore dispatch worker model
> >
> >  MAINTAINERS                                   |   3 +-
> >  app/test/test_graph.c                         | 130 ++++
> >  doc/guides/prog_guide/graph_lib.rst           |  71 ++-
> >  doc/guides/sample_app_ug/l3_forward_graph.rst |  16 +
> >  examples/l3fwd-graph/main.c                   | 230 +++++--
> >  lib/graph/graph.c                             | 161 +++++
> >  lib/graph/graph_debug.c                       |   6 +
> >  lib/graph/graph_populate.c                    |   1 +
> >  lib/graph/graph_private.h                     |  90 +++
> >  lib/graph/graph_stats.c                       |  76 ++-
> >  lib/graph/meson.build                         |   9 +-
> >  lib/graph/node.c                              |  27 +-
> >  lib/graph/rte_graph.h                         |  65 ++
> >  lib/graph/rte_graph_model_mcore_dispatch.c    | 191 ++++++
> >  lib/graph/rte_graph_model_mcore_dispatch.h    | 134 ++++
> >  lib/graph/rte_graph_model_rtc.h               |  46 ++
> >  lib/graph/rte_graph_worker.c                  |  39 ++
> >  lib/graph/rte_graph_worker.h                  | 503 +--------------
> >  lib/graph/rte_graph_worker_common.h           | 598 ++++++++++++++++++
> >  lib/graph/version.map                         |  11 +
> >  20 files changed, 1839 insertions(+), 568 deletions(-)  create mode
> > 100644 lib/graph/rte_graph_model_mcore_dispatch.c
> >  create mode 100644 lib/graph/rte_graph_model_mcore_dispatch.h
> >  create mode 100644 lib/graph/rte_graph_model_rtc.h  create mode
> > 100644 lib/graph/rte_graph_worker.c  create mode 100644
> > lib/graph/rte_graph_worker_common.h
> >
> > --
> > 2.37.2
> >