From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B904242CA8; Tue, 13 Jun 2023 13:13:03 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8D00340A8A; Tue, 13 Jun 2023 13:13:03 +0200 (CEST) Received: from mail-ua1-f54.google.com (mail-ua1-f54.google.com [209.85.222.54]) by mails.dpdk.org (Postfix) with ESMTP id D044540698 for ; Tue, 13 Jun 2023 13:13:01 +0200 (CEST) Received: by mail-ua1-f54.google.com with SMTP id a1e0cc1a2514c-7870821d9a1so584182241.1 for ; Tue, 13 Jun 2023 04:13:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686654781; x=1689246781; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=BXGFU+DBSGE633izozeOPwJ6/EXH5wYkJhvsGZQyiPs=; b=KCuWhk8ipAZIJ94CdQaD7BLQAGDU//h7J3JBZTswJb37uI9kuDWpxv8HBfbGO95Ooz AnJgGhKgiUCr1A22LUUI9CTOcpBnTgJFinVp+lxb68WhZFEowb4XABgo7qPXsBP97fX8 vGjSjxF42d2JGEoAEfDJGqwfnU+i1+qK0DguFDxVHyJMJQlkHX3QR1YsVZLmFLRzsC74 YLt0NjkUzih2U0SugE3KZwOnot2l5xhfOtnHAPsVvW9F5/WZXGXdl+MBYsgKNNDbv2mg 8ltdDHO84fJCSIFaJ/4iwlnG4W2jiN52zlNOC/XYD/n+IRwouQJxeOFYj6sXIvxn+3Qj YH2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686654781; x=1689246781; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BXGFU+DBSGE633izozeOPwJ6/EXH5wYkJhvsGZQyiPs=; b=kfDoeTbwzEJ+DmL64ihvyB64xgoouCsv51KMFNurCPEd0jtzdXKM623SAeiGntko0O B+F1Cec4juSuMGxIzOeIXwBufGdVF+zVTAUkCGOEmr/87q6A0YfL0/+XRwAHpL/jdkFX eGwny1V7CREysaJ4Gym1pNXJsZtOn2ARvBojNSaBp513zXQXDXsG3G5n8evFuWCFK1ut CBw8QjJrNQrYdmE4hwY4pkYz8W9bVQnUqsdJV7EkjLLgquwRW3/lloY4UJLjqrLHjyUE A6bcSTmEPCOeWZb3GcTI6C/OkPb9N+DZomVzvPKyU9EUHfvrFEq8ik92M60r6UGz/kzA o/fw== X-Gm-Message-State: AC+VfDxTksZOWQKHM8AiuS8U0ZjMxN43XOJwkeyiATOeVEvICFr0ioSu Hi3IoL7kB4AmSyk/xn4GCqaYiWCa8lEN91Z+n9g= X-Google-Smtp-Source: ACHHUZ6MgK94eQH8rlxMir4ssY4fSRdsoF/NZ2aU9UYgR89XH2+XLiu+E6YcB5btwDgPMVk+cy6f72sid5yVIM8Wdcg= X-Received: by 2002:a67:b147:0:b0:437:e6c4:c267 with SMTP id z7-20020a67b147000000b00437e6c4c267mr4507981vsl.9.1686654780996; Tue, 13 Jun 2023 04:13:00 -0700 (PDT) MIME-Version: 1.0 References: <20230609191245.252521-1-zhirun.yan@intel.com> <20230613101404.1787790-1-zhirun.yan@intel.com> In-Reply-To: <20230613101404.1787790-1-zhirun.yan@intel.com> From: Jerin Jacob Date: Tue, 13 Jun 2023 16:42:34 +0530 Message-ID: Subject: Re: [PATCH v13 00/16] graph enhancement for multi-core dispatch To: Zhirun Yan Cc: dev@dpdk.org, jerinj@marvell.com, kirankumark@marvell.com, ndabilpuram@marvell.com, stephen@networkplumber.org, pbhagavatula@marvell.com, david.marchand@redhat.com, cunming.liang@intel.com, haiyue.wang@intel.com, mattias.ronnblom@ericsson.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Jun 13, 2023 at 3:53=E2=80=AFPM Zhirun Yan w= rote: > > V13: > Add sub header into meson indirect_headers list to export.(change meson.b= uild in patch02, 05) Please rebase with dpdk.org/main. There are build issues now also make sure monitor CI after pushing the patches. [569/2998] Compiling C object lib/librte_node.a.p/node_ethdev_ctrl.c.o [570/2998] Compiling C object lib/librte_graph.a.p/graph_graph.c.o [571/2998] Compiling C object lib/librte_node.a.p/node_ip6_rewrite.c.o FAILED: lib/librte_node.a.p/node_ip6_rewrite.c.o ccache gcc -Ilib/librte_node.a.p -Ilib -I../lib -Ilib/node -I../lib/node -I. -I.. -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/metrics -I../lib/metrics -Ilib/telemetry -I../lib/telemetry -Ilib/graph -I../lib/graph -Ilib/pcapng -I../lib/pcapng -Ilib/ethdev -I../lib/ethdev -Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring -Ilib/meter -I../lib/meter -Ilib/lpm -I../lib/lpm -Ilib/hash -I../lib/hash -Ilib/rcu -I../lib/rcu -Ilib/cryptodev -I../lib/cryptodev -fd iagnostics-color=3Dalways -D_FILE_OFFSET_BITS=3D64 -Wall -Winvalid-pch -Wextra -Werror -O2 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-ext erns -Wold-style-definition -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-zero-length-bounds -D_GNU_SOURCE -fPIC -march=3D native -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -Wno-format-truncation -fno-strict-aliasing -DRTE_LOG_DEFAULT_LOGTYPE=3Dlib.node -MD -MQ lib/librte_node.a.p/node_ip6_rewrite.c.o -MF lib/librte_node.a.p/node_ip6_rewrite.c.o.d -o lib/librt e_node.a.p/node_ip6_rewrite.c.o -c ../lib/node/ip6_rewrite.c ../lib/node/ip6_rewrite.c:8:10: fatal error: rte_graph_worker.h: No such file or directory 8 | #include | ^~~~~~~~~~~~~~~~~~~~ compilation terminated. [572/2998] Compiling C object drivers/libtmp_rte_common_cnxk.a.p/common_cnxk_roc_idev.c.o > > V12: > Fix compilation broken at patch 1.(keep renamed header align with patch 1= ,2) > > V11: > Update comments and fix to add experimental flags for rte_graph_model_is_= valid() in patch 04. > Update added symbols in alphabetical order in version.map with patch 04,0= 5,06,08,10. > Update commit message in patch 16. > > V10: > Add rte_graph_worker_model_no_check_get() for fast path, extract rte_grap= h_model_is_valid() > in patch 04. > Change RTE_ASSERT to return in patch 06. > Change to treat not defined RTE_GRAPH_MODEL_SELECT as runtime pick in pat= ch 13. > Move stats into dispatch union in patch 14. > Change example to align with RTE_GRAPH_MODEL_SELECT scheme in patch 16. > Squash patch 17(doc) into patch 13(prog_guide), 16(example guide). > > > V9: > Fix CI build issues for doc building(move TAILQ next pointer out of union= ) in patch 09,10. > Fix graph model check in rte_graph_worker_model_set() in patch 04. > Fix typo in doc. > > V8: > No performance dorp for original l3fwd-graph and graph_perf_autotest. > > Update graph model set/get functions and add graph_model_is_valid() in pa= tch 04. > Update doc for new scheme usage(choose model in runtime or compile time). > Update dispatch schedule struct into union. > Change enum rte_graph_worker_model to macro define in rte_graph_worker_co= mmon.h. > Add model clone in graph_clone() in patch 08. > Remove unnecessary inline for slow path func graph_src_node_avail() in pa= tch 06. > > > V7: > Revert rte_rdtsc_precise() in fastpath to fix performance issues in patch= 03. > Introduce new scheme for model choosing. Use RTE_GRAPH_MODEL_SELECT to ch= oose in > compile-time in patch 13, 15.(must have rte_graph_worker_model_set() to= help > other config func to do model specific things like alloc wq, collect st= ats) > Extract the common func clone_name() into graph_private.h for graph/node = clone in > patch 07.(new patch) > Use rte_graph->model in rte_graph_worker_model_set() instead of RTE_PER_L= CORE_*. > Add test case for all new APIs in patch 16(new patch). > Remove *_END line in enum rte_graph_worker_model in patch 04. > Add model check for graph lcore binding. > Rename workqueue as graph_mcore_dispatch_wq_node in patch 09. > Change all new model files/APIs with prefix _mcore_dispatch_. > Change description of new API, comments of func/structure to explicitly m= ention for > mcore dispatch model only. Add Doxygen comments. > Update l3fwd-graph with new scheme, Update doc. > Update MAINTAINERS. > Fix typo and format issues. > > V6: > Change rte_rdtsc() to rte_rdtsc_precise(). > Add union in rte_graph_param to configure models. > Remove memset in fastpath, add RTE_ASSERT for cloned graph. > Update copyright in patch 02. > Update l3fwd-graph node affinity, start from rx core successively. > > V5: > Fix CI build issues about dynamically update doc. > > V4: > Fix CI build issues about undefined reference of sched apis. > Remove inline for model setting. > > V3: > Fix CI build issues about TLS and typo. > > V2: > Use git mv to keep git history. > Use TLS for per-thread local storage. > Change model name to mcore dispatch. > Change API with specific mode name. > Split big patch. > Fix CI issues. > Rebase l3fwd-graph example. > Update doc and maintainers files. > > > Currently, rte_graph supports RTC (Run-To-Completion) model within each > of a single core. > RTC is one of the typical model of packet processing. Others like > Pipeline or Hybrid are lack of support. > > The patch set introduces a 'multicore dispatch' model selection which > is a self-reacting scheme according to the core affinity. > The new model enables a cross-core dispatching mechanism which employs a > scheduling work-queue to dispatch streams to other worker cores which > being associated with the destination node. When core flavor of the > destination node is a default 'current', the stream can be continue > executed as normal. > > Example: > 3-node graph targets 3-core budget > > RTC: > Graph: node-0 -> node-1 -> node-2 @Core0. > > + - - - - - - - - - - - - - - - - - - - - - + > ' Core #0/1/2 ' > ' ' > ' +--------+ +---------+ +--------+ ' > ' | Node-0 | --> | Node-1 | --> | Node-2 | ' > ' +--------+ +---------+ +--------+ ' > ' ' > + - - - - - - - - - - - - - - - - - - - - - + > > Dispatch: > > Graph topo: node-0 -> Core1; node-1 -> node-2; node-2 -> node-3. > Config graph: node-0 @Core0; node-1/3 @Core1; node-2 @Core2. > > .. code-block:: diff > > + - - - - - -+ +- - - - - - - - - - - - - + + - - - - - -+ > ' Core #0 ' ' Core #1 ' ' Core #2 ' > ' ' ' ' ' ' > ' +--------+ ' ' +--------+ +--------+ ' ' +--------+ ' > ' | Node-0 | - - - ->| Node-1 | | Node-3 |<- - - - | Node-2 | ' > ' +--------+ ' ' +--------+ +--------+ ' ' +--------+ ' > ' ' ' | ' ' ^ ' > + - - - - - -+ +- - -|- - - - - - - - - - + + - - -|- - -+ > | | > + - - - - - - - - - - - - - - - - + > > > The patch set has been break down as below: > > 1. Split graph worker into common and default model part. > 2. Inline graph node processing to make it reusable. > 3. Add set/get APIs to choose worker model. > 4. Introduce core affinity API to set the node run on specific worker cor= e. > (only use in new model) > 5. Introduce graph affinity API to bind one graph with specific worker > core. > 6. Introduce graph clone API. > 7. Introduce stream moving with scheduler work-queue in patch 8~12. > 8. Add stats for new models. > 9. Abstract default graph config process and integrate new model into > example/l3fwd-graph. Add new parameters for model choosing. > > We could run with new worker model by this: > ./dpdk-l3fwd-graph -l 8,9,10,11 -n 4 -- -p 0x1 --config=3D"(0,0,9)" -P > --model=3D"dispatch" > > References: > https://static.sched.com/hosted_files/dpdkuserspace22/a6/graph%20introduc= e%20remote%20dispatch%20for%20mult-core%20scaling.pdf > > > > > Zhirun Yan (16): > graph: rename rte_graph_work as common > graph: split graph worker into common and default model > graph: move node process into inline function > graph: add get/set graph worker model APIs > graph: introduce graph node core affinity API > graph: introduce graph bind unbind API > graph: move node clone name func into private as common > graph: introduce graph clone API for other worker core > graph: add structure for stream moving between cores > graph: introduce stream moving cross cores > graph: enable create and destroy graph scheduling workqueue > graph: introduce graph walk by cross-core dispatch > graph: enable graph multicore dispatch scheduler model > graph: add stats for mcore dispatch model > test/graph: add functional tests for mcore dispatch model > examples/l3fwd-graph: introduce mcore dispatch worker model > > MAINTAINERS | 3 +- > app/test/test_graph.c | 130 ++++ > doc/guides/prog_guide/graph_lib.rst | 71 ++- > doc/guides/sample_app_ug/l3_forward_graph.rst | 16 + > examples/l3fwd-graph/main.c | 230 +++++-- > lib/graph/graph.c | 161 +++++ > lib/graph/graph_debug.c | 6 + > lib/graph/graph_populate.c | 1 + > lib/graph/graph_private.h | 90 +++ > lib/graph/graph_stats.c | 76 ++- > lib/graph/meson.build | 9 +- > lib/graph/node.c | 27 +- > lib/graph/rte_graph.h | 65 ++ > lib/graph/rte_graph_model_mcore_dispatch.c | 191 ++++++ > lib/graph/rte_graph_model_mcore_dispatch.h | 134 ++++ > lib/graph/rte_graph_model_rtc.h | 46 ++ > lib/graph/rte_graph_worker.c | 39 ++ > lib/graph/rte_graph_worker.h | 503 +-------------- > lib/graph/rte_graph_worker_common.h | 598 ++++++++++++++++++ > lib/graph/version.map | 11 + > 20 files changed, 1839 insertions(+), 568 deletions(-) > create mode 100644 lib/graph/rte_graph_model_mcore_dispatch.c > create mode 100644 lib/graph/rte_graph_model_mcore_dispatch.h > create mode 100644 lib/graph/rte_graph_model_rtc.h > create mode 100644 lib/graph/rte_graph_worker.c > create mode 100644 lib/graph/rte_graph_worker_common.h > > -- > 2.37.2 >