From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id D063842C40;
Tue, 6 Jun 2023 16:56:20 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 58A6242D98;
Tue, 6 Jun 2023 16:55:39 +0200 (CEST)
Received: from mga14.intel.com (mga14.intel.com [192.55.52.115])
by mails.dpdk.org (Postfix) with ESMTP id 2500842D82
for <dev@dpdk.org>; Tue, 6 Jun 2023 16:55:36 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;
d=intel.com; i=@intel.com; q=dns/txt; s=Intel;
t=1686063337; x=1717599337;
h=from:to:cc:subject:date:message-id:in-reply-to:
references:mime-version:content-transfer-encoding;
bh=JUa3REtKezppJy2yY4qVT5DBInLi5k/Hf4zZEP4w1Uc=;
b=PcI/Xc1w0J7NoEkDDkYhkAIGdADbZq2aLC1fmb/h5QkLTTpxAAEQlalg
gDl2J38LKh+f1S6P/XekEDKn1+l/rjk0kI9hwKBrfm7ivikerh/jzbQwE
jjoopFvYWeEeZyEaG+seYOMHZxdeuGowNU6/BiwCbuNqPtcF9Dw4Yj7aW
rKIJekRwb5NIQMtQD5dIKJ/Co7M+gG7Hevhc+JT54QyNzwajJYDSA4BbN
FUoJtQMPi5itYaEVJnhM0LxkL7fS7taDSk+J7ZsnRHZssE37vECrpqgiX
sfUETdPVEwWEzS2WJ22qLYCqcik/8cGoLMczKM/V3szGB/zccwtYVvajz A==;
X-IronPort-AV: E=McAfee;i="6600,9927,10733"; a="356705672"
X-IronPort-AV: E=Sophos;i="6.00,221,1681196400"; d="scan'208";a="356705672"
Received: from fmsmga005.fm.intel.com ([10.253.24.32])
by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
06 Jun 2023 07:55:27 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=McAfee;i="6600,9927,10733"; a="1039222978"
X-IronPort-AV: E=Sophos;i="6.00,221,1681196400"; d="scan'208";a="1039222978"
Received: from dpdk-zhirun-lmm.sh.intel.com ([10.67.119.94])
by fmsmga005.fm.intel.com with ESMTP; 06 Jun 2023 07:55:25 -0700
From: Zhirun Yan <zhirun.yan@intel.com>
To: dev@dpdk.org, jerinj@marvell.com, kirankumark@marvell.com,
ndabilpuram@marvell.com, stephen@networkplumber.org,
pbhagavatula@marvell.com, jerinjacobk@gmail.com
Cc: cunming.liang@intel.com, haiyue.wang@intel.com,
mattias.ronnblom@ericsson.com, Zhirun Yan <zhirun.yan@intel.com>
Subject: [PATCH v8 15/17] examples/l3fwd-graph: introduce multicore dispatch
worker model
Date: Tue, 6 Jun 2023 22:47:44 +0800
Message-Id: <20230606144746.708388-16-zhirun.yan@intel.com>
X-Mailer: git-send-email 2.37.2
In-Reply-To: <20230606144746.708388-1-zhirun.yan@intel.com>
References: <20230605111923.3772260-1-zhirun.yan@intel.com>
<20230606144746.708388-1-zhirun.yan@intel.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Add new parameter "model" to choose mcore dispatch or rtc model.
And in dispatch model, the node will affinity to worker core successively.
Note:
RTE_GRAPH_MODEL_SELECT is set to GRAPH_MODEL_RTC by default. Must set
model the same as RTE_GRAPH_MODEL_SELECT If set it as rtc or mcore
dispatch explicitly. GRAPH_MODEL_MCORE_RUNTIME_SELECT means it could
choose by model in runtime.
Only support one RX node for mcore dispatch model in current
implementation.
./dpdk-l3fwd-graph -l 8,9,10,11 -n 4 -- -p 0x1 --config="(0,0,9)" -P
--model="dispatch"
Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
Signed-off-by: Cunming Liang <cunming.liang@intel.com>
Signed-off-by: Zhirun Yan <zhirun.yan@intel.com>
---
examples/l3fwd-graph/main.c | 231 ++++++++++++++++++++++++++++++------
1 file changed, 193 insertions(+), 38 deletions(-)
diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c
index 5feeab4f0f..77a5a98aec 100644
--- a/examples/l3fwd-graph/main.c
+++ b/examples/l3fwd-graph/main.c
@@ -23,6 +23,12 @@
#include <rte_cycles.h>
#include <rte_eal.h>
#include <rte_ethdev.h>
+#define GRAPH_MODEL_RTC 0 /* Run-to-completion model, set by default. */
+#define GRAPH_MODEL_MCORE_DISPATCH 1 /* Dispatch model. */
+#define GRAPH_MODEL_MCORE_RUNTIME_SELECT 2 /* Support to select model by */
+ /* parsing model in cmdline. */
+#undef RTE_GRAPH_MODEL_SELECT
+#define RTE_GRAPH_MODEL_SELECT GRAPH_MODEL_RTC
#include <rte_graph_worker.h>
#include <rte_launch.h>
#include <rte_lcore.h>
@@ -55,6 +61,9 @@
#define NB_SOCKETS 8
+/* Graph module */
+#define WORKER_MODEL_RTC "rtc"
+#define WORKER_MODEL_MCORE_DISPATCH "dispatch"
/* Static global variables used within this file. */
static uint16_t nb_rxd = RX_DESC_DEFAULT;
static uint16_t nb_txd = TX_DESC_DEFAULT;
@@ -88,6 +97,8 @@ struct lcore_rx_queue {
char node_name[RTE_NODE_NAMESIZE];
};
+static uint32_t model_conf = RTE_GRAPH_MODEL_DEFAULT;
+
/* Lcore conf */
struct lcore_conf {
uint16_t n_rx_queue;
@@ -153,6 +164,19 @@ static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = {
{RTE_IPV4(198, 18, 6, 0), 24, 6}, {RTE_IPV4(198, 18, 7, 0), 24, 7},
};
+static int
+check_worker_model_params(void)
+{
+ if (model_conf == RTE_GRAPH_MODEL_MCORE_DISPATCH &&
+ nb_lcore_params > 1) {
+ printf("Exceeded max number of lcore params for remote model: %hu\n",
+ nb_lcore_params);
+ return -1;
+ }
+
+ return 0;
+}
+
static int
check_lcore_params(void)
{
@@ -276,6 +300,7 @@ print_usage(const char *prgname)
" --eth-dest=X,MM:MM:MM:MM:MM:MM: Ethernet destination for "
"port X\n"
" --max-pkt-len PKTLEN: maximum packet length in decimal (64-9600)\n"
+ " --model NAME: walking model name, dispatch or rtc(by default)\n"
" --no-numa: Disable numa awareness\n"
" --per-port-pool: Use separate buffer pool per port\n"
" --pcap-enable: Enables pcap capture\n"
@@ -318,6 +343,19 @@ parse_max_pkt_len(const char *pktlen)
return len;
}
+static void
+parse_worker_model(const char *model)
+{
+ if (strcmp(model, WORKER_MODEL_MCORE_DISPATCH) == 0)
+ model_conf = RTE_GRAPH_MODEL_MCORE_DISPATCH;
+ else if (strcmp(model, WORKER_MODEL_RTC) == 0)
+ model_conf = RTE_GRAPH_MODEL_RTC;
+
+ if (model_conf != RTE_GRAPH_MODEL_SELECT &&
+ RTE_GRAPH_MODEL_SELECT <= RTE_GRAPH_MODEL_MCORE_DISPATCH)
+ rte_exit(EXIT_FAILURE, "Invalid worker model: %s", model);
+}
+
static int
parse_portmask(const char *portmask)
{
@@ -434,6 +472,8 @@ static const char short_options[] = "p:" /* portmask */
#define CMD_LINE_OPT_PCAP_ENABLE "pcap-enable"
#define CMD_LINE_OPT_NUM_PKT_CAP "pcap-num-cap"
#define CMD_LINE_OPT_PCAP_FILENAME "pcap-file-name"
+#define CMD_LINE_OPT_WORKER_MODEL "model"
+
enum {
/* Long options mapped to a short option */
@@ -449,6 +489,7 @@ enum {
CMD_LINE_OPT_PARSE_PCAP_ENABLE,
CMD_LINE_OPT_PARSE_NUM_PKT_CAP,
CMD_LINE_OPT_PCAP_FILENAME_CAP,
+ CMD_LINE_OPT_WORKER_MODEL_TYPE,
};
static const struct option lgopts[] = {
@@ -460,6 +501,7 @@ static const struct option lgopts[] = {
{CMD_LINE_OPT_PCAP_ENABLE, 0, 0, CMD_LINE_OPT_PARSE_PCAP_ENABLE},
{CMD_LINE_OPT_NUM_PKT_CAP, 1, 0, CMD_LINE_OPT_PARSE_NUM_PKT_CAP},
{CMD_LINE_OPT_PCAP_FILENAME, 1, 0, CMD_LINE_OPT_PCAP_FILENAME_CAP},
+ {CMD_LINE_OPT_WORKER_MODEL, 1, 0, CMD_LINE_OPT_WORKER_MODEL_TYPE},
{NULL, 0, 0, 0},
};
@@ -551,6 +593,11 @@ parse_args(int argc, char **argv)
printf("Pcap file name: %s\n", pcap_filename);
break;
+ case CMD_LINE_OPT_WORKER_MODEL_TYPE:
+ printf("Use new worker model: %s\n", optarg);
+ parse_worker_model(optarg);
+ break;
+
default:
print_usage(prgname);
return -1;
@@ -788,6 +835,142 @@ config_port_max_pkt_len(struct rte_eth_conf *conf,
return 0;
}
+static void
+graph_config_mcore_dispatch(struct rte_graph_param graph_conf)
+{
+ uint16_t nb_patterns = graph_conf.nb_node_patterns;
+ int worker_count = rte_lcore_count() - 1;
+ int main_lcore_id = rte_get_main_lcore();
+ rte_graph_t main_graph_id = 0;
+ struct rte_node *node_tmp;
+ struct lcore_conf *qconf;
+ struct rte_graph *graph;
+ rte_graph_t graph_id;
+ rte_graph_off_t off;
+ int n_rx_node = 0;
+ int worker_lcore;
+ rte_node_t count;
+ int i, j;
+ int ret;
+
+ for (j = 0; j < nb_lcore_params; j++) {
+ qconf = &lcore_conf[lcore_params[j].lcore_id];
+ /* Add rx node patterns of all lcore */
+ for (i = 0; i < qconf->n_rx_queue; i++) {
+ char *node_name = qconf->rx_queue_list[i].node_name;
+ unsigned int lcore_id = lcore_params[j].lcore_id;
+
+ graph_conf.node_patterns[nb_patterns + n_rx_node + i] = node_name;
+ n_rx_node++;
+ ret = rte_graph_model_mcore_dispatch_node_lcore_affinity_set(node_name,
+ lcore_id);
+ if (ret == 0)
+ printf("Set node %s affinity to lcore %u\n", node_name,
+ lcore_params[j].lcore_id);
+ }
+ }
+
+ graph_conf.nb_node_patterns = nb_patterns + n_rx_node;
+ graph_conf.socket_id = rte_lcore_to_socket_id(main_lcore_id);
+
+ qconf = &lcore_conf[main_lcore_id];
+ snprintf(qconf->name, sizeof(qconf->name), "worker_%u",
+ main_lcore_id);
+
+ /* create main graph */
+ main_graph_id = rte_graph_create(qconf->name, &graph_conf);
+ if (main_graph_id == RTE_GRAPH_ID_INVALID)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_create(): main_graph_id invalid for lcore %u\n",
+ main_lcore_id);
+
+ /* set the graph model for the main graph */
+ rte_graph_worker_model_set(RTE_GRAPH_MODEL_MCORE_DISPATCH);
+ qconf->graph_id = main_graph_id;
+ qconf->graph = rte_graph_lookup(qconf->name);
+ if (!qconf->graph)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_lookup(): graph %s not found\n",
+ qconf->name);
+
+ graph = qconf->graph;
+ worker_lcore = lcore_params[nb_lcore_params - 1].lcore_id;
+ rte_graph_foreach_node(count, off, graph, node_tmp) {
+ /* Need to set the node Lcore affinity before clone graph for each lcore */
+ if (node_tmp->dispatch.lcore_id == RTE_MAX_LCORE) {
+ worker_lcore = rte_get_next_lcore(worker_lcore, true, 1);
+ ret = rte_graph_model_mcore_dispatch_node_lcore_affinity_set(node_tmp->name,
+ worker_lcore);
+ if (ret == 0)
+ printf("Set node %s affinity to lcore %u\n",
+ node_tmp->name, worker_lcore);
+ }
+ }
+
+ worker_lcore = main_lcore_id;
+ for (i = 0; i < worker_count; i++) {
+ worker_lcore = rte_get_next_lcore(worker_lcore, true, 1);
+
+ qconf = &lcore_conf[worker_lcore];
+ snprintf(qconf->name, sizeof(qconf->name), "cloned-%u", worker_lcore);
+ graph_id = rte_graph_clone(main_graph_id, qconf->name, &graph_conf);
+ ret = rte_graph_model_mcore_dispatch_core_bind(graph_id, worker_lcore);
+ if (ret == 0)
+ printf("bind graph %d to lcore %u\n", graph_id, worker_lcore);
+
+ /* full cloned graph name */
+ snprintf(qconf->name, sizeof(qconf->name), "%s",
+ rte_graph_id_to_name(graph_id));
+ qconf->graph_id = graph_id;
+ qconf->graph = rte_graph_lookup(qconf->name);
+ if (!qconf->graph)
+ rte_exit(EXIT_FAILURE,
+ "Failed to lookup graph %s\n",
+ qconf->name);
+ continue;
+ }
+}
+
+static void
+graph_config_rtc(struct rte_graph_param graph_conf)
+{
+ uint16_t nb_patterns = graph_conf.nb_node_patterns;
+ struct lcore_conf *qconf;
+ rte_graph_t graph_id;
+ uint32_t lcore_id;
+ rte_edge_t i;
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+
+ qconf = &lcore_conf[lcore_id];
+ /* Skip graph creation if no source exists */
+ if (!qconf->n_rx_queue)
+ continue;
+ /* Add rx node patterns of this lcore */
+ for (i = 0; i < qconf->n_rx_queue; i++) {
+ graph_conf.node_patterns[nb_patterns + i] =
+ qconf->rx_queue_list[i].node_name;
+ }
+ graph_conf.nb_node_patterns = nb_patterns + i;
+ graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
+ snprintf(qconf->name, sizeof(qconf->name), "worker_%u",
+ lcore_id);
+ graph_id = rte_graph_create(qconf->name, &graph_conf);
+ if (graph_id == RTE_GRAPH_ID_INVALID)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_create(): graph_id invalid for lcore %u\n",
+ lcore_id);
+ qconf->graph_id = graph_id;
+ qconf->graph = rte_graph_lookup(qconf->name);
+ if (!qconf->graph)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_lookup(): graph %s not found\n",
+ qconf->name);
+ }
+}
+
int
main(int argc, char **argv)
{
@@ -840,6 +1023,9 @@ main(int argc, char **argv)
if (check_lcore_params() < 0)
rte_exit(EXIT_FAILURE, "check_lcore_params() failed\n");
+ if (check_worker_model_params() < 0)
+ rte_exit(EXIT_FAILURE, "check_worker_model_params() failed\n");
+
ret = init_lcore_rx_queues();
if (ret < 0)
rte_exit(EXIT_FAILURE, "init_lcore_rx_queues() failed\n");
@@ -1079,51 +1265,20 @@ main(int argc, char **argv)
memset(&graph_conf, 0, sizeof(graph_conf));
graph_conf.node_patterns = node_patterns;
+ graph_conf.nb_node_patterns = nb_patterns;
/* Pcap config */
graph_conf.pcap_enable = pcap_trace_enable;
graph_conf.num_pkt_to_capture = packet_to_capture;
graph_conf.pcap_filename = pcap_filename;
- for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
- rte_graph_t graph_id;
- rte_edge_t i;
-
- if (rte_lcore_is_enabled(lcore_id) == 0)
- continue;
-
- qconf = &lcore_conf[lcore_id];
-
- /* Skip graph creation if no source exists */
- if (!qconf->n_rx_queue)
- continue;
-
- /* Add rx node patterns of this lcore */
- for (i = 0; i < qconf->n_rx_queue; i++) {
- graph_conf.node_patterns[nb_patterns + i] =
- qconf->rx_queue_list[i].node_name;
- }
-
- graph_conf.nb_node_patterns = nb_patterns + i;
- graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
-
- snprintf(qconf->name, sizeof(qconf->name), "worker_%u",
- lcore_id);
-
- graph_id = rte_graph_create(qconf->name, &graph_conf);
- if (graph_id == RTE_GRAPH_ID_INVALID)
- rte_exit(EXIT_FAILURE,
- "rte_graph_create(): graph_id invalid"
- " for lcore %u\n", lcore_id);
+ if (model_conf == RTE_GRAPH_MODEL_MCORE_DISPATCH)
+ graph_config_mcore_dispatch(graph_conf);
+ else
+ graph_config_rtc(graph_conf);
- qconf->graph_id = graph_id;
- qconf->graph = rte_graph_lookup(qconf->name);
- /* >8 End of graph initialization. */
- if (!qconf->graph)
- rte_exit(EXIT_FAILURE,
- "rte_graph_lookup(): graph %s not found\n",
- qconf->name);
- }
+ rte_graph_worker_model_set(model_conf);
+ /* >8 End of graph initialization. */
memset(&rewrite_data, 0, sizeof(rewrite_data));
rewrite_len = sizeof(rewrite_data);
--
2.37.2