From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D063842C40; Tue, 6 Jun 2023 16:56:20 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 58A6242D98; Tue, 6 Jun 2023 16:55:39 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 2500842D82 for ; Tue, 6 Jun 2023 16:55:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686063337; x=1717599337; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JUa3REtKezppJy2yY4qVT5DBInLi5k/Hf4zZEP4w1Uc=; b=PcI/Xc1w0J7NoEkDDkYhkAIGdADbZq2aLC1fmb/h5QkLTTpxAAEQlalg gDl2J38LKh+f1S6P/XekEDKn1+l/rjk0kI9hwKBrfm7ivikerh/jzbQwE jjoopFvYWeEeZyEaG+seYOMHZxdeuGowNU6/BiwCbuNqPtcF9Dw4Yj7aW rKIJekRwb5NIQMtQD5dIKJ/Co7M+gG7Hevhc+JT54QyNzwajJYDSA4BbN FUoJtQMPi5itYaEVJnhM0LxkL7fS7taDSk+J7ZsnRHZssE37vECrpqgiX sfUETdPVEwWEzS2WJ22qLYCqcik/8cGoLMczKM/V3szGB/zccwtYVvajz A==; X-IronPort-AV: E=McAfee;i="6600,9927,10733"; a="356705672" X-IronPort-AV: E=Sophos;i="6.00,221,1681196400"; d="scan'208";a="356705672" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jun 2023 07:55:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10733"; a="1039222978" X-IronPort-AV: E=Sophos;i="6.00,221,1681196400"; d="scan'208";a="1039222978" Received: from dpdk-zhirun-lmm.sh.intel.com ([10.67.119.94]) by fmsmga005.fm.intel.com with ESMTP; 06 Jun 2023 07:55:25 -0700 From: Zhirun Yan To: dev@dpdk.org, jerinj@marvell.com, kirankumark@marvell.com, ndabilpuram@marvell.com, stephen@networkplumber.org, pbhagavatula@marvell.com, jerinjacobk@gmail.com Cc: cunming.liang@intel.com, haiyue.wang@intel.com, mattias.ronnblom@ericsson.com, Zhirun Yan Subject: [PATCH v8 15/17] examples/l3fwd-graph: introduce multicore dispatch worker model Date: Tue, 6 Jun 2023 22:47:44 +0800 Message-Id: <20230606144746.708388-16-zhirun.yan@intel.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230606144746.708388-1-zhirun.yan@intel.com> References: <20230605111923.3772260-1-zhirun.yan@intel.com> <20230606144746.708388-1-zhirun.yan@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add new parameter "model" to choose mcore dispatch or rtc model. And in dispatch model, the node will affinity to worker core successively. Note: RTE_GRAPH_MODEL_SELECT is set to GRAPH_MODEL_RTC by default. Must set model the same as RTE_GRAPH_MODEL_SELECT If set it as rtc or mcore dispatch explicitly. GRAPH_MODEL_MCORE_RUNTIME_SELECT means it could choose by model in runtime. Only support one RX node for mcore dispatch model in current implementation. ./dpdk-l3fwd-graph -l 8,9,10,11 -n 4 -- -p 0x1 --config="(0,0,9)" -P --model="dispatch" Signed-off-by: Haiyue Wang Signed-off-by: Cunming Liang Signed-off-by: Zhirun Yan --- examples/l3fwd-graph/main.c | 231 ++++++++++++++++++++++++++++++------ 1 file changed, 193 insertions(+), 38 deletions(-) diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c index 5feeab4f0f..77a5a98aec 100644 --- a/examples/l3fwd-graph/main.c +++ b/examples/l3fwd-graph/main.c @@ -23,6 +23,12 @@ #include #include #include +#define GRAPH_MODEL_RTC 0 /* Run-to-completion model, set by default. */ +#define GRAPH_MODEL_MCORE_DISPATCH 1 /* Dispatch model. */ +#define GRAPH_MODEL_MCORE_RUNTIME_SELECT 2 /* Support to select model by */ + /* parsing model in cmdline. */ +#undef RTE_GRAPH_MODEL_SELECT +#define RTE_GRAPH_MODEL_SELECT GRAPH_MODEL_RTC #include #include #include @@ -55,6 +61,9 @@ #define NB_SOCKETS 8 +/* Graph module */ +#define WORKER_MODEL_RTC "rtc" +#define WORKER_MODEL_MCORE_DISPATCH "dispatch" /* Static global variables used within this file. */ static uint16_t nb_rxd = RX_DESC_DEFAULT; static uint16_t nb_txd = TX_DESC_DEFAULT; @@ -88,6 +97,8 @@ struct lcore_rx_queue { char node_name[RTE_NODE_NAMESIZE]; }; +static uint32_t model_conf = RTE_GRAPH_MODEL_DEFAULT; + /* Lcore conf */ struct lcore_conf { uint16_t n_rx_queue; @@ -153,6 +164,19 @@ static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = { {RTE_IPV4(198, 18, 6, 0), 24, 6}, {RTE_IPV4(198, 18, 7, 0), 24, 7}, }; +static int +check_worker_model_params(void) +{ + if (model_conf == RTE_GRAPH_MODEL_MCORE_DISPATCH && + nb_lcore_params > 1) { + printf("Exceeded max number of lcore params for remote model: %hu\n", + nb_lcore_params); + return -1; + } + + return 0; +} + static int check_lcore_params(void) { @@ -276,6 +300,7 @@ print_usage(const char *prgname) " --eth-dest=X,MM:MM:MM:MM:MM:MM: Ethernet destination for " "port X\n" " --max-pkt-len PKTLEN: maximum packet length in decimal (64-9600)\n" + " --model NAME: walking model name, dispatch or rtc(by default)\n" " --no-numa: Disable numa awareness\n" " --per-port-pool: Use separate buffer pool per port\n" " --pcap-enable: Enables pcap capture\n" @@ -318,6 +343,19 @@ parse_max_pkt_len(const char *pktlen) return len; } +static void +parse_worker_model(const char *model) +{ + if (strcmp(model, WORKER_MODEL_MCORE_DISPATCH) == 0) + model_conf = RTE_GRAPH_MODEL_MCORE_DISPATCH; + else if (strcmp(model, WORKER_MODEL_RTC) == 0) + model_conf = RTE_GRAPH_MODEL_RTC; + + if (model_conf != RTE_GRAPH_MODEL_SELECT && + RTE_GRAPH_MODEL_SELECT <= RTE_GRAPH_MODEL_MCORE_DISPATCH) + rte_exit(EXIT_FAILURE, "Invalid worker model: %s", model); +} + static int parse_portmask(const char *portmask) { @@ -434,6 +472,8 @@ static const char short_options[] = "p:" /* portmask */ #define CMD_LINE_OPT_PCAP_ENABLE "pcap-enable" #define CMD_LINE_OPT_NUM_PKT_CAP "pcap-num-cap" #define CMD_LINE_OPT_PCAP_FILENAME "pcap-file-name" +#define CMD_LINE_OPT_WORKER_MODEL "model" + enum { /* Long options mapped to a short option */ @@ -449,6 +489,7 @@ enum { CMD_LINE_OPT_PARSE_PCAP_ENABLE, CMD_LINE_OPT_PARSE_NUM_PKT_CAP, CMD_LINE_OPT_PCAP_FILENAME_CAP, + CMD_LINE_OPT_WORKER_MODEL_TYPE, }; static const struct option lgopts[] = { @@ -460,6 +501,7 @@ static const struct option lgopts[] = { {CMD_LINE_OPT_PCAP_ENABLE, 0, 0, CMD_LINE_OPT_PARSE_PCAP_ENABLE}, {CMD_LINE_OPT_NUM_PKT_CAP, 1, 0, CMD_LINE_OPT_PARSE_NUM_PKT_CAP}, {CMD_LINE_OPT_PCAP_FILENAME, 1, 0, CMD_LINE_OPT_PCAP_FILENAME_CAP}, + {CMD_LINE_OPT_WORKER_MODEL, 1, 0, CMD_LINE_OPT_WORKER_MODEL_TYPE}, {NULL, 0, 0, 0}, }; @@ -551,6 +593,11 @@ parse_args(int argc, char **argv) printf("Pcap file name: %s\n", pcap_filename); break; + case CMD_LINE_OPT_WORKER_MODEL_TYPE: + printf("Use new worker model: %s\n", optarg); + parse_worker_model(optarg); + break; + default: print_usage(prgname); return -1; @@ -788,6 +835,142 @@ config_port_max_pkt_len(struct rte_eth_conf *conf, return 0; } +static void +graph_config_mcore_dispatch(struct rte_graph_param graph_conf) +{ + uint16_t nb_patterns = graph_conf.nb_node_patterns; + int worker_count = rte_lcore_count() - 1; + int main_lcore_id = rte_get_main_lcore(); + rte_graph_t main_graph_id = 0; + struct rte_node *node_tmp; + struct lcore_conf *qconf; + struct rte_graph *graph; + rte_graph_t graph_id; + rte_graph_off_t off; + int n_rx_node = 0; + int worker_lcore; + rte_node_t count; + int i, j; + int ret; + + for (j = 0; j < nb_lcore_params; j++) { + qconf = &lcore_conf[lcore_params[j].lcore_id]; + /* Add rx node patterns of all lcore */ + for (i = 0; i < qconf->n_rx_queue; i++) { + char *node_name = qconf->rx_queue_list[i].node_name; + unsigned int lcore_id = lcore_params[j].lcore_id; + + graph_conf.node_patterns[nb_patterns + n_rx_node + i] = node_name; + n_rx_node++; + ret = rte_graph_model_mcore_dispatch_node_lcore_affinity_set(node_name, + lcore_id); + if (ret == 0) + printf("Set node %s affinity to lcore %u\n", node_name, + lcore_params[j].lcore_id); + } + } + + graph_conf.nb_node_patterns = nb_patterns + n_rx_node; + graph_conf.socket_id = rte_lcore_to_socket_id(main_lcore_id); + + qconf = &lcore_conf[main_lcore_id]; + snprintf(qconf->name, sizeof(qconf->name), "worker_%u", + main_lcore_id); + + /* create main graph */ + main_graph_id = rte_graph_create(qconf->name, &graph_conf); + if (main_graph_id == RTE_GRAPH_ID_INVALID) + rte_exit(EXIT_FAILURE, + "rte_graph_create(): main_graph_id invalid for lcore %u\n", + main_lcore_id); + + /* set the graph model for the main graph */ + rte_graph_worker_model_set(RTE_GRAPH_MODEL_MCORE_DISPATCH); + qconf->graph_id = main_graph_id; + qconf->graph = rte_graph_lookup(qconf->name); + if (!qconf->graph) + rte_exit(EXIT_FAILURE, + "rte_graph_lookup(): graph %s not found\n", + qconf->name); + + graph = qconf->graph; + worker_lcore = lcore_params[nb_lcore_params - 1].lcore_id; + rte_graph_foreach_node(count, off, graph, node_tmp) { + /* Need to set the node Lcore affinity before clone graph for each lcore */ + if (node_tmp->dispatch.lcore_id == RTE_MAX_LCORE) { + worker_lcore = rte_get_next_lcore(worker_lcore, true, 1); + ret = rte_graph_model_mcore_dispatch_node_lcore_affinity_set(node_tmp->name, + worker_lcore); + if (ret == 0) + printf("Set node %s affinity to lcore %u\n", + node_tmp->name, worker_lcore); + } + } + + worker_lcore = main_lcore_id; + for (i = 0; i < worker_count; i++) { + worker_lcore = rte_get_next_lcore(worker_lcore, true, 1); + + qconf = &lcore_conf[worker_lcore]; + snprintf(qconf->name, sizeof(qconf->name), "cloned-%u", worker_lcore); + graph_id = rte_graph_clone(main_graph_id, qconf->name, &graph_conf); + ret = rte_graph_model_mcore_dispatch_core_bind(graph_id, worker_lcore); + if (ret == 0) + printf("bind graph %d to lcore %u\n", graph_id, worker_lcore); + + /* full cloned graph name */ + snprintf(qconf->name, sizeof(qconf->name), "%s", + rte_graph_id_to_name(graph_id)); + qconf->graph_id = graph_id; + qconf->graph = rte_graph_lookup(qconf->name); + if (!qconf->graph) + rte_exit(EXIT_FAILURE, + "Failed to lookup graph %s\n", + qconf->name); + continue; + } +} + +static void +graph_config_rtc(struct rte_graph_param graph_conf) +{ + uint16_t nb_patterns = graph_conf.nb_node_patterns; + struct lcore_conf *qconf; + rte_graph_t graph_id; + uint32_t lcore_id; + rte_edge_t i; + + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + if (rte_lcore_is_enabled(lcore_id) == 0) + continue; + + qconf = &lcore_conf[lcore_id]; + /* Skip graph creation if no source exists */ + if (!qconf->n_rx_queue) + continue; + /* Add rx node patterns of this lcore */ + for (i = 0; i < qconf->n_rx_queue; i++) { + graph_conf.node_patterns[nb_patterns + i] = + qconf->rx_queue_list[i].node_name; + } + graph_conf.nb_node_patterns = nb_patterns + i; + graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id); + snprintf(qconf->name, sizeof(qconf->name), "worker_%u", + lcore_id); + graph_id = rte_graph_create(qconf->name, &graph_conf); + if (graph_id == RTE_GRAPH_ID_INVALID) + rte_exit(EXIT_FAILURE, + "rte_graph_create(): graph_id invalid for lcore %u\n", + lcore_id); + qconf->graph_id = graph_id; + qconf->graph = rte_graph_lookup(qconf->name); + if (!qconf->graph) + rte_exit(EXIT_FAILURE, + "rte_graph_lookup(): graph %s not found\n", + qconf->name); + } +} + int main(int argc, char **argv) { @@ -840,6 +1023,9 @@ main(int argc, char **argv) if (check_lcore_params() < 0) rte_exit(EXIT_FAILURE, "check_lcore_params() failed\n"); + if (check_worker_model_params() < 0) + rte_exit(EXIT_FAILURE, "check_worker_model_params() failed\n"); + ret = init_lcore_rx_queues(); if (ret < 0) rte_exit(EXIT_FAILURE, "init_lcore_rx_queues() failed\n"); @@ -1079,51 +1265,20 @@ main(int argc, char **argv) memset(&graph_conf, 0, sizeof(graph_conf)); graph_conf.node_patterns = node_patterns; + graph_conf.nb_node_patterns = nb_patterns; /* Pcap config */ graph_conf.pcap_enable = pcap_trace_enable; graph_conf.num_pkt_to_capture = packet_to_capture; graph_conf.pcap_filename = pcap_filename; - for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { - rte_graph_t graph_id; - rte_edge_t i; - - if (rte_lcore_is_enabled(lcore_id) == 0) - continue; - - qconf = &lcore_conf[lcore_id]; - - /* Skip graph creation if no source exists */ - if (!qconf->n_rx_queue) - continue; - - /* Add rx node patterns of this lcore */ - for (i = 0; i < qconf->n_rx_queue; i++) { - graph_conf.node_patterns[nb_patterns + i] = - qconf->rx_queue_list[i].node_name; - } - - graph_conf.nb_node_patterns = nb_patterns + i; - graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id); - - snprintf(qconf->name, sizeof(qconf->name), "worker_%u", - lcore_id); - - graph_id = rte_graph_create(qconf->name, &graph_conf); - if (graph_id == RTE_GRAPH_ID_INVALID) - rte_exit(EXIT_FAILURE, - "rte_graph_create(): graph_id invalid" - " for lcore %u\n", lcore_id); + if (model_conf == RTE_GRAPH_MODEL_MCORE_DISPATCH) + graph_config_mcore_dispatch(graph_conf); + else + graph_config_rtc(graph_conf); - qconf->graph_id = graph_id; - qconf->graph = rte_graph_lookup(qconf->name); - /* >8 End of graph initialization. */ - if (!qconf->graph) - rte_exit(EXIT_FAILURE, - "rte_graph_lookup(): graph %s not found\n", - qconf->name); - } + rte_graph_worker_model_set(model_conf); + /* >8 End of graph initialization. */ memset(&rewrite_data, 0, sizeof(rewrite_data)); rewrite_len = sizeof(rewrite_data); -- 2.37.2