From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id 2E4A8DED for ; Mon, 18 Jun 2018 16:36:37 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id z6-v6so15277459wma.0 for ; Mon, 18 Jun 2018 07:36:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:in-reply-to :references; bh=Myx9/mJrovJpoIAfW3EbrWjllXxrd+wjnuuSGNEU3eA=; b=qZ9LL4OHquYc5DnBuYRoE1osOCIpfn10siCJaWjH1OuJ05qlh8EU0n4riR/wajugXS 99C9XUpLmBXBHOmc3EUvurC/wRDo9xA1V6DlMpGqm3hRSFaAtieliiJFakv00JxPIDrU 1alTxtzJWsAqET2iwkmkAb2dRaCln2KH3lKVE+Zae/5SKYieDYEWXK0Bk2WyYzP7fCxx EPtIeQmheM1VC92qPy6ZWoWaW0vap9PlP1Fu6iK2uPPSfnSvk2JitE3GLtGkALyrJS3z DsVvppnDnxpxhNaHMNnDcbBn64tGIPGTnQBzgkhDtAUqsyz1/WS+NrV1gwYxfhKL8FCt a7Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=Myx9/mJrovJpoIAfW3EbrWjllXxrd+wjnuuSGNEU3eA=; b=Sy6WS73HlgFLEusNALl4A2/PeqjGtTDW+WZHtQOJZtGs8UIkzNGohSjUhEBYcLnW20 XjUb/eGZPgiBgSmQt2LfTVmLsMAberXT/ceR4dE8WSdxMk2IIU5ibBkYO8DA7ehYI2ds cn4MWdQPD9xosXMzZzz1SmHw3Eyf4ogJ/qvGlDx24hMRkN+d5dwvpI2OVE6SqT2JPSSU 8rVYabNGOHSLhT1zGZr8ZskELoyhpxFSHLOOiSuGGlfdPBTc2RJLC6/EJ53PLwnV8T0r yH3SjRgr9tv0P35Y9t8mnHFIdKlO4IOzGp+HoCjeNswMxJOxywwZA6S3xEapP3JmkejG bg+A== X-Gm-Message-State: APt69E0xA7P3TSAnmVRp6ORlNSK60pBjnM3+P5ftK4QLUFHPzgtQenD9 l8FEHx4SrcbsCzguvHfhF/YmRdraBw== X-Google-Smtp-Source: ADUXVKJrv0q+lCU8Es2mU7PEzI04UJDiJfKsinMyjY4j/oSGI88QUAirUffV51VVfY+qJ+iXWYJjHg== X-Received: by 2002:a1c:d287:: with SMTP id j129-v6mr8243613wmg.106.1529332596501; Mon, 18 Jun 2018 07:36:36 -0700 (PDT) Received: from laranjeiro-vm.dev.6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id a9-v6sm8609895wmh.38.2018.06.18.07.36.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Jun 2018 07:36:35 -0700 (PDT) From: Nelio Laranjeiro To: dev@dpdk.org, Adrien Mazarguil , Wenzhuo Lu , Jingjing Wu , Bernard Iremonger , Mohammad Abdul Awal Date: Mon, 18 Jun 2018 16:36:32 +0200 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: Subject: [dpdk-dev] [PATCH v3 1/2] app/testpmd: add VXLAN encap/decap support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 14:36:37 -0000 Due to the complex VXLAN_ENCAP flow action and based on the fact testpmd does not allocate memory, this patch adds a new command in testpmd to initialise a global structure containing the necessary information to make the outer layer of the packet. This same global structure will then be used by the flow command line in testpmd when the action vxlan_encap will be parsed, at this point, the conversion into such action becomes trivial. This global structure is only used for the encap action. Signed-off-by: Nelio Laranjeiro --- app/test-pmd/cmdline.c | 129 ++++++++++++++++++ app/test-pmd/cmdline_flow.c | 139 ++++++++++++++++++++ app/test-pmd/testpmd.c | 17 +++ app/test-pmd/testpmd.h | 17 +++ doc/guides/testpmd_app_ug/testpmd_funcs.rst | 13 ++ 5 files changed, 315 insertions(+) diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index 27e2aa8c8..93573606f 100644 --- a/app/test-pmd/cmdline.c +++ b/app/test-pmd/cmdline.c @@ -781,6 +781,14 @@ static void cmd_help_long_parsed(void *parsed_result, "port tm hierarchy commit (port_id) (clean_on_fail)\n" " Commit tm hierarchy.\n\n" + "vxlan ipv4|ipv6 vni udp-src udp-dst ip-src ip-dst" + " eth-src eth-dst\n" + " Configure the VXLAN encapsulation for flows.\n\n" + + "vxlan-with-vlan ipv4|ipv6 vni udp-src udp-dst ip-src ip-dst" + " vlan-tci eth-src eth-dst\n" + " Configure the VXLAN encapsulation for flows.\n\n" + , list_pkt_forwarding_modes() ); } @@ -14838,6 +14846,125 @@ cmdline_parse_inst_t cmd_set_port_tm_hierarchy_default = { }; #endif +/** Set VXLAN encapsulation details */ +struct cmd_set_vxlan_result { + cmdline_fixed_string_t set; + cmdline_fixed_string_t vxlan; + cmdline_fixed_string_t ip_version; + uint32_t vlan_present:1; + uint32_t vni; + uint16_t udp_src; + uint16_t udp_dst; + cmdline_ipaddr_t ip_src; + cmdline_ipaddr_t ip_dst; + uint16_t tci; + struct ether_addr eth_src; + struct ether_addr eth_dst; +}; + +cmdline_parse_token_string_t cmd_set_vxlan_set = + TOKEN_STRING_INITIALIZER(struct cmd_set_vxlan_result, set, "set"); +cmdline_parse_token_string_t cmd_set_vxlan_vxlan = + TOKEN_STRING_INITIALIZER(struct cmd_set_vxlan_result, vxlan, "vxlan"); +cmdline_parse_token_string_t cmd_set_vxlan_vxlan_with_vlan = + TOKEN_STRING_INITIALIZER(struct cmd_set_vxlan_result, vxlan, + "vxlan-with-vlan"); +cmdline_parse_token_string_t cmd_set_vxlan_ip_version = + TOKEN_STRING_INITIALIZER(struct cmd_set_vxlan_result, ip_version, + "ipv4#ipv6"); +cmdline_parse_token_num_t cmd_set_vxlan_vni = + TOKEN_NUM_INITIALIZER(struct cmd_set_vxlan_result, vni, UINT32); +cmdline_parse_token_num_t cmd_set_vxlan_udp_src = + TOKEN_NUM_INITIALIZER(struct cmd_set_vxlan_result, udp_src, UINT16); +cmdline_parse_token_num_t cmd_set_vxlan_udp_dst = + TOKEN_NUM_INITIALIZER(struct cmd_set_vxlan_result, udp_dst, UINT16); +cmdline_parse_token_ipaddr_t cmd_set_vxlan_ip_src = + TOKEN_IPADDR_INITIALIZER(struct cmd_set_vxlan_result, ip_src); +cmdline_parse_token_ipaddr_t cmd_set_vxlan_ip_dst = + TOKEN_IPADDR_INITIALIZER(struct cmd_set_vxlan_result, ip_dst); +cmdline_parse_token_num_t cmd_set_vxlan_vlan = + TOKEN_NUM_INITIALIZER(struct cmd_set_vxlan_result, tci, UINT16); +cmdline_parse_token_etheraddr_t cmd_set_vxlan_eth_src = + TOKEN_ETHERADDR_INITIALIZER(struct cmd_set_vxlan_result, eth_src); +cmdline_parse_token_etheraddr_t cmd_set_vxlan_eth_dst = + TOKEN_ETHERADDR_INITIALIZER(struct cmd_set_vxlan_result, eth_dst); + +static void cmd_set_vxlan_parsed(void *parsed_result, + __attribute__((unused)) struct cmdline *cl, + __attribute__((unused)) void *data) +{ + struct cmd_set_vxlan_result *res = parsed_result; + uint32_t vni = rte_cpu_to_be_32(res->vni) >> 8; + + if (strcmp(res->vxlan, "vxlan") == 0) + vxlan_encap_conf.select_vlan = 0; + else if (strcmp(res->vxlan, "vxlan-with-vlan") == 0) + vxlan_encap_conf.select_vlan = 1; + if (strcmp(res->ip_version, "ipv4") == 0) + vxlan_encap_conf.select_ipv4 = 1; + else if (strcmp(res->ip_version, "ipv6") == 0) + vxlan_encap_conf.select_ipv4 = 0; + else + return; + memcpy(vxlan_encap_conf.vni, &vni, 3); + vxlan_encap_conf.udp_src = rte_cpu_to_be_16(res->udp_src); + vxlan_encap_conf.udp_dst = rte_cpu_to_be_16(res->udp_dst); + if (vxlan_encap_conf.select_ipv4) { + IPV4_ADDR_TO_UINT(res->ip_src, vxlan_encap_conf.ipv4_src); + IPV4_ADDR_TO_UINT(res->ip_dst, vxlan_encap_conf.ipv4_dst); + } else { + IPV6_ADDR_TO_ARRAY(res->ip_src, vxlan_encap_conf.ipv6_src); + IPV6_ADDR_TO_ARRAY(res->ip_dst, vxlan_encap_conf.ipv6_dst); + } + if (vxlan_encap_conf.select_vlan) + vxlan_encap_conf.vlan_tci = rte_cpu_to_be_16(res->tci); + memcpy(vxlan_encap_conf.eth_src, res->eth_src.addr_bytes, + ETHER_ADDR_LEN); + memcpy(vxlan_encap_conf.eth_dst, res->eth_dst.addr_bytes, + ETHER_ADDR_LEN); +} + +cmdline_parse_inst_t cmd_set_vxlan = { + .f = cmd_set_vxlan_parsed, + .data = NULL, + .help_str = "set vxlan ipv4|ipv6 " + " ", + .tokens = { + (void *)&cmd_set_vxlan_set, + (void *)&cmd_set_vxlan_vxlan, + (void *)&cmd_set_vxlan_ip_version, + (void *)&cmd_set_vxlan_vni, + (void *)&cmd_set_vxlan_udp_src, + (void *)&cmd_set_vxlan_udp_dst, + (void *)&cmd_set_vxlan_ip_src, + (void *)&cmd_set_vxlan_ip_dst, + (void *)&cmd_set_vxlan_eth_src, + (void *)&cmd_set_vxlan_eth_dst, + NULL, + }, +}; + +cmdline_parse_inst_t cmd_set_vxlan_with_vlan = { + .f = cmd_set_vxlan_parsed, + .data = NULL, + .help_str = "set vxlan-with-vlan ipv4|ipv6 " + " ", + .tokens = { + (void *)&cmd_set_vxlan_set, + (void *)&cmd_set_vxlan_vxlan_with_vlan, + (void *)&cmd_set_vxlan_ip_version, + (void *)&cmd_set_vxlan_vni, + (void *)&cmd_set_vxlan_udp_src, + (void *)&cmd_set_vxlan_udp_dst, + (void *)&cmd_set_vxlan_ip_src, + (void *)&cmd_set_vxlan_ip_dst, + (void *)&cmd_set_vxlan_vlan, + (void *)&cmd_set_vxlan_eth_src, + (void *)&cmd_set_vxlan_eth_dst, + NULL, + }, +}; + /* Strict link priority scheduling mode setting */ static void cmd_strict_link_prio_parsed( @@ -17462,6 +17589,8 @@ cmdline_parse_ctx_t main_ctx[] = { #if defined RTE_LIBRTE_PMD_SOFTNIC && defined RTE_LIBRTE_SCHED (cmdline_parse_inst_t *)&cmd_set_port_tm_hierarchy_default, #endif + (cmdline_parse_inst_t *)&cmd_set_vxlan, + (cmdline_parse_inst_t *)&cmd_set_vxlan_with_vlan, (cmdline_parse_inst_t *)&cmd_ddp_add, (cmdline_parse_inst_t *)&cmd_ddp_del, (cmdline_parse_inst_t *)&cmd_ddp_get_list, diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c index 934cf7e90..a8b5221a6 100644 --- a/app/test-pmd/cmdline_flow.c +++ b/app/test-pmd/cmdline_flow.c @@ -239,6 +239,8 @@ enum index { ACTION_OF_POP_MPLS_ETHERTYPE, ACTION_OF_PUSH_MPLS, ACTION_OF_PUSH_MPLS_ETHERTYPE, + ACTION_VXLAN_ENCAP, + ACTION_VXLAN_DECAP, }; /** Maximum size for pattern in struct rte_flow_item_raw. */ @@ -258,6 +260,23 @@ struct action_rss_data { uint16_t queue[ACTION_RSS_QUEUE_NUM]; }; +/** Maximum number of items in struct rte_flow_action_vxlan_encap. */ +#define ACTION_VXLAN_ENCAP_ITEMS_NUM 6 + +/** Storage for struct rte_flow_action_vxlan_encap including external data. */ +struct action_vxlan_encap_data { + struct rte_flow_action_vxlan_encap conf; + struct rte_flow_item items[ACTION_VXLAN_ENCAP_ITEMS_NUM]; + struct rte_flow_item_eth item_eth; + struct rte_flow_item_vlan item_vlan; + union { + struct rte_flow_item_ipv4 item_ipv4; + struct rte_flow_item_ipv6 item_ipv6; + }; + struct rte_flow_item_udp item_udp; + struct rte_flow_item_vxlan item_vxlan; +}; + /** Maximum number of subsequent tokens and arguments on the stack. */ #define CTX_STACK_SIZE 16 @@ -775,6 +794,8 @@ static const enum index next_action[] = { ACTION_OF_SET_VLAN_PCP, ACTION_OF_POP_MPLS, ACTION_OF_PUSH_MPLS, + ACTION_VXLAN_ENCAP, + ACTION_VXLAN_DECAP, ZERO, }; @@ -905,6 +926,9 @@ static int parse_vc_action_rss_type(struct context *, const struct token *, static int parse_vc_action_rss_queue(struct context *, const struct token *, const char *, unsigned int, void *, unsigned int); +static int parse_vc_action_vxlan_encap(struct context *, const struct token *, + const char *, unsigned int, void *, + unsigned int); static int parse_destroy(struct context *, const struct token *, const char *, unsigned int, void *, unsigned int); @@ -2387,6 +2411,24 @@ static const struct token token_list[] = { ethertype)), .call = parse_vc_conf, }, + [ACTION_VXLAN_ENCAP] = { + .name = "vxlan_encap", + .help = "VXLAN encapsulation, uses configuration set by \"set" + " vxlan\"", + .priv = PRIV_ACTION(VXLAN_ENCAP, + sizeof(struct action_vxlan_encap_data)), + .next = NEXT(NEXT_ENTRY(ACTION_NEXT)), + .call = parse_vc_action_vxlan_encap, + }, + [ACTION_VXLAN_DECAP] = { + .name = "vxlan_decap", + .help = "Performs a decapsulation action by stripping all" + " headers of the VXLAN tunnel network overlay from the" + " matched flow.", + .priv = PRIV_ACTION(VXLAN_DECAP, 0), + .next = NEXT(NEXT_ENTRY(ACTION_NEXT)), + .call = parse_vc, + }, }; /** Remove and return last entry from argument stack. */ @@ -2951,6 +2993,103 @@ parse_vc_action_rss_queue(struct context *ctx, const struct token *token, return len; } +/** Parse VXLAN encap action. */ +static int +parse_vc_action_vxlan_encap(struct context *ctx, const struct token *token, + const char *str, unsigned int len, + void *buf, unsigned int size) +{ + struct buffer *out = buf; + struct rte_flow_action *action; + struct action_vxlan_encap_data *action_vxlan_encap_data; + int ret; + + ret = parse_vc(ctx, token, str, len, buf, size); + if (ret < 0) + return ret; + /* Nothing else to do if there is no buffer. */ + if (!out) + return ret; + if (!out->args.vc.actions_n) + return -1; + action = &out->args.vc.actions[out->args.vc.actions_n - 1]; + /* Point to selected object. */ + ctx->object = out->args.vc.data; + ctx->objmask = NULL; + /* Set up default configuration. */ + action_vxlan_encap_data = ctx->object; + *action_vxlan_encap_data = (struct action_vxlan_encap_data){ + .conf = (struct rte_flow_action_vxlan_encap){ + .definition = action_vxlan_encap_data->items, + }, + .items = { + { + .type = RTE_FLOW_ITEM_TYPE_ETH, + .spec = &action_vxlan_encap_data->item_eth, + .mask = &action_vxlan_encap_data->item_eth, + }, + { + .type = RTE_FLOW_ITEM_TYPE_VLAN, + .spec = &action_vxlan_encap_data->item_vlan, + .mask = &action_vxlan_encap_data->item_vlan, + }, + { + .type = RTE_FLOW_ITEM_TYPE_IPV4, + .spec = &action_vxlan_encap_data->item_ipv4, + .mask = &action_vxlan_encap_data->item_ipv4, + }, + { + .type = RTE_FLOW_ITEM_TYPE_UDP, + .spec = &action_vxlan_encap_data->item_udp, + .mask = &action_vxlan_encap_data->item_udp, + }, + { + .type = RTE_FLOW_ITEM_TYPE_VXLAN, + .spec = &action_vxlan_encap_data->item_vxlan, + .mask = &action_vxlan_encap_data->item_vxlan, + }, + { + .type = RTE_FLOW_ITEM_TYPE_END, + }, + }, + .item_eth.type = 0, + .item_vlan.tci = vxlan_encap_conf.vlan_tci, + .item_ipv4.hdr = { + .src_addr = vxlan_encap_conf.ipv4_src, + .dst_addr = vxlan_encap_conf.ipv4_dst, + }, + .item_udp.hdr = { + .src_port = vxlan_encap_conf.udp_src, + .dst_port = vxlan_encap_conf.udp_dst, + }, + .item_vxlan.flags = 0, + }; + memcpy(action_vxlan_encap_data->item_eth.dst.addr_bytes, + vxlan_encap_conf.eth_dst, ETHER_ADDR_LEN); + memcpy(action_vxlan_encap_data->item_eth.src.addr_bytes, + vxlan_encap_conf.eth_src, ETHER_ADDR_LEN); + if (!vxlan_encap_conf.select_ipv4) { + memcpy(&action_vxlan_encap_data->item_ipv6.hdr.src_addr, + &vxlan_encap_conf.ipv6_src, + sizeof(vxlan_encap_conf.ipv6_src)); + memcpy(&action_vxlan_encap_data->item_ipv6.hdr.dst_addr, + &vxlan_encap_conf.ipv6_dst, + sizeof(vxlan_encap_conf.ipv6_dst)); + action_vxlan_encap_data->items[2] = (struct rte_flow_item){ + .type = RTE_FLOW_ITEM_TYPE_IPV6, + .spec = &action_vxlan_encap_data->item_ipv6, + .mask = &action_vxlan_encap_data->item_ipv6, + }; + } + if (!vxlan_encap_conf.select_vlan) + action_vxlan_encap_data->items[1].type = + RTE_FLOW_ITEM_TYPE_VOID; + memcpy(action_vxlan_encap_data->item_vxlan.vni, vxlan_encap_conf.vni, + RTE_DIM(vxlan_encap_conf.vni)); + action->conf = &action_vxlan_encap_data->conf; + return ret; +} + /** Parse tokens for destroy command. */ static int parse_destroy(struct context *ctx, const struct token *token, diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 24c199844..af9b96f9b 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -393,6 +393,23 @@ uint8_t bitrate_enabled; struct gro_status gro_ports[RTE_MAX_ETHPORTS]; uint8_t gro_flush_cycles = GRO_DEFAULT_FLUSH_CYCLES; +struct vxlan_encap_conf vxlan_encap_conf = { + .select_ipv4 = 1, + .select_vlan = 0, + .vni = "\x00\x00\x00", + .udp_src = RTE_BE16(1), + .udp_dst = RTE_BE16(4789), + .ipv4_src = IPv4(127, 0, 0, 1), + .ipv4_dst = IPv4(255, 255, 255, 255), + .ipv6_src = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ipv6_dst = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x11\x11", + .vlan_tci = 0, + .eth_src = "\x00\x00\x00\x00\x00\x00", + .eth_dst = "\xff\xff\xff\xff\xff\xff", +}; + /* Forward function declarations */ static void map_port_queue_stats_mapping_registers(portid_t pi, struct rte_port *port); diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index f51cd9dd9..0d6618788 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -479,6 +479,23 @@ struct gso_status { extern struct gso_status gso_ports[RTE_MAX_ETHPORTS]; extern uint16_t gso_max_segment_size; +/* VXLAN encap/decap parameters. */ +struct vxlan_encap_conf { + uint32_t select_ipv4:1; + uint32_t select_vlan:1; + uint8_t vni[3]; + rte_be16_t udp_src; + rte_be16_t udp_dst; + rte_be32_t ipv4_src; + rte_be32_t ipv4_dst; + uint8_t ipv6_src[16]; + uint8_t ipv6_dst[16]; + rte_be16_t vlan_tci; + uint8_t eth_src[ETHER_ADDR_LEN]; + uint8_t eth_dst[ETHER_ADDR_LEN]; +}; +struct vxlan_encap_conf vxlan_encap_conf; + static inline unsigned int lcore_num(void) { diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst index 0d6fd50ca..30dc53046 100644 --- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst +++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst @@ -1534,6 +1534,13 @@ Enable or disable a per queue Tx offloading only on a specific Tx queue:: This command should be run when the port is stopped, or else it will fail. +Config VXLAN Encap outer layers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configure the outer layer to encapsulate a packet inside a VXLAN tunnel:: + + testpmd> set vxlan ipv4|ipv6 (vni) (udp-src) (udp-dst) (ip-src) (ip-dst) (mac-src) (mac-dst) + testpmd> set vxlan-with-vlan ipv4|ipv6 (vni) (udp-src) (udp-dst) (ip-src) (ip-dst) (vlan-tci) (mac-src) (mac-dst) Port Functions -------------- @@ -3650,6 +3657,12 @@ This section lists supported actions and their attributes, if any. - ``ethertype``: Ethertype. +- ``vxlan_encap``: Performs a VXLAN encapsulation, outer layer configuration + is done through `Config VXLAN Encap outer layers`_. + +- ``vxlan_decap``: Performs a decapsulation action by stripping all headers of + the VXLAN tunnel network overlay from the matched flow. + Destroying flow rules ~~~~~~~~~~~~~~~~~~~~~ -- 2.17.1