From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-eopbgr40045.outbound.protection.outlook.com [40.107.4.45]) by dpdk.org (Postfix) with ESMTP id A54741B499 for ; Wed, 10 Oct 2018 04:54:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cKhuTPZDof5qRy+YXH0fvFc2wE5n4z/YP9AL/s6qhGk=; b=wUwq44VQOcQwZRLCSwUqgQl8DG8TMU9myfMoLblT0zopfCLoyjdhUOrZ4jSodAYZTQsg0v5irHo6glMnk2DiYbpznSVtgjDmAb8J/ZBUReEjz1z0CqfvtIoP+YT4BUiGuyAsDGaLLHGIgk/jTyoDIwFKUwId7b0kpefTrNH3Csk= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4074.eurprd05.prod.outlook.com (52.134.72.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1207.22; Wed, 10 Oct 2018 02:54:14 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045%2]) with mapi id 15.20.1207.029; Wed, 10 Oct 2018 02:54:14 +0000 From: Yongseok Koh To: Shahaf Shuler CC: "dev@dpdk.org" , Yongseok Koh Thread-Topic: [PATCH v3] net/mlx5: support multiple groups and jump action Thread-Index: AQHUYESGx/H89kE530uHHoJw8okcLw== Date: Wed, 10 Oct 2018 02:54:13 +0000 Message-ID: <20181010025405.48555-1-yskoh@mellanox.com> References: <20181003015640.36306-1-yskoh@mellanox.com> In-Reply-To: <20181003015640.36306-1-yskoh@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR07CA0017.namprd07.prod.outlook.com (2603:10b6:a02:bc::30) To DB3PR0502MB3980.eurprd05.prod.outlook.com (2603:10a6:8:10::27) authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4074; 6:7J4mjQoyXLrwwlqqcac2DJTdSkO1h+7HSuX5+JnLHUxxp1y06foaVAihEZvfVkSUmGDrvV7CbKlEg1KgEdxsFR40TfxOeP2dNanYKuFUZd8zD7Ou0B/Vmhl2cmLZx+DYYccNcI7NvrFygKYuCF8eWBUADe8lU1JON0dd/xY+Fj3+FaNxWBn1SSdN0zfBMK5Hyopqw890n7x67CyAo592TqC1tDa4vCYDDK1eY5Fh6V2NEkRsVSSVw6FKhmQBkApANZNccP7Uar9wmhbV3Pcr8NFuyxjSkqdylo2lS6IYmPqGxkq3iDldckVG9nuSZ2Cf99Lx4raHuYkt90+WZssOU00/WvMj1aSzUnadXgbmxyCXoWs83faeJ0VjJ2sRMuIyMdL+kQSfoveztTEPzhbxj96y+tIXbHHqpdnEao/snhLspv6pTp1JoXKXrAx25n19f1bnjYA6fPK4hdlJtzYb0g==; 5:1Ca7RRuPF+Uc5/h3FP8OWR/cvBZuzNR1RV4p7j6MloLvcW3Hi06MUjkI3O36JWREVBnfa2nSppRlu4TtwETeOJVF+vTyTh0DTOmowddpGm7FymPnpQX0hnCPJYvTy2fnPLS8tFjUMcCDE6u1xZNMITk7Rk0x0MmoauzwYCVo6co=; 7:RmDuxERH9U2k+ApQVFtccwusOViMfUvaFUqwtIGUnchVvVWefDs2eHsdZRrbgrvoIPfpo3CgwnyPDSwodXBeXVMqDIyTXbkJZhk1qKFPeonokZ3Hh2I6AEb+fWpVAVIhjFv/UZ+rTpdWo4kQPSLSgJgMEUvCIkvQG2bvhDu7+yNa1dgm+P29SIvfWz07gjeMtQ2FQRdl7LJn54fPBDN2adibGPDD00H8tUdHQUHDoZV87eud5rCc7s2StFr/AJfT x-ms-office365-filtering-correlation-id: 8bbe5e16-80dc-40b8-801f-08d62e5ba914 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4074; x-ms-traffictypediagnostic: DB3PR0502MB4074: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(3231355)(944501410)(52105095)(6055026)(149066)(150057)(6041310)(20161123560045)(20161123562045)(20161123564045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051); SRVR:DB3PR0502MB4074; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4074; x-forefront-prvs: 08213D42D3 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(346002)(136003)(396003)(366004)(39860400002)(199004)(189003)(1076002)(71200400001)(71190400001)(52116002)(54906003)(6116002)(37006003)(3846002)(53936002)(575784001)(6436002)(6512007)(386003)(6506007)(6486002)(106356001)(105586002)(97736004)(446003)(11346002)(476003)(2616005)(102836004)(68736007)(486006)(186003)(76176011)(99286004)(2906002)(26005)(86362001)(107886003)(316002)(14454004)(5250100002)(8676002)(5660300001)(6636002)(6862004)(81166006)(81156014)(8936002)(478600001)(4744004)(2900100001)(256004)(14444005)(7736002)(66066001)(4326008)(305945005)(36756003)(25786009); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4074; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: G+AJof+RmkTRPuHaR98qdl1pQDC3l9ubV/fFqjFibFjLewg5CN+mrGiuEMO1v/Bvt/5sOHkYyhlvO0DfhAv9VhDDMcWFnEymJAvucLYx9i0bkYkPjBa4Krc5dOAYXac2QalccPpqksTQKHWTMiSZQunH5I61Hse7EUn4D1aKsLM/RnqoO0ofyMb7E1PLW+7i86GETwWFLm98GwbRkQY3sphipCGsZwoPxhHh+fcPe0fD/LHG8JtdNwGgi7CUQgyb1iUg3EgUMIHms0oK30DM4FJhWkNfXU7934IaBkV/hz7bfsp44e9AyjtJP7x/OmhGvXVRD1paih3bbP2gC2wmMVw4QshyKb46vTS+Si9fpt0= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8bbe5e16-80dc-40b8-801f-08d62e5ba914 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Oct 2018 02:54:13.9202 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4074 Subject: [dpdk-dev] [PATCH v3] net/mlx5: support multiple groups and jump action X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Oct 2018 02:54:16 -0000 rte_flow has 'group' attribute and 'jump' action in order to support multiple groups. This feature is known as multi-table support ('chain' in linux TC flower) in general because a group means a table of flows. Example commands are: flow create 0 transfer priority 1 ingress pattern eth / vlan vid is 100 / end actions jump group 1 / end flow create 0 transfer priority 1 ingress pattern eth / vlan vid is 200 / end actions jump group 2 / end flow create 0 transfer group 1 priority 2 ingress pattern eth / vlan vid is 100 / ipv4 dst spec 192.168.40.0 dst prefix 24 / end actions drop / end flow create 0 transfer group 1 priority 2 ingress pattern end actions of_pop_vlan / port_id id 1 / end flow create 0 transfer group 2 priority 2 ingress pattern eth / vlan vid is 200 / ipv4 dst spec 192.168.40.0 dst prefix 24 / end actions of_pop_vlan / port_id id 2 / end flow create 0 transfer group 2 priority 2 ingress pattern end actions port_id id 2 / end With theses flows, if a packet having vlan 200 and src_ip as 192.168.40.1, this packet will firstly hit the 1st flow. Then it will hit the 5th flow because of the 'jump' action. As a result, the packet will be forwarded to port 2 (VF representor) with vlan tag being stripped off. If the packet had vlan 100 instead, it would be dropped by the 3rd flow. Signed-off-by: Yongseok Koh Acked-by: Ori Kam --- v3: * reorder build macros v2: * drop ethdev patch as it had already been fixed by Adrien's patch * move TCF macros from mlx5_flow.h to mlx5_flow_tcf.c drivers/net/mlx5/Makefile | 10 +++++ drivers/net/mlx5/meson.build | 4 ++ drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_tcf.c | 82 +++++++++++++++++++++++++++++++++++-= ---- 4 files changed, 88 insertions(+), 9 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index ca1de9f210..928837bb85 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -207,6 +207,11 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config= -h.sh enum IFLA_PHYS_PORT_NAME \ $(AUTOCONF_OUTPUT) $Q sh -- '$<' '$@' \ + HAVE_TCA_CHAIN \ + linux/rtnetlink.h \ + enum TCA_CHAIN \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ HAVE_TCA_FLOWER_ACT \ linux/pkt_cls.h \ enum TCA_FLOWER_ACT \ @@ -342,6 +347,11 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config= -h.sh enum TCA_FLOWER_KEY_VLAN_ETH_TYPE \ $(AUTOCONF_OUTPUT) $Q sh -- '$<' '$@' \ + HAVE_TC_ACT_GOTO_CHAIN \ + linux/pkt_cls.h \ + define TC_ACT_GOTO_CHAIN \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ HAVE_TC_ACT_VLAN \ linux/tc_act/tc_vlan.h \ enum TCA_VLAN_PUSH_VLAN_PRIORITY \ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index fd93ac1625..025dcc9e96 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -126,6 +126,8 @@ if build 'IFLA_PHYS_SWITCH_ID' ], [ 'HAVE_IFLA_PHYS_PORT_NAME', 'linux/if_link.h', 'IFLA_PHYS_PORT_NAME' ], + [ 'HAVE_TCA_CHAIN', 'linux/rtnetlink.h', + 'TCA_CHAIN' ], [ 'HAVE_TCA_FLOWER_ACT', 'linux/pkt_cls.h', 'TCA_FLOWER_ACT' ], [ 'HAVE_TCA_FLOWER_FLAGS', 'linux/pkt_cls.h', @@ -180,6 +182,8 @@ if build 'TCA_FLOWER_KEY_VLAN_PRIO' ], [ 'HAVE_TCA_FLOWER_KEY_VLAN_ETH_TYPE', 'linux/pkt_cls.h', 'TCA_FLOWER_KEY_VLAN_ETH_TYPE' ], + [ 'HAVE_TC_ACT_GOTO_CHAIN', 'linux/pkt_cls.h', + 'TC_ACT_GOTO_CHAIN' ], [ 'HAVE_TC_ACT_VLAN', 'linux/tc_act/tc_vlan.h', 'TCA_VLAN_PUSH_VLAN_PRIORITY' ], [ 'HAVE_RDMA_NL_NLDEV', 'rdma/rdma_netlink.h', diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 12de841e86..3ed0ddc585 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -78,6 +78,7 @@ #define MLX5_FLOW_ACTION_OF_PUSH_VLAN (1u << 8) #define MLX5_FLOW_ACTION_OF_SET_VLAN_VID (1u << 9) #define MLX5_FLOW_ACTION_OF_SET_VLAN_PCP (1u << 10) +#define MLX5_FLOW_ACTION_JUMP (1u << 11) =20 #define MLX5_FLOW_FATE_ACTIONS \ (MLX5_FLOW_ACTION_DROP | MLX5_FLOW_ACTION_QUEUE | MLX5_FLOW_ACTION_RSS) diff --git a/drivers/net/mlx5/mlx5_flow_tcf.c b/drivers/net/mlx5/mlx5_flow_= tcf.c index 91f6ef6786..fbc4c2bb7c 100644 --- a/drivers/net/mlx5/mlx5_flow_tcf.c +++ b/drivers/net/mlx5/mlx5_flow_tcf.c @@ -148,6 +148,12 @@ struct tc_vlan { #ifndef HAVE_TCA_FLOWER_KEY_VLAN_ETH_TYPE #define TCA_FLOWER_KEY_VLAN_ETH_TYPE 25 #endif +#ifndef HAVE_TCA_CHAIN +#define TCA_CHAIN 11 +#endif +#ifndef HAVE_TC_ACT_GOTO_CHAIN +#define TC_ACT_GOTO_CHAIN 0x20000000 +#endif =20 #ifndef IPV6_ADDR_LEN #define IPV6_ADDR_LEN 16 @@ -225,7 +231,13 @@ struct flow_tcf_ptoi { unsigned int ifindex; /**< Network interface index. */ }; =20 -#define MLX5_TCF_FATE_ACTIONS (MLX5_FLOW_ACTION_DROP | MLX5_FLOW_ACTION_PO= RT_ID) +/* Due to a limitation on driver/FW. */ +#define MLX5_TCF_GROUP_ID_MAX 3 +#define MLX5_TCF_GROUP_PRIORITY_MAX 14 + +#define MLX5_TCF_FATE_ACTIONS \ + (MLX5_FLOW_ACTION_DROP | MLX5_FLOW_ACTION_PORT_ID | \ + MLX5_FLOW_ACTION_JUMP) #define MLX5_TCF_VLAN_ACTIONS \ (MLX5_FLOW_ACTION_OF_POP_VLAN | MLX5_FLOW_ACTION_OF_PUSH_VLAN | \ MLX5_FLOW_ACTION_OF_SET_VLAN_VID | MLX5_FLOW_ACTION_OF_SET_VLAN_PCP) @@ -370,14 +382,25 @@ flow_tcf_validate_attributes(const struct rte_flow_at= tr *attr, struct rte_flow_error *error) { /* - * Supported attributes: no groups, some priorities and ingress only. - * Don't care about transfer as it is the caller's problem. + * Supported attributes: groups, some priorities and ingress only. + * group is supported only if kernel supports chain. Don't care about + * transfer as it is the caller's problem. */ - if (attr->group) + if (attr->group > MLX5_TCF_GROUP_ID_MAX) return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ATTR_GROUP, attr, - "groups are not supported"); - if (attr->priority > 0xfffe) + "group ID larger than " + RTE_STR(MLX5_TCF_GROUP_ID_MAX) + " isn't supported"); + else if (attr->group > 0 && + attr->priority > MLX5_TCF_GROUP_PRIORITY_MAX) + return rte_flow_error_set(error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ATTR_PRIORITY, + attr, + "lowest priority level is " + RTE_STR(MLX5_TCF_GROUP_PRIORITY_MAX) + " when group is configured"); + else if (attr->priority > 0xfffe) return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ATTR_PRIORITY, attr, @@ -428,6 +451,7 @@ flow_tcf_validate(struct rte_eth_dev *dev, } spec, mask; union { const struct rte_flow_action_port_id *port_id; + const struct rte_flow_action_jump *jump; const struct rte_flow_action_of_push_vlan *of_push_vlan; const struct rte_flow_action_of_set_vlan_vid * of_set_vlan_vid; @@ -675,6 +699,16 @@ flow_tcf_validate(struct rte_eth_dev *dev, action_flags |=3D MLX5_FLOW_ACTION_PORT_ID; port_id_dev =3D &rte_eth_devices[conf.port_id->id]; break; + case RTE_FLOW_ACTION_TYPE_JUMP: + conf.jump =3D actions->conf; + if (attr->group >=3D conf.jump->group) + return rte_flow_error_set + (error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ACTION, + actions, + "can't jump to a group backward"); + action_flags |=3D MLX5_FLOW_ACTION_JUMP; + break; case RTE_FLOW_ACTION_TYPE_DROP: if (action_flags & MLX5_TCF_FATE_ACTIONS) return rte_flow_error_set @@ -757,7 +791,8 @@ flow_tcf_validate(struct rte_eth_dev *dev, * Maximum size of memory for items. */ static int -flow_tcf_get_items_and_size(const struct rte_flow_item items[], +flow_tcf_get_items_and_size(const struct rte_flow_attr *attr, + const struct rte_flow_item items[], uint64_t *item_flags) { int size =3D 0; @@ -766,6 +801,8 @@ flow_tcf_get_items_and_size(const struct rte_flow_item = items[], size +=3D SZ_NLATTR_STRZ_OF("flower") + SZ_NLATTR_NEST + /* TCA_OPTIONS. */ SZ_NLATTR_TYPE_OF(uint32_t); /* TCA_CLS_FLAGS_SKIP_SW. */ + if (attr->group > 0) + size +=3D SZ_NLATTR_TYPE_OF(uint32_t); /* TCA_CHAIN. */ for (; items->type !=3D RTE_FLOW_ITEM_TYPE_END; items++) { switch (items->type) { case RTE_FLOW_ITEM_TYPE_VOID: @@ -855,6 +892,13 @@ flow_tcf_get_actions_and_size(const struct rte_flow_ac= tion actions[], SZ_NLATTR_TYPE_OF(struct tc_mirred); flags |=3D MLX5_FLOW_ACTION_PORT_ID; break; + case RTE_FLOW_ACTION_TYPE_JUMP: + size +=3D SZ_NLATTR_NEST + /* na_act_index. */ + SZ_NLATTR_STRZ_OF("gact") + + SZ_NLATTR_NEST + /* TCA_ACT_OPTIONS. */ + SZ_NLATTR_TYPE_OF(struct tc_gact); + flags |=3D MLX5_FLOW_ACTION_JUMP; + break; case RTE_FLOW_ACTION_TYPE_DROP: size +=3D SZ_NLATTR_NEST + /* na_act_index. */ SZ_NLATTR_STRZ_OF("gact") + @@ -940,7 +984,7 @@ flow_tcf_nl_brand(struct nlmsghdr *nlh, uint32_t handle= ) * otherwise NULL and rte_ernno is set. */ static struct mlx5_flow * -flow_tcf_prepare(const struct rte_flow_attr *attr __rte_unused, +flow_tcf_prepare(const struct rte_flow_attr *attr, const struct rte_flow_item items[], const struct rte_flow_action actions[], uint64_t *item_flags, uint64_t *action_flags, @@ -953,7 +997,7 @@ flow_tcf_prepare(const struct rte_flow_attr *attr __rte= _unused, struct nlmsghdr *nlh; struct tcmsg *tcm; =20 - size +=3D flow_tcf_get_items_and_size(items, item_flags); + size +=3D flow_tcf_get_items_and_size(attr, items, item_flags); size +=3D flow_tcf_get_actions_and_size(actions, action_flags); dev_flow =3D rte_zmalloc(__func__, size, MNL_ALIGNTO); if (!dev_flow) { @@ -1024,6 +1068,7 @@ flow_tcf_translate(struct rte_eth_dev *dev, struct ml= x5_flow *dev_flow, } spec, mask; union { const struct rte_flow_action_port_id *port_id; + const struct rte_flow_action_jump *jump; const struct rte_flow_action_of_push_vlan *of_push_vlan; const struct rte_flow_action_of_set_vlan_vid * of_set_vlan_vid; @@ -1058,6 +1103,8 @@ flow_tcf_translate(struct rte_eth_dev *dev, struct ml= x5_flow *dev_flow, */ tcm->tcm_info =3D TC_H_MAKE((attr->priority + 1) << 16, RTE_BE16(ETH_P_ALL)); + if (attr->group > 0) + mnl_attr_put_u32(nlh, TCA_CHAIN, attr->group); mnl_attr_put_strz(nlh, TCA_KIND, "flower"); na_flower =3D mnl_attr_nest_start(nlh, TCA_OPTIONS); mnl_attr_put_u32(nlh, TCA_FLOWER_FLAGS, TCA_CLS_FLAGS_SKIP_SW); @@ -1332,6 +1379,23 @@ flow_tcf_translate(struct rte_eth_dev *dev, struct m= lx5_flow *dev_flow, mnl_attr_nest_end(nlh, na_act); mnl_attr_nest_end(nlh, na_act_index); break; + case RTE_FLOW_ACTION_TYPE_JUMP: + conf.jump =3D actions->conf; + na_act_index =3D + mnl_attr_nest_start(nlh, na_act_index_cur++); + assert(na_act_index); + mnl_attr_put_strz(nlh, TCA_ACT_KIND, "gact"); + na_act =3D mnl_attr_nest_start(nlh, TCA_ACT_OPTIONS); + assert(na_act); + mnl_attr_put(nlh, TCA_GACT_PARMS, + sizeof(struct tc_gact), + &(struct tc_gact){ + .action =3D TC_ACT_GOTO_CHAIN | + conf.jump->group, + }); + mnl_attr_nest_end(nlh, na_act); + mnl_attr_nest_end(nlh, na_act_index); + break; case RTE_FLOW_ACTION_TYPE_DROP: na_act_index =3D mnl_attr_nest_start(nlh, na_act_index_cur++); --=20 2.11.0