From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10089.outbound.protection.outlook.com [40.107.1.89]) by dpdk.org (Postfix) with ESMTP id 049F91B10B for ; Tue, 25 Sep 2018 01:17:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aLuGNwy9UpscyPlVpI74Dks+R555UvoH3M2LnD/Dk+U=; b=jGyi9KcF3F0BvhBoZLiv5gysp+yMzZ0A3GszZVC/paAjx8CcY4HM+PJMhDZDhDX9jMkwkfU+g5H/bzxteju5oq5ddTRYQRdrvim7it5az3VR29R2ALcMEmPBoL2DyWj+hSSA1EabTQEKlFarFmJvDVhBZNXxnL/FL4h9emdfRuI= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB3996.eurprd05.prod.outlook.com (52.134.65.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1164.25; Mon, 24 Sep 2018 23:17:39 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045%2]) with mapi id 15.20.1164.017; Mon, 24 Sep 2018 23:17:39 +0000 From: Yongseok Koh To: Thomas Monjalon , Shahaf Shuler CC: "dev@dpdk.org" , Ori Kam Thread-Topic: [PATCH v3 03/11] net/mlx5: add flow translate function Thread-Index: AQHUVFzJ7rw7u0kbqEuuJsXc6ioAQg== Date: Mon, 24 Sep 2018 23:17:39 +0000 Message-ID: <20180924231721.15799-4-yskoh@mellanox.com> References: <20180919072143.23211-1-yskoh@mellanox.com> <20180924231721.15799-1-yskoh@mellanox.com> In-Reply-To: <20180924231721.15799-1-yskoh@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BN6PR03CA0021.namprd03.prod.outlook.com (2603:10b6:404:23::31) To DB3PR0502MB3980.eurprd05.prod.outlook.com (2603:10a6:8:10::27) authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB3996; 6:09duA4e/5eRYpPjhdni8yOdzLAAdSkHKJRmT4sif0DryHWqatEbTIxB7mWzirl/taRhvemx5i1l8Wf8N991sbWYP03iM6jL/VZtXzfFHF854K5ly62tBadY1oxQe1Jauc5oZoGVVLgBjY45Z7Neoj+koCJDSdDQM2thQ+g/zK6M1qIdo2uJBvuKaIoTM6qqmVeIB9eQlx7VMB6bDv0fTf9++2HyDZW4bK1yoA8DnwcYMKCfdusQcSdk/99QCUxOA/P4fWp7upw7HQaGLynS4YSgzFeSKgFVZ7gAb3LtTvt3dZQ5sHCT88k8RqxduNcJFkJ+b+7/cm9bTUr346AjhQXxIV0tEgpxDpiKxE/Sl8viBbp9ry/VZMY3BK3HmjXGg9vJX4JugoVRxulQe6n+SNV5dwN8jL9pmqNyLXsNGiyKnD5M0wZXHbhm6z25J7qEjTIhmw8xoqE1/wkwn/Wg55g==; 5:Tf8byOmiesZhze1LvyJkrAickQsJ39gDeC5TOz67M1jORXnHCXJ344HeqdW6MDjx+CggkYDF2hyYG0vuN2gzjf0JaD5mw5QNY65TbaFxnw4W6lzPjHJGAYv9iT80eml08BL4B3APZeTKeX/pamW7CDYJRB6sYJ2/MZoDrOIqlxo=; 7:3NWxCBr39XCULY1i/PVBkvj4UoaWQE0QGdkbdQBqCW8sb0SMuiEtgQhqN2lFHrPFk98megR2tbTtvUmOtLgcStcunNpaLGxSXplpbdLBdmszJWwOANyF1dYtL7CnuL8X3IUzgXAC2ikjBbuGLjZ6/1f+wC34wRv4MnYX0FJ8HNpcWMlClrfyBNemST3WiQmR+RdCIIu9M6/i4OUiAEq6M32fNGV7bvyMnG7SqKq7LvQNJbQAfy3uVexZX3i9xKL6 x-ms-office365-filtering-correlation-id: c620af02-aad6-4079-3595-08d62273eb96 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(5600074)(711020)(4618075)(4534165)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB3996; x-ms-traffictypediagnostic: DB3PR0502MB3996: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(788757137089)(211171220733660)(269456686620040); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(3231355)(944501410)(52105095)(6055026)(149066)(150027)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123560045)(20161123562045)(20161123564045)(201708071742011)(7699051); SRVR:DB3PR0502MB3996; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB3996; x-forefront-prvs: 0805EC9467 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(376002)(346002)(366004)(39860400002)(136003)(199004)(189003)(51234002)(66066001)(2616005)(476003)(446003)(486006)(11346002)(16200700003)(6436002)(305945005)(7736002)(53946003)(386003)(6506007)(6486002)(6512007)(186003)(102836004)(5250100002)(76176011)(25786009)(26005)(68736007)(6116002)(54906003)(110136005)(99286004)(1076002)(3846002)(71200400001)(52116002)(86362001)(53936002)(71190400001)(97736004)(8936002)(107886003)(14444005)(256004)(4326008)(316002)(2900100001)(81156014)(8676002)(36756003)(81166006)(105586002)(2906002)(106356001)(6636002)(478600001)(5660300001)(14454004)(579004)(559001)(569006); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB3996; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: ZHnMn7POEt3ZBqC0f9Sv1sWtyThBCIEjAfX2HMrsBqW6zJZ9HGdDRRm+eB0gc2vQVUARzfXi9FdGeqJYdgQ/WQHDK8pyOyAMEc7tU44lZLWqptrBedhjFv8oBCi9p2kKqok+qdFdMQgP044Es0Cg+DHfVw0evnDyOf1eASknPXAcc9JJchZYm3djTnTWdu0vh/vsUQUGm+IqW+aHcudg7ATtydY5jr7QmjOMdbF9QQSNUTIB4v/fh8EpmoetBzGeoBwhV2SK1lHuGsOCG9pGzJOqQ3lgnaMoBMvvHFfO5BugAg2SWEsNuq5s6LGEN6xxIG/wAiWVWrO2uhcnwbnUgp2TqUei8O0sX/TwBRTkJxg= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: c620af02-aad6-4079-3595-08d62273eb96 X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Sep 2018 23:17:39.6858 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB3996 Subject: [dpdk-dev] [PATCH v3 03/11] net/mlx5: add flow translate function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2018 23:17:42 -0000 From: Ori Kam This commit modify the conversion of the input parameters into Verbs spec, in order to support all previous changes. Some of those changes are: removing the use of the parser, storing each flow in its own flow structure. Signed-off-by: Ori Kam Acked-by: Yongseok Koh --- drivers/net/mlx5/mlx5_flow.c | 1624 +++++++++++++++-----------------------= ---- 1 file changed, 580 insertions(+), 1044 deletions(-) diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 166fee555..2d55f08b8 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -296,6 +296,7 @@ struct mlx5_flow_verbs { struct mlx5_flow { LIST_ENTRY(mlx5_flow) next; struct rte_flow *flow; /**< Pointer to the main flow. */ + uint32_t layers; /**< Bit-fields that holds the detected layers. */ union { struct mlx5_flow_verbs verbs; /**< Holds the verbs dev-flow. */ }; @@ -316,15 +317,8 @@ struct mlx5_flow_counter { struct rte_flow { TAILQ_ENTRY(rte_flow) next; /**< Pointer to the next flow structure. */ struct rte_flow_attr attributes; /**< User flow attribute. */ - uint32_t layers; + uint32_t layers; /**< Bit-fields that holds the detected layers. */ /**< Bit-fields of present layers see MLX5_FLOW_LAYER_*. */ - uint32_t modifier; - /**< Bit-fields of present modifier see MLX5_FLOW_MOD_*. */ - uint32_t fate; - /**< Bit-fields of present fate see MLX5_FLOW_FATE_*. */ - LIST_HEAD(verbs, mlx5_flow_verbs) verbs; /**< Verbs flows list. */ - struct mlx5_flow_verbs *cur_verbs; - /**< Current Verbs flow structure being filled. */ struct mlx5_flow_counter *counter; /**< Holds Verbs flow counter. */ struct rte_flow_action_rss rss;/**< RSS context. */ uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */ @@ -332,6 +326,7 @@ struct rte_flow { void *nl_flow; /**< Netlink flow buffer if relevant. */ LIST_HEAD(dev_flows, mlx5_flow) dev_flows; /**< Device flows that are part of the flow. */ + uint32_t actions; /**< Bit-fields which mark all detected actions. */ }; =20 static const struct rte_flow_ops mlx5_flow_ops =3D { @@ -430,7 +425,7 @@ static struct mlx5_flow_tunnel_info tunnels_info[] =3D = { * Discover the maximum number of priority available. * * @param[in] dev - * Pointer to Ethernet device. + * Pointer to the Ethernet device structure. * * @return * number of supported flow priority on success, a negative errno @@ -497,34 +492,40 @@ mlx5_flow_discover_priorities(struct rte_eth_dev *dev= ) /** * Adjust flow priority. * - * @param dev - * Pointer to Ethernet device. - * @param flow - * Pointer to an rte flow. + * @param[in] dev + * Pointer to the Ethernet device structure. + * @param[in] priority + * The rule base priority. + * @param[in] subpriority + * The priority based on the items. + * + * @return + * The new priority. */ -static void -mlx5_flow_adjust_priority(struct rte_eth_dev *dev, struct rte_flow *flow) +static uint32_t +mlx5_flow_adjust_priority(struct rte_eth_dev *dev, + int32_t priority, + uint32_t subpriority) { + uint32_t res =3D 0; struct priv *priv =3D dev->data->dev_private; - uint32_t priority =3D flow->attributes.priority; - uint32_t subpriority =3D flow->cur_verbs->attr->priority; =20 switch (priv->config.flow_prio) { case RTE_DIM(priority_map_3): - priority =3D priority_map_3[priority][subpriority]; + res =3D priority_map_3[priority][subpriority]; break; case RTE_DIM(priority_map_5): - priority =3D priority_map_5[priority][subpriority]; + res =3D priority_map_5[priority][subpriority]; break; } - flow->cur_verbs->attr->priority =3D priority; + return res; } =20 /** * Get a flow counter. * * @param[in] dev - * Pointer to Ethernet device. + * Pointer to the Ethernet device structure. * @param[in] shared * Indicate if this counter is shared with other flows. * @param[in] id @@ -595,34 +596,6 @@ mlx5_flow_counter_release(struct mlx5_flow_counter *co= unter) } =20 /** - * Verify the @p attributes will be correctly understood by the NIC and st= ore - * them in the @p flow if everything is correct. - * - * @param[in] dev - * Pointer to Ethernet device structure. - * @param[in] attributes - * Pointer to flow attributes - * @param[in, out] flow - * Pointer to the rte_flow structure. - * - * @return - * 0 on success. - */ -static int -mlx5_flow_attributes(struct rte_eth_dev *dev, - const struct rte_flow_attr *attributes, - struct rte_flow *flow) -{ - struct priv *priv =3D dev->data->dev_private; - uint32_t priority_max =3D priv->config.flow_prio - 1; - - flow->attributes =3D *attributes; - if (attributes->priority =3D=3D MLX5_FLOW_PRIO_RSVD) - flow->attributes.priority =3D priority_max; - return 0; -} - -/** * Verify the @p item specifications (spec, last, mask) are compatible wit= h the * NIC capabilities. * @@ -693,9 +666,9 @@ mlx5_flow_item_acceptable(const struct rte_flow_item *i= tem, * Size in bytes of the specification to copy. */ static void -mlx5_flow_spec_verbs_add(struct rte_flow *flow, void *src, unsigned int si= ze) +mlx5_flow_spec_verbs_add(struct mlx5_flow *flow, void *src, unsigned int s= ize) { - struct mlx5_flow_verbs *verbs =3D flow->cur_verbs; + struct mlx5_flow_verbs *verbs =3D &flow->verbs; =20 if (verbs->specs) { void *dst; @@ -710,8 +683,8 @@ mlx5_flow_spec_verbs_add(struct rte_flow *flow, void *s= rc, unsigned int size) /** * Adjust verbs hash fields according to the @p flow information. * - * @param[in, out] flow. - * Pointer to flow structure. + * @param[in] dev_flow. + * Pointer to dev flow structure. * @param[in] tunnel * 1 when the hash field is for a tunnel item. * @param[in] layer_types @@ -720,49 +693,44 @@ mlx5_flow_spec_verbs_add(struct rte_flow *flow, void = *src, unsigned int size) * Item hash fields. */ static void -mlx5_flow_verbs_hashfields_adjust(struct rte_flow *flow, +mlx5_flow_verbs_hashfields_adjust(struct mlx5_flow *dev_flow, int tunnel __rte_unused, uint32_t layer_types, uint64_t hash_fields) { #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT + int rss_request_inner =3D dev_flow->flow->rss.level >=3D 2; + hash_fields |=3D (tunnel ? IBV_RX_HASH_INNER : 0); - if (flow->rss.level =3D=3D 2 && !tunnel) + if (rss_request_inner && !tunnel) hash_fields =3D 0; - else if (flow->rss.level < 2 && tunnel) + else if (!rss_request_inner && tunnel) hash_fields =3D 0; #endif - if (!(flow->rss.types & layer_types)) + if (!(dev_flow->flow->rss.types & layer_types)) hash_fields =3D 0; - flow->cur_verbs->hash_fields |=3D hash_fields; + dev_flow->verbs.hash_fields |=3D hash_fields; } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in] item_flags + * Bit field with all detected items. + * @param[in, out] dev_flow + * Pointer to dev_flow structure. */ -static int -mlx5_flow_item_eth(const struct rte_flow_item *item, struct rte_flow *flow= , - const size_t flow_size) +static void +flow_verbs_translate_item_eth(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_eth *spec =3D item->spec; const struct rte_flow_item_eth *mask =3D item->mask; - const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel =3D !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); const unsigned int size =3D sizeof(struct ibv_flow_spec_eth); struct ibv_flow_spec_eth eth =3D { .type =3D IBV_FLOW_SPEC_ETH | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -771,10 +739,6 @@ mlx5_flow_item_eth(const struct rte_flow_item *item, s= truct rte_flow *flow, =20 if (!mask) mask =3D &rte_flow_item_eth_mask; - flow->layers |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L2 : - MLX5_FLOW_LAYER_OUTER_L2; - if (size > flow_size) - return size; if (spec) { unsigned int i; =20 @@ -790,14 +754,18 @@ mlx5_flow_item_eth(const struct rte_flow_item *item, = struct rte_flow *flow, eth.val.src_mac[i] &=3D eth.mask.src_mac[i]; } eth.val.ether_type &=3D eth.mask.ether_type; + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L2; } - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L2; - mlx5_flow_spec_verbs_add(flow, ð, size); - return size; + mlx5_flow_spec_verbs_add(dev_flow, ð, size); + *item_flags |=3D tunnel ? + MLX5_FLOW_LAYER_INNER_L2 : + MLX5_FLOW_LAYER_OUTER_L2; } =20 /** * Update the VLAN tag in the Verbs Ethernet specification. + * This function assumes that the input is valid and there is space to add + * the requested item. * * @param[in, out] attr * Pointer to Verbs attributes structure. @@ -829,34 +797,26 @@ mlx5_flow_item_vlan_update(struct ibv_flow_attr *attr= , } =20 /** - * Convert the @p item into @p flow (or by updating the already present - * Ethernet Verbs) specification after ensuring the NIC will understand an= d - * process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in, out] item_flags + * Bit mask that holds all detected items. + * @param[in, out] dev_flow + * Pointer to dev_flow structure. */ -static int -mlx5_flow_item_vlan(const struct rte_flow_item *item, struct rte_flow *flo= w, - const size_t flow_size) +static void +flow_verbs_translate_item_vlan(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_vlan *spec =3D item->spec; const struct rte_flow_item_vlan *mask =3D item->mask; unsigned int size =3D sizeof(struct ibv_flow_spec_eth); - const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel =3D !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); struct ibv_flow_spec_eth eth =3D { .type =3D IBV_FLOW_SPEC_ETH | (tunnel ? IBV_FLOW_SPEC_INNER : 0), .size =3D size, @@ -874,49 +834,40 @@ mlx5_flow_item_vlan(const struct rte_flow_item *item,= struct rte_flow *flow, eth.mask.ether_type =3D mask->inner_type; eth.val.ether_type &=3D eth.mask.ether_type; } - if (!(flow->layers & l2m)) { - if (size <=3D flow_size) { - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L2; - mlx5_flow_spec_verbs_add(flow, ð, size); - } + if (!(*item_flags & l2m)) { + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L2; + mlx5_flow_spec_verbs_add(dev_flow, ð, size); } else { - if (flow->cur_verbs) - mlx5_flow_item_vlan_update(flow->cur_verbs->attr, + mlx5_flow_item_vlan_update(dev_flow->verbs.attr, ð); size =3D 0; /* Only an update is done in eth specification. */ } - flow->layers |=3D tunnel ? - (MLX5_FLOW_LAYER_INNER_L2 | MLX5_FLOW_LAYER_INNER_VLAN) : - (MLX5_FLOW_LAYER_OUTER_L2 | MLX5_FLOW_LAYER_OUTER_VLAN); - return size; + *item_flags |=3D tunnel ? + (MLX5_FLOW_LAYER_INNER_L2 | + MLX5_FLOW_LAYER_INNER_VLAN) : + (MLX5_FLOW_LAYER_OUTER_L2 | MLX5_FLOW_LAYER_OUTER_VLAN); } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_ipv4(const struct rte_flow_item *item, struct rte_flow *flo= w, - const size_t flow_size) +static void +flow_verbs_translate_item_ipv4(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_ipv4 *spec =3D item->spec; const struct rte_flow_item_ipv4 *mask =3D item->mask; - const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel =3D !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); unsigned int size =3D sizeof(struct ibv_flow_spec_ipv4_ext); struct ibv_flow_spec_ipv4_ext ipv4 =3D { .type =3D IBV_FLOW_SPEC_IPV4_EXT | @@ -926,7 +877,7 @@ mlx5_flow_item_ipv4(const struct rte_flow_item *item, s= truct rte_flow *flow, =20 if (!mask) mask =3D &rte_flow_item_ipv4_mask; - flow->layers |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 : + *item_flags |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 : MLX5_FLOW_LAYER_OUTER_L3_IPV4; if (spec) { ipv4.val =3D (struct ibv_flow_ipv4_ext_filter){ @@ -947,46 +898,37 @@ mlx5_flow_item_ipv4(const struct rte_flow_item *item,= struct rte_flow *flow, ipv4.val.proto &=3D ipv4.mask.proto; ipv4.val.tos &=3D ipv4.mask.tos; } - if (size <=3D flow_size) { - mlx5_flow_verbs_hashfields_adjust - (flow, tunnel, - (ETH_RSS_IPV4 | ETH_RSS_FRAG_IPV4 | - ETH_RSS_NONFRAG_IPV4_TCP | - ETH_RSS_NONFRAG_IPV4_UDP | - ETH_RSS_NONFRAG_IPV4_OTHER), - (IBV_RX_HASH_SRC_IPV4 | IBV_RX_HASH_DST_IPV4)); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L3; - mlx5_flow_spec_verbs_add(flow, &ipv4, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, tunnel, + (ETH_RSS_IPV4 | ETH_RSS_FRAG_IPV4 | + ETH_RSS_NONFRAG_IPV4_TCP | + ETH_RSS_NONFRAG_IPV4_UDP | + ETH_RSS_NONFRAG_IPV4_OTHER), + (IBV_RX_HASH_SRC_IPV4 | + IBV_RX_HASH_DST_IPV4)); + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L3; + mlx5_flow_spec_verbs_add(dev_flow, &ipv4, size); } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_ipv6(const struct rte_flow_item *item, struct rte_flow *flo= w, - const size_t flow_size) +static void +flow_verbs_translate_item_ipv6(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_ipv6 *spec =3D item->spec; const struct rte_flow_item_ipv6 *mask =3D item->mask; - const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel =3D !!(dev_flow->flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int size =3D sizeof(struct ibv_flow_spec_ipv6); struct ibv_flow_spec_ipv6 ipv6 =3D { .type =3D IBV_FLOW_SPEC_IPV6 | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -995,8 +937,8 @@ mlx5_flow_item_ipv6(const struct rte_flow_item *item, s= truct rte_flow *flow, =20 if (!mask) mask =3D &rte_flow_item_ipv6_mask; - flow->layers |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 : - MLX5_FLOW_LAYER_OUTER_L3_IPV6; + *item_flags |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 : + MLX5_FLOW_LAYER_OUTER_L3_IPV6; if (spec) { unsigned int i; uint32_t vtc_flow_val; @@ -1036,46 +978,40 @@ mlx5_flow_item_ipv6(const struct rte_flow_item *item= , struct rte_flow *flow, ipv6.val.next_hdr &=3D ipv6.mask.next_hdr; ipv6.val.hop_limit &=3D ipv6.mask.hop_limit; } - if (size <=3D flow_size) { - mlx5_flow_verbs_hashfields_adjust - (flow, tunnel, - (ETH_RSS_IPV6 | ETH_RSS_FRAG_IPV6 | - ETH_RSS_NONFRAG_IPV6_TCP | ETH_RSS_NONFRAG_IPV6_UDP | - ETH_RSS_NONFRAG_IPV6_OTHER | ETH_RSS_IPV6_EX | - ETH_RSS_IPV6_TCP_EX | ETH_RSS_IPV6_UDP_EX), - (IBV_RX_HASH_SRC_IPV6 | IBV_RX_HASH_DST_IPV6)); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L3; - mlx5_flow_spec_verbs_add(flow, &ipv6, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, tunnel, + (ETH_RSS_IPV6 | ETH_RSS_FRAG_IPV6 | + ETH_RSS_NONFRAG_IPV6_TCP | + ETH_RSS_NONFRAG_IPV6_UDP | + ETH_RSS_IPV6_EX | + ETH_RSS_IPV6_TCP_EX | + ETH_RSS_IPV6_UDP_EX | + ETH_RSS_NONFRAG_IPV6_OTHER), + (IBV_RX_HASH_SRC_IPV6 | + IBV_RX_HASH_DST_IPV6)); + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L3; + mlx5_flow_spec_verbs_add(dev_flow, &ipv6, size); } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_udp(const struct rte_flow_item *item, struct rte_flow *flow= , - const size_t flow_size) +static void +flow_verbs_translate_item_udp(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_udp *spec =3D item->spec; const struct rte_flow_item_udp *mask =3D item->mask; - const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel =3D !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); unsigned int size =3D sizeof(struct ibv_flow_spec_tcp_udp); struct ibv_flow_spec_tcp_udp udp =3D { .type =3D IBV_FLOW_SPEC_UDP | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -1084,8 +1020,8 @@ mlx5_flow_item_udp(const struct rte_flow_item *item, = struct rte_flow *flow, =20 if (!mask) mask =3D &rte_flow_item_udp_mask; - flow->layers |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP : - MLX5_FLOW_LAYER_OUTER_L4_UDP; + *item_flags |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP : + MLX5_FLOW_LAYER_OUTER_L4_UDP; if (spec) { udp.val.dst_port =3D spec->hdr.dst_port; udp.val.src_port =3D spec->hdr.src_port; @@ -1095,44 +1031,34 @@ mlx5_flow_item_udp(const struct rte_flow_item *item= , struct rte_flow *flow, udp.val.src_port &=3D udp.mask.src_port; udp.val.dst_port &=3D udp.mask.dst_port; } - if (size <=3D flow_size) { - mlx5_flow_verbs_hashfields_adjust(flow, tunnel, ETH_RSS_UDP, - (IBV_RX_HASH_SRC_PORT_UDP | - IBV_RX_HASH_DST_PORT_UDP)); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L4; - mlx5_flow_spec_verbs_add(flow, &udp, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, + tunnel, ETH_RSS_UDP, + (IBV_RX_HASH_SRC_PORT_UDP | + IBV_RX_HASH_DST_PORT_UDP)); + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L4; + mlx5_flow_spec_verbs_add(dev_flow, &udp, size); } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_tcp(const struct rte_flow_item *item, struct rte_flow *flow= , - const size_t flow_size) +static void +flow_verbs_translate_item_tcp(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_tcp *spec =3D item->spec; const struct rte_flow_item_tcp *mask =3D item->mask; - const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel =3D !!(dev_flow->flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int size =3D sizeof(struct ibv_flow_spec_tcp_udp); struct ibv_flow_spec_tcp_udp tcp =3D { .type =3D IBV_FLOW_SPEC_TCP | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -1141,8 +1067,8 @@ mlx5_flow_item_tcp(const struct rte_flow_item *item, = struct rte_flow *flow, =20 if (!mask) mask =3D &rte_flow_item_tcp_mask; - flow->layers |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP : - MLX5_FLOW_LAYER_OUTER_L4_TCP; + *item_flags |=3D tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP : + MLX5_FLOW_LAYER_OUTER_L4_TCP; if (spec) { tcp.val.dst_port =3D spec->hdr.dst_port; tcp.val.src_port =3D spec->hdr.src_port; @@ -1152,40 +1078,30 @@ mlx5_flow_item_tcp(const struct rte_flow_item *item= , struct rte_flow *flow, tcp.val.src_port &=3D tcp.mask.src_port; tcp.val.dst_port &=3D tcp.mask.dst_port; } - if (size <=3D flow_size) { - mlx5_flow_verbs_hashfields_adjust(flow, tunnel, ETH_RSS_TCP, - (IBV_RX_HASH_SRC_PORT_TCP | - IBV_RX_HASH_DST_PORT_TCP)); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L4; - mlx5_flow_spec_verbs_add(flow, &tcp, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, + tunnel, ETH_RSS_TCP, + (IBV_RX_HASH_SRC_PORT_TCP | + IBV_RX_HASH_DST_PORT_TCP)); + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L4; + mlx5_flow_spec_verbs_add(dev_flow, &tcp, size); } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_vxlan(const struct rte_flow_item *item, struct rte_flow *fl= ow, - const size_t flow_size) +static void +flow_verbs_translate_item_vxlan(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_vxlan *spec =3D item->spec; const struct rte_flow_item_vxlan *mask =3D item->mask; @@ -1209,38 +1125,27 @@ mlx5_flow_item_vxlan(const struct rte_flow_item *it= em, struct rte_flow *flow, /* Remove unwanted bits from values. */ vxlan.val.tunnel_id &=3D vxlan.mask.tunnel_id; } - if (size <=3D flow_size) { - mlx5_flow_spec_verbs_add(flow, &vxlan, size); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L2; - } - flow->layers |=3D MLX5_FLOW_LAYER_VXLAN; - return size; + mlx5_flow_spec_verbs_add(dev_flow, &vxlan, size); + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L2; + *item_flags |=3D MLX5_FLOW_LAYER_VXLAN; } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_vxlan_gpe(const struct rte_flow_item *item, - struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_item_vxlan_gpe(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_vxlan_gpe *spec =3D item->spec; const struct rte_flow_item_vxlan_gpe *mask =3D item->mask; @@ -1264,12 +1169,9 @@ mlx5_flow_item_vxlan_gpe(const struct rte_flow_item = *item, /* Remove unwanted bits from values. */ vxlan_gpe.val.tunnel_id &=3D vxlan_gpe.mask.tunnel_id; } - if (size <=3D flow_size) { - mlx5_flow_spec_verbs_add(flow, &vxlan_gpe, size); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L2; - } - flow->layers |=3D MLX5_FLOW_LAYER_VXLAN_GPE; - return size; + mlx5_flow_spec_verbs_add(dev_flow, &vxlan_gpe, size); + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L2; + *item_flags |=3D MLX5_FLOW_LAYER_VXLAN_GPE; } =20 /** @@ -1325,34 +1227,30 @@ mlx5_flow_item_gre_ip_protocol_update(struct ibv_fl= ow_attr *attr, } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * It will also update the previous L3 layer with the protocol value match= ing - * the GRE. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes t= hat + * the input is valid and that there is space to insert the requested item + * into the flow. * - * @param dev - * Pointer to Ethernet device. * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_gre(const struct rte_flow_item *item __rte_unused, - struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_item_gre(const struct rte_flow_item *item __rte_unuse= d, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { - struct mlx5_flow_verbs *verbs =3D flow->cur_verbs; -#ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT + struct mlx5_flow_verbs *verbs =3D &dev_flow->verbs; +#ifndef HAVE_IBV_DEVICE_MPLS_SUPPORT + unsigned int size =3D sizeof(struct ibv_flow_spec_tunnel); + struct ibv_flow_spec_tunnel tunnel =3D { + .type =3D IBV_FLOW_SPEC_VXLAN_TUNNEL, + .size =3D size, + }; +#else const struct rte_flow_item_gre *spec =3D item->spec; const struct rte_flow_item_gre *mask =3D item->mask; unsigned int size =3D sizeof(struct ibv_flow_spec_gre); @@ -1360,15 +1258,7 @@ mlx5_flow_item_gre(const struct rte_flow_item *item = __rte_unused, .type =3D IBV_FLOW_SPEC_GRE, .size =3D size, }; -#else - unsigned int size =3D sizeof(struct ibv_flow_spec_tunnel); - struct ibv_flow_spec_tunnel tunnel =3D { - .type =3D IBV_FLOW_SPEC_VXLAN_TUNNEL, - .size =3D size, - }; -#endif =20 -#ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT if (!mask) mask =3D &rte_flow_item_gre_mask; if (spec) { @@ -1381,51 +1271,36 @@ mlx5_flow_item_gre(const struct rte_flow_item *item= __rte_unused, tunnel.val.protocol &=3D tunnel.mask.protocol; tunnel.val.key &=3D tunnel.mask.key; } -#else -#endif /* !HAVE_IBV_DEVICE_MPLS_SUPPORT */ - if (size <=3D flow_size) { - if (flow->layers & MLX5_FLOW_LAYER_OUTER_L3_IPV4) - mlx5_flow_item_gre_ip_protocol_update - (verbs->attr, IBV_FLOW_SPEC_IPV4_EXT, - MLX5_IP_PROTOCOL_GRE); - else - mlx5_flow_item_gre_ip_protocol_update - (verbs->attr, IBV_FLOW_SPEC_IPV6, - MLX5_IP_PROTOCOL_GRE); - mlx5_flow_spec_verbs_add(flow, &tunnel, size); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L2; - } - flow->layers |=3D MLX5_FLOW_LAYER_GRE; - return size; +#endif + if (*item_flags & MLX5_FLOW_LAYER_OUTER_L3_IPV4) + mlx5_flow_item_gre_ip_protocol_update + (verbs->attr, IBV_FLOW_SPEC_IPV4_EXT, + MLX5_IP_PROTOCOL_GRE); + else + mlx5_flow_item_gre_ip_protocol_update + (verbs->attr, IBV_FLOW_SPEC_IPV6, + MLX5_IP_PROTOCOL_GRE); + mlx5_flow_spec_verbs_add(dev_flow, &tunnel, size); + verbs->attr->priority =3D MLX5_PRIORITY_MAP_L2; + *item_flags |=3D MLX5_FLOW_LAYER_GRE; } =20 /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p action into a Verbs specification. This function assumes= that + * the input is valid and that there is space to insert the requested acti= on + * into the flow. This function also return the action that was added. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p item has fully been conver= ted, - * otherwise another call with this returned memory size should be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_mpls(const struct rte_flow_item *item __rte_unused, - struct rte_flow *flow __rte_unused, - const size_t flow_size __rte_unused, - struct rte_flow_error *error) +static void +flow_verbs_translate_item_mpls(const struct rte_flow_item *item __rte_unus= ed, + uint64_t *action_flags __rte_unused, + struct mlx5_flow *dev_flow __rte_unused) { #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT const struct rte_flow_item_mpls *spec =3D item->spec; @@ -1444,135 +1319,25 @@ mlx5_flow_item_mpls(const struct rte_flow_item *it= em __rte_unused, /* Remove unwanted bits from values. */ mpls.val.label &=3D mpls.mask.label; } - if (size <=3D flow_size) { - mlx5_flow_spec_verbs_add(flow, &mpls, size); - flow->cur_verbs->attr->priority =3D MLX5_PRIORITY_MAP_L2; - } - flow->layers |=3D MLX5_FLOW_LAYER_MPLS; - return size; -#endif /* !HAVE_IBV_DEVICE_MPLS_SUPPORT */ - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ITEM, - item, - "MPLS is not supported by Verbs, please" - " update."); -} - -/** - * Convert the @p pattern into a Verbs specifications after ensuring the N= IC - * will understand and process it correctly. - * The conversion is performed item per item, each of them is written into - * the @p flow if its size is lesser or equal to @p flow_size. - * Validation and memory consumption computation are still performed until= the - * end of @p pattern, unless an error is encountered. - * - * @param[in] pattern - * Flow pattern. - * @param[in, out] flow - * Pointer to the rte_flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small some - * garbage may be present. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @pattern has fully been - * converted, otherwise another call with this returned memory size shou= ld - * be done. - * On error, a negative errno value is returned and rte_errno is set. - */ -static int -mlx5_flow_items(const struct rte_flow_item pattern[], - struct rte_flow *flow, const size_t flow_size, - struct rte_flow_error *error) -{ - int remain =3D flow_size; - size_t size =3D 0; - - for (; pattern->type !=3D RTE_FLOW_ITEM_TYPE_END; pattern++) { - int ret =3D 0; - - switch (pattern->type) { - case RTE_FLOW_ITEM_TYPE_VOID: - break; - case RTE_FLOW_ITEM_TYPE_ETH: - ret =3D mlx5_flow_item_eth(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_VLAN: - ret =3D mlx5_flow_item_vlan(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_IPV4: - ret =3D mlx5_flow_item_ipv4(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_IPV6: - ret =3D mlx5_flow_item_ipv6(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_UDP: - ret =3D mlx5_flow_item_udp(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_TCP: - ret =3D mlx5_flow_item_tcp(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_VXLAN: - ret =3D mlx5_flow_item_vxlan(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_VXLAN_GPE: - ret =3D mlx5_flow_item_vxlan_gpe(pattern, flow, - remain); - break; - case RTE_FLOW_ITEM_TYPE_GRE: - ret =3D mlx5_flow_item_gre(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_MPLS: - ret =3D mlx5_flow_item_mpls(pattern, flow, remain, error); - break; - default: - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ITEM, - pattern, - "item not supported"); - } - if (ret < 0) - return ret; - if (remain > ret) - remain -=3D ret; - else - remain =3D 0; - size +=3D ret; - } - if (!flow->layers) { - const struct rte_flow_item item =3D { - .type =3D RTE_FLOW_ITEM_TYPE_ETH, - }; - - return mlx5_flow_item_eth(&item, flow, flow_size); - } - return size; + mlx5_flow_spec_verbs_add(dev_flow, &mpls, size); + dev_flow->verbs.attr->priority =3D MLX5_PRIORITY_MAP_L2; + *action_flags |=3D MLX5_FLOW_LAYER_MPLS; +#endif } =20 /** - * Convert the @p action into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. - * - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. + * Convert the @p action into a Verbs specification. This function assumes= that + * the input is valid and that there is space to insert the requested acti= on + * into the flow. This function also return the action that was added. * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size shou= ld - * be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_drop(struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_action_drop(uint64_t *action_flags, + struct mlx5_flow *dev_flow) { unsigned int size =3D sizeof(struct ibv_flow_spec_action_drop); struct ibv_flow_spec_action_drop drop =3D { @@ -1580,53 +1345,55 @@ mlx5_flow_action_drop(struct rte_flow *flow, const = size_t flow_size) .size =3D size, }; =20 - if (size < flow_size) - mlx5_flow_spec_verbs_add(flow, &drop, size); - flow->fate |=3D MLX5_FLOW_FATE_DROP; - return size; + mlx5_flow_spec_verbs_add(dev_flow, &drop, size); + *action_flags |=3D MLX5_ACTION_DROP; } =20 /** - * Convert the @p action into @p flow after ensuring the NIC will understa= nd - * and process it correctly. + * Convert the @p action into a Verbs specification. This function assumes= that + * the input is valid and that there is space to insert the requested acti= on + * into the flow. This function also return the action that was added. * * @param[in] action * Action configuration. - * @param[in, out] flow - * Pointer to flow structure. - * - * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_queue(const struct rte_flow_action *action, - struct rte_flow *flow) +static void +flow_verbs_translate_action_queue(const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_action_queue *queue =3D action->conf; + struct rte_flow *flow =3D dev_flow->flow; =20 if (flow->queue) (*flow->queue)[0] =3D queue->index; flow->rss.queue_num =3D 1; - flow->fate |=3D MLX5_FLOW_FATE_QUEUE; - return 0; + *action_flags |=3D MLX5_ACTION_QUEUE; } =20 /** - * Ensure the @p action will be understood and used correctly by the NIC. + * Convert the @p action into a Verbs specification. This function assumes= that + * the input is valid and that there is space to insert the requested acti= on + * into the flow. This function also return the action that was added. * * @param[in] action * Action configuration. - * @param flow[in, out] - * Pointer to the rte_flow structure. - * - * @return - * 0 On success. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_rss(const struct rte_flow_action *action, - struct rte_flow *flow) +static void +flow_verbs_translate_action_rss(const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_action_rss *rss =3D action->conf; + struct rte_flow *flow =3D dev_flow->flow; =20 if (flow->queue) memcpy((*flow->queue), rss->queue, @@ -1635,30 +1402,26 @@ mlx5_flow_action_rss(const struct rte_flow_action *= action, memcpy(flow->key, rss->key, MLX5_RSS_HASH_KEY_LEN); flow->rss.types =3D rss->types; flow->rss.level =3D rss->level; - flow->fate |=3D MLX5_FLOW_FATE_RSS; - return 0; + *action_flags |=3D MLX5_ACTION_RSS; } =20 /** - * Convert the @p action into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p action into a Verbs specification. This function assumes= that + * the input is valid and that there is space to insert the requested acti= on + * into the flow. This function also return the action that was added. * - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size shou= ld - * be done. + * @param[in] action + * Action configuration. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_flag(struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_action_flag + (const struct rte_flow_action *action __rte_unused, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { unsigned int size =3D sizeof(struct ibv_flow_spec_action_tag); struct ibv_flow_spec_action_tag tag =3D { @@ -1666,14 +1429,8 @@ mlx5_flow_action_flag(struct rte_flow *flow, const s= ize_t flow_size) .size =3D size, .tag_id =3D mlx5_flow_mark_set(MLX5_FLOW_MARK_DEFAULT), }; - struct mlx5_flow_verbs *verbs =3D flow->cur_verbs; - - if (flow->modifier & MLX5_FLOW_MOD_MARK) - size =3D 0; - else if (size <=3D flow_size && verbs) - mlx5_flow_spec_verbs_add(flow, &tag, size); - flow->modifier |=3D MLX5_FLOW_MOD_FLAG; - return size; + *action_flags |=3D MLX5_ACTION_MARK; + mlx5_flow_spec_verbs_add(dev_flow, &tag, size); } =20 /** @@ -1685,499 +1442,115 @@ mlx5_flow_action_flag(struct rte_flow *flow, cons= t size_t flow_size) * Mark identifier to replace the flag. */ static void -mlx5_flow_verbs_mark_update(struct mlx5_flow_verbs *verbs, uint32_t mark_i= d) +flow_verbs_mark_update(struct mlx5_flow_verbs *verbs, uint32_t mark_id) { struct ibv_spec_header *hdr; int i; =20 - if (!verbs) - return; - /* Update Verbs specification. */ - hdr =3D (struct ibv_spec_header *)verbs->specs; - if (!hdr) - return; - for (i =3D 0; i !=3D verbs->attr->num_of_specs; ++i) { - if (hdr->type =3D=3D IBV_FLOW_SPEC_ACTION_TAG) { - struct ibv_flow_spec_action_tag *t =3D - (struct ibv_flow_spec_action_tag *)hdr; - - t->tag_id =3D mlx5_flow_mark_set(mark_id); - } - hdr =3D (struct ibv_spec_header *)((uintptr_t)hdr + hdr->size); - } -} - -/** - * Convert the @p action into @p flow (or by updating the already present - * Flag Verbs specification) after ensuring the NIC will understand and - * process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. - * - * @param[in] action - * Action configuration. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size shou= ld - * be done. - */ -static int -mlx5_flow_action_mark(const struct rte_flow_action *action, - struct rte_flow *flow, const size_t flow_size) -{ - const struct rte_flow_action_mark *mark =3D action->conf; - unsigned int size =3D sizeof(struct ibv_flow_spec_action_tag); - struct ibv_flow_spec_action_tag tag =3D { - .type =3D IBV_FLOW_SPEC_ACTION_TAG, - .size =3D size, - }; - struct mlx5_flow_verbs *verbs =3D flow->cur_verbs; - - if (flow->modifier & MLX5_FLOW_MOD_FLAG) { - mlx5_flow_verbs_mark_update(verbs, mark->id); - size =3D 0; - } else if (size <=3D flow_size) { - tag.tag_id =3D mlx5_flow_mark_set(mark->id); - mlx5_flow_spec_verbs_add(flow, &tag, size); - } - flow->modifier |=3D MLX5_FLOW_MOD_MARK; - return size; -} - -/** - * Convert the @p action into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_si= ze, - * nothing is written in @p flow, the validation is still performed. - * - * @param action[in] - * Action configuration. - * @param flow[in, out] - * Pointer to flow structure. - * @param flow_size[in] - * Size in bytes of the available space in @p flow, if too small, nothin= g is - * written. - * @param error[int, out] - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size shou= ld - * be done. - * On error, a negative errno value is returned and rte_errno is set. - */ -static int -mlx5_flow_action_count(struct rte_eth_dev *dev, - const struct rte_flow_action *action, - struct rte_flow *flow, - const size_t flow_size __rte_unused, - struct rte_flow_error *error) -{ - const struct rte_flow_action_count *count =3D action->conf; -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - unsigned int size =3D sizeof(struct ibv_flow_spec_counter_action); - struct ibv_flow_spec_counter_action counter =3D { - .type =3D IBV_FLOW_SPEC_ACTION_COUNT, - .size =3D size, - }; -#endif - - if (!flow->counter) { - flow->counter =3D mlx5_flow_counter_new(dev, count->shared, - count->id); - if (!flow->counter) - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ACTION, - action, - "cannot get counter" - " context."); - } - flow->modifier |=3D MLX5_FLOW_MOD_COUNT; -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - counter.counter_set_handle =3D flow->counter->cs->handle; - if (size <=3D flow_size) - mlx5_flow_spec_verbs_add(flow, &counter, size); - return size; -#endif - return 0; -} - -/** - * Convert the @p action into @p flow after ensuring the NIC will understa= nd - * and process it correctly. - * The conversion is performed action per action, each of them is written = into - * the @p flow if its size is lesser or equal to @p flow_size. - * Validation and memory consumption computation are still performed until= the - * end of @p action, unless an error is encountered. - * - * @param[in] dev - * Pointer to Ethernet device structure. - * @param[in] actions - * Pointer to flow actions array. - * @param[in, out] flow - * Pointer to the rte_flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small some - * garbage may be present. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the @p actions has fully been - * converted, otherwise another call with this returned memory size shou= ld - * be done. - * On error, a negative errno value is returned and rte_errno is set. - */ -static int -mlx5_flow_actions(struct rte_eth_dev *dev, - const struct rte_flow_action actions[], - struct rte_flow *flow, const size_t flow_size, - struct rte_flow_error *error) -{ - size_t size =3D 0; - int remain =3D flow_size; - int ret =3D 0; - - for (; actions->type !=3D RTE_FLOW_ACTION_TYPE_END; actions++) { - switch (actions->type) { - case RTE_FLOW_ACTION_TYPE_VOID: - break; - case RTE_FLOW_ACTION_TYPE_FLAG: - ret =3D mlx5_flow_action_flag(flow, remain); - break; - case RTE_FLOW_ACTION_TYPE_MARK: - ret =3D mlx5_flow_action_mark(actions, flow, remain); - break; - case RTE_FLOW_ACTION_TYPE_DROP: - ret =3D mlx5_flow_action_drop(flow, remain); - break; - case RTE_FLOW_ACTION_TYPE_QUEUE: - ret =3D mlx5_flow_action_queue(actions, flow); - break; - case RTE_FLOW_ACTION_TYPE_RSS: - ret =3D mlx5_flow_action_rss(actions, flow); - break; - case RTE_FLOW_ACTION_TYPE_COUNT: - ret =3D mlx5_flow_action_count(dev, actions, flow, remain, - error); - break; - default: - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ACTION, - actions, - "action not supported"); - } - if (ret < 0) - return ret; - if (remain > ret) - remain -=3D ret; - else - remain =3D 0; - size +=3D ret; - } - if (!flow->fate) - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_UNSPECIFIED, - NULL, - "no fate action found"); - return size; -} - -/** - * Validate flow rule and fill flow structure accordingly. - * - * @param dev - * Pointer to Ethernet device. - * @param[out] flow - * Pointer to flow structure. - * @param flow_size - * Size of allocated space for @p flow. - * @param[in] attr - * Flow rule attributes. - * @param[in] pattern - * Pattern specification (list terminated by the END pattern item). - * @param[in] actions - * Associated actions (list terminated by the END action). - * @param[out] error - * Perform verbose error reporting if not NULL. - * - * @return - * A positive value representing the size of the flow object in bytes - * regardless of @p flow_size on success, a negative errno value otherwi= se - * and rte_errno is set. - */ -static int -mlx5_flow_merge_switch(struct rte_eth_dev *dev, - struct rte_flow *flow, - size_t flow_size, - const struct rte_flow_attr *attr, - const struct rte_flow_item pattern[], - const struct rte_flow_action actions[], - struct rte_flow_error *error) -{ - unsigned int n =3D mlx5_dev_to_port_id(dev->device, NULL, 0); - uint16_t port_id[!n + n]; - struct mlx5_nl_flow_ptoi ptoi[!n + n + 1]; - size_t off =3D RTE_ALIGN_CEIL(sizeof(*flow), alignof(max_align_t)); - unsigned int i; - unsigned int own =3D 0; - int ret; - - /* At least one port is needed when no switch domain is present. */ - if (!n) { - n =3D 1; - port_id[0] =3D dev->data->port_id; - } else { - n =3D RTE_MIN(mlx5_dev_to_port_id(dev->device, port_id, n), n); - } - for (i =3D 0; i !=3D n; ++i) { - struct rte_eth_dev_info dev_info; - - rte_eth_dev_info_get(port_id[i], &dev_info); - if (port_id[i] =3D=3D dev->data->port_id) - own =3D i; - ptoi[i].port_id =3D port_id[i]; - ptoi[i].ifindex =3D dev_info.if_index; - } - /* Ensure first entry of ptoi[] is the current device. */ - if (own) { - ptoi[n] =3D ptoi[0]; - ptoi[0] =3D ptoi[own]; - ptoi[own] =3D ptoi[n]; - } - /* An entry with zero ifindex terminates ptoi[]. */ - ptoi[n].port_id =3D 0; - ptoi[n].ifindex =3D 0; - if (flow_size < off) - flow_size =3D 0; - ret =3D mlx5_nl_flow_transpose((uint8_t *)flow + off, - flow_size ? flow_size - off : 0, - ptoi, attr, pattern, actions, error); - if (ret < 0) - return ret; - if (flow_size) { - *flow =3D (struct rte_flow){ - .attributes =3D *attr, - .nl_flow =3D (uint8_t *)flow + off, - }; - /* - * Generate a reasonably unique handle based on the address - * of the target buffer. - * - * This is straightforward on 32-bit systems where the flow - * pointer can be used directly. Otherwise, its least - * significant part is taken after shifting it by the - * previous power of two of the pointed buffer size. - */ - if (sizeof(flow) <=3D 4) - mlx5_nl_flow_brand(flow->nl_flow, (uintptr_t)flow); - else - mlx5_nl_flow_brand - (flow->nl_flow, - (uintptr_t)flow >> - rte_log2_u32(rte_align32prevpow2(flow_size))); + if (!verbs) + return; + /* Update Verbs specification. */ + hdr =3D (struct ibv_spec_header *)verbs->specs; + if (!hdr) + return; + for (i =3D 0; i !=3D verbs->attr->num_of_specs; ++i) { + if (hdr->type =3D=3D IBV_FLOW_SPEC_ACTION_TAG) { + struct ibv_flow_spec_action_tag *t =3D + (struct ibv_flow_spec_action_tag *)hdr; + + t->tag_id =3D mlx5_flow_mark_set(mark_id); + } + hdr =3D (struct ibv_spec_header *)((uintptr_t)hdr + hdr->size); } - return off + ret; } =20 -static unsigned int -mlx5_find_graph_root(const struct rte_flow_item pattern[], uint32_t rss_le= vel) +/** + * Convert the @p action into a Verbs specification. This function assumes= that + * the input is valid and that there is space to insert the requested acti= on + * into the flow. This function also return the action that was added. + * + * @param[in] action + * Action configuration. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. + */ +static void +flow_verbs_translate_action_mark(const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { - const struct rte_flow_item *item; - unsigned int has_vlan =3D 0; + const struct rte_flow_action_mark *mark =3D action->conf; + unsigned int size =3D sizeof(struct ibv_flow_spec_action_tag); + struct ibv_flow_spec_action_tag tag =3D { + .type =3D IBV_FLOW_SPEC_ACTION_TAG, + .size =3D size, + }; + struct mlx5_flow_verbs *verbs =3D &dev_flow->verbs; =20 - for (item =3D pattern; item->type !=3D RTE_FLOW_ITEM_TYPE_END; item++) { - if (item->type =3D=3D RTE_FLOW_ITEM_TYPE_VLAN) { - has_vlan =3D 1; - break; - } + if (*action_flags & MLX5_ACTION_FLAG) { + flow_verbs_mark_update(verbs, mark->id); + size =3D 0; + } else { + tag.tag_id =3D mlx5_flow_mark_set(mark->id); + mlx5_flow_spec_verbs_add(dev_flow, &tag, size); } - if (has_vlan) - return rss_level < 2 ? MLX5_EXPANSION_ROOT_ETH_VLAN : - MLX5_EXPANSION_ROOT_OUTER_ETH_VLAN; - return rss_level < 2 ? MLX5_EXPANSION_ROOT : - MLX5_EXPANSION_ROOT_OUTER; + *action_flags |=3D MLX5_ACTION_MARK; } =20 /** - * Convert the @p attributes, @p pattern, @p action, into an flow for the = NIC - * after ensuring the NIC will understand and process it correctly. - * The conversion is only performed item/action per item/action, each of - * them is written into the @p flow if its size is lesser or equal to @p - * flow_size. - * Validation and memory consumption computation are still performed until= the - * end, unless an error is encountered. + * Convert the @p action into a Verbs specification. This function assumes= that + * the input is valid and that there is space to insert the requested acti= on + * into the flow. This function also return the action that was added. * * @param[in] dev - * Pointer to Ethernet device. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small some - * garbage may be present. - * @param[in] attributes - * Flow rule attributes. - * @param[in] pattern - * Pattern specification (list terminated by the END pattern item). - * @param[in] actions - * Associated actions (list terminated by the END action). + * Pointer to the Ethernet device structure. + * @param[in] action + * Action configuration. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. * @param[out] error - * Perform verbose error reporting if not NULL. + * Pointer to error structure. * * @return - * On success the number of bytes consumed/necessary, if the returned va= lue - * is lesser or equal to @p flow_size, the flow has fully been converted= and - * can be applied, otherwise another call with this returned memory size - * should be done. - * On error, a negative errno value is returned and rte_errno is set. + * 0 On success else a negative errno value is returned and rte_errno is= set. */ static int -mlx5_flow_merge(struct rte_eth_dev *dev, struct rte_flow *flow, - const size_t flow_size, - const struct rte_flow_attr *attributes, - const struct rte_flow_item pattern[], - const struct rte_flow_action actions[], - struct rte_flow_error *error) +flow_verbs_translate_action_count(struct rte_eth_dev *dev, + const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow, + struct rte_flow_error *error) { - struct rte_flow local_flow =3D { .layers =3D 0, }; - size_t size =3D sizeof(*flow); - union { - struct rte_flow_expand_rss buf; - uint8_t buffer[2048]; - } expand_buffer; - struct rte_flow_expand_rss *buf =3D &expand_buffer.buf; - struct mlx5_flow_verbs *original_verbs =3D NULL; - size_t original_verbs_size =3D 0; - uint32_t original_layers =3D 0; - int expanded_pattern_idx =3D 0; - int ret =3D 0; - uint32_t i; - - if (attributes->transfer) - return mlx5_flow_merge_switch(dev, flow, flow_size, - attributes, pattern, - actions, error); - if (size > flow_size) - flow =3D &local_flow; - ret =3D mlx5_flow_attributes(dev->data->dev_private, attributes, flow); - if (ret < 0) - return ret; - ret =3D mlx5_flow_actions(dev, actions, &local_flow, 0, error); - if (ret < 0) - return ret; - if (local_flow.rss.types) { - unsigned int graph_root; + const struct rte_flow_action_count *count =3D action->conf; + struct rte_flow *flow =3D dev_flow->flow; +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + unsigned int size =3D sizeof(struct ibv_flow_spec_counter_action); + struct ibv_flow_spec_counter_action counter =3D { + .type =3D IBV_FLOW_SPEC_ACTION_COUNT, + .size =3D size, + }; +#endif =20 - graph_root =3D mlx5_find_graph_root(pattern, - local_flow.rss.level); - ret =3D rte_flow_expand_rss(buf, sizeof(expand_buffer.buffer), - pattern, local_flow.rss.types, - mlx5_support_expansion, - graph_root); - assert(ret > 0 && - (unsigned int)ret < sizeof(expand_buffer.buffer)); - } else { - buf->entries =3D 1; - buf->entry[0].pattern =3D (void *)(uintptr_t)pattern; - } - size +=3D RTE_ALIGN_CEIL(local_flow.rss.queue_num * sizeof(uint16_t), - sizeof(void *)); - if (size <=3D flow_size) - flow->queue =3D (void *)(flow + 1); - LIST_INIT(&flow->verbs); - flow->layers =3D 0; - flow->modifier =3D 0; - flow->fate =3D 0; - for (i =3D 0; i !=3D buf->entries; ++i) { - size_t off =3D size; - size_t off2; - - flow->layers =3D original_layers; - size +=3D sizeof(struct ibv_flow_attr) + - sizeof(struct mlx5_flow_verbs); - off2 =3D size; - if (size < flow_size) { - flow->cur_verbs =3D (void *)((uintptr_t)flow + off); - flow->cur_verbs->attr =3D (void *)(flow->cur_verbs + 1); - flow->cur_verbs->specs =3D - (void *)(flow->cur_verbs->attr + 1); - } - /* First iteration convert the pattern into Verbs. */ - if (i =3D=3D 0) { - /* Actions don't need to be converted several time. */ - ret =3D mlx5_flow_actions(dev, actions, flow, - (size < flow_size) ? - flow_size - size : 0, - error); - if (ret < 0) - return ret; - size +=3D ret; - } else { - /* - * Next iteration means the pattern has already been - * converted and an expansion is necessary to match - * the user RSS request. For that only the expanded - * items will be converted, the common part with the - * user pattern are just copied into the next buffer - * zone. - */ - size +=3D original_verbs_size; - if (size < flow_size) { - rte_memcpy(flow->cur_verbs->attr, - original_verbs->attr, - original_verbs_size + - sizeof(struct ibv_flow_attr)); - flow->cur_verbs->size =3D original_verbs_size; - } - } - ret =3D mlx5_flow_items - ((const struct rte_flow_item *) - &buf->entry[i].pattern[expanded_pattern_idx], - flow, - (size < flow_size) ? flow_size - size : 0, error); - if (ret < 0) - return ret; - size +=3D ret; - if (size <=3D flow_size) { - mlx5_flow_adjust_priority(dev, flow); - LIST_INSERT_HEAD(&flow->verbs, flow->cur_verbs, next); - } - /* - * Keep a pointer of the first verbs conversion and the layers - * it has encountered. - */ - if (i =3D=3D 0) { - original_verbs =3D flow->cur_verbs; - original_verbs_size =3D size - off2; - original_layers =3D flow->layers; - /* - * move the index of the expanded pattern to the - * first item not addressed yet. - */ - if (pattern->type =3D=3D RTE_FLOW_ITEM_TYPE_END) { - expanded_pattern_idx++; - } else { - const struct rte_flow_item *item =3D pattern; - - for (item =3D pattern; - item->type !=3D RTE_FLOW_ITEM_TYPE_END; - ++item) - expanded_pattern_idx++; - } - } + if (!flow->counter) { + flow->counter =3D mlx5_flow_counter_new(dev, count->shared, + count->id); + if (!flow->counter) + return rte_flow_error_set(error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ACTION, + action, + "cannot get counter" + " context."); } - /* Restore the origin layers in the flow. */ - flow->layers =3D original_layers; - return size; + *action_flags |=3D MLX5_ACTION_COUNT; +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + counter.counter_set_handle =3D flow->counter->cs->handle; + mlx5_flow_spec_verbs_add(dev_flow, &counter, size); +#endif + return 0; } =20 /** @@ -2212,7 +1585,7 @@ mlx5_flow_rxq_tunnel_ptype_update(struct mlx5_rxq_ctr= l *rxq_ctrl) * Set the Rx queue flags (Mark/Flag and Tunnel Ptypes) according to the f= low. * * @param[in] dev - * Pointer to Ethernet device. + * Pointer to the Ethernet device structure. * @param[in] flow * Pointer to flow structure. */ @@ -2220,8 +1593,8 @@ static void mlx5_flow_rxq_flags_set(struct rte_eth_dev *dev, struct rte_flow *flow) { struct priv *priv =3D dev->data->dev_private; - const int mark =3D !!(flow->modifier & - (MLX5_FLOW_MOD_FLAG | MLX5_FLOW_MOD_MARK)); + const int mark =3D !!(flow->actions & + (MLX5_ACTION_FLAG | MLX5_ACTION_MARK)); const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; =20 @@ -2264,8 +1637,8 @@ static void mlx5_flow_rxq_flags_trim(struct rte_eth_dev *dev, struct rte_flow *flow) { struct priv *priv =3D dev->data->dev_private; - const int mark =3D !!(flow->modifier & - (MLX5_FLOW_MOD_FLAG | MLX5_FLOW_MOD_MARK)); + const int mark =3D !!(flow->actions & + (MLX5_ACTION_FLAG | MLX5_ACTION_MARK)); const int tunnel =3D !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; =20 @@ -3574,21 +2947,21 @@ mlx5_flow_verbs_get_items_and_size(const struct rte= _flow_item items[], size +=3D sizeof(struct ibv_flow_spec_tunnel); detected_items |=3D MLX5_FLOW_LAYER_VXLAN_GPE; break; - case RTE_FLOW_ITEM_TYPE_GRE: #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT + case RTE_FLOW_ITEM_TYPE_GRE: size +=3D sizeof(struct ibv_flow_spec_gre); detected_items |=3D MLX5_FLOW_LAYER_GRE; -#else - size +=3D sizeof(struct ibv_flow_spec_tunnel); - detected_items |=3D MLX5_FLOW_LAYER_TUNNEL; -#endif break; case RTE_FLOW_ITEM_TYPE_MPLS: -#ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT size +=3D sizeof(struct ibv_flow_spec_mpls); detected_items |=3D MLX5_FLOW_LAYER_MPLS; -#endif break; +#else + case RTE_FLOW_ITEM_TYPE_GRE: + size +=3D sizeof(struct ibv_flow_spec_tunnel); + detected_items |=3D MLX5_FLOW_LAYER_TUNNEL; + break; +#endif default: break; } @@ -3651,7 +3024,7 @@ mlx5_flow_verbs_prepare(const struct rte_flow_attr *a= ttr __rte_unused, uint64_t *action_flags, struct rte_flow_error *error) { - uint32_t size =3D sizeof(struct ibv_flow_attr); + uint32_t size =3D sizeof(struct mlx5_flow) + sizeof(struct ibv_flow_attr)= ; struct mlx5_flow *flow; =20 size +=3D mlx5_flow_verbs_get_actions_and_size(actions, action_flags); @@ -3664,10 +3037,150 @@ mlx5_flow_verbs_prepare(const struct rte_flow_attr= *attr __rte_unused, "not enough memory to create flow"); return NULL; } + flow->verbs.attr =3D (void *)(flow + 1); + flow->verbs.specs =3D (uint8_t *)(flow + 1) + + sizeof(struct ibv_flow_attr); return flow; } =20 /** + * + * Fill the flow with verb spec. + * + * @param[in] dev + * Pointer to Ethernet device. + * @param[in, out] dev_flow + * Pointer to the mlx5 flow. + * @param[in] attr + * Pointer to the flow attributes. + * @param[in] items + * Pointer to the list of items. + * @param[in] actions + * Pointer to the list of actions. + * @param[out] error + * Pointer to the error structure. + * + * @return + * 0 on success, else a negative errno value otherwise and rte_ernno is = set. + */ +static int mlx5_flow_verbs_translate(struct rte_eth_dev *dev, + struct mlx5_flow *dev_flow, + const struct rte_flow_attr *attr, + const struct rte_flow_item items[], + const struct rte_flow_action actions[], + struct rte_flow_error *error) +{ + uint64_t action_flags =3D 0; + uint64_t item_flags =3D 0; + uint64_t priority =3D attr->priority; + struct priv *priv =3D dev->data->dev_private; + + if (priority =3D=3D MLX5_FLOW_PRIO_RSVD) + priority =3D priv->config.flow_prio - 1; + for (; actions->type !=3D RTE_FLOW_ACTION_TYPE_END; actions++) { + int ret; + switch (actions->type) { + case RTE_FLOW_ACTION_TYPE_VOID: + break; + case RTE_FLOW_ACTION_TYPE_FLAG: + flow_verbs_translate_action_flag(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_MARK: + flow_verbs_translate_action_mark(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_DROP: + flow_verbs_translate_action_drop(&action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_QUEUE: + flow_verbs_translate_action_queue(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_RSS: + flow_verbs_translate_action_rss(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_COUNT: + ret =3D flow_verbs_translate_action_count(dev, + actions, + &action_flags, + dev_flow, + error); + if (ret < 0) + return ret; + break; + default: + return rte_flow_error_set(error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ACTION, + actions, + "action not supported"); + } + } + dev_flow->flow->actions |=3D action_flags; + for (; items->type !=3D RTE_FLOW_ITEM_TYPE_END; items++) { + switch (items->type) { + case RTE_FLOW_ITEM_TYPE_VOID: + break; + case RTE_FLOW_ITEM_TYPE_ETH: + flow_verbs_translate_item_eth(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_VLAN: + flow_verbs_translate_item_vlan(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_IPV4: + flow_verbs_translate_item_ipv4(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_IPV6: + flow_verbs_translate_item_ipv6(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_UDP: + flow_verbs_translate_item_udp(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_TCP: + flow_verbs_translate_item_tcp(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_VXLAN: + flow_verbs_translate_item_vxlan(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_VXLAN_GPE: + flow_verbs_translate_item_vxlan_gpe(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_GRE: + flow_verbs_translate_item_gre(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_MPLS: + flow_verbs_translate_item_mpls(items, &item_flags, + dev_flow); + break; + default: + return rte_flow_error_set(error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ITEM, + NULL, + "item not supported"); + } + } + dev_flow->verbs.attr->priority =3D mlx5_flow_adjust_priority(dev, + priority, + dev_flow->verbs.attr->priority); + return 0; +} + +/** * Remove the flow. * * @param[in] dev @@ -3680,16 +3193,18 @@ mlx5_flow_remove(struct rte_eth_dev *dev, struct rt= e_flow *flow) { struct priv *priv =3D dev->data->dev_private; struct mlx5_flow_verbs *verbs; + struct mlx5_flow *dev_flow; =20 if (flow->nl_flow && priv->mnl_socket) mlx5_nl_flow_destroy(priv->mnl_socket, flow->nl_flow, NULL); - LIST_FOREACH(verbs, &flow->verbs, next) { + LIST_FOREACH(dev_flow, &flow->dev_flows, next) { + verbs =3D &dev_flow->verbs; if (verbs->flow) { claim_zero(mlx5_glue->destroy_flow(verbs->flow)); verbs->flow =3D NULL; } if (verbs->hrxq) { - if (flow->fate & MLX5_FLOW_FATE_DROP) + if (flow->actions & MLX5_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); @@ -3721,10 +3236,12 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte= _flow *flow, { struct priv *priv =3D dev->data->dev_private; struct mlx5_flow_verbs *verbs; + struct mlx5_flow *dev_flow; int err; =20 - LIST_FOREACH(verbs, &flow->verbs, next) { - if (flow->fate & MLX5_FLOW_FATE_DROP) { + LIST_FOREACH(dev_flow, &flow->dev_flows, next) { + verbs =3D &dev_flow->verbs; + if (flow->actions & MLX5_ACTION_DROP) { verbs->hrxq =3D mlx5_hrxq_drop_new(dev); if (!verbs->hrxq) { rte_flow_error_set @@ -3760,8 +3277,8 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte_f= low *flow, } verbs->hrxq =3D hrxq; } - verbs->flow =3D - mlx5_glue->create_flow(verbs->hrxq->qp, verbs->attr); + verbs->flow =3D mlx5_glue->create_flow(verbs->hrxq->qp, + verbs->attr); if (!verbs->flow) { rte_flow_error_set(error, errno, RTE_FLOW_ERROR_TYPE_UNSPECIFIED, @@ -3777,9 +3294,10 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte_= flow *flow, return 0; error: err =3D rte_errno; /* Save rte_errno before cleanup. */ - LIST_FOREACH(verbs, &flow->verbs, next) { + LIST_FOREACH(dev_flow, &flow->dev_flows, next) { + verbs =3D &dev_flow->verbs; if (verbs->hrxq) { - if (flow->fate & MLX5_FLOW_FATE_DROP) + if (flow->actions & MLX5_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); @@ -3790,6 +3308,25 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte_= flow *flow, return -rte_errno; } =20 +static unsigned int +mlx5_find_graph_root(const struct rte_flow_item pattern[], uint32_t rss_le= vel) +{ + const struct rte_flow_item *item; + unsigned int has_vlan =3D 0; + + for (item =3D pattern; item->type !=3D RTE_FLOW_ITEM_TYPE_END; item++) { + if (item->type =3D=3D RTE_FLOW_ITEM_TYPE_VLAN) { + has_vlan =3D 1; + break; + } + } + if (has_vlan) + return rss_level < 2 ? MLX5_EXPANSION_ROOT_ETH_VLAN : + MLX5_EXPANSION_ROOT_OUTER_ETH_VLAN; + return rss_level < 2 ? MLX5_EXPANSION_ROOT : + MLX5_EXPANSION_ROOT_OUTER; +} + /** * Create a flow and add it to @p list. * @@ -3819,7 +3356,6 @@ mlx5_flow_list_create(struct rte_eth_dev *dev, { struct rte_flow *flow =3D NULL; struct mlx5_flow *dev_flow; - size_t size =3D 0; uint64_t action_flags =3D 0; uint64_t item_flags =3D 0; const struct rte_flow_action_rss *rss; @@ -3830,13 +3366,21 @@ mlx5_flow_list_create(struct rte_eth_dev *dev, struct rte_flow_expand_rss *buf =3D &expand_buffer.buf; int ret; uint32_t i; + uint32_t flow_size; =20 ret =3D mlx5_flow_validate(dev, attr, items, actions, error); if (ret < 0) return NULL; - flow =3D rte_calloc(__func__, 1, sizeof(*flow), 0); - LIST_INIT(&flow->dev_flows); + flow_size =3D sizeof(struct rte_flow); rss =3D mlx5_flow_get_rss_action(actions); + if (rss) + flow_size +=3D RTE_ALIGN_CEIL(rss->queue_num * sizeof(uint16_t), + sizeof(void *)); + else + flow_size +=3D RTE_ALIGN_CEIL(sizeof(uint16_t), sizeof(void *)); + flow =3D rte_calloc(__func__, 1, flow_size, 0); + flow->queue =3D (void *)(flow + 1); + LIST_INIT(&flow->dev_flows); if (rss && rss->types) { unsigned int graph_root; =20 @@ -3857,25 +3401,11 @@ mlx5_flow_list_create(struct rte_eth_dev *dev, &action_flags, error); dev_flow->flow =3D flow; LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); + mlx5_flow_verbs_translate(dev, dev_flow, attr, + buf->entry[i].pattern, + actions, + error); } - ret =3D mlx5_flow_merge(dev, flow, size, attr, items, actions, error); - if (ret < 0) - return NULL; - size =3D ret; - flow =3D rte_calloc(__func__, 1, size, 0); - if (!flow) { - rte_flow_error_set(error, ENOMEM, - RTE_FLOW_ERROR_TYPE_UNSPECIFIED, - NULL, - "not enough memory to create flow"); - return NULL; - } - ret =3D mlx5_flow_merge(dev, flow, size, attr, items, actions, error); - if (ret < 0) { - rte_free(flow); - return NULL; - } - assert((size_t)ret =3D=3D size); if (dev->data->dev_started) { ret =3D mlx5_flow_apply(dev, flow, error); if (ret < 0) { @@ -3933,6 +3463,12 @@ mlx5_flow_list_destroy(struct rte_eth_dev *dev, stru= ct mlx5_flows *list, */ if (dev->data->dev_started) mlx5_flow_rxq_flags_trim(dev, flow); + while (!LIST_EMPTY(&flow->dev_flows)) { + struct mlx5_flow *dev_flow; + dev_flow =3D LIST_FIRST(&flow->dev_flows); + LIST_REMOVE(dev_flow, next); + rte_free(dev_flow); + } rte_free(flow); } =20 @@ -4208,7 +3744,7 @@ mlx5_flow_query_count(struct rte_flow *flow __rte_unu= sed, struct rte_flow_error *error) { #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - if (flow->modifier & MLX5_FLOW_MOD_COUNT) { + if (flow->actions & MLX5_ACTION_COUNT) { struct rte_flow_query_count *qc =3D data; uint64_t counters[2] =3D {0, 0}; struct ibv_query_counter_set_attr query_cs_attr =3D { --=20 2.11.0