[RFC 0/9] add hws flex item matching and modify field

[RFC 0/9] add hws flex item matching and modify field

[Rongwei Liu]

Support flex item matching and modify field action in dv_flow_en=2.
The syntax follows current style exactly.

Rongwei Liu (9):
  ethdev: add flex item modify field support
  app/testpmd: add flex item modify field cmdline support
  app/testpmd: pass flex handle into matching mask
  net/mlx5: enable hws flex item create
  net/mlx5: add IPv6 protocol as flex item input
  net/mlx5/hws: add hws flex item matching support
  net/mlx5/hws: add flex item modify field implementation
  net/mlx5: return error for sws modify field
  app/testpmd: raw encap with flex item support

 app/test-pmd/cmdline_flow.c            | 122 ++++++++++++++---
 doc/guides/nics/features/mlx5.ini      |   1 +
 doc/guides/nics/mlx5.rst               |   2 +
 doc/guides/prog_guide/rte_flow.rst     |  36 ++---
 doc/guides/rel_notes/release_22_03.rst |   4 +
 drivers/common/mlx5/mlx5_devx_cmds.c   |  14 +-
 drivers/common/mlx5/mlx5_devx_cmds.h   |   7 +-
 drivers/common/mlx5/mlx5_prm.h         |  23 +++-
 drivers/net/mlx5/hws/mlx5dr_definer.c  | 141 +++++++++++++++++++
 drivers/net/mlx5/linux/mlx5_os.c       |  27 ++--
 drivers/net/mlx5/mlx5.c                |  17 ++-
 drivers/net/mlx5/mlx5.h                |   8 +-
 drivers/net/mlx5/mlx5_flow.h           |   4 +
 drivers/net/mlx5/mlx5_flow_dv.c        | 183 ++++++++++++++++++++++---
 drivers/net/mlx5/mlx5_flow_flex.c      | 102 +++++++++++---
 drivers/net/mlx5/mlx5_flow_hw.c        |  64 ++++++++-
 lib/ethdev/rte_flow.h                  |  10 +-
 17 files changed, 662 insertions(+), 103 deletions(-)

-- 
2.27.0

[RFC 1/9] ethdev: add flex item modify field support

[Rongwei Liu]

Add flex item as modify field destination.
Add "struct rte_flow_item_flex_handle *flex_handle" into
"struct rte_flow_action_modify_data" as union with existed
"level" member. This new member is dedicated for modifying
flex item.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 doc/guides/prog_guide/rte_flow.rst     | 36 ++++++++++++++------------
 doc/guides/rel_notes/release_22_03.rst |  4 +++
 lib/ethdev/rte_flow.h                  | 10 ++++---
 3 files changed, 30 insertions(+), 20 deletions(-)

diff --git a/doc/guides/prog_guide/rte_flow.rst b/doc/guides/prog_guide/rte_flow.rst
index 59932e82a6..db7063c4e0 100644
--- a/doc/guides/prog_guide/rte_flow.rst
+++ b/doc/guides/prog_guide/rte_flow.rst
@@ -2966,23 +2966,25 @@ value as sequence of bytes {xxx, xxx, 0x85, xxx, xxx, xxx}.
 
 .. table:: destination/source field definition
 
-   +---------------+----------------------------------------------------------+
-   | Field         | Value                                                    |
-   +===============+==========================================================+
-   | ``field``     | ID: packet field, mark, meta, tag, immediate, pointer    |
-   +---------------+----------------------------------------------------------+
-   | ``level``     | encapsulation level of a packet field or tag array index |
-   +---------------+----------------------------------------------------------+
-   | ``offset``    | number of bits to skip at the beginning                  |
-   +---------------+----------------------------------------------------------+
-   | ``value``     | immediate value buffer (source field only, not           |
-   |               | applicable to destination) for RTE_FLOW_FIELD_VALUE      |
-   |               | field type                                               |
-   +---------------+----------------------------------------------------------+
-   | ``pvalue``    | pointer to immediate value data (source field only, not  |
-   |               | applicable to destination) for RTE_FLOW_FIELD_POINTER    |
-   |               | field type                                               |
-   +---------------+----------------------------------------------------------+
+   +-----------------+----------------------------------------------------------+
+   | Field           | Value                                                    |
+   +=================+==========================================================+
+   | ``field``       | ID: packet field, mark, meta, tag, immediate, pointer    |
+   +-----------------+----------------------------------------------------------+
+   | ``level``       | encapsulation level of a packet field or tag array index |
+   +-----------------+----------------------------------------------------------+
+   | ``flex_handle`` | flex item handle of a packet field                       |
+   +-----------------+----------------------------------------------------------+
+   | ``offset``      | number of bits to skip at the beginning                  |
+   +-----------------+----------------------------------------------------------+
+   | ``value``       | immediate value buffer (source field only, not           |
+   |                 | applicable to destination) for RTE_FLOW_FIELD_VALUE      |
+   |                 | field type                                               |
+   +-----------------+----------------------------------------------------------+
+   | ``pvalue``      | pointer to immediate value data (source field only, not  |
+   |                 | applicable to destination) for RTE_FLOW_FIELD_POINTER    |
+   |                 | field type                                               |
+   +-----------------+----------------------------------------------------------+
 
 Action: ``CONNTRACK``
 ^^^^^^^^^^^^^^^^^^^^^
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 0923707cb8..5fc5aff8a4 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -207,6 +207,10 @@ API Changes
 * ethdev: Old public macros and enumeration constants without ``RTE_ETH_`` prefix,
   which are kept for backward compatibility, are marked as deprecated.
 
+* ethdev: added a new field:
+
+  - modify flex item: ``rte_flow_action_modify_data.flex_handle``
+
 * cryptodev: The asymmetric session handling was modified to use a single
   mempool object. An API ``rte_cryptodev_asym_session_pool_create`` was added
   to create a mempool with element size big enough to hold the generic asymmetric
diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index 21f7caf540..5f53a8ea19 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -3800,7 +3800,8 @@ enum rte_flow_field_id {
 	RTE_FLOW_FIELD_IPV6_ECN,	/**< IPv6 ECN. */
 	RTE_FLOW_FIELD_GTP_PSC_QFI,	/**< GTP QFI. */
 	RTE_FLOW_FIELD_METER_COLOR,	/**< Meter color marker. */
-	RTE_FLOW_FIELD_HASH_RESULT, /**< Hash result. */
+	RTE_FLOW_FIELD_HASH_RESULT,	/**< Hash result. */
+	RTE_FLOW_FIELD_FLEX_ITEM	/**< Flex item. */
 };
 
 /**
@@ -3814,8 +3815,11 @@ struct rte_flow_action_modify_data {
 	RTE_STD_C11
 	union {
 		struct {
-			/** Encapsulation level or tag index. */
-			uint32_t level;
+			/**< Encapsulation level or tag index or flex item handle. */
+			union {
+				uint32_t level;
+				struct rte_flow_item_flex_handle *flex_handle;
+			};
 			/** Number of bits to skip from a field. */
 			uint32_t offset;
 		};
-- 
2.27.0

[RFC 2/9] app/testpmd: add flex item modify field cmdline support

[Rongwei Liu]

Add flex item modify field cmdline support. Now user can use
testpmd cli to specify which flex item to be modified, either
source or destination.

Syntax is as below:
modify_field op set dst_type flex_item dst_level 0
dst_offset 16 src_type value src_value 0x123456781020 width 8

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 88 ++++++++++++++++++++++++++++++++++---
 1 file changed, 81 insertions(+), 7 deletions(-)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index 426585387f..d25bd2f348 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -651,10 +651,12 @@ enum index {
 	ACTION_MODIFY_FIELD_DST_TYPE,
 	ACTION_MODIFY_FIELD_DST_TYPE_VALUE,
 	ACTION_MODIFY_FIELD_DST_LEVEL,
+	ACTION_MODIFY_FIELD_DST_LEVEL_VALUE,
 	ACTION_MODIFY_FIELD_DST_OFFSET,
 	ACTION_MODIFY_FIELD_SRC_TYPE,
 	ACTION_MODIFY_FIELD_SRC_TYPE_VALUE,
 	ACTION_MODIFY_FIELD_SRC_LEVEL,
+	ACTION_MODIFY_FIELD_SRC_LEVEL_VALUE,
 	ACTION_MODIFY_FIELD_SRC_OFFSET,
 	ACTION_MODIFY_FIELD_SRC_VALUE,
 	ACTION_MODIFY_FIELD_SRC_POINTER,
@@ -881,7 +883,7 @@ static const char *const modify_field_ids[] = {
 	"vxlan_vni", "geneve_vni", "gtp_teid",
 	"tag", "mark", "meta", "pointer", "value",
 	"ipv4_ecn", "ipv6_ecn", "gtp_psc_qfi", "meter_color",
-	"hash_result",
+	"hash_result", "flex_item",
 	NULL
 };
 
@@ -2557,6 +2559,10 @@ parse_vc_modify_field_id(struct context *ctx, const struct token *token,
 				const char *str, unsigned int len, void *buf,
 				unsigned int size);
 static int
+parse_vc_modify_field_level(struct context *ctx, const struct token *token,
+				const char *str, unsigned int len, void *buf,
+				unsigned int size);
+static int
 parse_vc_action_conntrack_update(struct context *ctx, const struct token *token,
 			 const char *str, unsigned int len, void *buf,
 			 unsigned int size);
@@ -6461,11 +6467,15 @@ static const struct token token_list[] = {
 		.name = "dst_level",
 		.help = "destination field level",
 		.next = NEXT(action_modify_field_dst,
-			     NEXT_ENTRY(COMMON_UNSIGNED)),
-		.args = ARGS(ARGS_ENTRY(struct rte_flow_action_modify_field,
-					dst.level)),
+			     NEXT_ENTRY(ACTION_MODIFY_FIELD_DST_LEVEL_VALUE)),
 		.call = parse_vc_conf,
 	},
+	[ACTION_MODIFY_FIELD_DST_LEVEL_VALUE] = {
+		.name = "{dst_level}",
+		.help = "destination field level value",
+		.call = parse_vc_modify_field_level,
+		.comp = comp_none,
+	},
 	[ACTION_MODIFY_FIELD_DST_OFFSET] = {
 		.name = "dst_offset",
 		.help = "destination field bit offset",
@@ -6492,11 +6502,15 @@ static const struct token token_list[] = {
 		.name = "src_level",
 		.help = "source field level",
 		.next = NEXT(action_modify_field_src,
-			     NEXT_ENTRY(COMMON_UNSIGNED)),
-		.args = ARGS(ARGS_ENTRY(struct rte_flow_action_modify_field,
-					src.level)),
+			     NEXT_ENTRY(ACTION_MODIFY_FIELD_SRC_LEVEL_VALUE)),
 		.call = parse_vc_conf,
 	},
+	[ACTION_MODIFY_FIELD_SRC_LEVEL_VALUE] = {
+		.name = "{src_level}",
+		.help = "source field level value",
+		.call = parse_vc_modify_field_level,
+		.comp = comp_none,
+	},
 	[ACTION_MODIFY_FIELD_SRC_OFFSET] = {
 		.name = "src_offset",
 		.help = "source field bit offset",
@@ -9542,6 +9556,66 @@ parse_vc_modify_field_id(struct context *ctx, const struct token *token,
 	return len;
 }
 
+/** Parse level for modify_field command. */
+static int
+parse_vc_modify_field_level(struct context *ctx, const struct token *token,
+			 const char *str, unsigned int len, void *buf,
+			 unsigned int size)
+{
+	struct rte_flow_action_modify_field *action;
+	struct flex_item *fp;
+	uint32_t val;
+	struct buffer *out = buf;
+	char *end;
+
+	(void)token;
+	(void)size;
+	if (ctx->curr != ACTION_MODIFY_FIELD_DST_LEVEL_VALUE &&
+		ctx->curr != ACTION_MODIFY_FIELD_SRC_LEVEL_VALUE)
+		return -1;
+	if (!ctx->object)
+		return len;
+	action = ctx->object;
+	errno = 0;
+	val = strtoumax(str, &end, 0);
+	if (errno || (size_t)(end - str) != len)
+		return -1;
+	/* No need to validate action template mask value */
+	if (out->args.vc.masks) {
+		if (ctx->curr == ACTION_MODIFY_FIELD_DST_LEVEL_VALUE)
+			action->dst.level = val;
+		else
+			action->src.level = val;
+		return len;
+	}
+	if ((ctx->curr == ACTION_MODIFY_FIELD_DST_LEVEL_VALUE &&
+		action->dst.field == RTE_FLOW_FIELD_FLEX_ITEM) ||
+		(ctx->curr == ACTION_MODIFY_FIELD_SRC_LEVEL_VALUE &&
+		action->src.field == RTE_FLOW_FIELD_FLEX_ITEM)) {
+		if (val >= FLEX_MAX_PARSERS_NUM) {
+			printf("Bad flex item handle\n");
+			return -1;
+		}
+		fp = flex_items[ctx->port][val];
+		if (!fp) {
+			printf("Bad flex item handle\n");
+			return -1;
+		}
+	}
+	if (ctx->curr == ACTION_MODIFY_FIELD_DST_LEVEL_VALUE) {
+		if (action->dst.field != RTE_FLOW_FIELD_FLEX_ITEM)
+			action->dst.level = val;
+		else
+			action->dst.flex_handle = fp->flex_handle;
+	} else if (ctx->curr == ACTION_MODIFY_FIELD_SRC_LEVEL_VALUE) {
+		if (action->src.field != RTE_FLOW_FIELD_FLEX_ITEM)
+			action->src.level = val;
+		else
+			action->src.flex_handle = fp->flex_handle;
+	}
+	return len;
+}
+
 /** Parse the conntrack update, not a rte_flow_action. */
 static int
 parse_vc_action_conntrack_update(struct context *ctx, const struct token *token,
-- 
2.27.0

[RFC 3/9] app/testpmd: pass flex handle into matching mask

[Rongwei Liu]

In async flow create API, there is only mask information when
creating flow table but flex item handle is required to parse
the HW sample information.

Pass the flex item handle instead of UINT64/32_MAX to mask.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index d25bd2f348..1158da2122 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -11172,8 +11172,8 @@ parse_flex_handle(struct context *ctx, const struct token *token,
 	}
 	if (offset == offsetof(struct rte_flow_item_flex, handle)) {
 		const struct flex_item *fp;
-		struct rte_flow_item_flex *item_flex = ctx->object;
-		handle = (uint16_t)(uintptr_t)item_flex->handle;
+		spec = ctx->object;
+		handle = (uint16_t)(uintptr_t)spec->handle;
 		if (handle >= FLEX_MAX_PARSERS_NUM) {
 			printf("Bad flex item handle\n");
 			return -1;
@@ -11183,7 +11183,9 @@ parse_flex_handle(struct context *ctx, const struct token *token,
 			printf("Bad flex item handle\n");
 			return -1;
 		}
-		item_flex->handle = fp->flex_handle;
+		spec->handle = fp->flex_handle;
+		mask = spec + 2; /* spec, last, mask */
+		mask->handle = fp->flex_handle;
 	} else if (offset == offsetof(struct rte_flow_item_flex, pattern)) {
 		handle = (uint16_t)(uintptr_t)
 			((struct rte_flow_item_flex *)ctx->object)->pattern;
-- 
2.27.0

[RFC 4/9] net/mlx5: enable hws flex item create

[Rongwei Liu]

Enable flex item create and destroy with dv_flow_en=2

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 drivers/net/mlx5/linux/mlx5_os.c | 27 +++++++++++++++------------
 drivers/net/mlx5/mlx5_flow_hw.c  |  2 ++
 2 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index d48b9b68ac..3022c0ea1c 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -486,10 +486,20 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
 	err = mlx5_alloc_table_hash_list(priv);
 	if (err)
 		goto error;
-	if (priv->sh->config.dv_flow_en == 2)
-		return 0;
 	/* The resources below are only valid with DV support. */
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	/* Init shared flex parsers list, no need lcore_share */
+	snprintf(s, sizeof(s), "%s_flex_parsers_list", sh->ibdev_name);
+	sh->flex_parsers_dv = mlx5_list_create(s, sh, false,
+					       mlx5_flex_parser_create_cb,
+					       mlx5_flex_parser_match_cb,
+					       mlx5_flex_parser_remove_cb,
+					       mlx5_flex_parser_clone_cb,
+					       mlx5_flex_parser_clone_free_cb);
+	if (!sh->flex_parsers_dv)
+		goto error;
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 	/* Init port id action list. */
 	snprintf(s, sizeof(s), "%s_port_id_action_list", sh->ibdev_name);
 	sh->port_id_action_list = mlx5_list_create(s, sh, true,
@@ -530,16 +540,9 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
 					      flow_dv_dest_array_clone_free_cb);
 	if (!sh->dest_array_list)
 		goto error;
-	/* Init shared flex parsers list, no need lcore_share */
-	snprintf(s, sizeof(s), "%s_flex_parsers_list", sh->ibdev_name);
-	sh->flex_parsers_dv = mlx5_list_create(s, sh, false,
-					       mlx5_flex_parser_create_cb,
-					       mlx5_flex_parser_match_cb,
-					       mlx5_flex_parser_remove_cb,
-					       mlx5_flex_parser_clone_cb,
-					       mlx5_flex_parser_clone_free_cb);
-	if (!sh->flex_parsers_dv)
-		goto error;
+#else
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 #endif
 #ifdef HAVE_MLX5DV_DR
 	void *domain;
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 0705002d99..b9c7459646 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -8824,6 +8824,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
 	.query = flow_hw_query,
 	.get_aged_flows = flow_hw_get_aged_flows,
 	.get_q_aged_flows = flow_hw_get_q_aged_flows,
+	.item_create = flow_dv_item_create,
+	.item_release = flow_dv_item_release,
 };
 
 /**
-- 
2.27.0

[RFC 5/9] net/mlx5: add IPv6 protocol as flex item input

[Rongwei Liu]

Support IPv6 protocol as new flex item input link.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 drivers/net/mlx5/mlx5_flow_flex.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/net/mlx5/mlx5_flow_flex.c b/drivers/net/mlx5/mlx5_flow_flex.c
index fb08910ddb..bec07b13c1 100644
--- a/drivers/net/mlx5/mlx5_flow_flex.c
+++ b/drivers/net/mlx5/mlx5_flow_flex.c
@@ -1043,6 +1043,22 @@ mlx5_flex_arc_in_udp(const struct rte_flow_item *item,
 	return rte_be_to_cpu_16(spec->hdr.dst_port);
 }
 
+static int
+mlx5_flex_arc_in_ipv6(const struct rte_flow_item *item,
+		      struct rte_flow_error *error)
+{
+	const struct rte_flow_item_ipv6 *spec = item->spec;
+	const struct rte_flow_item_ipv6 *mask = item->mask;
+	struct rte_flow_item_ipv6 ip = { .hdr.proto = 0xff };
+
+	if (memcmp(mask, &ip, sizeof(struct rte_flow_item_ipv6))) {
+		return rte_flow_error_set
+			(error, EINVAL, RTE_FLOW_ERROR_TYPE_ITEM, item,
+			 "invalid ipv6 item mask, full mask is desired");
+	}
+	return spec->hdr.proto;
+}
+
 static int
 mlx5_flex_translate_arc_in(struct mlx5_hca_flex_attr *attr,
 			   const struct rte_flow_item_flex_conf *conf,
@@ -1089,6 +1105,9 @@ mlx5_flex_translate_arc_in(struct mlx5_hca_flex_attr *attr,
 		case RTE_FLOW_ITEM_TYPE_UDP:
 			ret = mlx5_flex_arc_in_udp(rte_item, error);
 			break;
+		case RTE_FLOW_ITEM_TYPE_IPV6:
+			ret = mlx5_flex_arc_in_ipv6(rte_item, error);
+			break;
 		default:
 			MLX5_ASSERT(false);
 			return rte_flow_error_set
-- 
2.27.0

[RFC 6/9] net/mlx5/hws: add hws flex item matching support

[Rongwei Liu]

Support flex item matching in hws and syntax follows
sws exactly.

Flex item should be created in advance and follow current
json mapping logic.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 doc/guides/nics/features/mlx5.ini     |   1 +
 doc/guides/nics/mlx5.rst              |   1 +
 drivers/common/mlx5/mlx5_devx_cmds.c  |  14 ++-
 drivers/common/mlx5/mlx5_devx_cmds.h  |   7 +-
 drivers/common/mlx5/mlx5_prm.h        |  22 +++-
 drivers/net/mlx5/hws/mlx5dr_definer.c | 141 ++++++++++++++++++++++++++
 drivers/net/mlx5/mlx5.c               |  17 +++-
 drivers/net/mlx5/mlx5.h               |   8 +-
 drivers/net/mlx5/mlx5_flow.h          |   1 +
 drivers/net/mlx5/mlx5_flow_flex.c     |  83 +++++++++++----
 drivers/net/mlx5/mlx5_flow_hw.c       |  48 ++++++++-
 11 files changed, 310 insertions(+), 33 deletions(-)

diff --git a/doc/guides/nics/features/mlx5.ini b/doc/guides/nics/features/mlx5.ini
index 62fd330e2b..135b5c035d 100644
--- a/doc/guides/nics/features/mlx5.ini
+++ b/doc/guides/nics/features/mlx5.ini
@@ -87,6 +87,7 @@ vlan                 = Y
 vxlan                = Y
 vxlan_gpe            = Y
 represented_port     = Y
+flex item            = Y
 
 [rte_flow actions]
 age                  = I
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 85a2b422c5..e442f9c015 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -106,6 +106,7 @@ Features
 - Sub-Function representors.
 - Sub-Function.
 - Matching on represented port.
+- Matching on flex item with specific pattern.
 
 
 Limitations
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 9e0b26fa11..deda33032c 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -607,7 +607,8 @@ mlx5_devx_cmd_query_hca_vdpa_attr(void *ctx,
 
 int
 mlx5_devx_cmd_query_parse_samples(struct mlx5_devx_obj *flex_obj,
-				  uint32_t ids[], uint32_t num)
+				  struct mlx5_ext_sample_id ids[],
+				  uint32_t num,  uint8_t *anchor)
 {
 	uint32_t in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {0};
 	uint32_t out[MLX5_ST_SZ_DW(create_flex_parser_out)] = {0};
@@ -636,6 +637,7 @@ mlx5_devx_cmd_query_parse_samples(struct mlx5_devx_obj *flex_obj,
 			(void *)flex_obj);
 		return -rte_errno;
 	}
+	*anchor = MLX5_GET(parse_graph_flex, flex, head_anchor_id);
 	for (i = 0; i < MLX5_GRAPH_NODE_SAMPLE_NUM; i++) {
 		void *s_off = (void *)((char *)sample + i *
 			      MLX5_ST_SZ_BYTES(parse_graph_flow_match_sample));
@@ -645,8 +647,8 @@ mlx5_devx_cmd_query_parse_samples(struct mlx5_devx_obj *flex_obj,
 			      flow_match_sample_en);
 		if (!en)
 			continue;
-		ids[idx++] = MLX5_GET(parse_graph_flow_match_sample, s_off,
-				  flow_match_sample_field_id);
+		ids[idx++].id = MLX5_GET(parse_graph_flow_match_sample, s_off,
+					 flow_match_sample_field_id);
 	}
 	if (num != idx) {
 		rte_errno = EINVAL;
@@ -794,6 +796,12 @@ mlx5_devx_cmd_query_hca_parse_graph_node_cap
 					 max_num_arc_out);
 	attr->max_num_sample = MLX5_GET(parse_graph_node_cap, hcattr,
 					max_num_sample);
+	attr->anchor_en = MLX5_GET(parse_graph_node_cap, hcattr, anchor_en);
+	attr->ext_sample_id = MLX5_GET(parse_graph_node_cap, hcattr, ext_sample_id);
+	attr->sample_tunnel_inner2 = MLX5_GET(parse_graph_node_cap, hcattr,
+					      sample_tunnel_inner2);
+	attr->zero_size_supported = MLX5_GET(parse_graph_node_cap, hcattr,
+					     zero_size_supported);
 	attr->sample_id_in_out = MLX5_GET(parse_graph_node_cap, hcattr,
 					  sample_id_in_out);
 	attr->max_base_header_length = MLX5_GET(parse_graph_node_cap, hcattr,
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 1c86426e71..eff5a31b2e 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -114,6 +114,10 @@ struct mlx5_hca_flex_attr {
 	uint8_t  max_num_arc_out;
 	uint8_t  max_num_sample;
 	uint8_t  max_num_prog_sample:5;	/* From HCA CAP 2 */
+	uint8_t  anchor_en:1;
+	uint8_t  ext_sample_id:1;
+	uint8_t  sample_tunnel_inner2:1;
+	uint8_t  zero_size_supported:1;
 	uint8_t  sample_id_in_out:1;
 	uint16_t max_base_header_length;
 	uint8_t  max_sample_base_offset;
@@ -736,7 +740,8 @@ int mlx5_devx_cmd_modify_tir(struct mlx5_devx_obj *tir,
 			     struct mlx5_devx_modify_tir_attr *tir_attr);
 __rte_internal
 int mlx5_devx_cmd_query_parse_samples(struct mlx5_devx_obj *flex_obj,
-				      uint32_t ids[], uint32_t num);
+				      struct mlx5_ext_sample_id ids[],
+				      uint32_t num, uint8_t *anchor);
 
 __rte_internal
 struct mlx5_devx_obj *
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 5b84657e08..97bc1eac21 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1899,7 +1899,11 @@ struct mlx5_ifc_parse_graph_node_cap_bits {
 	u8 max_num_arc_in[0x08];
 	u8 max_num_arc_out[0x08];
 	u8 max_num_sample[0x08];
-	u8 reserved_at_78[0x07];
+	u8 reserved_at_78[0x3];
+	u8 anchor_en[0x1];
+	u8 ext_sample_id[0x1];
+	u8 sample_tunnel_inner2[0x1];
+	u8 zero_size_supported[0x1];
 	u8 sample_id_in_out[0x1];
 	u8 max_base_header_length[0x10];
 	u8 reserved_at_90[0x08];
@@ -1909,6 +1913,18 @@ struct mlx5_ifc_parse_graph_node_cap_bits {
 	u8 header_length_mask_width[0x08];
 };
 
+/* ext_sample_id structure, see PRM Table 539. */
+struct mlx5_ext_sample_id {
+	union {
+			struct {
+				uint32_t format_select_dw:8;
+				uint32_t modify_field_id:12;
+				uint32_t sample_id:12;
+			};
+			uint32_t id;
+	};
+};
+
 struct mlx5_ifc_flow_table_prop_layout_bits {
 	u8 ft_support[0x1];
 	u8 flow_tag[0x1];
@@ -4574,7 +4590,9 @@ struct mlx5_ifc_parse_graph_flex_bits {
 	u8 header_length_mode[0x4];
 	u8 header_length_field_offset[0x10];
 	u8 next_header_field_offset[0x10];
-	u8 reserved_at_160[0x1b];
+	u8 reserved_at_160[0x12];
+	u8 head_anchor_id[0x6];
+	u8 reserved_at_178[0x3];
 	u8 next_header_field_size[0x5];
 	u8 header_length_field_mask[0x20];
 	u8 reserved_at_224[0x20];
diff --git a/drivers/net/mlx5/hws/mlx5dr_definer.c b/drivers/net/mlx5/hws/mlx5dr_definer.c
index ced1b61b72..e9f3bbf55b 100644
--- a/drivers/net/mlx5/hws/mlx5dr_definer.c
+++ b/drivers/net/mlx5/hws/mlx5dr_definer.c
@@ -299,6 +299,57 @@ mlx5dr_definer_integrity_set(struct mlx5dr_definer_fc *fc,
 	DR_SET(tag, ok1_bits, fc->byte_off, fc->bit_off, fc->bit_mask);
 }
 
+static uint32_t
+mlx5dr_definer_flex_parser_common(const struct mlx5dr_definer_fc *fc,
+				  const struct rte_flow_item_flex *flex,
+				  bool is_mask)
+{
+	struct mlx5_flex_item *tp = (struct mlx5_flex_item *)flex->handle;
+	struct mlx5_flex_pattern_field *map;
+	uint32_t i, val, pos, def;
+	int id;
+
+	tp = (struct mlx5_flex_item *)flex->handle;
+	for (i = 0, pos = 0, val = 0; i < tp->mapnum && pos < flex->length * CHAR_BIT; i++) {
+		map = tp->map + i;
+		id = mlx5_flex_get_sample_id(tp, i, &pos, fc->bit_off, &def);
+		if (id == -1)
+			continue;
+		MLX5_ASSERT(id < (int)tp->devx_fp->num_samples);
+		if (id >= (int)tp->devx_fp->num_samples || id >= MLX5_GRAPH_NODE_SAMPLE_NUM)
+			return -1;
+		if (tp->devx_fp->sample_ids[id].format_select_dw * 4 == fc->byte_off) {
+			val |= RTE_BE32(mlx5_flex_get_bitfield(flex, pos, map->width, map->shift)) &
+			       (is_mask ? def : UINT32_MAX);
+		}
+		pos += map->width;
+	}
+	return val;
+}
+
+static void
+mlx5dr_definer_flex_parser_set(struct mlx5dr_definer_fc *fc,
+			       const void *item,
+			       uint8_t *tag)
+{
+	uint32_t val;
+
+	val = mlx5dr_definer_flex_parser_common(fc, item, false);
+	DR_SET_BE32(tag, (val & fc->bit_mask), fc->byte_off, 0, fc->bit_mask);
+}
+
+static void
+mlx5dr_definer_flex_parser_mask_set(struct mlx5dr_definer_fc *fc,
+				    const void *item,
+				    uint8_t *tag)
+{
+	uint32_t mask;
+
+	mask = mlx5dr_definer_flex_parser_common(fc, item, true);
+	DR_SET_BE32(tag, (mask), fc->byte_off, 0, UINT32_MAX);
+	fc->bit_mask = mask;
+}
+
 static void
 mlx5dr_definer_gre_key_set(struct mlx5dr_definer_fc *fc,
 			   const void *item_spec,
@@ -1674,6 +1725,91 @@ mlx5dr_definer_conv_item_ipsec_syndrome(struct mlx5dr_definer_conv_data *cd,
 	return 0;
 }
 
+static struct mlx5dr_definer_fc *
+mlx5dr_definer_get_flex_parser_fc(struct mlx5dr_definer_conv_data *cd,
+				  struct mlx5_ext_sample_id reg, int item_idx)
+{
+	enum mlx5dr_definer_fname i = MLX5DR_DEFINER_FNAME_FLEX_PARSER_0;
+	struct mlx5dr_definer_fc *fc;
+
+	for (; i <= MLX5DR_DEFINER_FNAME_FLEX_PARSER_7; i++) {
+		fc = &cd->fc[i];
+		switch (i) {
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_0:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_0);
+			break;
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_1:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_1);
+			break;
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_2:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_2);
+			break;
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_3:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_3);
+			break;
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_4:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_4);
+			break;
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_5:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_5);
+			break;
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_6:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_6);
+			break;
+		case MLX5DR_DEFINER_FNAME_FLEX_PARSER_7:
+		default:
+			DR_CALC_SET_HDR(fc, flex_parser, flex_parser_7);
+			break;
+		}
+		if (fc->byte_off == reg.format_select_dw * 4)
+			break;
+	}
+	if (i > MLX5DR_DEFINER_FNAME_FLEX_PARSER_7) {
+		rte_errno = ENOTSUP;
+		return NULL;
+	}
+	fc->item_idx = item_idx;
+	fc->tag_set = &mlx5dr_definer_flex_parser_set;
+	fc->fname = i;
+	fc->tag_mask_set = &mlx5dr_definer_flex_parser_mask_set;
+	fc->bit_off = cd->tunnel;
+	return fc;
+}
+
+static int
+mlx5dr_definer_conv_item_flex(struct mlx5dr_definer_conv_data *cd,
+			      struct rte_flow_item *item,
+			      int item_idx)
+{
+	const struct rte_flow_item_flex *v, *m;
+	struct mlx5_flex_pattern_field *map;
+	bool is_inner = cd->tunnel;
+	struct mlx5_flex_item *tp;
+	uint32_t i, mask, def;
+	uint32_t pos;
+	int id;
+
+	MLX5_ASSERT(item->spec && item->mask);
+	m = item->mask;
+	v = item->spec;
+	tp = (struct mlx5_flex_item *)v->handle;
+	for (i = 0, pos = 0; i < tp->mapnum && pos < m->length * CHAR_BIT; i++) {
+		map = tp->map + i;
+		id = mlx5_flex_get_sample_id(tp, i, &pos, is_inner, &def);
+		if (id == -1)
+			continue;
+		MLX5_ASSERT(id < (int)tp->devx_fp->num_samples);
+		if (id >= (int)tp->devx_fp->num_samples || id >= MLX5_GRAPH_NODE_SAMPLE_NUM)
+			return -1;
+		mask = mlx5_flex_get_bitfield(m, pos, map->width, map->shift);
+		if (def & RTE_BE32(mask) &&
+		    !mlx5dr_definer_get_flex_parser_fc(cd, tp->devx_fp->sample_ids[id], item_idx))
+			return rte_errno;
+		pos += map->width;
+	}
+	return 0;
+}
+
 static int
 mlx5dr_definer_conv_items_to_hl(struct mlx5dr_context *ctx,
 				struct mlx5dr_match_template *mt,
@@ -1807,6 +1943,11 @@ mlx5dr_definer_conv_items_to_hl(struct mlx5dr_context *ctx,
 			ret = mlx5dr_definer_conv_item_ipsec_syndrome(&cd, items, i);
 			item_flags |= MLX5_FLOW_ITEM_IPSEC_SYNDROME;
 			break;
+		case RTE_FLOW_ITEM_TYPE_FLEX:
+			ret = mlx5dr_definer_conv_item_flex(&cd, items, i);
+			item_flags |= cd.tunnel ? MLX5_FLOW_ITEM_INNER_FLEX :
+				      MLX5_FLOW_ITEM_OUTER_FLEX;
+			break;
 		default:
 			DR_LOG(ERR, "Unsupported item type %d", items->type);
 			rte_errno = ENOTSUP;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index fe9897f83d..0791a6a155 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -974,11 +974,13 @@ int
 mlx5_flex_parser_ecpri_alloc(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_hca_flex_attr *attr = &priv->sh->cdev->config.hca_attr.flex;
 	struct mlx5_ecpri_parser_profile *prf =	&priv->sh->ecpri_parser;
 	struct mlx5_devx_graph_node_attr node = {
 		.modify_field_select = 0,
 	};
-	uint32_t ids[8];
+	struct mlx5_ext_sample_id ids[8];
+	uint8_t anchor_id;
 	int ret;
 
 	if (!priv->sh->cdev->config.hca_attr.parse_graph_flex_node) {
@@ -1014,15 +1016,20 @@ mlx5_flex_parser_ecpri_alloc(struct rte_eth_dev *dev)
 		return (rte_errno == 0) ? -ENODEV : -rte_errno;
 	}
 	prf->num = 2;
-	ret = mlx5_devx_cmd_query_parse_samples(prf->obj, ids, prf->num);
+	ret = mlx5_devx_cmd_query_parse_samples(prf->obj, ids, prf->num, &anchor_id);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to query sample IDs.");
 		return (rte_errno == 0) ? -ENODEV : -rte_errno;
 	}
 	prf->offset[0] = 0x0;
 	prf->offset[1] = sizeof(uint32_t);
-	prf->ids[0] = ids[0];
-	prf->ids[1] = ids[1];
+	if (attr->ext_sample_id) {
+		prf->ids[0] = ids[0].sample_id;
+		prf->ids[1] = ids[1].sample_id;
+	} else {
+		prf->ids[0] = ids[0].id;
+		prf->ids[1] = ids[1].id;
+	}
 	return 0;
 }
 
@@ -1037,7 +1044,7 @@ static void
 mlx5_flex_parser_ecpri_release(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_ecpri_parser_profile *prf =	&priv->sh->ecpri_parser;
+	struct mlx5_ecpri_parser_profile *prf = &priv->sh->ecpri_parser;
 
 	if (prf->obj)
 		mlx5_devx_cmd_destroy(prf->obj);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 761b5ac572..86a4c0a457 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1327,9 +1327,10 @@ struct mlx5_lag {
 struct mlx5_flex_parser_devx {
 	struct mlx5_list_entry entry;  /* List element at the beginning. */
 	uint32_t num_samples;
+	uint8_t anchor_id;
 	void *devx_obj;
 	struct mlx5_devx_graph_node_attr devx_conf;
-	uint32_t sample_ids[MLX5_GRAPH_NODE_SAMPLE_NUM];
+	struct mlx5_ext_sample_id sample_ids[MLX5_GRAPH_NODE_SAMPLE_NUM];
 };
 
 /* Pattern field descriptor - how to translate flex pattern into samples. */
@@ -2346,6 +2347,11 @@ void mlx5_flex_item_port_cleanup(struct rte_eth_dev *dev);
 void mlx5_flex_flow_translate_item(struct rte_eth_dev *dev, void *matcher,
 				   void *key, const struct rte_flow_item *item,
 				   bool is_inner);
+int mlx5_flex_get_sample_id(const struct mlx5_flex_item *tp,
+			    uint32_t idx, uint32_t *pos,
+			    bool is_inner, uint32_t *def);
+uint32_t mlx5_flex_get_bitfield(const struct rte_flow_item_flex *item,
+				uint32_t pos, uint32_t width, uint32_t shift);
 int mlx5_flex_acquire_index(struct rte_eth_dev *dev,
 			    struct rte_flow_item_flex_handle *handle,
 			    bool acquire);
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 7148c10e96..82b5a4a81f 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1280,6 +1280,7 @@ struct rte_flow_pattern_template {
 	 * tag pattern item for representor matching.
 	 */
 	bool implicit_tag;
+	uint8_t flex_item; /* flex item index. */
 };
 
 /* Flow action template struct. */
diff --git a/drivers/net/mlx5/mlx5_flow_flex.c b/drivers/net/mlx5/mlx5_flow_flex.c
index bec07b13c1..affec62384 100644
--- a/drivers/net/mlx5/mlx5_flow_flex.c
+++ b/drivers/net/mlx5/mlx5_flow_flex.c
@@ -113,7 +113,7 @@ mlx5_flex_free(struct mlx5_priv *priv, struct mlx5_flex_item *item)
 	}
 }
 
-static uint32_t
+uint32_t
 mlx5_flex_get_bitfield(const struct rte_flow_item_flex *item,
 		       uint32_t pos, uint32_t width, uint32_t shift)
 {
@@ -198,6 +198,50 @@ mlx5_flex_set_match_sample(void *misc4_m, void *misc4_v,
 	}
 #undef SET_FP_MATCH_SAMPLE_ID
 }
+
+/**
+ * Get the flex parser sample id and corresponding mask
+ * per shift and width information.
+ *
+ * @param[in] tp
+ *   Mlx5 flex item sample mapping handle.
+ * @param[in] idx
+ *   Mapping index.
+ * @param[in, out] pos
+ *   Where to search the value and mask.
+ * @param[in] is_inner
+ *   For inner matching or not.
+ * @param[in, def] def
+ *   Mask generated by mapping shift and width.
+ *
+ * @return
+ *   0 on success, -1 to ignore.
+ */
+int
+mlx5_flex_get_sample_id(const struct mlx5_flex_item *tp,
+			uint32_t idx, uint32_t *pos,
+			bool is_inner, uint32_t *def)
+{
+	const struct mlx5_flex_pattern_field *map = tp->map + idx;
+	uint32_t id = map->reg_id;
+
+	*def = (RTE_BIT64(map->width) - 1) << map->shift;
+	/* Skip placeholders for DUMMY fields. */
+	if (id == MLX5_INVALID_SAMPLE_REG_ID) {
+		*pos += map->width;
+		return -1;
+	}
+	MLX5_ASSERT(map->width);
+	MLX5_ASSERT(id < tp->devx_fp->num_samples);
+	if (tp->tunnel_mode == FLEX_TUNNEL_MODE_MULTI && is_inner) {
+		uint32_t num_samples = tp->devx_fp->num_samples / 2;
+
+		MLX5_ASSERT(tp->devx_fp->num_samples % 2 == 0);
+		MLX5_ASSERT(id < num_samples);
+		id += num_samples;
+	}
+	return id;
+}
 /**
  * Translate item pattern into matcher fields according to translation
  * array.
@@ -226,40 +270,38 @@ mlx5_flex_flow_translate_item(struct rte_eth_dev *dev,
 	void *misc4_m = MLX5_ADDR_OF(fte_match_param, matcher,
 				     misc_parameters_4);
 	void *misc4_v = MLX5_ADDR_OF(fte_match_param, key, misc_parameters_4);
+	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_hca_flex_attr *attr = &priv->sh->cdev->config.hca_attr.flex;
 	struct mlx5_flex_item *tp;
 	uint32_t i, pos = 0;
+	uint32_t sample_id;
 
 	RTE_SET_USED(dev);
 	MLX5_ASSERT(item->spec && item->mask);
 	spec = item->spec;
 	mask = item->mask;
 	tp = (struct mlx5_flex_item *)spec->handle;
-	MLX5_ASSERT(mlx5_flex_index(dev->data->dev_private, tp) >= 0);
+	MLX5_ASSERT(mlx5_flex_index(priv, tp) >= 0);
 	for (i = 0; i < tp->mapnum; i++) {
 		struct mlx5_flex_pattern_field *map = tp->map + i;
-		uint32_t id = map->reg_id;
-		uint32_t def = (RTE_BIT64(map->width) - 1) << map->shift;
-		uint32_t val, msk;
+		uint32_t val, msk, def;
+		int id = mlx5_flex_get_sample_id(tp, i, &pos, is_inner, &def);
 
-		/* Skip placeholders for DUMMY fields. */
-		if (id == MLX5_INVALID_SAMPLE_REG_ID) {
-			pos += map->width;
+		if (id == -1)
 			continue;
-		}
+		MLX5_ASSERT(id < (int)tp->devx_fp->num_samples);
+		if (id >= (int)tp->devx_fp->num_samples ||
+		    id >= MLX5_GRAPH_NODE_SAMPLE_NUM)
+			return;
 		val = mlx5_flex_get_bitfield(spec, pos, map->width, map->shift);
 		msk = mlx5_flex_get_bitfield(mask, pos, map->width, map->shift);
-		MLX5_ASSERT(map->width);
-		MLX5_ASSERT(id < tp->devx_fp->num_samples);
-		if (tp->tunnel_mode == FLEX_TUNNEL_MODE_MULTI && is_inner) {
-			uint32_t num_samples = tp->devx_fp->num_samples / 2;
-
-			MLX5_ASSERT(tp->devx_fp->num_samples % 2 == 0);
-			MLX5_ASSERT(id < num_samples);
-			id += num_samples;
-		}
+		if (attr->ext_sample_id)
+			sample_id = tp->devx_fp->sample_ids[id].sample_id;
+		else
+			sample_id = tp->devx_fp->sample_ids[id].id;
 		mlx5_flex_set_match_sample(misc4_m, misc4_v,
 					   def, msk & def, val & msk & def,
-					   tp->devx_fp->sample_ids[id], id);
+					   sample_id, id);
 		pos += map->width;
 	}
 }
@@ -1317,7 +1359,8 @@ mlx5_flex_parser_create_cb(void *list_ctx, void *ctx)
 	/* Query the firmware assigned sample ids. */
 	ret = mlx5_devx_cmd_query_parse_samples(fp->devx_obj,
 						fp->sample_ids,
-						fp->num_samples);
+						fp->num_samples,
+						&fp->anchor_id);
 	if (ret)
 		goto error;
 	DRV_LOG(DEBUG, "DEVx flex parser %p created, samples num: %u",
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index b9c7459646..56f1e699fa 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -4649,6 +4649,36 @@ flow_hw_actions_template_replace_container(const
 	}                                                             \
 })
 
+static int
+flow_hw_flex_item_acquire(struct rte_eth_dev *dev,
+			  struct rte_flow_item_flex_handle *handle,
+			  uint8_t *flex_item)
+{
+	int index = mlx5_flex_acquire_index(dev, handle, false);
+
+	MLX5_ASSERT(index >= 0 && index <= (int)(sizeof(uint32_t) * CHAR_BIT));
+	if (index < 0)
+		return -1;
+	if (!(*flex_item & RTE_BIT32(index))) {
+		/* Don't count same flex item again. */
+		if (mlx5_flex_acquire_index(dev, handle, true) != index)
+			MLX5_ASSERT(false);
+		*flex_item |= (uint8_t)RTE_BIT32(index);
+	}
+	return 0;
+}
+
+static void
+flow_hw_flex_item_release(struct rte_eth_dev *dev, uint8_t *flex_item)
+{
+	while (*flex_item) {
+		int index = rte_bsf32(*flex_item);
+
+		mlx5_flex_release_index(dev, index);
+		*flex_item &= ~(uint8_t)RTE_BIT32(index);
+	}
+}
+
 /**
  * Create flow action template.
  *
@@ -5037,6 +5067,7 @@ flow_hw_pattern_validate(struct rte_eth_dev *dev,
 		case RTE_FLOW_ITEM_TYPE_QUOTA:
 		case RTE_FLOW_ITEM_TYPE_ESP:
 		case MLX5_FLOW_ITEM_TYPE_IPSEC_SYNDROME:
+		case RTE_FLOW_ITEM_TYPE_FLEX:
 			break;
 		case RTE_FLOW_ITEM_TYPE_INTEGRITY:
 			/*
@@ -5114,6 +5145,7 @@ flow_hw_pattern_template_create(struct rte_eth_dev *dev,
 		.mask = &tag_m,
 		.last = NULL
 	};
+	unsigned int i = 0;
 
 	if (flow_hw_pattern_validate(dev, attr, items, error))
 		return NULL;
@@ -5173,6 +5205,19 @@ flow_hw_pattern_template_create(struct rte_eth_dev *dev,
 			it->implicit_tag = true;
 		mlx5_free(copied_items);
 	}
+	for (i = 0; items[i].type != RTE_FLOW_ITEM_TYPE_END; ++i) {
+		if (items[i].type == RTE_FLOW_ITEM_TYPE_FLEX) {
+			const struct rte_flow_item_flex *spec =
+				(const struct rte_flow_item_flex *)items[i].spec;
+			struct rte_flow_item_flex_handle *handle = spec->handle;
+
+			if (flow_hw_flex_item_acquire(dev, handle, &it->flex_item)) {
+				claim_zero(mlx5dr_match_template_destroy(it->mt));
+				mlx5_free(it);
+				return NULL;
+			}
+		}
+	}
 	__atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
 	LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
 	return it;
@@ -5192,7 +5237,7 @@ flow_hw_pattern_template_create(struct rte_eth_dev *dev,
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
 static int
-flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
+flow_hw_pattern_template_destroy(struct rte_eth_dev *dev,
 			      struct rte_flow_pattern_template *template,
 			      struct rte_flow_error *error __rte_unused)
 {
@@ -5205,6 +5250,7 @@ flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
 				   "item template in using");
 	}
 	LIST_REMOVE(template, next);
+	flow_hw_flex_item_release(dev, &template->flex_item);
 	claim_zero(mlx5dr_match_template_destroy(template->mt));
 	mlx5_free(template);
 	return 0;
-- 
2.27.0

[RFC 7/9] net/mlx5/hws: add flex item modify field implementation

[Rongwei Liu]

Add flex item modify field HWS implementation.
The minimum modify boundary is one byte.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 doc/guides/nics/mlx5.rst        |   1 +
 drivers/common/mlx5/mlx5_prm.h  |   1 +
 drivers/net/mlx5/mlx5_flow.h    |   3 +
 drivers/net/mlx5/mlx5_flow_dv.c | 164 +++++++++++++++++++++++++++++---
 drivers/net/mlx5/mlx5_flow_hw.c |  14 ++-
 5 files changed, 170 insertions(+), 13 deletions(-)

diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index e442f9c015..9f95ff9112 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -107,6 +107,7 @@ Features
 - Sub-Function.
 - Matching on represented port.
 - Matching on flex item with specific pattern.
+- Modify flex item field.
 
 
 Limitations
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 97bc1eac21..6c730ff85f 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -760,6 +760,7 @@ enum mlx5_modification_field {
 	MLX5_MODI_TUNNEL_HDR_DW_1 = 0x75,
 	MLX5_MODI_GTPU_FIRST_EXT_DW_0 = 0x76,
 	MLX5_MODI_HASH_RESULT = 0x81,
+	MLX5_MODI_INVALID = INT_MAX,
 };
 
 /* Total number of metadata reg_c's. */
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 82b5a4a81f..531f34c281 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1134,6 +1134,8 @@ struct field_modify_info {
 	uint32_t size; /* Size of field in protocol header, in bytes. */
 	uint32_t offset; /* Offset of field in protocol header, in bytes. */
 	enum mlx5_modification_field id;
+	uint32_t shift;
+	uint8_t is_flex; /* Temporary indicator for flex item modify filed WA. */
 };
 
 /* HW steering flow attributes. */
@@ -1299,6 +1301,7 @@ struct rte_flow_actions_template {
 	uint16_t mhdr_off; /* Offset of DR modify header action. */
 	uint32_t refcnt; /* Reference counter. */
 	uint16_t rx_cpy_pos; /* Action position of Rx metadata to be copied. */
+	uint8_t flex_item; /* flex item index. */
 };
 
 /* Jump action struct. */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 310fb7c5c3..501a190c5e 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -424,10 +424,15 @@ flow_dv_convert_modify_action(struct rte_flow_item *item,
 			++field;
 			continue;
 		}
-		/* Deduce actual data width in bits from mask value. */
-		off_b = rte_bsf32(mask) + carry_b;
-		size_b = sizeof(uint32_t) * CHAR_BIT -
-			 off_b - __builtin_clz(mask);
+		if (type == MLX5_MODIFICATION_TYPE_COPY && field->is_flex) {
+			off_b = 32 - field->shift + carry_b - field->size * CHAR_BIT;
+			size_b = field->size * CHAR_BIT - carry_b;
+		} else {
+			/* Deduce actual data width in bits from mask value. */
+			off_b = rte_bsf32(mask) + carry_b;
+			size_b = sizeof(uint32_t) * CHAR_BIT -
+				 off_b - __builtin_clz(mask);
+		}
 		MLX5_ASSERT(size_b);
 		actions[i] = (struct mlx5_modification_cmd) {
 			.action_type = type,
@@ -447,40 +452,46 @@ flow_dv_convert_modify_action(struct rte_flow_item *item,
 			 * Destination field overflow. Copy leftovers of
 			 * a source field to the next destination field.
 			 */
-			carry_b = 0;
 			if ((size_b > dcopy->size * CHAR_BIT - dcopy->offset) &&
 			    dcopy->size != 0) {
 				actions[i].length =
 					dcopy->size * CHAR_BIT - dcopy->offset;
-				carry_b = actions[i].length;
+				carry_b += actions[i].length;
 				next_field = false;
+			} else {
+				carry_b = 0;
 			}
 			/*
 			 * Not enough bits in a source filed to fill a
 			 * destination field. Switch to the next source.
 			 */
 			if ((size_b < dcopy->size * CHAR_BIT - dcopy->offset) &&
-			    (size_b == field->size * CHAR_BIT - off_b)) {
-				actions[i].length =
-					field->size * CHAR_BIT - off_b;
+			    ((size_b == field->size * CHAR_BIT - off_b) ||
+			     field->is_flex)) {
+				actions[i].length = size_b;
 				dcopy->offset += actions[i].length;
 				next_dcopy = false;
 			}
-			if (next_dcopy)
-				++dcopy;
 		} else {
 			MLX5_ASSERT(item->spec);
 			data = flow_dv_fetch_field((const uint8_t *)item->spec +
 						   field->offset, field->size);
 			/* Shift out the trailing masked bits from data. */
 			data = (data & mask) >> off_b;
+			if (field->is_flex)
+				actions[i].offset = 32 - field->shift - field->size * CHAR_BIT;
 			actions[i].data1 = rte_cpu_to_be_32(data);
 		}
 		/* Convert entire record to expected big-endian format. */
 		actions[i].data0 = rte_cpu_to_be_32(actions[i].data0);
+		if ((type != MLX5_MODIFICATION_TYPE_COPY ||
+		     dcopy->id != (enum mlx5_modification_field)UINT32_MAX) &&
+		    field->id != (enum mlx5_modification_field)UINT32_MAX)
+			++i;
+		if (next_dcopy && type == MLX5_MODIFICATION_TYPE_COPY)
+			++dcopy;
 		if (next_field)
 			++field;
-		++i;
 	} while (field->size);
 	if (resource->actions_num == i)
 		return rte_flow_error_set(error, EINVAL,
@@ -1476,6 +1487,130 @@ flow_modify_info_mask_32_masked(uint32_t length, uint32_t off, uint32_t post_mas
 	return rte_cpu_to_be_32(mask & post_mask);
 }
 
+static void
+mlx5_modify_flex_item(const struct rte_eth_dev *dev,
+		      const struct mlx5_flex_item *flex,
+		      const struct rte_flow_action_modify_data *data,
+		      struct field_modify_info *info,
+		      uint32_t *mask, uint32_t width)
+{
+	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_hca_flex_attr *attr = &priv->sh->cdev->config.hca_attr.flex;
+	uint32_t i, j;
+	int id = 0;
+	uint32_t pos = 0;
+	const struct mlx5_flex_pattern_field *map;
+	uint32_t offset = data->offset;
+	uint32_t width_left = width;
+	uint32_t def;
+	uint32_t cur_width = 0;
+	uint32_t tmp_ofs;
+	uint32_t idx = 0;
+	struct field_modify_info tmp;
+	int tmp_id;
+
+	if (!attr->ext_sample_id) {
+		DRV_LOG(ERR, "FW doesn't support modify field with flex item.");
+		return;
+	}
+	/*
+	 * search for the mapping instance until Accumulated width is no
+	 * less than data->offset.
+	 */
+	for (i = 0; i < flex->mapnum; i++) {
+		if (flex->map[i].width + pos > data->offset)
+			break;
+		pos += flex->map[i].width;
+	}
+	if (i >= flex->mapnum)
+		return;
+	tmp_ofs = pos < data->offset ? data->offset - pos : 0;
+	for (j = i; i < flex->mapnum && width_left > 0; ) {
+		map = flex->map + i;
+		id = mlx5_flex_get_sample_id(flex, i, &pos, false, &def);
+		if (id == -1) {
+			i++;
+			/* All left length is dummy */
+			if (pos >= data->offset + width)
+				return;
+			cur_width = map->width;
+		/* One mapping instance covers the whole width. */
+		} else if (pos + map->width >= (data->offset + width)) {
+			cur_width = width_left;
+		} else {
+			cur_width = cur_width + map->width - tmp_ofs;
+			pos += map->width;
+			/*
+			 * Continue to search next until:
+			 * 1. Another flex parser ID.
+			 * 2. Width has been covered.
+			 */
+			for (j = i + 1; j < flex->mapnum; j++) {
+				tmp_id = mlx5_flex_get_sample_id(flex, j, &pos, false, &def);
+				if (tmp_id == -1) {
+					i = j;
+					pos -= flex->map[j].width;
+					break;
+				}
+				if (id >= (int)flex->devx_fp->num_samples ||
+				    id >= MLX5_GRAPH_NODE_SAMPLE_NUM ||
+				    tmp_id >= (int)flex->devx_fp->num_samples ||
+				    tmp_id >= MLX5_GRAPH_NODE_SAMPLE_NUM)
+					return;
+				if (flex->devx_fp->sample_ids[id].id !=
+						flex->devx_fp->sample_ids[tmp_id].id ||
+				    flex->map[j].shift != flex->map[j - 1].width +
+							  flex->map[j - 1].shift) {
+					i = j;
+					break;
+				}
+				if ((pos + flex->map[j].width) >= (data->offset + width)) {
+					cur_width = width_left;
+					break;
+				}
+				pos += flex->map[j].width;
+				cur_width += flex->map[j].width;
+			}
+		}
+		if (cur_width > width_left)
+			cur_width = width_left;
+		else if (cur_width < width_left && (j == flex->mapnum || i == flex->mapnum))
+			return;
+
+		MLX5_ASSERT(id < (int)flex->devx_fp->num_samples);
+		if (id >= (int)flex->devx_fp->num_samples || id >= MLX5_GRAPH_NODE_SAMPLE_NUM)
+			return;
+		/* Use invalid entry as placeholder for DUMMY mapping. */
+		info[idx] = (struct field_modify_info){cur_width / CHAR_BIT, offset / CHAR_BIT,
+			     id == -1 ? MLX5_MODI_INVALID :
+			     (enum mlx5_modification_field)flex->devx_fp->sample_ids[id].modify_field_id,
+			     map->shift + tmp_ofs, 1};
+		offset += cur_width;
+		width_left -= cur_width;
+		if (!mask) {
+			info[idx].offset = (32 - cur_width - map->shift - tmp_ofs);
+			info[idx].size = cur_width / CHAR_BIT + info[idx].offset / CHAR_BIT;
+		}
+		cur_width = 0;
+		tmp_ofs = 0;
+		idx++;
+	}
+	if (unlikely(width_left > 0)) {
+		MLX5_ASSERT(false);
+		return;
+	}
+	if (mask)
+		memset(mask, 0xff, data->offset / CHAR_BIT + width / CHAR_BIT);
+	/* Re-order the info to follow IPv6 address. */
+	for (i = 0; i < idx / 2; i++) {
+		tmp = info[i];
+		MLX5_ASSERT(info[i].id);
+		MLX5_ASSERT(info[idx - 1 - i].id);
+		info[i] = info[idx - 1 - i];
+		info[idx - 1 - i] = tmp;
+	}
+}
+
 void
 mlx5_flow_field_id_to_modify_info
 		(const struct rte_flow_action_modify_data *data,
@@ -1950,6 +2085,11 @@ mlx5_flow_field_id_to_modify_info
 		else
 			info[idx].offset = off_be;
 		break;
+	case RTE_FLOW_FIELD_FLEX_ITEM:
+		MLX5_ASSERT(data->flex_handle != NULL && !(data->offset & 0x7));
+		mlx5_modify_flex_item(dev, (const struct mlx5_flex_item *)data->flex_handle,
+				      data, info, mask, width);
+		break;
 	case RTE_FLOW_FIELD_POINTER:
 	case RTE_FLOW_FIELD_VALUE:
 	default:
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 56f1e699fa..042f8cdc7f 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -4850,6 +4850,17 @@ flow_hw_actions_template_create(struct rte_eth_dev *dev,
 			at->actions[i].conf = actions->conf;
 			at->masks[i].conf = masks->conf;
 		}
+		if (actions->type == RTE_FLOW_ACTION_TYPE_MODIFY_FIELD) {
+			const struct rte_flow_action_modify_field *info = actions->conf;
+
+			if ((info->dst.field == RTE_FLOW_FIELD_FLEX_ITEM &&
+			     flow_hw_flex_item_acquire(dev, info->dst.flex_handle,
+						       &at->flex_item)) ||
+			     (info->src.field == RTE_FLOW_FIELD_FLEX_ITEM &&
+			      flow_hw_flex_item_acquire(dev, info->src.flex_handle,
+							&at->flex_item)))
+				goto error;
+		}
 	}
 	at->tmpl = flow_hw_dr_actions_template_create(at);
 	if (!at->tmpl)
@@ -4881,7 +4892,7 @@ flow_hw_actions_template_create(struct rte_eth_dev *dev,
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
 static int
-flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
+flow_hw_actions_template_destroy(struct rte_eth_dev *dev,
 				 struct rte_flow_actions_template *template,
 				 struct rte_flow_error *error __rte_unused)
 {
@@ -4894,6 +4905,7 @@ flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
 				   "action template in using");
 	}
 	LIST_REMOVE(template, next);
+	flow_hw_flex_item_release(dev, &template->flex_item);
 	if (template->tmpl)
 		mlx5dr_action_template_destroy(template->tmpl);
 	mlx5_free(template);
-- 
2.27.0

[RFC 8/9] net/mlx5: return error for sws modify field

[Rongwei Liu]

Return unsupported error message when application tries to
modify flex item field.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 drivers/net/mlx5/mlx5_flow_dv.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 501a190c5e..f09648d85f 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -4947,6 +4947,8 @@ flow_dv_validate_action_modify_hdr(const uint64_t action_flags,
 				   const struct rte_flow_action *action,
 				   struct rte_flow_error *error)
 {
+	const struct rte_flow_action_modify_field *action_modify_field = action->conf;
+
 	if (action->type != RTE_FLOW_ACTION_TYPE_DEC_TTL &&
 	    action->type != RTE_FLOW_ACTION_TYPE_DEC_IPV4_TTL &&
 	    action->type != RTE_FLOW_ACTION_TYPE_DEC_IPV6_HOP &&
@@ -4954,6 +4956,12 @@ flow_dv_validate_action_modify_hdr(const uint64_t action_flags,
 		return rte_flow_error_set(error, EINVAL,
 					  RTE_FLOW_ERROR_TYPE_ACTION_CONF,
 					  NULL, "action configuration not set");
+	if (action_modify_field->src.field == RTE_FLOW_FIELD_FLEX_ITEM ||
+	    action_modify_field->dst.field == RTE_FLOW_FIELD_FLEX_ITEM)
+		return rte_flow_error_set(error, ENOTSUP,
+				RTE_FLOW_ERROR_TYPE_ACTION, action,
+				"flex item fields modification"
+				" is not supported");
 	if (action_flags & MLX5_FLOW_ACTION_ENCAP)
 		return rte_flow_error_set(error, EINVAL,
 					  RTE_FLOW_ERROR_TYPE_ACTION, NULL,
@@ -5371,17 +5379,16 @@ flow_dv_validate_action_modify_field(struct rte_eth_dev *dev,
 	struct mlx5_hca_attr *hca_attr = &priv->sh->cdev->config.hca_attr;
 	const struct rte_flow_action_modify_field *action_modify_field =
 		action->conf;
-	uint32_t dst_width = mlx5_flow_item_field_width(dev,
-				action_modify_field->dst.field,
-				-1, attr, error);
-	uint32_t src_width = mlx5_flow_item_field_width(dev,
-				action_modify_field->src.field,
-				dst_width, attr, error);
+	uint32_t dst_width, src_width;
 
 	ret = flow_dv_validate_action_modify_hdr(action_flags, action, error);
 	if (ret)
 		return ret;
 
+	dst_width = mlx5_flow_item_field_width(dev, action_modify_field->dst.field,
+					       -1, attr, error);
+	src_width = mlx5_flow_item_field_width(dev, action_modify_field->src.field,
+					       dst_width, attr, error);
 	if (action_modify_field->width == 0)
 		return rte_flow_error_set(error, EINVAL,
 				RTE_FLOW_ERROR_TYPE_ACTION, action,
-- 
2.27.0

[RFC 9/9] app/testpmd: raw encap with flex item support

[Rongwei Liu]

Application should retrieve raw_encap buffer from
spec->pattern if it is flex item.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index 1158da2122..b5f13f797a 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -12540,6 +12540,7 @@ cmd_set_raw_parsed(const struct buffer *in)
 	uint16_t proto = 0;
 	uint16_t idx = in->port; /* We borrow port field as index */
 	int gtp_psc = -1; /* GTP PSC option index. */
+	const void *src_spec;
 
 	if (in->command == SET_SAMPLE_ACTIONS)
 		return cmd_set_raw_parsed_sample(in);
@@ -12563,6 +12564,7 @@ cmd_set_raw_parsed(const struct buffer *in)
 		item = in->args.vc.pattern + i;
 		if (item->spec == NULL)
 			item->spec = flow_item_default_mask(item);
+		src_spec = item->spec;
 		switch (item->type) {
 		case RTE_FLOW_ITEM_TYPE_ETH:
 			size = sizeof(struct rte_ether_hdr);
@@ -12690,9 +12692,13 @@ cmd_set_raw_parsed(const struct buffer *in)
 			size = sizeof(struct rte_flow_item_pfcp);
 			break;
 		case RTE_FLOW_ITEM_TYPE_FLEX:
-			size = item->spec ?
-				((const struct rte_flow_item_flex *)
-				item->spec)->length : 0;
+			if (item->spec != NULL) {
+				size = ((const struct rte_flow_item_flex *)item->spec)->length;
+				src_spec = ((const struct rte_flow_item_flex *)item->spec)->pattern;
+			} else {
+				size = 0;
+				src_spec = NULL;
+			}
 			break;
 		case RTE_FLOW_ITEM_TYPE_GRE_OPTION:
 			size = 0;
@@ -12725,12 +12731,14 @@ cmd_set_raw_parsed(const struct buffer *in)
 			fprintf(stderr, "Error - Not supported item\n");
 			goto error;
 		}
-		*total_size += size;
-		rte_memcpy(data_tail - (*total_size), item->spec, size);
-		/* update some fields which cannot be set by cmdline */
-		update_fields((data_tail - (*total_size)), item,
-			      upper_layer);
-		upper_layer = proto;
+		if (size) {
+			*total_size += size;
+			rte_memcpy(data_tail - (*total_size), src_spec, size);
+			/* update some fields which cannot be set by cmdline */
+			update_fields((data_tail - (*total_size)), item,
+				      upper_layer);
+			upper_layer = proto;
+		}
 	}
 	if (verbose_level & 0x1)
 		printf("total data size is %zu\n", (*total_size));
-- 
2.27.0

[RFC 7/9] net/mlx5/hws: add flex item modify field implementation

[Rongwei Liu]

Add flex item modify field HWS implementation.
The minimum modify boundary is one byte.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 doc/guides/nics/mlx5.rst        |   1 +
 drivers/common/mlx5/mlx5_prm.h  |   1 +
 drivers/net/mlx5/mlx5_flow.h    |   3 +
 drivers/net/mlx5/mlx5_flow_dv.c | 165 +++++++++++++++++++++++++++++---
 drivers/net/mlx5/mlx5_flow_hw.c |  14 ++-
 5 files changed, 171 insertions(+), 13 deletions(-)

diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 397e04fbed..26d68d86f4 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -107,6 +107,7 @@ Features
 - Sub-Function.
 - Matching on represented port.
 - Matching on flex item with specific pattern.
+- Modify flex item field.
 
 
 Limitations
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index b080599570..74fc8d9936 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -753,6 +753,7 @@ enum mlx5_modification_field {
 	MLX5_MODI_OUT_IP_ECN = 0x73,
 	MLX5_MODI_TUNNEL_HDR_DW_1 = 0x75,
 	MLX5_MODI_GTPU_FIRST_EXT_DW_0 = 0x76,
+	MLX5_MODI_INVALID = INT_MAX,
 };
 
 /* Total number of metadata reg_c's. */
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index c2f656e702..b54eaf64f4 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1080,6 +1080,8 @@ struct field_modify_info {
 	uint32_t size; /* Size of field in protocol header, in bytes. */
 	uint32_t offset; /* Offset of field in protocol header, in bytes. */
 	enum mlx5_modification_field id;
+	uint32_t shift;
+	uint8_t is_flex; /* Temporary indicator for flex item modify filed WA. */
 };
 
 /* HW steering flow attributes. */
@@ -1244,6 +1246,7 @@ struct rte_flow_actions_template {
 	uint16_t mhdr_off; /* Offset of DR modify header action. */
 	uint32_t refcnt; /* Reference counter. */
 	uint16_t rx_cpy_pos; /* Action position of Rx metadata to be copied. */
+	uint8_t flex_item; /* flex item index. */
 };
 
 /* Jump action struct. */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 7ca909999b..110d08b271 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -414,10 +414,15 @@ flow_dv_convert_modify_action(struct rte_flow_item *item,
 			++field;
 			continue;
 		}
-		/* Deduce actual data width in bits from mask value. */
-		off_b = rte_bsf32(mask) + carry_b;
-		size_b = sizeof(uint32_t) * CHAR_BIT -
-			 off_b - __builtin_clz(mask);
+		if (type == MLX5_MODIFICATION_TYPE_COPY && field->is_flex) {
+			off_b = 32 - field->shift + carry_b - field->size * CHAR_BIT;
+			size_b = field->size * CHAR_BIT - carry_b;
+		} else {
+			/* Deduce actual data width in bits from mask value. */
+			off_b = rte_bsf32(mask) + carry_b;
+			size_b = sizeof(uint32_t) * CHAR_BIT -
+				 off_b - __builtin_clz(mask);
+		}
 		MLX5_ASSERT(size_b);
 		actions[i] = (struct mlx5_modification_cmd) {
 			.action_type = type,
@@ -437,40 +442,46 @@ flow_dv_convert_modify_action(struct rte_flow_item *item,
 			 * Destination field overflow. Copy leftovers of
 			 * a source field to the next destination field.
 			 */
-			carry_b = 0;
 			if ((size_b > dcopy->size * CHAR_BIT - dcopy->offset) &&
 			    dcopy->size != 0) {
 				actions[i].length =
 					dcopy->size * CHAR_BIT - dcopy->offset;
-				carry_b = actions[i].length;
+				carry_b += actions[i].length;
 				next_field = false;
+			} else {
+				carry_b = 0;
 			}
 			/*
 			 * Not enough bits in a source filed to fill a
 			 * destination field. Switch to the next source.
 			 */
 			if ((size_b < dcopy->size * CHAR_BIT - dcopy->offset) &&
-			    (size_b == field->size * CHAR_BIT - off_b)) {
-				actions[i].length =
-					field->size * CHAR_BIT - off_b;
+			    ((size_b == field->size * CHAR_BIT - off_b) ||
+			     field->is_flex)) {
+				actions[i].length = size_b;
 				dcopy->offset += actions[i].length;
 				next_dcopy = false;
 			}
-			if (next_dcopy)
-				++dcopy;
 		} else {
 			MLX5_ASSERT(item->spec);
 			data = flow_dv_fetch_field((const uint8_t *)item->spec +
 						   field->offset, field->size);
 			/* Shift out the trailing masked bits from data. */
 			data = (data & mask) >> off_b;
+			if (field->is_flex)
+				actions[i].offset = 32 - field->shift - field->size * CHAR_BIT;
 			actions[i].data1 = rte_cpu_to_be_32(data);
 		}
 		/* Convert entire record to expected big-endian format. */
 		actions[i].data0 = rte_cpu_to_be_32(actions[i].data0);
+		if ((type != MLX5_MODIFICATION_TYPE_COPY ||
+		     dcopy->id != (enum mlx5_modification_field)UINT32_MAX) &&
+		    field->id != (enum mlx5_modification_field)UINT32_MAX)
+			++i;
+		if (next_dcopy && type == MLX5_MODIFICATION_TYPE_COPY)
+			++dcopy;
 		if (next_field)
 			++field;
-		++i;
 	} while (field->size);
 	if (resource->actions_num == i)
 		return rte_flow_error_set(error, EINVAL,
@@ -1421,6 +1432,131 @@ flow_modify_info_mask_32_masked(uint32_t length, uint32_t off, uint32_t post_mas
 	return rte_cpu_to_be_32(mask & post_mask);
 }
 
+static void
+mlx5_modify_flex_item(const struct rte_eth_dev *dev,
+		      const struct mlx5_flex_item *flex,
+		      const struct rte_flow_action_modify_data *data,
+		      struct field_modify_info *info,
+		      uint32_t *mask, uint32_t width)
+{
+	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_hca_flex_attr *attr = &priv->sh->cdev->config.hca_attr.flex;
+	uint32_t i, j;
+	int id = 0;
+	uint32_t pos = 0;
+	const struct mlx5_flex_pattern_field *map;
+	uint32_t offset = data->offset;
+	uint32_t width_left = width;
+	uint32_t def;
+	uint32_t cur_width = 0;
+	uint32_t tmp_ofs;
+	uint32_t idx = 0;
+	struct field_modify_info tmp;
+	int tmp_id;
+
+	if (!attr->ext_sample_id) {
+		DRV_LOG(ERR, "FW doesn't support modify field with flex item.");
+		return;
+	}
+	/*
+	 * search for the mapping instance until Accumulated width is no
+	 * less than data->offset.
+	 */
+	for (i = 0; i < flex->mapnum; i++) {
+		if (flex->map[i].width + pos > data->offset)
+			break;
+		pos += flex->map[i].width;
+	}
+	if (i >= flex->mapnum)
+		return;
+	tmp_ofs = pos < data->offset ? data->offset - pos : 0;
+	for (j = i; i < flex->mapnum && width_left > 0; ) {
+		map = flex->map + i;
+		id = mlx5_flex_get_sample_id(flex, i, &pos, false, &def);
+		if (id == -1) {
+			i++;
+			/* All left length is dummy */
+			if (pos >= data->offset + width)
+				return;
+			cur_width = map->width;
+		/* One mapping instance covers the whole width. */
+		} else if (pos + map->width >= (data->offset + width)) {
+			cur_width = width_left;
+		} else {
+			cur_width = cur_width + map->width - tmp_ofs;
+			pos += map->width;
+			/*
+			 * Continue to search next until:
+			 * 1. Another flex parser ID.
+			 * 2. Width has been covered.
+			 */
+			for (j = i + 1; j < flex->mapnum; j++) {
+				tmp_id = mlx5_flex_get_sample_id(flex, j, &pos, false, &def);
+				if (tmp_id == -1) {
+					i = j;
+					pos -= flex->map[j].width;
+					break;
+				}
+				if (id >= (int)flex->devx_fp->num_samples ||
+				    id >= MLX5_GRAPH_NODE_SAMPLE_NUM ||
+				    tmp_id >= (int)flex->devx_fp->num_samples ||
+				    tmp_id >= MLX5_GRAPH_NODE_SAMPLE_NUM)
+					return;
+				if (flex->devx_fp->sample_ids[id].id !=
+						flex->devx_fp->sample_ids[tmp_id].id ||
+				    flex->map[j].shift != flex->map[j - 1].width +
+							  flex->map[j - 1].shift) {
+					i = j;
+					break;
+				}
+				if ((pos + flex->map[j].width) >= (data->offset + width)) {
+					cur_width = width_left;
+					break;
+				}
+				pos += flex->map[j].width;
+				cur_width += flex->map[j].width;
+			}
+		}
+		if (cur_width > width_left)
+			cur_width = width_left;
+		else if (cur_width < width_left && (j == flex->mapnum || i == flex->mapnum))
+			return;
+
+		MLX5_ASSERT(id < (int)flex->devx_fp->num_samples);
+		if (id >= (int)flex->devx_fp->num_samples || id >= MLX5_GRAPH_NODE_SAMPLE_NUM)
+			return;
+		/* Use invalid entry as placeholder for DUMMY mapping. */
+		info[idx] = (struct field_modify_info){cur_width / CHAR_BIT, offset / CHAR_BIT,
+			     id == -1 ? MLX5_MODI_INVALID :
+			     (enum mlx5_modification_field)
+			     flex->devx_fp->sample_ids[id].modify_field_id,
+			     map->shift + tmp_ofs, 1};
+		offset += cur_width;
+		width_left -= cur_width;
+		if (!mask) {
+			info[idx].offset = (32 - cur_width - map->shift - tmp_ofs);
+			info[idx].size = cur_width / CHAR_BIT + info[idx].offset / CHAR_BIT;
+		}
+		cur_width = 0;
+		tmp_ofs = 0;
+		idx++;
+	}
+	if (unlikely(width_left > 0)) {
+		MLX5_ASSERT(false);
+		return;
+	}
+	if (mask)
+		memset(mask, 0xff, data->offset / CHAR_BIT + width / CHAR_BIT);
+	/* Re-order the info to follow IPv6 address. */
+	for (i = 0; i < idx / 2; i++) {
+		tmp = info[i];
+		MLX5_ASSERT(info[i].id);
+		MLX5_ASSERT(info[idx - 1 - i].id);
+		info[i] = info[idx - 1 - i];
+		info[idx - 1 - i] = tmp;
+	}
+}
+
 void
 mlx5_flow_field_id_to_modify_info
 		(const struct rte_flow_action_modify_data *data,
@@ -1883,6 +2019,11 @@ mlx5_flow_field_id_to_modify_info
 				info[idx].offset = data->offset;
 		}
 		break;
+	case RTE_FLOW_FIELD_FLEX_ITEM:
+		MLX5_ASSERT(data->flex_handle != NULL && !(data->offset & 0x7));
+		mlx5_modify_flex_item(dev, (const struct mlx5_flex_item *)data->flex_handle,
+				      data, info, mask, width);
+		break;
 	case RTE_FLOW_FIELD_POINTER:
 	case RTE_FLOW_FIELD_VALUE:
 	default:
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index abd0ebbb23..a023c2051b 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -4550,6 +4550,17 @@ flow_hw_actions_template_create(struct rte_eth_dev *dev,
 			at->actions[i].conf = actions->conf;
 			at->masks[i].conf = masks->conf;
 		}
+		if (actions->type == RTE_FLOW_ACTION_TYPE_MODIFY_FIELD) {
+			const struct rte_flow_action_modify_field *info = actions->conf;
+
+			if ((info->dst.field == RTE_FLOW_FIELD_FLEX_ITEM &&
+			     flow_hw_flex_item_acquire(dev, info->dst.flex_handle,
+						       &at->flex_item)) ||
+			     (info->src.field == RTE_FLOW_FIELD_FLEX_ITEM &&
+			      flow_hw_flex_item_acquire(dev, info->src.flex_handle,
+							&at->flex_item)))
+				goto error;
+		}
 	}
 	at->tmpl = flow_hw_dr_actions_template_create(at);
 	if (!at->tmpl)
@@ -4581,7 +4592,7 @@ flow_hw_actions_template_create(struct rte_eth_dev *dev,
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
 static int
-flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
+flow_hw_actions_template_destroy(struct rte_eth_dev *dev,
 				 struct rte_flow_actions_template *template,
 				 struct rte_flow_error *error __rte_unused)
 {
@@ -4594,6 +4605,7 @@ flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
 				   "action template in using");
 	}
 	LIST_REMOVE(template, next);
+	flow_hw_flex_item_release(dev, &template->flex_item);
 	if (template->tmpl)
 		mlx5dr_action_template_destroy(template->tmpl);
 	mlx5_free(template);
-- 
2.27.0