* [PATCH 00/13] net/mlx5: add hardware steering
@ 2022-02-10 16:29 Suanming Mou
2022-02-10 16:29 ` [PATCH 01/13] net/mlx5: introduce hardware steering operation Suanming Mou
` (15 more replies)
0 siblings, 16 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
As the new queue-based flow rules management mechanism is introduced
to the rte_flow. A new steering mode PMD code is also developed to
take advantage of the new API.
The HW steering prepares the required flow resources in advanced from
the hints when create template table. That means the matcher and most
of the actions have been prepared during flow table creation. Limited
left actions will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow creation/destruction
should be applied to the HW directly or not. An extra push function
has also been prepared to force push all the enqueued flows to HW.
Once the flow has been applied to the HW, the pull function will be
called to get the enqueued creation/destruction flows.
The asynchronous flow creation and destruction are handled by queue
jobs. The queue job descriptor is currently introduced to convey the
flow information and operation type between the flow management in
in pull function.
Flow rule memory is allocated in PMD layer instead of allocating from
HW layer. While destroying the flow, the flow rule memory can only be
freed after the event received.
*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW API ***
Suanming Mou (13):
net/mlx5: introduce hardware steering operation
net/mlx5: introduce hardware steering enable routine
net/mlx5: add port flow configuration
net/mlx5: add pattern template management
net/mlx5: add action template management
net/mlx5: add table management
net/mlx5: add basic flow queue operation
net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action
net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/linux/mlx5_os.c | 22 +-
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5.c | 50 +-
drivers/net/mlx5/mlx5.h | 63 +-
drivers/net/mlx5/mlx5_devx.c | 10 +
drivers/net/mlx5/mlx5_flow.c | 569 +++++-
drivers/net/mlx5/mlx5_flow.h | 276 +++
drivers/net/mlx5/mlx5_flow_dv.c | 175 +-
drivers/net/mlx5/mlx5_flow_hw.c | 2284 +++++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 78 +-
13 files changed, 3374 insertions(+), 171 deletions(-)
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 01/13] net/mlx5: introduce hardware steering operation
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 02/13] net/mlx5: introduce hardware steering enable routine Suanming Mou
` (14 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
A new hardware based steering operation is going to be introduced
for high insertion rate.
This commit adds the basic driver operation.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_flow.c | 1 +
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++
5 files changed, 17 insertions(+)
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.h b/drivers/net/mlx5/linux/mlx5_flow_os.h
index 1926d26410..e28a9e0436 100644
--- a/drivers/net/mlx5/linux/mlx5_flow_os.h
+++ b/drivers/net/mlx5/linux/mlx5_flow_os.h
@@ -9,6 +9,7 @@
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
#endif
/**
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 7d12dccdd4..edd4f126b3 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -16,6 +16,7 @@ sources = files(
'mlx5_flow.c',
'mlx5_flow_meter.c',
'mlx5_flow_dv.c',
+ 'mlx5_flow_hw.c',
'mlx5_flow_aso.c',
'mlx5_flow_flex.c',
'mlx5_mac.c',
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index d7cb1eb89b..21d17aca44 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+ [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
#endif
[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 7fec79afb3..26f5d97a7d 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
MLX5_FLOW_TYPE_MIN,
MLX5_FLOW_TYPE_DV,
MLX5_FLOW_TYPE_VERBS,
+ MLX5_FLOW_TYPE_HW,
MLX5_FLOW_TYPE_MAX,
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
new file mode 100644
index 0000000000..33875c7d08
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 02/13] net/mlx5: introduce hardware steering enable routine
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
2022-02-10 16:29 ` [PATCH 01/13] net/mlx5: introduce hardware steering operation Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 03/13] net/mlx5: add port flow configuration Suanming Mou
` (13 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
As the new hardware steering operation will be implemented under the
new rte_flow_q APIs. This is not compatible with the existing rte_flow
PMD's Direct Rules flow operation routine.
This commit introduces an extra dv_flow_en = 2 to specify the new flow
operation initialize routine.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 4 ++++
drivers/net/mlx5/mlx5.c | 2 +-
drivers/net/mlx5/mlx5.h | 3 ++-
3 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index aecdc5a68a..52e52a4ad7 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -295,6 +295,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
err = mlx5_alloc_table_hash_list(priv);
if (err)
goto error;
+ if (priv->config.dv_flow_en == 2)
+ return 0;
/* The resources below are only valid with DV support. */
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
/* Init port id action list. */
@@ -1712,6 +1714,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ if (priv->config.dv_flow_en == 2)
+ return eth_dev;
/* Port representor shares the same max priority with pf port. */
if (!priv->sh->flow_priority_check_flag) {
/* Supported Verbs flow priority number detection. */
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 67eda41a60..a4826a583b 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1933,7 +1933,7 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
config->dv_esw_en = !!tmp;
} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
- config->dv_flow_en = !!tmp;
+ config->dv_flow_en = tmp;
} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
if (tmp != MLX5_XMETA_MODE_LEGACY &&
tmp != MLX5_XMETA_MODE_META16 &&
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 737ad6895c..f3b991e549 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -259,7 +259,8 @@ struct mlx5_dev_config {
unsigned int l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
unsigned int vf_nl_en:1; /* Enable Netlink requests in VF mode. */
unsigned int dv_esw_en:1; /* Enable E-Switch DV flow. */
- unsigned int dv_flow_en:1; /* Enable DV flow. */
+ /* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+ unsigned int dv_flow_en:2;
unsigned int dv_xmeta_en:2; /* Enable extensive flow metadata. */
unsigned int lacp_by_user:1;
/* Enable user to manage LACP traffic. */
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 03/13] net/mlx5: add port flow configuration
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
2022-02-10 16:29 ` [PATCH 01/13] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-10 16:29 ` [PATCH 02/13] net/mlx5: introduce hardware steering enable routine Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 04/13] net/mlx5: add pattern template management Suanming Mou
` (12 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The hardware steering is backend to support rte_flow_q API in mlx5
PMD. The port configuration function creates the queues and needed
flow management resources.
The PMD layer configuration function allocates the queues' context
and per-queue job descriptor. The job descriptor size is equal to
the queue size, and the job descriptors will be popped from LIFO
to convey the flow information during flow insertion/destruction.
So when polling the result, the flow information will be extracted
from the descriptor and then the job descriptor will be push back
to the LIFO.
The commit creates the flow port queue and the job descriptors.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 3 +
drivers/net/mlx5/mlx5.h | 26 ++++++-
drivers/net/mlx5/mlx5_flow.c | 37 +++++++++
drivers/net/mlx5/mlx5_flow.h | 9 +++
drivers/net/mlx5/mlx5_flow_hw.c | 132 ++++++++++++++++++++++++++++++++
5 files changed, 206 insertions(+), 1 deletion(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index a4826a583b..f1933fd253 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1567,6 +1567,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
/* Free the eCPRI flex parser resource. */
mlx5_flex_parser_ecpri_release(dev);
mlx5_flex_item_port_cleanup(dev);
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ flow_hw_resource_release(dev);
+#endif
if (priv->rxq_privs != NULL) {
/* XXX race condition if mlx5_rx_burst() is still running. */
rte_delay_us_sleep(1000);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f3b991e549..31a13ca69a 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -33,7 +33,7 @@
#include "mlx5_utils.h"
#include "mlx5_os.h"
#include "mlx5_autoconf.h"
-
+#include "mlx5dr.h"
#define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -324,6 +324,26 @@ struct mlx5_lb_ctx {
uint16_t refcnt; /* Reference count for representors. */
};
+/* HW steering queue job descriptor type. */
+enum {
+ MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+ MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+ uint32_t type; /* Job type. */
+ struct rte_flow *flow; /* Flow attached to the job. */
+ void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO header . */
+struct mlx5_hw_q {
+ uint32_t job_idx; /* Free job index. */
+ uint32_t size; /* LIFO size. */
+ struct mlx5_hw_q_job **job; /* LIFO pointer. */
+} __rte_cache_aligned;
+
#define MLX5_COUNTERS_PER_POOL 512
#define MLX5_MAX_PENDING_QUERIES 4
#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1480,6 +1500,10 @@ struct mlx5_priv {
struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
+ struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+ uint32_t nb_queue; /* HW steering queue number. */
+ /* HW steering queue polling mechanism job descriptor LIFO. */
+ struct mlx5_hw_q *hw_q;
};
#define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 21d17aca44..5ff96642b4 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -805,6 +805,12 @@ static int
mlx5_flow_flex_item_release(struct rte_eth_dev *dev,
const struct rte_flow_item_flex_handle *handle,
struct rte_flow_error *error);
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -826,6 +832,7 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.get_restore_info = mlx5_flow_tunnel_get_restore_info,
.flex_item_create = mlx5_flow_flex_item_create,
.flex_item_release = mlx5_flow_flex_item_release,
+ .configure = mlx5_flow_port_configure,
};
/* Tunnel information. */
@@ -7814,6 +7821,36 @@ mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt,
return -ENOTSUP;
}
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->configure(dev, port_attr, nb_queue, queue_attr, err);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 26f5d97a7d..731478ff05 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1257,6 +1257,12 @@ typedef int (*mlx5_flow_item_update_t)
const struct rte_flow_item_flex_handle *handle,
const struct rte_flow_item_flex_conf *conf,
struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1295,6 +1301,7 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_create_t item_create;
mlx5_flow_item_release_t item_release;
mlx5_flow_item_update_t item_update;
+ mlx5_flow_port_configure_t configure;
};
/* mlx5_flow.c */
@@ -1767,4 +1774,6 @@ const struct mlx5_flow_tunnel *
mlx5_get_tof(const struct rte_flow_item *items,
const struct rte_flow_action *actions,
enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 33875c7d08..4194f81ee9 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -4,10 +4,142 @@
#include <rte_flow.h>
+#include <mlx5_malloc.h>
+#include "mlx5_defs.h"
#include "mlx5_flow.h"
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_context *dr_ctx = NULL;
+ struct mlx5dr_context_attr dr_ctx_attr = {0};
+ struct mlx5_hw_q *hw_q;
+ struct mlx5_hw_q_job *job = NULL;
+ uint32_t mem_size, i, j;
+
+ if (!port_attr || !nb_queue || !queue_attr) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ /* In case re-configuring, release existing context at first. */
+ if (priv->dr_ctx) {
+ /* */
+ for (i = 0; i < nb_queue; i++) {
+ hw_q = &priv->hw_q[i];
+ /* Make sure all queues are empty. */
+ if (hw_q->size != hw_q->job_idx) {
+ rte_errno = EBUSY;
+ goto err;
+ }
+ }
+ flow_hw_resource_release(dev);
+ }
+ /* Allocate the queue job descriptor LIFO. */
+ mem_size = sizeof(priv->hw_q[0]) * nb_queue;
+ for (i = 0; i < nb_queue; i++) {
+ /*
+ * Check if the queues' size are all the same as the
+ * limitation from HWS layer.
+ */
+ if (queue_attr[i]->size != queue_attr[0]->size) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(struct mlx5_hw_q_job)) *
+ queue_attr[0]->size;
+ }
+ priv->hw_q = mlx5_malloc(MLX5_MEM_ZERO, mem_size,
+ 64, SOCKET_ID_ANY);
+ if (!priv->hw_q) {
+ rte_errno = ENOMEM;
+ goto err;
+ }
+ for (i = 0; i < nb_queue; i++) {
+ priv->hw_q[i].job_idx = queue_attr[i]->size;
+ priv->hw_q[i].size = queue_attr[i]->size;
+ if (i == 0)
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &priv->hw_q[nb_queue];
+ else
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &job[queue_attr[i - 1]->size];
+ job = (struct mlx5_hw_q_job *)
+ &priv->hw_q[i].job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++)
+ priv->hw_q[i].job[j] = &job[j];
+ }
+ dr_ctx_attr.pd = priv->sh->cdev->pd;
+ dr_ctx_attr.queues = nb_queue;
+ /* Queue size should all be the same. Take the first one. */
+ dr_ctx_attr.queue_size = queue_attr[0]->size;
+ dr_ctx = mlx5dr_context_open(priv->sh->cdev->ctx, &dr_ctx_attr);
+ /* rte_errno has been updated by HWS layer. */
+ if (!dr_ctx)
+ goto err;
+ priv->dr_ctx = dr_ctx;
+ priv->nb_queue = nb_queue;
+ return 0;
+err:
+ if (dr_ctx)
+ claim_zero(mlx5dr_context_close(dr_ctx));
+ if (priv->hw_q) {
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ }
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to configure port");
+}
+
+/**
+ * Release HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ */
+void
+flow_hw_resource_release(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (!priv->dr_ctx)
+ return;
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ claim_zero(mlx5dr_context_close(priv->dr_ctx));
+ priv->dr_ctx = NULL;
+ priv->nb_queue = 0;
+}
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
+ .configure = flow_hw_configure,
+};
+
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 04/13] net/mlx5: add pattern template management
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (2 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 03/13] net/mlx5: add port flow configuration Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 05/13] net/mlx5: add action " Suanming Mou
` (11 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The pattern template defines flows that have the same matching
fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be
(eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while
the values for each rule will be different.
Due to the pattern template can be used in different domains, the
items will only be cached in pattern template create stage, while
the template is binded to a dedicated table, the HW criteria
will be created and saved to the table. And different tables
may create the same criteria and will not shared between each
other in order to have better performance.
This commit adds pattern template management.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 64 +++++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 20 ++++++++
drivers/net/mlx5/mlx5_flow_hw.c | 82 +++++++++++++++++++++++++++++++++
4 files changed, 168 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 31a13ca69a..96048ad0ea 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1500,6 +1500,8 @@ struct mlx5_priv {
struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
+ /* Item template list. */
+ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 5ff96642b4..27a40a9627 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -812,6 +812,17 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -833,6 +844,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.flex_item_create = mlx5_flow_flex_item_create,
.flex_item_release = mlx5_flow_flex_item_release,
.configure = mlx5_flow_port_configure,
+ .pattern_template_create = mlx5_flow_pattern_template_create,
+ .pattern_template_destroy = mlx5_flow_pattern_template_destroy,
};
/* Tunnel information. */
@@ -7851,6 +7864,57 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
return fops->configure(dev, port_attr, nb_queue, queue_attr, err);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->pattern_template_create(dev, attr, items, error);
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->pattern_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 731478ff05..88102f0991 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1015,6 +1015,15 @@ struct rte_flow {
uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
} __rte_packed;
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+ LIST_ENTRY(rte_flow_pattern_template) next;
+ /* Template attributes. */
+ struct rte_flow_pattern_template_attr attr;
+ struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint32_t refcnt; /* Reference counter. */
+};
+
/*
* Define list of valid combinations of RX Hash fields
* (see enum ibv_rx_hash_fields).
@@ -1263,6 +1272,15 @@ typedef int (*mlx5_flow_port_configure_t)
uint16_t nb_queue,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1302,6 +1320,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_release_t item_release;
mlx5_flow_item_update_t item_update;
mlx5_flow_port_configure_t configure;
+ mlx5_flow_pattern_template_create_t pattern_template_create;
+ mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 4194f81ee9..c984e520cd 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,81 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Item template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+flow_hw_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
+
+ it = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*it), 0, SOCKET_ID_ANY);
+ if (!it) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate item template");
+ return NULL;
+ }
+ it->attr = *attr;
+ it->mt = mlx5dr_match_template_create(items, attr->relaxed_matching);
+ if (!it->mt) {
+ mlx5_free(it);
+ return NULL;
+ }
+ __atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
+ return it;
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Item template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "item template in using");
+ }
+ LIST_REMOVE(template, next);
+ claim_zero(mlx5dr_match_template_destroy(template->mt));
+ mlx5_free(template);
+ return 0;
+}
+
/**
* Configure port HWS resources.
*
@@ -128,9 +203,14 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_itt)) {
+ it = LIST_FIRST(&priv->flow_hw_itt);
+ flow_hw_pattern_template_destroy(dev, it, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -140,6 +220,8 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.configure = flow_hw_configure,
+ .pattern_template_create = flow_hw_pattern_template_create,
+ .pattern_template_destroy = flow_hw_pattern_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 05/13] net/mlx5: add action template management
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (3 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 04/13] net/mlx5: add pattern template management Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 06/13] net/mlx5: add table management Suanming Mou
` (10 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The action template holds a list of action types that will be
used together on the same rule. The template's actions instances
will be created only when the template bind to the dedicated
group. And the created actions will be saved to each individual
group in order for best performance. The actions will not be
shared with each other.
This commit adds the action template management which caches the
flow action template.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 66 ++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 27 ++++++++
drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
4 files changed, 211 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 96048ad0ea..80dc72175d 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1502,6 +1502,8 @@ struct mlx5_priv {
uint32_t flex_item_map; /* Map of allocated flex item elements. */
/* Item template list. */
LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+ /* Action template list. */
+ LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 27a40a9627..be6a7ff336 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -822,6 +822,16 @@ static int
mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -846,6 +856,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.configure = mlx5_flow_port_configure,
.pattern_template_create = mlx5_flow_pattern_template_create,
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
+ .actions_template_create = mlx5_flow_actions_template_create,
+ .actions_template_destroy = mlx5_flow_actions_template_destroy,
};
/* Tunnel information. */
@@ -7915,6 +7927,60 @@ mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
return fops->pattern_template_destroy(dev, template, error);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->actions_template_create(dev, attr, actions, masks, error);
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->actions_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 88102f0991..f5ababb32f 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1024,6 +1024,21 @@ struct rte_flow_pattern_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Flow action template attribute. */
+struct rte_flow_actions_template_attr {
+ int32_t reserve;
+};
+
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+ LIST_ENTRY(rte_flow_actions_template) next;
+ /* Template attributes. */
+ struct rte_flow_actions_template_attr attr;
+ struct rte_flow_action *actions; /* Cached flow actions. */
+ struct rte_flow_action *masks; /* Cached action masks.*/
+ uint32_t refcnt; /* Reference counter. */
+};
+
/*
* Define list of valid combinations of RX Hash fields
* (see enum ibv_rx_hash_fields).
@@ -1281,6 +1296,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1322,6 +1347,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_port_configure_t configure;
mlx5_flow_pattern_template_create_t pattern_template_create;
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+ mlx5_flow_actions_template_create_t actions_template_create;
+ mlx5_flow_actions_template_destroy_t actions_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index c984e520cd..349c12a849 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,115 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Create flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+flow_hw_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int len, act_len, mask_len, i;
+ struct rte_flow_actions_template *at;
+
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, actions, error);
+ if (act_len <= 0)
+ return NULL;
+ len = RTE_ALIGN(act_len, 16);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, masks, error);
+ if (mask_len <= 0)
+ return NULL;
+ len += RTE_ALIGN(mask_len, 16);
+ at = mlx5_malloc(MLX5_MEM_ZERO, len + sizeof(*at), 64, SOCKET_ID_ANY);
+ if (!at) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate action template");
+ return NULL;
+ }
+ at->attr = *attr;
+ at->actions = (struct rte_flow_action *)(at + 1);
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->actions, len,
+ actions, error);
+ if (act_len <= 0)
+ goto error;
+ at->masks = (struct rte_flow_action *)
+ (((uint8_t *)at->actions) + act_len);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->masks,
+ len - act_len, masks, error);
+ if (mask_len <= 0)
+ goto error;
+ /*
+ * mlx5 PMD hacks indirect action index directly to the action conf.
+ * The rte_flow_conv() function copies the content from conf pointer.
+ * Need to restore the indirect action index from action conf here.
+ */
+ for (i = 0; actions->type != RTE_FLOW_ACTION_TYPE_END;
+ actions++, masks++, i++) {
+ if (actions->type == RTE_FLOW_ACTION_TYPE_INDIRECT) {
+ at->actions[i].conf = actions->conf;
+ at->masks[i].conf = masks->conf;
+ }
+ }
+ __atomic_fetch_add(&at->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_at, at, next);
+ return at;
+error:
+ mlx5_free(at);
+ return NULL;
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Action template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action template in using");
+ }
+ LIST_REMOVE(template, next);
+ mlx5_free(template);
+ return 0;
+}
+
/**
* Create flow item template.
*
@@ -204,6 +313,7 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct rte_flow_pattern_template *it;
+ struct rte_flow_actions_template *at;
if (!priv->dr_ctx)
return;
@@ -211,6 +321,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
}
+ while (!LIST_EMPTY(&priv->flow_hw_at)) {
+ at = LIST_FIRST(&priv->flow_hw_at);
+ flow_hw_actions_template_destroy(dev, at, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -222,6 +336,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.configure = flow_hw_configure,
.pattern_template_create = flow_hw_pattern_template_create,
.pattern_template_destroy = flow_hw_pattern_template_destroy,
+ .actions_template_create = flow_hw_actions_template_create,
+ .actions_template_destroy = flow_hw_actions_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 06/13] net/mlx5: add table management
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (4 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 05/13] net/mlx5: add action " Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 07/13] net/mlx5: add basic flow queue operation Suanming Mou
` (9 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Flow table is a group of flows with the same matching criteria
and the same actions defined for them. The table defines rules
that have the same matching fields but with different matching
values. For example, matching on 5 tuple, the table will be
(IPv4 source + IPv4 dest + s_port + d_port + next_proto)
while the values for each rule will be different.
The templates' relevant matching criteria and action instances
will be created in the table creation and saved in the table.
As table attributes indicate the supported flow number, the flow
memory will also be allocated at the same time.
This commit adds the table management functions.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 45 ++-
drivers/net/mlx5/mlx5.h | 21 +-
drivers/net/mlx5/mlx5_flow.c | 81 +++++
drivers/net/mlx5/mlx5_flow.h | 72 +++++
drivers/net/mlx5/mlx5_flow_hw.c | 522 ++++++++++++++++++++++++++++++++
5 files changed, 735 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f1933fd253..51f3d9bf99 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1354,12 +1354,46 @@ void
mlx5_free_table_hash_list(struct mlx5_priv *priv)
{
struct mlx5_dev_ctx_shared *sh = priv->sh;
-
- if (!sh->flow_tbls)
+ struct mlx5_hlist **tbls = (priv->config.dv_flow_en == 2) ?
+ &sh->groups : &sh->flow_tbls;
+ if (*tbls == NULL)
return;
- mlx5_hlist_destroy(sh->flow_tbls);
- sh->flow_tbls = NULL;
+ mlx5_hlist_destroy(*tbls);
+ *tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ * Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+ int err = 0;
+ struct mlx5_dev_ctx_shared *sh = priv->sh;
+ char s[MLX5_NAME_SIZE];
+
+ MLX5_ASSERT(sh);
+ snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+ sh->groups = mlx5_hlist_create
+ (s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+ false, true, sh,
+ flow_hw_grp_create_cb,
+ flow_hw_grp_match_cb,
+ flow_hw_grp_remove_cb,
+ flow_hw_grp_clone_cb,
+ flow_hw_grp_clone_free_cb);
+ if (!sh->groups) {
+ DRV_LOG(ERR, "flow groups with hash creation failed.");
+ err = ENOMEM;
+ }
+ return err;
}
+#endif
+
/**
* Initialize flow table hash list and create the root tables entry
@@ -1375,11 +1409,14 @@ int
mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
{
int err = 0;
+
/* Tables are only used in DV and DR modes. */
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
struct mlx5_dev_ctx_shared *sh = priv->sh;
char s[MLX5_NAME_SIZE];
+ if (priv->config.dv_flow_en == 2)
+ return mlx5_alloc_hw_group_hash_list(priv);
MLX5_ASSERT(sh);
snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 80dc72175d..a4bc8d1fb7 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -62,7 +62,9 @@ enum mlx5_ipool_index {
MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
MLX5_IPOOL_TAG, /* Pool for tag resource. */
MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
- MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+ MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+ /* Pool for HWS group. Jump action will be created internally. */
+ MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -106,6 +108,13 @@ enum mlx5_delay_drop_mode {
MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
};
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+ MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+ MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+ MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
/* Hlist and list callback context. */
struct mlx5_flow_cb_ctx {
struct rte_eth_dev *dev;
@@ -1203,7 +1212,10 @@ struct mlx5_dev_ctx_shared {
rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
/* UAR same-page access control required in 32bit implementations. */
#endif
- struct mlx5_hlist *flow_tbls;
+ union {
+ struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+ struct mlx5_hlist *groups; /* HWS flow group. */
+ };
struct mlx5_flow_tunnel_hub *tunnel_hub;
/* Direct Rules tables for FDB, NIC TX+RX */
void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1508,6 +1520,11 @@ struct mlx5_priv {
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
struct mlx5_hw_q *hw_q;
+ /* HW steering rte flow table list header. */
+ LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX];
};
#define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index be6a7ff336..2e70e1eaaf 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -833,6 +833,19 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -858,6 +871,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
.actions_template_create = mlx5_flow_actions_template_create,
.actions_template_destroy = mlx5_flow_actions_template_destroy,
+ .template_table_create = mlx5_flow_table_create,
+ .template_table_destroy = mlx5_flow_table_destroy,
};
/* Tunnel information. */
@@ -7981,6 +7996,72 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
return fops->actions_template_destroy(dev, template, error);
}
+/**
+ * Create flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->template_table_create(dev,
+ attr,
+ item_templates,
+ nb_item_templates,
+ action_templates,
+ nb_action_templates,
+ error);
+}
+
+/**
+ * PMD destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->template_table_destroy(dev, table, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index f5ababb32f..02eb03e4cf 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1039,6 +1039,54 @@ struct rte_flow_actions_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Jump action struct. */
+struct mlx5_hw_jump_action {
+ /* Action jump from root. */
+ struct mlx5dr_action *root_action;
+ /* HW steering jump action. */
+ struct mlx5dr_action *hws_action;
+};
+
+/* DR action set struct. */
+struct mlx5_hw_actions {
+ struct mlx5dr_action *drop; /* Drop action. */
+};
+
+/* mlx5 action template struct. */
+struct mlx5_hw_action_template {
+ /* Action template pointer. */
+ struct rte_flow_actions_template *action_template;
+ struct mlx5_hw_actions acts; /* Template actions. */
+};
+
+/* mlx5 flow group struct. */
+struct mlx5_flow_group {
+ struct mlx5_list_entry entry;
+ struct mlx5dr_table *tbl; /* HWS table object. */
+ struct mlx5_hw_jump_action jump; /* Jump action. */
+ enum mlx5dr_table_type type; /* Table type. */
+ uint32_t group_id; /* Group id. */
+ uint32_t idx; /* Group memory index. */
+};
+
+#define MLX5_HW_TBL_MAX_ITEM_TEMPLATE 2
+#define MLX5_HW_TBL_MAX_ACTION_TEMPLATE 32
+
+struct rte_flow_template_table {
+ LIST_ENTRY(rte_flow_template_table) next;
+ struct mlx5_flow_group *grp; /* The group rte_flow_template_table uses. */
+ struct mlx5dr_matcher *matcher; /* Template matcher. */
+ /* Item templates bind to the table. */
+ struct rte_flow_pattern_template *its[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ /* Action templates bind to the table. */
+ struct mlx5_hw_action_template ats[MLX5_HW_TBL_MAX_ACTION_TEMPLATE];
+ struct mlx5_indexed_pool *flow; /* The table's flow ipool. */
+ uint32_t type; /* Flow table type RX/TX/FDB. */
+ uint8_t nb_item_templates; /* Item template number. */
+ uint8_t nb_action_templates; /* Action template number. */
+ uint32_t refcnt; /* Table reference counter. */
+};
+
/*
* Define list of valid combinations of RX Hash fields
* (see enum ibv_rx_hash_fields).
@@ -1306,6 +1354,18 @@ typedef int (*mlx5_flow_actions_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_template_table *(*mlx5_flow_table_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_table_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1349,6 +1409,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
mlx5_flow_actions_template_create_t actions_template_create;
mlx5_flow_actions_template_destroy_t actions_template_destroy;
+ mlx5_flow_table_create_t template_table_create;
+ mlx5_flow_table_destroy_t template_table_destroy;
};
/* mlx5_flow.c */
@@ -1784,6 +1846,16 @@ int
flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
struct rte_flow_error *error);
+struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
+void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+int flow_hw_grp_match_cb(void *tool_ctx,
+ struct mlx5_list_entry *entry,
+ void *cb_ctx);
+struct mlx5_list_entry *flow_hw_grp_clone_cb(void *tool_ctx,
+ struct mlx5_list_entry *oentry,
+ void *cb_ctx);
+void flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+
struct mlx5_aso_age_action *flow_aso_age_get_by_idx(struct rte_eth_dev *dev,
uint32_t age_idx);
int flow_dev_geneve_tlv_option_resource_register(struct rte_eth_dev *dev,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 349c12a849..5cb5e2ebb9 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,303 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/* DR action flags with different table. */
+static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX] = {
+ {
+ MLX5DR_ACTION_FLAG_ROOT_RX,
+ MLX5DR_ACTION_FLAG_ROOT_TX,
+ MLX5DR_ACTION_FLAG_ROOT_FDB,
+ },
+ {
+ MLX5DR_ACTION_FLAG_HWS_RX,
+ MLX5DR_ACTION_FLAG_HWS_TX,
+ MLX5DR_ACTION_FLAG_HWS_FDB,
+ },
+};
+
+/**
+ * Destroy DR actions created by action template.
+ *
+ * For DR actions created during table creation's action translate.
+ * Need to destroy the DR action when destroying the table.
+ *
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ */
+static void
+__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+{
+}
+
+/**
+ * Translate rte_flow actions to DR action.
+ *
+ * As the action template has already indicated the actions. Translate
+ * the rte_flow actions to DR action if possbile. So in flow create
+ * stage we will save cycles from handing the actions' organizing.
+ * For the actions with limited information, need to add these to a
+ * list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in/out] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] at
+ * Action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *table_attr,
+ struct mlx5_hw_actions *acts,
+ struct rte_flow_actions_template *at,
+ struct rte_flow_error *error __rte_unused)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ const struct rte_flow_attr *attr = &table_attr->flow_attr;
+ struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *masks = at->masks;
+ bool actions_end = false;
+ uint32_t type;
+
+ if (attr->transfer)
+ type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ type = MLX5DR_TABLE_TYPE_NIC_RX;
+ for (; !actions_end; actions++, masks++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ acts->drop = priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ return 0;
+}
+
+/**
+ * Create flow table.
+ *
+ * The input item and action templates will be binded to the table.
+ * Flow memory will also be allocated. Matcher will be created based
+ * on the item template. Action will be translated to the dedicated
+ * DR action if possible.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+flow_hw_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_matcher_attr matcher_attr = {0};
+ struct rte_flow_template_table *tbl = NULL;
+ struct mlx5_flow_group *grp;
+ struct mlx5dr_match_template *mt[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ struct rte_flow_attr flow_attr = attr->flow_attr;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &flow_attr,
+ };
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow),
+ .trunk_size = 1 << 12,
+ .per_core_cache = 1 << 13,
+ .need_lock = 1,
+ .release_mem_en = !!priv->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_table_flow",
+ };
+ struct mlx5_list_entry *ge;
+ uint32_t i, max_tpl = MLX5_HW_TBL_MAX_ITEM_TEMPLATE;
+ uint32_t nb_flows = rte_align32pow2(attr->nb_flows);
+ int err;
+
+ /* HWS layer accepts only 1 item template with root table. */
+ if (!attr->flow_attr.group)
+ max_tpl = 1;
+ cfg.max_idx = nb_flows;
+ /* For table has limited flows, resize the cache and trunk size. */
+ if (nb_flows < cfg.trunk_size) {
+ cfg.per_core_cache = nb_flows >> 2;
+ cfg.trunk_size = nb_flows;
+ }
+ /* Check if we requires too many templates. */
+ if (nb_item_templates > max_tpl ||
+ nb_action_templates > MLX5_HW_TBL_MAX_ACTION_TEMPLATE) {
+ rte_errno = EINVAL;
+ goto error;
+ }
+ /* Allocate the table memory. */
+ tbl = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*tbl), 0, SOCKET_ID_ANY);
+ if (!tbl)
+ goto error;
+ /* Allocate flow indexed pool. */
+ tbl->flow = mlx5_ipool_create(&cfg);
+ if (!tbl->flow)
+ goto error;
+ /* Register the flow group. */
+ ge = mlx5_hlist_register(priv->sh->groups, attr->flow_attr.group, &ctx);
+ if (!ge)
+ goto error;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ tbl->grp = grp;
+ /* Prepare matcher information. */
+ matcher_attr.priority = attr->flow_attr.priority;
+ matcher_attr.mode = MLX5DR_MATCHER_RESOURCE_MODE_RULE;
+ matcher_attr.rule.num_log = rte_log2_u32(nb_flows);
+ /* Build the item template. */
+ for (i = 0; i < nb_item_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&item_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto it_error;
+ }
+ mt[i] = item_templates[i]->mt;
+ tbl->its[i] = item_templates[i];
+ }
+ tbl->matcher = mlx5dr_matcher_create
+ (tbl->grp->tbl, mt, nb_item_templates, &matcher_attr);
+ if (!tbl->matcher)
+ goto it_error;
+ tbl->nb_item_templates = nb_item_templates;
+ /* Build the action template. */
+ for (i = 0; i < nb_action_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&action_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto at_error;
+ }
+ err = flow_hw_actions_translate(dev, attr,
+ &tbl->ats[i].acts,
+ action_templates[i], error);
+ if (err)
+ goto at_error;
+ tbl->ats[i].action_template = action_templates[i];
+ }
+ tbl->nb_action_templates = nb_action_templates;
+ tbl->type = attr->flow_attr.transfer ? MLX5DR_TABLE_TYPE_FDB :
+ (attr->flow_attr.egress ? MLX5DR_TABLE_TYPE_NIC_TX :
+ MLX5DR_TABLE_TYPE_NIC_RX);
+ LIST_INSERT_HEAD(&priv->flow_hw_tbl, tbl, next);
+ return tbl;
+at_error:
+ while (i--) {
+ __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __atomic_sub_fetch(&action_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ i = nb_item_templates;
+it_error:
+ while (i--)
+ __atomic_sub_fetch(&item_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ mlx5dr_matcher_destroy(tbl->matcher);
+error:
+ err = rte_errno;
+ if (tbl) {
+ if (tbl->flow)
+ mlx5_ipool_destroy(tbl->flow);
+ mlx5_free(tbl);
+ }
+ rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
+ return NULL;
+}
+
+/**
+ * Destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error __rte_unused)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int i;
+
+ if (table->refcnt) {
+ DRV_LOG(WARNING, "Table %p is still in using.", (void *)table);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table in using");
+ }
+ LIST_REMOVE(table, next);
+ for (i = 0; i < table->nb_item_templates; i++)
+ __atomic_sub_fetch(&table->its[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ for (i = 0; i < table->nb_action_templates; i++) {
+ __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __atomic_sub_fetch(&table->ats[i].action_template->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ mlx5dr_matcher_destroy(table->matcher);
+ mlx5_hlist_unregister(priv->sh->groups, &table->grp->entry);
+ mlx5_ipool_destroy(table->flow);
+ mlx5_free(table);
+ return 0;
+}
+
/**
* Create flow action template.
*
@@ -196,6 +493,199 @@ flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
return 0;
}
+/**
+ * Create group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] cb_ctx
+ * Pointer to the group creation context.
+ *
+ * @return
+ * Group entry on success, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct rte_eth_dev *dev = ctx->dev;
+ struct rte_flow_attr *attr = (struct rte_flow_attr *)ctx->data;
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_table_attr dr_tbl_attr = {0};
+ struct rte_flow_error *error = ctx->error;
+ struct mlx5_flow_group *grp_data;
+ struct mlx5dr_table *tbl = NULL;
+ struct mlx5dr_action *jump;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_zmalloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ dr_tbl_attr.level = attr->group;
+ if (attr->transfer)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_RX;
+ tbl = mlx5dr_table_create(priv->dr_ctx, &dr_tbl_attr);
+ if (!tbl)
+ goto error;
+ grp_data->tbl = tbl;
+ if (attr->group) {
+ /* Jump action be used by non-root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[!!attr->group][dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.hws_action = jump;
+ /* Jump action be used by root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_ROOT]
+ [dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.root_action = jump;
+ }
+ grp_data->idx = idx;
+ grp_data->group_id = attr->group;
+ grp_data->type = dr_tbl_attr.type;
+ return &grp_data->entry;
+error:
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (tbl)
+ mlx5dr_table_destroy(tbl);
+ if (idx)
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], idx);
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow dr table");
+ return NULL;
+}
+
+/**
+ * Remove group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the entry to be removed.
+ */
+void
+flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ MLX5_ASSERT(entry && sh);
+ /* To use the wrapper glue functions instead. */
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ mlx5dr_table_destroy(grp_data->tbl);
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
+/**
+ * Match group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+int
+flow_hw_grp_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
+ void *cb_ctx)
+{
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+ struct rte_flow_attr *attr =
+ (struct rte_flow_attr *)ctx->data;
+
+ return (grp_data->group_id != attr->group) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_FDB) &&
+ attr->transfer) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_TX) &&
+ attr->egress) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_RX) &&
+ attr->ingress);
+}
+
+/**
+ * Clone group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_clone_cb(void *tool_ctx, struct mlx5_list_entry *oentry,
+ void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data;
+ struct rte_flow_error *error = ctx->error;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_malloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ memcpy(grp_data, oentry, sizeof(*grp_data));
+ grp_data->idx = idx;
+ return &grp_data->entry;
+}
+
+/**
+ * Free cloned group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be freed.
+ */
+void
+flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
/**
* Configure port HWS resources.
*
@@ -213,6 +703,7 @@ flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
+
static int
flow_hw_configure(struct rte_eth_dev *dev,
const struct rte_flow_port_attr *port_attr,
@@ -289,8 +780,24 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
priv->dr_ctx = dr_ctx;
priv->nb_queue = nb_queue;
+ /* Add global actions. */
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ priv->hw_drop[i][j] = mlx5dr_action_create_dest_drop
+ (priv->dr_ctx, mlx5_hw_act_flag[i][j]);
+ if (!priv->hw_drop[i][j])
+ goto err;
+ }
+ }
return 0;
err:
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
if (dr_ctx)
claim_zero(mlx5dr_context_close(dr_ctx));
if (priv->hw_q) {
@@ -312,11 +819,17 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_template_table *tbl;
struct rte_flow_pattern_template *it;
struct rte_flow_actions_template *at;
+ int i, j;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_tbl)) {
+ tbl = LIST_FIRST(&priv->flow_hw_tbl);
+ flow_hw_table_destroy(dev, tbl, NULL);
+ }
while (!LIST_EMPTY(&priv->flow_hw_itt)) {
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
@@ -325,6 +838,13 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
at = LIST_FIRST(&priv->flow_hw_at);
flow_hw_actions_template_destroy(dev, at, NULL);
}
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -338,6 +858,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.pattern_template_destroy = flow_hw_pattern_template_destroy,
.actions_template_create = flow_hw_actions_template_create,
.actions_template_destroy = flow_hw_actions_template_destroy,
+ .template_table_create = flow_hw_table_create,
+ .template_table_destroy = flow_hw_table_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 07/13] net/mlx5: add basic flow queue operation
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (5 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 06/13] net/mlx5: add table management Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 08/13] net/mlx5: add flow flush function Suanming Mou
` (8 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The HW steering uses queue-based flow rules management mechanism.
The matcher and part of the actions have been prepared during
flow table creation. Some left actions will be constructed during
flow creation if needed.
A flow postpone attribute bit decribles if flow creation/destruction
should be applied to the HW directly. An extra drain function has
also been prepared to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function will be
called to get the enqueued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead of
allocating from HW steering layer. While destroying the flow, the
flow rule memory can only be freed after the CQE received.
The HW queue job descriptor is currently introduced to convey
the flow information and operation type between the flow
insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:
rte_flow_q_flow_create();
rte_flow_q_flow_destroy();
rte_flow_q_push();
rte_flow_q_pull();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +-
drivers/net/mlx5/mlx5_flow.c | 158 ++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 39 +++++
drivers/net/mlx5/mlx5_flow_hw.c | 278 +++++++++++++++++++++++++++++++-
4 files changed, 475 insertions(+), 2 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a4bc8d1fb7..ec4eb7ee94 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -342,7 +342,7 @@ enum {
/* HW steering flow management job descriptor. */
struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
- struct rte_flow *flow; /* Flow attached to the job. */
+ struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
};
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 2e70e1eaaf..b48a3af0fb 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -845,6 +845,33 @@ static int
mlx5_flow_table_destroy(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+static struct rte_flow *
+mlx5_flow_q_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_q_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow *flow,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_q_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_q_op_res res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_q_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -873,6 +900,10 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.actions_template_destroy = mlx5_flow_actions_template_destroy,
.template_table_create = mlx5_flow_table_create,
.template_table_destroy = mlx5_flow_table_destroy,
+ .q_flow_create = mlx5_flow_q_flow_create,
+ .q_flow_destroy = mlx5_flow_q_flow_destroy,
+ .q_pull = mlx5_flow_q_pull,
+ .q_push = mlx5_flow_q_push,
};
/* Tunnel information. */
@@ -8062,6 +8093,133 @@ mlx5_flow_table_destroy(struct rte_eth_dev *dev,
return fops->template_table_destroy(dev, table, error);
}
+/**
+ * Enqueue flow creation.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue_id
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+mlx5_flow_q_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue_id,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->q_flow_create(dev, queue_id, attr, table,
+ items, pattern_template_index,
+ actions, action_template_index,
+ error);
+}
+
+/**
+ * Enqueue flow destruction.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_q_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow *flow,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->q_flow_destroy(dev, queue, attr, flow, error);
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_q_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_q_op_res res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->q_pull(dev, queue, res, n_res, error);
+}
+
+/**
+ * Push the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flows.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_q_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->q_push(dev, queue, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 02eb03e4cf..40eb8d79aa 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1015,6 +1015,13 @@ struct rte_flow {
uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
} __rte_packed;
+/* HWS flow struct. */
+struct rte_flow_hw {
+ uint32_t idx; /* Flow index from indexed pool. */
+ struct rte_flow_template_table *table; /* The table flow allcated from. */
+ struct mlx5dr_rule rule; /* HWS layer data struct. */
+} __rte_packed;
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1366,6 +1373,32 @@ typedef int (*mlx5_flow_table_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+typedef struct rte_flow *(*mlx5_flow_q_flow_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_q_flow_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow *flow,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_q_pull_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_q_op_res res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_q_push_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1411,6 +1444,10 @@ struct mlx5_flow_driver_ops {
mlx5_flow_actions_template_destroy_t actions_template_destroy;
mlx5_flow_table_create_t template_table_create;
mlx5_flow_table_destroy_t template_table_destroy;
+ mlx5_flow_q_flow_create_t q_flow_create;
+ mlx5_flow_q_flow_destroy_t q_flow_destroy;
+ mlx5_flow_q_pull_t q_pull;
+ mlx5_flow_q_push_t q_push;
};
/* mlx5_flow.c */
@@ -1581,6 +1618,8 @@ mlx5_translate_tunnel_etypes(uint64_t pattern_flags)
return 0;
}
+int flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error);
int mlx5_flow_group_to_table(struct rte_eth_dev *dev,
const struct mlx5_flow_tunnel *tunnel,
uint32_t group, uint32_t *table,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 5cb5e2ebb9..a74825312f 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -10,6 +10,9 @@
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/* DR action flags with different table. */
@@ -105,6 +108,275 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Construct flow action array.
+ *
+ * For action template contains dynamic actions, these actions need to
+ * be updated according to the rte_flow action during flow creation.
+ *
+ * @param[in] hw_acts
+ * Pointer to translated actions from template.
+ * @param[in] actions
+ * Array of rte_flow action need to be checked.
+ * @param[in] rule_acts
+ * Array of DR rule actions to be used during flow creation..
+ * @param[in] acts_num
+ * Pointer to the real acts_num flow has.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+ const struct rte_flow_action actions[],
+ struct mlx5dr_rule_action *rule_acts,
+ uint32_t *acts_num)
+{
+ bool actions_end = false;
+ uint32_t i;
+
+ for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ rule_acts[i++].action = hw_acts->drop;
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ *acts_num = i;
+ return 0;
+}
+
+/**
+ * Enqueue HW steering flow creation.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow creation status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+flow_hw_q_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = attr->user_data,
+ .burst = attr->postpone,
+ };
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
+ struct mlx5_hw_actions *hw_acts;
+ struct rte_flow_hw *flow;
+ struct mlx5_hw_q_job *job;
+ uint32_t acts_num, flow_idx;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ flow = mlx5_ipool_zmalloc(table->flow, &flow_idx);
+ if (!flow)
+ goto error;
+ /*
+ * Set the table here in order to know the destination table
+ * when free the flow afterwards.
+ */
+ flow->table = table;
+ flow->idx = flow_idx;
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ /*
+ * Set the job type here in order to know if the flow memory
+ * should be freed or not when get the result from dequeue.
+ */
+ job->type = MLX5_HW_Q_JOB_TYPE_CREATE;
+ job->flow = flow;
+ job->user_data = attr->user_data;
+ rule_attr.user_data = job;
+ hw_acts = &table->ats[action_template_index].acts;
+ /* Construct the flow action array based on the input actions.*/
+ flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ ret = mlx5dr_rule_create(table->matcher,
+ pattern_template_index, items,
+ rule_acts, acts_num,
+ &rule_attr, &flow->rule);
+ if (likely(!ret))
+ return (struct rte_flow *)flow;
+ /* Flow created fail, return the descriptor and flow memory. */
+ mlx5_ipool_free(table->flow, flow_idx);
+ priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
+error:
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+ return NULL;
+}
+
+/**
+ * Enqueue HW steering flow destruction.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow destruction status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_q_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow *flow,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = attr->user_data,
+ .burst = attr->postpone,
+ };
+ struct rte_flow_hw *fh = (struct rte_flow_hw *)flow;
+ struct mlx5_hw_q_job *job;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ job->type = MLX5_HW_Q_JOB_TYPE_DESTROY;
+ job->user_data = attr->user_data;
+ job->flow = fh;
+ rule_attr.user_data = job;
+ ret = mlx5dr_rule_destroy(&fh->rule, &rule_attr);
+ if (ret)
+ goto error;
+ return 0;
+error:
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * For flows enqueued from creation/destruction, the status should be
+ * checked from the dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_q_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_q_op_res res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q_job *job;
+ int ret, i;
+
+ ret = mlx5dr_send_queue_poll(priv->dr_ctx, queue, res, n_res);
+ if (ret < 0)
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to query flow queue");
+ for (i = 0; i < ret; i++) {
+ job = (struct mlx5_hw_q_job *)res[i].user_data;
+ /* Restore user data. */
+ res[i].user_data = job->user_data;
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
+ }
+ return ret;
+}
+
+/**
+ * Push the enqueued flows to HW.
+ *
+ * Force apply all the enqueued flows to the HW.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flow.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_q_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error __rte_unused)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ return mlx5dr_send_queue_action(priv->dr_ctx, queue,
+ MLX5DR_SEND_QUEUE_ACTION_DRAIN);
+}
+
/**
* Create flow table.
*
@@ -152,7 +424,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
.data = &flow_attr,
};
struct mlx5_indexed_pool_config cfg = {
- .size = sizeof(struct rte_flow),
+ .size = sizeof(struct rte_flow_hw),
.trunk_size = 1 << 12,
.per_core_cache = 1 << 13,
.need_lock = 1,
@@ -860,6 +1132,10 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.actions_template_destroy = flow_hw_actions_template_destroy,
.template_table_create = flow_hw_table_create,
.template_table_destroy = flow_hw_table_destroy,
+ .q_flow_create = flow_hw_q_flow_create,
+ .q_flow_destroy = flow_hw_q_flow_destroy,
+ .q_pull = flow_hw_q_pull,
+ .q_push = flow_hw_q_push,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 08/13] net/mlx5: add flow flush function
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (6 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 07/13] net/mlx5: add basic flow queue operation Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 09/13] net/mlx5: add flow jump action Suanming Mou
` (7 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
In case port restarting, all created flows should be flushed.
This commit adds the flow flush helper function.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 8 +++
drivers/net/mlx5/mlx5_flow_hw.c | 117 ++++++++++++++++++++++++++++++++
2 files changed, 125 insertions(+)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index b48a3af0fb..9ac96ac979 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -6991,6 +6991,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
uint32_t num_flushed = 0, fidx = 1;
struct rte_flow *flow;
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ if (priv->config.dv_flow_en == 2 &&
+ type == MLX5_FLOW_TYPE_GEN) {
+ flow_hw_q_flow_flush(dev, NULL);
+ return;
+ }
+#endif
+
MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
flow_list_destroy(dev, type, fidx);
num_flushed++;
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index a74825312f..dcf72ab89f 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -377,6 +377,123 @@ flow_hw_q_push(struct rte_eth_dev *dev,
MLX5DR_SEND_QUEUE_ACTION_DRAIN);
}
+/**
+ * Drain the enqueued flows' completion.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the flow.
+ * @param[in] pending_rules
+ * The pending flow number.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+__flow_hw_pull_comp(struct rte_eth_dev *dev,
+ uint32_t queue,
+ uint32_t pending_rules,
+ struct rte_flow_error *error)
+{
+#define BURST_THR 32u
+ struct rte_flow_q_op_res comp[BURST_THR];
+ int ret, i, empty_loop = 0;
+
+ flow_hw_q_push(dev, queue, error);
+ while (pending_rules) {
+ ret = flow_hw_q_pull(dev, 0, comp, BURST_THR, error);
+ if (ret < 0)
+ return -1;
+ if (!ret) {
+ usleep(200);
+ if (++empty_loop > 5) {
+ DRV_LOG(WARNING, "No available dequeue, quit.");
+ break;
+ }
+ continue;
+ }
+ for (i = 0; i < ret; i++) {
+ if (comp[i].status == RTE_FLOW_Q_OP_ERROR)
+ DRV_LOG(WARNING, "Flow flush get error CQE.");
+ }
+ if ((uint32_t)ret > pending_rules) {
+ DRV_LOG(WARNING, "Flow flush get extra CQE.");
+ return -1;
+ }
+ pending_rules -= ret;
+ empty_loop = 0;
+ }
+ return 0;
+}
+
+/**
+ * Flush created flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+int
+flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error)
+{
+#define DEFAULT_QUEUE 0
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q *hw_q;
+ struct rte_flow_template_table *tbl;
+ struct rte_flow_hw *flow;
+ struct rte_flow_q_ops_attr attr = {
+ .postpone = 0,
+ };
+ uint32_t pending_rules = 0;
+ uint32_t queue;
+ uint32_t fidx;
+
+ /*
+ * Ensure to push and dequeue all the enqueued flows in case user
+ * forgot to dequeue. Or the enqueued created flows will be leaked.
+ * The forgot dequeue will also cause flow flush get extra CQEs as
+ * expected and pending_rules be minus value.
+ */
+ for (queue = 0; queue < priv->nb_queue; queue++) {
+ hw_q = &priv->hw_q[queue];
+ if (__flow_hw_pull_comp(dev, queue, hw_q->size - hw_q->job_idx,
+ error))
+ return -1;
+ }
+ /* Flush flow per-table from DEFAULT_QUEUE. */
+ hw_q = &priv->hw_q[DEFAULT_QUEUE];
+ LIST_FOREACH(tbl, &priv->flow_hw_tbl, next) {
+ MLX5_IPOOL_FOREACH(tbl->flow, fidx, flow) {
+ if (flow_hw_q_flow_destroy(dev, DEFAULT_QUEUE, &attr,
+ (struct rte_flow *)flow,
+ error))
+ return -1;
+ pending_rules++;
+ /* Drain completion with queue size. */
+ if (pending_rules >= hw_q->size) {
+ if (__flow_hw_pull_comp(dev, DEFAULT_QUEUE,
+ pending_rules, error))
+ return -1;
+ pending_rules = 0;
+ }
+ }
+ }
+ /* Drain left completion. */
+ if (pending_rules &&
+ __flow_hw_pull_comp(dev, DEFAULT_QUEUE, pending_rules,
+ error))
+ return -1;
+ return 0;
+}
+
/**
* Create flow table.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 09/13] net/mlx5: add flow jump action
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (7 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 08/13] net/mlx5: add flow flush function Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 10/13] net/mlx5: add queue and RSS action Suanming Mou
` (6 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Jump action connects different level of flow tables as a complete data
flow.
A new action construct data struct is also added in this commit to help
handle the dynamic actions.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 25 ++-
drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
3 files changed, 275 insertions(+), 21 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index ec4eb7ee94..0bc9897101 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1525,6 +1525,7 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
};
#define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 40eb8d79aa..a1ab9173d9 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1018,10 +1018,25 @@ struct rte_flow {
/* HWS flow struct. */
struct rte_flow_hw {
uint32_t idx; /* Flow index from indexed pool. */
+ uint32_t fate_type; /* Fate action type. */
+ union {
+ /* Jump action. */
+ struct mlx5_hw_jump_action *jump;
+ };
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
} __rte_packed;
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+ LIST_ENTRY(mlx5_action_construct_data) next;
+ /* Ensure the action types are matched. */
+ int type;
+ uint32_t idx; /* Data index. */
+ uint16_t action_src; /* rte_flow_action src offset. */
+ uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1054,9 +1069,17 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
/* DR action set struct. */
struct mlx5_hw_actions {
- struct mlx5dr_action *drop; /* Drop action. */
+ /* Dynamic action list. */
+ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+ struct mlx5_hw_jump_action *jump; /* Jump action. */
+ uint32_t acts_num:4; /* Total action number. */
+ /* Translated DR action array from action template. */
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
/* mlx5 action template struct. */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index dcf72ab89f..a825766245 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -30,18 +30,158 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Register destination table DR jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the flow attributes.
+ * @param[in] dest_group
+ * The destination group ID.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct mlx5_hw_jump_action *
+flow_hw_jump_action_register(struct rte_eth_dev *dev,
+ const struct rte_flow_attr *attr,
+ uint32_t dest_group,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_attr jattr = *attr;
+ struct mlx5_flow_group *grp;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &jattr,
+ };
+ struct mlx5_list_entry *ge;
+
+ jattr.group = dest_group;
+ ge = mlx5_hlist_register(priv->sh->flow_tbls, dest_group, &ctx);
+ if (!ge)
+ return NULL;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ return &grp->jump;
+}
+
+/**
+ * Release jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] jump
+ * Pointer to the jump action.
+ */
+
+static void
+flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+}
+
/**
* Destroy DR actions created by action template.
*
* For DR actions created during table creation's action translate.
* Need to destroy the DR action when destroying the table.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
* @param[in] acts
* Pointer to the template HW steering DR actions.
*/
static void
-__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+__flow_hw_action_template_destroy(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (acts->jump) {
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (acts->jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+ acts->jump = NULL;
+ }
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline struct mlx5_action_construct_data *
+__flow_hw_act_data_alloc(struct mlx5_priv *priv,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
+{
+ struct mlx5_action_construct_data *act_data;
+ uint32_t idx = 0;
+
+ act_data = mlx5_ipool_zmalloc(priv->acts_ipool, &idx);
+ if (!act_data)
+ return NULL;
+ act_data->idx = idx;
+ act_data->type = type;
+ act_data->action_src = action_src;
+ act_data->action_dst = action_dst;
+ return act_data;
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_general_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
}
/**
@@ -74,14 +214,16 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
const struct rte_flow_template_table_attr *table_attr,
struct mlx5_hw_actions *acts,
struct rte_flow_actions_template *at,
- struct rte_flow_error *error __rte_unused)
+ struct rte_flow_error *error)
{
struct mlx5_priv *priv = dev->data->dev_private;
const struct rte_flow_attr *attr = &table_attr->flow_attr;
struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
bool actions_end = false;
- uint32_t type;
+ uint32_t type, i;
+ int err;
if (attr->transfer)
type = MLX5DR_TABLE_TYPE_FDB;
@@ -89,14 +231,34 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
type = MLX5DR_TABLE_TYPE_NIC_TX;
else
type = MLX5DR_TABLE_TYPE_NIC_RX;
- for (; !actions_end; actions++, masks++) {
+ for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
case RTE_FLOW_ACTION_TYPE_DROP:
- acts->drop = priv->hw_drop[!!attr->group][type];
+ acts->rule_acts[i++].action =
+ priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ if (masks->conf) {
+ uint32_t jump_group =
+ ((const struct rte_flow_action_jump *)
+ actions->conf)->group;
+ acts->jump = flow_hw_jump_action_register
+ (dev, attr, jump_group, error);
+ if (!acts->jump)
+ goto err;
+ acts->rule_acts[i].action = (!!attr->group) ?
+ acts->jump->hws_action :
+ acts->jump->root_action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
@@ -105,7 +267,14 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ acts->acts_num = i;
return 0;
+err:
+ err = rte_errno;
+ __flow_hw_action_template_destroy(dev, acts);
+ return rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
}
/**
@@ -114,6 +283,10 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* For action template contains dynamic actions, these actions need to
* be updated according to the rte_flow action during flow creation.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] job
+ * Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
* @param[in] actions
@@ -127,31 +300,63 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* 0 on success, negative value otherwise and rte_errno is set.
*/
static __rte_always_inline int
-flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+flow_hw_actions_construct(struct rte_eth_dev *dev,
+ struct mlx5_hw_q_job *job,
+ struct mlx5_hw_actions *hw_acts,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
{
- bool actions_end = false;
- uint32_t i;
+ struct rte_flow_template_table *table = job->flow->table;
+ struct mlx5_action_construct_data *act_data;
+ const struct rte_flow_action *action;
+ struct rte_flow_attr attr = {
+ .ingress = 1,
+ };
- for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
- switch (actions->type) {
+ memcpy(rule_acts, hw_acts->rule_acts,
+ sizeof(*rule_acts) * hw_acts->acts_num);
+ *acts_num = hw_acts->acts_num;
+ if (LIST_EMPTY(&hw_acts->act_list))
+ return 0;
+ attr.group = table->grp->group_id;
+ if (table->type == MLX5DR_TABLE_TYPE_FDB) {
+ attr.transfer = 1;
+ attr.ingress = 1;
+ } else if (table->type == MLX5DR_TABLE_TYPE_NIC_TX) {
+ attr.egress = 1;
+ attr.ingress = 0;
+ } else {
+ attr.ingress = 1;
+ }
+ LIST_FOREACH(act_data, &hw_acts->act_list, next) {
+ uint32_t jump_group;
+ struct mlx5_hw_jump_action *jump;
+
+ action = &actions[act_data->action_src];
+ MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
+ (int)action->type == act_data->type);
+ switch (action->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
- case RTE_FLOW_ACTION_TYPE_DROP:
- rule_acts[i++].action = hw_acts->drop;
- break;
- case RTE_FLOW_ACTION_TYPE_END:
- actions_end = true;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ jump_group = ((const struct rte_flow_action_jump *)
+ action->conf)->group;
+ jump = flow_hw_jump_action_register
+ (dev, &attr, jump_group, NULL);
+ if (!jump)
+ return -1;
+ rule_acts[act_data->action_dst].action =
+ (!!attr.group) ? jump->hws_action : jump->root_action;
+ job->flow->jump = jump;
+ job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
default:
break;
}
}
- *acts_num = i;
return 0;
}
@@ -230,7 +435,8 @@ flow_hw_q_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, actions,
+ rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -344,8 +550,11 @@ flow_hw_q_pull(struct rte_eth_dev *dev,
job = (struct mlx5_hw_q_job *)res[i].user_data;
/* Restore user data. */
res[i].user_data = job->user_data;
- if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
+ if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
+ flow_hw_jump_release(dev, job->flow->jump);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ }
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
}
return ret;
@@ -616,6 +825,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
rte_errno = EINVAL;
goto at_error;
}
+ LIST_INIT(&tbl->ats[i].acts.act_list);
err = flow_hw_actions_translate(dev, attr,
&tbl->ats[i].acts,
action_templates[i], error);
@@ -631,7 +841,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
return tbl;
at_error:
while (i--) {
- __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &tbl->ats[i].acts);
__atomic_sub_fetch(&action_templates[i]->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -687,7 +897,7 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
- __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -1106,6 +1316,15 @@ flow_hw_configure(struct rte_eth_dev *dev,
struct mlx5_hw_q *hw_q;
struct mlx5_hw_q_job *job = NULL;
uint32_t mem_size, i, j;
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow_hw),
+ .trunk_size = 4096,
+ .need_lock = 1,
+ .release_mem_en = !!priv->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_action_construct_data",
+ };
if (!port_attr || !nb_queue || !queue_attr) {
rte_errno = EINVAL;
@@ -1124,6 +1343,9 @@ flow_hw_configure(struct rte_eth_dev *dev,
}
flow_hw_resource_release(dev);
}
+ priv->acts_ipool = mlx5_ipool_create(&cfg);
+ if (!priv->acts_ipool)
+ goto err;
/* Allocate the queue job descriptor LIFO. */
mem_size = sizeof(priv->hw_q[0]) * nb_queue;
for (i = 0; i < nb_queue; i++) {
@@ -1193,6 +1415,10 @@ flow_hw_configure(struct rte_eth_dev *dev,
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
}
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"fail to configure port");
@@ -1234,6 +1460,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
}
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 10/13] net/mlx5: add queue and RSS action
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (8 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 09/13] net/mlx5: add flow jump action Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 11/13] net/mlx5: add mark action Suanming Mou
` (5 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
This commit adds the queue and RSS action. Similar to the jump action,
dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed
during port restart, the actions are created with standalone indirect
table as indirect action does. When port stops, detaches the indirect
table from action, when port starts, attaches the indirect table back
to the action.
One more change is made to accelerate the action creation. Currently
the mlx5_hrxq_get() function returns the object index instead of object
pointer. This introduced an extra converting the index to the object by
calling mlx5_ipool_get() in most of the case. And that extra converting
hurts multi-thread performance since mlx5_ipool_get() uses the global
lock inside. As the hash Rx queue object itself also contains the index,
returns the object directly will achieve better performance without the
global lock.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 18 ++--
drivers/net/mlx5/mlx5.h | 4 +
drivers/net/mlx5/mlx5_devx.c | 10 ++
drivers/net/mlx5/mlx5_flow.c | 38 +++-----
drivers/net/mlx5/mlx5_flow.h | 7 ++
drivers/net/mlx5/mlx5_flow_dv.c | 150 ++++++++++++++---------------
drivers/net/mlx5/mlx5_flow_hw.c | 101 +++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 78 +++++++++------
10 files changed, 271 insertions(+), 151 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 52e52a4ad7..8f0b15aad0 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1714,6 +1714,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+ mlx5_hrxq_create_cb,
+ mlx5_hrxq_match_cb,
+ mlx5_hrxq_remove_cb,
+ mlx5_hrxq_clone_cb,
+ mlx5_hrxq_clone_free_cb);
+ if (!priv->hrxqs)
+ goto error;
+ rte_rwlock_init(&priv->ind_tbls_lock);
if (priv->config.dv_flow_en == 2)
return eth_dev;
/* Port representor shares the same max priority with pf port. */
@@ -1744,15 +1753,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = ENOTSUP;
goto error;
}
- priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
- mlx5_hrxq_create_cb,
- mlx5_hrxq_match_cb,
- mlx5_hrxq_remove_cb,
- mlx5_hrxq_clone_cb,
- mlx5_hrxq_clone_free_cb);
- if (!priv->hrxqs)
- goto error;
- rte_rwlock_init(&priv->ind_tbls_lock);
/* Query availability of metadata reg_c's. */
if (!priv->sh->metadata_regc_check_flag) {
err = mlx5_flow_discover_mreg_c(eth_dev);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 0bc9897101..6fb82bf1f3 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1286,6 +1286,7 @@ struct mlx5_flow_rss_desc {
uint64_t hash_fields; /* Verbs Hash fields. */
uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
uint32_t key_len; /**< RSS hash key len. */
+ uint32_t hws_flags; /**< HW steering action. */
uint32_t tunnel; /**< Queue in tunnel. */
uint32_t shared_rss; /**< Shared RSS index. */
struct mlx5_ind_table_obj *ind_tbl;
@@ -1347,6 +1348,7 @@ struct mlx5_hrxq {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
void *action; /* DV QP action pointer. */
#endif
+ uint32_t hws_flags; /* Hw steering flags. */
uint64_t hash_fields; /* Verbs Hash fields. */
uint32_t rss_key_len; /* Hash key length in bytes. */
uint32_t idx; /* Hash Rx queue index. */
@@ -1477,6 +1479,8 @@ struct mlx5_priv {
LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
/* Indirection tables. */
LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+ /* Standalone indirect tables. */
+ LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
/* Pointer to next element. */
rte_rwlock_t ind_tbls_lock;
uint32_t refcnt; /**< Reference counter. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 91243f684f..af131bcd1b 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
goto error;
}
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ if (hrxq->hws_flags) {
+ hrxq->action = mlx5dr_action_create_dest_tir
+ (priv->dr_ctx,
+ (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+ if (!hrxq->action)
+ goto error;
+ return 0;
+ }
if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
&hrxq->action)) {
rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
DRV_LOG(ERR, "Cannot create drop RX queue");
return ret;
}
+ if (priv->config.dv_flow_en == 2)
+ return 0;
/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
if (ret != 0) {
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 9ac96ac979..9cad84ebc6 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -9302,14 +9302,10 @@ int
mlx5_action_handle_attach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
const char *message;
uint32_t queue_idx;
@@ -9325,9 +9321,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
}
if (ret != 0)
return ret;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not attach "
@@ -9336,13 +9330,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
goto error;
}
}
+
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not detach "
@@ -9365,15 +9358,10 @@ int
mlx5_action_handle_detach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
-
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not detach "
@@ -9384,11 +9372,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
}
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not attach "
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index a1ab9173d9..33094c8c07 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1022,6 +1022,7 @@ struct rte_flow_hw {
union {
/* Jump action. */
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq; /* TIR action. */
};
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1077,6 +1078,7 @@ struct mlx5_hw_actions {
/* Dynamic action list. */
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
+ struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1907,6 +1909,11 @@ int flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
int
flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
struct rte_flow_error *error);
+void flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index ef9c66eddf..c3d9d30dba 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -10966,78 +10966,83 @@ flow_dv_translate_item_tx_queue(struct rte_eth_dev *dev,
/**
* Set the hash fields according to the @p flow information.
*
- * @param[in] dev_flow
- * Pointer to the mlx5_flow.
+ * @param[in] item_flags
+ * The match pattern item flags.
* @param[in] rss_desc
* Pointer to the mlx5_flow_rss_desc.
+ * @param[out] hash_fields
+ * Pointer to the RSS hash fields.
*/
-static void
-flow_dv_hashfields_set(struct mlx5_flow *dev_flow,
- struct mlx5_flow_rss_desc *rss_desc)
+void
+flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields)
{
- uint64_t items = dev_flow->handle->layers;
+ uint64_t items = item_flags;
+ uint64_t fields = 0;
int rss_inner = 0;
uint64_t rss_types = rte_eth_rss_hf_refine(rss_desc->types);
- dev_flow->hash_fields = 0;
+ *hash_fields = 0;
#ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT
if (rss_desc->level >= 2)
rss_inner = 1;
#endif
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV4)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4)) ||
+ !items) {
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV4;
+ fields |= IBV_RX_HASH_SRC_IPV4;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV4;
+ fields |= IBV_RX_HASH_DST_IPV4;
else
- dev_flow->hash_fields |= MLX5_IPV4_IBV_RX_HASH;
+ fields |= MLX5_IPV4_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV6)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6)) ||
+ !items) {
if (rss_types & MLX5_IPV6_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV6;
+ fields |= IBV_RX_HASH_SRC_IPV6;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV6;
+ fields |= IBV_RX_HASH_DST_IPV6;
else
- dev_flow->hash_fields |= MLX5_IPV6_IBV_RX_HASH;
+ fields |= MLX5_IPV6_IBV_RX_HASH;
}
}
- if (dev_flow->hash_fields == 0)
+ if (fields == 0)
/*
* There is no match between the RSS types and the
* L3 protocol (IPv4/IPv6) defined in the flow rule.
*/
return;
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_UDP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_UDP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_UDP;
+ fields |= IBV_RX_HASH_SRC_PORT_UDP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_UDP;
+ fields |= IBV_RX_HASH_DST_PORT_UDP;
else
- dev_flow->hash_fields |= MLX5_UDP_IBV_RX_HASH;
+ fields |= MLX5_UDP_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_TCP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_TCP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_TCP;
+ fields |= IBV_RX_HASH_SRC_PORT_TCP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_TCP;
+ fields |= IBV_RX_HASH_DST_PORT_TCP;
else
- dev_flow->hash_fields |= MLX5_TCP_IBV_RX_HASH;
+ fields |= MLX5_TCP_IBV_RX_HASH;
}
}
if (rss_inner)
- dev_flow->hash_fields |= IBV_RX_HASH_INNER;
+ fields |= IBV_RX_HASH_INNER;
+ *hash_fields = fields;
}
/**
@@ -11061,7 +11066,6 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc,
uint32_t *hrxq_idx)
{
- struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_handle *dh = dev_flow->handle;
struct mlx5_hrxq *hrxq;
@@ -11072,11 +11076,8 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
rss_desc->shared_rss = 0;
if (rss_desc->hash_fields == 0)
rss_desc->queue_num = 1;
- *hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- if (!*hrxq_idx)
- return NULL;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- *hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
+ *hrxq_idx = hrxq ? hrxq->idx : 0;
return hrxq;
}
@@ -11622,7 +11623,9 @@ flow_dv_translate_action_sample(struct rte_eth_dev *dev,
* rss->level and rss.types should be set in advance
* when expanding items for RSS.
*/
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
hrxq = flow_dv_hrxq_prepare(dev, dev_flow,
rss_desc, &hrxq_idx);
if (!hrxq)
@@ -13647,7 +13650,9 @@ flow_dv_translate(struct rte_eth_dev *dev,
*/
handle->layers |= item_flags;
if (action_flags & MLX5_FLOW_ACTION_RSS)
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
/* If has RSS action in the sample action, the Sample/Mirror resource
* should be registered after the hash filed be update.
*/
@@ -14596,20 +14601,18 @@ __flow_dv_action_rss_hrxqs_release(struct rte_eth_dev *dev,
* MLX5_RSS_HASH_IPV4_DST_ONLY are mutually exclusive so they can share
* same slot in mlx5_rss_hash_fields.
*
- * @param[in] rss
- * Pointer to the shared action RSS conf.
+ * @param[in] rss_types
+ * RSS type.
* @param[in, out] hash_field
* hash_field variable needed to be adjusted.
*
* @return
* void
*/
-static void
-__flow_dv_action_rss_l34_hash_adjust(struct mlx5_shared_action_rss *rss,
- uint64_t *hash_field)
+void
+flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field)
{
- uint64_t rss_types = rss->origin.types;
-
switch (*hash_field & ~IBV_RX_HASH_INNER) {
case MLX5_RSS_HASH_IPV4:
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
@@ -14692,12 +14695,15 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
size_t i;
int err;
- if (mlx5_ind_table_obj_setup(dev, shared_rss->ind_tbl,
- !!dev->data->dev_started)) {
+ shared_rss->ind_tbl = mlx5_ind_table_obj_new
+ (dev, shared_rss->origin.queue,
+ shared_rss->origin.queue_num,
+ true,
+ !!dev->data->dev_started);
+ if (!shared_rss->ind_tbl)
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot setup indirection table");
- }
memcpy(rss_desc.key, shared_rss->origin.key, MLX5_RSS_HASH_KEY_LEN);
rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
rss_desc.const_q = shared_rss->origin.queue;
@@ -14706,19 +14712,20 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
- uint32_t hrxq_idx;
+ struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
int tunnel = 0;
- __flow_dv_action_rss_l34_hash_adjust(shared_rss, &hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(shared_rss->origin.types,
+ &hash_fields);
if (shared_rss->origin.level > 1) {
hash_fields |= IBV_RX_HASH_INNER;
tunnel = 1;
}
rss_desc.tunnel = tunnel;
rss_desc.hash_fields = hash_fields;
- hrxq_idx = mlx5_hrxq_get(dev, &rss_desc);
- if (!hrxq_idx) {
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
@@ -14726,14 +14733,14 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
goto error_hrxq_new;
}
err = __flow_dv_action_rss_hrxq_set
- (shared_rss, hash_fields, hrxq_idx);
+ (shared_rss, hash_fields, hrxq->idx);
MLX5_ASSERT(!err);
}
return 0;
error_hrxq_new:
err = rte_errno;
__flow_dv_action_rss_hrxqs_release(dev, shared_rss);
- if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true, true))
+ if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true))
shared_rss->ind_tbl = NULL;
rte_errno = err;
return -rte_errno;
@@ -14764,18 +14771,14 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss = NULL;
- void *queue = NULL;
struct rte_flow_action_rss *origin;
const uint8_t *rss_key;
- uint32_t queue_size = rss->queue_num * sizeof(uint16_t);
uint32_t idx;
RTE_SET_USED(conf);
- queue = mlx5_malloc(0, RTE_ALIGN_CEIL(queue_size, sizeof(void *)),
- 0, SOCKET_ID_ANY);
shared_rss = mlx5_ipool_zmalloc
(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], &idx);
- if (!shared_rss || !queue) {
+ if (!shared_rss) {
rte_flow_error_set(error, ENOMEM,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot allocate resource memory");
@@ -14787,18 +14790,6 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
"rss action number out of range");
goto error_rss_init;
}
- shared_rss->ind_tbl = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(*shared_rss->ind_tbl),
- 0, SOCKET_ID_ANY);
- if (!shared_rss->ind_tbl) {
- rte_flow_error_set(error, ENOMEM,
- RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
- "cannot allocate resource memory");
- goto error_rss_init;
- }
- memcpy(queue, rss->queue, queue_size);
- shared_rss->ind_tbl->queues = queue;
- shared_rss->ind_tbl->queues_n = rss->queue_num;
origin = &shared_rss->origin;
origin->func = rss->func;
origin->level = rss->level;
@@ -14809,10 +14800,12 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
memcpy(shared_rss->key, rss_key, MLX5_RSS_HASH_KEY_LEN);
origin->key = &shared_rss->key[0];
origin->key_len = MLX5_RSS_HASH_KEY_LEN;
- origin->queue = queue;
+ origin->queue = rss->queue;
origin->queue_num = rss->queue_num;
if (__flow_dv_action_rss_setup(dev, idx, shared_rss, error))
goto error_rss_init;
+ /* Update queue with indirect table queue memoyr. */
+ origin->queue = shared_rss->ind_tbl->queues;
rte_spinlock_init(&shared_rss->action_rss_sl);
__atomic_add_fetch(&shared_rss->refcnt, 1, __ATOMIC_RELAXED);
rte_spinlock_lock(&priv->shared_act_sl);
@@ -14823,12 +14816,11 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
error_rss_init:
if (shared_rss) {
if (shared_rss->ind_tbl)
- mlx5_free(shared_rss->ind_tbl);
+ mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
+ !!dev->data->dev_started);
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
idx);
}
- if (queue)
- mlx5_free(queue);
return 0;
}
@@ -14856,7 +14848,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
uint32_t old_refcnt = 1;
int remaining;
- uint16_t *queue = NULL;
if (!shared_rss)
return rte_flow_error_set(error, EINVAL,
@@ -14875,8 +14866,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
RTE_FLOW_ERROR_TYPE_ACTION,
NULL,
"shared rss hrxq has references");
- queue = shared_rss->ind_tbl->queues;
- remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true,
+ remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
!!dev->data->dev_started);
if (remaining)
return rte_flow_error_set(error, EBUSY,
@@ -14884,7 +14874,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
NULL,
"shared rss indirection table has"
" references");
- mlx5_free(queue);
rte_spinlock_lock(&priv->shared_act_sl);
ILIST_REMOVE(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
&priv->rss_shared_actions, idx, shared_rss, next);
@@ -16878,11 +16867,12 @@ __flow_dv_meter_get_rss_sub_policy(struct rte_eth_dev *dev,
for (i = 0; i < MLX5_MTR_RTE_COLORS; i++) {
if (!rss_desc[i])
continue;
- hrxq_idx[i] = mlx5_hrxq_get(dev, rss_desc[i]);
- if (!hrxq_idx[i]) {
+ hrxq = mlx5_hrxq_get(dev, rss_desc[i]);
+ if (!hrxq) {
rte_spinlock_unlock(&mtr_policy->sl);
return NULL;
}
+ hrxq_idx[i] = hrxq->idx;
}
sub_policy_num = (mtr_policy->sub_policy_num >>
(MLX5_MTR_SUB_POLICY_NUM_SHIFT * domain)) &
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index a825766245..e59d812072 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -7,6 +7,7 @@
#include <mlx5_malloc.h>
#include "mlx5_defs.h"
#include "mlx5_flow.h"
+#include "mlx5_rx.h"
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
@@ -89,6 +90,56 @@ flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
}
+/**
+ * Register queue/RSS action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] hws_flags
+ * DR action flags.
+ * @param[in] action
+ * rte flow action.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static inline struct mlx5_hrxq*
+flow_hw_tir_action_register(struct rte_eth_dev *dev,
+ uint32_t hws_flags,
+ const struct rte_flow_action *action)
+{
+ struct mlx5_flow_rss_desc rss_desc = {
+ .hws_flags = hws_flags,
+ };
+ struct mlx5_hrxq *hrxq;
+
+ if (action->type == RTE_FLOW_ACTION_TYPE_QUEUE) {
+ const struct rte_flow_action_queue *queue = action->conf;
+
+ rss_desc.const_q = &queue->index;
+ rss_desc.queue_num = 1;
+ } else {
+ const struct rte_flow_action_rss *rss = action->conf;
+
+ rss_desc.queue_num = rss->queue_num;
+ rss_desc.const_q = rss->queue;
+ memcpy(rss_desc.key,
+ !rss->key ? rss_hash_default_key : rss->key,
+ MLX5_RSS_HASH_KEY_LEN);
+ rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
+ rss_desc.types = !rss->types ? RTE_ETH_RSS_IP : rss->types;
+ flow_dv_hashfields_set(0, &rss_desc, &rss_desc.hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(rss->types,
+ &rss_desc.hash_fields);
+ if (rss->level > 1) {
+ rss_desc.hash_fields |= IBV_RX_HASH_INNER;
+ rss_desc.tunnel = 1;
+ }
+ }
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ return hrxq;
+}
+
/**
* Destroy DR actions created by action template.
*
@@ -260,6 +311,40 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -313,6 +398,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_attr attr = {
.ingress = 1,
};
+ uint32_t ft_flag;
memcpy(rule_acts, hw_acts->rule_acts,
sizeof(*rule_acts) * hw_acts->acts_num);
@@ -320,6 +406,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
if (LIST_EMPTY(&hw_acts->act_list))
return 0;
attr.group = table->grp->group_id;
+ ft_flag = mlx5_hw_act_flag[!!table->grp->group_id][table->type];
if (table->type == MLX5DR_TABLE_TYPE_FDB) {
attr.transfer = 1;
attr.ingress = 1;
@@ -332,6 +419,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
@@ -353,6 +441,17 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->jump = jump;
job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ hrxq = flow_hw_tir_action_register(dev,
+ ft_flag,
+ action);
+ if (!hrxq)
+ return -1;
+ rule_acts[act_data->action_dst].action = hrxq->action;
+ job->flow->hrxq = hrxq;
+ job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
+ break;
default:
break;
}
@@ -553,6 +652,8 @@ flow_hw_q_pull(struct rte_eth_dev *dev,
if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
flow_hw_jump_release(dev, job->flow->jump);
+ else if (job->flow->fate_type == MLX5_FLOW_FATE_QUEUE)
+ mlx5_hrxq_obj_release(dev, job->flow->hrxq);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
}
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c
index 90ccb9aaff..f08aa7a770 100644
--- a/drivers/net/mlx5/mlx5_flow_verbs.c
+++ b/drivers/net/mlx5/mlx5_flow_verbs.c
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
MLX5_ASSERT(priv->drop_queue.hrxq);
hrxq = priv->drop_queue.hrxq;
} else {
- uint32_t hrxq_idx;
struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
rss_desc->tunnel = !!(handle->layers &
MLX5_FLOW_LAYER_TUNNEL);
rss_desc->shared_rss = 0;
- hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
"cannot get hash queue");
goto error;
}
- handle->rix_hrxq = hrxq_idx;
+ handle->rix_hrxq = hrxq->idx;
}
MLX5_ASSERT(hrxq);
handle->drv_flow = mlx5_glue->create_flow
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index cb5d51340d..468772ee27 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -225,9 +225,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
const uint16_t *queues,
uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+ const uint16_t *queues,
+ uint32_t queues_n,
+ bool standalone,
+ bool ref_qs);
int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs);
int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
@@ -250,8 +254,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
void *cb_ctx __rte_unused);
void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 580d7ae868..a892675646 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -2284,8 +2284,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
* Pointer to Ethernet device.
* @param ind_table
* Indirection table to release.
- * @param standalone
- * Indirection table for Standalone queue.
* @param deref_rxqs
* If true, then dereference RX queues related to indirection table.
* Otherwise, no additional action will be taken.
@@ -2296,7 +2294,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
int
mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs)
{
struct mlx5_priv *priv = dev->data->dev_private;
@@ -2304,7 +2301,7 @@ mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
rte_rwlock_write_lock(&priv->ind_tbls_lock);
ret = __atomic_sub_fetch(&ind_tbl->refcnt, 1, __ATOMIC_RELAXED);
- if (!ret && !standalone)
+ if (!ret)
LIST_REMOVE(ind_tbl, next);
rte_rwlock_write_unlock(&priv->ind_tbls_lock);
if (ret)
@@ -2413,7 +2410,7 @@ mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
* @return
* The Verbs/DevX object initialized, NULL otherwise and rte_errno is set.
*/
-static struct mlx5_ind_table_obj *
+struct mlx5_ind_table_obj *
mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
uint32_t queues_n, bool standalone, bool ref_qs)
{
@@ -2435,11 +2432,13 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
mlx5_free(ind_tbl);
return NULL;
}
- if (!standalone) {
- rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ if (!standalone)
LIST_INSERT_HEAD(&priv->ind_tbls, ind_tbl, next);
- rte_rwlock_write_unlock(&priv->ind_tbls_lock);
- }
+ else
+ LIST_INSERT_HEAD(&priv->standalone_ind_tbls, ind_tbl, next);
+ rte_rwlock_write_unlock(&priv->ind_tbls_lock);
+
return ind_tbl;
}
@@ -2605,6 +2604,7 @@ mlx5_hrxq_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
return (hrxq->rss_key_len != rss_desc->key_len ||
memcmp(hrxq->rss_key, rss_desc->key, rss_desc->key_len) ||
+ hrxq->hws_flags != rss_desc->hws_flags ||
hrxq->hash_fields != rss_desc->hash_fields ||
hrxq->ind_table->queues_n != rss_desc->queue_num ||
memcmp(hrxq->ind_table->queues, rss_desc->queue,
@@ -2689,8 +2689,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
}
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ mlx5_ind_table_obj_release(dev, hrxq->ind_table, true);
hrxq->ind_table = ind_tbl;
}
hrxq->hash_fields = hash_fields;
@@ -2700,8 +2699,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
err = rte_errno;
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, ind_tbl, hrxq->standalone,
- true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
}
rte_errno = err;
return -rte_errno;
@@ -2713,12 +2711,16 @@ __mlx5_hrxq_remove(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
struct mlx5_priv *priv = dev->data->dev_private;
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- mlx5_glue->destroy_flow_action(hrxq->action);
+ if (hrxq->hws_flags)
+ mlx5dr_action_destroy(hrxq->action);
+ else
+ mlx5_glue->destroy_flow_action(hrxq->action);
#endif
priv->obj_ops.hrxq_destroy(hrxq);
if (!hrxq->standalone) {
mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ hrxq->hws_flags ?
+ (!!dev->data->dev_started) : true);
}
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq->idx);
}
@@ -2762,11 +2764,12 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
int ret;
queues_n = rss_desc->hash_fields ? queues_n : 1;
- if (!ind_tbl)
+ if (!ind_tbl && !rss_desc->hws_flags)
ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n);
if (!ind_tbl)
ind_tbl = mlx5_ind_table_obj_new(dev, queues, queues_n,
- standalone,
+ standalone ||
+ rss_desc->hws_flags,
!!dev->data->dev_started);
if (!ind_tbl)
return NULL;
@@ -2778,6 +2781,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
hrxq->ind_table = ind_tbl;
hrxq->rss_key_len = rss_key_len;
hrxq->hash_fields = rss_desc->hash_fields;
+ hrxq->hws_flags = rss_desc->hws_flags;
memcpy(hrxq->rss_key, rss_key, rss_key_len);
ret = priv->obj_ops.hrxq_new(dev, hrxq, rss_desc->tunnel);
if (ret < 0)
@@ -2785,7 +2789,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
return hrxq;
error:
if (!rss_desc->ind_tbl)
- mlx5_ind_table_obj_release(dev, ind_tbl, standalone, true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
if (hrxq)
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
return NULL;
@@ -2839,13 +2843,13 @@ mlx5_hrxq_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
* RSS configuration for the Rx hash queue.
*
* @return
- * An hash Rx queue index on success.
+ * An hash Rx queue on success.
*/
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
+ struct mlx5_hrxq *hrxq = NULL;
struct mlx5_list_entry *entry;
struct mlx5_flow_cb_ctx ctx = {
.data = rss_desc,
@@ -2856,12 +2860,10 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
} else {
entry = mlx5_list_register(priv->hrxqs, &ctx);
if (!entry)
- return 0;
+ return NULL;
hrxq = container_of(entry, typeof(*hrxq), entry);
}
- if (hrxq)
- return hrxq->idx;
- return 0;
+ return hrxq;
}
/**
@@ -2870,17 +2872,15 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
* @param dev
* Pointer to Ethernet device.
* @param hrxq_idx
- * Index to Hash Rx queue to release.
+ * Hash Rx queue to release.
*
* @return
* 1 while a reference on it exists, 0 when freed.
*/
-int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
if (!hrxq)
return 0;
if (!hrxq->standalone)
@@ -2889,6 +2889,26 @@ int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
return 0;
}
+/**
+ * Release the hash Rx queue with index.
+ *
+ * @param dev
+ * Pointer to Ethernet device.
+ * @param hrxq_idx
+ * Index to Hash Rx queue to release.
+ *
+ * @return
+ * 1 while a reference on it exists, 0 when freed.
+ */
+int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hrxq *hrxq;
+
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
+ return mlx5_hrxq_obj_release(dev, hrxq);
+}
+
/**
* Create a drop Rx Hash queue.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 11/13] net/mlx5: add mark action
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (9 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 10/13] net/mlx5: add queue and RSS action Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 12/13] net/mlx5: add indirect action Suanming Mou
` (4 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The mark action is covered by tag action internally. While it is added
the HW will add a tag to the packet. The mark value can be set as fixed
or dynamic as the action mask indicates.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 3 ++
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 87 ++++++++++++++++++++++++++++++---
3 files changed, 85 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 6fb82bf1f3..c78dc3c431 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1529,6 +1529,9 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX];
struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
};
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 33094c8c07..8e65486a1f 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1080,6 +1080,7 @@ struct mlx5_hw_actions {
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
+ uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index e59d812072..a754cdd084 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -31,6 +31,50 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Trim rxq flag refcnt.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ */
+static void
+flow_hw_rxq_flag_trim(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ unsigned int i;
+
+ if (!priv->mark_enabled)
+ return;
+ for (i = 0; i < priv->rxqs_n; ++i) {
+ struct mlx5_rxq_ctrl *rxq_ctrl = mlx5_rxq_ctrl_get(dev, i);
+
+ rxq_ctrl->rxq.mark = 0;
+ }
+ priv->mark_enabled = 0;
+}
+
+/**
+ * Set rxq flag refcnt.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ */
+static void
+flow_hw_rxq_flag_set(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ unsigned int i;
+
+ if (priv->mark_enabled)
+ return;
+ for (i = 0; i < priv->rxqs_n; ++i) {
+ struct mlx5_rxq_ctrl *rxq_ctrl = mlx5_rxq_ctrl_get(dev, i);
+
+ rxq_ctrl->rxq.mark = 1;
+ }
+ priv->mark_enabled = 1;
+}
+
/**
* Register destination table DR jump action.
*
@@ -292,6 +336,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
acts->rule_acts[i++].action =
priv->hw_drop[!!attr->group][type];
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ acts->mark = true;
+ if (masks->conf)
+ acts->rule_acts[i].tag.value =
+ mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (masks->conf))->id);
+ else if (__flow_hw_act_data_general_append(priv, acts,
+ actions->type, actions - action_start, i))
+ goto err;
+ acts->rule_acts[i++].action =
+ priv->hw_tag[!!attr->group][type];
+ flow_hw_rxq_flag_set(dev);
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
if (masks->conf) {
uint32_t jump_group =
@@ -418,6 +476,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
}
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
+ uint32_t tag;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
@@ -429,6 +488,12 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ tag = mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (action->conf))->id);
+ rule_acts[act_data->action_dst].tag.value = tag;
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
jump_group = ((const struct rte_flow_action_jump *)
action->conf)->group;
@@ -998,6 +1063,8 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
+ if (table->ats[i].acts.mark)
+ flow_hw_rxq_flag_trim(dev);
__flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
@@ -1499,15 +1566,21 @@ flow_hw_configure(struct rte_eth_dev *dev,
(priv->dr_ctx, mlx5_hw_act_flag[i][j]);
if (!priv->hw_drop[i][j])
goto err;
+ priv->hw_tag[i][j] = mlx5dr_action_create_tag
+ (priv->dr_ctx, mlx5_hw_act_flag[i][j]);
+ if (!priv->hw_tag[i][j])
+ goto err;
}
}
return 0;
err:
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_tag[i][j])
+ mlx5dr_action_destroy(priv->hw_tag[i][j]);
+
}
}
if (dr_ctx)
@@ -1556,9 +1629,11 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
}
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_tag[i][j])
+ mlx5dr_action_destroy(priv->hw_tag[i][j]);
+
}
}
if (priv->acts_ipool) {
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 12/13] net/mlx5: add indirect action
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (10 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 11/13] net/mlx5: add mark action Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-10 16:29 ` [PATCH 13/13] net/mlx5: add header reformat action Suanming Mou
` (3 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering can support indirect action as well. With indirect action,
the flow can be created with more flexible shared RSS action selection.
This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:
rte_flow_q_action_handle_create();
rte_flow_q_action_handle_destroy();
rte_flow_q_action_handle_update();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 116 +++++++++
drivers/net/mlx5/mlx5_flow.h | 56 +++++
drivers/net/mlx5/mlx5_flow_dv.c | 21 +-
drivers/net/mlx5/mlx5_flow_hw.c | 402 +++++++++++++++++++++++++++++++-
4 files changed, 582 insertions(+), 13 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 9cad84ebc6..46950044e0 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -873,6 +873,26 @@ mlx5_flow_q_push(struct rte_eth_dev *dev,
uint32_t queue,
struct rte_flow_error *error);
+static struct rte_flow_action_handle *
+mlx5_flow_q_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_q_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_q_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -904,6 +924,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.q_flow_destroy = mlx5_flow_q_flow_destroy,
.q_pull = mlx5_flow_q_pull,
.q_push = mlx5_flow_q_push,
+ .q_action_handle_create = mlx5_flow_q_action_handle_create,
+ .q_action_handle_update = mlx5_flow_q_action_handle_update,
+ .q_action_handle_destroy = mlx5_flow_q_action_handle_destroy,
};
/* Tunnel information. */
@@ -8228,6 +8251,99 @@ mlx5_flow_q_push(struct rte_eth_dev *dev,
return fops->q_push(dev, queue, error);
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_q_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->q_action_create(dev, queue, attr, conf, action, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_q_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->q_action_update(dev, queue, attr, handle, update, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_q_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->q_action_destroy(dev, queue, attr, handle, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 8e65486a1f..097e5bf587 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -41,6 +41,7 @@ enum mlx5_rte_flow_action_type {
MLX5_RTE_FLOW_ACTION_TYPE_AGE,
MLX5_RTE_FLOW_ACTION_TYPE_COUNT,
MLX5_RTE_FLOW_ACTION_TYPE_JUMP,
+ MLX5_RTE_FLOW_ACTION_TYPE_RSS,
};
#define MLX5_INDIRECT_ACTION_TYPE_OFFSET 30
@@ -1036,6 +1037,13 @@ struct mlx5_action_construct_data {
uint32_t idx; /* Data index. */
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+ union {
+ struct {
+ uint64_t types; /* RSS hash types. */
+ uint32_t level; /* RSS level. */
+ uint32_t idx; /* Shared action index. */
+ } shared_rss;
+ };
};
/* Flow item template struct. */
@@ -1044,6 +1052,7 @@ struct rte_flow_pattern_template {
/* Template attributes. */
struct rte_flow_pattern_template_attr attr;
struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint64_t item_flags; /* Item layer flags. */
uint32_t refcnt; /* Reference counter. */
};
@@ -1426,6 +1435,29 @@ typedef int (*mlx5_flow_q_push_t)
uint32_t queue,
struct rte_flow_error *error);
+typedef struct rte_flow_action_handle *(*mlx5_flow_q_action_handle_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_q_action_handle_update_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_q_action_handle_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error);
+
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
mlx5_flow_prepare_t prepare;
@@ -1474,6 +1506,9 @@ struct mlx5_flow_driver_ops {
mlx5_flow_q_flow_destroy_t q_flow_destroy;
mlx5_flow_q_pull_t q_pull;
mlx5_flow_q_push_t q_push;
+ mlx5_flow_q_action_handle_create_t q_action_create;
+ mlx5_flow_q_action_handle_update_t q_action_update;
+ mlx5_flow_q_action_handle_destroy_t q_action_destroy;
};
/* mlx5_flow.c */
@@ -1915,6 +1950,8 @@ void flow_dv_hashfields_set(uint64_t item_flags,
uint64_t *hash_fields);
void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
uint64_t *hash_field);
+uint32_t flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
@@ -1965,4 +2002,23 @@ mlx5_get_tof(const struct rte_flow_item *items,
enum mlx5_tof_rule_type *rule_type);
void
flow_hw_resource_release(struct rte_eth_dev *dev);
+int flow_dv_action_validate(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+struct rte_flow_action_handle *flow_dv_action_create(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+int flow_dv_action_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error);
+int flow_dv_action_update(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *err);
+int flow_dv_action_query(struct rte_eth_dev *dev,
+ const struct rte_flow_action_handle *handle,
+ void *data,
+ struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index c3d9d30dba..ca8ae4214b 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -13837,9 +13837,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
* @return
* Valid hash RX queue index, otherwise 0.
*/
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
- const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss =
@@ -13967,7 +13967,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
struct mlx5_hrxq *hrxq = NULL;
uint32_t hrxq_idx;
- hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
rss_desc->shared_rss,
dev_flow->hash_fields);
if (hrxq_idx)
@@ -14691,6 +14691,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
struct mlx5_shared_action_rss *shared_rss,
struct rte_flow_error *error)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_rss_desc rss_desc = { 0 };
size_t i;
int err;
@@ -14711,6 +14712,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
/* Set non-zero value to indicate a shared RSS. */
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
+ if (priv->config.dv_flow_en == 2)
+ rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14902,7 +14905,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
* A valid shared action handle in case of success, NULL otherwise and
* rte_errno is set.
*/
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
flow_dv_action_create(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
@@ -14972,7 +14975,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_destroy(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
struct rte_flow_error *error)
@@ -15181,7 +15184,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_update(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
const void *update,
@@ -15895,7 +15898,7 @@ flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
"counters are not available");
}
-static int
+int
flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle, void *data,
struct rte_flow_error *error)
@@ -17584,7 +17587,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_validate(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index a754cdd084..9fc6f24542 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -75,6 +75,72 @@ flow_hw_rxq_flag_set(struct rte_eth_dev *dev)
priv->mark_enabled = 1;
}
+/**
+ * Generate the pattern item flags.
+ * Will be used for shared RSS action.
+ *
+ * @param[in] items
+ * Pointer to the list of items.
+ *
+ * @return
+ * Item flags.
+ */
+static uint64_t
+flow_hw_rss_item_flags_get(const struct rte_flow_item items[])
+{
+ uint64_t item_flags = 0;
+ uint64_t last_item = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) {
+ int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL);
+ int item_type = items->type;
+
+ switch (item_type) {
+ case RTE_FLOW_ITEM_TYPE_IPV4:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV4;
+ break;
+ case RTE_FLOW_ITEM_TYPE_IPV6:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV6;
+ break;
+ case RTE_FLOW_ITEM_TYPE_TCP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP :
+ MLX5_FLOW_LAYER_OUTER_L4_TCP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_UDP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP :
+ MLX5_FLOW_LAYER_OUTER_L4_UDP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_NVGRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN:
+ last_item = MLX5_FLOW_LAYER_VXLAN;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN_GPE:
+ last_item = MLX5_FLOW_LAYER_VXLAN_GPE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GENEVE:
+ last_item = MLX5_FLOW_LAYER_GENEVE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_MPLS:
+ last_item = MLX5_FLOW_LAYER_MPLS;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GTP:
+ last_item = MLX5_FLOW_LAYER_GTP;
+ break;
+ default:
+ break;
+ }
+ item_flags |= last_item;
+ }
+ return item_flags;
+}
+
/**
* Register destination table DR jump action.
*
@@ -279,6 +345,96 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append shared RSS action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] idx
+ * Shared RSS index.
+ * @param[in] rss
+ * Pointer to the shared RSS info.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_shared_rss_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint32_t idx,
+ struct mlx5_shared_action_rss *rss)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->shared_rss.level = rss->origin.level;
+ act_data->shared_rss.types = !rss->origin.types ? RTE_ETH_RSS_IP :
+ rss->origin.types;
+ act_data->shared_rss.idx = idx;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
+/**
+ * Translate shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct mlx5_hw_actions *acts,
+ uint16_t action_src,
+ uint16_t action_dst)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss || __flow_hw_act_data_shared_rss_append
+ (priv, acts,
+ (enum rte_flow_action_type)MLX5_RTE_FLOW_ACTION_TYPE_RSS,
+ action_src, action_dst, idx, shared_rss))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -329,6 +485,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (!attr->group) {
+ DRV_LOG(ERR, "Indirect action is not supported in root table.");
+ goto err;
+ }
+ if (actions->conf && masks->conf) {
+ if (flow_hw_shared_action_translate
+ (dev, actions, acts, actions - action_start, i))
+ goto err;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -420,6 +590,115 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
"fail to create rte table");
}
+/**
+ * Get shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] act_data
+ * Pointer to the recorded action construct data.
+ * @param[in] item_flags
+ * The matcher itme_flags used for RSS lookup.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_get(struct rte_eth_dev *dev,
+ struct mlx5_action_construct_data *act_data,
+ const uint64_t item_flags,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_rss_desc rss_desc = { 0 };
+ uint64_t hash_fields = 0;
+ uint32_t hrxq_idx = 0;
+ struct mlx5_hrxq *hrxq = NULL;
+ int act_type = act_data->type;
+
+ switch (act_type) {
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ rss_desc.level = act_data->shared_rss.level;
+ rss_desc.types = act_data->shared_rss.types;
+ flow_dv_hashfields_set(item_flags, &rss_desc, &hash_fields);
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup
+ (dev, act_data->shared_rss.idx, hash_fields);
+ if (hrxq_idx)
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
+ hrxq_idx);
+ if (hrxq) {
+ rule_act->action = hrxq->action;
+ return 0;
+ }
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d",
+ act_data->type);
+ break;
+ }
+ return -1;
+}
+
+/**
+ * Construct shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] table
+ * Pointer to the flow table.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_construct(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct rte_flow_template_table *table,
+ const uint8_t it_idx,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_action_construct_data act_data;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+ uint64_t item_flags;
+
+ memset(&act_data, 0, sizeof(act_data));
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ act_data.type = MLX5_RTE_FLOW_ACTION_TYPE_RSS;
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss)
+ return -1;
+ act_data.shared_rss.idx = idx;
+ act_data.shared_rss.level = shared_rss->origin.level;
+ act_data.shared_rss.types = !shared_rss->origin.types ?
+ RTE_ETH_RSS_IP :
+ shared_rss->origin.types;
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, &act_data, item_flags, rule_act))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Construct flow action array.
*
@@ -432,6 +711,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
* @param[in] actions
* Array of rte_flow action need to be checked.
* @param[in] rule_acts
@@ -445,7 +726,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
static __rte_always_inline int
flow_hw_actions_construct(struct rte_eth_dev *dev,
struct mlx5_hw_q_job *job,
- struct mlx5_hw_actions *hw_acts,
+ const struct mlx5_hw_actions *hw_acts,
+ const uint8_t it_idx,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
@@ -477,14 +759,19 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
+ uint64_t item_flags;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
(int)action->type == act_data->type);
- switch (action->type) {
+ switch (act_data->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (flow_hw_shared_action_construct
+ (dev, action, table, it_idx,
+ &rule_acts[act_data->action_dst]))
+ return -1;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -517,6 +804,13 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->hrxq = hrxq;
job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
break;
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, act_data, item_flags,
+ &rule_acts[act_data->action_dst]))
+ return -1;
+ break;
default:
break;
}
@@ -599,8 +893,8 @@ flow_hw_q_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(dev, job, hw_acts, actions,
- rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, pattern_template_index,
+ actions, rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -1223,6 +1517,7 @@ flow_hw_pattern_template_create(struct rte_eth_dev *dev,
mlx5_free(it);
return NULL;
}
+ it->item_flags = flow_hw_rss_item_flags_get(items);
__atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
return it;
@@ -1647,6 +1942,97 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
priv->nb_queue = 0;
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+flow_hw_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ return flow_dv_action_create(dev, conf, action, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ return flow_dv_action_update(dev, handle, update, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_q_ops_attr *attr,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ return flow_dv_action_destroy(dev, handle, error);
+}
+
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.configure = flow_hw_configure,
.pattern_template_create = flow_hw_pattern_template_create,
@@ -1659,6 +2045,14 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.q_flow_destroy = flow_hw_q_flow_destroy,
.q_pull = flow_hw_q_pull,
.q_push = flow_hw_q_push,
+ .q_action_create = flow_hw_action_handle_create,
+ .q_action_destroy = flow_hw_action_handle_destroy,
+ .q_action_update = flow_hw_action_handle_update,
+ .action_validate = flow_dv_action_validate,
+ .action_create = flow_dv_action_create,
+ .action_destroy = flow_dv_action_destroy,
+ .action_update = flow_dv_action_update,
+ .action_query = flow_dv_action_query,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH 13/13] net/mlx5: add header reformat action
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (11 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 12/13] net/mlx5: add indirect action Suanming Mou
@ 2022-02-10 16:29 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (2 subsequent siblings)
15 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-10 16:29 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering header reformat action can work under bulk mode. In
this case, when create the table, bulk size of header reformat
actions will be allocated in low level. Afterwards, when create
flow, just simply specify the action index in the bulk and the
encapsulation data to the action will be enough.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 21 +++
drivers/net/mlx5/mlx5_flow_dv.c | 4 +-
drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
4 files changed, 251 insertions(+), 3 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index c78dc3c431..e10f55bf8c 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -344,6 +344,7 @@ struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
+ uint8_t *encap_data; /* Encap data. */
};
/* HW steering job descriptor LIFO header . */
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 097e5bf587..16fb6e643b 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1038,6 +1038,14 @@ struct mlx5_action_construct_data {
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
union {
+ struct {
+ /* encap src(item) offset. */
+ uint16_t src;
+ /* encap dst data offset. */
+ uint16_t dst;
+ /* encap data len. */
+ uint16_t len;
+ } encap;
struct {
uint64_t types; /* RSS hash types. */
uint32_t level; /* RSS level. */
@@ -1079,6 +1087,13 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+ struct mlx5dr_action *action; /* Action object. */
+ size_t data_size; /* Action metadata size. */
+ uint8_t data[]; /* Action data. */
+};
+
/* The maximum actions support in the flow. */
#define MLX5_HW_MAX_ACTS 16
@@ -1088,6 +1103,9 @@ struct mlx5_hw_actions {
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
+ /* Encap/Decap action. */
+ struct mlx5_hw_encap_decap_action *encap_decap;
+ uint16_t encap_decap_pos; /* Encap/Decap action position. */
uint32_t acts_num:4; /* Total action number. */
uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
@@ -2021,4 +2039,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle,
void *data,
struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+ size_t *size, struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index ca8ae4214b..377ed6c1db 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -4024,7 +4024,7 @@ flow_dv_push_vlan_action_resource_register
* @return
* sizeof struct item_type, 0 if void or irrelevant.
*/
-static size_t
+size_t
flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
{
size_t retval;
@@ -4090,7 +4090,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-static int
+int
flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
size_t *size, struct rte_flow_error *error)
{
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 9fc6f24542..5a652ac8e6 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -345,6 +345,50 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append dynamic encap action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] encap_src
+ * Offset of source encap raw data.
+ * @param[in] encap_dst
+ * Offset of destination encap raw data.
+ * @param[in] len
+ * Length of the data to be updated.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_encap_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint16_t encap_src,
+ uint16_t encap_dst,
+ uint16_t len)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->encap.src = encap_src;
+ act_data->encap.dst = encap_dst;
+ act_data->encap.len = len;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
/**
* Append shared RSS action to the dynamic action list.
*
@@ -435,6 +479,53 @@ flow_hw_shared_action_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Translate encap items to encapsulation list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] items
+ * Encap item pattern.
+ * @param[in] items_m
+ * Encap item mask indicates which part are constant and dynamic.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_encap_item_translate(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ const struct rte_flow_item *items,
+ const struct rte_flow_item *items_m)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ size_t len, total_len = 0;
+ uint32_t i = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++, items_m++, i++) {
+ len = flow_dv_get_item_hdr_len(items->type);
+ if ((!items_m->spec ||
+ memcmp(items_m->spec, items->spec, len)) &&
+ __flow_hw_act_data_encap_append(priv, acts, type,
+ action_src, action_dst, i,
+ total_len, len))
+ return -1;
+ total_len += len;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -472,6 +563,12 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
struct rte_flow_action *actions = at->actions;
struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
+ enum mlx5dr_action_reformat_type refmt_type = 0;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL, *enc_item_m = NULL;
+ uint16_t reformat_pos = MLX5_HW_MAX_ACTS, reformat_src = 0;
+ uint8_t *encap_data = NULL;
+ size_t data_size = 0;
bool actions_end = false;
uint32_t type, i;
int err;
@@ -573,6 +670,56 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_vxlan_encap *)
+ masks->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_DECAP:
+ case RTE_FLOW_ACTION_TYPE_NVGRE_DECAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ actions->conf;
+ encap_data = raw_encap_data->data;
+ data_size = raw_encap_data->size;
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ refmt_type = data_size <
+ MLX5_ENCAPSULATION_DECISION_SIZE ?
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L3_TO_L2 :
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L3;
+ } else {
+ reformat_pos = i++;
+ refmt_type =
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ }
+ reformat_src = actions - action_start;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_DECAP:
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -580,6 +727,45 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ uint8_t buf[MLX5_ENCAP_MAX_LEN];
+
+ if (enc_item) {
+ MLX5_ASSERT(!encap_data);
+ if (flow_dv_convert_encap_data
+ (enc_item, buf, &data_size, error) ||
+ flow_hw_encap_item_translate
+ (dev, acts, (action_start + reformat_src)->type,
+ reformat_src, reformat_pos,
+ enc_item, enc_item_m))
+ goto err;
+ encap_data = buf;
+ } else if (encap_data && __flow_hw_act_data_encap_append
+ (priv, acts,
+ (action_start + reformat_src)->type,
+ reformat_src, reformat_pos, 0, 0, data_size)) {
+ goto err;
+ }
+ acts->encap_decap = mlx5_malloc(MLX5_MEM_ZERO,
+ sizeof(*acts->encap_decap) + data_size,
+ 0, SOCKET_ID_ANY);
+ if (!acts->encap_decap)
+ goto err;
+ if (data_size) {
+ acts->encap_decap->data_size = data_size;
+ memcpy(acts->encap_decap->data, encap_data, data_size);
+ }
+ acts->encap_decap->action = mlx5dr_action_create_reformat
+ (priv->dr_ctx, refmt_type,
+ data_size, encap_data,
+ rte_log2_u32(table_attr->nb_flows),
+ mlx5_hw_act_flag[!!attr->group][type]);
+ if (!acts->encap_decap->action)
+ goto err;
+ acts->rule_acts[reformat_pos].action =
+ acts->encap_decap->action;
+ acts->encap_decap_pos = reformat_pos;
+ }
acts->acts_num = i;
return 0;
err:
@@ -735,6 +921,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_template_table *table = job->flow->table;
struct mlx5_action_construct_data *act_data;
const struct rte_flow_action *action;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL;
+ uint8_t *buf = job->encap_data;
struct rte_flow_attr attr = {
.ingress = 1,
};
@@ -756,6 +945,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
} else {
attr.ingress = 1;
}
+ if (hw_acts->encap_decap && hw_acts->encap_decap->data_size)
+ memcpy(buf, hw_acts->encap_decap->data,
+ hw_acts->encap_decap->data_size);
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
@@ -811,10 +1003,38 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
&rule_acts[act_data->action_dst]))
return -1;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ action->conf;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ raw_encap_data->data, act_data->encap.len);
+ MLX5_ASSERT(raw_encap_data->size ==
+ act_data->encap.len);
+ break;
default:
break;
}
}
+ if (hw_acts->encap_decap) {
+ rule_acts[hw_acts->encap_decap_pos].reformat.offset =
+ job->flow->idx - 1;
+ rule_acts[hw_acts->encap_decap_pos].reformat.data = buf;
+ }
return 0;
}
@@ -1821,6 +2041,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(uint8_t) * MLX5_ENCAP_MAX_LEN +
sizeof(struct mlx5_hw_q_job)) *
queue_attr[0]->size;
}
@@ -1831,6 +2052,8 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
for (i = 0; i < nb_queue; i++) {
+ uint8_t *encap = NULL;
+
priv->hw_q[i].job_idx = queue_attr[i]->size;
priv->hw_q[i].size = queue_attr[i]->size;
if (i == 0)
@@ -1841,8 +2064,11 @@ flow_hw_configure(struct rte_eth_dev *dev,
&job[queue_attr[i - 1]->size];
job = (struct mlx5_hw_q_job *)
&priv->hw_q[i].job[queue_attr[i]->size];
- for (j = 0; j < queue_attr[i]->size; j++)
+ encap = (uint8_t *)&job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++) {
+ job[j].encap_data = &encap[j * MLX5_ENCAP_MAX_LEN];
priv->hw_q[i].job[j] = &job[j];
+ }
}
dr_ctx_attr.pd = priv->sh->cdev->pd;
dr_ctx_attr.queues = nb_queue;
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 00/14] net/mlx5: add hardware steering
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (12 preceding siblings ...)
2022-02-10 16:29 ` [PATCH 13/13] net/mlx5: add header reformat action Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
` (13 more replies)
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
15 siblings, 14 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The Connect-X steering is a lookup hardware mechanism that accesses
flow tables, matches packets to the rules, and performs specified actions.
Historically, mlx5 PMD implements several software engines to manage
steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to
manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers,
groups, etc.) on the flow inserting is sure to cause slow flow
insertion
- security, exposing the low-level steering entries directly to the
userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering"
is going to be introduced to get rid of the security risks. And it will
take advantage of the recently new introduced async queue-based rte_flow
APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API
will not be supported in the first implementation, only the new async
queue-based flow operations is going to be supported. A new steering
mode parameter for dv_flow_en will be introduced and user will be
able to engage the new steering engine.
*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW ASYNC API ***
---
v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.
Suanming Mou (14):
net/mlx5: introduce hardware steering operation
net/mlx5: add HW steering low-level abstract code
net/mlx5: introduce hardware steering enable routine
net/mlx5: add port flow configuration
net/mlx5: add pattern template management
net/mlx5: add action template management
net/mlx5: add table management
net/mlx5: add basic flow queue operation
net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action
net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
doc/guides/nics/mlx5.rst | 19 +-
doc/guides/rel_notes/release_22_03.rst | 6 +
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/linux/mlx5_os.c | 22 +-
drivers/net/mlx5/meson.build | 2 +
drivers/net/mlx5/mlx5.c | 55 +-
drivers/net/mlx5/mlx5.h | 66 +-
drivers/net/mlx5/mlx5_devx.c | 10 +
drivers/net/mlx5/mlx5_dr.c | 383 ++++
drivers/net/mlx5/mlx5_dr.h | 456 +++++
drivers/net/mlx5/mlx5_flow.c | 721 ++++++-
drivers/net/mlx5/mlx5_flow.h | 287 +++
drivers/net/mlx5/mlx5_flow_dv.c | 186 +-
drivers/net/mlx5/mlx5_flow_hw.c | 2335 +++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 85 +-
drivers/net/mlx5/windows/mlx5_flow_os.h | 1 +
18 files changed, 4471 insertions(+), 180 deletions(-)
create mode 100644 drivers/net/mlx5/mlx5_dr.c
create mode 100644 drivers/net/mlx5/mlx5_dr.h
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 01/14] net/mlx5: introduce hardware steering operation
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
` (12 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The Connect-X steering is a lookup hardware mechanism that accesses
flow tables, matches packets to the rules, and performs specified actions.
Historically, mlx5 PMD implements several software engines to manage
steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to
manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers,
groups, etc.) on the flow inserting is sure to cause slow flow
insertion
- security, exposing the low-level steering entries directly to the
userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering"
is going to be introduced to get rid of the security risks. And it will
take advantage of the recently new introduced async queue-based rte_flow
APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API
will not be supported in the first implementation, only the new async
queue-based flow operations is going to be supported. A new steering
mode parameter for dv_flow_en will be introduced and user will be
able to engage the new steering engine.
This commit adds the basic driver operation.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_flow.c | 1 +
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++
drivers/net/mlx5/windows/mlx5_flow_os.h | 1 +
6 files changed, 18 insertions(+)
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.h b/drivers/net/mlx5/linux/mlx5_flow_os.h
index 1926d26410..e28a9e0436 100644
--- a/drivers/net/mlx5/linux/mlx5_flow_os.h
+++ b/drivers/net/mlx5/linux/mlx5_flow_os.h
@@ -9,6 +9,7 @@
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
#endif
/**
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 2f6d8cbb3d..39a2b8c523 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -16,6 +16,7 @@ sources = files(
'mlx5_flow.c',
'mlx5_flow_meter.c',
'mlx5_flow_dv.c',
+ 'mlx5_flow_hw.c',
'mlx5_flow_aso.c',
'mlx5_flow_flex.c',
'mlx5_mac.c',
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 5b5b00cc3e..de203c7b33 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+ [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
#endif
[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 583e8b7321..5e88a23f22 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
MLX5_FLOW_TYPE_MIN,
MLX5_FLOW_TYPE_DV,
MLX5_FLOW_TYPE_VERBS,
+ MLX5_FLOW_TYPE_HW,
MLX5_FLOW_TYPE_MAX,
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
new file mode 100644
index 0000000000..729d5914a8
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2022 NVIDIA Corporation & Affiliates
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
diff --git a/drivers/net/mlx5/windows/mlx5_flow_os.h b/drivers/net/mlx5/windows/mlx5_flow_os.h
index dfcb012334..52013b06a0 100644
--- a/drivers/net/mlx5/windows/mlx5_flow_os.h
+++ b/drivers/net/mlx5/windows/mlx5_flow_os.h
@@ -10,6 +10,7 @@
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
#endif
/**
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 02/14] net/mlx5: add HW steering low-level abstract code
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-22 8:51 ` [PATCH v2 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
` (11 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The HW steering low-level implementation will be added later in another
patch series. To avoid the linkage issues the abstract stub replacement
is provided currently.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++
3 files changed, 840 insertions(+)
create mode 100644 drivers/net/mlx5/mlx5_dr.c
create mode 100644 drivers/net/mlx5/mlx5_dr.h
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 39a2b8c523..393b2c97ac 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -14,6 +14,7 @@ sources = files(
'mlx5.c',
'mlx5_ethdev.c',
'mlx5_flow.c',
+ 'mlx5_dr.c',
'mlx5_flow_meter.c',
'mlx5_flow_dv.c',
'mlx5_flow_hw.c',
diff --git a/drivers/net/mlx5/mlx5_dr.c b/drivers/net/mlx5/mlx5_dr.c
new file mode 100644
index 0000000000..7218708986
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_dr.c
@@ -0,0 +1,383 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA CORPORATION. All rights reserved.
+ */
+#include <rte_flow.h>
+
+#include "mlx5_defs.h"
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+
+/*
+ * The following null stubs are prepared in order not to break the linkage
+ * before the HW steering low-level implementation is added.
+ */
+
+/* Open a context used for direct rule insertion using hardware steering.
+ * Each context can contain multiple tables of different types.
+ *
+ * @param[in] ibv_ctx
+ * The ibv context to used for HWS.
+ * @param[in] attr
+ * Attributes used for context open.
+ * @return pointer to mlx5dr_context on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_context *
+mlx5dr_context_open(void *ibv_ctx,
+ struct mlx5dr_context_attr *attr)
+{
+ (void)ibv_ctx;
+ (void)attr;
+ return NULL;
+}
+
+/* Close a context used for direct hardware steering.
+ *
+ * @param[in] ctx
+ * mlx5dr context to close.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_context_close(struct mlx5dr_context *ctx)
+{
+ (void)ctx;
+ return 0;
+}
+
+/* Create a new direct rule table. Each table can contain multiple matchers.
+ *
+ * @param[in] ctx
+ * The context in which the new table will be opened.
+ * @param[in] attr
+ * Attributes used for table creation.
+ * @return pointer to mlx5dr_table on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_table *
+mlx5dr_table_create(struct mlx5dr_context *ctx,
+ struct mlx5dr_table_attr *attr)
+{
+ (void)ctx;
+ (void)attr;
+ return NULL;
+}
+
+/* Destroy direct rule table.
+ *
+ * @param[in] tbl
+ * mlx5dr table to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int mlx5dr_table_destroy(struct mlx5dr_table *tbl)
+{
+ (void)tbl;
+ return 0;
+}
+
+/* Create new match template based on items mask, the match template
+ * will be used for matcher creation.
+ *
+ * @param[in] items
+ * Describe the mask for template creation
+ * @param[in] flags
+ * Template creation flags
+ * @return pointer to mlx5dr_match_template on success NULL otherwise
+ */
+__rte_weak struct mlx5dr_match_template *
+mlx5dr_match_template_create(const struct rte_flow_item items[],
+ enum mlx5dr_match_template_flags flags)
+{
+ (void)items;
+ (void)flags;
+ return NULL;
+}
+
+/* Destroy match template.
+ *
+ * @param[in] mt
+ * Match template to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_match_template_destroy(struct mlx5dr_match_template *mt)
+{
+ (void)mt;
+ return 0;
+}
+
+/* Create a new direct rule matcher. Each matcher can contain multiple rules.
+ * Matchers on the table will be processed by priority. Matching fields and
+ * mask are described by the match template. In some cases multiple match
+ * templates can be used on the same matcher.
+ *
+ * @param[in] table
+ * The table in which the new matcher will be opened.
+ * @param[in] mt
+ * Array of match templates to be used on matcher.
+ * @param[in] num_of_mt
+ * Number of match templates in mt array.
+ * @param[in] attr
+ * Attributes used for matcher creation.
+ * @return pointer to mlx5dr_matcher on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_matcher *
+mlx5dr_matcher_create(struct mlx5dr_table *table __rte_unused,
+ struct mlx5dr_match_template *mt[] __rte_unused,
+ uint8_t num_of_mt __rte_unused,
+ struct mlx5dr_matcher_attr *attr __rte_unused)
+{
+ return NULL;
+}
+
+/* Destroy direct rule matcher.
+ *
+ * @param[in] matcher
+ * Matcher to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher __rte_unused)
+{
+ return 0;
+}
+
+/* Enqueue create rule operation.
+ *
+ * @param[in] matcher
+ * The matcher in which the new rule will be created.
+ * @param[in] mt_idx
+ * Match template index to create the rule with.
+ * @param[in] items
+ * The items used for the value matching.
+ * @param[in] rule_actions
+ * Rule action to be executed on match.
+ * @param[in] num_of_actions
+ * Number of rule actions.
+ * @param[in] attr
+ * Rule creation attributes.
+ * @param[in, out] rule_handle
+ * A valid rule handle. The handle doesn't require any initialization.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_rule_create(struct mlx5dr_matcher *matcher __rte_unused,
+ uint8_t mt_idx __rte_unused,
+ const struct rte_flow_item items[] __rte_unused,
+ struct mlx5dr_rule_action rule_actions[] __rte_unused,
+ uint8_t num_of_actions __rte_unused,
+ struct mlx5dr_rule_attr *attr __rte_unused,
+ struct mlx5dr_rule *rule_handle __rte_unused)
+{
+ return 0;
+}
+
+/* Enqueue destroy rule operation.
+ *
+ * @param[in] rule
+ * The rule destruction to enqueue.
+ * @param[in] attr
+ * Rule destruction attributes.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_rule_destroy(struct mlx5dr_rule *rule __rte_unused,
+ struct mlx5dr_rule_attr *attr __rte_unused)
+{
+ return 0;
+}
+
+/* Create direct rule drop action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_drop(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule default miss action.
+ * Defaults are RX: Drop TX: Wire.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_default_miss(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule goto table action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] tbl
+ * Destination table.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx __rte_unused,
+ struct mlx5dr_table *tbl __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule goto TIR action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule TIR devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx __rte_unused,
+ struct mlx5dr_devx_obj *obj __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule TAG action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_tag(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule counter action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule counter devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_counter(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule reformat action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] reformat_type
+ * Type of reformat.
+ * @param[in] data_sz
+ * Size in bytes of data.
+ * @param[in] inline_data
+ * Header data array in case of inline action.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_reformat(struct mlx5dr_context *ctx __rte_unused,
+ enum mlx5dr_action_reformat_type reformat_type __rte_unused,
+ size_t data_sz __rte_unused,
+ void *inline_data __rte_unused,
+ uint32_t log_bulk_size __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule modify header action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] pattern_sz
+ * Byte size of the pattern array.
+ * @param[in] pattern
+ * PRM format modify pattern action array.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_modify_header(struct mlx5dr_context *ctx,
+ size_t pattern_sz,
+ rte_be64_t pattern[],
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Destroy direct rule action.
+ *
+ * @param[in] action
+ * The action to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_action_destroy(struct mlx5dr_action *action __rte_unused)
+{
+ return 0;
+}
+
+/* Poll queue for rule creation and deletions completions.
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to poll.
+ * @param[in, out] res
+ * Completion array.
+ * @param[in] res_nb
+ * Maximum number of results to return.
+ * @return negative number on failure, the number of completions otherwise.
+ */
+__rte_weak int
+mlx5dr_send_queue_poll(struct mlx5dr_context *ctx __rte_unused,
+ uint16_t queue_id __rte_unused,
+ struct rte_flow_op_result res[] __rte_unused,
+ uint32_t res_nb __rte_unused)
+{
+ return 0;
+}
+
+/* Perform an action on the queue
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to perform the action on.
+ * @param[in] actions
+ * Actions to perform on the queue. (enum mlx5dr_send_queue_actions)
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_send_queue_action(struct mlx5dr_context *ctx __rte_unused,
+ uint16_t queue_id __rte_unused,
+ uint32_t actions __rte_unused)
+{
+ return 0;
+}
+
+#endif
diff --git a/drivers/net/mlx5/mlx5_dr.h b/drivers/net/mlx5/mlx5_dr.h
new file mode 100644
index 0000000000..d0b2c15652
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_dr.h
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA CORPORATION. All rights reserved.
+ */
+
+#ifndef MLX5_DR_H_
+#define MLX5_DR_H_
+
+#include <rte_flow.h>
+
+struct mlx5dr_context;
+struct mlx5dr_table;
+struct mlx5dr_matcher;
+struct mlx5dr_rule;
+
+enum mlx5dr_table_type {
+ MLX5DR_TABLE_TYPE_NIC_RX,
+ MLX5DR_TABLE_TYPE_NIC_TX,
+ MLX5DR_TABLE_TYPE_FDB,
+ MLX5DR_TABLE_TYPE_MAX,
+};
+
+enum mlx5dr_matcher_resource_mode {
+ /* Allocate resources based on number of rules with minimal failure probability */
+ MLX5DR_MATCHER_RESOURCE_MODE_RULE,
+ /* Allocate fixed size hash table based on given column and rows */
+ MLX5DR_MATCHER_RESOURCE_MODE_HTABLE,
+};
+
+enum mlx5dr_action_flags {
+ MLX5DR_ACTION_FLAG_ROOT_RX = 1 << 0,
+ MLX5DR_ACTION_FLAG_ROOT_TX = 1 << 1,
+ MLX5DR_ACTION_FLAG_ROOT_FDB = 1 << 2,
+ MLX5DR_ACTION_FLAG_HWS_RX = 1 << 3,
+ MLX5DR_ACTION_FLAG_HWS_TX = 1 << 4,
+ MLX5DR_ACTION_FLAG_HWS_FDB = 1 << 5,
+ MLX5DR_ACTION_FLAG_INLINE = 1 << 6,
+};
+
+enum mlx5dr_action_reformat_type {
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L3_TO_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L3,
+};
+
+enum mlx5dr_match_template_flags {
+ /* Allow relaxed matching by skipping derived dependent match fields. */
+ MLX5DR_MATCH_TEMPLATE_FLAG_RELAXED_MATCH = 1,
+};
+
+enum mlx5dr_send_queue_actions {
+ /* Start executing all pending queued rules and write to HW */
+ MLX5DR_SEND_QUEUE_ACTION_DRAIN = 1 << 0,
+};
+
+struct mlx5dr_context_attr {
+ uint16_t queues;
+ uint16_t queue_size;
+ size_t initial_log_ste_memory;
+ /* Optional PD used for allocating res ources */
+ struct ibv_pd *pd;
+};
+
+struct mlx5dr_table_attr {
+ enum mlx5dr_table_type type;
+ uint32_t level;
+};
+
+struct mlx5dr_matcher_attr {
+ uint32_t priority;
+ enum mlx5dr_matcher_resource_mode mode;
+ union {
+ struct {
+ uint8_t sz_row_log;
+ uint8_t sz_col_log;
+ } table;
+
+ struct {
+ uint8_t num_log;
+ } rule;
+ };
+};
+
+struct mlx5dr_rule_attr {
+ uint16_t queue_id;
+ void *user_data;
+ uint32_t burst:1;
+};
+
+struct mlx5dr_devx_obj {
+ struct mlx5dv_devx_obj *obj;
+ uint32_t id;
+};
+
+struct mlx5dr_rule_action {
+ struct mlx5dr_action *action;
+ union {
+ struct {
+ uint32_t value;
+ } tag;
+
+ struct {
+ uint32_t offset;
+ } counter;
+
+ struct {
+ uint32_t offset;
+ uint8_t *data;
+ } modify_header;
+
+ struct {
+ uint32_t offset;
+ uint8_t *data;
+ } reformat;
+
+ struct {
+ rte_be32_t vlan_hdr;
+ } push_vlan;
+ };
+};
+
+enum {
+ MLX5DR_MATCH_TAG_SZ = 32,
+ MLX5DR_JAMBO_TAG_SZ = 44,
+};
+
+enum mlx5dr_rule_status {
+ MLX5DR_RULE_STATUS_UNKNOWN,
+ MLX5DR_RULE_STATUS_CREATING,
+ MLX5DR_RULE_STATUS_CREATED,
+ MLX5DR_RULE_STATUS_DELETING,
+ MLX5DR_RULE_STATUS_DELETED,
+ MLX5DR_RULE_STATUS_FAILED,
+};
+
+struct mlx5dr_rule {
+ struct mlx5dr_matcher *matcher;
+ union {
+ uint8_t match_tag[MLX5DR_MATCH_TAG_SZ];
+ struct ibv_flow *flow;
+ };
+ enum mlx5dr_rule_status status;
+ uint32_t rtc_used; /* The RTC into which the STE was inserted */
+};
+
+/* Open a context used for direct rule insertion using hardware steering.
+ * Each context can contain multiple tables of different types.
+ *
+ * @param[in] ibv_ctx
+ * The ibv context to used for HWS.
+ * @param[in] attr
+ * Attributes used for context open.
+ * @return pointer to mlx5dr_context on success NULL otherwise.
+ */
+struct mlx5dr_context *
+mlx5dr_context_open(void *ibv_ctx,
+ struct mlx5dr_context_attr *attr);
+
+/* Close a context used for direct hardware steering.
+ *
+ * @param[in] ctx
+ * mlx5dr context to close.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_context_close(struct mlx5dr_context *ctx);
+
+/* Create a new direct rule table. Each table can contain multiple matchers.
+ *
+ * @param[in] ctx
+ * The context in which the new table will be opened.
+ * @param[in] attr
+ * Attributes used for table creation.
+ * @return pointer to mlx5dr_table on success NULL otherwise.
+ */
+struct mlx5dr_table *
+mlx5dr_table_create(struct mlx5dr_context *ctx,
+ struct mlx5dr_table_attr *attr);
+
+/* Destroy direct rule table.
+ *
+ * @param[in] tbl
+ * mlx5dr table to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_table_destroy(struct mlx5dr_table *tbl);
+
+/* Create new match template based on items mask, the match template
+ * will be used for matcher creation.
+ *
+ * @param[in] items
+ * Describe the mask for template creation
+ * @param[in] flags
+ * Template creation flags
+ * @return pointer to mlx5dr_match_template on success NULL otherwise
+ */
+struct mlx5dr_match_template *
+mlx5dr_match_template_create(const struct rte_flow_item items[],
+ enum mlx5dr_match_template_flags flags);
+
+/* Destroy match template.
+ *
+ * @param[in] mt
+ * Match template to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_match_template_destroy(struct mlx5dr_match_template *mt);
+
+/* Create a new direct rule matcher. Each matcher can contain multiple rules.
+ * Matchers on the table will be processed by priority. Matching fields and
+ * mask are described by the match template. In some cases multiple match
+ * templates can be used on the same matcher.
+ *
+ * @param[in] table
+ * The table in which the new matcher will be opened.
+ * @param[in] mt
+ * Array of match templates to be used on matcher.
+ * @param[in] num_of_mt
+ * Number of match templates in mt array.
+ * @param[in] attr
+ * Attributes used for matcher creation.
+ * @return pointer to mlx5dr_matcher on success NULL otherwise.
+ */
+struct mlx5dr_matcher *
+mlx5dr_matcher_create(struct mlx5dr_table *table,
+ struct mlx5dr_match_template *mt[],
+ uint8_t num_of_mt,
+ struct mlx5dr_matcher_attr *attr);
+
+/* Destroy direct rule matcher.
+ *
+ * @param[in] matcher
+ * Matcher to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher);
+
+/* Get the size of the rule handle (mlx5dr_rule) to be used on rule creation.
+ *
+ * @return size in bytes of rule handle struct.
+ */
+size_t mlx5dr_rule_get_handle_size(void);
+
+/* Enqueue create rule operation.
+ *
+ * @param[in] matcher
+ * The matcher in which the new rule will be created.
+ * @param[in] mt_idx
+ * Match template index to create the rule with.
+ * @param[in] items
+ * The items used for the value matching.
+ * @param[in] rule_actions
+ * Rule action to be executed on match.
+ * @param[in] num_of_actions
+ * Number of rule actions.
+ * @param[in] attr
+ * Rule creation attributes.
+ * @param[in, out] rule_handle
+ * A valid rule handle. The handle doesn't require any initialization.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+int mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
+ uint8_t mt_idx,
+ const struct rte_flow_item items[],
+ struct mlx5dr_rule_action rule_actions[],
+ uint8_t num_of_actions,
+ struct mlx5dr_rule_attr *attr,
+ struct mlx5dr_rule *rule_handle);
+
+/* Enqueue destroy rule operation.
+ *
+ * @param[in] rule
+ * The rule destruction to enqueue.
+ * @param[in] attr
+ * Rule destruction attributes.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+int mlx5dr_rule_destroy(struct mlx5dr_rule *rule,
+ struct mlx5dr_rule_attr *attr);
+
+/* Create direct rule drop action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_drop(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule default miss action.
+ * Defaults are RX: Drop TX: Wire.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_default_miss(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule goto table action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] tbl
+ * Destination table.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx,
+ struct mlx5dr_table *tbl,
+ uint32_t flags);
+
+/* Create direct rule goto TIR action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule TIR devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule TAG action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_tag(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule counter action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule counter devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_counter(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule reformat action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] reformat_type
+ * Type of reformat.
+ * @param[in] data_sz
+ * Size in bytes of data.
+ * @param[in] inline_data
+ * Header data array in case of inline action.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_reformat(struct mlx5dr_context *ctx,
+ enum mlx5dr_action_reformat_type reformat_type,
+ size_t data_sz,
+ void *inline_data,
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Create direct rule modify header action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] pattern_sz
+ * Byte size of the pattern array.
+ * @param[in] pattern
+ * PRM format modify pattern action array.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_modify_header(struct mlx5dr_context *ctx,
+ size_t pattern_sz,
+ rte_be64_t pattern[],
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Destroy direct rule action.
+ *
+ * @param[in] action
+ * The action to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_action_destroy(struct mlx5dr_action *action);
+
+/* Poll queue for rule creation and deletions completions.
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to poll.
+ * @param[in, out] res
+ * Completion array.
+ * @param[in] res_nb
+ * Maximum number of results to return.
+ * @return negative number on failure, the number of completions otherwise.
+ */
+int mlx5dr_send_queue_poll(struct mlx5dr_context *ctx,
+ uint16_t queue_id,
+ struct rte_flow_op_result res[],
+ uint32_t res_nb);
+
+/* Perform an action on the queue
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to perform the action on.
+ * @param[in] actions
+ * Actions to perform on the queue. (enum mlx5dr_send_queue_actions)
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_send_queue_action(struct mlx5dr_context *ctx,
+ uint16_t queue_id,
+ uint32_t actions);
+
+/* Dump HWS info
+ *
+ * @param[in] ctx
+ * The context which to dump the info from.
+ * @param[in] f
+ * The file to write the dump to.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_debug_dump(struct mlx5dr_context *ctx, FILE *f);
+
+#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 03/14] net/mlx5: introduce hardware steering enable routine
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-22 8:51 ` [PATCH v2 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-22 8:51 ` [PATCH v2 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 04/14] net/mlx5: add port flow configuration Suanming Mou
` (10 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The new hardware steering engine relies on using dedicated steering WQEs
instead of direct writing to the low-level steering table entries directly.
In the first introduce implementation the hardware steering engine supports
the new queue based Flow API, the existing synchronous non-queue based Flow
API is not supported.
A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:
dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support
This commit introduces the extra dv_flow_en = 2 to specify the new
flow initialize and manage operation routine.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
doc/guides/nics/mlx5.rst | 13 ++++++++++---
drivers/net/mlx5/linux/mlx5_os.c | 4 ++++
drivers/net/mlx5/mlx5.c | 7 ++++++-
drivers/net/mlx5/mlx5.h | 3 ++-
drivers/net/mlx5/mlx5_flow.c | 22 ++++++++++++++++++++++
5 files changed, 44 insertions(+), 5 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index c3cc0c0f41..c3f7d61e7d 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -1042,10 +1042,17 @@ Driver options
- ``dv_flow_en`` parameter [int]
- A nonzero value enables the DV flow steering assuming it is supported
- by the driver (RDMA Core library version is rdma-core-24.0 or higher).
+ Value 0 means legacy Verbs flow offloading.
- Enabled by default if supported.
+ Value 1 enables the DV flow steering assuming it is supported by the
+ driver (RDMA Core library version is rdma-core-24.0 or higher).
+
+ Value 2 enables the WQE based hardware steering. In this mode only
+ the queue-based rte_flow_q flow management is supported.
+
+ Configured by default to 1 DV flow steering if the driver(RDMA CORE library)
+ supported. Otherwise, the value will be 0 which indicates legacy Verbs flow
+ offloading.
- ``dv_esw_en`` parameter [int]
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index ecf823da56..0faf26f5b8 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
err = mlx5_alloc_table_hash_list(priv);
if (err)
goto error;
+ if (priv->sh->config.dv_flow_en == 2)
+ return 0;
/* The resources below are only valid with DV support. */
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
/* Init port id action list. */
@@ -1519,6 +1521,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ if (priv->sh->config.dv_flow_en == 2)
+ return eth_dev;
/* Port representor shares the same max priority with pf port. */
if (!priv->sh->flow_priority_check_flag) {
/* Supported Verbs flow priority number detection. */
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 9f65a8f901..f49d30c05c 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque)
} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
config->dv_esw_en = !!tmp;
} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
- config->dv_flow_en = !!tmp;
+ if (tmp > 2) {
+ DRV_LOG(ERR, "Invalid %s parameter.", key);
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ config->dv_flow_en = tmp;
} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
if (tmp != MLX5_XMETA_MODE_LEGACY &&
tmp != MLX5_XMETA_MODE_META16 &&
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 35ea3fb47c..42e725be63 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -287,7 +287,8 @@ struct mlx5_sh_config {
int tx_skew; /* Tx scheduling skew between WQE and data on wire. */
uint32_t reclaim_mode:2; /* Memory reclaim mode. */
uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */
- uint32_t dv_flow_en:1; /* Enable DV flow. */
+ /* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+ unsigned int dv_flow_en:2;
uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */
uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */
uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index de203c7b33..0f384cc13f 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -6838,6 +6838,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,
const struct rte_flow_action actions[],
struct rte_flow_error *error)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (priv->sh->config.dv_flow_en == 2) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q creation not supported");
+ return NULL;
+ }
/*
* If the device is not started yet, it is not allowed to created a
* flow from application. PMD default flows and traffic control flows
@@ -7334,6 +7343,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,
struct rte_flow *flow,
struct rte_flow_error *error __rte_unused)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (priv->sh->config.dv_flow_en == 2)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q destruction not supported");
flow_list_destroy(dev, MLX5_FLOW_TYPE_GEN,
(uintptr_t)(void *)flow);
return 0;
@@ -7431,7 +7447,13 @@ mlx5_flow_query(struct rte_eth_dev *dev,
struct rte_flow_error *error)
{
int ret;
+ struct mlx5_priv *priv = dev->data->dev_private;
+ if (priv->sh->config.dv_flow_en == 2)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q query not supported");
ret = flow_drv_query(dev, (uintptr_t)(void *)flow, actions, data,
error);
if (ret < 0)
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 04/14] net/mlx5: add port flow configuration
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (2 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 05/14] net/mlx5: add pattern template management Suanming Mou
` (9 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The hardware steering is backend to support rte_flow_async API in
mlx5 PMD. The port configuration function creates the queues and
needed flow management resources.
The PMD layer configuration function allocates the queues' context
and per-queue job descriptor pool. The job descriptor pool size
is equal to the queue size, and the job descriptors will be popped
from pool with LIFO strategy to convey the flow information during
flow insertion/destruction. Then, while polling the queued operation
result, the flow information will be extracted from the job descriptor
and the descriptor will be pushed back to the LIFO pool.
The commit creates the flow port queues and the job descriptor pools.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 3 +
drivers/net/mlx5/mlx5.h | 30 +++++-
drivers/net/mlx5/mlx5_flow.c | 86 +++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 15 +++
drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++
5 files changed, 292 insertions(+), 1 deletion(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f49d30c05c..0079aa83c1 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
/* Free the eCPRI flex parser resource. */
mlx5_flex_parser_ecpri_release(dev);
mlx5_flex_item_port_cleanup(dev);
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ flow_hw_resource_release(dev);
+#endif
if (priv->rxq_privs != NULL) {
/* XXX race condition if mlx5_rx_burst() is still running. */
rte_delay_us_sleep(1000);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 42e725be63..5a6c4937a5 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -33,7 +33,9 @@
#include "mlx5_utils.h"
#include "mlx5_os.h"
#include "mlx5_autoconf.h"
-
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+#endif
#define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {
uint16_t refcnt; /* Reference count for representors. */
};
+/* HW steering queue job descriptor type. */
+enum {
+ MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+ MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+ uint32_t type; /* Job type. */
+ struct rte_flow *flow; /* Flow attached to the job. */
+ void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO pool. */
+struct mlx5_hw_q {
+ uint32_t job_idx; /* Free job index. */
+ uint32_t size; /* LIFO size. */
+ struct mlx5_hw_q_job **job; /* LIFO header. */
+} __rte_cache_aligned;
+
#define MLX5_COUNTERS_PER_POOL 512
#define MLX5_MAX_PENDING_QUERIES 4
#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1476,6 +1498,12 @@ struct mlx5_priv {
struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+ uint32_t nb_queue; /* HW steering queue number. */
+ /* HW steering queue polling mechanism job descriptor LIFO. */
+ struct mlx5_hw_q *hw_q;
+#endif
};
#define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 0f384cc13f..01da304e0b 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -805,6 +805,17 @@ static int
mlx5_flow_flex_item_release(struct rte_eth_dev *dev,
const struct rte_flow_item_flex_handle *handle,
struct rte_flow_error *error);
+static int
+mlx5_flow_info_get(struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -826,6 +837,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.get_restore_info = mlx5_flow_tunnel_get_restore_info,
.flex_item_create = mlx5_flow_flex_item_create,
.flex_item_release = mlx5_flow_flex_item_release,
+ .info_get = mlx5_flow_info_get,
+ .configure = mlx5_flow_port_configure,
};
/* Tunnel information. */
@@ -3429,6 +3442,12 @@ flow_get_drv_type(struct rte_eth_dev *dev, const struct rte_flow_attr *attr)
if (type != MLX5_FLOW_TYPE_MAX)
return type;
+ /*
+ * Currently when dv_flow_en == 2, only HW steering engine is
+ * supported. New engines can also be chosen here if ready.
+ */
+ if (priv->sh->config.dv_flow_en == 2)
+ return MLX5_FLOW_TYPE_HW;
/* If no OS specific type - continue with DV/VERBS selection */
if (attr->transfer && priv->sh->config.dv_esw_en)
type = MLX5_FLOW_TYPE_DV;
@@ -7835,6 +7854,73 @@ mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt,
return -ENOTSUP;
}
+/**
+ * Get information about HWS pre-configurable resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] port_info
+ * Pointer to port information.
+ * @param[out] queue_info
+ * Pointer to queue information.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_info_get(struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "info get with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->info_get(dev, port_info, queue_info, error);
+}
+
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "port configure with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->configure(dev, port_attr, nb_queue, queue_attr, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 5e88a23f22..1c7dee8b29 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)
const struct rte_flow_item_flex_handle *handle,
const struct rte_flow_item_flex_conf *conf,
struct rte_flow_error *error);
+typedef int (*mlx5_flow_info_get_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_create_t item_create;
mlx5_flow_item_release_t item_release;
mlx5_flow_item_update_t item_update;
+ mlx5_flow_info_get_t info_get;
+ mlx5_flow_port_configure_t configure;
};
/* mlx5_flow.c */
@@ -1767,4 +1780,6 @@ const struct mlx5_flow_tunnel *
mlx5_get_tof(const struct rte_flow_item *items,
const struct rte_flow_action *actions,
enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 729d5914a8..e5b2ae91d8 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -4,10 +4,169 @@
#include <rte_flow.h>
+#include <mlx5_malloc.h>
+#include "mlx5_defs.h"
#include "mlx5_flow.h"
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Get information about HWS pre-configurable resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] port_info
+ * Pointer to port information.
+ * @param[out] queue_info
+ * Pointer to queue information.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_port_info *port_info __rte_unused,
+ struct rte_flow_queue_info *queue_info __rte_unused,
+ struct rte_flow_error *error __rte_unused)
+{
+ /* Nothing to be updated currently. */
+ memset(port_info, 0, sizeof(*port_info));
+ /* Queue size is unlimited from low-level. */
+ queue_info->max_size = UINT32_MAX;
+ return 0;
+}
+
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_context *dr_ctx = NULL;
+ struct mlx5dr_context_attr dr_ctx_attr = {0};
+ struct mlx5_hw_q *hw_q;
+ struct mlx5_hw_q_job *job = NULL;
+ uint32_t mem_size, i, j;
+
+ if (!port_attr || !nb_queue || !queue_attr) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ /* In case re-configuring, release existing context at first. */
+ if (priv->dr_ctx) {
+ /* */
+ for (i = 0; i < nb_queue; i++) {
+ hw_q = &priv->hw_q[i];
+ /* Make sure all queues are empty. */
+ if (hw_q->size != hw_q->job_idx) {
+ rte_errno = EBUSY;
+ goto err;
+ }
+ }
+ flow_hw_resource_release(dev);
+ }
+ /* Allocate the queue job descriptor LIFO. */
+ mem_size = sizeof(priv->hw_q[0]) * nb_queue;
+ for (i = 0; i < nb_queue; i++) {
+ /*
+ * Check if the queues' size are all the same as the
+ * limitation from HWS layer.
+ */
+ if (queue_attr[i]->size != queue_attr[0]->size) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(struct mlx5_hw_q_job)) *
+ queue_attr[0]->size;
+ }
+ priv->hw_q = mlx5_malloc(MLX5_MEM_ZERO, mem_size,
+ 64, SOCKET_ID_ANY);
+ if (!priv->hw_q) {
+ rte_errno = ENOMEM;
+ goto err;
+ }
+ for (i = 0; i < nb_queue; i++) {
+ priv->hw_q[i].job_idx = queue_attr[i]->size;
+ priv->hw_q[i].size = queue_attr[i]->size;
+ if (i == 0)
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &priv->hw_q[nb_queue];
+ else
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &job[queue_attr[i - 1]->size];
+ job = (struct mlx5_hw_q_job *)
+ &priv->hw_q[i].job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++)
+ priv->hw_q[i].job[j] = &job[j];
+ }
+ dr_ctx_attr.pd = priv->sh->cdev->pd;
+ dr_ctx_attr.queues = nb_queue;
+ /* Queue size should all be the same. Take the first one. */
+ dr_ctx_attr.queue_size = queue_attr[0]->size;
+ dr_ctx = mlx5dr_context_open(priv->sh->cdev->ctx, &dr_ctx_attr);
+ /* rte_errno has been updated by HWS layer. */
+ if (!dr_ctx)
+ goto err;
+ priv->dr_ctx = dr_ctx;
+ priv->nb_queue = nb_queue;
+ return 0;
+err:
+ if (dr_ctx)
+ claim_zero(mlx5dr_context_close(dr_ctx));
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to configure port");
+}
+
+/**
+ * Release HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ */
+void
+flow_hw_resource_release(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (!priv->dr_ctx)
+ return;
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ claim_zero(mlx5dr_context_close(priv->dr_ctx));
+ priv->dr_ctx = NULL;
+ priv->nb_queue = 0;
+}
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
+ .info_get = flow_hw_info_get,
+ .configure = flow_hw_configure,
+};
+
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 05/14] net/mlx5: add pattern template management
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (3 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 04/14] net/mlx5: add port flow configuration Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 06/14] net/mlx5: add action " Suanming Mou
` (8 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The pattern template defines flows that have the same matching
fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be
(eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while
the values for each rule will be different.
Due to the pattern template can be used in different domains, the
items will only be cached in pattern template create stage, while
the template is binded to a dedicated table, the HW criteria will
be created and saved to the table. The pattern templates can be
used by multiple tables. But different tables create the same
criteria and will not share the matcher between each other in order
to have better performance.
This commit adds pattern template management.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 76 +++++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 24 +++++++++
drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++
4 files changed, 188 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 5a6c4937a5..44694daf1b 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1499,6 +1499,8 @@ struct mlx5_priv {
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ /* Item template list. */
+ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 01da304e0b..e6363ec765 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -817,6 +817,17 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -839,6 +850,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.flex_item_release = mlx5_flow_flex_item_release,
.info_get = mlx5_flow_info_get,
.configure = mlx5_flow_port_configure,
+ .pattern_template_create = mlx5_flow_pattern_template_create,
+ .pattern_template_destroy = mlx5_flow_pattern_template_destroy,
};
/* Tunnel information. */
@@ -7921,6 +7934,69 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
return fops->configure(dev, port_attr, nb_queue, queue_attr, error);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "pattern create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pattern_template_create(dev, attr, items, error);
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "pattern destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pattern_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 1c7dee8b29..12dbb4655b 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1015,6 +1015,19 @@ struct rte_flow {
uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
} __rte_packed;
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+ LIST_ENTRY(rte_flow_pattern_template) next;
+ /* Template attributes. */
+ struct rte_flow_pattern_template_attr attr;
+ struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint32_t refcnt; /* Reference counter. */
+};
+
+#endif
+
/*
* Define list of valid combinations of RX Hash fields
* (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)
uint16_t nb_queue,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_update_t item_update;
mlx5_flow_info_get_t info_get;
mlx5_flow_port_configure_t configure;
+ mlx5_flow_pattern_template_create_t pattern_template_create;
+ mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index e5b2ae91d8..66c5825084 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -13,6 +13,85 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Item template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+flow_hw_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
+
+ it = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*it), 0, rte_socket_id());
+ if (!it) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate item template");
+ return NULL;
+ }
+ it->attr = *attr;
+ it->mt = mlx5dr_match_template_create(items, attr->relaxed_matching);
+ if (!it->mt) {
+ mlx5_free(it);
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot create match template");
+ return NULL;
+ }
+ __atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
+ return it;
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Item template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "item template in using");
+ }
+ LIST_REMOVE(template, next);
+ claim_zero(mlx5dr_match_template_destroy(template->mt));
+ mlx5_free(template);
+ return 0;
+}
+
+/*
* Get information about HWS pre-configurable resources.
*
* @param[in] dev
@@ -154,9 +233,14 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_itt)) {
+ it = LIST_FIRST(&priv->flow_hw_itt);
+ flow_hw_pattern_template_destroy(dev, it, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -167,6 +251,8 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.info_get = flow_hw_info_get,
.configure = flow_hw_configure,
+ .pattern_template_create = flow_hw_pattern_template_create,
+ .pattern_template_destroy = flow_hw_pattern_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 06/14] net/mlx5: add action template management
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (4 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 05/14] net/mlx5: add pattern template management Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 07/14] net/mlx5: add table management Suanming Mou
` (7 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The action template holds a list of action types that will be
used together on the same rule. The template's actions instances
will be created only when the template bind to the dedicated
group. And the created actions will be saved to each individual
group in order for best performance. The actions in a group will
not be shared with each other unless shared actions are specified.
This commit adds the action template management which stores the
flow action template.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 78 +++++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 22 ++++++
drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
4 files changed, 218 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 44694daf1b..8e36805e54 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1501,6 +1501,8 @@ struct mlx5_priv {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
/* Item template list. */
LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+ /* Action template list. */
+ LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index e6363ec765..e937110054 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -827,6 +827,16 @@ static int
mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -852,6 +862,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.configure = mlx5_flow_port_configure,
.pattern_template_create = mlx5_flow_pattern_template_create,
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
+ .actions_template_create = mlx5_flow_actions_template_create,
+ .actions_template_destroy = mlx5_flow_actions_template_destroy,
};
/* Tunnel information. */
@@ -7997,6 +8009,72 @@ mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
return fops->pattern_template_destroy(dev, template, error);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->actions_template_create(dev, attr, actions, masks, error);
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->actions_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 12dbb4655b..7e2d6bf4eb 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+ LIST_ENTRY(rte_flow_actions_template) next;
+ /* Template attributes. */
+ struct rte_flow_actions_template_attr attr;
+ struct rte_flow_action *actions; /* Cached flow actions. */
+ struct rte_flow_action *masks; /* Cached action masks.*/
+ uint32_t refcnt; /* Reference counter. */
+};
+
#endif
/*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_port_configure_t configure;
mlx5_flow_pattern_template_create_t pattern_template_create;
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+ mlx5_flow_actions_template_create_t actions_template_create;
+ mlx5_flow_actions_template_destroy_t actions_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 66c5825084..4214a63a73 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,115 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Create flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+flow_hw_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int len, act_len, mask_len, i;
+ struct rte_flow_actions_template *at;
+
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, actions, error);
+ if (act_len <= 0)
+ return NULL;
+ len = RTE_ALIGN(act_len, 16);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, masks, error);
+ if (mask_len <= 0)
+ return NULL;
+ len += RTE_ALIGN(mask_len, 16);
+ at = mlx5_malloc(MLX5_MEM_ZERO, len + sizeof(*at), 64, rte_socket_id());
+ if (!at) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate action template");
+ return NULL;
+ }
+ at->attr = *attr;
+ at->actions = (struct rte_flow_action *)(at + 1);
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->actions, len,
+ actions, error);
+ if (act_len <= 0)
+ goto error;
+ at->masks = (struct rte_flow_action *)
+ (((uint8_t *)at->actions) + act_len);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->masks,
+ len - act_len, masks, error);
+ if (mask_len <= 0)
+ goto error;
+ /*
+ * mlx5 PMD hacks indirect action index directly to the action conf.
+ * The rte_flow_conv() function copies the content from conf pointer.
+ * Need to restore the indirect action index from action conf here.
+ */
+ for (i = 0; actions->type != RTE_FLOW_ACTION_TYPE_END;
+ actions++, masks++, i++) {
+ if (actions->type == RTE_FLOW_ACTION_TYPE_INDIRECT) {
+ at->actions[i].conf = actions->conf;
+ at->masks[i].conf = masks->conf;
+ }
+ }
+ __atomic_fetch_add(&at->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_at, at, next);
+ return at;
+error:
+ mlx5_free(at);
+ return NULL;
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Action template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action template in using");
+ }
+ LIST_REMOVE(template, next);
+ mlx5_free(template);
+ return 0;
+}
+
/**
* Create flow item template.
*
@@ -234,6 +343,7 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct rte_flow_pattern_template *it;
+ struct rte_flow_actions_template *at;
if (!priv->dr_ctx)
return;
@@ -241,6 +351,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
}
+ while (!LIST_EMPTY(&priv->flow_hw_at)) {
+ at = LIST_FIRST(&priv->flow_hw_at);
+ flow_hw_actions_template_destroy(dev, at, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -253,6 +367,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.configure = flow_hw_configure,
.pattern_template_create = flow_hw_pattern_template_create,
.pattern_template_destroy = flow_hw_pattern_template_destroy,
+ .actions_template_create = flow_hw_actions_template_create,
+ .actions_template_destroy = flow_hw_actions_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 07/14] net/mlx5: add table management
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (5 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 06/14] net/mlx5: add action " Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 08/14] net/mlx5: add basic flow queue operation Suanming Mou
` (6 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Flow table is a group of flows with the same matching criteria
and the same actions defined for them. The table defines rules
that have the same matching fields but with different matching
values. For example, matching on 5 tuple, the table will be
(IPv4 source + IPv4 dest + s_port + d_port + next_proto)
while the values for each rule will be different.
The templates' relevant matching criteria and action instances
will be created in the table creation and saved in the table.
As table attributes indicate the supported flow number, the flow
memory will also be allocated at the same time.
This commit adds the table management functions.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 45 ++-
drivers/net/mlx5/mlx5.h | 21 +-
drivers/net/mlx5/mlx5_flow.c | 93 ++++++
drivers/net/mlx5/mlx5_flow.h | 73 +++++
drivers/net/mlx5/mlx5_flow_hw.c | 525 ++++++++++++++++++++++++++++++++
5 files changed, 751 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 0079aa83c1..7611fdd62b 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1607,12 +1607,46 @@ void
mlx5_free_table_hash_list(struct mlx5_priv *priv)
{
struct mlx5_dev_ctx_shared *sh = priv->sh;
-
- if (!sh->flow_tbls)
+ struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en == 2) ?
+ &sh->groups : &sh->flow_tbls;
+ if (*tbls == NULL)
return;
- mlx5_hlist_destroy(sh->flow_tbls);
- sh->flow_tbls = NULL;
+ mlx5_hlist_destroy(*tbls);
+ *tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ * Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+ int err = 0;
+ struct mlx5_dev_ctx_shared *sh = priv->sh;
+ char s[MLX5_NAME_SIZE];
+
+ MLX5_ASSERT(sh);
+ snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+ sh->groups = mlx5_hlist_create
+ (s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+ false, true, sh,
+ flow_hw_grp_create_cb,
+ flow_hw_grp_match_cb,
+ flow_hw_grp_remove_cb,
+ flow_hw_grp_clone_cb,
+ flow_hw_grp_clone_free_cb);
+ if (!sh->groups) {
+ DRV_LOG(ERR, "flow groups with hash creation failed.");
+ err = ENOMEM;
+ }
+ return err;
}
+#endif
+
/**
* Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int
mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
{
int err = 0;
+
/* Tables are only used in DV and DR modes. */
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
struct mlx5_dev_ctx_shared *sh = priv->sh;
char s[MLX5_NAME_SIZE];
+ if (priv->sh->config.dv_flow_en == 2)
+ return mlx5_alloc_hw_group_hash_list(priv);
MLX5_ASSERT(sh);
snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 8e36805e54..1cc63e1d9f 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {
MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
MLX5_IPOOL_TAG, /* Pool for tag resource. */
MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
- MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+ MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+ /* Pool for HWS group. Jump action will be created internally. */
+ MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode {
MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
};
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+ MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+ MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+ MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
/* Hlist and list callback context. */
struct mlx5_flow_cb_ctx {
struct rte_eth_dev *dev;
@@ -1201,7 +1210,10 @@ struct mlx5_dev_ctx_shared {
rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
/* UAR same-page access control required in 32bit implementations. */
#endif
- struct mlx5_hlist *flow_tbls;
+ union {
+ struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+ struct mlx5_hlist *groups; /* HWS flow group. */
+ };
struct mlx5_flow_tunnel_hub *tunnel_hub;
/* Direct Rules tables for FDB, NIC TX+RX */
void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1507,6 +1519,11 @@ struct mlx5_priv {
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
struct mlx5_hw_q *hw_q;
+ /* HW steering rte flow table list header. */
+ LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX];
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index e937110054..e4e7ff1875 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -838,6 +838,19 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -864,6 +877,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
.actions_template_create = mlx5_flow_actions_template_create,
.actions_template_destroy = mlx5_flow_actions_template_destroy,
+ .template_table_create = mlx5_flow_table_create,
+ .template_table_destroy = mlx5_flow_table_destroy,
};
/* Tunnel information. */
@@ -8075,6 +8090,84 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
return fops->actions_template_destroy(dev, template, error);
}
+/**
+ * Create flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->template_table_create(dev,
+ attr,
+ item_templates,
+ nb_item_templates,
+ action_templates,
+ nb_action_templates,
+ error);
+}
+
+/**
+ * PMD destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->template_table_destroy(dev, table, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 7e2d6bf4eb..40041a8438 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1036,6 +1036,55 @@ struct rte_flow_actions_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Jump action struct. */
+struct mlx5_hw_jump_action {
+ /* Action jump from root. */
+ struct mlx5dr_action *root_action;
+ /* HW steering jump action. */
+ struct mlx5dr_action *hws_action;
+};
+
+/* DR action set struct. */
+struct mlx5_hw_actions {
+ struct mlx5dr_action *drop; /* Drop action. */
+};
+
+/* mlx5 action template struct. */
+struct mlx5_hw_action_template {
+ /* Action template pointer. */
+ struct rte_flow_actions_template *action_template;
+ struct mlx5_hw_actions acts; /* Template actions. */
+};
+
+/* mlx5 flow group struct. */
+struct mlx5_flow_group {
+ struct mlx5_list_entry entry;
+ struct mlx5dr_table *tbl; /* HWS table object. */
+ struct mlx5_hw_jump_action jump; /* Jump action. */
+ enum mlx5dr_table_type type; /* Table type. */
+ uint32_t group_id; /* Group id. */
+ uint32_t idx; /* Group memory index. */
+};
+
+
+#define MLX5_HW_TBL_MAX_ITEM_TEMPLATE 2
+#define MLX5_HW_TBL_MAX_ACTION_TEMPLATE 32
+
+struct rte_flow_template_table {
+ LIST_ENTRY(rte_flow_template_table) next;
+ struct mlx5_flow_group *grp; /* The group rte_flow_template_table uses. */
+ struct mlx5dr_matcher *matcher; /* Template matcher. */
+ /* Item templates bind to the table. */
+ struct rte_flow_pattern_template *its[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ /* Action templates bind to the table. */
+ struct mlx5_hw_action_template ats[MLX5_HW_TBL_MAX_ACTION_TEMPLATE];
+ struct mlx5_indexed_pool *flow; /* The table's flow ipool. */
+ uint32_t type; /* Flow table type RX/TX/FDB. */
+ uint8_t nb_item_templates; /* Item template number. */
+ uint8_t nb_action_templates; /* Action template number. */
+ uint32_t refcnt; /* Table reference counter. */
+};
+
#endif
/*
@@ -1310,6 +1359,18 @@ typedef int (*mlx5_flow_actions_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_template_table *(*mlx5_flow_table_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_table_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1354,6 +1415,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
mlx5_flow_actions_template_create_t actions_template_create;
mlx5_flow_actions_template_destroy_t actions_template_destroy;
+ mlx5_flow_table_create_t template_table_create;
+ mlx5_flow_table_destroy_t template_table_destroy;
};
/* mlx5_flow.c */
@@ -1789,6 +1852,16 @@ int
flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
struct rte_flow_error *error);
+struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
+void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+int flow_hw_grp_match_cb(void *tool_ctx,
+ struct mlx5_list_entry *entry,
+ void *cb_ctx);
+struct mlx5_list_entry *flow_hw_grp_clone_cb(void *tool_ctx,
+ struct mlx5_list_entry *oentry,
+ void *cb_ctx);
+void flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+
struct mlx5_aso_age_action *flow_aso_age_get_by_idx(struct rte_eth_dev *dev,
uint32_t age_idx);
int flow_dev_geneve_tlv_option_resource_register(struct rte_eth_dev *dev,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 4214a63a73..8cb1ef842a 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,306 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/* DR action flags with different table. */
+static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX] = {
+ {
+ MLX5DR_ACTION_FLAG_ROOT_RX,
+ MLX5DR_ACTION_FLAG_ROOT_TX,
+ MLX5DR_ACTION_FLAG_ROOT_FDB,
+ },
+ {
+ MLX5DR_ACTION_FLAG_HWS_RX,
+ MLX5DR_ACTION_FLAG_HWS_TX,
+ MLX5DR_ACTION_FLAG_HWS_FDB,
+ },
+};
+
+/**
+ * Destroy DR actions created by action template.
+ *
+ * For DR actions created during table creation's action translate.
+ * Need to destroy the DR action when destroying the table.
+ *
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ */
+static void
+__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+{
+}
+
+/**
+ * Translate rte_flow actions to DR action.
+ *
+ * As the action template has already indicated the actions. Translate
+ * the rte_flow actions to DR action if possbile. So in flow create
+ * stage we will save cycles from handing the actions' organizing.
+ * For the actions with limited information, need to add these to a
+ * list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in/out] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] at
+ * Action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *table_attr,
+ struct mlx5_hw_actions *acts,
+ struct rte_flow_actions_template *at,
+ struct rte_flow_error *error __rte_unused)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ const struct rte_flow_attr *attr = &table_attr->flow_attr;
+ struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *masks = at->masks;
+ bool actions_end = false;
+ uint32_t type;
+
+ if (attr->transfer)
+ type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ type = MLX5DR_TABLE_TYPE_NIC_RX;
+ for (; !actions_end; actions++, masks++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ acts->drop = priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ return 0;
+}
+
+/**
+ * Create flow table.
+ *
+ * The input item and action templates will be binded to the table.
+ * Flow memory will also be allocated. Matcher will be created based
+ * on the item template. Action will be translated to the dedicated
+ * DR action if possible.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+flow_hw_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_matcher_attr matcher_attr = {0};
+ struct rte_flow_template_table *tbl = NULL;
+ struct mlx5_flow_group *grp;
+ struct mlx5dr_match_template *mt[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ struct rte_flow_attr flow_attr = attr->flow_attr;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &flow_attr,
+ };
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow),
+ .trunk_size = 1 << 12,
+ .per_core_cache = 1 << 13,
+ .need_lock = 1,
+ .release_mem_en = !!priv->sh->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_table_flow",
+ };
+ struct mlx5_list_entry *ge;
+ uint32_t i, max_tpl = MLX5_HW_TBL_MAX_ITEM_TEMPLATE;
+ uint32_t nb_flows = rte_align32pow2(attr->nb_flows);
+ int err;
+
+ /* HWS layer accepts only 1 item template with root table. */
+ if (!attr->flow_attr.group)
+ max_tpl = 1;
+ cfg.max_idx = nb_flows;
+ /* For table has limited flows, resize the cache and trunk size. */
+ if (nb_flows < cfg.trunk_size) {
+ cfg.per_core_cache = nb_flows >> 2;
+ cfg.trunk_size = nb_flows;
+ }
+ /* Check if we requires too many templates. */
+ if (nb_item_templates > max_tpl ||
+ nb_action_templates > MLX5_HW_TBL_MAX_ACTION_TEMPLATE) {
+ rte_errno = EINVAL;
+ goto error;
+ }
+ /* Allocate the table memory. */
+ tbl = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*tbl), 0, rte_socket_id());
+ if (!tbl)
+ goto error;
+ /* Allocate flow indexed pool. */
+ tbl->flow = mlx5_ipool_create(&cfg);
+ if (!tbl->flow)
+ goto error;
+ /* Register the flow group. */
+ ge = mlx5_hlist_register(priv->sh->groups, attr->flow_attr.group, &ctx);
+ if (!ge)
+ goto error;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ tbl->grp = grp;
+ /* Prepare matcher information. */
+ matcher_attr.priority = attr->flow_attr.priority;
+ matcher_attr.mode = MLX5DR_MATCHER_RESOURCE_MODE_RULE;
+ matcher_attr.rule.num_log = rte_log2_u32(nb_flows);
+ /* Build the item template. */
+ for (i = 0; i < nb_item_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&item_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto it_error;
+ }
+ mt[i] = item_templates[i]->mt;
+ tbl->its[i] = item_templates[i];
+ }
+ tbl->matcher = mlx5dr_matcher_create
+ (tbl->grp->tbl, mt, nb_item_templates, &matcher_attr);
+ if (!tbl->matcher)
+ goto it_error;
+ tbl->nb_item_templates = nb_item_templates;
+ /* Build the action template. */
+ for (i = 0; i < nb_action_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&action_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto at_error;
+ }
+ err = flow_hw_actions_translate(dev, attr,
+ &tbl->ats[i].acts,
+ action_templates[i], error);
+ if (err)
+ goto at_error;
+ tbl->ats[i].action_template = action_templates[i];
+ }
+ tbl->nb_action_templates = nb_action_templates;
+ tbl->type = attr->flow_attr.transfer ? MLX5DR_TABLE_TYPE_FDB :
+ (attr->flow_attr.egress ? MLX5DR_TABLE_TYPE_NIC_TX :
+ MLX5DR_TABLE_TYPE_NIC_RX);
+ LIST_INSERT_HEAD(&priv->flow_hw_tbl, tbl, next);
+ return tbl;
+at_error:
+ while (i--) {
+ __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __atomic_sub_fetch(&action_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ i = nb_item_templates;
+it_error:
+ while (i--)
+ __atomic_sub_fetch(&item_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ mlx5dr_matcher_destroy(tbl->matcher);
+error:
+ err = rte_errno;
+ if (tbl) {
+ if (tbl->grp)
+ mlx5_hlist_unregister(priv->sh->groups,
+ &tbl->grp->entry);
+ if (tbl->flow)
+ mlx5_ipool_destroy(tbl->flow);
+ mlx5_free(tbl);
+ }
+ rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
+ return NULL;
+}
+
+/**
+ * Destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int i;
+
+ if (table->refcnt) {
+ DRV_LOG(WARNING, "Table %p is still in using.", (void *)table);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table in using");
+ }
+ LIST_REMOVE(table, next);
+ for (i = 0; i < table->nb_item_templates; i++)
+ __atomic_sub_fetch(&table->its[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ for (i = 0; i < table->nb_action_templates; i++) {
+ __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __atomic_sub_fetch(&table->ats[i].action_template->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ mlx5dr_matcher_destroy(table->matcher);
+ mlx5_hlist_unregister(priv->sh->groups, &table->grp->entry);
+ mlx5_ipool_destroy(table->flow);
+ mlx5_free(table);
+ return 0;
+}
+
/**
* Create flow action template.
*
@@ -228,6 +528,199 @@ flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
return 0;
}
+/**
+ * Create group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] cb_ctx
+ * Pointer to the group creation context.
+ *
+ * @return
+ * Group entry on success, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct rte_eth_dev *dev = ctx->dev;
+ struct rte_flow_attr *attr = (struct rte_flow_attr *)ctx->data;
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_table_attr dr_tbl_attr = {0};
+ struct rte_flow_error *error = ctx->error;
+ struct mlx5_flow_group *grp_data;
+ struct mlx5dr_table *tbl = NULL;
+ struct mlx5dr_action *jump;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_zmalloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ dr_tbl_attr.level = attr->group;
+ if (attr->transfer)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_RX;
+ tbl = mlx5dr_table_create(priv->dr_ctx, &dr_tbl_attr);
+ if (!tbl)
+ goto error;
+ grp_data->tbl = tbl;
+ if (attr->group) {
+ /* Jump action be used by non-root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[!!attr->group][dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.hws_action = jump;
+ /* Jump action be used by root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_ROOT]
+ [dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.root_action = jump;
+ }
+ grp_data->idx = idx;
+ grp_data->group_id = attr->group;
+ grp_data->type = dr_tbl_attr.type;
+ return &grp_data->entry;
+error:
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (tbl)
+ mlx5dr_table_destroy(tbl);
+ if (idx)
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], idx);
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow dr table");
+ return NULL;
+}
+
+/**
+ * Remove group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the entry to be removed.
+ */
+void
+flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ MLX5_ASSERT(entry && sh);
+ /* To use the wrapper glue functions instead. */
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ mlx5dr_table_destroy(grp_data->tbl);
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
+/**
+ * Match group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+int
+flow_hw_grp_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
+ void *cb_ctx)
+{
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+ struct rte_flow_attr *attr =
+ (struct rte_flow_attr *)ctx->data;
+
+ return (grp_data->group_id != attr->group) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_FDB) &&
+ attr->transfer) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_TX) &&
+ attr->egress) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_RX) &&
+ attr->ingress);
+}
+
+/**
+ * Clone group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_clone_cb(void *tool_ctx, struct mlx5_list_entry *oentry,
+ void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data;
+ struct rte_flow_error *error = ctx->error;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_malloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ memcpy(grp_data, oentry, sizeof(*grp_data));
+ grp_data->idx = idx;
+ return &grp_data->entry;
+}
+
+/**
+ * Free cloned group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be freed.
+ */
+void
+flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
/**
* Configure port HWS resources.
*
@@ -245,6 +738,7 @@ flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
+
static int
flow_hw_configure(struct rte_eth_dev *dev,
const struct rte_flow_port_attr *port_attr,
@@ -321,8 +815,24 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
priv->dr_ctx = dr_ctx;
priv->nb_queue = nb_queue;
+ /* Add global actions. */
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ priv->hw_drop[i][j] = mlx5dr_action_create_dest_drop
+ (priv->dr_ctx, mlx5_hw_act_flag[i][j]);
+ if (!priv->hw_drop[i][j])
+ goto err;
+ }
+ }
return 0;
err:
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
if (dr_ctx)
claim_zero(mlx5dr_context_close(dr_ctx));
mlx5_free(priv->hw_q);
@@ -342,11 +852,17 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_template_table *tbl;
struct rte_flow_pattern_template *it;
struct rte_flow_actions_template *at;
+ int i, j;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_tbl)) {
+ tbl = LIST_FIRST(&priv->flow_hw_tbl);
+ flow_hw_table_destroy(dev, tbl, NULL);
+ }
while (!LIST_EMPTY(&priv->flow_hw_itt)) {
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
@@ -355,6 +871,13 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
at = LIST_FIRST(&priv->flow_hw_at);
flow_hw_actions_template_destroy(dev, at, NULL);
}
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -369,6 +892,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.pattern_template_destroy = flow_hw_pattern_template_destroy,
.actions_template_create = flow_hw_actions_template_create,
.actions_template_destroy = flow_hw_actions_template_destroy,
+ .template_table_create = flow_hw_table_create,
+ .template_table_destroy = flow_hw_table_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 08/14] net/mlx5: add basic flow queue operation
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (6 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 07/14] net/mlx5: add table management Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 09/14] net/mlx5: add flow flush function Suanming Mou
` (5 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The HW steering uses async queue-based flow rules management
mechanism. The matcher and part of the actions have been
prepared during flow table creation. Some remaining actions
will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow management
should be applied to the HW directly. An extra push function
is provided to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function
will be called to get the queued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead
of allocating from HW steering layer. While destroying the
flow, the flow rule memory can only be freed after the CQE
received.
The HW queue job descriptor is currently introduced to convey
the flow information and operation type between the flow
insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:
rte_flow_async_create();
rte_flow_async_destroy();
rte_flow_push();
rte_flow_pull();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
doc/guides/nics/mlx5.rst | 6 +
doc/guides/rel_notes/release_22_03.rst | 6 +
drivers/net/mlx5/mlx5.h | 2 +-
drivers/net/mlx5/mlx5_flow.c | 188 ++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 41 ++++
drivers/net/mlx5/mlx5_flow_hw.c | 292 ++++++++++++++++++++++++-
6 files changed, 533 insertions(+), 2 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index c3f7d61e7d..a1b3161c4a 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -511,6 +511,12 @@ Limitations
from the reference "Clock Queue" completions,
the scheduled send timestamps should not be specified with non-zero MSB.
+ - HW steering:
+
+ - WQE based high scaling and safer flow insertion/destruction.
+ - Set ``dv_flow_en`` to 2 in order to enable HW steering.
+ - Async queue-based ``rte_flow_q`` APIs supported only.
+
Statistics
----------
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index da186315a5..5c39380320 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -156,6 +156,12 @@ New Features
* Added LED OEM support.
+* **Updated Mellanox mlx5 driver.**
+
+ Updated the Mellanox mlx5 driver with new features and improvements, including:
+
+ * Added WQE based hardware steering support with ``rte_flow_q`` API.
+
* **Added an API for private user data in asymmetric crypto session.**
An API was added to get/set an asymmetric crypto session's user data.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 1cc63e1d9f..17aa4b683b 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -340,7 +340,7 @@ enum {
/* HW steering flow management job descriptor. */
struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
- struct rte_flow *flow; /* Flow attached to the job. */
+ struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
};
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index e4e7ff1875..c6224e4f06 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -850,6 +850,34 @@ static int
mlx5_flow_table_destroy(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+static struct rte_flow *
+mlx5_flow_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -879,6 +907,10 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.actions_template_destroy = mlx5_flow_actions_template_destroy,
.template_table_create = mlx5_flow_table_create,
.template_table_destroy = mlx5_flow_table_destroy,
+ .async_create = mlx5_flow_async_flow_create,
+ .async_destroy = mlx5_flow_async_flow_destroy,
+ .pull = mlx5_flow_pull,
+ .push = mlx5_flow_push,
};
/* Tunnel information. */
@@ -8168,6 +8200,162 @@ mlx5_flow_table_destroy(struct rte_eth_dev *dev,
return fops->template_table_destroy(dev, table, error);
}
+/**
+ * Enqueue flow creation.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue_id
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+mlx5_flow_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue_id,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->async_flow_create(dev, queue_id, attr, table,
+ items, pattern_template_index,
+ actions, action_template_index,
+ user_data, error);
+}
+
+/**
+ * Enqueue flow destruction.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->async_flow_destroy(dev, queue, attr, flow,
+ user_data, error);
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q pull with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pull(dev, queue, res, n_res, error);
+}
+
+/**
+ * Push the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flows.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q push with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->push(dev, queue, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 40041a8438..a0cc4028fa 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1017,6 +1017,13 @@ struct rte_flow {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/* HWS flow struct. */
+struct rte_flow_hw {
+ uint32_t idx; /* Flow index from indexed pool. */
+ struct rte_flow_template_table *table; /* The table flow allcated from. */
+ struct mlx5dr_rule rule; /* HWS layer data struct. */
+} __rte_packed;
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1371,6 +1378,34 @@ typedef int (*mlx5_flow_table_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+typedef struct rte_flow *(*mlx5_flow_async_flow_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_async_flow_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_pull_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_push_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1417,6 +1452,10 @@ struct mlx5_flow_driver_ops {
mlx5_flow_actions_template_destroy_t actions_template_destroy;
mlx5_flow_table_create_t template_table_create;
mlx5_flow_table_destroy_t template_table_destroy;
+ mlx5_flow_async_flow_create_t async_flow_create;
+ mlx5_flow_async_flow_destroy_t async_flow_destroy;
+ mlx5_flow_pull_t pull;
+ mlx5_flow_push_t push;
};
/* mlx5_flow.c */
@@ -1587,6 +1626,8 @@ mlx5_translate_tunnel_etypes(uint64_t pattern_flags)
return 0;
}
+int flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error);
int mlx5_flow_group_to_table(struct rte_eth_dev *dev,
const struct mlx5_flow_tunnel *tunnel,
uint32_t group, uint32_t *table,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 8cb1ef842a..accc3a96d9 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -10,6 +10,9 @@
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/* DR action flags with different table. */
@@ -105,6 +108,289 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Construct flow action array.
+ *
+ * For action template contains dynamic actions, these actions need to
+ * be updated according to the rte_flow action during flow creation.
+ *
+ * @param[in] hw_acts
+ * Pointer to translated actions from template.
+ * @param[in] actions
+ * Array of rte_flow action need to be checked.
+ * @param[in] rule_acts
+ * Array of DR rule actions to be used during flow creation..
+ * @param[in] acts_num
+ * Pointer to the real acts_num flow has.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+ const struct rte_flow_action actions[],
+ struct mlx5dr_rule_action *rule_acts,
+ uint32_t *acts_num)
+{
+ bool actions_end = false;
+ uint32_t i;
+
+ for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ rule_acts[i++].action = hw_acts->drop;
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ *acts_num = i;
+ return 0;
+}
+
+/**
+ * Enqueue HW steering flow creation.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow creation status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+flow_hw_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = user_data,
+ .burst = attr->postpone,
+ };
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
+ struct mlx5_hw_actions *hw_acts;
+ struct rte_flow_hw *flow;
+ struct mlx5_hw_q_job *job;
+ uint32_t acts_num, flow_idx;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ flow = mlx5_ipool_zmalloc(table->flow, &flow_idx);
+ if (!flow)
+ goto error;
+ /*
+ * Set the table here in order to know the destination table
+ * when free the flow afterwards.
+ */
+ flow->table = table;
+ flow->idx = flow_idx;
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ /*
+ * Set the job type here in order to know if the flow memory
+ * should be freed or not when get the result from dequeue.
+ */
+ job->type = MLX5_HW_Q_JOB_TYPE_CREATE;
+ job->flow = flow;
+ job->user_data = user_data;
+ rule_attr.user_data = job;
+ hw_acts = &table->ats[action_template_index].acts;
+ /* Construct the flow action array based on the input actions.*/
+ flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ ret = mlx5dr_rule_create(table->matcher,
+ pattern_template_index, items,
+ rule_acts, acts_num,
+ &rule_attr, &flow->rule);
+ if (likely(!ret))
+ return (struct rte_flow *)flow;
+ /* Flow created fail, return the descriptor and flow memory. */
+ mlx5_ipool_free(table->flow, flow_idx);
+ priv->hw_q[queue].job_idx++;
+error:
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+ return NULL;
+}
+
+/**
+ * Enqueue HW steering flow destruction.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow destruction status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = user_data,
+ .burst = attr->postpone,
+ };
+ struct rte_flow_hw *fh = (struct rte_flow_hw *)flow;
+ struct mlx5_hw_q_job *job;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ job->type = MLX5_HW_Q_JOB_TYPE_DESTROY;
+ job->user_data = user_data;
+ job->flow = fh;
+ rule_attr.user_data = job;
+ ret = mlx5dr_rule_destroy(&fh->rule, &rule_attr);
+ if (likely(!ret))
+ return 0;
+ priv->hw_q[queue].job_idx++;
+error:
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * For flows enqueued from creation/destruction, the status should be
+ * checked from the dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q_job *job;
+ int ret, i;
+
+ ret = mlx5dr_send_queue_poll(priv->dr_ctx, queue, res, n_res);
+ if (ret < 0)
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to query flow queue");
+ for (i = 0; i < ret; i++) {
+ job = (struct mlx5_hw_q_job *)res[i].user_data;
+ /* Restore user data. */
+ res[i].user_data = job->user_data;
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
+ }
+ return ret;
+}
+
+/**
+ * Push the enqueued flows to HW.
+ *
+ * Force apply all the enqueued flows to the HW.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flow.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int ret;
+
+ ret = mlx5dr_send_queue_action(priv->dr_ctx, queue,
+ MLX5DR_SEND_QUEUE_ACTION_DRAIN);
+ if (ret) {
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to push flows");
+ return ret;
+ }
+ return 0;
+}
+
/**
* Create flow table.
*
@@ -152,7 +438,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
.data = &flow_attr,
};
struct mlx5_indexed_pool_config cfg = {
- .size = sizeof(struct rte_flow),
+ .size = sizeof(struct rte_flow_hw),
.trunk_size = 1 << 12,
.per_core_cache = 1 << 13,
.need_lock = 1,
@@ -894,6 +1180,10 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.actions_template_destroy = flow_hw_actions_template_destroy,
.template_table_create = flow_hw_table_create,
.template_table_destroy = flow_hw_table_destroy,
+ .async_flow_create = flow_hw_async_flow_create,
+ .async_flow_destroy = flow_hw_async_flow_destroy,
+ .pull = flow_hw_pull,
+ .push = flow_hw_push,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 09/14] net/mlx5: add flow flush function
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (7 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 08/14] net/mlx5: add basic flow queue operation Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 10/14] net/mlx5: add flow jump action Suanming Mou
` (4 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
In case port is being stopped, all created flows should be flushed.
This commit adds the flow flush helper function.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 8 ++
drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++
2 files changed, 137 insertions(+)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index c6224e4f06..ff1c9f0067 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7013,6 +7013,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
uint32_t num_flushed = 0, fidx = 1;
struct rte_flow *flow;
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ if (priv->sh->config.dv_flow_en == 2 &&
+ type == MLX5_FLOW_TYPE_GEN) {
+ flow_hw_q_flow_flush(dev, NULL);
+ return;
+ }
+#endif
+
MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
flow_list_destroy(dev, type, fidx);
num_flushed++;
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index accc3a96d9..ed14eacce2 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -13,6 +13,12 @@
/* The maximum actions support in the flow. */
#define MLX5_HW_MAX_ACTS 16
+/* Default push burst threshold. */
+#define BURST_THR 32u
+
+/* Default queue to flush the flows. */
+#define MLX5_DEFAULT_FLUSH_QUEUE 0
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/* DR action flags with different table. */
@@ -391,6 +397,129 @@ flow_hw_push(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Drain the enqueued flows' completion.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the flow.
+ * @param[in] pending_rules
+ * The pending flow number.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+__flow_hw_pull_comp(struct rte_eth_dev *dev,
+ uint32_t queue,
+ uint32_t pending_rules,
+ struct rte_flow_error *error)
+{
+ struct rte_flow_op_result comp[BURST_THR];
+ int ret, i, empty_loop = 0;
+
+ flow_hw_push(dev, queue, error);
+ while (pending_rules) {
+ ret = flow_hw_pull(dev, queue, comp, BURST_THR, error);
+ if (ret < 0)
+ return -1;
+ if (!ret) {
+ rte_delay_us_sleep(20000);
+ if (++empty_loop > 5) {
+ DRV_LOG(WARNING, "No available dequeue, quit.");
+ break;
+ }
+ continue;
+ }
+ for (i = 0; i < ret; i++) {
+ if (comp[i].status == RTE_FLOW_OP_ERROR)
+ DRV_LOG(WARNING, "Flow flush get error CQE.");
+ }
+ if ((uint32_t)ret > pending_rules) {
+ DRV_LOG(WARNING, "Flow flush get extra CQE.");
+ return rte_flow_error_set(error, ERANGE,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "get extra CQE");
+ }
+ pending_rules -= ret;
+ empty_loop = 0;
+ }
+ return 0;
+}
+
+/**
+ * Flush created flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+int
+flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q *hw_q;
+ struct rte_flow_template_table *tbl;
+ struct rte_flow_hw *flow;
+ struct rte_flow_op_attr attr = {
+ .postpone = 0,
+ };
+ uint32_t pending_rules = 0;
+ uint32_t queue;
+ uint32_t fidx;
+
+ /*
+ * Ensure to push and dequeue all the enqueued flow
+ * creation/destruction jobs in case user forgot to
+ * dequeue. Or the enqueued created flows will be
+ * leaked. The forgotten dequeues would also cause
+ * flow flush get extra CQEs as expected and pending_rules
+ * be minus value.
+ */
+ for (queue = 0; queue < priv->nb_queue; queue++) {
+ hw_q = &priv->hw_q[queue];
+ if (__flow_hw_pull_comp(dev, queue, hw_q->size - hw_q->job_idx,
+ error))
+ return -1;
+ }
+ /* Flush flow per-table from MLX5_DEFAULT_FLUSH_QUEUE. */
+ hw_q = &priv->hw_q[MLX5_DEFAULT_FLUSH_QUEUE];
+ LIST_FOREACH(tbl, &priv->flow_hw_tbl, next) {
+ MLX5_IPOOL_FOREACH(tbl->flow, fidx, flow) {
+ if (flow_hw_async_flow_destroy(dev,
+ MLX5_DEFAULT_FLUSH_QUEUE,
+ &attr,
+ (struct rte_flow *)flow,
+ NULL,
+ error))
+ return -1;
+ pending_rules++;
+ /* Drain completion with queue size. */
+ if (pending_rules >= hw_q->size) {
+ if (__flow_hw_pull_comp(dev,
+ MLX5_DEFAULT_FLUSH_QUEUE,
+ pending_rules, error))
+ return -1;
+ pending_rules = 0;
+ }
+ }
+ }
+ /* Drain left completion. */
+ if (pending_rules &&
+ __flow_hw_pull_comp(dev, MLX5_DEFAULT_FLUSH_QUEUE, pending_rules,
+ error))
+ return -1;
+ return 0;
+}
+
/**
* Create flow table.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 10/14] net/mlx5: add flow jump action
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (8 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 09/14] net/mlx5: add flow flush function Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 11/14] net/mlx5: add queue and RSS action Suanming Mou
` (3 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Jump action connects different level of flow tables and allows packet
handling in the chain of flows.
A new action construct data struct is also added in this commit to help
to handle not only the dynamic jump action but also for the other generic
dynamic actions. The actions with empty mask configuration means dynamic
action, and the dedicated action will be created with the flow action
configuration during flow creation. In that dynamic action case, the action
will be appended to the table template's action list during table creation.
When creating the flows, traverse the action list and pick the dynamic
action configuration details from flow actions as the action construct data
struct describes, then create the dedicated dynamic actions.
This commit adds the jump action and the generic dynamic action construct
mechanism.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 25 ++-
drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
3 files changed, 275 insertions(+), 21 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 17aa4b683b..bb0d07d870 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1524,6 +1524,7 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index a0cc4028fa..675a5bfee5 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1020,10 +1020,25 @@ struct rte_flow {
/* HWS flow struct. */
struct rte_flow_hw {
uint32_t idx; /* Flow index from indexed pool. */
+ uint32_t fate_type; /* Fate action type. */
+ union {
+ /* Jump action. */
+ struct mlx5_hw_jump_action *jump;
+ };
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
} __rte_packed;
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+ LIST_ENTRY(mlx5_action_construct_data) next;
+ /* Ensure the action types are matched. */
+ int type;
+ uint32_t idx; /* Data index. */
+ uint16_t action_src; /* rte_flow_action src offset. */
+ uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
/* DR action set struct. */
struct mlx5_hw_actions {
- struct mlx5dr_action *drop; /* Drop action. */
+ /* Dynamic action list. */
+ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+ struct mlx5_hw_jump_action *jump; /* Jump action. */
+ uint32_t acts_num:4; /* Total action number. */
+ /* Translated DR action array from action template. */
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
/* mlx5 action template struct. */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index ed14eacce2..f320d0db8c 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -36,18 +36,158 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Register destination table DR jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the flow attributes.
+ * @param[in] dest_group
+ * The destination group ID.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct mlx5_hw_jump_action *
+flow_hw_jump_action_register(struct rte_eth_dev *dev,
+ const struct rte_flow_attr *attr,
+ uint32_t dest_group,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_attr jattr = *attr;
+ struct mlx5_flow_group *grp;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &jattr,
+ };
+ struct mlx5_list_entry *ge;
+
+ jattr.group = dest_group;
+ ge = mlx5_hlist_register(priv->sh->flow_tbls, dest_group, &ctx);
+ if (!ge)
+ return NULL;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ return &grp->jump;
+}
+
+/**
+ * Release jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] jump
+ * Pointer to the jump action.
+ */
+
+static void
+flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+}
+
/**
* Destroy DR actions created by action template.
*
* For DR actions created during table creation's action translate.
* Need to destroy the DR action when destroying the table.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
* @param[in] acts
* Pointer to the template HW steering DR actions.
*/
static void
-__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+__flow_hw_action_template_destroy(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (acts->jump) {
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (acts->jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+ acts->jump = NULL;
+ }
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline struct mlx5_action_construct_data *
+__flow_hw_act_data_alloc(struct mlx5_priv *priv,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
+{
+ struct mlx5_action_construct_data *act_data;
+ uint32_t idx = 0;
+
+ act_data = mlx5_ipool_zmalloc(priv->acts_ipool, &idx);
+ if (!act_data)
+ return NULL;
+ act_data->idx = idx;
+ act_data->type = type;
+ act_data->action_src = action_src;
+ act_data->action_dst = action_dst;
+ return act_data;
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_general_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
}
/**
@@ -80,14 +220,16 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
const struct rte_flow_template_table_attr *table_attr,
struct mlx5_hw_actions *acts,
struct rte_flow_actions_template *at,
- struct rte_flow_error *error __rte_unused)
+ struct rte_flow_error *error)
{
struct mlx5_priv *priv = dev->data->dev_private;
const struct rte_flow_attr *attr = &table_attr->flow_attr;
struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
bool actions_end = false;
- uint32_t type;
+ uint32_t type, i;
+ int err;
if (attr->transfer)
type = MLX5DR_TABLE_TYPE_FDB;
@@ -95,14 +237,34 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
type = MLX5DR_TABLE_TYPE_NIC_TX;
else
type = MLX5DR_TABLE_TYPE_NIC_RX;
- for (; !actions_end; actions++, masks++) {
+ for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
case RTE_FLOW_ACTION_TYPE_DROP:
- acts->drop = priv->hw_drop[!!attr->group][type];
+ acts->rule_acts[i++].action =
+ priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ if (masks->conf) {
+ uint32_t jump_group =
+ ((const struct rte_flow_action_jump *)
+ actions->conf)->group;
+ acts->jump = flow_hw_jump_action_register
+ (dev, attr, jump_group, error);
+ if (!acts->jump)
+ goto err;
+ acts->rule_acts[i].action = (!!attr->group) ?
+ acts->jump->hws_action :
+ acts->jump->root_action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
@@ -111,7 +273,14 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ acts->acts_num = i;
return 0;
+err:
+ err = rte_errno;
+ __flow_hw_action_template_destroy(dev, acts);
+ return rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
}
/**
@@ -120,6 +289,10 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* For action template contains dynamic actions, these actions need to
* be updated according to the rte_flow action during flow creation.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] job
+ * Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
* @param[in] actions
@@ -133,31 +306,63 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* 0 on success, negative value otherwise and rte_errno is set.
*/
static __rte_always_inline int
-flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+flow_hw_actions_construct(struct rte_eth_dev *dev,
+ struct mlx5_hw_q_job *job,
+ struct mlx5_hw_actions *hw_acts,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
{
- bool actions_end = false;
- uint32_t i;
+ struct rte_flow_template_table *table = job->flow->table;
+ struct mlx5_action_construct_data *act_data;
+ const struct rte_flow_action *action;
+ struct rte_flow_attr attr = {
+ .ingress = 1,
+ };
- for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
- switch (actions->type) {
+ memcpy(rule_acts, hw_acts->rule_acts,
+ sizeof(*rule_acts) * hw_acts->acts_num);
+ *acts_num = hw_acts->acts_num;
+ if (LIST_EMPTY(&hw_acts->act_list))
+ return 0;
+ attr.group = table->grp->group_id;
+ if (table->type == MLX5DR_TABLE_TYPE_FDB) {
+ attr.transfer = 1;
+ attr.ingress = 1;
+ } else if (table->type == MLX5DR_TABLE_TYPE_NIC_TX) {
+ attr.egress = 1;
+ attr.ingress = 0;
+ } else {
+ attr.ingress = 1;
+ }
+ LIST_FOREACH(act_data, &hw_acts->act_list, next) {
+ uint32_t jump_group;
+ struct mlx5_hw_jump_action *jump;
+
+ action = &actions[act_data->action_src];
+ MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
+ (int)action->type == act_data->type);
+ switch (action->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
- case RTE_FLOW_ACTION_TYPE_DROP:
- rule_acts[i++].action = hw_acts->drop;
- break;
- case RTE_FLOW_ACTION_TYPE_END:
- actions_end = true;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ jump_group = ((const struct rte_flow_action_jump *)
+ action->conf)->group;
+ jump = flow_hw_jump_action_register
+ (dev, &attr, jump_group, NULL);
+ if (!jump)
+ return -1;
+ rule_acts[act_data->action_dst].action =
+ (!!attr.group) ? jump->hws_action : jump->root_action;
+ job->flow->jump = jump;
+ job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
default:
break;
}
}
- *acts_num = i;
return 0;
}
@@ -239,7 +444,8 @@ flow_hw_async_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, actions,
+ rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -356,8 +562,11 @@ flow_hw_pull(struct rte_eth_dev *dev,
job = (struct mlx5_hw_q_job *)res[i].user_data;
/* Restore user data. */
res[i].user_data = job->user_data;
- if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
+ if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
+ flow_hw_jump_release(dev, job->flow->jump);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ }
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
}
return ret;
@@ -642,6 +851,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
rte_errno = EINVAL;
goto at_error;
}
+ LIST_INIT(&tbl->ats[i].acts.act_list);
err = flow_hw_actions_translate(dev, attr,
&tbl->ats[i].acts,
action_templates[i], error);
@@ -657,7 +867,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
return tbl;
at_error:
while (i--) {
- __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &tbl->ats[i].acts);
__atomic_sub_fetch(&action_templates[i]->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -716,7 +926,7 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
- __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -1167,6 +1377,15 @@ flow_hw_configure(struct rte_eth_dev *dev,
struct mlx5_hw_q *hw_q;
struct mlx5_hw_q_job *job = NULL;
uint32_t mem_size, i, j;
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow_hw),
+ .trunk_size = 4096,
+ .need_lock = 1,
+ .release_mem_en = !!priv->sh->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_action_construct_data",
+ };
if (!port_attr || !nb_queue || !queue_attr) {
rte_errno = EINVAL;
@@ -1185,6 +1404,9 @@ flow_hw_configure(struct rte_eth_dev *dev,
}
flow_hw_resource_release(dev);
}
+ priv->acts_ipool = mlx5_ipool_create(&cfg);
+ if (!priv->acts_ipool)
+ goto err;
/* Allocate the queue job descriptor LIFO. */
mem_size = sizeof(priv->hw_q[0]) * nb_queue;
for (i = 0; i < nb_queue; i++) {
@@ -1252,6 +1474,10 @@ flow_hw_configure(struct rte_eth_dev *dev,
claim_zero(mlx5dr_context_close(dr_ctx));
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"fail to configure port");
@@ -1293,6 +1519,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
}
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 11/14] net/mlx5: add queue and RSS action
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (9 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 10/14] net/mlx5: add flow jump action Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 12/14] net/mlx5: add mark action Suanming Mou
` (2 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
This commit adds the queue and RSS action. Similar to the jump action,
dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed
during port restart, the actions are created with standalone indirect
table as indirect action does. When port stops, detaches the indirect
table from action, when port starts, attaches the indirect table back
to the action.
One more change is made to accelerate the action creation. Currently
the mlx5_hrxq_get() function returns the object index instead of object
pointer. This introduced an extra converting the index to the object by
calling mlx5_ipool_get() in most of the case. And that extra converting
hurts multi-thread performance since mlx5_ipool_get() uses the global
lock inside. As the hash Rx queue object itself also contains the index,
returns the object directly will achieve better performance without the
global lock.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 18 ++--
drivers/net/mlx5/mlx5.h | 4 +
drivers/net/mlx5/mlx5_devx.c | 10 ++
drivers/net/mlx5/mlx5_flow.c | 38 +++----
drivers/net/mlx5/mlx5_flow.h | 7 ++
drivers/net/mlx5/mlx5_flow_dv.c | 161 ++++++++++++++---------------
drivers/net/mlx5/mlx5_flow_hw.c | 101 ++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 85 +++++++++------
10 files changed, 283 insertions(+), 157 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 0faf26f5b8..2e1606a733 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1521,6 +1521,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+ mlx5_hrxq_create_cb,
+ mlx5_hrxq_match_cb,
+ mlx5_hrxq_remove_cb,
+ mlx5_hrxq_clone_cb,
+ mlx5_hrxq_clone_free_cb);
+ if (!priv->hrxqs)
+ goto error;
+ rte_rwlock_init(&priv->ind_tbls_lock);
if (priv->sh->config.dv_flow_en == 2)
return eth_dev;
/* Port representor shares the same max priority with pf port. */
@@ -1545,15 +1554,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = ENOTSUP;
goto error;
}
- priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
- mlx5_hrxq_create_cb,
- mlx5_hrxq_match_cb,
- mlx5_hrxq_remove_cb,
- mlx5_hrxq_clone_cb,
- mlx5_hrxq_clone_free_cb);
- if (!priv->hrxqs)
- goto error;
- rte_rwlock_init(&priv->ind_tbls_lock);
/* Query availability of metadata reg_c's. */
if (!priv->sh->metadata_regc_check_flag) {
err = mlx5_flow_discover_mreg_c(eth_dev);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index bb0d07d870..43a913fcc5 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1284,6 +1284,7 @@ struct mlx5_flow_rss_desc {
uint64_t hash_fields; /* Verbs Hash fields. */
uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
uint32_t key_len; /**< RSS hash key len. */
+ uint32_t hws_flags; /**< HW steering action. */
uint32_t tunnel; /**< Queue in tunnel. */
uint32_t shared_rss; /**< Shared RSS index. */
struct mlx5_ind_table_obj *ind_tbl;
@@ -1345,6 +1346,7 @@ struct mlx5_hrxq {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
void *action; /* DV QP action pointer. */
#endif
+ uint32_t hws_flags; /* Hw steering flags. */
uint64_t hash_fields; /* Verbs Hash fields. */
uint32_t rss_key_len; /* Hash key length in bytes. */
uint32_t idx; /* Hash Rx queue index. */
@@ -1475,6 +1477,8 @@ struct mlx5_priv {
LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
/* Indirection tables. */
LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+ /* Standalone indirect tables. */
+ LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
/* Pointer to next element. */
rte_rwlock_t ind_tbls_lock;
uint32_t refcnt; /**< Reference counter. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index e57787cfec..cafd7ead0d 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
goto error;
}
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ if (hrxq->hws_flags) {
+ hrxq->action = mlx5dr_action_create_dest_tir
+ (priv->dr_ctx,
+ (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+ if (!hrxq->action)
+ goto error;
+ return 0;
+ }
if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
&hrxq->action)) {
rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
DRV_LOG(ERR, "Cannot create drop RX queue");
return ret;
}
+ if (priv->sh->config.dv_flow_en == 2)
+ return 0;
/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
if (ret != 0) {
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index ff1c9f0067..0b3134764d 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -9438,14 +9438,10 @@ int
mlx5_action_handle_attach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
const char *message;
uint32_t queue_idx;
@@ -9461,9 +9457,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
}
if (ret != 0)
return ret;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not attach "
@@ -9472,13 +9466,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
goto error;
}
}
+
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not detach "
@@ -9501,15 +9494,10 @@ int
mlx5_action_handle_detach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
-
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not detach "
@@ -9520,11 +9508,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
}
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not attach "
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 675a5bfee5..aa1709e7cb 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {
union {
/* Jump action. */
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq; /* TIR action. */
};
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {
/* Dynamic action list. */
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
+ struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1915,6 +1917,11 @@ int flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
int
flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
struct rte_flow_error *error);
+void flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index c30cb4c203..15d8535e6e 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -10965,78 +10965,83 @@ flow_dv_translate_item_tx_queue(struct rte_eth_dev *dev,
/**
* Set the hash fields according to the @p flow information.
*
- * @param[in] dev_flow
- * Pointer to the mlx5_flow.
+ * @param[in] item_flags
+ * The match pattern item flags.
* @param[in] rss_desc
* Pointer to the mlx5_flow_rss_desc.
+ * @param[out] hash_fields
+ * Pointer to the RSS hash fields.
*/
-static void
-flow_dv_hashfields_set(struct mlx5_flow *dev_flow,
- struct mlx5_flow_rss_desc *rss_desc)
+void
+flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields)
{
- uint64_t items = dev_flow->handle->layers;
+ uint64_t items = item_flags;
+ uint64_t fields = 0;
int rss_inner = 0;
uint64_t rss_types = rte_eth_rss_hf_refine(rss_desc->types);
- dev_flow->hash_fields = 0;
+ *hash_fields = 0;
#ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT
if (rss_desc->level >= 2)
rss_inner = 1;
#endif
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV4)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4)) ||
+ !items) {
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV4;
+ fields |= IBV_RX_HASH_SRC_IPV4;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV4;
+ fields |= IBV_RX_HASH_DST_IPV4;
else
- dev_flow->hash_fields |= MLX5_IPV4_IBV_RX_HASH;
+ fields |= MLX5_IPV4_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV6)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6)) ||
+ !items) {
if (rss_types & MLX5_IPV6_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV6;
+ fields |= IBV_RX_HASH_SRC_IPV6;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV6;
+ fields |= IBV_RX_HASH_DST_IPV6;
else
- dev_flow->hash_fields |= MLX5_IPV6_IBV_RX_HASH;
+ fields |= MLX5_IPV6_IBV_RX_HASH;
}
}
- if (dev_flow->hash_fields == 0)
+ if (fields == 0)
/*
* There is no match between the RSS types and the
* L3 protocol (IPv4/IPv6) defined in the flow rule.
*/
return;
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_UDP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_UDP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_UDP;
+ fields |= IBV_RX_HASH_SRC_PORT_UDP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_UDP;
+ fields |= IBV_RX_HASH_DST_PORT_UDP;
else
- dev_flow->hash_fields |= MLX5_UDP_IBV_RX_HASH;
+ fields |= MLX5_UDP_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_TCP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_TCP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_TCP;
+ fields |= IBV_RX_HASH_SRC_PORT_TCP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_TCP;
+ fields |= IBV_RX_HASH_DST_PORT_TCP;
else
- dev_flow->hash_fields |= MLX5_TCP_IBV_RX_HASH;
+ fields |= MLX5_TCP_IBV_RX_HASH;
}
}
if (rss_inner)
- dev_flow->hash_fields |= IBV_RX_HASH_INNER;
+ fields |= IBV_RX_HASH_INNER;
+ *hash_fields = fields;
}
/**
@@ -11060,7 +11065,6 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc,
uint32_t *hrxq_idx)
{
- struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_handle *dh = dev_flow->handle;
struct mlx5_hrxq *hrxq;
@@ -11071,11 +11075,8 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
rss_desc->shared_rss = 0;
if (rss_desc->hash_fields == 0)
rss_desc->queue_num = 1;
- *hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- if (!*hrxq_idx)
- return NULL;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- *hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
+ *hrxq_idx = hrxq ? hrxq->idx : 0;
return hrxq;
}
@@ -11621,7 +11622,9 @@ flow_dv_translate_action_sample(struct rte_eth_dev *dev,
* rss->level and rss.types should be set in advance
* when expanding items for RSS.
*/
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
hrxq = flow_dv_hrxq_prepare(dev, dev_flow,
rss_desc, &hrxq_idx);
if (!hrxq)
@@ -13645,7 +13648,9 @@ flow_dv_translate(struct rte_eth_dev *dev,
*/
handle->layers |= item_flags;
if (action_flags & MLX5_FLOW_ACTION_RSS)
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
/* If has RSS action in the sample action, the Sample/Mirror resource
* should be registered after the hash filed be update.
*/
@@ -14594,20 +14599,18 @@ __flow_dv_action_rss_hrxqs_release(struct rte_eth_dev *dev,
* MLX5_RSS_HASH_IPV4_DST_ONLY are mutually exclusive so they can share
* same slot in mlx5_rss_hash_fields.
*
- * @param[in] rss
- * Pointer to the shared action RSS conf.
+ * @param[in] rss_types
+ * RSS type.
* @param[in, out] hash_field
* hash_field variable needed to be adjusted.
*
* @return
* void
*/
-static void
-__flow_dv_action_rss_l34_hash_adjust(struct mlx5_shared_action_rss *rss,
- uint64_t *hash_field)
+void
+flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field)
{
- uint64_t rss_types = rss->origin.types;
-
switch (*hash_field & ~IBV_RX_HASH_INNER) {
case MLX5_RSS_HASH_IPV4:
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
@@ -14690,12 +14693,15 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
size_t i;
int err;
- if (mlx5_ind_table_obj_setup(dev, shared_rss->ind_tbl,
- !!dev->data->dev_started)) {
+ shared_rss->ind_tbl = mlx5_ind_table_obj_new
+ (dev, shared_rss->origin.queue,
+ shared_rss->origin.queue_num,
+ true,
+ !!dev->data->dev_started);
+ if (!shared_rss->ind_tbl)
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot setup indirection table");
- }
memcpy(rss_desc.key, shared_rss->origin.key, MLX5_RSS_HASH_KEY_LEN);
rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
rss_desc.const_q = shared_rss->origin.queue;
@@ -14704,19 +14710,20 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
- uint32_t hrxq_idx;
+ struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
int tunnel = 0;
- __flow_dv_action_rss_l34_hash_adjust(shared_rss, &hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(shared_rss->origin.types,
+ &hash_fields);
if (shared_rss->origin.level > 1) {
hash_fields |= IBV_RX_HASH_INNER;
tunnel = 1;
}
rss_desc.tunnel = tunnel;
rss_desc.hash_fields = hash_fields;
- hrxq_idx = mlx5_hrxq_get(dev, &rss_desc);
- if (!hrxq_idx) {
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
@@ -14724,14 +14731,14 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
goto error_hrxq_new;
}
err = __flow_dv_action_rss_hrxq_set
- (shared_rss, hash_fields, hrxq_idx);
+ (shared_rss, hash_fields, hrxq->idx);
MLX5_ASSERT(!err);
}
return 0;
error_hrxq_new:
err = rte_errno;
__flow_dv_action_rss_hrxqs_release(dev, shared_rss);
- if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true, true))
+ if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true))
shared_rss->ind_tbl = NULL;
rte_errno = err;
return -rte_errno;
@@ -14762,18 +14769,14 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss = NULL;
- void *queue = NULL;
struct rte_flow_action_rss *origin;
const uint8_t *rss_key;
- uint32_t queue_size = rss->queue_num * sizeof(uint16_t);
uint32_t idx;
RTE_SET_USED(conf);
- queue = mlx5_malloc(0, RTE_ALIGN_CEIL(queue_size, sizeof(void *)),
- 0, SOCKET_ID_ANY);
shared_rss = mlx5_ipool_zmalloc
(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], &idx);
- if (!shared_rss || !queue) {
+ if (!shared_rss) {
rte_flow_error_set(error, ENOMEM,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot allocate resource memory");
@@ -14785,18 +14788,6 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
"rss action number out of range");
goto error_rss_init;
}
- shared_rss->ind_tbl = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(*shared_rss->ind_tbl),
- 0, SOCKET_ID_ANY);
- if (!shared_rss->ind_tbl) {
- rte_flow_error_set(error, ENOMEM,
- RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
- "cannot allocate resource memory");
- goto error_rss_init;
- }
- memcpy(queue, rss->queue, queue_size);
- shared_rss->ind_tbl->queues = queue;
- shared_rss->ind_tbl->queues_n = rss->queue_num;
origin = &shared_rss->origin;
origin->func = rss->func;
origin->level = rss->level;
@@ -14807,10 +14798,12 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
memcpy(shared_rss->key, rss_key, MLX5_RSS_HASH_KEY_LEN);
origin->key = &shared_rss->key[0];
origin->key_len = MLX5_RSS_HASH_KEY_LEN;
- origin->queue = queue;
+ origin->queue = rss->queue;
origin->queue_num = rss->queue_num;
if (__flow_dv_action_rss_setup(dev, idx, shared_rss, error))
goto error_rss_init;
+ /* Update queue with indirect table queue memoyr. */
+ origin->queue = shared_rss->ind_tbl->queues;
rte_spinlock_init(&shared_rss->action_rss_sl);
__atomic_add_fetch(&shared_rss->refcnt, 1, __ATOMIC_RELAXED);
rte_spinlock_lock(&priv->shared_act_sl);
@@ -14821,12 +14814,11 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
error_rss_init:
if (shared_rss) {
if (shared_rss->ind_tbl)
- mlx5_free(shared_rss->ind_tbl);
+ mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
+ !!dev->data->dev_started);
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
idx);
}
- if (queue)
- mlx5_free(queue);
return 0;
}
@@ -14854,7 +14846,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
uint32_t old_refcnt = 1;
int remaining;
- uint16_t *queue = NULL;
if (!shared_rss)
return rte_flow_error_set(error, EINVAL,
@@ -14873,8 +14864,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
RTE_FLOW_ERROR_TYPE_ACTION,
NULL,
"shared rss hrxq has references");
- queue = shared_rss->ind_tbl->queues;
- remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true,
+ remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
!!dev->data->dev_started);
if (remaining)
return rte_flow_error_set(error, EBUSY,
@@ -14882,7 +14872,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
NULL,
"shared rss indirection table has"
" references");
- mlx5_free(queue);
rte_spinlock_lock(&priv->shared_act_sl);
ILIST_REMOVE(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
&priv->rss_shared_actions, idx, shared_rss, next);
@@ -15061,7 +15050,7 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx,
mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
int ret = 0;
void *queue = NULL;
- uint16_t *queue_old = NULL;
+ void *queue_i = NULL;
uint32_t queue_size = action_conf->queue_num * sizeof(uint16_t);
bool dev_started = !!dev->data->dev_started;
@@ -15084,22 +15073,23 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx,
memcpy(queue, action_conf->queue, queue_size);
MLX5_ASSERT(shared_rss->ind_tbl);
rte_spinlock_lock(&shared_rss->action_rss_sl);
- queue_old = shared_rss->ind_tbl->queues;
+ queue_i = shared_rss->ind_tbl->queues;
ret = mlx5_ind_table_obj_modify(dev, shared_rss->ind_tbl,
queue, action_conf->queue_num,
true /* standalone */,
dev_started /* ref_new_qs */,
dev_started /* deref_old_qs */);
if (ret) {
- mlx5_free(queue);
ret = rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_ACTION, NULL,
"cannot update indirection table");
} else {
- mlx5_free(queue_old);
- shared_rss->origin.queue = queue;
+ /* Restore the queue to indirect table internal queue. */
+ memcpy(queue_i, queue, queue_size);
+ shared_rss->ind_tbl->queues = queue_i;
shared_rss->origin.queue_num = action_conf->queue_num;
}
+ mlx5_free(queue);
rte_spinlock_unlock(&shared_rss->action_rss_sl);
return ret;
}
@@ -16877,11 +16867,12 @@ __flow_dv_meter_get_rss_sub_policy(struct rte_eth_dev *dev,
for (i = 0; i < MLX5_MTR_RTE_COLORS; i++) {
if (!rss_desc[i])
continue;
- hrxq_idx[i] = mlx5_hrxq_get(dev, rss_desc[i]);
- if (!hrxq_idx[i]) {
+ hrxq = mlx5_hrxq_get(dev, rss_desc[i]);
+ if (!hrxq) {
rte_spinlock_unlock(&mtr_policy->sl);
return NULL;
}
+ hrxq_idx[i] = hrxq->idx;
}
sub_policy_num = (mtr_policy->sub_policy_num >>
(MLX5_MTR_SUB_POLICY_NUM_SHIFT * domain)) &
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index f320d0db8c..0d49ab0bb2 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -7,6 +7,7 @@
#include <mlx5_malloc.h>
#include "mlx5_defs.h"
#include "mlx5_flow.h"
+#include "mlx5_rx.h"
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
@@ -95,6 +96,56 @@ flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
}
+/**
+ * Register queue/RSS action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] hws_flags
+ * DR action flags.
+ * @param[in] action
+ * rte flow action.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static inline struct mlx5_hrxq*
+flow_hw_tir_action_register(struct rte_eth_dev *dev,
+ uint32_t hws_flags,
+ const struct rte_flow_action *action)
+{
+ struct mlx5_flow_rss_desc rss_desc = {
+ .hws_flags = hws_flags,
+ };
+ struct mlx5_hrxq *hrxq;
+
+ if (action->type == RTE_FLOW_ACTION_TYPE_QUEUE) {
+ const struct rte_flow_action_queue *queue = action->conf;
+
+ rss_desc.const_q = &queue->index;
+ rss_desc.queue_num = 1;
+ } else {
+ const struct rte_flow_action_rss *rss = action->conf;
+
+ rss_desc.queue_num = rss->queue_num;
+ rss_desc.const_q = rss->queue;
+ memcpy(rss_desc.key,
+ !rss->key ? rss_hash_default_key : rss->key,
+ MLX5_RSS_HASH_KEY_LEN);
+ rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
+ rss_desc.types = !rss->types ? RTE_ETH_RSS_IP : rss->types;
+ flow_dv_hashfields_set(0, &rss_desc, &rss_desc.hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(rss->types,
+ &rss_desc.hash_fields);
+ if (rss->level > 1) {
+ rss_desc.hash_fields |= IBV_RX_HASH_INNER;
+ rss_desc.tunnel = 1;
+ }
+ }
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ return hrxq;
+}
+
/**
* Destroy DR actions created by action template.
*
@@ -266,6 +317,40 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -319,6 +404,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_attr attr = {
.ingress = 1,
};
+ uint32_t ft_flag;
memcpy(rule_acts, hw_acts->rule_acts,
sizeof(*rule_acts) * hw_acts->acts_num);
@@ -326,6 +412,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
if (LIST_EMPTY(&hw_acts->act_list))
return 0;
attr.group = table->grp->group_id;
+ ft_flag = mlx5_hw_act_flag[!!table->grp->group_id][table->type];
if (table->type == MLX5DR_TABLE_TYPE_FDB) {
attr.transfer = 1;
attr.ingress = 1;
@@ -338,6 +425,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
@@ -359,6 +447,17 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->jump = jump;
job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ hrxq = flow_hw_tir_action_register(dev,
+ ft_flag,
+ action);
+ if (!hrxq)
+ return -1;
+ rule_acts[act_data->action_dst].action = hrxq->action;
+ job->flow->hrxq = hrxq;
+ job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
+ break;
default:
break;
}
@@ -565,6 +664,8 @@ flow_hw_pull(struct rte_eth_dev *dev,
if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
flow_hw_jump_release(dev, job->flow->jump);
+ else if (job->flow->fate_type == MLX5_FLOW_FATE_QUEUE)
+ mlx5_hrxq_obj_release(dev, job->flow->hrxq);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
}
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c
index 90ccb9aaff..f08aa7a770 100644
--- a/drivers/net/mlx5/mlx5_flow_verbs.c
+++ b/drivers/net/mlx5/mlx5_flow_verbs.c
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
MLX5_ASSERT(priv->drop_queue.hrxq);
hrxq = priv->drop_queue.hrxq;
} else {
- uint32_t hrxq_idx;
struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
rss_desc->tunnel = !!(handle->layers &
MLX5_FLOW_LAYER_TUNNEL);
rss_desc->shared_rss = 0;
- hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
"cannot get hash queue");
goto error;
}
- handle->rix_hrxq = hrxq_idx;
+ handle->rix_hrxq = hrxq->idx;
}
MLX5_ASSERT(hrxq);
handle->drv_flow = mlx5_glue->create_flow
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 7e417819f7..221f4ded97 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -225,9 +225,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
const uint16_t *queues,
uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+ const uint16_t *queues,
+ uint32_t queues_n,
+ bool standalone,
+ bool ref_qs);
int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs);
int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
@@ -250,8 +254,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
void *cb_ctx __rte_unused);
void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 2625fa3308..acd24ae494 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -2285,8 +2285,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
* Pointer to Ethernet device.
* @param ind_table
* Indirection table to release.
- * @param standalone
- * Indirection table for Standalone queue.
* @param deref_rxqs
* If true, then dereference RX queues related to indirection table.
* Otherwise, no additional action will be taken.
@@ -2297,7 +2295,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
int
mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs)
{
struct mlx5_priv *priv = dev->data->dev_private;
@@ -2305,7 +2302,7 @@ mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
rte_rwlock_write_lock(&priv->ind_tbls_lock);
ret = __atomic_sub_fetch(&ind_tbl->refcnt, 1, __ATOMIC_RELAXED);
- if (!ret && !standalone)
+ if (!ret)
LIST_REMOVE(ind_tbl, next);
rte_rwlock_write_unlock(&priv->ind_tbls_lock);
if (ret)
@@ -2414,7 +2411,7 @@ mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
* @return
* The Verbs/DevX object initialized, NULL otherwise and rte_errno is set.
*/
-static struct mlx5_ind_table_obj *
+struct mlx5_ind_table_obj *
mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
uint32_t queues_n, bool standalone, bool ref_qs)
{
@@ -2422,8 +2419,13 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
struct mlx5_ind_table_obj *ind_tbl;
int ret;
+ /*
+ * Allocate maximum queues for shared action as queue number
+ * maybe modified later.
+ */
ind_tbl = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*ind_tbl) +
- queues_n * sizeof(uint16_t), 0, SOCKET_ID_ANY);
+ (standalone ? priv->rxqs_n : queues_n) *
+ sizeof(uint16_t), 0, SOCKET_ID_ANY);
if (!ind_tbl) {
rte_errno = ENOMEM;
return NULL;
@@ -2436,11 +2438,13 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
mlx5_free(ind_tbl);
return NULL;
}
- if (!standalone) {
- rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ if (!standalone)
LIST_INSERT_HEAD(&priv->ind_tbls, ind_tbl, next);
- rte_rwlock_write_unlock(&priv->ind_tbls_lock);
- }
+ else
+ LIST_INSERT_HEAD(&priv->standalone_ind_tbls, ind_tbl, next);
+ rte_rwlock_write_unlock(&priv->ind_tbls_lock);
+
return ind_tbl;
}
@@ -2606,6 +2610,7 @@ mlx5_hrxq_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
return (hrxq->rss_key_len != rss_desc->key_len ||
memcmp(hrxq->rss_key, rss_desc->key, rss_desc->key_len) ||
+ hrxq->hws_flags != rss_desc->hws_flags ||
hrxq->hash_fields != rss_desc->hash_fields ||
hrxq->ind_table->queues_n != rss_desc->queue_num ||
memcmp(hrxq->ind_table->queues, rss_desc->queue,
@@ -2690,8 +2695,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
}
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ mlx5_ind_table_obj_release(dev, hrxq->ind_table, true);
hrxq->ind_table = ind_tbl;
}
hrxq->hash_fields = hash_fields;
@@ -2701,8 +2705,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
err = rte_errno;
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, ind_tbl, hrxq->standalone,
- true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
}
rte_errno = err;
return -rte_errno;
@@ -2714,12 +2717,16 @@ __mlx5_hrxq_remove(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
struct mlx5_priv *priv = dev->data->dev_private;
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- mlx5_glue->destroy_flow_action(hrxq->action);
+ if (hrxq->hws_flags)
+ mlx5dr_action_destroy(hrxq->action);
+ else
+ mlx5_glue->destroy_flow_action(hrxq->action);
#endif
priv->obj_ops.hrxq_destroy(hrxq);
if (!hrxq->standalone) {
mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ hrxq->hws_flags ?
+ (!!dev->data->dev_started) : true);
}
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq->idx);
}
@@ -2763,11 +2770,12 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
int ret;
queues_n = rss_desc->hash_fields ? queues_n : 1;
- if (!ind_tbl)
+ if (!ind_tbl && !rss_desc->hws_flags)
ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n);
if (!ind_tbl)
ind_tbl = mlx5_ind_table_obj_new(dev, queues, queues_n,
- standalone,
+ standalone ||
+ rss_desc->hws_flags,
!!dev->data->dev_started);
if (!ind_tbl)
return NULL;
@@ -2779,6 +2787,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
hrxq->ind_table = ind_tbl;
hrxq->rss_key_len = rss_key_len;
hrxq->hash_fields = rss_desc->hash_fields;
+ hrxq->hws_flags = rss_desc->hws_flags;
memcpy(hrxq->rss_key, rss_key, rss_key_len);
ret = priv->obj_ops.hrxq_new(dev, hrxq, rss_desc->tunnel);
if (ret < 0)
@@ -2786,7 +2795,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
return hrxq;
error:
if (!rss_desc->ind_tbl)
- mlx5_ind_table_obj_release(dev, ind_tbl, standalone, true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
if (hrxq)
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
return NULL;
@@ -2840,13 +2849,13 @@ mlx5_hrxq_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
* RSS configuration for the Rx hash queue.
*
* @return
- * An hash Rx queue index on success.
+ * An hash Rx queue on success.
*/
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
+ struct mlx5_hrxq *hrxq = NULL;
struct mlx5_list_entry *entry;
struct mlx5_flow_cb_ctx ctx = {
.data = rss_desc,
@@ -2857,12 +2866,10 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
} else {
entry = mlx5_list_register(priv->hrxqs, &ctx);
if (!entry)
- return 0;
+ return NULL;
hrxq = container_of(entry, typeof(*hrxq), entry);
}
- if (hrxq)
- return hrxq->idx;
- return 0;
+ return hrxq;
}
/**
@@ -2871,17 +2878,15 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
* @param dev
* Pointer to Ethernet device.
* @param hrxq_idx
- * Index to Hash Rx queue to release.
+ * Hash Rx queue to release.
*
* @return
* 1 while a reference on it exists, 0 when freed.
*/
-int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
if (!hrxq)
return 0;
if (!hrxq->standalone)
@@ -2890,6 +2895,26 @@ int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
return 0;
}
+/**
+ * Release the hash Rx queue with index.
+ *
+ * @param dev
+ * Pointer to Ethernet device.
+ * @param hrxq_idx
+ * Index to Hash Rx queue to release.
+ *
+ * @return
+ * 1 while a reference on it exists, 0 when freed.
+ */
+int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hrxq *hrxq;
+
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
+ return mlx5_hrxq_obj_release(dev, hrxq);
+}
+
/**
* Create a drop Rx Hash queue.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 12/14] net/mlx5: add mark action
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (10 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 11/14] net/mlx5: add queue and RSS action Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 13/14] net/mlx5: add indirect action Suanming Mou
2022-02-22 8:51 ` [PATCH v2 14/14] net/mlx5: add header reformat action Suanming Mou
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The mark action is covered by tag action internally. While it is added
the HW will add a tag to the packet. The mark value can be set as fixed
or dynamic as the action mask indicates.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++---
3 files changed, 63 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 43a913fcc5..e78eb5e380 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1528,6 +1528,8 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX];
struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index aa1709e7cb..ec759c1aa4 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
+ uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 0d49ab0bb2..a28e3c00b3 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -37,6 +37,31 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Set rxq flag.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] enable
+ * Flag to enable or not.
+ */
+static void
+flow_hw_rxq_flag_set(struct rte_eth_dev *dev, bool enable)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ unsigned int i;
+
+ if ((!priv->mark_enabled && !enable) ||
+ (priv->mark_enabled && enable))
+ return;
+ for (i = 0; i < priv->rxqs_n; ++i) {
+ struct mlx5_rxq_ctrl *rxq_ctrl = mlx5_rxq_ctrl_get(dev, i);
+
+ rxq_ctrl->rxq.mark = enable;
+ }
+ priv->mark_enabled = enable;
+}
+
/**
* Register destination table DR jump action.
*
@@ -298,6 +323,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
acts->rule_acts[i++].action =
priv->hw_drop[!!attr->group][type];
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ acts->mark = true;
+ if (masks->conf)
+ acts->rule_acts[i].tag.value =
+ mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (masks->conf))->id);
+ else if (__flow_hw_act_data_general_append(priv, acts,
+ actions->type, actions - action_start, i))
+ goto err;
+ acts->rule_acts[i++].action =
+ priv->hw_tag[!!attr->group];
+ flow_hw_rxq_flag_set(dev, true);
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
if (masks->conf) {
uint32_t jump_group =
@@ -424,6 +463,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
}
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
+ uint32_t tag;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
@@ -435,6 +475,12 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ tag = mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (action->conf))->id);
+ rule_acts[act_data->action_dst].tag.value = tag;
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
jump_group = ((const struct rte_flow_action_jump *)
action->conf)->group;
@@ -1027,6 +1073,8 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
+ if (table->ats[i].acts.mark)
+ flow_hw_rxq_flag_set(dev, false);
__flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
@@ -1561,15 +1609,20 @@ flow_hw_configure(struct rte_eth_dev *dev,
if (!priv->hw_drop[i][j])
goto err;
}
+ priv->hw_tag[i] = mlx5dr_action_create_tag
+ (priv->dr_ctx, mlx5_hw_act_flag[i][0]);
+ if (!priv->hw_tag[i])
+ goto err;
}
return 0;
err:
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
+ if (priv->hw_tag[i])
+ mlx5dr_action_destroy(priv->hw_tag[i]);
}
if (dr_ctx)
claim_zero(mlx5dr_context_close(dr_ctx));
@@ -1615,10 +1668,11 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
}
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
+ if (priv->hw_tag[i])
+ mlx5dr_action_destroy(priv->hw_tag[i]);
}
if (priv->acts_ipool) {
mlx5_ipool_destroy(priv->acts_ipool);
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 13/14] net/mlx5: add indirect action
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (11 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 12/14] net/mlx5: add mark action Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
2022-02-22 8:51 ` [PATCH v2 14/14] net/mlx5: add header reformat action Suanming Mou
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering can support indirect action as well. With indirect action,
the flow can be created with more flexible shared RSS action selection.
This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:
rte_flow_async_action_handle_create();
rte_flow_async_action_handle_destroy();
rte_flow_async_action_handle_update();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 131 ++++++++++
drivers/net/mlx5/mlx5_flow.h | 59 +++++
drivers/net/mlx5/mlx5_flow_dv.c | 21 +-
drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++-
4 files changed, 612 insertions(+), 13 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 0b3134764d..bb4d2f6bae 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,
uint32_t queue,
struct rte_flow_error *error);
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.async_destroy = mlx5_flow_async_flow_destroy,
.pull = mlx5_flow_pull,
.push = mlx5_flow_push,
+ .async_action_handle_create = mlx5_flow_async_action_handle_create,
+ .async_action_handle_update = mlx5_flow_async_action_handle_update,
+ .async_action_handle_destroy = mlx5_flow_async_action_handle_destroy,
};
/* Tunnel information. */
@@ -8364,6 +8390,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,
return fops->push(dev, queue, error);
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_create(dev, queue, attr, conf, action,
+ user_data, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_update(dev, queue, attr, handle,
+ update, user_data, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_destroy(dev, queue, attr, handle,
+ user_data, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index ec759c1aa4..9ac6745597 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -41,6 +41,7 @@ enum mlx5_rte_flow_action_type {
MLX5_RTE_FLOW_ACTION_TYPE_AGE,
MLX5_RTE_FLOW_ACTION_TYPE_COUNT,
MLX5_RTE_FLOW_ACTION_TYPE_JUMP,
+ MLX5_RTE_FLOW_ACTION_TYPE_RSS,
};
#define MLX5_INDIRECT_ACTION_TYPE_OFFSET 30
@@ -1038,6 +1039,13 @@ struct mlx5_action_construct_data {
uint32_t idx; /* Data index. */
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+ union {
+ struct {
+ uint64_t types; /* RSS hash types. */
+ uint32_t level; /* RSS level. */
+ uint32_t idx; /* Shared action index. */
+ } shared_rss;
+ };
};
/* Flow item template struct. */
@@ -1046,6 +1054,7 @@ struct rte_flow_pattern_template {
/* Template attributes. */
struct rte_flow_pattern_template_attr attr;
struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint64_t item_flags; /* Item layer flags. */
uint32_t refcnt; /* Reference counter. */
};
@@ -1433,6 +1442,32 @@ typedef int (*mlx5_flow_push_t)
uint32_t queue,
struct rte_flow_error *error);
+typedef struct rte_flow_action_handle *(*mlx5_flow_async_action_handle_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_async_action_handle_update_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_async_action_handle_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error);
+
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
mlx5_flow_prepare_t prepare;
@@ -1482,6 +1517,9 @@ struct mlx5_flow_driver_ops {
mlx5_flow_async_flow_destroy_t async_flow_destroy;
mlx5_flow_pull_t pull;
mlx5_flow_push_t push;
+ mlx5_flow_async_action_handle_create_t async_action_create;
+ mlx5_flow_async_action_handle_update_t async_action_update;
+ mlx5_flow_async_action_handle_destroy_t async_action_destroy;
};
/* mlx5_flow.c */
@@ -1923,6 +1961,8 @@ void flow_dv_hashfields_set(uint64_t item_flags,
uint64_t *hash_fields);
void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
uint64_t *hash_field);
+uint32_t flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
@@ -1973,4 +2013,23 @@ mlx5_get_tof(const struct rte_flow_item *items,
enum mlx5_tof_rule_type *rule_type);
void
flow_hw_resource_release(struct rte_eth_dev *dev);
+int flow_dv_action_validate(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+struct rte_flow_action_handle *flow_dv_action_create(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+int flow_dv_action_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error);
+int flow_dv_action_update(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *err);
+int flow_dv_action_query(struct rte_eth_dev *dev,
+ const struct rte_flow_action_handle *handle,
+ void *data,
+ struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 15d8535e6e..b49b88a13f 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -13835,9 +13835,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
* @return
* Valid hash RX queue index, otherwise 0.
*/
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
- const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss =
@@ -13965,7 +13965,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
struct mlx5_hrxq *hrxq = NULL;
uint32_t hrxq_idx;
- hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
rss_desc->shared_rss,
dev_flow->hash_fields);
if (hrxq_idx)
@@ -14689,6 +14689,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
struct mlx5_shared_action_rss *shared_rss,
struct rte_flow_error *error)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_rss_desc rss_desc = { 0 };
size_t i;
int err;
@@ -14709,6 +14710,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
/* Set non-zero value to indicate a shared RSS. */
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
+ if (priv->sh->config.dv_flow_en == 2)
+ rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14900,7 +14903,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
* A valid shared action handle in case of success, NULL otherwise and
* rte_errno is set.
*/
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
flow_dv_action_create(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
@@ -14970,7 +14973,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_destroy(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
struct rte_flow_error *error)
@@ -15180,7 +15183,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_update(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
const void *update,
@@ -15894,7 +15897,7 @@ flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
"counters are not available");
}
-static int
+int
flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle, void *data,
struct rte_flow_error *error)
@@ -17584,7 +17587,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_validate(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index a28e3c00b3..95df6e5190 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -62,6 +62,72 @@ flow_hw_rxq_flag_set(struct rte_eth_dev *dev, bool enable)
priv->mark_enabled = enable;
}
+/**
+ * Generate the pattern item flags.
+ * Will be used for shared RSS action.
+ *
+ * @param[in] items
+ * Pointer to the list of items.
+ *
+ * @return
+ * Item flags.
+ */
+static uint64_t
+flow_hw_rss_item_flags_get(const struct rte_flow_item items[])
+{
+ uint64_t item_flags = 0;
+ uint64_t last_item = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) {
+ int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL);
+ int item_type = items->type;
+
+ switch (item_type) {
+ case RTE_FLOW_ITEM_TYPE_IPV4:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV4;
+ break;
+ case RTE_FLOW_ITEM_TYPE_IPV6:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV6;
+ break;
+ case RTE_FLOW_ITEM_TYPE_TCP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP :
+ MLX5_FLOW_LAYER_OUTER_L4_TCP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_UDP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP :
+ MLX5_FLOW_LAYER_OUTER_L4_UDP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_NVGRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN:
+ last_item = MLX5_FLOW_LAYER_VXLAN;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN_GPE:
+ last_item = MLX5_FLOW_LAYER_VXLAN_GPE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GENEVE:
+ last_item = MLX5_FLOW_LAYER_GENEVE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_MPLS:
+ last_item = MLX5_FLOW_LAYER_MPLS;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GTP:
+ last_item = MLX5_FLOW_LAYER_GTP;
+ break;
+ default:
+ break;
+ }
+ item_flags |= last_item;
+ }
+ return item_flags;
+}
+
/**
* Register destination table DR jump action.
*
@@ -266,6 +332,96 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append shared RSS action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] idx
+ * Shared RSS index.
+ * @param[in] rss
+ * Pointer to the shared RSS info.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_shared_rss_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint32_t idx,
+ struct mlx5_shared_action_rss *rss)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->shared_rss.level = rss->origin.level;
+ act_data->shared_rss.types = !rss->origin.types ? RTE_ETH_RSS_IP :
+ rss->origin.types;
+ act_data->shared_rss.idx = idx;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
+/**
+ * Translate shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct mlx5_hw_actions *acts,
+ uint16_t action_src,
+ uint16_t action_dst)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss || __flow_hw_act_data_shared_rss_append
+ (priv, acts,
+ (enum rte_flow_action_type)MLX5_RTE_FLOW_ACTION_TYPE_RSS,
+ action_src, action_dst, idx, shared_rss))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -316,6 +472,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (!attr->group) {
+ DRV_LOG(ERR, "Indirect action is not supported in root table.");
+ goto err;
+ }
+ if (actions->conf && masks->conf) {
+ if (flow_hw_shared_action_translate
+ (dev, actions, acts, actions - action_start, i))
+ goto err;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -407,6 +577,115 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
"fail to create rte table");
}
+/**
+ * Get shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] act_data
+ * Pointer to the recorded action construct data.
+ * @param[in] item_flags
+ * The matcher itme_flags used for RSS lookup.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_get(struct rte_eth_dev *dev,
+ struct mlx5_action_construct_data *act_data,
+ const uint64_t item_flags,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_rss_desc rss_desc = { 0 };
+ uint64_t hash_fields = 0;
+ uint32_t hrxq_idx = 0;
+ struct mlx5_hrxq *hrxq = NULL;
+ int act_type = act_data->type;
+
+ switch (act_type) {
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ rss_desc.level = act_data->shared_rss.level;
+ rss_desc.types = act_data->shared_rss.types;
+ flow_dv_hashfields_set(item_flags, &rss_desc, &hash_fields);
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup
+ (dev, act_data->shared_rss.idx, hash_fields);
+ if (hrxq_idx)
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
+ hrxq_idx);
+ if (hrxq) {
+ rule_act->action = hrxq->action;
+ return 0;
+ }
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d",
+ act_data->type);
+ break;
+ }
+ return -1;
+}
+
+/**
+ * Construct shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] table
+ * Pointer to the flow table.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_construct(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct rte_flow_template_table *table,
+ const uint8_t it_idx,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_action_construct_data act_data;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+ uint64_t item_flags;
+
+ memset(&act_data, 0, sizeof(act_data));
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ act_data.type = MLX5_RTE_FLOW_ACTION_TYPE_RSS;
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss)
+ return -1;
+ act_data.shared_rss.idx = idx;
+ act_data.shared_rss.level = shared_rss->origin.level;
+ act_data.shared_rss.types = !shared_rss->origin.types ?
+ RTE_ETH_RSS_IP :
+ shared_rss->origin.types;
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, &act_data, item_flags, rule_act))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Construct flow action array.
*
@@ -419,6 +698,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
* @param[in] actions
* Array of rte_flow action need to be checked.
* @param[in] rule_acts
@@ -432,7 +713,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
static __rte_always_inline int
flow_hw_actions_construct(struct rte_eth_dev *dev,
struct mlx5_hw_q_job *job,
- struct mlx5_hw_actions *hw_acts,
+ const struct mlx5_hw_actions *hw_acts,
+ const uint8_t it_idx,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
@@ -464,14 +746,19 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
+ uint64_t item_flags;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
(int)action->type == act_data->type);
- switch (action->type) {
+ switch (act_data->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (flow_hw_shared_action_construct
+ (dev, action, table, it_idx,
+ &rule_acts[act_data->action_dst]))
+ return -1;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -504,6 +791,13 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->hrxq = hrxq;
job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
break;
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, act_data, item_flags,
+ &rule_acts[act_data->action_dst]))
+ return -1;
+ break;
default:
break;
}
@@ -589,8 +883,8 @@ flow_hw_async_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(dev, job, hw_acts, actions,
- rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, pattern_template_index,
+ actions, rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -1237,6 +1531,7 @@ flow_hw_pattern_template_create(struct rte_eth_dev *dev,
"cannot create match template");
return NULL;
}
+ it->item_flags = flow_hw_rss_item_flags_get(items);
__atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
return it;
@@ -1685,6 +1980,109 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
priv->nb_queue = 0;
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+flow_hw_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_create(dev, conf, action, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_update(dev, handle, update, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_destroy(dev, handle, error);
+}
+
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.info_get = flow_hw_info_get,
.configure = flow_hw_configure,
@@ -1698,6 +2096,14 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.async_flow_destroy = flow_hw_async_flow_destroy,
.pull = flow_hw_pull,
.push = flow_hw_push,
+ .async_action_create = flow_hw_action_handle_create,
+ .async_action_destroy = flow_hw_action_handle_destroy,
+ .async_action_update = flow_hw_action_handle_update,
+ .action_validate = flow_dv_action_validate,
+ .action_create = flow_dv_action_create,
+ .action_destroy = flow_dv_action_destroy,
+ .action_update = flow_dv_action_update,
+ .action_query = flow_dv_action_query,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v2 14/14] net/mlx5: add header reformat action
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
` (12 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 13/14] net/mlx5: add indirect action Suanming Mou
@ 2022-02-22 8:51 ` Suanming Mou
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-22 8:51 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering header reformat action can work under bulk mode. In
this case, when create the table, bulk size of header reformat
actions will be allocated in low level. Afterwards, when create
flow, just simply specify the action index in the bulk and the
encapsulation data to the action will be enough.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 21 +++
drivers/net/mlx5/mlx5_flow_dv.c | 4 +-
drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
4 files changed, 251 insertions(+), 3 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index e78eb5e380..bb7067d62d 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
+ uint8_t *encap_data; /* Encap data. */
};
/* HW steering job descriptor LIFO pool. */
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 9ac6745597..ca9011958e 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
union {
+ struct {
+ /* encap src(item) offset. */
+ uint16_t src;
+ /* encap dst data offset. */
+ uint16_t dst;
+ /* encap data len. */
+ uint16_t len;
+ } encap;
struct {
uint64_t types; /* RSS hash types. */
uint32_t level; /* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+ struct mlx5dr_action *action; /* Action object. */
+ size_t data_size; /* Action metadata size. */
+ uint8_t data[]; /* Action data. */
+};
+
/* The maximum actions support in the flow. */
#define MLX5_HW_MAX_ACTS 16
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions {
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
+ /* Encap/Decap action. */
+ struct mlx5_hw_encap_decap_action *encap_decap;
+ uint16_t encap_decap_pos; /* Encap/Decap action position. */
uint32_t acts_num:4; /* Total action number. */
uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
@@ -2032,4 +2050,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle,
void *data,
struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+ size_t *size, struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index b49b88a13f..48ea079dd5 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -4026,7 +4026,7 @@ flow_dv_push_vlan_action_resource_register
* @return
* sizeof struct item_type, 0 if void or irrelevant.
*/
-static size_t
+size_t
flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
{
size_t retval;
@@ -4092,7 +4092,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-static int
+int
flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
size_t *size, struct rte_flow_error *error)
{
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 95df6e5190..cc70b85369 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -332,6 +332,50 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append dynamic encap action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] encap_src
+ * Offset of source encap raw data.
+ * @param[in] encap_dst
+ * Offset of destination encap raw data.
+ * @param[in] len
+ * Length of the data to be updated.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_encap_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint16_t encap_src,
+ uint16_t encap_dst,
+ uint16_t len)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->encap.src = encap_src;
+ act_data->encap.dst = encap_dst;
+ act_data->encap.len = len;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
/**
* Append shared RSS action to the dynamic action list.
*
@@ -422,6 +466,53 @@ flow_hw_shared_action_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Translate encap items to encapsulation list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] items
+ * Encap item pattern.
+ * @param[in] items_m
+ * Encap item mask indicates which part are constant and dynamic.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_encap_item_translate(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ const struct rte_flow_item *items,
+ const struct rte_flow_item *items_m)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ size_t len, total_len = 0;
+ uint32_t i = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++, items_m++, i++) {
+ len = flow_dv_get_item_hdr_len(items->type);
+ if ((!items_m->spec ||
+ memcmp(items_m->spec, items->spec, len)) &&
+ __flow_hw_act_data_encap_append(priv, acts, type,
+ action_src, action_dst, i,
+ total_len, len))
+ return -1;
+ total_len += len;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -459,6 +550,12 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
struct rte_flow_action *actions = at->actions;
struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
+ enum mlx5dr_action_reformat_type refmt_type = 0;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL, *enc_item_m = NULL;
+ uint16_t reformat_pos = MLX5_HW_MAX_ACTS, reformat_src = 0;
+ uint8_t *encap_data = NULL;
+ size_t data_size = 0;
bool actions_end = false;
uint32_t type, i;
int err;
@@ -560,6 +657,56 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_vxlan_encap *)
+ masks->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_DECAP:
+ case RTE_FLOW_ACTION_TYPE_NVGRE_DECAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ actions->conf;
+ encap_data = raw_encap_data->data;
+ data_size = raw_encap_data->size;
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ refmt_type = data_size <
+ MLX5_ENCAPSULATION_DECISION_SIZE ?
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L3_TO_L2 :
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L3;
+ } else {
+ reformat_pos = i++;
+ refmt_type =
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ }
+ reformat_src = actions - action_start;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_DECAP:
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -567,6 +714,45 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ uint8_t buf[MLX5_ENCAP_MAX_LEN];
+
+ if (enc_item) {
+ MLX5_ASSERT(!encap_data);
+ if (flow_dv_convert_encap_data
+ (enc_item, buf, &data_size, error) ||
+ flow_hw_encap_item_translate
+ (dev, acts, (action_start + reformat_src)->type,
+ reformat_src, reformat_pos,
+ enc_item, enc_item_m))
+ goto err;
+ encap_data = buf;
+ } else if (encap_data && __flow_hw_act_data_encap_append
+ (priv, acts,
+ (action_start + reformat_src)->type,
+ reformat_src, reformat_pos, 0, 0, data_size)) {
+ goto err;
+ }
+ acts->encap_decap = mlx5_malloc(MLX5_MEM_ZERO,
+ sizeof(*acts->encap_decap) + data_size,
+ 0, SOCKET_ID_ANY);
+ if (!acts->encap_decap)
+ goto err;
+ if (data_size) {
+ acts->encap_decap->data_size = data_size;
+ memcpy(acts->encap_decap->data, encap_data, data_size);
+ }
+ acts->encap_decap->action = mlx5dr_action_create_reformat
+ (priv->dr_ctx, refmt_type,
+ data_size, encap_data,
+ rte_log2_u32(table_attr->nb_flows),
+ mlx5_hw_act_flag[!!attr->group][type]);
+ if (!acts->encap_decap->action)
+ goto err;
+ acts->rule_acts[reformat_pos].action =
+ acts->encap_decap->action;
+ acts->encap_decap_pos = reformat_pos;
+ }
acts->acts_num = i;
return 0;
err:
@@ -722,6 +908,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_template_table *table = job->flow->table;
struct mlx5_action_construct_data *act_data;
const struct rte_flow_action *action;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL;
+ uint8_t *buf = job->encap_data;
struct rte_flow_attr attr = {
.ingress = 1,
};
@@ -743,6 +932,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
} else {
attr.ingress = 1;
}
+ if (hw_acts->encap_decap && hw_acts->encap_decap->data_size)
+ memcpy(buf, hw_acts->encap_decap->data,
+ hw_acts->encap_decap->data_size);
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
@@ -798,10 +990,38 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
&rule_acts[act_data->action_dst]))
return -1;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ action->conf;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ raw_encap_data->data, act_data->encap.len);
+ MLX5_ASSERT(raw_encap_data->size ==
+ act_data->encap.len);
+ break;
default:
break;
}
}
+ if (hw_acts->encap_decap) {
+ rule_acts[hw_acts->encap_decap_pos].reformat.offset =
+ job->flow->idx - 1;
+ rule_acts[hw_acts->encap_decap_pos].reformat.data = buf;
+ }
return 0;
}
@@ -1863,6 +2083,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(uint8_t) * MLX5_ENCAP_MAX_LEN +
sizeof(struct mlx5_hw_q_job)) *
queue_attr[0]->size;
}
@@ -1873,6 +2094,8 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
for (i = 0; i < nb_queue; i++) {
+ uint8_t *encap = NULL;
+
priv->hw_q[i].job_idx = queue_attr[i]->size;
priv->hw_q[i].size = queue_attr[i]->size;
if (i == 0)
@@ -1883,8 +2106,11 @@ flow_hw_configure(struct rte_eth_dev *dev,
&job[queue_attr[i - 1]->size];
job = (struct mlx5_hw_q_job *)
&priv->hw_q[i].job[queue_attr[i]->size];
- for (j = 0; j < queue_attr[i]->size; j++)
+ encap = (uint8_t *)&job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++) {
+ job[j].encap_data = &encap[j * MLX5_ENCAP_MAX_LEN];
priv->hw_q[i].job[j] = &job[j];
+ }
}
dr_ctx_attr.pd = priv->sh->cdev->pd;
dr_ctx_attr.queues = nb_queue;
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 00/14] net/mlx5: add hardware steering
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (13 preceding siblings ...)
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
` (13 more replies)
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
15 siblings, 14 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The Connect-X steering is a lookup hardware mechanism that accesses
flow tables, matches packets to the rules, and performs specified actions.
Historically, mlx5 PMD implements several software engines to manage
steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to
manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers,
groups, etc.) on the flow inserting is sure to cause slow flow
insertion
- security, exposing the low-level steering entries directly to the
userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering"
is going to be introduced to get rid of the security risks. And it will
take advantage of the recently new introduced async queue-based rte_flow
APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API
will not be supported in the first implementation, only the new async
queue-based flow operations is going to be supported. A new steering
mode parameter for dv_flow_en will be introduced and user will be
able to engage the new steering engine.
*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW ASYNC API ***
---
v3:
- rebase to the latest version.
v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.
Suanming Mou (14):
net/mlx5: introduce hardware steering operation
net/mlx5: add HW steering low-level abstract code
net/mlx5: introduce hardware steering enable routine
net/mlx5: add port flow configuration
net/mlx5: add pattern template management
net/mlx5: add action template management
net/mlx5: add table management
net/mlx5: add basic flow queue operation
net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action
net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
doc/guides/nics/mlx5.rst | 19 +-
doc/guides/rel_notes/release_22_03.rst | 6 +
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/linux/mlx5_os.c | 22 +-
drivers/net/mlx5/meson.build | 2 +
drivers/net/mlx5/mlx5.c | 55 +-
drivers/net/mlx5/mlx5.h | 66 +-
drivers/net/mlx5/mlx5_devx.c | 10 +
drivers/net/mlx5/mlx5_dr.c | 383 ++++
drivers/net/mlx5/mlx5_dr.h | 456 +++++
drivers/net/mlx5/mlx5_flow.c | 721 ++++++-
drivers/net/mlx5/mlx5_flow.h | 287 +++
drivers/net/mlx5/mlx5_flow_dv.c | 186 +-
drivers/net/mlx5/mlx5_flow_hw.c | 2335 +++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 85 +-
drivers/net/mlx5/windows/mlx5_flow_os.h | 1 +
18 files changed, 4471 insertions(+), 180 deletions(-)
create mode 100644 drivers/net/mlx5/mlx5_dr.c
create mode 100644 drivers/net/mlx5/mlx5_dr.h
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 01/14] net/mlx5: introduce hardware steering operation
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
` (12 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The Connect-X steering is a lookup hardware mechanism that accesses
flow tables, matches packets to the rules, and performs specified actions.
Historically, mlx5 PMD implements several software engines to manage
steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to
manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers,
groups, etc.) on the flow inserting is sure to cause slow flow
insertion
- security, exposing the low-level steering entries directly to the
userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering"
is going to be introduced to get rid of the security risks. And it will
take advantage of the recently new introduced async queue-based rte_flow
APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API
will not be supported in the first implementation, only the new async
queue-based flow operations is going to be supported. A new steering
mode parameter for dv_flow_en will be introduced and user will be
able to engage the new steering engine.
This commit adds the basic driver operation.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_flow.c | 1 +
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++
drivers/net/mlx5/windows/mlx5_flow_os.h | 1 +
6 files changed, 18 insertions(+)
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.h b/drivers/net/mlx5/linux/mlx5_flow_os.h
index 1926d26410..e28a9e0436 100644
--- a/drivers/net/mlx5/linux/mlx5_flow_os.h
+++ b/drivers/net/mlx5/linux/mlx5_flow_os.h
@@ -9,6 +9,7 @@
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
#endif
/**
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 2f6d8cbb3d..39a2b8c523 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -16,6 +16,7 @@ sources = files(
'mlx5_flow.c',
'mlx5_flow_meter.c',
'mlx5_flow_dv.c',
+ 'mlx5_flow_hw.c',
'mlx5_flow_aso.c',
'mlx5_flow_flex.c',
'mlx5_mac.c',
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index a87ac8e6d7..b289f13fc0 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+ [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
#endif
[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index a20773eeb2..b70ef0c1b8 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
MLX5_FLOW_TYPE_MIN,
MLX5_FLOW_TYPE_DV,
MLX5_FLOW_TYPE_VERBS,
+ MLX5_FLOW_TYPE_HW,
MLX5_FLOW_TYPE_MAX,
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
new file mode 100644
index 0000000000..729d5914a8
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2022 NVIDIA Corporation & Affiliates
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
diff --git a/drivers/net/mlx5/windows/mlx5_flow_os.h b/drivers/net/mlx5/windows/mlx5_flow_os.h
index dfcb012334..52013b06a0 100644
--- a/drivers/net/mlx5/windows/mlx5_flow_os.h
+++ b/drivers/net/mlx5/windows/mlx5_flow_os.h
@@ -10,6 +10,7 @@
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
#endif
/**
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 02/14] net/mlx5: add HW steering low-level abstract code
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-24 3:10 ` [PATCH v3 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
` (11 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The HW steering low-level implementation will be added later in another
patch series. To avoid the linkage issues the abstract stub replacement
is provided currently.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++
3 files changed, 840 insertions(+)
create mode 100644 drivers/net/mlx5/mlx5_dr.c
create mode 100644 drivers/net/mlx5/mlx5_dr.h
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 39a2b8c523..393b2c97ac 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -14,6 +14,7 @@ sources = files(
'mlx5.c',
'mlx5_ethdev.c',
'mlx5_flow.c',
+ 'mlx5_dr.c',
'mlx5_flow_meter.c',
'mlx5_flow_dv.c',
'mlx5_flow_hw.c',
diff --git a/drivers/net/mlx5/mlx5_dr.c b/drivers/net/mlx5/mlx5_dr.c
new file mode 100644
index 0000000000..7218708986
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_dr.c
@@ -0,0 +1,383 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA CORPORATION. All rights reserved.
+ */
+#include <rte_flow.h>
+
+#include "mlx5_defs.h"
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+
+/*
+ * The following null stubs are prepared in order not to break the linkage
+ * before the HW steering low-level implementation is added.
+ */
+
+/* Open a context used for direct rule insertion using hardware steering.
+ * Each context can contain multiple tables of different types.
+ *
+ * @param[in] ibv_ctx
+ * The ibv context to used for HWS.
+ * @param[in] attr
+ * Attributes used for context open.
+ * @return pointer to mlx5dr_context on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_context *
+mlx5dr_context_open(void *ibv_ctx,
+ struct mlx5dr_context_attr *attr)
+{
+ (void)ibv_ctx;
+ (void)attr;
+ return NULL;
+}
+
+/* Close a context used for direct hardware steering.
+ *
+ * @param[in] ctx
+ * mlx5dr context to close.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_context_close(struct mlx5dr_context *ctx)
+{
+ (void)ctx;
+ return 0;
+}
+
+/* Create a new direct rule table. Each table can contain multiple matchers.
+ *
+ * @param[in] ctx
+ * The context in which the new table will be opened.
+ * @param[in] attr
+ * Attributes used for table creation.
+ * @return pointer to mlx5dr_table on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_table *
+mlx5dr_table_create(struct mlx5dr_context *ctx,
+ struct mlx5dr_table_attr *attr)
+{
+ (void)ctx;
+ (void)attr;
+ return NULL;
+}
+
+/* Destroy direct rule table.
+ *
+ * @param[in] tbl
+ * mlx5dr table to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int mlx5dr_table_destroy(struct mlx5dr_table *tbl)
+{
+ (void)tbl;
+ return 0;
+}
+
+/* Create new match template based on items mask, the match template
+ * will be used for matcher creation.
+ *
+ * @param[in] items
+ * Describe the mask for template creation
+ * @param[in] flags
+ * Template creation flags
+ * @return pointer to mlx5dr_match_template on success NULL otherwise
+ */
+__rte_weak struct mlx5dr_match_template *
+mlx5dr_match_template_create(const struct rte_flow_item items[],
+ enum mlx5dr_match_template_flags flags)
+{
+ (void)items;
+ (void)flags;
+ return NULL;
+}
+
+/* Destroy match template.
+ *
+ * @param[in] mt
+ * Match template to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_match_template_destroy(struct mlx5dr_match_template *mt)
+{
+ (void)mt;
+ return 0;
+}
+
+/* Create a new direct rule matcher. Each matcher can contain multiple rules.
+ * Matchers on the table will be processed by priority. Matching fields and
+ * mask are described by the match template. In some cases multiple match
+ * templates can be used on the same matcher.
+ *
+ * @param[in] table
+ * The table in which the new matcher will be opened.
+ * @param[in] mt
+ * Array of match templates to be used on matcher.
+ * @param[in] num_of_mt
+ * Number of match templates in mt array.
+ * @param[in] attr
+ * Attributes used for matcher creation.
+ * @return pointer to mlx5dr_matcher on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_matcher *
+mlx5dr_matcher_create(struct mlx5dr_table *table __rte_unused,
+ struct mlx5dr_match_template *mt[] __rte_unused,
+ uint8_t num_of_mt __rte_unused,
+ struct mlx5dr_matcher_attr *attr __rte_unused)
+{
+ return NULL;
+}
+
+/* Destroy direct rule matcher.
+ *
+ * @param[in] matcher
+ * Matcher to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher __rte_unused)
+{
+ return 0;
+}
+
+/* Enqueue create rule operation.
+ *
+ * @param[in] matcher
+ * The matcher in which the new rule will be created.
+ * @param[in] mt_idx
+ * Match template index to create the rule with.
+ * @param[in] items
+ * The items used for the value matching.
+ * @param[in] rule_actions
+ * Rule action to be executed on match.
+ * @param[in] num_of_actions
+ * Number of rule actions.
+ * @param[in] attr
+ * Rule creation attributes.
+ * @param[in, out] rule_handle
+ * A valid rule handle. The handle doesn't require any initialization.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_rule_create(struct mlx5dr_matcher *matcher __rte_unused,
+ uint8_t mt_idx __rte_unused,
+ const struct rte_flow_item items[] __rte_unused,
+ struct mlx5dr_rule_action rule_actions[] __rte_unused,
+ uint8_t num_of_actions __rte_unused,
+ struct mlx5dr_rule_attr *attr __rte_unused,
+ struct mlx5dr_rule *rule_handle __rte_unused)
+{
+ return 0;
+}
+
+/* Enqueue destroy rule operation.
+ *
+ * @param[in] rule
+ * The rule destruction to enqueue.
+ * @param[in] attr
+ * Rule destruction attributes.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_rule_destroy(struct mlx5dr_rule *rule __rte_unused,
+ struct mlx5dr_rule_attr *attr __rte_unused)
+{
+ return 0;
+}
+
+/* Create direct rule drop action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_drop(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule default miss action.
+ * Defaults are RX: Drop TX: Wire.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_default_miss(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule goto table action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] tbl
+ * Destination table.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx __rte_unused,
+ struct mlx5dr_table *tbl __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule goto TIR action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule TIR devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx __rte_unused,
+ struct mlx5dr_devx_obj *obj __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule TAG action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_tag(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule counter action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule counter devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_counter(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule reformat action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] reformat_type
+ * Type of reformat.
+ * @param[in] data_sz
+ * Size in bytes of data.
+ * @param[in] inline_data
+ * Header data array in case of inline action.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_reformat(struct mlx5dr_context *ctx __rte_unused,
+ enum mlx5dr_action_reformat_type reformat_type __rte_unused,
+ size_t data_sz __rte_unused,
+ void *inline_data __rte_unused,
+ uint32_t log_bulk_size __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule modify header action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] pattern_sz
+ * Byte size of the pattern array.
+ * @param[in] pattern
+ * PRM format modify pattern action array.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_modify_header(struct mlx5dr_context *ctx,
+ size_t pattern_sz,
+ rte_be64_t pattern[],
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Destroy direct rule action.
+ *
+ * @param[in] action
+ * The action to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_action_destroy(struct mlx5dr_action *action __rte_unused)
+{
+ return 0;
+}
+
+/* Poll queue for rule creation and deletions completions.
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to poll.
+ * @param[in, out] res
+ * Completion array.
+ * @param[in] res_nb
+ * Maximum number of results to return.
+ * @return negative number on failure, the number of completions otherwise.
+ */
+__rte_weak int
+mlx5dr_send_queue_poll(struct mlx5dr_context *ctx __rte_unused,
+ uint16_t queue_id __rte_unused,
+ struct rte_flow_op_result res[] __rte_unused,
+ uint32_t res_nb __rte_unused)
+{
+ return 0;
+}
+
+/* Perform an action on the queue
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to perform the action on.
+ * @param[in] actions
+ * Actions to perform on the queue. (enum mlx5dr_send_queue_actions)
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_send_queue_action(struct mlx5dr_context *ctx __rte_unused,
+ uint16_t queue_id __rte_unused,
+ uint32_t actions __rte_unused)
+{
+ return 0;
+}
+
+#endif
diff --git a/drivers/net/mlx5/mlx5_dr.h b/drivers/net/mlx5/mlx5_dr.h
new file mode 100644
index 0000000000..d0b2c15652
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_dr.h
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA CORPORATION. All rights reserved.
+ */
+
+#ifndef MLX5_DR_H_
+#define MLX5_DR_H_
+
+#include <rte_flow.h>
+
+struct mlx5dr_context;
+struct mlx5dr_table;
+struct mlx5dr_matcher;
+struct mlx5dr_rule;
+
+enum mlx5dr_table_type {
+ MLX5DR_TABLE_TYPE_NIC_RX,
+ MLX5DR_TABLE_TYPE_NIC_TX,
+ MLX5DR_TABLE_TYPE_FDB,
+ MLX5DR_TABLE_TYPE_MAX,
+};
+
+enum mlx5dr_matcher_resource_mode {
+ /* Allocate resources based on number of rules with minimal failure probability */
+ MLX5DR_MATCHER_RESOURCE_MODE_RULE,
+ /* Allocate fixed size hash table based on given column and rows */
+ MLX5DR_MATCHER_RESOURCE_MODE_HTABLE,
+};
+
+enum mlx5dr_action_flags {
+ MLX5DR_ACTION_FLAG_ROOT_RX = 1 << 0,
+ MLX5DR_ACTION_FLAG_ROOT_TX = 1 << 1,
+ MLX5DR_ACTION_FLAG_ROOT_FDB = 1 << 2,
+ MLX5DR_ACTION_FLAG_HWS_RX = 1 << 3,
+ MLX5DR_ACTION_FLAG_HWS_TX = 1 << 4,
+ MLX5DR_ACTION_FLAG_HWS_FDB = 1 << 5,
+ MLX5DR_ACTION_FLAG_INLINE = 1 << 6,
+};
+
+enum mlx5dr_action_reformat_type {
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L3_TO_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L3,
+};
+
+enum mlx5dr_match_template_flags {
+ /* Allow relaxed matching by skipping derived dependent match fields. */
+ MLX5DR_MATCH_TEMPLATE_FLAG_RELAXED_MATCH = 1,
+};
+
+enum mlx5dr_send_queue_actions {
+ /* Start executing all pending queued rules and write to HW */
+ MLX5DR_SEND_QUEUE_ACTION_DRAIN = 1 << 0,
+};
+
+struct mlx5dr_context_attr {
+ uint16_t queues;
+ uint16_t queue_size;
+ size_t initial_log_ste_memory;
+ /* Optional PD used for allocating res ources */
+ struct ibv_pd *pd;
+};
+
+struct mlx5dr_table_attr {
+ enum mlx5dr_table_type type;
+ uint32_t level;
+};
+
+struct mlx5dr_matcher_attr {
+ uint32_t priority;
+ enum mlx5dr_matcher_resource_mode mode;
+ union {
+ struct {
+ uint8_t sz_row_log;
+ uint8_t sz_col_log;
+ } table;
+
+ struct {
+ uint8_t num_log;
+ } rule;
+ };
+};
+
+struct mlx5dr_rule_attr {
+ uint16_t queue_id;
+ void *user_data;
+ uint32_t burst:1;
+};
+
+struct mlx5dr_devx_obj {
+ struct mlx5dv_devx_obj *obj;
+ uint32_t id;
+};
+
+struct mlx5dr_rule_action {
+ struct mlx5dr_action *action;
+ union {
+ struct {
+ uint32_t value;
+ } tag;
+
+ struct {
+ uint32_t offset;
+ } counter;
+
+ struct {
+ uint32_t offset;
+ uint8_t *data;
+ } modify_header;
+
+ struct {
+ uint32_t offset;
+ uint8_t *data;
+ } reformat;
+
+ struct {
+ rte_be32_t vlan_hdr;
+ } push_vlan;
+ };
+};
+
+enum {
+ MLX5DR_MATCH_TAG_SZ = 32,
+ MLX5DR_JAMBO_TAG_SZ = 44,
+};
+
+enum mlx5dr_rule_status {
+ MLX5DR_RULE_STATUS_UNKNOWN,
+ MLX5DR_RULE_STATUS_CREATING,
+ MLX5DR_RULE_STATUS_CREATED,
+ MLX5DR_RULE_STATUS_DELETING,
+ MLX5DR_RULE_STATUS_DELETED,
+ MLX5DR_RULE_STATUS_FAILED,
+};
+
+struct mlx5dr_rule {
+ struct mlx5dr_matcher *matcher;
+ union {
+ uint8_t match_tag[MLX5DR_MATCH_TAG_SZ];
+ struct ibv_flow *flow;
+ };
+ enum mlx5dr_rule_status status;
+ uint32_t rtc_used; /* The RTC into which the STE was inserted */
+};
+
+/* Open a context used for direct rule insertion using hardware steering.
+ * Each context can contain multiple tables of different types.
+ *
+ * @param[in] ibv_ctx
+ * The ibv context to used for HWS.
+ * @param[in] attr
+ * Attributes used for context open.
+ * @return pointer to mlx5dr_context on success NULL otherwise.
+ */
+struct mlx5dr_context *
+mlx5dr_context_open(void *ibv_ctx,
+ struct mlx5dr_context_attr *attr);
+
+/* Close a context used for direct hardware steering.
+ *
+ * @param[in] ctx
+ * mlx5dr context to close.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_context_close(struct mlx5dr_context *ctx);
+
+/* Create a new direct rule table. Each table can contain multiple matchers.
+ *
+ * @param[in] ctx
+ * The context in which the new table will be opened.
+ * @param[in] attr
+ * Attributes used for table creation.
+ * @return pointer to mlx5dr_table on success NULL otherwise.
+ */
+struct mlx5dr_table *
+mlx5dr_table_create(struct mlx5dr_context *ctx,
+ struct mlx5dr_table_attr *attr);
+
+/* Destroy direct rule table.
+ *
+ * @param[in] tbl
+ * mlx5dr table to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_table_destroy(struct mlx5dr_table *tbl);
+
+/* Create new match template based on items mask, the match template
+ * will be used for matcher creation.
+ *
+ * @param[in] items
+ * Describe the mask for template creation
+ * @param[in] flags
+ * Template creation flags
+ * @return pointer to mlx5dr_match_template on success NULL otherwise
+ */
+struct mlx5dr_match_template *
+mlx5dr_match_template_create(const struct rte_flow_item items[],
+ enum mlx5dr_match_template_flags flags);
+
+/* Destroy match template.
+ *
+ * @param[in] mt
+ * Match template to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_match_template_destroy(struct mlx5dr_match_template *mt);
+
+/* Create a new direct rule matcher. Each matcher can contain multiple rules.
+ * Matchers on the table will be processed by priority. Matching fields and
+ * mask are described by the match template. In some cases multiple match
+ * templates can be used on the same matcher.
+ *
+ * @param[in] table
+ * The table in which the new matcher will be opened.
+ * @param[in] mt
+ * Array of match templates to be used on matcher.
+ * @param[in] num_of_mt
+ * Number of match templates in mt array.
+ * @param[in] attr
+ * Attributes used for matcher creation.
+ * @return pointer to mlx5dr_matcher on success NULL otherwise.
+ */
+struct mlx5dr_matcher *
+mlx5dr_matcher_create(struct mlx5dr_table *table,
+ struct mlx5dr_match_template *mt[],
+ uint8_t num_of_mt,
+ struct mlx5dr_matcher_attr *attr);
+
+/* Destroy direct rule matcher.
+ *
+ * @param[in] matcher
+ * Matcher to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher);
+
+/* Get the size of the rule handle (mlx5dr_rule) to be used on rule creation.
+ *
+ * @return size in bytes of rule handle struct.
+ */
+size_t mlx5dr_rule_get_handle_size(void);
+
+/* Enqueue create rule operation.
+ *
+ * @param[in] matcher
+ * The matcher in which the new rule will be created.
+ * @param[in] mt_idx
+ * Match template index to create the rule with.
+ * @param[in] items
+ * The items used for the value matching.
+ * @param[in] rule_actions
+ * Rule action to be executed on match.
+ * @param[in] num_of_actions
+ * Number of rule actions.
+ * @param[in] attr
+ * Rule creation attributes.
+ * @param[in, out] rule_handle
+ * A valid rule handle. The handle doesn't require any initialization.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+int mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
+ uint8_t mt_idx,
+ const struct rte_flow_item items[],
+ struct mlx5dr_rule_action rule_actions[],
+ uint8_t num_of_actions,
+ struct mlx5dr_rule_attr *attr,
+ struct mlx5dr_rule *rule_handle);
+
+/* Enqueue destroy rule operation.
+ *
+ * @param[in] rule
+ * The rule destruction to enqueue.
+ * @param[in] attr
+ * Rule destruction attributes.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+int mlx5dr_rule_destroy(struct mlx5dr_rule *rule,
+ struct mlx5dr_rule_attr *attr);
+
+/* Create direct rule drop action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_drop(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule default miss action.
+ * Defaults are RX: Drop TX: Wire.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_default_miss(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule goto table action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] tbl
+ * Destination table.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx,
+ struct mlx5dr_table *tbl,
+ uint32_t flags);
+
+/* Create direct rule goto TIR action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule TIR devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule TAG action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_tag(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule counter action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule counter devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_counter(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule reformat action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] reformat_type
+ * Type of reformat.
+ * @param[in] data_sz
+ * Size in bytes of data.
+ * @param[in] inline_data
+ * Header data array in case of inline action.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_reformat(struct mlx5dr_context *ctx,
+ enum mlx5dr_action_reformat_type reformat_type,
+ size_t data_sz,
+ void *inline_data,
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Create direct rule modify header action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] pattern_sz
+ * Byte size of the pattern array.
+ * @param[in] pattern
+ * PRM format modify pattern action array.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_modify_header(struct mlx5dr_context *ctx,
+ size_t pattern_sz,
+ rte_be64_t pattern[],
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Destroy direct rule action.
+ *
+ * @param[in] action
+ * The action to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_action_destroy(struct mlx5dr_action *action);
+
+/* Poll queue for rule creation and deletions completions.
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to poll.
+ * @param[in, out] res
+ * Completion array.
+ * @param[in] res_nb
+ * Maximum number of results to return.
+ * @return negative number on failure, the number of completions otherwise.
+ */
+int mlx5dr_send_queue_poll(struct mlx5dr_context *ctx,
+ uint16_t queue_id,
+ struct rte_flow_op_result res[],
+ uint32_t res_nb);
+
+/* Perform an action on the queue
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to perform the action on.
+ * @param[in] actions
+ * Actions to perform on the queue. (enum mlx5dr_send_queue_actions)
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_send_queue_action(struct mlx5dr_context *ctx,
+ uint16_t queue_id,
+ uint32_t actions);
+
+/* Dump HWS info
+ *
+ * @param[in] ctx
+ * The context which to dump the info from.
+ * @param[in] f
+ * The file to write the dump to.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_debug_dump(struct mlx5dr_context *ctx, FILE *f);
+
+#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 03/14] net/mlx5: introduce hardware steering enable routine
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-24 3:10 ` [PATCH v3 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-24 3:10 ` [PATCH v3 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 04/14] net/mlx5: add port flow configuration Suanming Mou
` (10 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The new hardware steering engine relies on using dedicated steering WQEs
instead of direct writing to the low-level steering table entries directly.
In the first introduce implementation the hardware steering engine supports
the new queue based Flow API, the existing synchronous non-queue based Flow
API is not supported.
A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:
dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support
This commit introduces the extra dv_flow_en = 2 to specify the new
flow initialize and manage operation routine.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
doc/guides/nics/mlx5.rst | 13 ++++++++++---
drivers/net/mlx5/linux/mlx5_os.c | 4 ++++
drivers/net/mlx5/mlx5.c | 7 ++++++-
drivers/net/mlx5/mlx5.h | 3 ++-
drivers/net/mlx5/mlx5_flow.c | 22 ++++++++++++++++++++++
5 files changed, 44 insertions(+), 5 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 8956cd1dd8..0e0169c8bb 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -942,10 +942,17 @@ for an additional list of options shared with other mlx5 drivers.
- ``dv_flow_en`` parameter [int]
- A nonzero value enables the DV flow steering assuming it is supported
- by the driver (RDMA Core library version is rdma-core-24.0 or higher).
+ Value 0 means legacy Verbs flow offloading.
- Enabled by default if supported.
+ Value 1 enables the DV flow steering assuming it is supported by the
+ driver (RDMA Core library version is rdma-core-24.0 or higher).
+
+ Value 2 enables the WQE based hardware steering. In this mode only
+ the queue-based rte_flow_q flow management is supported.
+
+ Configured by default to 1 DV flow steering if the driver(RDMA CORE library)
+ supported. Otherwise, the value will be 0 which indicates legacy Verbs flow
+ offloading.
- ``dv_esw_en`` parameter [int]
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index ecf823da56..0faf26f5b8 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
err = mlx5_alloc_table_hash_list(priv);
if (err)
goto error;
+ if (priv->sh->config.dv_flow_en == 2)
+ return 0;
/* The resources below are only valid with DV support. */
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
/* Init port id action list. */
@@ -1519,6 +1521,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ if (priv->sh->config.dv_flow_en == 2)
+ return eth_dev;
/* Port representor shares the same max priority with pf port. */
if (!priv->sh->flow_priority_check_flag) {
/* Supported Verbs flow priority number detection. */
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 9f65a8f901..f49d30c05c 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque)
} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
config->dv_esw_en = !!tmp;
} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
- config->dv_flow_en = !!tmp;
+ if (tmp > 2) {
+ DRV_LOG(ERR, "Invalid %s parameter.", key);
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ config->dv_flow_en = tmp;
} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
if (tmp != MLX5_XMETA_MODE_LEGACY &&
tmp != MLX5_XMETA_MODE_META16 &&
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 0f465d0e9e..b2259fc1fb 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -287,7 +287,8 @@ struct mlx5_sh_config {
int tx_skew; /* Tx scheduling skew between WQE and data on wire. */
uint32_t reclaim_mode:2; /* Memory reclaim mode. */
uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */
- uint32_t dv_flow_en:1; /* Enable DV flow. */
+ /* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+ unsigned int dv_flow_en:2;
uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */
uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */
uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index b289f13fc0..cdb40c0756 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -6840,6 +6840,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,
const struct rte_flow_action actions[],
struct rte_flow_error *error)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (priv->sh->config.dv_flow_en == 2) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q creation not supported");
+ return NULL;
+ }
/*
* If the device is not started yet, it is not allowed to created a
* flow from application. PMD default flows and traffic control flows
@@ -7336,6 +7345,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,
struct rte_flow *flow,
struct rte_flow_error *error __rte_unused)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (priv->sh->config.dv_flow_en == 2)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q destruction not supported");
flow_list_destroy(dev, MLX5_FLOW_TYPE_GEN,
(uintptr_t)(void *)flow);
return 0;
@@ -7433,7 +7449,13 @@ mlx5_flow_query(struct rte_eth_dev *dev,
struct rte_flow_error *error)
{
int ret;
+ struct mlx5_priv *priv = dev->data->dev_private;
+ if (priv->sh->config.dv_flow_en == 2)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q query not supported");
ret = flow_drv_query(dev, (uintptr_t)(void *)flow, actions, data,
error);
if (ret < 0)
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 04/14] net/mlx5: add port flow configuration
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (2 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 05/14] net/mlx5: add pattern template management Suanming Mou
` (9 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The hardware steering is backend to support rte_flow_async API in
mlx5 PMD. The port configuration function creates the queues and
needed flow management resources.
The PMD layer configuration function allocates the queues' context
and per-queue job descriptor pool. The job descriptor pool size
is equal to the queue size, and the job descriptors will be popped
from pool with LIFO strategy to convey the flow information during
flow insertion/destruction. Then, while polling the queued operation
result, the flow information will be extracted from the job descriptor
and the descriptor will be pushed back to the LIFO pool.
The commit creates the flow port queues and the job descriptor pools.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 3 +
drivers/net/mlx5/mlx5.h | 30 +++++-
drivers/net/mlx5/mlx5_flow.c | 86 +++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 15 +++
drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++
5 files changed, 292 insertions(+), 1 deletion(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f49d30c05c..0079aa83c1 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
/* Free the eCPRI flex parser resource. */
mlx5_flex_parser_ecpri_release(dev);
mlx5_flex_item_port_cleanup(dev);
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ flow_hw_resource_release(dev);
+#endif
if (priv->rxq_privs != NULL) {
/* XXX race condition if mlx5_rx_burst() is still running. */
rte_delay_us_sleep(1000);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index b2259fc1fb..f0edf7f559 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -33,7 +33,9 @@
#include "mlx5_utils.h"
#include "mlx5_os.h"
#include "mlx5_autoconf.h"
-
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+#endif
#define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {
uint16_t refcnt; /* Reference count for representors. */
};
+/* HW steering queue job descriptor type. */
+enum {
+ MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+ MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+ uint32_t type; /* Job type. */
+ struct rte_flow *flow; /* Flow attached to the job. */
+ void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO pool. */
+struct mlx5_hw_q {
+ uint32_t job_idx; /* Free job index. */
+ uint32_t size; /* LIFO size. */
+ struct mlx5_hw_q_job **job; /* LIFO header. */
+} __rte_cache_aligned;
+
#define MLX5_COUNTERS_PER_POOL 512
#define MLX5_MAX_PENDING_QUERIES 4
#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1479,6 +1501,12 @@ struct mlx5_priv {
struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+ uint32_t nb_queue; /* HW steering queue number. */
+ /* HW steering queue polling mechanism job descriptor LIFO. */
+ struct mlx5_hw_q *hw_q;
+#endif
};
#define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index cdb40c0756..554ebc804d 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -805,6 +805,17 @@ static int
mlx5_flow_flex_item_release(struct rte_eth_dev *dev,
const struct rte_flow_item_flex_handle *handle,
struct rte_flow_error *error);
+static int
+mlx5_flow_info_get(struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -826,6 +837,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.get_restore_info = mlx5_flow_tunnel_get_restore_info,
.flex_item_create = mlx5_flow_flex_item_create,
.flex_item_release = mlx5_flow_flex_item_release,
+ .info_get = mlx5_flow_info_get,
+ .configure = mlx5_flow_port_configure,
};
/* Tunnel information. */
@@ -3429,6 +3442,12 @@ flow_get_drv_type(struct rte_eth_dev *dev, const struct rte_flow_attr *attr)
if (type != MLX5_FLOW_TYPE_MAX)
return type;
+ /*
+ * Currently when dv_flow_en == 2, only HW steering engine is
+ * supported. New engines can also be chosen here if ready.
+ */
+ if (priv->sh->config.dv_flow_en == 2)
+ return MLX5_FLOW_TYPE_HW;
/* If no OS specific type - continue with DV/VERBS selection */
if (attr->transfer && priv->sh->config.dv_esw_en)
type = MLX5_FLOW_TYPE_DV;
@@ -7838,6 +7857,73 @@ mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt,
return -ENOTSUP;
}
+/**
+ * Get information about HWS pre-configurable resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] port_info
+ * Pointer to port information.
+ * @param[out] queue_info
+ * Pointer to queue information.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_info_get(struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "info get with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->info_get(dev, port_info, queue_info, error);
+}
+
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "port configure with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->configure(dev, port_attr, nb_queue, queue_attr, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index b70ef0c1b8..9f0dc4bde7 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)
const struct rte_flow_item_flex_handle *handle,
const struct rte_flow_item_flex_conf *conf,
struct rte_flow_error *error);
+typedef int (*mlx5_flow_info_get_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_create_t item_create;
mlx5_flow_item_release_t item_release;
mlx5_flow_item_update_t item_update;
+ mlx5_flow_info_get_t info_get;
+ mlx5_flow_port_configure_t configure;
};
/* mlx5_flow.c */
@@ -1762,4 +1775,6 @@ const struct mlx5_flow_tunnel *
mlx5_get_tof(const struct rte_flow_item *items,
const struct rte_flow_action *actions,
enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 729d5914a8..e5b2ae91d8 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -4,10 +4,169 @@
#include <rte_flow.h>
+#include <mlx5_malloc.h>
+#include "mlx5_defs.h"
#include "mlx5_flow.h"
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Get information about HWS pre-configurable resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] port_info
+ * Pointer to port information.
+ * @param[out] queue_info
+ * Pointer to queue information.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_port_info *port_info __rte_unused,
+ struct rte_flow_queue_info *queue_info __rte_unused,
+ struct rte_flow_error *error __rte_unused)
+{
+ /* Nothing to be updated currently. */
+ memset(port_info, 0, sizeof(*port_info));
+ /* Queue size is unlimited from low-level. */
+ queue_info->max_size = UINT32_MAX;
+ return 0;
+}
+
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_context *dr_ctx = NULL;
+ struct mlx5dr_context_attr dr_ctx_attr = {0};
+ struct mlx5_hw_q *hw_q;
+ struct mlx5_hw_q_job *job = NULL;
+ uint32_t mem_size, i, j;
+
+ if (!port_attr || !nb_queue || !queue_attr) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ /* In case re-configuring, release existing context at first. */
+ if (priv->dr_ctx) {
+ /* */
+ for (i = 0; i < nb_queue; i++) {
+ hw_q = &priv->hw_q[i];
+ /* Make sure all queues are empty. */
+ if (hw_q->size != hw_q->job_idx) {
+ rte_errno = EBUSY;
+ goto err;
+ }
+ }
+ flow_hw_resource_release(dev);
+ }
+ /* Allocate the queue job descriptor LIFO. */
+ mem_size = sizeof(priv->hw_q[0]) * nb_queue;
+ for (i = 0; i < nb_queue; i++) {
+ /*
+ * Check if the queues' size are all the same as the
+ * limitation from HWS layer.
+ */
+ if (queue_attr[i]->size != queue_attr[0]->size) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(struct mlx5_hw_q_job)) *
+ queue_attr[0]->size;
+ }
+ priv->hw_q = mlx5_malloc(MLX5_MEM_ZERO, mem_size,
+ 64, SOCKET_ID_ANY);
+ if (!priv->hw_q) {
+ rte_errno = ENOMEM;
+ goto err;
+ }
+ for (i = 0; i < nb_queue; i++) {
+ priv->hw_q[i].job_idx = queue_attr[i]->size;
+ priv->hw_q[i].size = queue_attr[i]->size;
+ if (i == 0)
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &priv->hw_q[nb_queue];
+ else
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &job[queue_attr[i - 1]->size];
+ job = (struct mlx5_hw_q_job *)
+ &priv->hw_q[i].job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++)
+ priv->hw_q[i].job[j] = &job[j];
+ }
+ dr_ctx_attr.pd = priv->sh->cdev->pd;
+ dr_ctx_attr.queues = nb_queue;
+ /* Queue size should all be the same. Take the first one. */
+ dr_ctx_attr.queue_size = queue_attr[0]->size;
+ dr_ctx = mlx5dr_context_open(priv->sh->cdev->ctx, &dr_ctx_attr);
+ /* rte_errno has been updated by HWS layer. */
+ if (!dr_ctx)
+ goto err;
+ priv->dr_ctx = dr_ctx;
+ priv->nb_queue = nb_queue;
+ return 0;
+err:
+ if (dr_ctx)
+ claim_zero(mlx5dr_context_close(dr_ctx));
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to configure port");
+}
+
+/**
+ * Release HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ */
+void
+flow_hw_resource_release(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (!priv->dr_ctx)
+ return;
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ claim_zero(mlx5dr_context_close(priv->dr_ctx));
+ priv->dr_ctx = NULL;
+ priv->nb_queue = 0;
+}
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
+ .info_get = flow_hw_info_get,
+ .configure = flow_hw_configure,
+};
+
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 05/14] net/mlx5: add pattern template management
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (3 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 04/14] net/mlx5: add port flow configuration Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 06/14] net/mlx5: add action " Suanming Mou
` (8 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The pattern template defines flows that have the same matching
fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be
(eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while
the values for each rule will be different.
Due to the pattern template can be used in different domains, the
items will only be cached in pattern template create stage, while
the template is binded to a dedicated table, the HW criteria will
be created and saved to the table. The pattern templates can be
used by multiple tables. But different tables create the same
criteria and will not share the matcher between each other in order
to have better performance.
This commit adds pattern template management.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 76 +++++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 24 +++++++++
drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++
4 files changed, 188 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f0edf7f559..e36adde708 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1502,6 +1502,8 @@ struct mlx5_priv {
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ /* Item template list. */
+ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 554ebc804d..673e0ec55f 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -817,6 +817,17 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -839,6 +850,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.flex_item_release = mlx5_flow_flex_item_release,
.info_get = mlx5_flow_info_get,
.configure = mlx5_flow_port_configure,
+ .pattern_template_create = mlx5_flow_pattern_template_create,
+ .pattern_template_destroy = mlx5_flow_pattern_template_destroy,
};
/* Tunnel information. */
@@ -7924,6 +7937,69 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
return fops->configure(dev, port_attr, nb_queue, queue_attr, error);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "pattern create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pattern_template_create(dev, attr, items, error);
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "pattern destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pattern_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 9f0dc4bde7..49027d6a3a 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1015,6 +1015,19 @@ struct rte_flow {
uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
} __rte_packed;
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+ LIST_ENTRY(rte_flow_pattern_template) next;
+ /* Template attributes. */
+ struct rte_flow_pattern_template_attr attr;
+ struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint32_t refcnt; /* Reference counter. */
+};
+
+#endif
+
/*
* Define list of valid combinations of RX Hash fields
* (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)
uint16_t nb_queue,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_update_t item_update;
mlx5_flow_info_get_t info_get;
mlx5_flow_port_configure_t configure;
+ mlx5_flow_pattern_template_create_t pattern_template_create;
+ mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index e5b2ae91d8..66c5825084 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -13,6 +13,85 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Item template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+flow_hw_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
+
+ it = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*it), 0, rte_socket_id());
+ if (!it) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate item template");
+ return NULL;
+ }
+ it->attr = *attr;
+ it->mt = mlx5dr_match_template_create(items, attr->relaxed_matching);
+ if (!it->mt) {
+ mlx5_free(it);
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot create match template");
+ return NULL;
+ }
+ __atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
+ return it;
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Item template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "item template in using");
+ }
+ LIST_REMOVE(template, next);
+ claim_zero(mlx5dr_match_template_destroy(template->mt));
+ mlx5_free(template);
+ return 0;
+}
+
+/*
* Get information about HWS pre-configurable resources.
*
* @param[in] dev
@@ -154,9 +233,14 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_itt)) {
+ it = LIST_FIRST(&priv->flow_hw_itt);
+ flow_hw_pattern_template_destroy(dev, it, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -167,6 +251,8 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.info_get = flow_hw_info_get,
.configure = flow_hw_configure,
+ .pattern_template_create = flow_hw_pattern_template_create,
+ .pattern_template_destroy = flow_hw_pattern_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 06/14] net/mlx5: add action template management
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (4 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 05/14] net/mlx5: add pattern template management Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 07/14] net/mlx5: add table management Suanming Mou
` (7 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The action template holds a list of action types that will be
used together on the same rule. The template's actions instances
will be created only when the template bind to the dedicated
group. And the created actions will be saved to each individual
group in order for best performance. The actions in a group will
not be shared with each other unless shared actions are specified.
This commit adds the action template management which stores the
flow action template.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 78 +++++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 22 ++++++
drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
4 files changed, 218 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index e36adde708..156b8613d8 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1504,6 +1504,8 @@ struct mlx5_priv {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
/* Item template list. */
LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+ /* Action template list. */
+ LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 673e0ec55f..18c313c6f0 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -827,6 +827,16 @@ static int
mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -852,6 +862,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.configure = mlx5_flow_port_configure,
.pattern_template_create = mlx5_flow_pattern_template_create,
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
+ .actions_template_create = mlx5_flow_actions_template_create,
+ .actions_template_destroy = mlx5_flow_actions_template_destroy,
};
/* Tunnel information. */
@@ -8000,6 +8012,72 @@ mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
return fops->pattern_template_destroy(dev, template, error);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->actions_template_create(dev, attr, actions, masks, error);
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->actions_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 49027d6a3a..9a643fe0a8 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+ LIST_ENTRY(rte_flow_actions_template) next;
+ /* Template attributes. */
+ struct rte_flow_actions_template_attr attr;
+ struct rte_flow_action *actions; /* Cached flow actions. */
+ struct rte_flow_action *masks; /* Cached action masks.*/
+ uint32_t refcnt; /* Reference counter. */
+};
+
#endif
/*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_port_configure_t configure;
mlx5_flow_pattern_template_create_t pattern_template_create;
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+ mlx5_flow_actions_template_create_t actions_template_create;
+ mlx5_flow_actions_template_destroy_t actions_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 66c5825084..4214a63a73 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,115 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Create flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+flow_hw_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int len, act_len, mask_len, i;
+ struct rte_flow_actions_template *at;
+
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, actions, error);
+ if (act_len <= 0)
+ return NULL;
+ len = RTE_ALIGN(act_len, 16);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, masks, error);
+ if (mask_len <= 0)
+ return NULL;
+ len += RTE_ALIGN(mask_len, 16);
+ at = mlx5_malloc(MLX5_MEM_ZERO, len + sizeof(*at), 64, rte_socket_id());
+ if (!at) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate action template");
+ return NULL;
+ }
+ at->attr = *attr;
+ at->actions = (struct rte_flow_action *)(at + 1);
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->actions, len,
+ actions, error);
+ if (act_len <= 0)
+ goto error;
+ at->masks = (struct rte_flow_action *)
+ (((uint8_t *)at->actions) + act_len);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->masks,
+ len - act_len, masks, error);
+ if (mask_len <= 0)
+ goto error;
+ /*
+ * mlx5 PMD hacks indirect action index directly to the action conf.
+ * The rte_flow_conv() function copies the content from conf pointer.
+ * Need to restore the indirect action index from action conf here.
+ */
+ for (i = 0; actions->type != RTE_FLOW_ACTION_TYPE_END;
+ actions++, masks++, i++) {
+ if (actions->type == RTE_FLOW_ACTION_TYPE_INDIRECT) {
+ at->actions[i].conf = actions->conf;
+ at->masks[i].conf = masks->conf;
+ }
+ }
+ __atomic_fetch_add(&at->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_at, at, next);
+ return at;
+error:
+ mlx5_free(at);
+ return NULL;
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Action template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action template in using");
+ }
+ LIST_REMOVE(template, next);
+ mlx5_free(template);
+ return 0;
+}
+
/**
* Create flow item template.
*
@@ -234,6 +343,7 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct rte_flow_pattern_template *it;
+ struct rte_flow_actions_template *at;
if (!priv->dr_ctx)
return;
@@ -241,6 +351,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
}
+ while (!LIST_EMPTY(&priv->flow_hw_at)) {
+ at = LIST_FIRST(&priv->flow_hw_at);
+ flow_hw_actions_template_destroy(dev, at, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -253,6 +367,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.configure = flow_hw_configure,
.pattern_template_create = flow_hw_pattern_template_create,
.pattern_template_destroy = flow_hw_pattern_template_destroy,
+ .actions_template_create = flow_hw_actions_template_create,
+ .actions_template_destroy = flow_hw_actions_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 07/14] net/mlx5: add table management
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (5 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 06/14] net/mlx5: add action " Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 08/14] net/mlx5: add basic flow queue operation Suanming Mou
` (6 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Flow table is a group of flows with the same matching criteria
and the same actions defined for them. The table defines rules
that have the same matching fields but with different matching
values. For example, matching on 5 tuple, the table will be
(IPv4 source + IPv4 dest + s_port + d_port + next_proto)
while the values for each rule will be different.
The templates' relevant matching criteria and action instances
will be created in the table creation and saved in the table.
As table attributes indicate the supported flow number, the flow
memory will also be allocated at the same time.
This commit adds the table management functions.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 45 ++-
drivers/net/mlx5/mlx5.h | 21 +-
drivers/net/mlx5/mlx5_flow.c | 93 ++++++
drivers/net/mlx5/mlx5_flow.h | 73 +++++
drivers/net/mlx5/mlx5_flow_hw.c | 525 ++++++++++++++++++++++++++++++++
5 files changed, 751 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 0079aa83c1..7611fdd62b 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1607,12 +1607,46 @@ void
mlx5_free_table_hash_list(struct mlx5_priv *priv)
{
struct mlx5_dev_ctx_shared *sh = priv->sh;
-
- if (!sh->flow_tbls)
+ struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en == 2) ?
+ &sh->groups : &sh->flow_tbls;
+ if (*tbls == NULL)
return;
- mlx5_hlist_destroy(sh->flow_tbls);
- sh->flow_tbls = NULL;
+ mlx5_hlist_destroy(*tbls);
+ *tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ * Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+ int err = 0;
+ struct mlx5_dev_ctx_shared *sh = priv->sh;
+ char s[MLX5_NAME_SIZE];
+
+ MLX5_ASSERT(sh);
+ snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+ sh->groups = mlx5_hlist_create
+ (s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+ false, true, sh,
+ flow_hw_grp_create_cb,
+ flow_hw_grp_match_cb,
+ flow_hw_grp_remove_cb,
+ flow_hw_grp_clone_cb,
+ flow_hw_grp_clone_free_cb);
+ if (!sh->groups) {
+ DRV_LOG(ERR, "flow groups with hash creation failed.");
+ err = ENOMEM;
+ }
+ return err;
}
+#endif
+
/**
* Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int
mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
{
int err = 0;
+
/* Tables are only used in DV and DR modes. */
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
struct mlx5_dev_ctx_shared *sh = priv->sh;
char s[MLX5_NAME_SIZE];
+ if (priv->sh->config.dv_flow_en == 2)
+ return mlx5_alloc_hw_group_hash_list(priv);
MLX5_ASSERT(sh);
snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 156b8613d8..f523173ad5 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {
MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
MLX5_IPOOL_TAG, /* Pool for tag resource. */
MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
- MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+ MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+ /* Pool for HWS group. Jump action will be created internally. */
+ MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode {
MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
};
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+ MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+ MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+ MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
/* Hlist and list callback context. */
struct mlx5_flow_cb_ctx {
struct rte_eth_dev *dev;
@@ -1204,7 +1213,10 @@ struct mlx5_dev_ctx_shared {
rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
/* UAR same-page access control required in 32bit implementations. */
#endif
- struct mlx5_hlist *flow_tbls;
+ union {
+ struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+ struct mlx5_hlist *groups; /* HWS flow group. */
+ };
struct mlx5_flow_tunnel_hub *tunnel_hub;
/* Direct Rules tables for FDB, NIC TX+RX */
void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1510,6 +1522,11 @@ struct mlx5_priv {
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
struct mlx5_hw_q *hw_q;
+ /* HW steering rte flow table list header. */
+ LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX];
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 18c313c6f0..ee7fc35e1a 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -838,6 +838,19 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -864,6 +877,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
.actions_template_create = mlx5_flow_actions_template_create,
.actions_template_destroy = mlx5_flow_actions_template_destroy,
+ .template_table_create = mlx5_flow_table_create,
+ .template_table_destroy = mlx5_flow_table_destroy,
};
/* Tunnel information. */
@@ -8078,6 +8093,84 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
return fops->actions_template_destroy(dev, template, error);
}
+/**
+ * Create flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->template_table_create(dev,
+ attr,
+ item_templates,
+ nb_item_templates,
+ action_templates,
+ nb_action_templates,
+ error);
+}
+
+/**
+ * PMD destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->template_table_destroy(dev, table, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 9a643fe0a8..1579036f58 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1036,6 +1036,55 @@ struct rte_flow_actions_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Jump action struct. */
+struct mlx5_hw_jump_action {
+ /* Action jump from root. */
+ struct mlx5dr_action *root_action;
+ /* HW steering jump action. */
+ struct mlx5dr_action *hws_action;
+};
+
+/* DR action set struct. */
+struct mlx5_hw_actions {
+ struct mlx5dr_action *drop; /* Drop action. */
+};
+
+/* mlx5 action template struct. */
+struct mlx5_hw_action_template {
+ /* Action template pointer. */
+ struct rte_flow_actions_template *action_template;
+ struct mlx5_hw_actions acts; /* Template actions. */
+};
+
+/* mlx5 flow group struct. */
+struct mlx5_flow_group {
+ struct mlx5_list_entry entry;
+ struct mlx5dr_table *tbl; /* HWS table object. */
+ struct mlx5_hw_jump_action jump; /* Jump action. */
+ enum mlx5dr_table_type type; /* Table type. */
+ uint32_t group_id; /* Group id. */
+ uint32_t idx; /* Group memory index. */
+};
+
+
+#define MLX5_HW_TBL_MAX_ITEM_TEMPLATE 2
+#define MLX5_HW_TBL_MAX_ACTION_TEMPLATE 32
+
+struct rte_flow_template_table {
+ LIST_ENTRY(rte_flow_template_table) next;
+ struct mlx5_flow_group *grp; /* The group rte_flow_template_table uses. */
+ struct mlx5dr_matcher *matcher; /* Template matcher. */
+ /* Item templates bind to the table. */
+ struct rte_flow_pattern_template *its[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ /* Action templates bind to the table. */
+ struct mlx5_hw_action_template ats[MLX5_HW_TBL_MAX_ACTION_TEMPLATE];
+ struct mlx5_indexed_pool *flow; /* The table's flow ipool. */
+ uint32_t type; /* Flow table type RX/TX/FDB. */
+ uint8_t nb_item_templates; /* Item template number. */
+ uint8_t nb_action_templates; /* Action template number. */
+ uint32_t refcnt; /* Table reference counter. */
+};
+
#endif
/*
@@ -1310,6 +1359,18 @@ typedef int (*mlx5_flow_actions_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_template_table *(*mlx5_flow_table_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_table_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1354,6 +1415,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
mlx5_flow_actions_template_create_t actions_template_create;
mlx5_flow_actions_template_destroy_t actions_template_destroy;
+ mlx5_flow_table_create_t template_table_create;
+ mlx5_flow_table_destroy_t template_table_destroy;
};
/* mlx5_flow.c */
@@ -1784,6 +1847,16 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx,
void flow_dv_dest_array_clone_free_cb(void *tool_ctx,
struct mlx5_list_entry *entry);
+struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
+void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+int flow_hw_grp_match_cb(void *tool_ctx,
+ struct mlx5_list_entry *entry,
+ void *cb_ctx);
+struct mlx5_list_entry *flow_hw_grp_clone_cb(void *tool_ctx,
+ struct mlx5_list_entry *oentry,
+ void *cb_ctx);
+void flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+
struct mlx5_aso_age_action *flow_aso_age_get_by_idx(struct rte_eth_dev *dev,
uint32_t age_idx);
int flow_dev_geneve_tlv_option_resource_register(struct rte_eth_dev *dev,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 4214a63a73..8cb1ef842a 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,306 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/* DR action flags with different table. */
+static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX] = {
+ {
+ MLX5DR_ACTION_FLAG_ROOT_RX,
+ MLX5DR_ACTION_FLAG_ROOT_TX,
+ MLX5DR_ACTION_FLAG_ROOT_FDB,
+ },
+ {
+ MLX5DR_ACTION_FLAG_HWS_RX,
+ MLX5DR_ACTION_FLAG_HWS_TX,
+ MLX5DR_ACTION_FLAG_HWS_FDB,
+ },
+};
+
+/**
+ * Destroy DR actions created by action template.
+ *
+ * For DR actions created during table creation's action translate.
+ * Need to destroy the DR action when destroying the table.
+ *
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ */
+static void
+__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+{
+}
+
+/**
+ * Translate rte_flow actions to DR action.
+ *
+ * As the action template has already indicated the actions. Translate
+ * the rte_flow actions to DR action if possbile. So in flow create
+ * stage we will save cycles from handing the actions' organizing.
+ * For the actions with limited information, need to add these to a
+ * list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in/out] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] at
+ * Action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *table_attr,
+ struct mlx5_hw_actions *acts,
+ struct rte_flow_actions_template *at,
+ struct rte_flow_error *error __rte_unused)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ const struct rte_flow_attr *attr = &table_attr->flow_attr;
+ struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *masks = at->masks;
+ bool actions_end = false;
+ uint32_t type;
+
+ if (attr->transfer)
+ type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ type = MLX5DR_TABLE_TYPE_NIC_RX;
+ for (; !actions_end; actions++, masks++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ acts->drop = priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ return 0;
+}
+
+/**
+ * Create flow table.
+ *
+ * The input item and action templates will be binded to the table.
+ * Flow memory will also be allocated. Matcher will be created based
+ * on the item template. Action will be translated to the dedicated
+ * DR action if possible.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+flow_hw_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_matcher_attr matcher_attr = {0};
+ struct rte_flow_template_table *tbl = NULL;
+ struct mlx5_flow_group *grp;
+ struct mlx5dr_match_template *mt[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ struct rte_flow_attr flow_attr = attr->flow_attr;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &flow_attr,
+ };
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow),
+ .trunk_size = 1 << 12,
+ .per_core_cache = 1 << 13,
+ .need_lock = 1,
+ .release_mem_en = !!priv->sh->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_table_flow",
+ };
+ struct mlx5_list_entry *ge;
+ uint32_t i, max_tpl = MLX5_HW_TBL_MAX_ITEM_TEMPLATE;
+ uint32_t nb_flows = rte_align32pow2(attr->nb_flows);
+ int err;
+
+ /* HWS layer accepts only 1 item template with root table. */
+ if (!attr->flow_attr.group)
+ max_tpl = 1;
+ cfg.max_idx = nb_flows;
+ /* For table has limited flows, resize the cache and trunk size. */
+ if (nb_flows < cfg.trunk_size) {
+ cfg.per_core_cache = nb_flows >> 2;
+ cfg.trunk_size = nb_flows;
+ }
+ /* Check if we requires too many templates. */
+ if (nb_item_templates > max_tpl ||
+ nb_action_templates > MLX5_HW_TBL_MAX_ACTION_TEMPLATE) {
+ rte_errno = EINVAL;
+ goto error;
+ }
+ /* Allocate the table memory. */
+ tbl = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*tbl), 0, rte_socket_id());
+ if (!tbl)
+ goto error;
+ /* Allocate flow indexed pool. */
+ tbl->flow = mlx5_ipool_create(&cfg);
+ if (!tbl->flow)
+ goto error;
+ /* Register the flow group. */
+ ge = mlx5_hlist_register(priv->sh->groups, attr->flow_attr.group, &ctx);
+ if (!ge)
+ goto error;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ tbl->grp = grp;
+ /* Prepare matcher information. */
+ matcher_attr.priority = attr->flow_attr.priority;
+ matcher_attr.mode = MLX5DR_MATCHER_RESOURCE_MODE_RULE;
+ matcher_attr.rule.num_log = rte_log2_u32(nb_flows);
+ /* Build the item template. */
+ for (i = 0; i < nb_item_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&item_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto it_error;
+ }
+ mt[i] = item_templates[i]->mt;
+ tbl->its[i] = item_templates[i];
+ }
+ tbl->matcher = mlx5dr_matcher_create
+ (tbl->grp->tbl, mt, nb_item_templates, &matcher_attr);
+ if (!tbl->matcher)
+ goto it_error;
+ tbl->nb_item_templates = nb_item_templates;
+ /* Build the action template. */
+ for (i = 0; i < nb_action_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&action_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto at_error;
+ }
+ err = flow_hw_actions_translate(dev, attr,
+ &tbl->ats[i].acts,
+ action_templates[i], error);
+ if (err)
+ goto at_error;
+ tbl->ats[i].action_template = action_templates[i];
+ }
+ tbl->nb_action_templates = nb_action_templates;
+ tbl->type = attr->flow_attr.transfer ? MLX5DR_TABLE_TYPE_FDB :
+ (attr->flow_attr.egress ? MLX5DR_TABLE_TYPE_NIC_TX :
+ MLX5DR_TABLE_TYPE_NIC_RX);
+ LIST_INSERT_HEAD(&priv->flow_hw_tbl, tbl, next);
+ return tbl;
+at_error:
+ while (i--) {
+ __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __atomic_sub_fetch(&action_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ i = nb_item_templates;
+it_error:
+ while (i--)
+ __atomic_sub_fetch(&item_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ mlx5dr_matcher_destroy(tbl->matcher);
+error:
+ err = rte_errno;
+ if (tbl) {
+ if (tbl->grp)
+ mlx5_hlist_unregister(priv->sh->groups,
+ &tbl->grp->entry);
+ if (tbl->flow)
+ mlx5_ipool_destroy(tbl->flow);
+ mlx5_free(tbl);
+ }
+ rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
+ return NULL;
+}
+
+/**
+ * Destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int i;
+
+ if (table->refcnt) {
+ DRV_LOG(WARNING, "Table %p is still in using.", (void *)table);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table in using");
+ }
+ LIST_REMOVE(table, next);
+ for (i = 0; i < table->nb_item_templates; i++)
+ __atomic_sub_fetch(&table->its[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ for (i = 0; i < table->nb_action_templates; i++) {
+ __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __atomic_sub_fetch(&table->ats[i].action_template->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ mlx5dr_matcher_destroy(table->matcher);
+ mlx5_hlist_unregister(priv->sh->groups, &table->grp->entry);
+ mlx5_ipool_destroy(table->flow);
+ mlx5_free(table);
+ return 0;
+}
+
/**
* Create flow action template.
*
@@ -228,6 +528,199 @@ flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
return 0;
}
+/**
+ * Create group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] cb_ctx
+ * Pointer to the group creation context.
+ *
+ * @return
+ * Group entry on success, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct rte_eth_dev *dev = ctx->dev;
+ struct rte_flow_attr *attr = (struct rte_flow_attr *)ctx->data;
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_table_attr dr_tbl_attr = {0};
+ struct rte_flow_error *error = ctx->error;
+ struct mlx5_flow_group *grp_data;
+ struct mlx5dr_table *tbl = NULL;
+ struct mlx5dr_action *jump;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_zmalloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ dr_tbl_attr.level = attr->group;
+ if (attr->transfer)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_RX;
+ tbl = mlx5dr_table_create(priv->dr_ctx, &dr_tbl_attr);
+ if (!tbl)
+ goto error;
+ grp_data->tbl = tbl;
+ if (attr->group) {
+ /* Jump action be used by non-root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[!!attr->group][dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.hws_action = jump;
+ /* Jump action be used by root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_ROOT]
+ [dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.root_action = jump;
+ }
+ grp_data->idx = idx;
+ grp_data->group_id = attr->group;
+ grp_data->type = dr_tbl_attr.type;
+ return &grp_data->entry;
+error:
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (tbl)
+ mlx5dr_table_destroy(tbl);
+ if (idx)
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], idx);
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow dr table");
+ return NULL;
+}
+
+/**
+ * Remove group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the entry to be removed.
+ */
+void
+flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ MLX5_ASSERT(entry && sh);
+ /* To use the wrapper glue functions instead. */
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ mlx5dr_table_destroy(grp_data->tbl);
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
+/**
+ * Match group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+int
+flow_hw_grp_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
+ void *cb_ctx)
+{
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+ struct rte_flow_attr *attr =
+ (struct rte_flow_attr *)ctx->data;
+
+ return (grp_data->group_id != attr->group) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_FDB) &&
+ attr->transfer) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_TX) &&
+ attr->egress) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_RX) &&
+ attr->ingress);
+}
+
+/**
+ * Clone group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_clone_cb(void *tool_ctx, struct mlx5_list_entry *oentry,
+ void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data;
+ struct rte_flow_error *error = ctx->error;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_malloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ memcpy(grp_data, oentry, sizeof(*grp_data));
+ grp_data->idx = idx;
+ return &grp_data->entry;
+}
+
+/**
+ * Free cloned group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be freed.
+ */
+void
+flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
/**
* Configure port HWS resources.
*
@@ -245,6 +738,7 @@ flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
+
static int
flow_hw_configure(struct rte_eth_dev *dev,
const struct rte_flow_port_attr *port_attr,
@@ -321,8 +815,24 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
priv->dr_ctx = dr_ctx;
priv->nb_queue = nb_queue;
+ /* Add global actions. */
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ priv->hw_drop[i][j] = mlx5dr_action_create_dest_drop
+ (priv->dr_ctx, mlx5_hw_act_flag[i][j]);
+ if (!priv->hw_drop[i][j])
+ goto err;
+ }
+ }
return 0;
err:
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
if (dr_ctx)
claim_zero(mlx5dr_context_close(dr_ctx));
mlx5_free(priv->hw_q);
@@ -342,11 +852,17 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_template_table *tbl;
struct rte_flow_pattern_template *it;
struct rte_flow_actions_template *at;
+ int i, j;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_tbl)) {
+ tbl = LIST_FIRST(&priv->flow_hw_tbl);
+ flow_hw_table_destroy(dev, tbl, NULL);
+ }
while (!LIST_EMPTY(&priv->flow_hw_itt)) {
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
@@ -355,6 +871,13 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
at = LIST_FIRST(&priv->flow_hw_at);
flow_hw_actions_template_destroy(dev, at, NULL);
}
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -369,6 +892,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.pattern_template_destroy = flow_hw_pattern_template_destroy,
.actions_template_create = flow_hw_actions_template_create,
.actions_template_destroy = flow_hw_actions_template_destroy,
+ .template_table_create = flow_hw_table_create,
+ .template_table_destroy = flow_hw_table_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 08/14] net/mlx5: add basic flow queue operation
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (6 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 07/14] net/mlx5: add table management Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 09/14] net/mlx5: add flow flush function Suanming Mou
` (5 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The HW steering uses async queue-based flow rules management
mechanism. The matcher and part of the actions have been
prepared during flow table creation. Some remaining actions
will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow management
should be applied to the HW directly. An extra push function
is provided to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function
will be called to get the queued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead
of allocating from HW steering layer. While destroying the
flow, the flow rule memory can only be freed after the CQE
received.
The HW queue job descriptor is currently introduced to convey
the flow information and operation type between the flow
insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:
rte_flow_async_create();
rte_flow_async_destroy();
rte_flow_push();
rte_flow_pull();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
doc/guides/nics/mlx5.rst | 6 +
doc/guides/rel_notes/release_22_03.rst | 6 +
drivers/net/mlx5/mlx5.h | 2 +-
drivers/net/mlx5/mlx5_flow.c | 188 ++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 41 ++++
drivers/net/mlx5/mlx5_flow_hw.c | 292 ++++++++++++++++++++++++-
6 files changed, 533 insertions(+), 2 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 0e0169c8bb..7b04e9bac5 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -487,6 +487,12 @@ Limitations
from the reference "Clock Queue" completions,
the scheduled send timestamps should not be specified with non-zero MSB.
+ - HW steering:
+
+ - WQE based high scaling and safer flow insertion/destruction.
+ - Set ``dv_flow_en`` to 2 in order to enable HW steering.
+ - Async queue-based ``rte_flow_q`` APIs supported only.
+
Statistics
----------
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 512c727e81..375df6ba74 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -162,6 +162,12 @@ New Features
* Added LED OEM support.
+* **Updated Mellanox mlx5 driver.**
+
+ Updated the Mellanox mlx5 driver with new features and improvements, including:
+
+ * Added WQE based hardware steering support with ``rte_flow_q`` API.
+
* **Added an API for private user data in asymmetric crypto session.**
An API was added to get/set an asymmetric crypto session's user data.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f523173ad5..d94e98db77 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -340,7 +340,7 @@ enum {
/* HW steering flow management job descriptor. */
struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
- struct rte_flow *flow; /* Flow attached to the job. */
+ struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
};
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index ee7fc35e1a..ad131c1b22 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -850,6 +850,34 @@ static int
mlx5_flow_table_destroy(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+static struct rte_flow *
+mlx5_flow_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -879,6 +907,10 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.actions_template_destroy = mlx5_flow_actions_template_destroy,
.template_table_create = mlx5_flow_table_create,
.template_table_destroy = mlx5_flow_table_destroy,
+ .async_create = mlx5_flow_async_flow_create,
+ .async_destroy = mlx5_flow_async_flow_destroy,
+ .pull = mlx5_flow_pull,
+ .push = mlx5_flow_push,
};
/* Tunnel information. */
@@ -8171,6 +8203,162 @@ mlx5_flow_table_destroy(struct rte_eth_dev *dev,
return fops->template_table_destroy(dev, table, error);
}
+/**
+ * Enqueue flow creation.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue_id
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+mlx5_flow_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue_id,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->async_flow_create(dev, queue_id, attr, table,
+ items, pattern_template_index,
+ actions, action_template_index,
+ user_data, error);
+}
+
+/**
+ * Enqueue flow destruction.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->async_flow_destroy(dev, queue, attr, flow,
+ user_data, error);
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q pull with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pull(dev, queue, res, n_res, error);
+}
+
+/**
+ * Push the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flows.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q push with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->push(dev, queue, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 1579036f58..3add4c4a81 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1017,6 +1017,13 @@ struct rte_flow {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/* HWS flow struct. */
+struct rte_flow_hw {
+ uint32_t idx; /* Flow index from indexed pool. */
+ struct rte_flow_template_table *table; /* The table flow allcated from. */
+ struct mlx5dr_rule rule; /* HWS layer data struct. */
+} __rte_packed;
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1371,6 +1378,34 @@ typedef int (*mlx5_flow_table_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+typedef struct rte_flow *(*mlx5_flow_async_flow_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_async_flow_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_pull_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_push_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1417,6 +1452,10 @@ struct mlx5_flow_driver_ops {
mlx5_flow_actions_template_destroy_t actions_template_destroy;
mlx5_flow_table_create_t template_table_create;
mlx5_flow_table_destroy_t template_table_destroy;
+ mlx5_flow_async_flow_create_t async_flow_create;
+ mlx5_flow_async_flow_destroy_t async_flow_destroy;
+ mlx5_flow_pull_t pull;
+ mlx5_flow_push_t push;
};
/* mlx5_flow.c */
@@ -1587,6 +1626,8 @@ mlx5_translate_tunnel_etypes(uint64_t pattern_flags)
return 0;
}
+int flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error);
int mlx5_flow_group_to_table(struct rte_eth_dev *dev,
const struct mlx5_flow_tunnel *tunnel,
uint32_t group, uint32_t *table,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 8cb1ef842a..accc3a96d9 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -10,6 +10,9 @@
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/* DR action flags with different table. */
@@ -105,6 +108,289 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Construct flow action array.
+ *
+ * For action template contains dynamic actions, these actions need to
+ * be updated according to the rte_flow action during flow creation.
+ *
+ * @param[in] hw_acts
+ * Pointer to translated actions from template.
+ * @param[in] actions
+ * Array of rte_flow action need to be checked.
+ * @param[in] rule_acts
+ * Array of DR rule actions to be used during flow creation..
+ * @param[in] acts_num
+ * Pointer to the real acts_num flow has.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+ const struct rte_flow_action actions[],
+ struct mlx5dr_rule_action *rule_acts,
+ uint32_t *acts_num)
+{
+ bool actions_end = false;
+ uint32_t i;
+
+ for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ rule_acts[i++].action = hw_acts->drop;
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ *acts_num = i;
+ return 0;
+}
+
+/**
+ * Enqueue HW steering flow creation.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow creation status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+flow_hw_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = user_data,
+ .burst = attr->postpone,
+ };
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
+ struct mlx5_hw_actions *hw_acts;
+ struct rte_flow_hw *flow;
+ struct mlx5_hw_q_job *job;
+ uint32_t acts_num, flow_idx;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ flow = mlx5_ipool_zmalloc(table->flow, &flow_idx);
+ if (!flow)
+ goto error;
+ /*
+ * Set the table here in order to know the destination table
+ * when free the flow afterwards.
+ */
+ flow->table = table;
+ flow->idx = flow_idx;
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ /*
+ * Set the job type here in order to know if the flow memory
+ * should be freed or not when get the result from dequeue.
+ */
+ job->type = MLX5_HW_Q_JOB_TYPE_CREATE;
+ job->flow = flow;
+ job->user_data = user_data;
+ rule_attr.user_data = job;
+ hw_acts = &table->ats[action_template_index].acts;
+ /* Construct the flow action array based on the input actions.*/
+ flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ ret = mlx5dr_rule_create(table->matcher,
+ pattern_template_index, items,
+ rule_acts, acts_num,
+ &rule_attr, &flow->rule);
+ if (likely(!ret))
+ return (struct rte_flow *)flow;
+ /* Flow created fail, return the descriptor and flow memory. */
+ mlx5_ipool_free(table->flow, flow_idx);
+ priv->hw_q[queue].job_idx++;
+error:
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+ return NULL;
+}
+
+/**
+ * Enqueue HW steering flow destruction.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow destruction status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = user_data,
+ .burst = attr->postpone,
+ };
+ struct rte_flow_hw *fh = (struct rte_flow_hw *)flow;
+ struct mlx5_hw_q_job *job;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ job->type = MLX5_HW_Q_JOB_TYPE_DESTROY;
+ job->user_data = user_data;
+ job->flow = fh;
+ rule_attr.user_data = job;
+ ret = mlx5dr_rule_destroy(&fh->rule, &rule_attr);
+ if (likely(!ret))
+ return 0;
+ priv->hw_q[queue].job_idx++;
+error:
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * For flows enqueued from creation/destruction, the status should be
+ * checked from the dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q_job *job;
+ int ret, i;
+
+ ret = mlx5dr_send_queue_poll(priv->dr_ctx, queue, res, n_res);
+ if (ret < 0)
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to query flow queue");
+ for (i = 0; i < ret; i++) {
+ job = (struct mlx5_hw_q_job *)res[i].user_data;
+ /* Restore user data. */
+ res[i].user_data = job->user_data;
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
+ }
+ return ret;
+}
+
+/**
+ * Push the enqueued flows to HW.
+ *
+ * Force apply all the enqueued flows to the HW.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flow.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int ret;
+
+ ret = mlx5dr_send_queue_action(priv->dr_ctx, queue,
+ MLX5DR_SEND_QUEUE_ACTION_DRAIN);
+ if (ret) {
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to push flows");
+ return ret;
+ }
+ return 0;
+}
+
/**
* Create flow table.
*
@@ -152,7 +438,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
.data = &flow_attr,
};
struct mlx5_indexed_pool_config cfg = {
- .size = sizeof(struct rte_flow),
+ .size = sizeof(struct rte_flow_hw),
.trunk_size = 1 << 12,
.per_core_cache = 1 << 13,
.need_lock = 1,
@@ -894,6 +1180,10 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.actions_template_destroy = flow_hw_actions_template_destroy,
.template_table_create = flow_hw_table_create,
.template_table_destroy = flow_hw_table_destroy,
+ .async_flow_create = flow_hw_async_flow_create,
+ .async_flow_destroy = flow_hw_async_flow_destroy,
+ .pull = flow_hw_pull,
+ .push = flow_hw_push,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 09/14] net/mlx5: add flow flush function
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (7 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 08/14] net/mlx5: add basic flow queue operation Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 10/14] net/mlx5: add flow jump action Suanming Mou
` (4 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
In case port is being stopped, all created flows should be flushed.
This commit adds the flow flush helper function.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 8 ++
drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++
2 files changed, 137 insertions(+)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index ad131c1b22..1672939200 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7015,6 +7015,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
uint32_t num_flushed = 0, fidx = 1;
struct rte_flow *flow;
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ if (priv->sh->config.dv_flow_en == 2 &&
+ type == MLX5_FLOW_TYPE_GEN) {
+ flow_hw_q_flow_flush(dev, NULL);
+ return;
+ }
+#endif
+
MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
flow_list_destroy(dev, type, fidx);
num_flushed++;
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index accc3a96d9..ed14eacce2 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -13,6 +13,12 @@
/* The maximum actions support in the flow. */
#define MLX5_HW_MAX_ACTS 16
+/* Default push burst threshold. */
+#define BURST_THR 32u
+
+/* Default queue to flush the flows. */
+#define MLX5_DEFAULT_FLUSH_QUEUE 0
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/* DR action flags with different table. */
@@ -391,6 +397,129 @@ flow_hw_push(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Drain the enqueued flows' completion.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the flow.
+ * @param[in] pending_rules
+ * The pending flow number.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+__flow_hw_pull_comp(struct rte_eth_dev *dev,
+ uint32_t queue,
+ uint32_t pending_rules,
+ struct rte_flow_error *error)
+{
+ struct rte_flow_op_result comp[BURST_THR];
+ int ret, i, empty_loop = 0;
+
+ flow_hw_push(dev, queue, error);
+ while (pending_rules) {
+ ret = flow_hw_pull(dev, queue, comp, BURST_THR, error);
+ if (ret < 0)
+ return -1;
+ if (!ret) {
+ rte_delay_us_sleep(20000);
+ if (++empty_loop > 5) {
+ DRV_LOG(WARNING, "No available dequeue, quit.");
+ break;
+ }
+ continue;
+ }
+ for (i = 0; i < ret; i++) {
+ if (comp[i].status == RTE_FLOW_OP_ERROR)
+ DRV_LOG(WARNING, "Flow flush get error CQE.");
+ }
+ if ((uint32_t)ret > pending_rules) {
+ DRV_LOG(WARNING, "Flow flush get extra CQE.");
+ return rte_flow_error_set(error, ERANGE,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "get extra CQE");
+ }
+ pending_rules -= ret;
+ empty_loop = 0;
+ }
+ return 0;
+}
+
+/**
+ * Flush created flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+int
+flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q *hw_q;
+ struct rte_flow_template_table *tbl;
+ struct rte_flow_hw *flow;
+ struct rte_flow_op_attr attr = {
+ .postpone = 0,
+ };
+ uint32_t pending_rules = 0;
+ uint32_t queue;
+ uint32_t fidx;
+
+ /*
+ * Ensure to push and dequeue all the enqueued flow
+ * creation/destruction jobs in case user forgot to
+ * dequeue. Or the enqueued created flows will be
+ * leaked. The forgotten dequeues would also cause
+ * flow flush get extra CQEs as expected and pending_rules
+ * be minus value.
+ */
+ for (queue = 0; queue < priv->nb_queue; queue++) {
+ hw_q = &priv->hw_q[queue];
+ if (__flow_hw_pull_comp(dev, queue, hw_q->size - hw_q->job_idx,
+ error))
+ return -1;
+ }
+ /* Flush flow per-table from MLX5_DEFAULT_FLUSH_QUEUE. */
+ hw_q = &priv->hw_q[MLX5_DEFAULT_FLUSH_QUEUE];
+ LIST_FOREACH(tbl, &priv->flow_hw_tbl, next) {
+ MLX5_IPOOL_FOREACH(tbl->flow, fidx, flow) {
+ if (flow_hw_async_flow_destroy(dev,
+ MLX5_DEFAULT_FLUSH_QUEUE,
+ &attr,
+ (struct rte_flow *)flow,
+ NULL,
+ error))
+ return -1;
+ pending_rules++;
+ /* Drain completion with queue size. */
+ if (pending_rules >= hw_q->size) {
+ if (__flow_hw_pull_comp(dev,
+ MLX5_DEFAULT_FLUSH_QUEUE,
+ pending_rules, error))
+ return -1;
+ pending_rules = 0;
+ }
+ }
+ }
+ /* Drain left completion. */
+ if (pending_rules &&
+ __flow_hw_pull_comp(dev, MLX5_DEFAULT_FLUSH_QUEUE, pending_rules,
+ error))
+ return -1;
+ return 0;
+}
+
/**
* Create flow table.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 10/14] net/mlx5: add flow jump action
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (8 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 09/14] net/mlx5: add flow flush function Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 11/14] net/mlx5: add queue and RSS action Suanming Mou
` (3 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Jump action connects different level of flow tables and allows packet
handling in the chain of flows.
A new action construct data struct is also added in this commit to help
to handle not only the dynamic jump action but also for the other generic
dynamic actions. The actions with empty mask configuration means dynamic
action, and the dedicated action will be created with the flow action
configuration during flow creation. In that dynamic action case, the action
will be appended to the table template's action list during table creation.
When creating the flows, traverse the action list and pick the dynamic
action configuration details from flow actions as the action construct data
struct describes, then create the dedicated dynamic actions.
This commit adds the jump action and the generic dynamic action construct
mechanism.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 25 ++-
drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
3 files changed, 275 insertions(+), 21 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index d94e98db77..f3732958a2 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1527,6 +1527,7 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 3add4c4a81..963dbd7806 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1020,10 +1020,25 @@ struct rte_flow {
/* HWS flow struct. */
struct rte_flow_hw {
uint32_t idx; /* Flow index from indexed pool. */
+ uint32_t fate_type; /* Fate action type. */
+ union {
+ /* Jump action. */
+ struct mlx5_hw_jump_action *jump;
+ };
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
} __rte_packed;
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+ LIST_ENTRY(mlx5_action_construct_data) next;
+ /* Ensure the action types are matched. */
+ int type;
+ uint32_t idx; /* Data index. */
+ uint16_t action_src; /* rte_flow_action src offset. */
+ uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
/* DR action set struct. */
struct mlx5_hw_actions {
- struct mlx5dr_action *drop; /* Drop action. */
+ /* Dynamic action list. */
+ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+ struct mlx5_hw_jump_action *jump; /* Jump action. */
+ uint32_t acts_num:4; /* Total action number. */
+ /* Translated DR action array from action template. */
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
/* mlx5 action template struct. */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index ed14eacce2..f320d0db8c 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -36,18 +36,158 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Register destination table DR jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the flow attributes.
+ * @param[in] dest_group
+ * The destination group ID.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct mlx5_hw_jump_action *
+flow_hw_jump_action_register(struct rte_eth_dev *dev,
+ const struct rte_flow_attr *attr,
+ uint32_t dest_group,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_attr jattr = *attr;
+ struct mlx5_flow_group *grp;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &jattr,
+ };
+ struct mlx5_list_entry *ge;
+
+ jattr.group = dest_group;
+ ge = mlx5_hlist_register(priv->sh->flow_tbls, dest_group, &ctx);
+ if (!ge)
+ return NULL;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ return &grp->jump;
+}
+
+/**
+ * Release jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] jump
+ * Pointer to the jump action.
+ */
+
+static void
+flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+}
+
/**
* Destroy DR actions created by action template.
*
* For DR actions created during table creation's action translate.
* Need to destroy the DR action when destroying the table.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
* @param[in] acts
* Pointer to the template HW steering DR actions.
*/
static void
-__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+__flow_hw_action_template_destroy(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (acts->jump) {
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (acts->jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+ acts->jump = NULL;
+ }
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline struct mlx5_action_construct_data *
+__flow_hw_act_data_alloc(struct mlx5_priv *priv,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
+{
+ struct mlx5_action_construct_data *act_data;
+ uint32_t idx = 0;
+
+ act_data = mlx5_ipool_zmalloc(priv->acts_ipool, &idx);
+ if (!act_data)
+ return NULL;
+ act_data->idx = idx;
+ act_data->type = type;
+ act_data->action_src = action_src;
+ act_data->action_dst = action_dst;
+ return act_data;
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_general_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
}
/**
@@ -80,14 +220,16 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
const struct rte_flow_template_table_attr *table_attr,
struct mlx5_hw_actions *acts,
struct rte_flow_actions_template *at,
- struct rte_flow_error *error __rte_unused)
+ struct rte_flow_error *error)
{
struct mlx5_priv *priv = dev->data->dev_private;
const struct rte_flow_attr *attr = &table_attr->flow_attr;
struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
bool actions_end = false;
- uint32_t type;
+ uint32_t type, i;
+ int err;
if (attr->transfer)
type = MLX5DR_TABLE_TYPE_FDB;
@@ -95,14 +237,34 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
type = MLX5DR_TABLE_TYPE_NIC_TX;
else
type = MLX5DR_TABLE_TYPE_NIC_RX;
- for (; !actions_end; actions++, masks++) {
+ for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
case RTE_FLOW_ACTION_TYPE_DROP:
- acts->drop = priv->hw_drop[!!attr->group][type];
+ acts->rule_acts[i++].action =
+ priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ if (masks->conf) {
+ uint32_t jump_group =
+ ((const struct rte_flow_action_jump *)
+ actions->conf)->group;
+ acts->jump = flow_hw_jump_action_register
+ (dev, attr, jump_group, error);
+ if (!acts->jump)
+ goto err;
+ acts->rule_acts[i].action = (!!attr->group) ?
+ acts->jump->hws_action :
+ acts->jump->root_action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
@@ -111,7 +273,14 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ acts->acts_num = i;
return 0;
+err:
+ err = rte_errno;
+ __flow_hw_action_template_destroy(dev, acts);
+ return rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
}
/**
@@ -120,6 +289,10 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* For action template contains dynamic actions, these actions need to
* be updated according to the rte_flow action during flow creation.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] job
+ * Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
* @param[in] actions
@@ -133,31 +306,63 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* 0 on success, negative value otherwise and rte_errno is set.
*/
static __rte_always_inline int
-flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+flow_hw_actions_construct(struct rte_eth_dev *dev,
+ struct mlx5_hw_q_job *job,
+ struct mlx5_hw_actions *hw_acts,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
{
- bool actions_end = false;
- uint32_t i;
+ struct rte_flow_template_table *table = job->flow->table;
+ struct mlx5_action_construct_data *act_data;
+ const struct rte_flow_action *action;
+ struct rte_flow_attr attr = {
+ .ingress = 1,
+ };
- for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
- switch (actions->type) {
+ memcpy(rule_acts, hw_acts->rule_acts,
+ sizeof(*rule_acts) * hw_acts->acts_num);
+ *acts_num = hw_acts->acts_num;
+ if (LIST_EMPTY(&hw_acts->act_list))
+ return 0;
+ attr.group = table->grp->group_id;
+ if (table->type == MLX5DR_TABLE_TYPE_FDB) {
+ attr.transfer = 1;
+ attr.ingress = 1;
+ } else if (table->type == MLX5DR_TABLE_TYPE_NIC_TX) {
+ attr.egress = 1;
+ attr.ingress = 0;
+ } else {
+ attr.ingress = 1;
+ }
+ LIST_FOREACH(act_data, &hw_acts->act_list, next) {
+ uint32_t jump_group;
+ struct mlx5_hw_jump_action *jump;
+
+ action = &actions[act_data->action_src];
+ MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
+ (int)action->type == act_data->type);
+ switch (action->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
- case RTE_FLOW_ACTION_TYPE_DROP:
- rule_acts[i++].action = hw_acts->drop;
- break;
- case RTE_FLOW_ACTION_TYPE_END:
- actions_end = true;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ jump_group = ((const struct rte_flow_action_jump *)
+ action->conf)->group;
+ jump = flow_hw_jump_action_register
+ (dev, &attr, jump_group, NULL);
+ if (!jump)
+ return -1;
+ rule_acts[act_data->action_dst].action =
+ (!!attr.group) ? jump->hws_action : jump->root_action;
+ job->flow->jump = jump;
+ job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
default:
break;
}
}
- *acts_num = i;
return 0;
}
@@ -239,7 +444,8 @@ flow_hw_async_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, actions,
+ rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -356,8 +562,11 @@ flow_hw_pull(struct rte_eth_dev *dev,
job = (struct mlx5_hw_q_job *)res[i].user_data;
/* Restore user data. */
res[i].user_data = job->user_data;
- if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
+ if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
+ flow_hw_jump_release(dev, job->flow->jump);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ }
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
}
return ret;
@@ -642,6 +851,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
rte_errno = EINVAL;
goto at_error;
}
+ LIST_INIT(&tbl->ats[i].acts.act_list);
err = flow_hw_actions_translate(dev, attr,
&tbl->ats[i].acts,
action_templates[i], error);
@@ -657,7 +867,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
return tbl;
at_error:
while (i--) {
- __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &tbl->ats[i].acts);
__atomic_sub_fetch(&action_templates[i]->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -716,7 +926,7 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
- __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -1167,6 +1377,15 @@ flow_hw_configure(struct rte_eth_dev *dev,
struct mlx5_hw_q *hw_q;
struct mlx5_hw_q_job *job = NULL;
uint32_t mem_size, i, j;
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow_hw),
+ .trunk_size = 4096,
+ .need_lock = 1,
+ .release_mem_en = !!priv->sh->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_action_construct_data",
+ };
if (!port_attr || !nb_queue || !queue_attr) {
rte_errno = EINVAL;
@@ -1185,6 +1404,9 @@ flow_hw_configure(struct rte_eth_dev *dev,
}
flow_hw_resource_release(dev);
}
+ priv->acts_ipool = mlx5_ipool_create(&cfg);
+ if (!priv->acts_ipool)
+ goto err;
/* Allocate the queue job descriptor LIFO. */
mem_size = sizeof(priv->hw_q[0]) * nb_queue;
for (i = 0; i < nb_queue; i++) {
@@ -1252,6 +1474,10 @@ flow_hw_configure(struct rte_eth_dev *dev,
claim_zero(mlx5dr_context_close(dr_ctx));
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"fail to configure port");
@@ -1293,6 +1519,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
}
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 11/14] net/mlx5: add queue and RSS action
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (9 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 10/14] net/mlx5: add flow jump action Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 12/14] net/mlx5: add mark action Suanming Mou
` (2 subsequent siblings)
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
This commit adds the queue and RSS action. Similar to the jump action,
dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed
during port restart, the actions are created with standalone indirect
table as indirect action does. When port stops, detaches the indirect
table from action, when port starts, attaches the indirect table back
to the action.
One more change is made to accelerate the action creation. Currently
the mlx5_hrxq_get() function returns the object index instead of object
pointer. This introduced an extra converting the index to the object by
calling mlx5_ipool_get() in most of the case. And that extra converting
hurts multi-thread performance since mlx5_ipool_get() uses the global
lock inside. As the hash Rx queue object itself also contains the index,
returns the object directly will achieve better performance without the
global lock.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 18 ++--
drivers/net/mlx5/mlx5.h | 4 +
drivers/net/mlx5/mlx5_devx.c | 10 ++
drivers/net/mlx5/mlx5_flow.c | 38 +++----
drivers/net/mlx5/mlx5_flow.h | 7 ++
drivers/net/mlx5/mlx5_flow_dv.c | 161 ++++++++++++++---------------
drivers/net/mlx5/mlx5_flow_hw.c | 101 ++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 85 +++++++++------
10 files changed, 283 insertions(+), 157 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 0faf26f5b8..2e1606a733 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1521,6 +1521,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+ mlx5_hrxq_create_cb,
+ mlx5_hrxq_match_cb,
+ mlx5_hrxq_remove_cb,
+ mlx5_hrxq_clone_cb,
+ mlx5_hrxq_clone_free_cb);
+ if (!priv->hrxqs)
+ goto error;
+ rte_rwlock_init(&priv->ind_tbls_lock);
if (priv->sh->config.dv_flow_en == 2)
return eth_dev;
/* Port representor shares the same max priority with pf port. */
@@ -1545,15 +1554,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = ENOTSUP;
goto error;
}
- priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
- mlx5_hrxq_create_cb,
- mlx5_hrxq_match_cb,
- mlx5_hrxq_remove_cb,
- mlx5_hrxq_clone_cb,
- mlx5_hrxq_clone_free_cb);
- if (!priv->hrxqs)
- goto error;
- rte_rwlock_init(&priv->ind_tbls_lock);
/* Query availability of metadata reg_c's. */
if (!priv->sh->metadata_regc_check_flag) {
err = mlx5_flow_discover_mreg_c(eth_dev);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f3732958a2..f60a40d669 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1287,6 +1287,7 @@ struct mlx5_flow_rss_desc {
uint64_t hash_fields; /* Verbs Hash fields. */
uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
uint32_t key_len; /**< RSS hash key len. */
+ uint32_t hws_flags; /**< HW steering action. */
uint32_t tunnel; /**< Queue in tunnel. */
uint32_t shared_rss; /**< Shared RSS index. */
struct mlx5_ind_table_obj *ind_tbl;
@@ -1348,6 +1349,7 @@ struct mlx5_hrxq {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
void *action; /* DV QP action pointer. */
#endif
+ uint32_t hws_flags; /* Hw steering flags. */
uint64_t hash_fields; /* Verbs Hash fields. */
uint32_t rss_key_len; /* Hash key length in bytes. */
uint32_t idx; /* Hash Rx queue index. */
@@ -1478,6 +1480,8 @@ struct mlx5_priv {
LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
/* Indirection tables. */
LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+ /* Standalone indirect tables. */
+ LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
/* Pointer to next element. */
rte_rwlock_t ind_tbls_lock;
uint32_t refcnt; /**< Reference counter. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index a9b8c2a1b7..8d151fa4ab 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
goto error;
}
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ if (hrxq->hws_flags) {
+ hrxq->action = mlx5dr_action_create_dest_tir
+ (priv->dr_ctx,
+ (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+ if (!hrxq->action)
+ goto error;
+ return 0;
+ }
if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
&hrxq->action)) {
rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
DRV_LOG(ERR, "Cannot create drop RX queue");
return ret;
}
+ if (priv->sh->config.dv_flow_en == 2)
+ return 0;
/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
if (ret != 0) {
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 1672939200..cbd8408e30 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -9439,14 +9439,10 @@ int
mlx5_action_handle_attach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
const char *message;
uint32_t queue_idx;
@@ -9462,9 +9458,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
}
if (ret != 0)
return ret;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not attach "
@@ -9473,13 +9467,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
goto error;
}
}
+
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not detach "
@@ -9502,15 +9495,10 @@ int
mlx5_action_handle_detach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
-
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not detach "
@@ -9521,11 +9509,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
}
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not attach "
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 963dbd7806..70e6cf633f 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {
union {
/* Jump action. */
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq; /* TIR action. */
};
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {
/* Dynamic action list. */
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
+ struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1910,6 +1912,11 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx,
struct mlx5_list_entry *entry, void *cb_ctx);
void flow_dv_dest_array_clone_free_cb(void *tool_ctx,
struct mlx5_list_entry *entry);
+void flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index abd1c27538..d48726cf05 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -10975,78 +10975,83 @@ flow_dv_translate_item_tx_queue(struct rte_eth_dev *dev,
/**
* Set the hash fields according to the @p flow information.
*
- * @param[in] dev_flow
- * Pointer to the mlx5_flow.
+ * @param[in] item_flags
+ * The match pattern item flags.
* @param[in] rss_desc
* Pointer to the mlx5_flow_rss_desc.
+ * @param[out] hash_fields
+ * Pointer to the RSS hash fields.
*/
-static void
-flow_dv_hashfields_set(struct mlx5_flow *dev_flow,
- struct mlx5_flow_rss_desc *rss_desc)
+void
+flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields)
{
- uint64_t items = dev_flow->handle->layers;
+ uint64_t items = item_flags;
+ uint64_t fields = 0;
int rss_inner = 0;
uint64_t rss_types = rte_eth_rss_hf_refine(rss_desc->types);
- dev_flow->hash_fields = 0;
+ *hash_fields = 0;
#ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT
if (rss_desc->level >= 2)
rss_inner = 1;
#endif
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV4)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4)) ||
+ !items) {
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV4;
+ fields |= IBV_RX_HASH_SRC_IPV4;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV4;
+ fields |= IBV_RX_HASH_DST_IPV4;
else
- dev_flow->hash_fields |= MLX5_IPV4_IBV_RX_HASH;
+ fields |= MLX5_IPV4_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV6)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6)) ||
+ !items) {
if (rss_types & MLX5_IPV6_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV6;
+ fields |= IBV_RX_HASH_SRC_IPV6;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV6;
+ fields |= IBV_RX_HASH_DST_IPV6;
else
- dev_flow->hash_fields |= MLX5_IPV6_IBV_RX_HASH;
+ fields |= MLX5_IPV6_IBV_RX_HASH;
}
}
- if (dev_flow->hash_fields == 0)
+ if (fields == 0)
/*
* There is no match between the RSS types and the
* L3 protocol (IPv4/IPv6) defined in the flow rule.
*/
return;
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_UDP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_UDP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_UDP;
+ fields |= IBV_RX_HASH_SRC_PORT_UDP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_UDP;
+ fields |= IBV_RX_HASH_DST_PORT_UDP;
else
- dev_flow->hash_fields |= MLX5_UDP_IBV_RX_HASH;
+ fields |= MLX5_UDP_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_TCP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_TCP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_TCP;
+ fields |= IBV_RX_HASH_SRC_PORT_TCP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_TCP;
+ fields |= IBV_RX_HASH_DST_PORT_TCP;
else
- dev_flow->hash_fields |= MLX5_TCP_IBV_RX_HASH;
+ fields |= MLX5_TCP_IBV_RX_HASH;
}
}
if (rss_inner)
- dev_flow->hash_fields |= IBV_RX_HASH_INNER;
+ fields |= IBV_RX_HASH_INNER;
+ *hash_fields = fields;
}
/**
@@ -11070,7 +11075,6 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc,
uint32_t *hrxq_idx)
{
- struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_handle *dh = dev_flow->handle;
struct mlx5_hrxq *hrxq;
@@ -11081,11 +11085,8 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
rss_desc->shared_rss = 0;
if (rss_desc->hash_fields == 0)
rss_desc->queue_num = 1;
- *hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- if (!*hrxq_idx)
- return NULL;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- *hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
+ *hrxq_idx = hrxq ? hrxq->idx : 0;
return hrxq;
}
@@ -11631,7 +11632,9 @@ flow_dv_translate_action_sample(struct rte_eth_dev *dev,
* rss->level and rss.types should be set in advance
* when expanding items for RSS.
*/
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
hrxq = flow_dv_hrxq_prepare(dev, dev_flow,
rss_desc, &hrxq_idx);
if (!hrxq)
@@ -13655,7 +13658,9 @@ flow_dv_translate(struct rte_eth_dev *dev,
*/
handle->layers |= item_flags;
if (action_flags & MLX5_FLOW_ACTION_RSS)
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
/* If has RSS action in the sample action, the Sample/Mirror resource
* should be registered after the hash filed be update.
*/
@@ -14604,20 +14609,18 @@ __flow_dv_action_rss_hrxqs_release(struct rte_eth_dev *dev,
* MLX5_RSS_HASH_IPV4_DST_ONLY are mutually exclusive so they can share
* same slot in mlx5_rss_hash_fields.
*
- * @param[in] rss
- * Pointer to the shared action RSS conf.
+ * @param[in] rss_types
+ * RSS type.
* @param[in, out] hash_field
* hash_field variable needed to be adjusted.
*
* @return
* void
*/
-static void
-__flow_dv_action_rss_l34_hash_adjust(struct mlx5_shared_action_rss *rss,
- uint64_t *hash_field)
+void
+flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field)
{
- uint64_t rss_types = rss->origin.types;
-
switch (*hash_field & ~IBV_RX_HASH_INNER) {
case MLX5_RSS_HASH_IPV4:
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
@@ -14700,12 +14703,15 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
size_t i;
int err;
- if (mlx5_ind_table_obj_setup(dev, shared_rss->ind_tbl,
- !!dev->data->dev_started)) {
+ shared_rss->ind_tbl = mlx5_ind_table_obj_new
+ (dev, shared_rss->origin.queue,
+ shared_rss->origin.queue_num,
+ true,
+ !!dev->data->dev_started);
+ if (!shared_rss->ind_tbl)
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot setup indirection table");
- }
memcpy(rss_desc.key, shared_rss->origin.key, MLX5_RSS_HASH_KEY_LEN);
rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
rss_desc.const_q = shared_rss->origin.queue;
@@ -14714,19 +14720,20 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
- uint32_t hrxq_idx;
+ struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
int tunnel = 0;
- __flow_dv_action_rss_l34_hash_adjust(shared_rss, &hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(shared_rss->origin.types,
+ &hash_fields);
if (shared_rss->origin.level > 1) {
hash_fields |= IBV_RX_HASH_INNER;
tunnel = 1;
}
rss_desc.tunnel = tunnel;
rss_desc.hash_fields = hash_fields;
- hrxq_idx = mlx5_hrxq_get(dev, &rss_desc);
- if (!hrxq_idx) {
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
@@ -14734,14 +14741,14 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
goto error_hrxq_new;
}
err = __flow_dv_action_rss_hrxq_set
- (shared_rss, hash_fields, hrxq_idx);
+ (shared_rss, hash_fields, hrxq->idx);
MLX5_ASSERT(!err);
}
return 0;
error_hrxq_new:
err = rte_errno;
__flow_dv_action_rss_hrxqs_release(dev, shared_rss);
- if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true, true))
+ if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true))
shared_rss->ind_tbl = NULL;
rte_errno = err;
return -rte_errno;
@@ -14772,18 +14779,14 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss = NULL;
- void *queue = NULL;
struct rte_flow_action_rss *origin;
const uint8_t *rss_key;
- uint32_t queue_size = rss->queue_num * sizeof(uint16_t);
uint32_t idx;
RTE_SET_USED(conf);
- queue = mlx5_malloc(0, RTE_ALIGN_CEIL(queue_size, sizeof(void *)),
- 0, SOCKET_ID_ANY);
shared_rss = mlx5_ipool_zmalloc
(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], &idx);
- if (!shared_rss || !queue) {
+ if (!shared_rss) {
rte_flow_error_set(error, ENOMEM,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot allocate resource memory");
@@ -14795,18 +14798,6 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
"rss action number out of range");
goto error_rss_init;
}
- shared_rss->ind_tbl = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(*shared_rss->ind_tbl),
- 0, SOCKET_ID_ANY);
- if (!shared_rss->ind_tbl) {
- rte_flow_error_set(error, ENOMEM,
- RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
- "cannot allocate resource memory");
- goto error_rss_init;
- }
- memcpy(queue, rss->queue, queue_size);
- shared_rss->ind_tbl->queues = queue;
- shared_rss->ind_tbl->queues_n = rss->queue_num;
origin = &shared_rss->origin;
origin->func = rss->func;
origin->level = rss->level;
@@ -14817,10 +14808,12 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
memcpy(shared_rss->key, rss_key, MLX5_RSS_HASH_KEY_LEN);
origin->key = &shared_rss->key[0];
origin->key_len = MLX5_RSS_HASH_KEY_LEN;
- origin->queue = queue;
+ origin->queue = rss->queue;
origin->queue_num = rss->queue_num;
if (__flow_dv_action_rss_setup(dev, idx, shared_rss, error))
goto error_rss_init;
+ /* Update queue with indirect table queue memoyr. */
+ origin->queue = shared_rss->ind_tbl->queues;
rte_spinlock_init(&shared_rss->action_rss_sl);
__atomic_add_fetch(&shared_rss->refcnt, 1, __ATOMIC_RELAXED);
rte_spinlock_lock(&priv->shared_act_sl);
@@ -14831,12 +14824,11 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
error_rss_init:
if (shared_rss) {
if (shared_rss->ind_tbl)
- mlx5_free(shared_rss->ind_tbl);
+ mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
+ !!dev->data->dev_started);
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
idx);
}
- if (queue)
- mlx5_free(queue);
return 0;
}
@@ -14864,7 +14856,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
uint32_t old_refcnt = 1;
int remaining;
- uint16_t *queue = NULL;
if (!shared_rss)
return rte_flow_error_set(error, EINVAL,
@@ -14883,8 +14874,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
RTE_FLOW_ERROR_TYPE_ACTION,
NULL,
"shared rss hrxq has references");
- queue = shared_rss->ind_tbl->queues;
- remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true,
+ remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
!!dev->data->dev_started);
if (remaining)
return rte_flow_error_set(error, EBUSY,
@@ -14892,7 +14882,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
NULL,
"shared rss indirection table has"
" references");
- mlx5_free(queue);
rte_spinlock_lock(&priv->shared_act_sl);
ILIST_REMOVE(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
&priv->rss_shared_actions, idx, shared_rss, next);
@@ -15071,7 +15060,7 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx,
mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
int ret = 0;
void *queue = NULL;
- uint16_t *queue_old = NULL;
+ void *queue_i = NULL;
uint32_t queue_size = action_conf->queue_num * sizeof(uint16_t);
bool dev_started = !!dev->data->dev_started;
@@ -15094,22 +15083,23 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx,
memcpy(queue, action_conf->queue, queue_size);
MLX5_ASSERT(shared_rss->ind_tbl);
rte_spinlock_lock(&shared_rss->action_rss_sl);
- queue_old = shared_rss->ind_tbl->queues;
+ queue_i = shared_rss->ind_tbl->queues;
ret = mlx5_ind_table_obj_modify(dev, shared_rss->ind_tbl,
queue, action_conf->queue_num,
true /* standalone */,
dev_started /* ref_new_qs */,
dev_started /* deref_old_qs */);
if (ret) {
- mlx5_free(queue);
ret = rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_ACTION, NULL,
"cannot update indirection table");
} else {
- mlx5_free(queue_old);
- shared_rss->origin.queue = queue;
+ /* Restore the queue to indirect table internal queue. */
+ memcpy(queue_i, queue, queue_size);
+ shared_rss->ind_tbl->queues = queue_i;
shared_rss->origin.queue_num = action_conf->queue_num;
}
+ mlx5_free(queue);
rte_spinlock_unlock(&shared_rss->action_rss_sl);
return ret;
}
@@ -16845,11 +16835,12 @@ __flow_dv_meter_get_rss_sub_policy(struct rte_eth_dev *dev,
for (i = 0; i < MLX5_MTR_RTE_COLORS; i++) {
if (!rss_desc[i])
continue;
- hrxq_idx[i] = mlx5_hrxq_get(dev, rss_desc[i]);
- if (!hrxq_idx[i]) {
+ hrxq = mlx5_hrxq_get(dev, rss_desc[i]);
+ if (!hrxq) {
rte_spinlock_unlock(&mtr_policy->sl);
return NULL;
}
+ hrxq_idx[i] = hrxq->idx;
}
sub_policy_num = (mtr_policy->sub_policy_num >>
(MLX5_MTR_SUB_POLICY_NUM_SHIFT * domain)) &
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index f320d0db8c..0d49ab0bb2 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -7,6 +7,7 @@
#include <mlx5_malloc.h>
#include "mlx5_defs.h"
#include "mlx5_flow.h"
+#include "mlx5_rx.h"
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
@@ -95,6 +96,56 @@ flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
}
+/**
+ * Register queue/RSS action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] hws_flags
+ * DR action flags.
+ * @param[in] action
+ * rte flow action.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static inline struct mlx5_hrxq*
+flow_hw_tir_action_register(struct rte_eth_dev *dev,
+ uint32_t hws_flags,
+ const struct rte_flow_action *action)
+{
+ struct mlx5_flow_rss_desc rss_desc = {
+ .hws_flags = hws_flags,
+ };
+ struct mlx5_hrxq *hrxq;
+
+ if (action->type == RTE_FLOW_ACTION_TYPE_QUEUE) {
+ const struct rte_flow_action_queue *queue = action->conf;
+
+ rss_desc.const_q = &queue->index;
+ rss_desc.queue_num = 1;
+ } else {
+ const struct rte_flow_action_rss *rss = action->conf;
+
+ rss_desc.queue_num = rss->queue_num;
+ rss_desc.const_q = rss->queue;
+ memcpy(rss_desc.key,
+ !rss->key ? rss_hash_default_key : rss->key,
+ MLX5_RSS_HASH_KEY_LEN);
+ rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
+ rss_desc.types = !rss->types ? RTE_ETH_RSS_IP : rss->types;
+ flow_dv_hashfields_set(0, &rss_desc, &rss_desc.hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(rss->types,
+ &rss_desc.hash_fields);
+ if (rss->level > 1) {
+ rss_desc.hash_fields |= IBV_RX_HASH_INNER;
+ rss_desc.tunnel = 1;
+ }
+ }
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ return hrxq;
+}
+
/**
* Destroy DR actions created by action template.
*
@@ -266,6 +317,40 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -319,6 +404,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_attr attr = {
.ingress = 1,
};
+ uint32_t ft_flag;
memcpy(rule_acts, hw_acts->rule_acts,
sizeof(*rule_acts) * hw_acts->acts_num);
@@ -326,6 +412,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
if (LIST_EMPTY(&hw_acts->act_list))
return 0;
attr.group = table->grp->group_id;
+ ft_flag = mlx5_hw_act_flag[!!table->grp->group_id][table->type];
if (table->type == MLX5DR_TABLE_TYPE_FDB) {
attr.transfer = 1;
attr.ingress = 1;
@@ -338,6 +425,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
@@ -359,6 +447,17 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->jump = jump;
job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ hrxq = flow_hw_tir_action_register(dev,
+ ft_flag,
+ action);
+ if (!hrxq)
+ return -1;
+ rule_acts[act_data->action_dst].action = hrxq->action;
+ job->flow->hrxq = hrxq;
+ job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
+ break;
default:
break;
}
@@ -565,6 +664,8 @@ flow_hw_pull(struct rte_eth_dev *dev,
if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
flow_hw_jump_release(dev, job->flow->jump);
+ else if (job->flow->fate_type == MLX5_FLOW_FATE_QUEUE)
+ mlx5_hrxq_obj_release(dev, job->flow->hrxq);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
}
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c
index 90ccb9aaff..f08aa7a770 100644
--- a/drivers/net/mlx5/mlx5_flow_verbs.c
+++ b/drivers/net/mlx5/mlx5_flow_verbs.c
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
MLX5_ASSERT(priv->drop_queue.hrxq);
hrxq = priv->drop_queue.hrxq;
} else {
- uint32_t hrxq_idx;
struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
rss_desc->tunnel = !!(handle->layers &
MLX5_FLOW_LAYER_TUNNEL);
rss_desc->shared_rss = 0;
- hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
"cannot get hash queue");
goto error;
}
- handle->rix_hrxq = hrxq_idx;
+ handle->rix_hrxq = hrxq->idx;
}
MLX5_ASSERT(hrxq);
handle->drv_flow = mlx5_glue->create_flow
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 38335fd744..295dba063b 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -224,9 +224,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
const uint16_t *queues,
uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+ const uint16_t *queues,
+ uint32_t queues_n,
+ bool standalone,
+ bool ref_qs);
int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs);
int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
@@ -249,8 +253,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
void *cb_ctx __rte_unused);
void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 809006f66a..e7284f9da9 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -2279,8 +2279,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
* Pointer to Ethernet device.
* @param ind_table
* Indirection table to release.
- * @param standalone
- * Indirection table for Standalone queue.
* @param deref_rxqs
* If true, then dereference RX queues related to indirection table.
* Otherwise, no additional action will be taken.
@@ -2291,7 +2289,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
int
mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs)
{
struct mlx5_priv *priv = dev->data->dev_private;
@@ -2299,7 +2296,7 @@ mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
rte_rwlock_write_lock(&priv->ind_tbls_lock);
ret = __atomic_sub_fetch(&ind_tbl->refcnt, 1, __ATOMIC_RELAXED);
- if (!ret && !standalone)
+ if (!ret)
LIST_REMOVE(ind_tbl, next);
rte_rwlock_write_unlock(&priv->ind_tbls_lock);
if (ret)
@@ -2408,7 +2405,7 @@ mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
* @return
* The Verbs/DevX object initialized, NULL otherwise and rte_errno is set.
*/
-static struct mlx5_ind_table_obj *
+struct mlx5_ind_table_obj *
mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
uint32_t queues_n, bool standalone, bool ref_qs)
{
@@ -2416,8 +2413,13 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
struct mlx5_ind_table_obj *ind_tbl;
int ret;
+ /*
+ * Allocate maximum queues for shared action as queue number
+ * maybe modified later.
+ */
ind_tbl = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*ind_tbl) +
- queues_n * sizeof(uint16_t), 0, SOCKET_ID_ANY);
+ (standalone ? priv->rxqs_n : queues_n) *
+ sizeof(uint16_t), 0, SOCKET_ID_ANY);
if (!ind_tbl) {
rte_errno = ENOMEM;
return NULL;
@@ -2430,11 +2432,13 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
mlx5_free(ind_tbl);
return NULL;
}
- if (!standalone) {
- rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ if (!standalone)
LIST_INSERT_HEAD(&priv->ind_tbls, ind_tbl, next);
- rte_rwlock_write_unlock(&priv->ind_tbls_lock);
- }
+ else
+ LIST_INSERT_HEAD(&priv->standalone_ind_tbls, ind_tbl, next);
+ rte_rwlock_write_unlock(&priv->ind_tbls_lock);
+
return ind_tbl;
}
@@ -2600,6 +2604,7 @@ mlx5_hrxq_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
return (hrxq->rss_key_len != rss_desc->key_len ||
memcmp(hrxq->rss_key, rss_desc->key, rss_desc->key_len) ||
+ hrxq->hws_flags != rss_desc->hws_flags ||
hrxq->hash_fields != rss_desc->hash_fields ||
hrxq->ind_table->queues_n != rss_desc->queue_num ||
memcmp(hrxq->ind_table->queues, rss_desc->queue,
@@ -2684,8 +2689,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
}
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ mlx5_ind_table_obj_release(dev, hrxq->ind_table, true);
hrxq->ind_table = ind_tbl;
}
hrxq->hash_fields = hash_fields;
@@ -2695,8 +2699,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
err = rte_errno;
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, ind_tbl, hrxq->standalone,
- true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
}
rte_errno = err;
return -rte_errno;
@@ -2708,12 +2711,16 @@ __mlx5_hrxq_remove(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
struct mlx5_priv *priv = dev->data->dev_private;
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- mlx5_glue->destroy_flow_action(hrxq->action);
+ if (hrxq->hws_flags)
+ mlx5dr_action_destroy(hrxq->action);
+ else
+ mlx5_glue->destroy_flow_action(hrxq->action);
#endif
priv->obj_ops.hrxq_destroy(hrxq);
if (!hrxq->standalone) {
mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ hrxq->hws_flags ?
+ (!!dev->data->dev_started) : true);
}
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq->idx);
}
@@ -2757,11 +2764,12 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
int ret;
queues_n = rss_desc->hash_fields ? queues_n : 1;
- if (!ind_tbl)
+ if (!ind_tbl && !rss_desc->hws_flags)
ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n);
if (!ind_tbl)
ind_tbl = mlx5_ind_table_obj_new(dev, queues, queues_n,
- standalone,
+ standalone ||
+ rss_desc->hws_flags,
!!dev->data->dev_started);
if (!ind_tbl)
return NULL;
@@ -2773,6 +2781,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
hrxq->ind_table = ind_tbl;
hrxq->rss_key_len = rss_key_len;
hrxq->hash_fields = rss_desc->hash_fields;
+ hrxq->hws_flags = rss_desc->hws_flags;
memcpy(hrxq->rss_key, rss_key, rss_key_len);
ret = priv->obj_ops.hrxq_new(dev, hrxq, rss_desc->tunnel);
if (ret < 0)
@@ -2780,7 +2789,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
return hrxq;
error:
if (!rss_desc->ind_tbl)
- mlx5_ind_table_obj_release(dev, ind_tbl, standalone, true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
if (hrxq)
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
return NULL;
@@ -2834,13 +2843,13 @@ mlx5_hrxq_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
* RSS configuration for the Rx hash queue.
*
* @return
- * An hash Rx queue index on success.
+ * An hash Rx queue on success.
*/
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
+ struct mlx5_hrxq *hrxq = NULL;
struct mlx5_list_entry *entry;
struct mlx5_flow_cb_ctx ctx = {
.data = rss_desc,
@@ -2851,12 +2860,10 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
} else {
entry = mlx5_list_register(priv->hrxqs, &ctx);
if (!entry)
- return 0;
+ return NULL;
hrxq = container_of(entry, typeof(*hrxq), entry);
}
- if (hrxq)
- return hrxq->idx;
- return 0;
+ return hrxq;
}
/**
@@ -2865,17 +2872,15 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
* @param dev
* Pointer to Ethernet device.
* @param hrxq_idx
- * Index to Hash Rx queue to release.
+ * Hash Rx queue to release.
*
* @return
* 1 while a reference on it exists, 0 when freed.
*/
-int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
if (!hrxq)
return 0;
if (!hrxq->standalone)
@@ -2884,6 +2889,26 @@ int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
return 0;
}
+/**
+ * Release the hash Rx queue with index.
+ *
+ * @param dev
+ * Pointer to Ethernet device.
+ * @param hrxq_idx
+ * Index to Hash Rx queue to release.
+ *
+ * @return
+ * 1 while a reference on it exists, 0 when freed.
+ */
+int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hrxq *hrxq;
+
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
+ return mlx5_hrxq_obj_release(dev, hrxq);
+}
+
/**
* Create a drop Rx Hash queue.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 12/14] net/mlx5: add mark action
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (10 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 11/14] net/mlx5: add queue and RSS action Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 13/14] net/mlx5: add indirect action Suanming Mou
2022-02-24 3:10 ` [PATCH v3 14/14] net/mlx5: add header reformat action Suanming Mou
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The mark action is covered by tag action internally. While it is added
the HW will add a tag to the packet. The mark value can be set as fixed
or dynamic as the action mask indicates.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++---
3 files changed, 63 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f60a40d669..a0f9fbe365 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1531,6 +1531,8 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX];
struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 70e6cf633f..c83e73c793 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
+ uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 0d49ab0bb2..a28e3c00b3 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -37,6 +37,31 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Set rxq flag.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] enable
+ * Flag to enable or not.
+ */
+static void
+flow_hw_rxq_flag_set(struct rte_eth_dev *dev, bool enable)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ unsigned int i;
+
+ if ((!priv->mark_enabled && !enable) ||
+ (priv->mark_enabled && enable))
+ return;
+ for (i = 0; i < priv->rxqs_n; ++i) {
+ struct mlx5_rxq_ctrl *rxq_ctrl = mlx5_rxq_ctrl_get(dev, i);
+
+ rxq_ctrl->rxq.mark = enable;
+ }
+ priv->mark_enabled = enable;
+}
+
/**
* Register destination table DR jump action.
*
@@ -298,6 +323,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
acts->rule_acts[i++].action =
priv->hw_drop[!!attr->group][type];
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ acts->mark = true;
+ if (masks->conf)
+ acts->rule_acts[i].tag.value =
+ mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (masks->conf))->id);
+ else if (__flow_hw_act_data_general_append(priv, acts,
+ actions->type, actions - action_start, i))
+ goto err;
+ acts->rule_acts[i++].action =
+ priv->hw_tag[!!attr->group];
+ flow_hw_rxq_flag_set(dev, true);
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
if (masks->conf) {
uint32_t jump_group =
@@ -424,6 +463,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
}
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
+ uint32_t tag;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
@@ -435,6 +475,12 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ tag = mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (action->conf))->id);
+ rule_acts[act_data->action_dst].tag.value = tag;
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
jump_group = ((const struct rte_flow_action_jump *)
action->conf)->group;
@@ -1027,6 +1073,8 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
+ if (table->ats[i].acts.mark)
+ flow_hw_rxq_flag_set(dev, false);
__flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
@@ -1561,15 +1609,20 @@ flow_hw_configure(struct rte_eth_dev *dev,
if (!priv->hw_drop[i][j])
goto err;
}
+ priv->hw_tag[i] = mlx5dr_action_create_tag
+ (priv->dr_ctx, mlx5_hw_act_flag[i][0]);
+ if (!priv->hw_tag[i])
+ goto err;
}
return 0;
err:
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
+ if (priv->hw_tag[i])
+ mlx5dr_action_destroy(priv->hw_tag[i]);
}
if (dr_ctx)
claim_zero(mlx5dr_context_close(dr_ctx));
@@ -1615,10 +1668,11 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
}
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
+ if (priv->hw_tag[i])
+ mlx5dr_action_destroy(priv->hw_tag[i]);
}
if (priv->acts_ipool) {
mlx5_ipool_destroy(priv->acts_ipool);
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 13/14] net/mlx5: add indirect action
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (11 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 12/14] net/mlx5: add mark action Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
2022-02-24 3:10 ` [PATCH v3 14/14] net/mlx5: add header reformat action Suanming Mou
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering can support indirect action as well. With indirect action,
the flow can be created with more flexible shared RSS action selection.
This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:
rte_flow_async_action_handle_create();
rte_flow_async_action_handle_destroy();
rte_flow_async_action_handle_update();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 131 ++++++++++
drivers/net/mlx5/mlx5_flow.h | 59 +++++
drivers/net/mlx5/mlx5_flow_dv.c | 21 +-
drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++-
4 files changed, 612 insertions(+), 13 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index cbd8408e30..5a4e000c12 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,
uint32_t queue,
struct rte_flow_error *error);
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.async_destroy = mlx5_flow_async_flow_destroy,
.pull = mlx5_flow_pull,
.push = mlx5_flow_push,
+ .async_action_handle_create = mlx5_flow_async_action_handle_create,
+ .async_action_handle_update = mlx5_flow_async_action_handle_update,
+ .async_action_handle_destroy = mlx5_flow_async_action_handle_destroy,
};
/* Tunnel information. */
@@ -8367,6 +8393,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,
return fops->push(dev, queue, error);
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_create(dev, queue, attr, conf, action,
+ user_data, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_update(dev, queue, attr, handle,
+ update, user_data, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_destroy(dev, queue, attr, handle,
+ user_data, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index c83e73c793..4c224bbf52 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -41,6 +41,7 @@ enum mlx5_rte_flow_action_type {
MLX5_RTE_FLOW_ACTION_TYPE_AGE,
MLX5_RTE_FLOW_ACTION_TYPE_COUNT,
MLX5_RTE_FLOW_ACTION_TYPE_JUMP,
+ MLX5_RTE_FLOW_ACTION_TYPE_RSS,
};
#define MLX5_INDIRECT_ACTION_TYPE_OFFSET 30
@@ -1038,6 +1039,13 @@ struct mlx5_action_construct_data {
uint32_t idx; /* Data index. */
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+ union {
+ struct {
+ uint64_t types; /* RSS hash types. */
+ uint32_t level; /* RSS level. */
+ uint32_t idx; /* Shared action index. */
+ } shared_rss;
+ };
};
/* Flow item template struct. */
@@ -1046,6 +1054,7 @@ struct rte_flow_pattern_template {
/* Template attributes. */
struct rte_flow_pattern_template_attr attr;
struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint64_t item_flags; /* Item layer flags. */
uint32_t refcnt; /* Reference counter. */
};
@@ -1433,6 +1442,32 @@ typedef int (*mlx5_flow_push_t)
uint32_t queue,
struct rte_flow_error *error);
+typedef struct rte_flow_action_handle *(*mlx5_flow_async_action_handle_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_async_action_handle_update_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_async_action_handle_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error);
+
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
mlx5_flow_prepare_t prepare;
@@ -1482,6 +1517,9 @@ struct mlx5_flow_driver_ops {
mlx5_flow_async_flow_destroy_t async_flow_destroy;
mlx5_flow_pull_t pull;
mlx5_flow_push_t push;
+ mlx5_flow_async_action_handle_create_t async_action_create;
+ mlx5_flow_async_action_handle_update_t async_action_update;
+ mlx5_flow_async_action_handle_destroy_t async_action_destroy;
};
/* mlx5_flow.c */
@@ -1918,6 +1956,8 @@ void flow_dv_hashfields_set(uint64_t item_flags,
uint64_t *hash_fields);
void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
uint64_t *hash_field);
+uint32_t flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
@@ -1968,4 +2008,23 @@ mlx5_get_tof(const struct rte_flow_item *items,
enum mlx5_tof_rule_type *rule_type);
void
flow_hw_resource_release(struct rte_eth_dev *dev);
+int flow_dv_action_validate(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+struct rte_flow_action_handle *flow_dv_action_create(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+int flow_dv_action_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error);
+int flow_dv_action_update(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *err);
+int flow_dv_action_query(struct rte_eth_dev *dev,
+ const struct rte_flow_action_handle *handle,
+ void *data,
+ struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index d48726cf05..5f85100324 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -13845,9 +13845,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
* @return
* Valid hash RX queue index, otherwise 0.
*/
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
- const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss =
@@ -13975,7 +13975,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
struct mlx5_hrxq *hrxq = NULL;
uint32_t hrxq_idx;
- hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
rss_desc->shared_rss,
dev_flow->hash_fields);
if (hrxq_idx)
@@ -14699,6 +14699,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
struct mlx5_shared_action_rss *shared_rss,
struct rte_flow_error *error)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_rss_desc rss_desc = { 0 };
size_t i;
int err;
@@ -14719,6 +14720,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
/* Set non-zero value to indicate a shared RSS. */
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
+ if (priv->sh->config.dv_flow_en == 2)
+ rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14910,7 +14913,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
* A valid shared action handle in case of success, NULL otherwise and
* rte_errno is set.
*/
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
flow_dv_action_create(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
@@ -14980,7 +14983,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_destroy(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
struct rte_flow_error *error)
@@ -15190,7 +15193,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_update(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
const void *update,
@@ -15862,7 +15865,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
"counters are not available");
}
-static int
+int
flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle, void *data,
struct rte_flow_error *error)
@@ -17555,7 +17558,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_validate(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index a28e3c00b3..95df6e5190 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -62,6 +62,72 @@ flow_hw_rxq_flag_set(struct rte_eth_dev *dev, bool enable)
priv->mark_enabled = enable;
}
+/**
+ * Generate the pattern item flags.
+ * Will be used for shared RSS action.
+ *
+ * @param[in] items
+ * Pointer to the list of items.
+ *
+ * @return
+ * Item flags.
+ */
+static uint64_t
+flow_hw_rss_item_flags_get(const struct rte_flow_item items[])
+{
+ uint64_t item_flags = 0;
+ uint64_t last_item = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) {
+ int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL);
+ int item_type = items->type;
+
+ switch (item_type) {
+ case RTE_FLOW_ITEM_TYPE_IPV4:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV4;
+ break;
+ case RTE_FLOW_ITEM_TYPE_IPV6:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV6;
+ break;
+ case RTE_FLOW_ITEM_TYPE_TCP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP :
+ MLX5_FLOW_LAYER_OUTER_L4_TCP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_UDP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP :
+ MLX5_FLOW_LAYER_OUTER_L4_UDP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_NVGRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN:
+ last_item = MLX5_FLOW_LAYER_VXLAN;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN_GPE:
+ last_item = MLX5_FLOW_LAYER_VXLAN_GPE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GENEVE:
+ last_item = MLX5_FLOW_LAYER_GENEVE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_MPLS:
+ last_item = MLX5_FLOW_LAYER_MPLS;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GTP:
+ last_item = MLX5_FLOW_LAYER_GTP;
+ break;
+ default:
+ break;
+ }
+ item_flags |= last_item;
+ }
+ return item_flags;
+}
+
/**
* Register destination table DR jump action.
*
@@ -266,6 +332,96 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append shared RSS action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] idx
+ * Shared RSS index.
+ * @param[in] rss
+ * Pointer to the shared RSS info.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_shared_rss_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint32_t idx,
+ struct mlx5_shared_action_rss *rss)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->shared_rss.level = rss->origin.level;
+ act_data->shared_rss.types = !rss->origin.types ? RTE_ETH_RSS_IP :
+ rss->origin.types;
+ act_data->shared_rss.idx = idx;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
+/**
+ * Translate shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct mlx5_hw_actions *acts,
+ uint16_t action_src,
+ uint16_t action_dst)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss || __flow_hw_act_data_shared_rss_append
+ (priv, acts,
+ (enum rte_flow_action_type)MLX5_RTE_FLOW_ACTION_TYPE_RSS,
+ action_src, action_dst, idx, shared_rss))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -316,6 +472,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (!attr->group) {
+ DRV_LOG(ERR, "Indirect action is not supported in root table.");
+ goto err;
+ }
+ if (actions->conf && masks->conf) {
+ if (flow_hw_shared_action_translate
+ (dev, actions, acts, actions - action_start, i))
+ goto err;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -407,6 +577,115 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
"fail to create rte table");
}
+/**
+ * Get shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] act_data
+ * Pointer to the recorded action construct data.
+ * @param[in] item_flags
+ * The matcher itme_flags used for RSS lookup.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_get(struct rte_eth_dev *dev,
+ struct mlx5_action_construct_data *act_data,
+ const uint64_t item_flags,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_rss_desc rss_desc = { 0 };
+ uint64_t hash_fields = 0;
+ uint32_t hrxq_idx = 0;
+ struct mlx5_hrxq *hrxq = NULL;
+ int act_type = act_data->type;
+
+ switch (act_type) {
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ rss_desc.level = act_data->shared_rss.level;
+ rss_desc.types = act_data->shared_rss.types;
+ flow_dv_hashfields_set(item_flags, &rss_desc, &hash_fields);
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup
+ (dev, act_data->shared_rss.idx, hash_fields);
+ if (hrxq_idx)
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
+ hrxq_idx);
+ if (hrxq) {
+ rule_act->action = hrxq->action;
+ return 0;
+ }
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d",
+ act_data->type);
+ break;
+ }
+ return -1;
+}
+
+/**
+ * Construct shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] table
+ * Pointer to the flow table.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_construct(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct rte_flow_template_table *table,
+ const uint8_t it_idx,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_action_construct_data act_data;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+ uint64_t item_flags;
+
+ memset(&act_data, 0, sizeof(act_data));
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ act_data.type = MLX5_RTE_FLOW_ACTION_TYPE_RSS;
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss)
+ return -1;
+ act_data.shared_rss.idx = idx;
+ act_data.shared_rss.level = shared_rss->origin.level;
+ act_data.shared_rss.types = !shared_rss->origin.types ?
+ RTE_ETH_RSS_IP :
+ shared_rss->origin.types;
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, &act_data, item_flags, rule_act))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Construct flow action array.
*
@@ -419,6 +698,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
* @param[in] actions
* Array of rte_flow action need to be checked.
* @param[in] rule_acts
@@ -432,7 +713,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
static __rte_always_inline int
flow_hw_actions_construct(struct rte_eth_dev *dev,
struct mlx5_hw_q_job *job,
- struct mlx5_hw_actions *hw_acts,
+ const struct mlx5_hw_actions *hw_acts,
+ const uint8_t it_idx,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
@@ -464,14 +746,19 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
+ uint64_t item_flags;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
(int)action->type == act_data->type);
- switch (action->type) {
+ switch (act_data->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (flow_hw_shared_action_construct
+ (dev, action, table, it_idx,
+ &rule_acts[act_data->action_dst]))
+ return -1;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -504,6 +791,13 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->hrxq = hrxq;
job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
break;
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, act_data, item_flags,
+ &rule_acts[act_data->action_dst]))
+ return -1;
+ break;
default:
break;
}
@@ -589,8 +883,8 @@ flow_hw_async_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(dev, job, hw_acts, actions,
- rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, pattern_template_index,
+ actions, rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -1237,6 +1531,7 @@ flow_hw_pattern_template_create(struct rte_eth_dev *dev,
"cannot create match template");
return NULL;
}
+ it->item_flags = flow_hw_rss_item_flags_get(items);
__atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
return it;
@@ -1685,6 +1980,109 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
priv->nb_queue = 0;
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+flow_hw_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_create(dev, conf, action, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_update(dev, handle, update, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_destroy(dev, handle, error);
+}
+
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.info_get = flow_hw_info_get,
.configure = flow_hw_configure,
@@ -1698,6 +2096,14 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.async_flow_destroy = flow_hw_async_flow_destroy,
.pull = flow_hw_pull,
.push = flow_hw_push,
+ .async_action_create = flow_hw_action_handle_create,
+ .async_action_destroy = flow_hw_action_handle_destroy,
+ .async_action_update = flow_hw_action_handle_update,
+ .action_validate = flow_dv_action_validate,
+ .action_create = flow_dv_action_create,
+ .action_destroy = flow_dv_action_destroy,
+ .action_update = flow_dv_action_update,
+ .action_query = flow_dv_action_query,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v3 14/14] net/mlx5: add header reformat action
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
` (12 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 13/14] net/mlx5: add indirect action Suanming Mou
@ 2022-02-24 3:10 ` Suanming Mou
13 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 3:10 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering header reformat action can work under bulk mode. In
this case, when create the table, bulk size of header reformat
actions will be allocated in low level. Afterwards, when create
flow, just simply specify the action index in the bulk and the
encapsulation data to the action will be enough.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 21 +++
drivers/net/mlx5/mlx5_flow_dv.c | 4 +-
drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
4 files changed, 251 insertions(+), 3 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a0f9fbe365..bd69aa2334 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
+ uint8_t *encap_data; /* Encap data. */
};
/* HW steering job descriptor LIFO pool. */
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 4c224bbf52..09f0d7a75d 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
union {
+ struct {
+ /* encap src(item) offset. */
+ uint16_t src;
+ /* encap dst data offset. */
+ uint16_t dst;
+ /* encap data len. */
+ uint16_t len;
+ } encap;
struct {
uint64_t types; /* RSS hash types. */
uint32_t level; /* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+ struct mlx5dr_action *action; /* Action object. */
+ size_t data_size; /* Action metadata size. */
+ uint8_t data[]; /* Action data. */
+};
+
/* The maximum actions support in the flow. */
#define MLX5_HW_MAX_ACTS 16
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions {
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
+ /* Encap/Decap action. */
+ struct mlx5_hw_encap_decap_action *encap_decap;
+ uint16_t encap_decap_pos; /* Encap/Decap action position. */
uint32_t acts_num:4; /* Total action number. */
uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
@@ -2027,4 +2045,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle,
void *data,
struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+ size_t *size, struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 5f85100324..7a012f7bb9 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -4034,7 +4034,7 @@ flow_dv_push_vlan_action_resource_register
* @return
* sizeof struct item_type, 0 if void or irrelevant.
*/
-static size_t
+size_t
flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
{
size_t retval;
@@ -4100,7 +4100,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-static int
+int
flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
size_t *size, struct rte_flow_error *error)
{
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 95df6e5190..cc70b85369 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -332,6 +332,50 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append dynamic encap action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] encap_src
+ * Offset of source encap raw data.
+ * @param[in] encap_dst
+ * Offset of destination encap raw data.
+ * @param[in] len
+ * Length of the data to be updated.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_encap_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint16_t encap_src,
+ uint16_t encap_dst,
+ uint16_t len)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->encap.src = encap_src;
+ act_data->encap.dst = encap_dst;
+ act_data->encap.len = len;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
/**
* Append shared RSS action to the dynamic action list.
*
@@ -422,6 +466,53 @@ flow_hw_shared_action_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Translate encap items to encapsulation list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] items
+ * Encap item pattern.
+ * @param[in] items_m
+ * Encap item mask indicates which part are constant and dynamic.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_encap_item_translate(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ const struct rte_flow_item *items,
+ const struct rte_flow_item *items_m)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ size_t len, total_len = 0;
+ uint32_t i = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++, items_m++, i++) {
+ len = flow_dv_get_item_hdr_len(items->type);
+ if ((!items_m->spec ||
+ memcmp(items_m->spec, items->spec, len)) &&
+ __flow_hw_act_data_encap_append(priv, acts, type,
+ action_src, action_dst, i,
+ total_len, len))
+ return -1;
+ total_len += len;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -459,6 +550,12 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
struct rte_flow_action *actions = at->actions;
struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
+ enum mlx5dr_action_reformat_type refmt_type = 0;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL, *enc_item_m = NULL;
+ uint16_t reformat_pos = MLX5_HW_MAX_ACTS, reformat_src = 0;
+ uint8_t *encap_data = NULL;
+ size_t data_size = 0;
bool actions_end = false;
uint32_t type, i;
int err;
@@ -560,6 +657,56 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_vxlan_encap *)
+ masks->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_DECAP:
+ case RTE_FLOW_ACTION_TYPE_NVGRE_DECAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ actions->conf;
+ encap_data = raw_encap_data->data;
+ data_size = raw_encap_data->size;
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ refmt_type = data_size <
+ MLX5_ENCAPSULATION_DECISION_SIZE ?
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L3_TO_L2 :
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L3;
+ } else {
+ reformat_pos = i++;
+ refmt_type =
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ }
+ reformat_src = actions - action_start;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_DECAP:
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -567,6 +714,45 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ uint8_t buf[MLX5_ENCAP_MAX_LEN];
+
+ if (enc_item) {
+ MLX5_ASSERT(!encap_data);
+ if (flow_dv_convert_encap_data
+ (enc_item, buf, &data_size, error) ||
+ flow_hw_encap_item_translate
+ (dev, acts, (action_start + reformat_src)->type,
+ reformat_src, reformat_pos,
+ enc_item, enc_item_m))
+ goto err;
+ encap_data = buf;
+ } else if (encap_data && __flow_hw_act_data_encap_append
+ (priv, acts,
+ (action_start + reformat_src)->type,
+ reformat_src, reformat_pos, 0, 0, data_size)) {
+ goto err;
+ }
+ acts->encap_decap = mlx5_malloc(MLX5_MEM_ZERO,
+ sizeof(*acts->encap_decap) + data_size,
+ 0, SOCKET_ID_ANY);
+ if (!acts->encap_decap)
+ goto err;
+ if (data_size) {
+ acts->encap_decap->data_size = data_size;
+ memcpy(acts->encap_decap->data, encap_data, data_size);
+ }
+ acts->encap_decap->action = mlx5dr_action_create_reformat
+ (priv->dr_ctx, refmt_type,
+ data_size, encap_data,
+ rte_log2_u32(table_attr->nb_flows),
+ mlx5_hw_act_flag[!!attr->group][type]);
+ if (!acts->encap_decap->action)
+ goto err;
+ acts->rule_acts[reformat_pos].action =
+ acts->encap_decap->action;
+ acts->encap_decap_pos = reformat_pos;
+ }
acts->acts_num = i;
return 0;
err:
@@ -722,6 +908,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_template_table *table = job->flow->table;
struct mlx5_action_construct_data *act_data;
const struct rte_flow_action *action;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL;
+ uint8_t *buf = job->encap_data;
struct rte_flow_attr attr = {
.ingress = 1,
};
@@ -743,6 +932,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
} else {
attr.ingress = 1;
}
+ if (hw_acts->encap_decap && hw_acts->encap_decap->data_size)
+ memcpy(buf, hw_acts->encap_decap->data,
+ hw_acts->encap_decap->data_size);
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
@@ -798,10 +990,38 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
&rule_acts[act_data->action_dst]))
return -1;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ action->conf;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ raw_encap_data->data, act_data->encap.len);
+ MLX5_ASSERT(raw_encap_data->size ==
+ act_data->encap.len);
+ break;
default:
break;
}
}
+ if (hw_acts->encap_decap) {
+ rule_acts[hw_acts->encap_decap_pos].reformat.offset =
+ job->flow->idx - 1;
+ rule_acts[hw_acts->encap_decap_pos].reformat.data = buf;
+ }
return 0;
}
@@ -1863,6 +2083,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(uint8_t) * MLX5_ENCAP_MAX_LEN +
sizeof(struct mlx5_hw_q_job)) *
queue_attr[0]->size;
}
@@ -1873,6 +2094,8 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
for (i = 0; i < nb_queue; i++) {
+ uint8_t *encap = NULL;
+
priv->hw_q[i].job_idx = queue_attr[i]->size;
priv->hw_q[i].size = queue_attr[i]->size;
if (i == 0)
@@ -1883,8 +2106,11 @@ flow_hw_configure(struct rte_eth_dev *dev,
&job[queue_attr[i - 1]->size];
job = (struct mlx5_hw_q_job *)
&priv->hw_q[i].job[queue_attr[i]->size];
- for (j = 0; j < queue_attr[i]->size; j++)
+ encap = (uint8_t *)&job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++) {
+ job[j].encap_data = &encap[j * MLX5_ENCAP_MAX_LEN];
priv->hw_q[i].job[j] = &job[j];
+ }
}
dr_ctx_attr.pd = priv->sh->cdev->pd;
dr_ctx_attr.queues = nb_queue;
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 00/14] net/mlx5: add hardware steering
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
` (14 preceding siblings ...)
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
` (14 more replies)
15 siblings, 15 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The Connect-X steering is a lookup hardware mechanism that accesses
flow tables, matches packets to the rules, and performs specified actions.
Historically, mlx5 PMD implements several software engines to manage
steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to
manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers,
groups, etc.) on the flow inserting is sure to cause slow flow
insertion
- security, exposing the low-level steering entries directly to the
userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering"
is going to be introduced to get rid of the security risks. And it will
take advantage of the recently new introduced async queue-based rte_flow
APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API
will not be supported in the first implementation, only the new async
queue-based flow operations is going to be supported. A new steering
mode parameter for dv_flow_en will be introduced and user will be
able to engage the new steering engine.
---
v4:
v3:
- rebase to the latest version.
v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.
Suanming Mou (14):
net/mlx5: introduce hardware steering operation
net/mlx5: add HW steering low-level abstract code
net/mlx5: introduce hardware steering enable routine
net/mlx5: add port flow configuration
net/mlx5: add pattern template management
net/mlx5: add action template management
net/mlx5: add table management
net/mlx5: add basic flow queue operation
net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action
net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
doc/guides/nics/mlx5.rst | 19 +-
doc/guides/rel_notes/release_22_03.rst | 1 +
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/linux/mlx5_os.c | 22 +-
drivers/net/mlx5/meson.build | 2 +
drivers/net/mlx5/mlx5.c | 55 +-
drivers/net/mlx5/mlx5.h | 66 +-
drivers/net/mlx5/mlx5_devx.c | 10 +
drivers/net/mlx5/mlx5_dr.c | 383 ++++
drivers/net/mlx5/mlx5_dr.h | 456 +++++
drivers/net/mlx5/mlx5_flow.c | 721 ++++++-
drivers/net/mlx5/mlx5_flow.h | 287 +++
drivers/net/mlx5/mlx5_flow_dv.c | 186 +-
drivers/net/mlx5/mlx5_flow_hw.c | 2337 +++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 85 +-
drivers/net/mlx5/windows/mlx5_flow_os.h | 1 +
18 files changed, 4468 insertions(+), 180 deletions(-)
create mode 100644 drivers/net/mlx5/mlx5_dr.c
create mode 100644 drivers/net/mlx5/mlx5_dr.h
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 01/14] net/mlx5: introduce hardware steering operation
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
` (13 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The Connect-X steering is a lookup hardware mechanism that accesses
flow tables, matches packets to the rules, and performs specified actions.
Historically, mlx5 PMD implements several software engines to manage
steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to
manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers,
groups, etc.) on the flow inserting is sure to cause slow flow
insertion
- security, exposing the low-level steering entries directly to the
userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering"
is going to be introduced to get rid of the security risks. And it will
take advantage of the recently new introduced async queue-based rte_flow
APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API
will not be supported in the first implementation, only the new async
queue-based flow operations is going to be supported. A new steering
mode parameter for dv_flow_en will be introduced and user will be
able to engage the new steering engine.
This commit adds the basic driver operation.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_flow.c | 1 +
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++
drivers/net/mlx5/windows/mlx5_flow_os.h | 1 +
6 files changed, 18 insertions(+)
create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.h b/drivers/net/mlx5/linux/mlx5_flow_os.h
index 1926d26410..e28a9e0436 100644
--- a/drivers/net/mlx5/linux/mlx5_flow_os.h
+++ b/drivers/net/mlx5/linux/mlx5_flow_os.h
@@ -9,6 +9,7 @@
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
#endif
/**
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 2f6d8cbb3d..39a2b8c523 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -16,6 +16,7 @@ sources = files(
'mlx5_flow.c',
'mlx5_flow_meter.c',
'mlx5_flow_dv.c',
+ 'mlx5_flow_hw.c',
'mlx5_flow_aso.c',
'mlx5_flow_flex.c',
'mlx5_mac.c',
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 96f3402418..aec14ea39d 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+ [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
#endif
[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index a20773eeb2..b70ef0c1b8 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
MLX5_FLOW_TYPE_MIN,
MLX5_FLOW_TYPE_DV,
MLX5_FLOW_TYPE_VERBS,
+ MLX5_FLOW_TYPE_HW,
MLX5_FLOW_TYPE_MAX,
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
new file mode 100644
index 0000000000..729d5914a8
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2022 NVIDIA Corporation & Affiliates
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
diff --git a/drivers/net/mlx5/windows/mlx5_flow_os.h b/drivers/net/mlx5/windows/mlx5_flow_os.h
index dfcb012334..52013b06a0 100644
--- a/drivers/net/mlx5/windows/mlx5_flow_os.h
+++ b/drivers/net/mlx5/windows/mlx5_flow_os.h
@@ -10,6 +10,7 @@
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
#endif
/**
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-24 13:40 ` [PATCH v4 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 22:57 ` Ferruh Yigit
2022-02-24 13:40 ` [PATCH v4 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
` (12 subsequent siblings)
14 siblings, 1 reply; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The HW steering low-level implementation will be added later in another
patch series. To avoid the linkage issues the abstract stub replacement
is provided currently.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++
3 files changed, 840 insertions(+)
create mode 100644 drivers/net/mlx5/mlx5_dr.c
create mode 100644 drivers/net/mlx5/mlx5_dr.h
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 39a2b8c523..393b2c97ac 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -14,6 +14,7 @@ sources = files(
'mlx5.c',
'mlx5_ethdev.c',
'mlx5_flow.c',
+ 'mlx5_dr.c',
'mlx5_flow_meter.c',
'mlx5_flow_dv.c',
'mlx5_flow_hw.c',
diff --git a/drivers/net/mlx5/mlx5_dr.c b/drivers/net/mlx5/mlx5_dr.c
new file mode 100644
index 0000000000..7218708986
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_dr.c
@@ -0,0 +1,383 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA CORPORATION. All rights reserved.
+ */
+#include <rte_flow.h>
+
+#include "mlx5_defs.h"
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+
+/*
+ * The following null stubs are prepared in order not to break the linkage
+ * before the HW steering low-level implementation is added.
+ */
+
+/* Open a context used for direct rule insertion using hardware steering.
+ * Each context can contain multiple tables of different types.
+ *
+ * @param[in] ibv_ctx
+ * The ibv context to used for HWS.
+ * @param[in] attr
+ * Attributes used for context open.
+ * @return pointer to mlx5dr_context on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_context *
+mlx5dr_context_open(void *ibv_ctx,
+ struct mlx5dr_context_attr *attr)
+{
+ (void)ibv_ctx;
+ (void)attr;
+ return NULL;
+}
+
+/* Close a context used for direct hardware steering.
+ *
+ * @param[in] ctx
+ * mlx5dr context to close.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_context_close(struct mlx5dr_context *ctx)
+{
+ (void)ctx;
+ return 0;
+}
+
+/* Create a new direct rule table. Each table can contain multiple matchers.
+ *
+ * @param[in] ctx
+ * The context in which the new table will be opened.
+ * @param[in] attr
+ * Attributes used for table creation.
+ * @return pointer to mlx5dr_table on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_table *
+mlx5dr_table_create(struct mlx5dr_context *ctx,
+ struct mlx5dr_table_attr *attr)
+{
+ (void)ctx;
+ (void)attr;
+ return NULL;
+}
+
+/* Destroy direct rule table.
+ *
+ * @param[in] tbl
+ * mlx5dr table to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int mlx5dr_table_destroy(struct mlx5dr_table *tbl)
+{
+ (void)tbl;
+ return 0;
+}
+
+/* Create new match template based on items mask, the match template
+ * will be used for matcher creation.
+ *
+ * @param[in] items
+ * Describe the mask for template creation
+ * @param[in] flags
+ * Template creation flags
+ * @return pointer to mlx5dr_match_template on success NULL otherwise
+ */
+__rte_weak struct mlx5dr_match_template *
+mlx5dr_match_template_create(const struct rte_flow_item items[],
+ enum mlx5dr_match_template_flags flags)
+{
+ (void)items;
+ (void)flags;
+ return NULL;
+}
+
+/* Destroy match template.
+ *
+ * @param[in] mt
+ * Match template to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_match_template_destroy(struct mlx5dr_match_template *mt)
+{
+ (void)mt;
+ return 0;
+}
+
+/* Create a new direct rule matcher. Each matcher can contain multiple rules.
+ * Matchers on the table will be processed by priority. Matching fields and
+ * mask are described by the match template. In some cases multiple match
+ * templates can be used on the same matcher.
+ *
+ * @param[in] table
+ * The table in which the new matcher will be opened.
+ * @param[in] mt
+ * Array of match templates to be used on matcher.
+ * @param[in] num_of_mt
+ * Number of match templates in mt array.
+ * @param[in] attr
+ * Attributes used for matcher creation.
+ * @return pointer to mlx5dr_matcher on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_matcher *
+mlx5dr_matcher_create(struct mlx5dr_table *table __rte_unused,
+ struct mlx5dr_match_template *mt[] __rte_unused,
+ uint8_t num_of_mt __rte_unused,
+ struct mlx5dr_matcher_attr *attr __rte_unused)
+{
+ return NULL;
+}
+
+/* Destroy direct rule matcher.
+ *
+ * @param[in] matcher
+ * Matcher to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher __rte_unused)
+{
+ return 0;
+}
+
+/* Enqueue create rule operation.
+ *
+ * @param[in] matcher
+ * The matcher in which the new rule will be created.
+ * @param[in] mt_idx
+ * Match template index to create the rule with.
+ * @param[in] items
+ * The items used for the value matching.
+ * @param[in] rule_actions
+ * Rule action to be executed on match.
+ * @param[in] num_of_actions
+ * Number of rule actions.
+ * @param[in] attr
+ * Rule creation attributes.
+ * @param[in, out] rule_handle
+ * A valid rule handle. The handle doesn't require any initialization.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_rule_create(struct mlx5dr_matcher *matcher __rte_unused,
+ uint8_t mt_idx __rte_unused,
+ const struct rte_flow_item items[] __rte_unused,
+ struct mlx5dr_rule_action rule_actions[] __rte_unused,
+ uint8_t num_of_actions __rte_unused,
+ struct mlx5dr_rule_attr *attr __rte_unused,
+ struct mlx5dr_rule *rule_handle __rte_unused)
+{
+ return 0;
+}
+
+/* Enqueue destroy rule operation.
+ *
+ * @param[in] rule
+ * The rule destruction to enqueue.
+ * @param[in] attr
+ * Rule destruction attributes.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_rule_destroy(struct mlx5dr_rule *rule __rte_unused,
+ struct mlx5dr_rule_attr *attr __rte_unused)
+{
+ return 0;
+}
+
+/* Create direct rule drop action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_drop(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule default miss action.
+ * Defaults are RX: Drop TX: Wire.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_default_miss(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule goto table action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] tbl
+ * Destination table.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx __rte_unused,
+ struct mlx5dr_table *tbl __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule goto TIR action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule TIR devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx __rte_unused,
+ struct mlx5dr_devx_obj *obj __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule TAG action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_tag(struct mlx5dr_context *ctx __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule counter action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule counter devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_counter(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule reformat action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] reformat_type
+ * Type of reformat.
+ * @param[in] data_sz
+ * Size in bytes of data.
+ * @param[in] inline_data
+ * Header data array in case of inline action.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_reformat(struct mlx5dr_context *ctx __rte_unused,
+ enum mlx5dr_action_reformat_type reformat_type __rte_unused,
+ size_t data_sz __rte_unused,
+ void *inline_data __rte_unused,
+ uint32_t log_bulk_size __rte_unused,
+ uint32_t flags __rte_unused)
+{
+ return NULL;
+}
+
+/* Create direct rule modify header action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] pattern_sz
+ * Byte size of the pattern array.
+ * @param[in] pattern
+ * PRM format modify pattern action array.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+__rte_weak struct mlx5dr_action *
+mlx5dr_action_create_modify_header(struct mlx5dr_context *ctx,
+ size_t pattern_sz,
+ rte_be64_t pattern[],
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Destroy direct rule action.
+ *
+ * @param[in] action
+ * The action to destroy.
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_action_destroy(struct mlx5dr_action *action __rte_unused)
+{
+ return 0;
+}
+
+/* Poll queue for rule creation and deletions completions.
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to poll.
+ * @param[in, out] res
+ * Completion array.
+ * @param[in] res_nb
+ * Maximum number of results to return.
+ * @return negative number on failure, the number of completions otherwise.
+ */
+__rte_weak int
+mlx5dr_send_queue_poll(struct mlx5dr_context *ctx __rte_unused,
+ uint16_t queue_id __rte_unused,
+ struct rte_flow_op_result res[] __rte_unused,
+ uint32_t res_nb __rte_unused)
+{
+ return 0;
+}
+
+/* Perform an action on the queue
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to perform the action on.
+ * @param[in] actions
+ * Actions to perform on the queue. (enum mlx5dr_send_queue_actions)
+ * @return zero on success non zero otherwise.
+ */
+__rte_weak int
+mlx5dr_send_queue_action(struct mlx5dr_context *ctx __rte_unused,
+ uint16_t queue_id __rte_unused,
+ uint32_t actions __rte_unused)
+{
+ return 0;
+}
+
+#endif
diff --git a/drivers/net/mlx5/mlx5_dr.h b/drivers/net/mlx5/mlx5_dr.h
new file mode 100644
index 0000000000..d0b2c15652
--- /dev/null
+++ b/drivers/net/mlx5/mlx5_dr.h
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA CORPORATION. All rights reserved.
+ */
+
+#ifndef MLX5_DR_H_
+#define MLX5_DR_H_
+
+#include <rte_flow.h>
+
+struct mlx5dr_context;
+struct mlx5dr_table;
+struct mlx5dr_matcher;
+struct mlx5dr_rule;
+
+enum mlx5dr_table_type {
+ MLX5DR_TABLE_TYPE_NIC_RX,
+ MLX5DR_TABLE_TYPE_NIC_TX,
+ MLX5DR_TABLE_TYPE_FDB,
+ MLX5DR_TABLE_TYPE_MAX,
+};
+
+enum mlx5dr_matcher_resource_mode {
+ /* Allocate resources based on number of rules with minimal failure probability */
+ MLX5DR_MATCHER_RESOURCE_MODE_RULE,
+ /* Allocate fixed size hash table based on given column and rows */
+ MLX5DR_MATCHER_RESOURCE_MODE_HTABLE,
+};
+
+enum mlx5dr_action_flags {
+ MLX5DR_ACTION_FLAG_ROOT_RX = 1 << 0,
+ MLX5DR_ACTION_FLAG_ROOT_TX = 1 << 1,
+ MLX5DR_ACTION_FLAG_ROOT_FDB = 1 << 2,
+ MLX5DR_ACTION_FLAG_HWS_RX = 1 << 3,
+ MLX5DR_ACTION_FLAG_HWS_TX = 1 << 4,
+ MLX5DR_ACTION_FLAG_HWS_FDB = 1 << 5,
+ MLX5DR_ACTION_FLAG_INLINE = 1 << 6,
+};
+
+enum mlx5dr_action_reformat_type {
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L3_TO_L2,
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L3,
+};
+
+enum mlx5dr_match_template_flags {
+ /* Allow relaxed matching by skipping derived dependent match fields. */
+ MLX5DR_MATCH_TEMPLATE_FLAG_RELAXED_MATCH = 1,
+};
+
+enum mlx5dr_send_queue_actions {
+ /* Start executing all pending queued rules and write to HW */
+ MLX5DR_SEND_QUEUE_ACTION_DRAIN = 1 << 0,
+};
+
+struct mlx5dr_context_attr {
+ uint16_t queues;
+ uint16_t queue_size;
+ size_t initial_log_ste_memory;
+ /* Optional PD used for allocating res ources */
+ struct ibv_pd *pd;
+};
+
+struct mlx5dr_table_attr {
+ enum mlx5dr_table_type type;
+ uint32_t level;
+};
+
+struct mlx5dr_matcher_attr {
+ uint32_t priority;
+ enum mlx5dr_matcher_resource_mode mode;
+ union {
+ struct {
+ uint8_t sz_row_log;
+ uint8_t sz_col_log;
+ } table;
+
+ struct {
+ uint8_t num_log;
+ } rule;
+ };
+};
+
+struct mlx5dr_rule_attr {
+ uint16_t queue_id;
+ void *user_data;
+ uint32_t burst:1;
+};
+
+struct mlx5dr_devx_obj {
+ struct mlx5dv_devx_obj *obj;
+ uint32_t id;
+};
+
+struct mlx5dr_rule_action {
+ struct mlx5dr_action *action;
+ union {
+ struct {
+ uint32_t value;
+ } tag;
+
+ struct {
+ uint32_t offset;
+ } counter;
+
+ struct {
+ uint32_t offset;
+ uint8_t *data;
+ } modify_header;
+
+ struct {
+ uint32_t offset;
+ uint8_t *data;
+ } reformat;
+
+ struct {
+ rte_be32_t vlan_hdr;
+ } push_vlan;
+ };
+};
+
+enum {
+ MLX5DR_MATCH_TAG_SZ = 32,
+ MLX5DR_JAMBO_TAG_SZ = 44,
+};
+
+enum mlx5dr_rule_status {
+ MLX5DR_RULE_STATUS_UNKNOWN,
+ MLX5DR_RULE_STATUS_CREATING,
+ MLX5DR_RULE_STATUS_CREATED,
+ MLX5DR_RULE_STATUS_DELETING,
+ MLX5DR_RULE_STATUS_DELETED,
+ MLX5DR_RULE_STATUS_FAILED,
+};
+
+struct mlx5dr_rule {
+ struct mlx5dr_matcher *matcher;
+ union {
+ uint8_t match_tag[MLX5DR_MATCH_TAG_SZ];
+ struct ibv_flow *flow;
+ };
+ enum mlx5dr_rule_status status;
+ uint32_t rtc_used; /* The RTC into which the STE was inserted */
+};
+
+/* Open a context used for direct rule insertion using hardware steering.
+ * Each context can contain multiple tables of different types.
+ *
+ * @param[in] ibv_ctx
+ * The ibv context to used for HWS.
+ * @param[in] attr
+ * Attributes used for context open.
+ * @return pointer to mlx5dr_context on success NULL otherwise.
+ */
+struct mlx5dr_context *
+mlx5dr_context_open(void *ibv_ctx,
+ struct mlx5dr_context_attr *attr);
+
+/* Close a context used for direct hardware steering.
+ *
+ * @param[in] ctx
+ * mlx5dr context to close.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_context_close(struct mlx5dr_context *ctx);
+
+/* Create a new direct rule table. Each table can contain multiple matchers.
+ *
+ * @param[in] ctx
+ * The context in which the new table will be opened.
+ * @param[in] attr
+ * Attributes used for table creation.
+ * @return pointer to mlx5dr_table on success NULL otherwise.
+ */
+struct mlx5dr_table *
+mlx5dr_table_create(struct mlx5dr_context *ctx,
+ struct mlx5dr_table_attr *attr);
+
+/* Destroy direct rule table.
+ *
+ * @param[in] tbl
+ * mlx5dr table to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_table_destroy(struct mlx5dr_table *tbl);
+
+/* Create new match template based on items mask, the match template
+ * will be used for matcher creation.
+ *
+ * @param[in] items
+ * Describe the mask for template creation
+ * @param[in] flags
+ * Template creation flags
+ * @return pointer to mlx5dr_match_template on success NULL otherwise
+ */
+struct mlx5dr_match_template *
+mlx5dr_match_template_create(const struct rte_flow_item items[],
+ enum mlx5dr_match_template_flags flags);
+
+/* Destroy match template.
+ *
+ * @param[in] mt
+ * Match template to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_match_template_destroy(struct mlx5dr_match_template *mt);
+
+/* Create a new direct rule matcher. Each matcher can contain multiple rules.
+ * Matchers on the table will be processed by priority. Matching fields and
+ * mask are described by the match template. In some cases multiple match
+ * templates can be used on the same matcher.
+ *
+ * @param[in] table
+ * The table in which the new matcher will be opened.
+ * @param[in] mt
+ * Array of match templates to be used on matcher.
+ * @param[in] num_of_mt
+ * Number of match templates in mt array.
+ * @param[in] attr
+ * Attributes used for matcher creation.
+ * @return pointer to mlx5dr_matcher on success NULL otherwise.
+ */
+struct mlx5dr_matcher *
+mlx5dr_matcher_create(struct mlx5dr_table *table,
+ struct mlx5dr_match_template *mt[],
+ uint8_t num_of_mt,
+ struct mlx5dr_matcher_attr *attr);
+
+/* Destroy direct rule matcher.
+ *
+ * @param[in] matcher
+ * Matcher to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher);
+
+/* Get the size of the rule handle (mlx5dr_rule) to be used on rule creation.
+ *
+ * @return size in bytes of rule handle struct.
+ */
+size_t mlx5dr_rule_get_handle_size(void);
+
+/* Enqueue create rule operation.
+ *
+ * @param[in] matcher
+ * The matcher in which the new rule will be created.
+ * @param[in] mt_idx
+ * Match template index to create the rule with.
+ * @param[in] items
+ * The items used for the value matching.
+ * @param[in] rule_actions
+ * Rule action to be executed on match.
+ * @param[in] num_of_actions
+ * Number of rule actions.
+ * @param[in] attr
+ * Rule creation attributes.
+ * @param[in, out] rule_handle
+ * A valid rule handle. The handle doesn't require any initialization.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+int mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
+ uint8_t mt_idx,
+ const struct rte_flow_item items[],
+ struct mlx5dr_rule_action rule_actions[],
+ uint8_t num_of_actions,
+ struct mlx5dr_rule_attr *attr,
+ struct mlx5dr_rule *rule_handle);
+
+/* Enqueue destroy rule operation.
+ *
+ * @param[in] rule
+ * The rule destruction to enqueue.
+ * @param[in] attr
+ * Rule destruction attributes.
+ * @return zero on successful enqueue non zero otherwise.
+ */
+int mlx5dr_rule_destroy(struct mlx5dr_rule *rule,
+ struct mlx5dr_rule_attr *attr);
+
+/* Create direct rule drop action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_drop(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule default miss action.
+ * Defaults are RX: Drop TX: Wire.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_default_miss(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule goto table action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] tbl
+ * Destination table.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx,
+ struct mlx5dr_table *tbl,
+ uint32_t flags);
+
+/* Create direct rule goto TIR action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule TIR devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule TAG action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_tag(struct mlx5dr_context *ctx,
+ uint32_t flags);
+
+/* Create direct rule counter action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] obj
+ * Direct rule counter devx object.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_counter(struct mlx5dr_context *ctx,
+ struct mlx5dr_devx_obj *obj,
+ uint32_t flags);
+
+/* Create direct rule reformat action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] reformat_type
+ * Type of reformat.
+ * @param[in] data_sz
+ * Size in bytes of data.
+ * @param[in] inline_data
+ * Header data array in case of inline action.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_reformat(struct mlx5dr_context *ctx,
+ enum mlx5dr_action_reformat_type reformat_type,
+ size_t data_sz,
+ void *inline_data,
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Create direct rule modify header action.
+ *
+ * @param[in] ctx
+ * The context in which the new action will be created.
+ * @param[in] pattern_sz
+ * Byte size of the pattern array.
+ * @param[in] pattern
+ * PRM format modify pattern action array.
+ * @param[in] log_bulk_size
+ * Number of unique values used with this pattern.
+ * @param[in] flags
+ * Action creation flags. (enum mlx5dr_action_flags)
+ * @return pointer to mlx5dr_action on success NULL otherwise.
+ */
+struct mlx5dr_action *
+mlx5dr_action_create_modify_header(struct mlx5dr_context *ctx,
+ size_t pattern_sz,
+ rte_be64_t pattern[],
+ uint32_t log_bulk_size,
+ uint32_t flags);
+
+/* Destroy direct rule action.
+ *
+ * @param[in] action
+ * The action to destroy.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_action_destroy(struct mlx5dr_action *action);
+
+/* Poll queue for rule creation and deletions completions.
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to poll.
+ * @param[in, out] res
+ * Completion array.
+ * @param[in] res_nb
+ * Maximum number of results to return.
+ * @return negative number on failure, the number of completions otherwise.
+ */
+int mlx5dr_send_queue_poll(struct mlx5dr_context *ctx,
+ uint16_t queue_id,
+ struct rte_flow_op_result res[],
+ uint32_t res_nb);
+
+/* Perform an action on the queue
+ *
+ * @param[in] ctx
+ * The context to which the queue belong to.
+ * @param[in] queue_id
+ * The id of the queue to perform the action on.
+ * @param[in] actions
+ * Actions to perform on the queue. (enum mlx5dr_send_queue_actions)
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_send_queue_action(struct mlx5dr_context *ctx,
+ uint16_t queue_id,
+ uint32_t actions);
+
+/* Dump HWS info
+ *
+ * @param[in] ctx
+ * The context which to dump the info from.
+ * @param[in] f
+ * The file to write the dump to.
+ * @return zero on success non zero otherwise.
+ */
+int mlx5dr_debug_dump(struct mlx5dr_context *ctx, FILE *f);
+
+#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 03/14] net/mlx5: introduce hardware steering enable routine
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-24 13:40 ` [PATCH v4 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-24 13:40 ` [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 04/14] net/mlx5: add port flow configuration Suanming Mou
` (11 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The new hardware steering engine relies on using dedicated steering WQEs
instead of direct writing to the low-level steering table entries directly.
In the first introduce implementation the hardware steering engine supports
the new queue based Flow API, the existing synchronous non-queue based Flow
API is not supported.
A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:
dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support
This commit introduces the extra dv_flow_en = 2 to specify the new
flow initialize and manage operation routine.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
doc/guides/nics/mlx5.rst | 13 ++++++++++---
drivers/net/mlx5/linux/mlx5_os.c | 4 ++++
drivers/net/mlx5/mlx5.c | 7 ++++++-
drivers/net/mlx5/mlx5.h | 3 ++-
drivers/net/mlx5/mlx5_flow.c | 22 ++++++++++++++++++++++
5 files changed, 44 insertions(+), 5 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 34be031360..92127675a3 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -943,10 +943,17 @@ for an additional list of options shared with other mlx5 drivers.
- ``dv_flow_en`` parameter [int]
- A nonzero value enables the DV flow steering assuming it is supported
- by the driver (RDMA Core library version is rdma-core-24.0 or higher).
+ Value 0 means legacy Verbs flow offloading.
- Enabled by default if supported.
+ Value 1 enables the DV flow steering assuming it is supported by the
+ driver (RDMA Core library version is rdma-core-24.0 or higher).
+
+ Value 2 enables the WQE based hardware steering. In this mode only
+ the queue-based rte_flow_q flow management is supported.
+
+ Configured by default to 1 DV flow steering if the driver(RDMA CORE library)
+ supported. Otherwise, the value will be 0 which indicates legacy Verbs flow
+ offloading.
- ``dv_esw_en`` parameter [int]
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 058c140fe1..5b91e057b2 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
err = mlx5_alloc_table_hash_list(priv);
if (err)
goto error;
+ if (priv->sh->config.dv_flow_en == 2)
+ return 0;
/* The resources below are only valid with DV support. */
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
/* Init port id action list. */
@@ -1535,6 +1537,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ if (priv->sh->config.dv_flow_en == 2)
+ return eth_dev;
/* Port representor shares the same max priority with pf port. */
if (!priv->sh->flow_priority_check_flag) {
/* Supported Verbs flow priority number detection. */
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 9760f52b46..baa4ae75f3 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque)
} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
config->dv_esw_en = !!tmp;
} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
- config->dv_flow_en = !!tmp;
+ if (tmp > 2) {
+ DRV_LOG(ERR, "Invalid %s parameter.", key);
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ config->dv_flow_en = tmp;
} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
if (tmp != MLX5_XMETA_MODE_LEGACY &&
tmp != MLX5_XMETA_MODE_META16 &&
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index fa27f65a36..d36d679625 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -287,7 +287,8 @@ struct mlx5_sh_config {
int tx_skew; /* Tx scheduling skew between WQE and data on wire. */
uint32_t reclaim_mode:2; /* Memory reclaim mode. */
uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */
- uint32_t dv_flow_en:1; /* Enable DV flow. */
+ /* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+ unsigned int dv_flow_en:2;
uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */
uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */
uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index aec14ea39d..ef7f2ceaee 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -6849,6 +6849,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,
const struct rte_flow_action actions[],
struct rte_flow_error *error)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (priv->sh->config.dv_flow_en == 2) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q creation not supported");
+ return NULL;
+ }
/*
* If the device is not started yet, it is not allowed to created a
* flow from application. PMD default flows and traffic control flows
@@ -7345,6 +7354,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,
struct rte_flow *flow,
struct rte_flow_error *error __rte_unused)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (priv->sh->config.dv_flow_en == 2)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q destruction not supported");
flow_list_destroy(dev, MLX5_FLOW_TYPE_GEN,
(uintptr_t)(void *)flow);
return 0;
@@ -7442,7 +7458,13 @@ mlx5_flow_query(struct rte_eth_dev *dev,
struct rte_flow_error *error)
{
int ret;
+ struct mlx5_priv *priv = dev->data->dev_private;
+ if (priv->sh->config.dv_flow_en == 2)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "Flow non-Q query not supported");
ret = flow_drv_query(dev, (uintptr_t)(void *)flow, actions, data,
error);
if (ret < 0)
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 04/14] net/mlx5: add port flow configuration
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (2 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 05/14] net/mlx5: add pattern template management Suanming Mou
` (10 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The hardware steering is backend to support rte_flow_async API in
mlx5 PMD. The port configuration function creates the queues and
needed flow management resources.
The PMD layer configuration function allocates the queues' context
and per-queue job descriptor pool. The job descriptor pool size
is equal to the queue size, and the job descriptors will be popped
from pool with LIFO strategy to convey the flow information during
flow insertion/destruction. Then, while polling the queued operation
result, the flow information will be extracted from the job descriptor
and the descriptor will be pushed back to the LIFO pool.
The commit creates the flow port queues and the job descriptor pools.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 3 +
drivers/net/mlx5/mlx5.h | 30 +++++-
drivers/net/mlx5/mlx5_flow.c | 86 +++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 15 +++
drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++
5 files changed, 292 insertions(+), 1 deletion(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index baa4ae75f3..24bbfee519 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
/* Free the eCPRI flex parser resource. */
mlx5_flex_parser_ecpri_release(dev);
mlx5_flex_item_port_cleanup(dev);
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ flow_hw_resource_release(dev);
+#endif
if (priv->rxq_privs != NULL) {
/* XXX race condition if mlx5_rx_burst() is still running. */
rte_delay_us_sleep(1000);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index d36d679625..86c84d1510 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -33,7 +33,9 @@
#include "mlx5_utils.h"
#include "mlx5_os.h"
#include "mlx5_autoconf.h"
-
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+#endif
#define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {
uint16_t refcnt; /* Reference count for representors. */
};
+/* HW steering queue job descriptor type. */
+enum {
+ MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+ MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+ uint32_t type; /* Job type. */
+ struct rte_flow *flow; /* Flow attached to the job. */
+ void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO pool. */
+struct mlx5_hw_q {
+ uint32_t job_idx; /* Free job index. */
+ uint32_t size; /* LIFO size. */
+ struct mlx5_hw_q_job **job; /* LIFO header. */
+} __rte_cache_aligned;
+
#define MLX5_COUNTERS_PER_POOL 512
#define MLX5_MAX_PENDING_QUERIES 4
#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1480,6 +1502,12 @@ struct mlx5_priv {
struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+ uint32_t nb_queue; /* HW steering queue number. */
+ /* HW steering queue polling mechanism job descriptor LIFO. */
+ struct mlx5_hw_q *hw_q;
+#endif
};
#define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index ef7f2ceaee..156fb35ce9 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -805,6 +805,17 @@ static int
mlx5_flow_flex_item_release(struct rte_eth_dev *dev,
const struct rte_flow_item_flex_handle *handle,
struct rte_flow_error *error);
+static int
+mlx5_flow_info_get(struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -826,6 +837,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.get_restore_info = mlx5_flow_tunnel_get_restore_info,
.flex_item_create = mlx5_flow_flex_item_create,
.flex_item_release = mlx5_flow_flex_item_release,
+ .info_get = mlx5_flow_info_get,
+ .configure = mlx5_flow_port_configure,
};
/* Tunnel information. */
@@ -3440,6 +3453,12 @@ flow_get_drv_type(struct rte_eth_dev *dev, const struct rte_flow_attr *attr)
if (type != MLX5_FLOW_TYPE_MAX)
return type;
+ /*
+ * Currently when dv_flow_en == 2, only HW steering engine is
+ * supported. New engines can also be chosen here if ready.
+ */
+ if (priv->sh->config.dv_flow_en == 2)
+ return MLX5_FLOW_TYPE_HW;
/* If no OS specific type - continue with DV/VERBS selection */
if (attr->transfer && priv->sh->config.dv_esw_en)
type = MLX5_FLOW_TYPE_DV;
@@ -7847,6 +7866,73 @@ mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt,
return -ENOTSUP;
}
+/**
+ * Get information about HWS pre-configurable resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] port_info
+ * Pointer to port information.
+ * @param[out] queue_info
+ * Pointer to queue information.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_info_get(struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "info get with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->info_get(dev, port_info, queue_info, error);
+}
+
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "port configure with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->configure(dev, port_attr, nb_queue, queue_attr, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index b70ef0c1b8..9f0dc4bde7 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)
const struct rte_flow_item_flex_handle *handle,
const struct rte_flow_item_flex_conf *conf,
struct rte_flow_error *error);
+typedef int (*mlx5_flow_info_get_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_port_info *port_info,
+ struct rte_flow_queue_info *queue_info,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *err);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_create_t item_create;
mlx5_flow_item_release_t item_release;
mlx5_flow_item_update_t item_update;
+ mlx5_flow_info_get_t info_get;
+ mlx5_flow_port_configure_t configure;
};
/* mlx5_flow.c */
@@ -1762,4 +1775,6 @@ const struct mlx5_flow_tunnel *
mlx5_get_tof(const struct rte_flow_item *items,
const struct rte_flow_action *actions,
enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 729d5914a8..e5b2ae91d8 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -4,10 +4,169 @@
#include <rte_flow.h>
+#include <mlx5_malloc.h>
+#include "mlx5_defs.h"
#include "mlx5_flow.h"
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Get information about HWS pre-configurable resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] port_info
+ * Pointer to port information.
+ * @param[out] queue_info
+ * Pointer to queue information.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_port_info *port_info __rte_unused,
+ struct rte_flow_queue_info *queue_info __rte_unused,
+ struct rte_flow_error *error __rte_unused)
+{
+ /* Nothing to be updated currently. */
+ memset(port_info, 0, sizeof(*port_info));
+ /* Queue size is unlimited from low-level. */
+ queue_info->max_size = UINT32_MAX;
+ return 0;
+}
+
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ * Port configuration attributes.
+ * @param[in] nb_queue
+ * Number of queue.
+ * @param[in] queue_attr
+ * Array that holds attributes for each flow queue.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_configure(struct rte_eth_dev *dev,
+ const struct rte_flow_port_attr *port_attr,
+ uint16_t nb_queue,
+ const struct rte_flow_queue_attr *queue_attr[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_context *dr_ctx = NULL;
+ struct mlx5dr_context_attr dr_ctx_attr = {0};
+ struct mlx5_hw_q *hw_q;
+ struct mlx5_hw_q_job *job = NULL;
+ uint32_t mem_size, i, j;
+
+ if (!port_attr || !nb_queue || !queue_attr) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ /* In case re-configuring, release existing context at first. */
+ if (priv->dr_ctx) {
+ /* */
+ for (i = 0; i < nb_queue; i++) {
+ hw_q = &priv->hw_q[i];
+ /* Make sure all queues are empty. */
+ if (hw_q->size != hw_q->job_idx) {
+ rte_errno = EBUSY;
+ goto err;
+ }
+ }
+ flow_hw_resource_release(dev);
+ }
+ /* Allocate the queue job descriptor LIFO. */
+ mem_size = sizeof(priv->hw_q[0]) * nb_queue;
+ for (i = 0; i < nb_queue; i++) {
+ /*
+ * Check if the queues' size are all the same as the
+ * limitation from HWS layer.
+ */
+ if (queue_attr[i]->size != queue_attr[0]->size) {
+ rte_errno = EINVAL;
+ goto err;
+ }
+ mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(struct mlx5_hw_q_job)) *
+ queue_attr[0]->size;
+ }
+ priv->hw_q = mlx5_malloc(MLX5_MEM_ZERO, mem_size,
+ 64, SOCKET_ID_ANY);
+ if (!priv->hw_q) {
+ rte_errno = ENOMEM;
+ goto err;
+ }
+ for (i = 0; i < nb_queue; i++) {
+ priv->hw_q[i].job_idx = queue_attr[i]->size;
+ priv->hw_q[i].size = queue_attr[i]->size;
+ if (i == 0)
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &priv->hw_q[nb_queue];
+ else
+ priv->hw_q[i].job = (struct mlx5_hw_q_job **)
+ &job[queue_attr[i - 1]->size];
+ job = (struct mlx5_hw_q_job *)
+ &priv->hw_q[i].job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++)
+ priv->hw_q[i].job[j] = &job[j];
+ }
+ dr_ctx_attr.pd = priv->sh->cdev->pd;
+ dr_ctx_attr.queues = nb_queue;
+ /* Queue size should all be the same. Take the first one. */
+ dr_ctx_attr.queue_size = queue_attr[0]->size;
+ dr_ctx = mlx5dr_context_open(priv->sh->cdev->ctx, &dr_ctx_attr);
+ /* rte_errno has been updated by HWS layer. */
+ if (!dr_ctx)
+ goto err;
+ priv->dr_ctx = dr_ctx;
+ priv->nb_queue = nb_queue;
+ return 0;
+err:
+ if (dr_ctx)
+ claim_zero(mlx5dr_context_close(dr_ctx));
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to configure port");
+}
+
+/**
+ * Release HWS resources.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ */
+void
+flow_hw_resource_release(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (!priv->dr_ctx)
+ return;
+ mlx5_free(priv->hw_q);
+ priv->hw_q = NULL;
+ claim_zero(mlx5dr_context_close(priv->dr_ctx));
+ priv->dr_ctx = NULL;
+ priv->nb_queue = 0;
+}
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
+ .info_get = flow_hw_info_get,
+ .configure = flow_hw_configure,
+};
+
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 05/14] net/mlx5: add pattern template management
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (3 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 04/14] net/mlx5: add port flow configuration Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 06/14] net/mlx5: add action " Suanming Mou
` (9 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The pattern template defines flows that have the same matching
fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be
(eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while
the values for each rule will be different.
Due to the pattern template can be used in different domains, the
items will only be cached in pattern template create stage, while
the template is binded to a dedicated table, the HW criteria will
be created and saved to the table. The pattern templates can be
used by multiple tables. But different tables create the same
criteria and will not share the matcher between each other in order
to have better performance.
This commit adds pattern template management.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 76 +++++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 24 +++++++++
drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++
4 files changed, 188 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 86c84d1510..7b458c410d 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1503,6 +1503,8 @@ struct mlx5_priv {
/* Flex items have been created on the port. */
uint32_t flex_item_map; /* Map of allocated flex item elements. */
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ /* Item template list. */
+ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 156fb35ce9..a603b69d36 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -817,6 +817,17 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -839,6 +850,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.flex_item_release = mlx5_flow_flex_item_release,
.info_get = mlx5_flow_info_get,
.configure = mlx5_flow_port_configure,
+ .pattern_template_create = mlx5_flow_pattern_template_create,
+ .pattern_template_destroy = mlx5_flow_pattern_template_destroy,
};
/* Tunnel information. */
@@ -7933,6 +7946,69 @@ mlx5_flow_port_configure(struct rte_eth_dev *dev,
return fops->configure(dev, port_attr, nb_queue, queue_attr, error);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+mlx5_flow_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "pattern create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pattern_template_create(dev, attr, items, error);
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "pattern destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pattern_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 9f0dc4bde7..49027d6a3a 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1015,6 +1015,19 @@ struct rte_flow {
uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
} __rte_packed;
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+ LIST_ENTRY(rte_flow_pattern_template) next;
+ /* Template attributes. */
+ struct rte_flow_pattern_template_attr attr;
+ struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint32_t refcnt; /* Reference counter. */
+};
+
+#endif
+
/*
* Define list of valid combinations of RX Hash fields
* (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)
uint16_t nb_queue,
const struct rte_flow_queue_attr *queue_attr[],
struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_item_update_t item_update;
mlx5_flow_info_get_t info_get;
mlx5_flow_port_configure_t configure;
+ mlx5_flow_pattern_template_create_t pattern_template_create;
+ mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index e5b2ae91d8..66c5825084 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -13,6 +13,85 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the item template attributes.
+ * @param[in] items
+ * The template item pattern.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Item template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_pattern_template *
+flow_hw_pattern_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_pattern_template_attr *attr,
+ const struct rte_flow_item items[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
+
+ it = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*it), 0, rte_socket_id());
+ if (!it) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate item template");
+ return NULL;
+ }
+ it->attr = *attr;
+ it->mt = mlx5dr_match_template_create(items, attr->relaxed_matching);
+ if (!it->mt) {
+ mlx5_free(it);
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot create match template");
+ return NULL;
+ }
+ __atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
+ return it;
+}
+
+/**
+ * Destroy flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the item template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_pattern_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_pattern_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Item template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "item template in using");
+ }
+ LIST_REMOVE(template, next);
+ claim_zero(mlx5dr_match_template_destroy(template->mt));
+ mlx5_free(template);
+ return 0;
+}
+
+/*
* Get information about HWS pre-configurable resources.
*
* @param[in] dev
@@ -154,9 +233,14 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_pattern_template *it;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_itt)) {
+ it = LIST_FIRST(&priv->flow_hw_itt);
+ flow_hw_pattern_template_destroy(dev, it, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -167,6 +251,8 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.info_get = flow_hw_info_get,
.configure = flow_hw_configure,
+ .pattern_template_create = flow_hw_pattern_template_create,
+ .pattern_template_destroy = flow_hw_pattern_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 06/14] net/mlx5: add action template management
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (4 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 05/14] net/mlx5: add pattern template management Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 07/14] net/mlx5: add table management Suanming Mou
` (8 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The action template holds a list of action types that will be
used together on the same rule. The template's actions instances
will be created only when the template bind to the dedicated
group. And the created actions will be saved to each individual
group in order for best performance. The actions in a group will
not be shared with each other unless shared actions are specified.
This commit adds the action template management which stores the
flow action template.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.c | 78 +++++++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 22 ++++++
drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
4 files changed, 218 insertions(+)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 7b458c410d..1722d38aad 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1505,6 +1505,8 @@ struct mlx5_priv {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
/* Item template list. */
LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+ /* Action template list. */
+ LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index a603b69d36..26af6d0106 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -827,6 +827,16 @@ static int
mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -852,6 +862,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.configure = mlx5_flow_port_configure,
.pattern_template_create = mlx5_flow_pattern_template_create,
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
+ .actions_template_create = mlx5_flow_actions_template_create,
+ .actions_template_destroy = mlx5_flow_actions_template_destroy,
};
/* Tunnel information. */
@@ -8009,6 +8021,72 @@ mlx5_flow_pattern_template_destroy(struct rte_eth_dev *dev,
return fops->pattern_template_destroy(dev, template, error);
}
+/**
+ * Create flow item template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+mlx5_flow_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->actions_template_create(dev, attr, actions, masks, error);
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->actions_template_destroy(dev, template, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 49027d6a3a..9a643fe0a8 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+ LIST_ENTRY(rte_flow_actions_template) next;
+ /* Template attributes. */
+ struct rte_flow_actions_template_attr attr;
+ struct rte_flow_action *actions; /* Cached flow actions. */
+ struct rte_flow_action *masks; /* Cached action masks.*/
+ uint32_t refcnt; /* Reference counter. */
+};
+
#endif
/*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_pattern_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_port_configure_t configure;
mlx5_flow_pattern_template_create_t pattern_template_create;
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+ mlx5_flow_actions_template_create_t actions_template_create;
+ mlx5_flow_actions_template_destroy_t actions_template_destroy;
};
/* mlx5_flow.c */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 66c5825084..4214a63a73 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,115 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/**
+ * Create flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the action template attributes.
+ * @param[in] actions
+ * Associated actions (list terminated by the END action).
+ * @param[in] masks
+ * List of actions that marks which of the action's member is constant.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action template pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_actions_template *
+flow_hw_actions_template_create(struct rte_eth_dev *dev,
+ const struct rte_flow_actions_template_attr *attr,
+ const struct rte_flow_action actions[],
+ const struct rte_flow_action masks[],
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int len, act_len, mask_len, i;
+ struct rte_flow_actions_template *at;
+
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, actions, error);
+ if (act_len <= 0)
+ return NULL;
+ len = RTE_ALIGN(act_len, 16);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS,
+ NULL, 0, masks, error);
+ if (mask_len <= 0)
+ return NULL;
+ len += RTE_ALIGN(mask_len, 16);
+ at = mlx5_malloc(MLX5_MEM_ZERO, len + sizeof(*at), 64, rte_socket_id());
+ if (!at) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate action template");
+ return NULL;
+ }
+ at->attr = *attr;
+ at->actions = (struct rte_flow_action *)(at + 1);
+ act_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->actions, len,
+ actions, error);
+ if (act_len <= 0)
+ goto error;
+ at->masks = (struct rte_flow_action *)
+ (((uint8_t *)at->actions) + act_len);
+ mask_len = rte_flow_conv(RTE_FLOW_CONV_OP_ACTIONS, at->masks,
+ len - act_len, masks, error);
+ if (mask_len <= 0)
+ goto error;
+ /*
+ * mlx5 PMD hacks indirect action index directly to the action conf.
+ * The rte_flow_conv() function copies the content from conf pointer.
+ * Need to restore the indirect action index from action conf here.
+ */
+ for (i = 0; actions->type != RTE_FLOW_ACTION_TYPE_END;
+ actions++, masks++, i++) {
+ if (actions->type == RTE_FLOW_ACTION_TYPE_INDIRECT) {
+ at->actions[i].conf = actions->conf;
+ at->masks[i].conf = masks->conf;
+ }
+ }
+ __atomic_fetch_add(&at->refcnt, 1, __ATOMIC_RELAXED);
+ LIST_INSERT_HEAD(&priv->flow_hw_at, at, next);
+ return at;
+error:
+ mlx5_free(at);
+ return NULL;
+}
+
+/**
+ * Destroy flow action template.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] template
+ * Pointer to the action template to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused,
+ struct rte_flow_actions_template *template,
+ struct rte_flow_error *error __rte_unused)
+{
+ if (__atomic_load_n(&template->refcnt, __ATOMIC_RELAXED) > 1) {
+ DRV_LOG(WARNING, "Action template %p is still in use.",
+ (void *)template);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "action template in using");
+ }
+ LIST_REMOVE(template, next);
+ mlx5_free(template);
+ return 0;
+}
+
/**
* Create flow item template.
*
@@ -234,6 +343,7 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct rte_flow_pattern_template *it;
+ struct rte_flow_actions_template *at;
if (!priv->dr_ctx)
return;
@@ -241,6 +351,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
}
+ while (!LIST_EMPTY(&priv->flow_hw_at)) {
+ at = LIST_FIRST(&priv->flow_hw_at);
+ flow_hw_actions_template_destroy(dev, at, NULL);
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -253,6 +367,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.configure = flow_hw_configure,
.pattern_template_create = flow_hw_pattern_template_create,
.pattern_template_destroy = flow_hw_pattern_template_destroy,
+ .actions_template_create = flow_hw_actions_template_create,
+ .actions_template_destroy = flow_hw_actions_template_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 07/14] net/mlx5: add table management
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (5 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 06/14] net/mlx5: add action " Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 08/14] net/mlx5: add basic flow queue operation Suanming Mou
` (7 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Flow table is a group of flows with the same matching criteria
and the same actions defined for them. The table defines rules
that have the same matching fields but with different matching
values. For example, matching on 5 tuple, the table will be
(IPv4 source + IPv4 dest + s_port + d_port + next_proto)
while the values for each rule will be different.
The templates' relevant matching criteria and action instances
will be created in the table creation and saved in the table.
As table attributes indicate the supported flow number, the flow
memory will also be allocated at the same time.
This commit adds the table management functions.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.c | 45 ++-
drivers/net/mlx5/mlx5.h | 21 +-
drivers/net/mlx5/mlx5_flow.c | 93 ++++++
drivers/net/mlx5/mlx5_flow.h | 73 +++++
drivers/net/mlx5/mlx5_flow_hw.c | 527 ++++++++++++++++++++++++++++++++
5 files changed, 753 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 24bbfee519..44e3e76737 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1607,12 +1607,46 @@ void
mlx5_free_table_hash_list(struct mlx5_priv *priv)
{
struct mlx5_dev_ctx_shared *sh = priv->sh;
-
- if (!sh->flow_tbls)
+ struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en == 2) ?
+ &sh->groups : &sh->flow_tbls;
+ if (*tbls == NULL)
return;
- mlx5_hlist_destroy(sh->flow_tbls);
- sh->flow_tbls = NULL;
+ mlx5_hlist_destroy(*tbls);
+ *tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ * Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+ int err = 0;
+ struct mlx5_dev_ctx_shared *sh = priv->sh;
+ char s[MLX5_NAME_SIZE];
+
+ MLX5_ASSERT(sh);
+ snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+ sh->groups = mlx5_hlist_create
+ (s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+ false, true, sh,
+ flow_hw_grp_create_cb,
+ flow_hw_grp_match_cb,
+ flow_hw_grp_remove_cb,
+ flow_hw_grp_clone_cb,
+ flow_hw_grp_clone_free_cb);
+ if (!sh->groups) {
+ DRV_LOG(ERR, "flow groups with hash creation failed.");
+ err = ENOMEM;
+ }
+ return err;
}
+#endif
+
/**
* Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int
mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
{
int err = 0;
+
/* Tables are only used in DV and DR modes. */
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
struct mlx5_dev_ctx_shared *sh = priv->sh;
char s[MLX5_NAME_SIZE];
+ if (priv->sh->config.dv_flow_en == 2)
+ return mlx5_alloc_hw_group_hash_list(priv);
MLX5_ASSERT(sh);
snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 1722d38aad..4eedc8859f 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {
MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
MLX5_IPOOL_TAG, /* Pool for tag resource. */
MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
- MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+ MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+ /* Pool for HWS group. Jump action will be created internally. */
+ MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode {
MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
};
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+ MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+ MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+ MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
/* Hlist and list callback context. */
struct mlx5_flow_cb_ctx {
struct rte_eth_dev *dev;
@@ -1204,7 +1213,10 @@ struct mlx5_dev_ctx_shared {
rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
/* UAR same-page access control required in 32bit implementations. */
#endif
- struct mlx5_hlist *flow_tbls;
+ union {
+ struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+ struct mlx5_hlist *groups; /* HWS flow group. */
+ };
struct mlx5_flow_tunnel_hub *tunnel_hub;
/* Direct Rules tables for FDB, NIC TX+RX */
void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1511,6 +1523,11 @@ struct mlx5_priv {
uint32_t nb_queue; /* HW steering queue number. */
/* HW steering queue polling mechanism job descriptor LIFO. */
struct mlx5_hw_q *hw_q;
+ /* HW steering rte flow table list header. */
+ LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX];
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 26af6d0106..472a5b975d 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -838,6 +838,19 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -864,6 +877,8 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.pattern_template_destroy = mlx5_flow_pattern_template_destroy,
.actions_template_create = mlx5_flow_actions_template_create,
.actions_template_destroy = mlx5_flow_actions_template_destroy,
+ .template_table_create = mlx5_flow_table_create,
+ .template_table_destroy = mlx5_flow_table_destroy,
};
/* Tunnel information. */
@@ -8087,6 +8102,84 @@ mlx5_flow_actions_template_destroy(struct rte_eth_dev *dev,
return fops->actions_template_destroy(dev, template, error);
}
+/**
+ * Create flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+mlx5_flow_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->template_table_create(dev,
+ attr,
+ item_templates,
+ nb_item_templates,
+ action_templates,
+ nb_action_templates,
+ error);
+}
+
+/**
+ * PMD destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->template_table_destroy(dev, table, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 9a643fe0a8..1579036f58 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1036,6 +1036,55 @@ struct rte_flow_actions_template {
uint32_t refcnt; /* Reference counter. */
};
+/* Jump action struct. */
+struct mlx5_hw_jump_action {
+ /* Action jump from root. */
+ struct mlx5dr_action *root_action;
+ /* HW steering jump action. */
+ struct mlx5dr_action *hws_action;
+};
+
+/* DR action set struct. */
+struct mlx5_hw_actions {
+ struct mlx5dr_action *drop; /* Drop action. */
+};
+
+/* mlx5 action template struct. */
+struct mlx5_hw_action_template {
+ /* Action template pointer. */
+ struct rte_flow_actions_template *action_template;
+ struct mlx5_hw_actions acts; /* Template actions. */
+};
+
+/* mlx5 flow group struct. */
+struct mlx5_flow_group {
+ struct mlx5_list_entry entry;
+ struct mlx5dr_table *tbl; /* HWS table object. */
+ struct mlx5_hw_jump_action jump; /* Jump action. */
+ enum mlx5dr_table_type type; /* Table type. */
+ uint32_t group_id; /* Group id. */
+ uint32_t idx; /* Group memory index. */
+};
+
+
+#define MLX5_HW_TBL_MAX_ITEM_TEMPLATE 2
+#define MLX5_HW_TBL_MAX_ACTION_TEMPLATE 32
+
+struct rte_flow_template_table {
+ LIST_ENTRY(rte_flow_template_table) next;
+ struct mlx5_flow_group *grp; /* The group rte_flow_template_table uses. */
+ struct mlx5dr_matcher *matcher; /* Template matcher. */
+ /* Item templates bind to the table. */
+ struct rte_flow_pattern_template *its[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ /* Action templates bind to the table. */
+ struct mlx5_hw_action_template ats[MLX5_HW_TBL_MAX_ACTION_TEMPLATE];
+ struct mlx5_indexed_pool *flow; /* The table's flow ipool. */
+ uint32_t type; /* Flow table type RX/TX/FDB. */
+ uint8_t nb_item_templates; /* Item template number. */
+ uint8_t nb_action_templates; /* Action template number. */
+ uint32_t refcnt; /* Table reference counter. */
+};
+
#endif
/*
@@ -1310,6 +1359,18 @@ typedef int (*mlx5_flow_actions_template_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_actions_template *template,
struct rte_flow_error *error);
+typedef struct rte_flow_template_table *(*mlx5_flow_table_create_t)
+ (struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_table_destroy_t)
+ (struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1354,6 +1415,8 @@ struct mlx5_flow_driver_ops {
mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
mlx5_flow_actions_template_create_t actions_template_create;
mlx5_flow_actions_template_destroy_t actions_template_destroy;
+ mlx5_flow_table_create_t template_table_create;
+ mlx5_flow_table_destroy_t template_table_destroy;
};
/* mlx5_flow.c */
@@ -1784,6 +1847,16 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx,
void flow_dv_dest_array_clone_free_cb(void *tool_ctx,
struct mlx5_list_entry *entry);
+struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
+void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+int flow_hw_grp_match_cb(void *tool_ctx,
+ struct mlx5_list_entry *entry,
+ void *cb_ctx);
+struct mlx5_list_entry *flow_hw_grp_clone_cb(void *tool_ctx,
+ struct mlx5_list_entry *oentry,
+ void *cb_ctx);
+void flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry);
+
struct mlx5_aso_age_action *flow_aso_age_get_by_idx(struct rte_eth_dev *dev,
uint32_t age_idx);
int flow_dev_geneve_tlv_option_resource_register(struct rte_eth_dev *dev,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 4214a63a73..c1e3b56f23 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -12,6 +12,308 @@
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+/* DR action flags with different table. */
+static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
+ [MLX5DR_TABLE_TYPE_MAX] = {
+ {
+ MLX5DR_ACTION_FLAG_ROOT_RX,
+ MLX5DR_ACTION_FLAG_ROOT_TX,
+ MLX5DR_ACTION_FLAG_ROOT_FDB,
+ },
+ {
+ MLX5DR_ACTION_FLAG_HWS_RX,
+ MLX5DR_ACTION_FLAG_HWS_TX,
+ MLX5DR_ACTION_FLAG_HWS_FDB,
+ },
+};
+
+/**
+ * Destroy DR actions created by action template.
+ *
+ * For DR actions created during table creation's action translate.
+ * Need to destroy the DR action when destroying the table.
+ *
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ */
+static void
+__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+{
+}
+
+/**
+ * Translate rte_flow actions to DR action.
+ *
+ * As the action template has already indicated the actions. Translate
+ * the rte_flow actions to DR action if possbile. So in flow create
+ * stage we will save cycles from handing the actions' organizing.
+ * For the actions with limited information, need to add these to a
+ * list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in/out] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] at
+ * Action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static int
+flow_hw_actions_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *table_attr,
+ struct mlx5_hw_actions *acts,
+ struct rte_flow_actions_template *at,
+ struct rte_flow_error *error __rte_unused)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ const struct rte_flow_attr *attr = &table_attr->flow_attr;
+ struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *masks = at->masks;
+ bool actions_end = false;
+ uint32_t type;
+
+ if (attr->transfer)
+ type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ type = MLX5DR_TABLE_TYPE_NIC_RX;
+ for (; !actions_end; actions++, masks++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ acts->drop = priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ return 0;
+}
+
+/**
+ * Create flow table.
+ *
+ * The input item and action templates will be binded to the table.
+ * Flow memory will also be allocated. Matcher will be created based
+ * on the item template. Action will be translated to the dedicated
+ * DR action if possible.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] attr
+ * Pointer to the table attributes.
+ * @param[in] item_templates
+ * Item template array to be binded to the table.
+ * @param[in] nb_item_templates
+ * Number of item template.
+ * @param[in] action_templates
+ * Action template array to be binded to the table.
+ * @param[in] nb_action_templates
+ * Number of action template.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_template_table *
+flow_hw_table_create(struct rte_eth_dev *dev,
+ const struct rte_flow_template_table_attr *attr,
+ struct rte_flow_pattern_template *item_templates[],
+ uint8_t nb_item_templates,
+ struct rte_flow_actions_template *action_templates[],
+ uint8_t nb_action_templates,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_matcher_attr matcher_attr = {0};
+ struct rte_flow_template_table *tbl = NULL;
+ struct mlx5_flow_group *grp;
+ struct mlx5dr_match_template *mt[MLX5_HW_TBL_MAX_ITEM_TEMPLATE];
+ struct rte_flow_attr flow_attr = attr->flow_attr;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &flow_attr,
+ };
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow),
+ .trunk_size = 1 << 12,
+ .per_core_cache = 1 << 13,
+ .need_lock = 1,
+ .release_mem_en = !!priv->sh->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_table_flow",
+ };
+ struct mlx5_list_entry *ge;
+ uint32_t i, max_tpl = MLX5_HW_TBL_MAX_ITEM_TEMPLATE;
+ uint32_t nb_flows = rte_align32pow2(attr->nb_flows);
+ int err;
+
+ /* HWS layer accepts only 1 item template with root table. */
+ if (!attr->flow_attr.group)
+ max_tpl = 1;
+ cfg.max_idx = nb_flows;
+ /* For table has very limited flows, disable cache. */
+ if (nb_flows < cfg.trunk_size) {
+ cfg.per_core_cache = 0;
+ cfg.trunk_size = nb_flows;
+ }
+ /* Check if we requires too many templates. */
+ if (nb_item_templates > max_tpl ||
+ nb_action_templates > MLX5_HW_TBL_MAX_ACTION_TEMPLATE) {
+ rte_errno = EINVAL;
+ goto error;
+ }
+ /* Allocate the table memory. */
+ tbl = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*tbl), 0, rte_socket_id());
+ if (!tbl)
+ goto error;
+ /* Allocate flow indexed pool. */
+ tbl->flow = mlx5_ipool_create(&cfg);
+ if (!tbl->flow)
+ goto error;
+ /* Register the flow group. */
+ ge = mlx5_hlist_register(priv->sh->groups, attr->flow_attr.group, &ctx);
+ if (!ge)
+ goto error;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ tbl->grp = grp;
+ /* Prepare matcher information. */
+ matcher_attr.priority = attr->flow_attr.priority;
+ matcher_attr.mode = MLX5DR_MATCHER_RESOURCE_MODE_RULE;
+ matcher_attr.rule.num_log = rte_log2_u32(nb_flows);
+ /* Build the item template. */
+ for (i = 0; i < nb_item_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&item_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto it_error;
+ }
+ mt[i] = item_templates[i]->mt;
+ tbl->its[i] = item_templates[i];
+ }
+ tbl->matcher = mlx5dr_matcher_create
+ (tbl->grp->tbl, mt, nb_item_templates, &matcher_attr);
+ if (!tbl->matcher)
+ goto it_error;
+ tbl->nb_item_templates = nb_item_templates;
+ /* Build the action template. */
+ for (i = 0; i < nb_action_templates; i++) {
+ uint32_t ret;
+
+ ret = __atomic_add_fetch(&action_templates[i]->refcnt, 1,
+ __ATOMIC_RELAXED);
+ if (ret <= 1) {
+ rte_errno = EINVAL;
+ goto at_error;
+ }
+ err = flow_hw_actions_translate(dev, attr,
+ &tbl->ats[i].acts,
+ action_templates[i], error);
+ if (err) {
+ i++;
+ goto at_error;
+ }
+ tbl->ats[i].action_template = action_templates[i];
+ }
+ tbl->nb_action_templates = nb_action_templates;
+ tbl->type = attr->flow_attr.transfer ? MLX5DR_TABLE_TYPE_FDB :
+ (attr->flow_attr.egress ? MLX5DR_TABLE_TYPE_NIC_TX :
+ MLX5DR_TABLE_TYPE_NIC_RX);
+ LIST_INSERT_HEAD(&priv->flow_hw_tbl, tbl, next);
+ return tbl;
+at_error:
+ while (i--) {
+ __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __atomic_sub_fetch(&action_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ i = nb_item_templates;
+it_error:
+ while (i--)
+ __atomic_sub_fetch(&item_templates[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ mlx5dr_matcher_destroy(tbl->matcher);
+error:
+ err = rte_errno;
+ if (tbl) {
+ if (tbl->grp)
+ mlx5_hlist_unregister(priv->sh->groups,
+ &tbl->grp->entry);
+ if (tbl->flow)
+ mlx5_ipool_destroy(tbl->flow);
+ mlx5_free(tbl);
+ }
+ rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
+ return NULL;
+}
+
+/**
+ * Destroy flow table.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table
+ * Pointer to the table to be destroyed.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_table_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_template_table *table,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int i;
+
+ if (table->refcnt) {
+ DRV_LOG(WARNING, "Table %p is still in using.", (void *)table);
+ return rte_flow_error_set(error, EBUSY,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "table in using");
+ }
+ LIST_REMOVE(table, next);
+ for (i = 0; i < table->nb_item_templates; i++)
+ __atomic_sub_fetch(&table->its[i]->refcnt,
+ 1, __ATOMIC_RELAXED);
+ for (i = 0; i < table->nb_action_templates; i++) {
+ __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __atomic_sub_fetch(&table->ats[i].action_template->refcnt,
+ 1, __ATOMIC_RELAXED);
+ }
+ mlx5dr_matcher_destroy(table->matcher);
+ mlx5_hlist_unregister(priv->sh->groups, &table->grp->entry);
+ mlx5_ipool_destroy(table->flow);
+ mlx5_free(table);
+ return 0;
+}
+
/**
* Create flow action template.
*
@@ -228,6 +530,199 @@ flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
return 0;
}
+/**
+ * Create group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] cb_ctx
+ * Pointer to the group creation context.
+ *
+ * @return
+ * Group entry on success, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct rte_eth_dev *dev = ctx->dev;
+ struct rte_flow_attr *attr = (struct rte_flow_attr *)ctx->data;
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_table_attr dr_tbl_attr = {0};
+ struct rte_flow_error *error = ctx->error;
+ struct mlx5_flow_group *grp_data;
+ struct mlx5dr_table *tbl = NULL;
+ struct mlx5dr_action *jump;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_zmalloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ dr_tbl_attr.level = attr->group;
+ if (attr->transfer)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_FDB;
+ else if (attr->egress)
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_TX;
+ else
+ dr_tbl_attr.type = MLX5DR_TABLE_TYPE_NIC_RX;
+ tbl = mlx5dr_table_create(priv->dr_ctx, &dr_tbl_attr);
+ if (!tbl)
+ goto error;
+ grp_data->tbl = tbl;
+ if (attr->group) {
+ /* Jump action be used by non-root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[!!attr->group][dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.hws_action = jump;
+ /* Jump action be used by root table. */
+ jump = mlx5dr_action_create_dest_table
+ (priv->dr_ctx, tbl,
+ mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_ROOT]
+ [dr_tbl_attr.type]);
+ if (!jump)
+ goto error;
+ grp_data->jump.root_action = jump;
+ }
+ grp_data->idx = idx;
+ grp_data->group_id = attr->group;
+ grp_data->type = dr_tbl_attr.type;
+ return &grp_data->entry;
+error:
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (tbl)
+ mlx5dr_table_destroy(tbl);
+ if (idx)
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], idx);
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow dr table");
+ return NULL;
+}
+
+/**
+ * Remove group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the entry to be removed.
+ */
+void
+flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ MLX5_ASSERT(entry && sh);
+ /* To use the wrapper glue functions instead. */
+ if (grp_data->jump.hws_action)
+ mlx5dr_action_destroy(grp_data->jump.hws_action);
+ if (grp_data->jump.root_action)
+ mlx5dr_action_destroy(grp_data->jump.root_action);
+ mlx5dr_table_destroy(grp_data->tbl);
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
+/**
+ * Match group callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+int
+flow_hw_grp_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
+ void *cb_ctx)
+{
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+ struct rte_flow_attr *attr =
+ (struct rte_flow_attr *)ctx->data;
+
+ return (grp_data->group_id != attr->group) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_FDB) &&
+ attr->transfer) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_TX) &&
+ attr->egress) ||
+ ((grp_data->type != MLX5DR_TABLE_TYPE_NIC_RX) &&
+ attr->ingress);
+}
+
+/**
+ * Clone group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be matched.
+ * @param[in] cb_ctx
+ * Pointer to the group matching context.
+ *
+ * @return
+ * 0 on matched, 1 on miss matched.
+ */
+struct mlx5_list_entry *
+flow_hw_grp_clone_cb(void *tool_ctx, struct mlx5_list_entry *oentry,
+ void *cb_ctx)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_cb_ctx *ctx = cb_ctx;
+ struct mlx5_flow_group *grp_data;
+ struct rte_flow_error *error = ctx->error;
+ uint32_t idx = 0;
+
+ grp_data = mlx5_ipool_malloc(sh->ipool[MLX5_IPOOL_HW_GRP], &idx);
+ if (!grp_data) {
+ rte_flow_error_set(error, ENOMEM,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "cannot allocate flow table data entry");
+ return NULL;
+ }
+ memcpy(grp_data, oentry, sizeof(*grp_data));
+ grp_data->idx = idx;
+ return &grp_data->entry;
+}
+
+/**
+ * Free cloned group entry callback.
+ *
+ * @param[in] tool_ctx
+ * Pointer to the hash list related context.
+ * @param[in] entry
+ * Pointer to the group to be freed.
+ */
+void
+flow_hw_grp_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
+{
+ struct mlx5_dev_ctx_shared *sh = tool_ctx;
+ struct mlx5_flow_group *grp_data =
+ container_of(entry, struct mlx5_flow_group, entry);
+
+ mlx5_ipool_free(sh->ipool[MLX5_IPOOL_HW_GRP], grp_data->idx);
+}
+
/**
* Configure port HWS resources.
*
@@ -245,6 +740,7 @@ flow_hw_info_get(struct rte_eth_dev *dev __rte_unused,
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
+
static int
flow_hw_configure(struct rte_eth_dev *dev,
const struct rte_flow_port_attr *port_attr,
@@ -321,8 +817,24 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
priv->dr_ctx = dr_ctx;
priv->nb_queue = nb_queue;
+ /* Add global actions. */
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ priv->hw_drop[i][j] = mlx5dr_action_create_dest_drop
+ (priv->dr_ctx, mlx5_hw_act_flag[i][j]);
+ if (!priv->hw_drop[i][j])
+ goto err;
+ }
+ }
return 0;
err:
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
if (dr_ctx)
claim_zero(mlx5dr_context_close(dr_ctx));
mlx5_free(priv->hw_q);
@@ -342,11 +854,17 @@ void
flow_hw_resource_release(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_template_table *tbl;
struct rte_flow_pattern_template *it;
struct rte_flow_actions_template *at;
+ int i, j;
if (!priv->dr_ctx)
return;
+ while (!LIST_EMPTY(&priv->flow_hw_tbl)) {
+ tbl = LIST_FIRST(&priv->flow_hw_tbl);
+ flow_hw_table_destroy(dev, tbl, NULL);
+ }
while (!LIST_EMPTY(&priv->flow_hw_itt)) {
it = LIST_FIRST(&priv->flow_hw_itt);
flow_hw_pattern_template_destroy(dev, it, NULL);
@@ -355,6 +873,13 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
at = LIST_FIRST(&priv->flow_hw_at);
flow_hw_actions_template_destroy(dev, at, NULL);
}
+ for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
+ for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
+ if (!priv->hw_drop[i][j])
+ continue;
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ }
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
@@ -369,6 +894,8 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.pattern_template_destroy = flow_hw_pattern_template_destroy,
.actions_template_create = flow_hw_actions_template_create,
.actions_template_destroy = flow_hw_actions_template_destroy,
+ .template_table_create = flow_hw_table_create,
+ .template_table_destroy = flow_hw_table_destroy,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 08/14] net/mlx5: add basic flow queue operation
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (6 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 07/14] net/mlx5: add table management Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 09/14] net/mlx5: add flow flush function Suanming Mou
` (6 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The HW steering uses async queue-based flow rules management
mechanism. The matcher and part of the actions have been
prepared during flow table creation. Some remaining actions
will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow management
should be applied to the HW directly. An extra push function
is provided to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function
will be called to get the queued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead
of allocating from HW steering layer. While destroying the
flow, the flow rule memory can only be freed after the CQE
received.
The HW queue job descriptor is currently introduced to convey
the flow information and operation type between the flow
insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:
rte_flow_async_create();
rte_flow_async_destroy();
rte_flow_push();
rte_flow_pull();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
doc/guides/nics/mlx5.rst | 6 +
doc/guides/rel_notes/release_22_03.rst | 1 +
drivers/net/mlx5/mlx5.h | 2 +-
drivers/net/mlx5/mlx5_flow.c | 188 ++++++++++++++++
drivers/net/mlx5/mlx5_flow.h | 41 ++++
drivers/net/mlx5/mlx5_flow_hw.c | 292 ++++++++++++++++++++++++-
6 files changed, 528 insertions(+), 2 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 92127675a3..a5b3298f0c 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -488,6 +488,12 @@ Limitations
from the reference "Clock Queue" completions,
the scheduled send timestamps should not be specified with non-zero MSB.
+ - HW steering:
+
+ - WQE based high scaling and safer flow insertion/destruction.
+ - Set ``dv_flow_en`` to 2 in order to enable HW steering.
+ - Async queue-based ``rte_flow_q`` APIs supported only.
+
Statistics
----------
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 3b4a6814f3..0c39011b24 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -164,6 +164,7 @@ New Features
* Support ConnectX-7 capability to schedule traffic sending on timestamp
* Support steering for external Rx queue created outside the PMD.
+ * Added WQE based hardware steering support with ``rte_flow_async`` API.
* **Updated Wangxun ngbe driver.**
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 4eedc8859f..6f7ec2f067 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -340,7 +340,7 @@ enum {
/* HW steering flow management job descriptor. */
struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
- struct rte_flow *flow; /* Flow attached to the job. */
+ struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
};
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 472a5b975d..1702d60fc0 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -850,6 +850,34 @@ static int
mlx5_flow_table_destroy(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+static struct rte_flow *
+mlx5_flow_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+static int
+mlx5_flow_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
@@ -879,6 +907,10 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.actions_template_destroy = mlx5_flow_actions_template_destroy,
.template_table_create = mlx5_flow_table_create,
.template_table_destroy = mlx5_flow_table_destroy,
+ .async_create = mlx5_flow_async_flow_create,
+ .async_destroy = mlx5_flow_async_flow_destroy,
+ .pull = mlx5_flow_pull,
+ .push = mlx5_flow_push,
};
/* Tunnel information. */
@@ -8180,6 +8212,162 @@ mlx5_flow_table_destroy(struct rte_eth_dev *dev,
return fops->template_table_destroy(dev, table, error);
}
+/**
+ * Enqueue flow creation.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue_id
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+mlx5_flow_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue_id,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW) {
+ rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q create with incorrect steering mode");
+ return NULL;
+ }
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->async_flow_create(dev, queue_id, attr, table,
+ items, pattern_template_index,
+ actions, action_template_index,
+ user_data, error);
+}
+
+/**
+ * Enqueue flow destruction.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q destroy with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->async_flow_destroy(dev, queue, attr, flow,
+ user_data, error);
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q pull with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->pull(dev, queue, res, n_res, error);
+}
+
+/**
+ * Push the enqueued flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flows.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops;
+
+ if (flow_get_drv_type(dev, NULL) != MLX5_FLOW_TYPE_HW)
+ return rte_flow_error_set(error, ENOTSUP,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+ NULL,
+ "flow_q push with incorrect steering mode");
+ fops = flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+ return fops->push(dev, queue, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 1579036f58..3add4c4a81 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1017,6 +1017,13 @@ struct rte_flow {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/* HWS flow struct. */
+struct rte_flow_hw {
+ uint32_t idx; /* Flow index from indexed pool. */
+ struct rte_flow_template_table *table; /* The table flow allcated from. */
+ struct mlx5dr_rule rule; /* HWS layer data struct. */
+} __rte_packed;
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1371,6 +1378,34 @@ typedef int (*mlx5_flow_table_destroy_t)
(struct rte_eth_dev *dev,
struct rte_flow_template_table *table,
struct rte_flow_error *error);
+typedef struct rte_flow *(*mlx5_flow_async_flow_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_async_flow_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_pull_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error);
+typedef int (*mlx5_flow_push_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error);
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
@@ -1417,6 +1452,10 @@ struct mlx5_flow_driver_ops {
mlx5_flow_actions_template_destroy_t actions_template_destroy;
mlx5_flow_table_create_t template_table_create;
mlx5_flow_table_destroy_t template_table_destroy;
+ mlx5_flow_async_flow_create_t async_flow_create;
+ mlx5_flow_async_flow_destroy_t async_flow_destroy;
+ mlx5_flow_pull_t pull;
+ mlx5_flow_push_t push;
};
/* mlx5_flow.c */
@@ -1587,6 +1626,8 @@ mlx5_translate_tunnel_etypes(uint64_t pattern_flags)
return 0;
}
+int flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error);
int mlx5_flow_group_to_table(struct rte_eth_dev *dev,
const struct mlx5_flow_tunnel *tunnel,
uint32_t group, uint32_t *table,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index c1e3b56f23..f0cb530524 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -10,6 +10,9 @@
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/* DR action flags with different table. */
@@ -105,6 +108,289 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Construct flow action array.
+ *
+ * For action template contains dynamic actions, these actions need to
+ * be updated according to the rte_flow action during flow creation.
+ *
+ * @param[in] hw_acts
+ * Pointer to translated actions from template.
+ * @param[in] actions
+ * Array of rte_flow action need to be checked.
+ * @param[in] rule_acts
+ * Array of DR rule actions to be used during flow creation..
+ * @param[in] acts_num
+ * Pointer to the real acts_num flow has.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+ const struct rte_flow_action actions[],
+ struct mlx5dr_rule_action *rule_acts,
+ uint32_t *acts_num)
+{
+ bool actions_end = false;
+ uint32_t i;
+
+ for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
+ switch (actions->type) {
+ case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ break;
+ case RTE_FLOW_ACTION_TYPE_VOID:
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ rule_acts[i++].action = hw_acts->drop;
+ break;
+ case RTE_FLOW_ACTION_TYPE_END:
+ actions_end = true;
+ break;
+ default:
+ break;
+ }
+ }
+ *acts_num = i;
+ return 0;
+}
+
+/**
+ * Enqueue HW steering flow creation.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow creation status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to create the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] items
+ * Items with flow spec value.
+ * @param[in] pattern_template_index
+ * The item pattern flow follows from the table.
+ * @param[in] actions
+ * Action with flow spec value.
+ * @param[in] action_template_index
+ * The action pattern flow follows from the table.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Flow pointer on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow *
+flow_hw_async_flow_create(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_template_table *table,
+ const struct rte_flow_item items[],
+ uint8_t pattern_template_index,
+ const struct rte_flow_action actions[],
+ uint8_t action_template_index,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = user_data,
+ .burst = attr->postpone,
+ };
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
+ struct mlx5_hw_actions *hw_acts;
+ struct rte_flow_hw *flow;
+ struct mlx5_hw_q_job *job;
+ uint32_t acts_num, flow_idx;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ flow = mlx5_ipool_zmalloc(table->flow, &flow_idx);
+ if (!flow)
+ goto error;
+ /*
+ * Set the table here in order to know the destination table
+ * when free the flow afterwards.
+ */
+ flow->table = table;
+ flow->idx = flow_idx;
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ /*
+ * Set the job type here in order to know if the flow memory
+ * should be freed or not when get the result from dequeue.
+ */
+ job->type = MLX5_HW_Q_JOB_TYPE_CREATE;
+ job->flow = flow;
+ job->user_data = user_data;
+ rule_attr.user_data = job;
+ hw_acts = &table->ats[action_template_index].acts;
+ /* Construct the flow action array based on the input actions.*/
+ flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ ret = mlx5dr_rule_create(table->matcher,
+ pattern_template_index, items,
+ rule_acts, acts_num,
+ &rule_attr, &flow->rule);
+ if (likely(!ret))
+ return (struct rte_flow *)flow;
+ /* Flow created fail, return the descriptor and flow memory. */
+ mlx5_ipool_free(table->flow, flow_idx);
+ priv->hw_q[queue].job_idx++;
+error:
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+ return NULL;
+}
+
+/**
+ * Enqueue HW steering flow destruction.
+ *
+ * The flow will be applied to the HW only if the postpone bit is not set or
+ * the extra push function is called.
+ * The flow destruction status should be checked from dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to destroy the flow.
+ * @param[in] attr
+ * Pointer to the flow operation attributes.
+ * @param[in] flow
+ * Pointer to the flow to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_async_flow_destroy(struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow *flow,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5dr_rule_attr rule_attr = {
+ .queue_id = queue,
+ .user_data = user_data,
+ .burst = attr->postpone,
+ };
+ struct rte_flow_hw *fh = (struct rte_flow_hw *)flow;
+ struct mlx5_hw_q_job *job;
+ int ret;
+
+ if (unlikely(!priv->hw_q[queue].job_idx)) {
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ job = priv->hw_q[queue].job[--priv->hw_q[queue].job_idx];
+ job->type = MLX5_HW_Q_JOB_TYPE_DESTROY;
+ job->user_data = user_data;
+ job->flow = fh;
+ rule_attr.user_data = job;
+ ret = mlx5dr_rule_destroy(&fh->rule, &rule_attr);
+ if (likely(!ret))
+ return 0;
+ priv->hw_q[queue].job_idx++;
+error:
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte flow");
+}
+
+/**
+ * Pull the enqueued flows.
+ *
+ * For flows enqueued from creation/destruction, the status should be
+ * checked from the dequeue result.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the result.
+ * @param[in/out] res
+ * Array to save the results.
+ * @param[in] n_res
+ * Available result with the array.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Result number on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_pull(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_op_result res[],
+ uint16_t n_res,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q_job *job;
+ int ret, i;
+
+ ret = mlx5dr_send_queue_poll(priv->dr_ctx, queue, res, n_res);
+ if (ret < 0)
+ return rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to query flow queue");
+ for (i = 0; i < ret; i++) {
+ job = (struct mlx5_hw_q_job *)res[i].user_data;
+ /* Restore user data. */
+ res[i].user_data = job->user_data;
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
+ }
+ return ret;
+}
+
+/**
+ * Push the enqueued flows to HW.
+ *
+ * Force apply all the enqueued flows to the HW.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to push the flow.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_push(struct rte_eth_dev *dev,
+ uint32_t queue,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ int ret;
+
+ ret = mlx5dr_send_queue_action(priv->dr_ctx, queue,
+ MLX5DR_SEND_QUEUE_ACTION_DRAIN);
+ if (ret) {
+ rte_flow_error_set(error, rte_errno,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to push flows");
+ return ret;
+ }
+ return 0;
+}
+
/**
* Create flow table.
*
@@ -152,7 +438,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
.data = &flow_attr,
};
struct mlx5_indexed_pool_config cfg = {
- .size = sizeof(struct rte_flow),
+ .size = sizeof(struct rte_flow_hw),
.trunk_size = 1 << 12,
.per_core_cache = 1 << 13,
.need_lock = 1,
@@ -896,6 +1182,10 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.actions_template_destroy = flow_hw_actions_template_destroy,
.template_table_create = flow_hw_table_create,
.template_table_destroy = flow_hw_table_destroy,
+ .async_flow_create = flow_hw_async_flow_create,
+ .async_flow_destroy = flow_hw_async_flow_destroy,
+ .pull = flow_hw_pull,
+ .push = flow_hw_push,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 09/14] net/mlx5: add flow flush function
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (7 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 08/14] net/mlx5: add basic flow queue operation Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 10/14] net/mlx5: add flow jump action Suanming Mou
` (5 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
In case port is being stopped, all created flows should be flushed.
This commit adds the flow flush helper function.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 8 ++
drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++
2 files changed, 137 insertions(+)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 1702d60fc0..5af71585fc 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7024,6 +7024,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
uint32_t num_flushed = 0, fidx = 1;
struct rte_flow *flow;
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ if (priv->sh->config.dv_flow_en == 2 &&
+ type == MLX5_FLOW_TYPE_GEN) {
+ flow_hw_q_flow_flush(dev, NULL);
+ return;
+ }
+#endif
+
MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
flow_list_destroy(dev, type, fidx);
num_flushed++;
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index f0cb530524..74f8ee1d6a 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -13,6 +13,12 @@
/* The maximum actions support in the flow. */
#define MLX5_HW_MAX_ACTS 16
+/* Default push burst threshold. */
+#define BURST_THR 32u
+
+/* Default queue to flush the flows. */
+#define MLX5_DEFAULT_FLUSH_QUEUE 0
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
/* DR action flags with different table. */
@@ -391,6 +397,129 @@ flow_hw_push(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Drain the enqueued flows' completion.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * The queue to pull the flow.
+ * @param[in] pending_rules
+ * The pending flow number.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+__flow_hw_pull_comp(struct rte_eth_dev *dev,
+ uint32_t queue,
+ uint32_t pending_rules,
+ struct rte_flow_error *error)
+{
+ struct rte_flow_op_result comp[BURST_THR];
+ int ret, i, empty_loop = 0;
+
+ flow_hw_push(dev, queue, error);
+ while (pending_rules) {
+ ret = flow_hw_pull(dev, queue, comp, BURST_THR, error);
+ if (ret < 0)
+ return -1;
+ if (!ret) {
+ rte_delay_us_sleep(20000);
+ if (++empty_loop > 5) {
+ DRV_LOG(WARNING, "No available dequeue, quit.");
+ break;
+ }
+ continue;
+ }
+ for (i = 0; i < ret; i++) {
+ if (comp[i].status == RTE_FLOW_OP_ERROR)
+ DRV_LOG(WARNING, "Flow flush get error CQE.");
+ }
+ if ((uint32_t)ret > pending_rules) {
+ DRV_LOG(WARNING, "Flow flush get extra CQE.");
+ return rte_flow_error_set(error, ERANGE,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "get extra CQE");
+ }
+ pending_rules -= ret;
+ empty_loop = 0;
+ }
+ return 0;
+}
+
+/**
+ * Flush created flows.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+int
+flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hw_q *hw_q;
+ struct rte_flow_template_table *tbl;
+ struct rte_flow_hw *flow;
+ struct rte_flow_op_attr attr = {
+ .postpone = 0,
+ };
+ uint32_t pending_rules = 0;
+ uint32_t queue;
+ uint32_t fidx;
+
+ /*
+ * Ensure to push and dequeue all the enqueued flow
+ * creation/destruction jobs in case user forgot to
+ * dequeue. Or the enqueued created flows will be
+ * leaked. The forgotten dequeues would also cause
+ * flow flush get extra CQEs as expected and pending_rules
+ * be minus value.
+ */
+ for (queue = 0; queue < priv->nb_queue; queue++) {
+ hw_q = &priv->hw_q[queue];
+ if (__flow_hw_pull_comp(dev, queue, hw_q->size - hw_q->job_idx,
+ error))
+ return -1;
+ }
+ /* Flush flow per-table from MLX5_DEFAULT_FLUSH_QUEUE. */
+ hw_q = &priv->hw_q[MLX5_DEFAULT_FLUSH_QUEUE];
+ LIST_FOREACH(tbl, &priv->flow_hw_tbl, next) {
+ MLX5_IPOOL_FOREACH(tbl->flow, fidx, flow) {
+ if (flow_hw_async_flow_destroy(dev,
+ MLX5_DEFAULT_FLUSH_QUEUE,
+ &attr,
+ (struct rte_flow *)flow,
+ NULL,
+ error))
+ return -1;
+ pending_rules++;
+ /* Drain completion with queue size. */
+ if (pending_rules >= hw_q->size) {
+ if (__flow_hw_pull_comp(dev,
+ MLX5_DEFAULT_FLUSH_QUEUE,
+ pending_rules, error))
+ return -1;
+ pending_rules = 0;
+ }
+ }
+ }
+ /* Drain left completion. */
+ if (pending_rules &&
+ __flow_hw_pull_comp(dev, MLX5_DEFAULT_FLUSH_QUEUE, pending_rules,
+ error))
+ return -1;
+ return 0;
+}
+
/**
* Create flow table.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 10/14] net/mlx5: add flow jump action
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (8 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 09/14] net/mlx5: add flow flush function Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 11/14] net/mlx5: add queue and RSS action Suanming Mou
` (4 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
Jump action connects different level of flow tables and allows packet
handling in the chain of flows.
A new action construct data struct is also added in this commit to help
to handle not only the dynamic jump action but also for the other generic
dynamic actions. The actions with empty mask configuration means dynamic
action, and the dedicated action will be created with the flow action
configuration during flow creation. In that dynamic action case, the action
will be appended to the table template's action list during table creation.
When creating the flows, traverse the action list and pick the dynamic
action configuration details from flow actions as the action construct data
struct describes, then create the dedicated dynamic actions.
This commit adds the jump action and the generic dynamic action construct
mechanism.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 25 ++-
drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
3 files changed, 275 insertions(+), 21 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 6f7ec2f067..3d4e9cf8ee 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1528,6 +1528,7 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 3add4c4a81..963dbd7806 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1020,10 +1020,25 @@ struct rte_flow {
/* HWS flow struct. */
struct rte_flow_hw {
uint32_t idx; /* Flow index from indexed pool. */
+ uint32_t fate_type; /* Fate action type. */
+ union {
+ /* Jump action. */
+ struct mlx5_hw_jump_action *jump;
+ };
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
} __rte_packed;
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+ LIST_ENTRY(mlx5_action_construct_data) next;
+ /* Ensure the action types are matched. */
+ int type;
+ uint32_t idx; /* Data index. */
+ uint16_t action_src; /* rte_flow_action src offset. */
+ uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
/* Flow item template struct. */
struct rte_flow_pattern_template {
LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
/* DR action set struct. */
struct mlx5_hw_actions {
- struct mlx5dr_action *drop; /* Drop action. */
+ /* Dynamic action list. */
+ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+ struct mlx5_hw_jump_action *jump; /* Jump action. */
+ uint32_t acts_num:4; /* Total action number. */
+ /* Translated DR action array from action template. */
+ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
/* mlx5 action template struct. */
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 74f8ee1d6a..d9471eaf48 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -36,18 +36,158 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Register destination table DR jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] table_attr
+ * Pointer to the flow attributes.
+ * @param[in] dest_group
+ * The destination group ID.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static struct mlx5_hw_jump_action *
+flow_hw_jump_action_register(struct rte_eth_dev *dev,
+ const struct rte_flow_attr *attr,
+ uint32_t dest_group,
+ struct rte_flow_error *error)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct rte_flow_attr jattr = *attr;
+ struct mlx5_flow_group *grp;
+ struct mlx5_flow_cb_ctx ctx = {
+ .dev = dev,
+ .error = error,
+ .data = &jattr,
+ };
+ struct mlx5_list_entry *ge;
+
+ jattr.group = dest_group;
+ ge = mlx5_hlist_register(priv->sh->flow_tbls, dest_group, &ctx);
+ if (!ge)
+ return NULL;
+ grp = container_of(ge, struct mlx5_flow_group, entry);
+ return &grp->jump;
+}
+
+/**
+ * Release jump action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] jump
+ * Pointer to the jump action.
+ */
+
+static void
+flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+}
+
/**
* Destroy DR actions created by action template.
*
* For DR actions created during table creation's action translate.
* Need to destroy the DR action when destroying the table.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
* @param[in] acts
* Pointer to the template HW steering DR actions.
*/
static void
-__flow_hw_action_template_destroy(struct mlx5_hw_actions *acts __rte_unused)
+__flow_hw_action_template_destroy(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ if (acts->jump) {
+ struct mlx5_flow_group *grp;
+
+ grp = container_of
+ (acts->jump, struct mlx5_flow_group, jump);
+ mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
+ acts->jump = NULL;
+ }
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline struct mlx5_action_construct_data *
+__flow_hw_act_data_alloc(struct mlx5_priv *priv,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
{
+ struct mlx5_action_construct_data *act_data;
+ uint32_t idx = 0;
+
+ act_data = mlx5_ipool_zmalloc(priv->acts_ipool, &idx);
+ if (!act_data)
+ return NULL;
+ act_data->idx = idx;
+ act_data->type = type;
+ act_data->action_src = action_src;
+ act_data->action_dst = action_dst;
+ return act_data;
+}
+
+/**
+ * Append dynamic action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_general_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
}
/**
@@ -80,14 +220,16 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
const struct rte_flow_template_table_attr *table_attr,
struct mlx5_hw_actions *acts,
struct rte_flow_actions_template *at,
- struct rte_flow_error *error __rte_unused)
+ struct rte_flow_error *error)
{
struct mlx5_priv *priv = dev->data->dev_private;
const struct rte_flow_attr *attr = &table_attr->flow_attr;
struct rte_flow_action *actions = at->actions;
+ struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
bool actions_end = false;
- uint32_t type;
+ uint32_t type, i;
+ int err;
if (attr->transfer)
type = MLX5DR_TABLE_TYPE_FDB;
@@ -95,14 +237,34 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
type = MLX5DR_TABLE_TYPE_NIC_TX;
else
type = MLX5DR_TABLE_TYPE_NIC_RX;
- for (; !actions_end; actions++, masks++) {
+ for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
case RTE_FLOW_ACTION_TYPE_DROP:
- acts->drop = priv->hw_drop[!!attr->group][type];
+ acts->rule_acts[i++].action =
+ priv->hw_drop[!!attr->group][type];
+ break;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ if (masks->conf) {
+ uint32_t jump_group =
+ ((const struct rte_flow_action_jump *)
+ actions->conf)->group;
+ acts->jump = flow_hw_jump_action_register
+ (dev, attr, jump_group, error);
+ if (!acts->jump)
+ goto err;
+ acts->rule_acts[i].action = (!!attr->group) ?
+ acts->jump->hws_action :
+ acts->jump->root_action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
@@ -111,7 +273,14 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ acts->acts_num = i;
return 0;
+err:
+ err = rte_errno;
+ __flow_hw_action_template_destroy(dev, acts);
+ return rte_flow_error_set(error, err,
+ RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
+ "fail to create rte table");
}
/**
@@ -120,6 +289,10 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* For action template contains dynamic actions, these actions need to
* be updated according to the rte_flow action during flow creation.
*
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] job
+ * Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
* @param[in] actions
@@ -133,31 +306,63 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* 0 on success, negative value otherwise and rte_errno is set.
*/
static __rte_always_inline int
-flow_hw_actions_construct(struct mlx5_hw_actions *hw_acts,
+flow_hw_actions_construct(struct rte_eth_dev *dev,
+ struct mlx5_hw_q_job *job,
+ struct mlx5_hw_actions *hw_acts,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
{
- bool actions_end = false;
- uint32_t i;
+ struct rte_flow_template_table *table = job->flow->table;
+ struct mlx5_action_construct_data *act_data;
+ const struct rte_flow_action *action;
+ struct rte_flow_attr attr = {
+ .ingress = 1,
+ };
- for (i = 0; !actions_end || (i >= MLX5_HW_MAX_ACTS); actions++) {
- switch (actions->type) {
+ memcpy(rule_acts, hw_acts->rule_acts,
+ sizeof(*rule_acts) * hw_acts->acts_num);
+ *acts_num = hw_acts->acts_num;
+ if (LIST_EMPTY(&hw_acts->act_list))
+ return 0;
+ attr.group = table->grp->group_id;
+ if (table->type == MLX5DR_TABLE_TYPE_FDB) {
+ attr.transfer = 1;
+ attr.ingress = 1;
+ } else if (table->type == MLX5DR_TABLE_TYPE_NIC_TX) {
+ attr.egress = 1;
+ attr.ingress = 0;
+ } else {
+ attr.ingress = 1;
+ }
+ LIST_FOREACH(act_data, &hw_acts->act_list, next) {
+ uint32_t jump_group;
+ struct mlx5_hw_jump_action *jump;
+
+ action = &actions[act_data->action_src];
+ MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
+ (int)action->type == act_data->type);
+ switch (action->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
- case RTE_FLOW_ACTION_TYPE_DROP:
- rule_acts[i++].action = hw_acts->drop;
- break;
- case RTE_FLOW_ACTION_TYPE_END:
- actions_end = true;
+ case RTE_FLOW_ACTION_TYPE_JUMP:
+ jump_group = ((const struct rte_flow_action_jump *)
+ action->conf)->group;
+ jump = flow_hw_jump_action_register
+ (dev, &attr, jump_group, NULL);
+ if (!jump)
+ return -1;
+ rule_acts[act_data->action_dst].action =
+ (!!attr.group) ? jump->hws_action : jump->root_action;
+ job->flow->jump = jump;
+ job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
default:
break;
}
}
- *acts_num = i;
return 0;
}
@@ -239,7 +444,8 @@ flow_hw_async_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(hw_acts, actions, rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, actions,
+ rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -356,8 +562,11 @@ flow_hw_pull(struct rte_eth_dev *dev,
job = (struct mlx5_hw_q_job *)res[i].user_data;
/* Restore user data. */
res[i].user_data = job->user_data;
- if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY)
+ if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
+ if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
+ flow_hw_jump_release(dev, job->flow->jump);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
+ }
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
}
return ret;
@@ -642,6 +851,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
rte_errno = EINVAL;
goto at_error;
}
+ LIST_INIT(&tbl->ats[i].acts.act_list);
err = flow_hw_actions_translate(dev, attr,
&tbl->ats[i].acts,
action_templates[i], error);
@@ -659,7 +869,7 @@ flow_hw_table_create(struct rte_eth_dev *dev,
return tbl;
at_error:
while (i--) {
- __flow_hw_action_template_destroy(&tbl->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &tbl->ats[i].acts);
__atomic_sub_fetch(&action_templates[i]->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -718,7 +928,7 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
- __flow_hw_action_template_destroy(&table->ats[i].acts);
+ __flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
}
@@ -1169,6 +1379,15 @@ flow_hw_configure(struct rte_eth_dev *dev,
struct mlx5_hw_q *hw_q;
struct mlx5_hw_q_job *job = NULL;
uint32_t mem_size, i, j;
+ struct mlx5_indexed_pool_config cfg = {
+ .size = sizeof(struct rte_flow_hw),
+ .trunk_size = 4096,
+ .need_lock = 1,
+ .release_mem_en = !!priv->sh->config.reclaim_mode,
+ .malloc = mlx5_malloc,
+ .free = mlx5_free,
+ .type = "mlx5_hw_action_construct_data",
+ };
if (!port_attr || !nb_queue || !queue_attr) {
rte_errno = EINVAL;
@@ -1187,6 +1406,9 @@ flow_hw_configure(struct rte_eth_dev *dev,
}
flow_hw_resource_release(dev);
}
+ priv->acts_ipool = mlx5_ipool_create(&cfg);
+ if (!priv->acts_ipool)
+ goto err;
/* Allocate the queue job descriptor LIFO. */
mem_size = sizeof(priv->hw_q[0]) * nb_queue;
for (i = 0; i < nb_queue; i++) {
@@ -1254,6 +1476,10 @@ flow_hw_configure(struct rte_eth_dev *dev,
claim_zero(mlx5dr_context_close(dr_ctx));
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"fail to configure port");
@@ -1295,6 +1521,10 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
}
+ if (priv->acts_ipool) {
+ mlx5_ipool_destroy(priv->acts_ipool);
+ priv->acts_ipool = NULL;
+ }
mlx5_free(priv->hw_q);
priv->hw_q = NULL;
claim_zero(mlx5dr_context_close(priv->dr_ctx));
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 11/14] net/mlx5: add queue and RSS action
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (9 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 10/14] net/mlx5: add flow jump action Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 12/14] net/mlx5: add mark action Suanming Mou
` (3 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
This commit adds the queue and RSS action. Similar to the jump action,
dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed
during port restart, the actions are created with standalone indirect
table as indirect action does. When port stops, detaches the indirect
table from action, when port starts, attaches the indirect table back
to the action.
One more change is made to accelerate the action creation. Currently
the mlx5_hrxq_get() function returns the object index instead of object
pointer. This introduced an extra converting the index to the object by
calling mlx5_ipool_get() in most of the case. And that extra converting
hurts multi-thread performance since mlx5_ipool_get() uses the global
lock inside. As the hash Rx queue object itself also contains the index,
returns the object directly will achieve better performance without the
global lock.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 18 ++--
drivers/net/mlx5/mlx5.h | 4 +
drivers/net/mlx5/mlx5_devx.c | 10 ++
drivers/net/mlx5/mlx5_flow.c | 38 +++----
drivers/net/mlx5/mlx5_flow.h | 7 ++
drivers/net/mlx5/mlx5_flow_dv.c | 161 ++++++++++++++---------------
drivers/net/mlx5/mlx5_flow_hw.c | 101 ++++++++++++++++++
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_rx.h | 9 +-
drivers/net/mlx5/mlx5_rxq.c | 85 +++++++++------
10 files changed, 283 insertions(+), 157 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 5b91e057b2..c9e8017ff7 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1537,6 +1537,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
if (!priv->drop_queue.hrxq)
goto error;
+ priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+ mlx5_hrxq_create_cb,
+ mlx5_hrxq_match_cb,
+ mlx5_hrxq_remove_cb,
+ mlx5_hrxq_clone_cb,
+ mlx5_hrxq_clone_free_cb);
+ if (!priv->hrxqs)
+ goto error;
+ rte_rwlock_init(&priv->ind_tbls_lock);
if (priv->sh->config.dv_flow_en == 2)
return eth_dev;
/* Port representor shares the same max priority with pf port. */
@@ -1561,15 +1570,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = ENOTSUP;
goto error;
}
- priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
- mlx5_hrxq_create_cb,
- mlx5_hrxq_match_cb,
- mlx5_hrxq_remove_cb,
- mlx5_hrxq_clone_cb,
- mlx5_hrxq_clone_free_cb);
- if (!priv->hrxqs)
- goto error;
- rte_rwlock_init(&priv->ind_tbls_lock);
/* Query availability of metadata reg_c's. */
if (!priv->sh->metadata_regc_check_flag) {
err = mlx5_flow_discover_mreg_c(eth_dev);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 3d4e9cf8ee..fe422f7fd1 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1287,6 +1287,7 @@ struct mlx5_flow_rss_desc {
uint64_t hash_fields; /* Verbs Hash fields. */
uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
uint32_t key_len; /**< RSS hash key len. */
+ uint32_t hws_flags; /**< HW steering action. */
uint32_t tunnel; /**< Queue in tunnel. */
uint32_t shared_rss; /**< Shared RSS index. */
struct mlx5_ind_table_obj *ind_tbl;
@@ -1348,6 +1349,7 @@ struct mlx5_hrxq {
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
void *action; /* DV QP action pointer. */
#endif
+ uint32_t hws_flags; /* Hw steering flags. */
uint64_t hash_fields; /* Verbs Hash fields. */
uint32_t rss_key_len; /* Hash key length in bytes. */
uint32_t idx; /* Hash Rx queue index. */
@@ -1479,6 +1481,8 @@ struct mlx5_priv {
LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
/* Indirection tables. */
LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+ /* Standalone indirect tables. */
+ LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
/* Pointer to next element. */
rte_rwlock_t ind_tbls_lock;
uint32_t refcnt; /**< Reference counter. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 8aa68d9658..af106bda50 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -815,6 +815,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
goto error;
}
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+ if (hrxq->hws_flags) {
+ hrxq->action = mlx5dr_action_create_dest_tir
+ (priv->dr_ctx,
+ (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+ if (!hrxq->action)
+ goto error;
+ return 0;
+ }
if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
&hrxq->action)) {
rte_errno = errno;
@@ -1050,6 +1058,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
DRV_LOG(ERR, "Cannot create drop RX queue");
return ret;
}
+ if (priv->sh->config.dv_flow_en == 2)
+ return 0;
/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
if (ret != 0) {
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 5af71585fc..7c3e4ec47a 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -9448,14 +9448,10 @@ int
mlx5_action_handle_attach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
const char *message;
uint32_t queue_idx;
@@ -9471,9 +9467,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
}
if (ret != 0)
return ret;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not attach "
@@ -9482,13 +9476,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
goto error;
}
}
+
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not detach "
@@ -9511,15 +9504,10 @@ int
mlx5_action_handle_detach(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_indexed_pool *ipool =
- priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
- struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
int ret = 0;
- uint32_t idx;
-
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+ struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
if (ret != 0) {
DRV_LOG(ERR, "Port %u could not detach "
@@ -9530,11 +9518,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
}
return 0;
error:
- shared_rss_last = shared_rss;
- ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
- struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
- if (shared_rss == shared_rss_last)
+ ind_tbl_last = ind_tbl;
+ LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+ if (ind_tbl == ind_tbl_last)
break;
if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
DRV_LOG(CRIT, "Port %u could not attach "
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 963dbd7806..70e6cf633f 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {
union {
/* Jump action. */
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq; /* TIR action. */
};
struct rte_flow_template_table *table; /* The table flow allcated from. */
struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {
/* Dynamic action list. */
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
+ struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1910,6 +1912,11 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx,
struct mlx5_list_entry *entry, void *cb_ctx);
void flow_dv_dest_array_clone_free_cb(void *tool_ctx,
struct mlx5_list_entry *entry);
+void flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index c4cd5c894b..2ffdfad61f 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -10971,78 +10971,83 @@ flow_dv_translate_item_tx_queue(struct rte_eth_dev *dev,
/**
* Set the hash fields according to the @p flow information.
*
- * @param[in] dev_flow
- * Pointer to the mlx5_flow.
+ * @param[in] item_flags
+ * The match pattern item flags.
* @param[in] rss_desc
* Pointer to the mlx5_flow_rss_desc.
+ * @param[out] hash_fields
+ * Pointer to the RSS hash fields.
*/
-static void
-flow_dv_hashfields_set(struct mlx5_flow *dev_flow,
- struct mlx5_flow_rss_desc *rss_desc)
+void
+flow_dv_hashfields_set(uint64_t item_flags,
+ struct mlx5_flow_rss_desc *rss_desc,
+ uint64_t *hash_fields)
{
- uint64_t items = dev_flow->handle->layers;
+ uint64_t items = item_flags;
+ uint64_t fields = 0;
int rss_inner = 0;
uint64_t rss_types = rte_eth_rss_hf_refine(rss_desc->types);
- dev_flow->hash_fields = 0;
+ *hash_fields = 0;
#ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT
if (rss_desc->level >= 2)
rss_inner = 1;
#endif
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV4)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV4)) ||
+ !items) {
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV4;
+ fields |= IBV_RX_HASH_SRC_IPV4;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV4;
+ fields |= IBV_RX_HASH_DST_IPV4;
else
- dev_flow->hash_fields |= MLX5_IPV4_IBV_RX_HASH;
+ fields |= MLX5_IPV4_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L3_IPV6)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L3_IPV6)) ||
+ !items) {
if (rss_types & MLX5_IPV6_LAYER_TYPES) {
if (rss_types & RTE_ETH_RSS_L3_SRC_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_SRC_IPV6;
+ fields |= IBV_RX_HASH_SRC_IPV6;
else if (rss_types & RTE_ETH_RSS_L3_DST_ONLY)
- dev_flow->hash_fields |= IBV_RX_HASH_DST_IPV6;
+ fields |= IBV_RX_HASH_DST_IPV6;
else
- dev_flow->hash_fields |= MLX5_IPV6_IBV_RX_HASH;
+ fields |= MLX5_IPV6_IBV_RX_HASH;
}
}
- if (dev_flow->hash_fields == 0)
+ if (fields == 0)
/*
* There is no match between the RSS types and the
* L3 protocol (IPv4/IPv6) defined in the flow rule.
*/
return;
if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_UDP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_UDP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_UDP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_UDP;
+ fields |= IBV_RX_HASH_SRC_PORT_UDP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_UDP;
+ fields |= IBV_RX_HASH_DST_PORT_UDP;
else
- dev_flow->hash_fields |= MLX5_UDP_IBV_RX_HASH;
+ fields |= MLX5_UDP_IBV_RX_HASH;
}
} else if ((rss_inner && (items & MLX5_FLOW_LAYER_INNER_L4_TCP)) ||
- (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP))) {
+ (!rss_inner && (items & MLX5_FLOW_LAYER_OUTER_L4_TCP)) ||
+ !items) {
if (rss_types & RTE_ETH_RSS_TCP) {
if (rss_types & RTE_ETH_RSS_L4_SRC_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_SRC_PORT_TCP;
+ fields |= IBV_RX_HASH_SRC_PORT_TCP;
else if (rss_types & RTE_ETH_RSS_L4_DST_ONLY)
- dev_flow->hash_fields |=
- IBV_RX_HASH_DST_PORT_TCP;
+ fields |= IBV_RX_HASH_DST_PORT_TCP;
else
- dev_flow->hash_fields |= MLX5_TCP_IBV_RX_HASH;
+ fields |= MLX5_TCP_IBV_RX_HASH;
}
}
if (rss_inner)
- dev_flow->hash_fields |= IBV_RX_HASH_INNER;
+ fields |= IBV_RX_HASH_INNER;
+ *hash_fields = fields;
}
/**
@@ -11066,7 +11071,6 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc,
uint32_t *hrxq_idx)
{
- struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_handle *dh = dev_flow->handle;
struct mlx5_hrxq *hrxq;
@@ -11077,11 +11081,8 @@ flow_dv_hrxq_prepare(struct rte_eth_dev *dev,
rss_desc->shared_rss = 0;
if (rss_desc->hash_fields == 0)
rss_desc->queue_num = 1;
- *hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- if (!*hrxq_idx)
- return NULL;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- *hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
+ *hrxq_idx = hrxq ? hrxq->idx : 0;
return hrxq;
}
@@ -11627,7 +11628,9 @@ flow_dv_translate_action_sample(struct rte_eth_dev *dev,
* rss->level and rss.types should be set in advance
* when expanding items for RSS.
*/
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
hrxq = flow_dv_hrxq_prepare(dev, dev_flow,
rss_desc, &hrxq_idx);
if (!hrxq)
@@ -13651,7 +13654,9 @@ flow_dv_translate(struct rte_eth_dev *dev,
*/
handle->layers |= item_flags;
if (action_flags & MLX5_FLOW_ACTION_RSS)
- flow_dv_hashfields_set(dev_flow, rss_desc);
+ flow_dv_hashfields_set(dev_flow->handle->layers,
+ rss_desc,
+ &dev_flow->hash_fields);
/* If has RSS action in the sample action, the Sample/Mirror resource
* should be registered after the hash filed be update.
*/
@@ -14600,20 +14605,18 @@ __flow_dv_action_rss_hrxqs_release(struct rte_eth_dev *dev,
* MLX5_RSS_HASH_IPV4_DST_ONLY are mutually exclusive so they can share
* same slot in mlx5_rss_hash_fields.
*
- * @param[in] rss
- * Pointer to the shared action RSS conf.
+ * @param[in] rss_types
+ * RSS type.
* @param[in, out] hash_field
* hash_field variable needed to be adjusted.
*
* @return
* void
*/
-static void
-__flow_dv_action_rss_l34_hash_adjust(struct mlx5_shared_action_rss *rss,
- uint64_t *hash_field)
+void
+flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+ uint64_t *hash_field)
{
- uint64_t rss_types = rss->origin.types;
-
switch (*hash_field & ~IBV_RX_HASH_INNER) {
case MLX5_RSS_HASH_IPV4:
if (rss_types & MLX5_IPV4_LAYER_TYPES) {
@@ -14696,12 +14699,15 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
size_t i;
int err;
- if (mlx5_ind_table_obj_setup(dev, shared_rss->ind_tbl,
- !!dev->data->dev_started)) {
+ shared_rss->ind_tbl = mlx5_ind_table_obj_new
+ (dev, shared_rss->origin.queue,
+ shared_rss->origin.queue_num,
+ true,
+ !!dev->data->dev_started);
+ if (!shared_rss->ind_tbl)
return rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot setup indirection table");
- }
memcpy(rss_desc.key, shared_rss->origin.key, MLX5_RSS_HASH_KEY_LEN);
rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
rss_desc.const_q = shared_rss->origin.queue;
@@ -14710,19 +14716,20 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
- uint32_t hrxq_idx;
+ struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
int tunnel = 0;
- __flow_dv_action_rss_l34_hash_adjust(shared_rss, &hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(shared_rss->origin.types,
+ &hash_fields);
if (shared_rss->origin.level > 1) {
hash_fields |= IBV_RX_HASH_INNER;
tunnel = 1;
}
rss_desc.tunnel = tunnel;
rss_desc.hash_fields = hash_fields;
- hrxq_idx = mlx5_hrxq_get(dev, &rss_desc);
- if (!hrxq_idx) {
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
@@ -14730,14 +14737,14 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
goto error_hrxq_new;
}
err = __flow_dv_action_rss_hrxq_set
- (shared_rss, hash_fields, hrxq_idx);
+ (shared_rss, hash_fields, hrxq->idx);
MLX5_ASSERT(!err);
}
return 0;
error_hrxq_new:
err = rte_errno;
__flow_dv_action_rss_hrxqs_release(dev, shared_rss);
- if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true, true))
+ if (!mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true))
shared_rss->ind_tbl = NULL;
rte_errno = err;
return -rte_errno;
@@ -14768,18 +14775,14 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss = NULL;
- void *queue = NULL;
struct rte_flow_action_rss *origin;
const uint8_t *rss_key;
- uint32_t queue_size = rss->queue_num * sizeof(uint16_t);
uint32_t idx;
RTE_SET_USED(conf);
- queue = mlx5_malloc(0, RTE_ALIGN_CEIL(queue_size, sizeof(void *)),
- 0, SOCKET_ID_ANY);
shared_rss = mlx5_ipool_zmalloc
(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], &idx);
- if (!shared_rss || !queue) {
+ if (!shared_rss) {
rte_flow_error_set(error, ENOMEM,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
"cannot allocate resource memory");
@@ -14791,18 +14794,6 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
"rss action number out of range");
goto error_rss_init;
}
- shared_rss->ind_tbl = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(*shared_rss->ind_tbl),
- 0, SOCKET_ID_ANY);
- if (!shared_rss->ind_tbl) {
- rte_flow_error_set(error, ENOMEM,
- RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
- "cannot allocate resource memory");
- goto error_rss_init;
- }
- memcpy(queue, rss->queue, queue_size);
- shared_rss->ind_tbl->queues = queue;
- shared_rss->ind_tbl->queues_n = rss->queue_num;
origin = &shared_rss->origin;
origin->func = rss->func;
origin->level = rss->level;
@@ -14813,10 +14804,12 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
memcpy(shared_rss->key, rss_key, MLX5_RSS_HASH_KEY_LEN);
origin->key = &shared_rss->key[0];
origin->key_len = MLX5_RSS_HASH_KEY_LEN;
- origin->queue = queue;
+ origin->queue = rss->queue;
origin->queue_num = rss->queue_num;
if (__flow_dv_action_rss_setup(dev, idx, shared_rss, error))
goto error_rss_init;
+ /* Update queue with indirect table queue memoyr. */
+ origin->queue = shared_rss->ind_tbl->queues;
rte_spinlock_init(&shared_rss->action_rss_sl);
__atomic_add_fetch(&shared_rss->refcnt, 1, __ATOMIC_RELAXED);
rte_spinlock_lock(&priv->shared_act_sl);
@@ -14827,12 +14820,11 @@ __flow_dv_action_rss_create(struct rte_eth_dev *dev,
error_rss_init:
if (shared_rss) {
if (shared_rss->ind_tbl)
- mlx5_free(shared_rss->ind_tbl);
+ mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
+ !!dev->data->dev_started);
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
idx);
}
- if (queue)
- mlx5_free(queue);
return 0;
}
@@ -14860,7 +14852,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
uint32_t old_refcnt = 1;
int remaining;
- uint16_t *queue = NULL;
if (!shared_rss)
return rte_flow_error_set(error, EINVAL,
@@ -14879,8 +14870,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
RTE_FLOW_ERROR_TYPE_ACTION,
NULL,
"shared rss hrxq has references");
- queue = shared_rss->ind_tbl->queues;
- remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl, true,
+ remaining = mlx5_ind_table_obj_release(dev, shared_rss->ind_tbl,
!!dev->data->dev_started);
if (remaining)
return rte_flow_error_set(error, EBUSY,
@@ -14888,7 +14878,6 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
NULL,
"shared rss indirection table has"
" references");
- mlx5_free(queue);
rte_spinlock_lock(&priv->shared_act_sl);
ILIST_REMOVE(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS],
&priv->rss_shared_actions, idx, shared_rss, next);
@@ -15067,7 +15056,7 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx,
mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
int ret = 0;
void *queue = NULL;
- uint16_t *queue_old = NULL;
+ void *queue_i = NULL;
uint32_t queue_size = action_conf->queue_num * sizeof(uint16_t);
bool dev_started = !!dev->data->dev_started;
@@ -15090,22 +15079,23 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx,
memcpy(queue, action_conf->queue, queue_size);
MLX5_ASSERT(shared_rss->ind_tbl);
rte_spinlock_lock(&shared_rss->action_rss_sl);
- queue_old = shared_rss->ind_tbl->queues;
+ queue_i = shared_rss->ind_tbl->queues;
ret = mlx5_ind_table_obj_modify(dev, shared_rss->ind_tbl,
queue, action_conf->queue_num,
true /* standalone */,
dev_started /* ref_new_qs */,
dev_started /* deref_old_qs */);
if (ret) {
- mlx5_free(queue);
ret = rte_flow_error_set(error, rte_errno,
RTE_FLOW_ERROR_TYPE_ACTION, NULL,
"cannot update indirection table");
} else {
- mlx5_free(queue_old);
- shared_rss->origin.queue = queue;
+ /* Restore the queue to indirect table internal queue. */
+ memcpy(queue_i, queue, queue_size);
+ shared_rss->ind_tbl->queues = queue_i;
shared_rss->origin.queue_num = action_conf->queue_num;
}
+ mlx5_free(queue);
rte_spinlock_unlock(&shared_rss->action_rss_sl);
return ret;
}
@@ -16841,11 +16831,12 @@ __flow_dv_meter_get_rss_sub_policy(struct rte_eth_dev *dev,
for (i = 0; i < MLX5_MTR_RTE_COLORS; i++) {
if (!rss_desc[i])
continue;
- hrxq_idx[i] = mlx5_hrxq_get(dev, rss_desc[i]);
- if (!hrxq_idx[i]) {
+ hrxq = mlx5_hrxq_get(dev, rss_desc[i]);
+ if (!hrxq) {
rte_spinlock_unlock(&mtr_policy->sl);
return NULL;
}
+ hrxq_idx[i] = hrxq->idx;
}
sub_policy_num = (mtr_policy->sub_policy_num >>
(MLX5_MTR_SUB_POLICY_NUM_SHIFT * domain)) &
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index d9471eaf48..81542e0fc5 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -7,6 +7,7 @@
#include <mlx5_malloc.h>
#include "mlx5_defs.h"
#include "mlx5_flow.h"
+#include "mlx5_rx.h"
#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
@@ -95,6 +96,56 @@ flow_hw_jump_release(struct rte_eth_dev *dev, struct mlx5_hw_jump_action *jump)
mlx5_hlist_unregister(priv->sh->flow_tbls, &grp->entry);
}
+/**
+ * Register queue/RSS action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] hws_flags
+ * DR action flags.
+ * @param[in] action
+ * rte flow action.
+ *
+ * @return
+ * Table on success, NULL otherwise and rte_errno is set.
+ */
+static inline struct mlx5_hrxq*
+flow_hw_tir_action_register(struct rte_eth_dev *dev,
+ uint32_t hws_flags,
+ const struct rte_flow_action *action)
+{
+ struct mlx5_flow_rss_desc rss_desc = {
+ .hws_flags = hws_flags,
+ };
+ struct mlx5_hrxq *hrxq;
+
+ if (action->type == RTE_FLOW_ACTION_TYPE_QUEUE) {
+ const struct rte_flow_action_queue *queue = action->conf;
+
+ rss_desc.const_q = &queue->index;
+ rss_desc.queue_num = 1;
+ } else {
+ const struct rte_flow_action_rss *rss = action->conf;
+
+ rss_desc.queue_num = rss->queue_num;
+ rss_desc.const_q = rss->queue;
+ memcpy(rss_desc.key,
+ !rss->key ? rss_hash_default_key : rss->key,
+ MLX5_RSS_HASH_KEY_LEN);
+ rss_desc.key_len = MLX5_RSS_HASH_KEY_LEN;
+ rss_desc.types = !rss->types ? RTE_ETH_RSS_IP : rss->types;
+ flow_dv_hashfields_set(0, &rss_desc, &rss_desc.hash_fields);
+ flow_dv_action_rss_l34_hash_adjust(rss->types,
+ &rss_desc.hash_fields);
+ if (rss->level > 1) {
+ rss_desc.hash_fields |= IBV_RX_HASH_INNER;
+ rss_desc.tunnel = 1;
+ }
+ }
+ hrxq = mlx5_hrxq_get(dev, &rss_desc);
+ return hrxq;
+}
+
/**
* Destroy DR actions created by action template.
*
@@ -266,6 +317,40 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ if (masks->conf) {
+ acts->tir = flow_hw_tir_action_register
+ (dev,
+ mlx5_hw_act_flag[!!attr->group][type],
+ actions);
+ if (!acts->tir)
+ goto err;
+ acts->rule_acts[i].action =
+ acts->tir->action;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)) {
+ goto err;
+ }
+ i++;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -319,6 +404,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_attr attr = {
.ingress = 1,
};
+ uint32_t ft_flag;
memcpy(rule_acts, hw_acts->rule_acts,
sizeof(*rule_acts) * hw_acts->acts_num);
@@ -326,6 +412,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
if (LIST_EMPTY(&hw_acts->act_list))
return 0;
attr.group = table->grp->group_id;
+ ft_flag = mlx5_hw_act_flag[!!table->grp->group_id][table->type];
if (table->type == MLX5DR_TABLE_TYPE_FDB) {
attr.transfer = 1;
attr.ingress = 1;
@@ -338,6 +425,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
struct mlx5_hw_jump_action *jump;
+ struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
@@ -359,6 +447,17 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->jump = jump;
job->flow->fate_type = MLX5_FLOW_FATE_JUMP;
break;
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ hrxq = flow_hw_tir_action_register(dev,
+ ft_flag,
+ action);
+ if (!hrxq)
+ return -1;
+ rule_acts[act_data->action_dst].action = hrxq->action;
+ job->flow->hrxq = hrxq;
+ job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
+ break;
default:
break;
}
@@ -565,6 +664,8 @@ flow_hw_pull(struct rte_eth_dev *dev,
if (job->type == MLX5_HW_Q_JOB_TYPE_DESTROY) {
if (job->flow->fate_type == MLX5_FLOW_FATE_JUMP)
flow_hw_jump_release(dev, job->flow->jump);
+ else if (job->flow->fate_type == MLX5_FLOW_FATE_QUEUE)
+ mlx5_hrxq_obj_release(dev, job->flow->hrxq);
mlx5_ipool_free(job->flow->table->flow, job->flow->idx);
}
priv->hw_q[queue].job[priv->hw_q[queue].job_idx++] = job;
diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c
index 90ccb9aaff..f08aa7a770 100644
--- a/drivers/net/mlx5/mlx5_flow_verbs.c
+++ b/drivers/net/mlx5/mlx5_flow_verbs.c
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
MLX5_ASSERT(priv->drop_queue.hrxq);
hrxq = priv->drop_queue.hrxq;
} else {
- uint32_t hrxq_idx;
struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
rss_desc->tunnel = !!(handle->layers &
MLX5_FLOW_LAYER_TUNNEL);
rss_desc->shared_rss = 0;
- hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
- hrxq_idx);
+ hrxq = mlx5_hrxq_get(dev, rss_desc);
if (!hrxq) {
rte_flow_error_set
(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
"cannot get hash queue");
goto error;
}
- handle->rix_hrxq = hrxq_idx;
+ handle->rix_hrxq = hrxq->idx;
}
MLX5_ASSERT(hrxq);
handle->drv_flow = mlx5_glue->create_flow
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 29652a8c9f..acebe3348c 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -231,9 +231,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
const uint16_t *queues,
uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+ const uint16_t *queues,
+ uint32_t queues_n,
+ bool standalone,
+ bool ref_qs);
int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs);
int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
@@ -256,8 +260,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
void *cb_ctx __rte_unused);
void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
bool mlx5_rxq_is_hairpin(struct rte_eth_dev *dev, uint16_t idx);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 22679755a4..1df383c827 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -2364,8 +2364,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
* Pointer to Ethernet device.
* @param ind_table
* Indirection table to release.
- * @param standalone
- * Indirection table for Standalone queue.
* @param deref_rxqs
* If true, then dereference RX queues related to indirection table.
* Otherwise, no additional action will be taken.
@@ -2376,7 +2374,6 @@ mlx5_ind_table_obj_get(struct rte_eth_dev *dev, const uint16_t *queues,
int
mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
struct mlx5_ind_table_obj *ind_tbl,
- bool standalone,
bool deref_rxqs)
{
struct mlx5_priv *priv = dev->data->dev_private;
@@ -2384,7 +2381,7 @@ mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
rte_rwlock_write_lock(&priv->ind_tbls_lock);
ret = __atomic_sub_fetch(&ind_tbl->refcnt, 1, __ATOMIC_RELAXED);
- if (!ret && !standalone)
+ if (!ret)
LIST_REMOVE(ind_tbl, next);
rte_rwlock_write_unlock(&priv->ind_tbls_lock);
if (ret)
@@ -2504,7 +2501,7 @@ mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
* @return
* The Verbs/DevX object initialized, NULL otherwise and rte_errno is set.
*/
-static struct mlx5_ind_table_obj *
+struct mlx5_ind_table_obj *
mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
uint32_t queues_n, bool standalone, bool ref_qs)
{
@@ -2512,8 +2509,13 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
struct mlx5_ind_table_obj *ind_tbl;
int ret;
+ /*
+ * Allocate maximum queues for shared action as queue number
+ * maybe modified later.
+ */
ind_tbl = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*ind_tbl) +
- queues_n * sizeof(uint16_t), 0, SOCKET_ID_ANY);
+ (standalone ? priv->rxqs_n : queues_n) *
+ sizeof(uint16_t), 0, SOCKET_ID_ANY);
if (!ind_tbl) {
rte_errno = ENOMEM;
return NULL;
@@ -2526,11 +2528,13 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues,
mlx5_free(ind_tbl);
return NULL;
}
- if (!standalone) {
- rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ rte_rwlock_write_lock(&priv->ind_tbls_lock);
+ if (!standalone)
LIST_INSERT_HEAD(&priv->ind_tbls, ind_tbl, next);
- rte_rwlock_write_unlock(&priv->ind_tbls_lock);
- }
+ else
+ LIST_INSERT_HEAD(&priv->standalone_ind_tbls, ind_tbl, next);
+ rte_rwlock_write_unlock(&priv->ind_tbls_lock);
+
return ind_tbl;
}
@@ -2696,6 +2700,7 @@ mlx5_hrxq_match_cb(void *tool_ctx __rte_unused, struct mlx5_list_entry *entry,
return (hrxq->rss_key_len != rss_desc->key_len ||
memcmp(hrxq->rss_key, rss_desc->key, rss_desc->key_len) ||
+ hrxq->hws_flags != rss_desc->hws_flags ||
hrxq->hash_fields != rss_desc->hash_fields ||
hrxq->ind_table->queues_n != rss_desc->queue_num ||
memcmp(hrxq->ind_table->queues, rss_desc->queue,
@@ -2780,8 +2785,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
}
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ mlx5_ind_table_obj_release(dev, hrxq->ind_table, true);
hrxq->ind_table = ind_tbl;
}
hrxq->hash_fields = hash_fields;
@@ -2791,8 +2795,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx,
err = rte_errno;
if (ind_tbl != hrxq->ind_table) {
MLX5_ASSERT(!hrxq->standalone);
- mlx5_ind_table_obj_release(dev, ind_tbl, hrxq->standalone,
- true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
}
rte_errno = err;
return -rte_errno;
@@ -2804,12 +2807,16 @@ __mlx5_hrxq_remove(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
struct mlx5_priv *priv = dev->data->dev_private;
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- mlx5_glue->destroy_flow_action(hrxq->action);
+ if (hrxq->hws_flags)
+ mlx5dr_action_destroy(hrxq->action);
+ else
+ mlx5_glue->destroy_flow_action(hrxq->action);
#endif
priv->obj_ops.hrxq_destroy(hrxq);
if (!hrxq->standalone) {
mlx5_ind_table_obj_release(dev, hrxq->ind_table,
- hrxq->standalone, true);
+ hrxq->hws_flags ?
+ (!!dev->data->dev_started) : true);
}
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq->idx);
}
@@ -2853,11 +2860,12 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
int ret;
queues_n = rss_desc->hash_fields ? queues_n : 1;
- if (!ind_tbl)
+ if (!ind_tbl && !rss_desc->hws_flags)
ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n);
if (!ind_tbl)
ind_tbl = mlx5_ind_table_obj_new(dev, queues, queues_n,
- standalone,
+ standalone ||
+ rss_desc->hws_flags,
!!dev->data->dev_started);
if (!ind_tbl)
return NULL;
@@ -2869,6 +2877,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
hrxq->ind_table = ind_tbl;
hrxq->rss_key_len = rss_key_len;
hrxq->hash_fields = rss_desc->hash_fields;
+ hrxq->hws_flags = rss_desc->hws_flags;
memcpy(hrxq->rss_key, rss_key, rss_key_len);
ret = priv->obj_ops.hrxq_new(dev, hrxq, rss_desc->tunnel);
if (ret < 0)
@@ -2876,7 +2885,7 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev,
return hrxq;
error:
if (!rss_desc->ind_tbl)
- mlx5_ind_table_obj_release(dev, ind_tbl, standalone, true);
+ mlx5_ind_table_obj_release(dev, ind_tbl, true);
if (hrxq)
mlx5_ipool_free(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
return NULL;
@@ -2930,13 +2939,13 @@ mlx5_hrxq_clone_free_cb(void *tool_ctx, struct mlx5_list_entry *entry)
* RSS configuration for the Rx hash queue.
*
* @return
- * An hash Rx queue index on success.
+ * An hash Rx queue on success.
*/
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
struct mlx5_flow_rss_desc *rss_desc)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
+ struct mlx5_hrxq *hrxq = NULL;
struct mlx5_list_entry *entry;
struct mlx5_flow_cb_ctx ctx = {
.data = rss_desc,
@@ -2947,12 +2956,10 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
} else {
entry = mlx5_list_register(priv->hrxqs, &ctx);
if (!entry)
- return 0;
+ return NULL;
hrxq = container_of(entry, typeof(*hrxq), entry);
}
- if (hrxq)
- return hrxq->idx;
- return 0;
+ return hrxq;
}
/**
@@ -2961,17 +2968,15 @@ uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
* @param dev
* Pointer to Ethernet device.
* @param hrxq_idx
- * Index to Hash Rx queue to release.
+ * Hash Rx queue to release.
*
* @return
* 1 while a reference on it exists, 0 when freed.
*/
-int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_hrxq *hrxq;
- hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
if (!hrxq)
return 0;
if (!hrxq->standalone)
@@ -2980,6 +2985,26 @@ int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
return 0;
}
+/**
+ * Release the hash Rx queue with index.
+ *
+ * @param dev
+ * Pointer to Ethernet device.
+ * @param hrxq_idx
+ * Index to Hash Rx queue to release.
+ *
+ * @return
+ * 1 while a reference on it exists, 0 when freed.
+ */
+int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hrxq_idx)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_hrxq *hrxq;
+
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx);
+ return mlx5_hrxq_obj_release(dev, hrxq);
+}
+
/**
* Create a drop Rx Hash queue.
*
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 12/14] net/mlx5: add mark action
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (10 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 11/14] net/mlx5: add queue and RSS action Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 13/14] net/mlx5: add indirect action Suanming Mou
` (2 subsequent siblings)
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
The mark action is covered by tag action internally. While it is added
the HW will add a tag to the packet. The mark value can be set as fixed
or dynamic as the action mask indicates.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 2 +
drivers/net/mlx5/mlx5_flow.h | 1 +
drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++---
3 files changed, 63 insertions(+), 6 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index fe422f7fd1..df0257ff23 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1532,6 +1532,8 @@ struct mlx5_priv {
/* HW steering global drop action. */
struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
[MLX5DR_TABLE_TYPE_MAX];
+ /* HW steering global drop action. */
+ struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX];
struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
#endif
};
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 70e6cf633f..c83e73c793 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
uint32_t acts_num:4; /* Total action number. */
+ uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
};
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 81542e0fc5..c3423597bd 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -37,6 +37,31 @@ static uint32_t mlx5_hw_act_flag[MLX5_HW_ACTION_FLAG_MAX]
},
};
+/**
+ * Set rxq flag.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] enable
+ * Flag to enable or not.
+ */
+static void
+flow_hw_rxq_flag_set(struct rte_eth_dev *dev, bool enable)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ unsigned int i;
+
+ if ((!priv->mark_enabled && !enable) ||
+ (priv->mark_enabled && enable))
+ return;
+ for (i = 0; i < priv->rxqs_n; ++i) {
+ struct mlx5_rxq_ctrl *rxq_ctrl = mlx5_rxq_ctrl_get(dev, i);
+
+ rxq_ctrl->rxq.mark = enable;
+ }
+ priv->mark_enabled = enable;
+}
+
/**
* Register destination table DR jump action.
*
@@ -298,6 +323,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
acts->rule_acts[i++].action =
priv->hw_drop[!!attr->group][type];
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ acts->mark = true;
+ if (masks->conf)
+ acts->rule_acts[i].tag.value =
+ mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (masks->conf))->id);
+ else if (__flow_hw_act_data_general_append(priv, acts,
+ actions->type, actions - action_start, i))
+ goto err;
+ acts->rule_acts[i++].action =
+ priv->hw_tag[!!attr->group];
+ flow_hw_rxq_flag_set(dev, true);
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
if (masks->conf) {
uint32_t jump_group =
@@ -424,6 +463,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
}
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
+ uint32_t tag;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
@@ -435,6 +475,12 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
+ case RTE_FLOW_ACTION_TYPE_MARK:
+ tag = mlx5_flow_mark_set
+ (((const struct rte_flow_action_mark *)
+ (action->conf))->id);
+ rule_acts[act_data->action_dst].tag.value = tag;
+ break;
case RTE_FLOW_ACTION_TYPE_JUMP:
jump_group = ((const struct rte_flow_action_jump *)
action->conf)->group;
@@ -1029,6 +1075,8 @@ flow_hw_table_destroy(struct rte_eth_dev *dev,
__atomic_sub_fetch(&table->its[i]->refcnt,
1, __ATOMIC_RELAXED);
for (i = 0; i < table->nb_action_templates; i++) {
+ if (table->ats[i].acts.mark)
+ flow_hw_rxq_flag_set(dev, false);
__flow_hw_action_template_destroy(dev, &table->ats[i].acts);
__atomic_sub_fetch(&table->ats[i].action_template->refcnt,
1, __ATOMIC_RELAXED);
@@ -1563,15 +1611,20 @@ flow_hw_configure(struct rte_eth_dev *dev,
if (!priv->hw_drop[i][j])
goto err;
}
+ priv->hw_tag[i] = mlx5dr_action_create_tag
+ (priv->dr_ctx, mlx5_hw_act_flag[i][0]);
+ if (!priv->hw_tag[i])
+ goto err;
}
return 0;
err:
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
+ if (priv->hw_tag[i])
+ mlx5dr_action_destroy(priv->hw_tag[i]);
}
if (dr_ctx)
claim_zero(mlx5dr_context_close(dr_ctx));
@@ -1617,10 +1670,11 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
}
for (i = 0; i < MLX5_HW_ACTION_FLAG_MAX; i++) {
for (j = 0; j < MLX5DR_TABLE_TYPE_MAX; j++) {
- if (!priv->hw_drop[i][j])
- continue;
- mlx5dr_action_destroy(priv->hw_drop[i][j]);
+ if (priv->hw_drop[i][j])
+ mlx5dr_action_destroy(priv->hw_drop[i][j]);
}
+ if (priv->hw_tag[i])
+ mlx5dr_action_destroy(priv->hw_tag[i]);
}
if (priv->acts_ipool) {
mlx5_ipool_destroy(priv->acts_ipool);
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 13/14] net/mlx5: add indirect action
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (11 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 12/14] net/mlx5: add mark action Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 14/14] net/mlx5: add header reformat action Suanming Mou
2022-02-24 21:12 ` [PATCH v4 00/14] net/mlx5: add hardware steering Raslan Darawsheh
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering can support indirect action as well. With indirect action,
the flow can be created with more flexible shared RSS action selection.
This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:
rte_flow_async_action_handle_create();
rte_flow_async_action_handle_destroy();
rte_flow_async_action_handle_update();
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5_flow.c | 131 ++++++++++
drivers/net/mlx5/mlx5_flow.h | 59 +++++
drivers/net/mlx5/mlx5_flow_dv.c | 21 +-
drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++-
4 files changed, 612 insertions(+), 13 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 7c3e4ec47a..3875160708 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,
uint32_t queue,
struct rte_flow_error *error);
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error);
+
static const struct rte_flow_ops mlx5_flow_ops = {
.validate = mlx5_flow_validate,
.create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
.async_destroy = mlx5_flow_async_flow_destroy,
.pull = mlx5_flow_pull,
.push = mlx5_flow_push,
+ .async_action_handle_create = mlx5_flow_async_action_handle_create,
+ .async_action_handle_update = mlx5_flow_async_action_handle_update,
+ .async_action_handle_destroy = mlx5_flow_async_action_handle_destroy,
};
/* Tunnel information. */
@@ -8376,6 +8402,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,
return fops->push(dev, queue, error);
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_create(dev, queue, attr, conf, action,
+ user_data, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_update(dev, queue, attr, handle,
+ update, user_data, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ const struct mlx5_flow_driver_ops *fops =
+ flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+ return fops->async_action_destroy(dev, queue, attr, handle,
+ user_data, error);
+}
+
/**
* Allocate a new memory for the counter values wrapped by all the needed
* management.
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index c83e73c793..4c224bbf52 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -41,6 +41,7 @@ enum mlx5_rte_flow_action_type {
MLX5_RTE_FLOW_ACTION_TYPE_AGE,
MLX5_RTE_FLOW_ACTION_TYPE_COUNT,
MLX5_RTE_FLOW_ACTION_TYPE_JUMP,
+ MLX5_RTE_FLOW_ACTION_TYPE_RSS,
};
#define MLX5_INDIRECT_ACTION_TYPE_OFFSET 30
@@ -1038,6 +1039,13 @@ struct mlx5_action_construct_data {
uint32_t idx; /* Data index. */
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+ union {
+ struct {
+ uint64_t types; /* RSS hash types. */
+ uint32_t level; /* RSS level. */
+ uint32_t idx; /* Shared action index. */
+ } shared_rss;
+ };
};
/* Flow item template struct. */
@@ -1046,6 +1054,7 @@ struct rte_flow_pattern_template {
/* Template attributes. */
struct rte_flow_pattern_template_attr attr;
struct mlx5dr_match_template *mt; /* mlx5 match template. */
+ uint64_t item_flags; /* Item layer flags. */
uint32_t refcnt; /* Reference counter. */
};
@@ -1433,6 +1442,32 @@ typedef int (*mlx5_flow_push_t)
uint32_t queue,
struct rte_flow_error *error);
+typedef struct rte_flow_action_handle *(*mlx5_flow_async_action_handle_create_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_async_action_handle_update_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error);
+
+typedef int (*mlx5_flow_async_action_handle_destroy_t)
+ (struct rte_eth_dev *dev,
+ uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error);
+
struct mlx5_flow_driver_ops {
mlx5_flow_validate_t validate;
mlx5_flow_prepare_t prepare;
@@ -1482,6 +1517,9 @@ struct mlx5_flow_driver_ops {
mlx5_flow_async_flow_destroy_t async_flow_destroy;
mlx5_flow_pull_t pull;
mlx5_flow_push_t push;
+ mlx5_flow_async_action_handle_create_t async_action_create;
+ mlx5_flow_async_action_handle_update_t async_action_update;
+ mlx5_flow_async_action_handle_destroy_t async_action_destroy;
};
/* mlx5_flow.c */
@@ -1918,6 +1956,8 @@ void flow_dv_hashfields_set(uint64_t item_flags,
uint64_t *hash_fields);
void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
uint64_t *hash_field);
+uint32_t flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields);
struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
@@ -1968,4 +2008,23 @@ mlx5_get_tof(const struct rte_flow_item *items,
enum mlx5_tof_rule_type *rule_type);
void
flow_hw_resource_release(struct rte_eth_dev *dev);
+int flow_dv_action_validate(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+struct rte_flow_action_handle *flow_dv_action_create(struct rte_eth_dev *dev,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ struct rte_flow_error *err);
+int flow_dv_action_destroy(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ struct rte_flow_error *error);
+int flow_dv_action_update(struct rte_eth_dev *dev,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ struct rte_flow_error *err);
+int flow_dv_action_query(struct rte_eth_dev *dev,
+ const struct rte_flow_action_handle *handle,
+ void *data,
+ struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 2ffdfad61f..5686986f5c 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -13841,9 +13841,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
* @return
* Valid hash RX queue index, otherwise 0.
*/
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
- const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+ const uint64_t hash_fields)
{
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_shared_action_rss *shared_rss =
@@ -13971,7 +13971,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
struct mlx5_hrxq *hrxq = NULL;
uint32_t hrxq_idx;
- hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
rss_desc->shared_rss,
dev_flow->hash_fields);
if (hrxq_idx)
@@ -14695,6 +14695,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
struct mlx5_shared_action_rss *shared_rss,
struct rte_flow_error *error)
{
+ struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_flow_rss_desc rss_desc = { 0 };
size_t i;
int err;
@@ -14715,6 +14716,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
/* Set non-zero value to indicate a shared RSS. */
rss_desc.shared_rss = action_idx;
rss_desc.ind_tbl = shared_rss->ind_tbl;
+ if (priv->sh->config.dv_flow_en == 2)
+ rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
struct mlx5_hrxq *hrxq;
uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14906,7 +14909,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
* A valid shared action handle in case of success, NULL otherwise and
* rte_errno is set.
*/
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
flow_dv_action_create(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
@@ -14976,7 +14979,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_destroy(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
struct rte_flow_error *error)
@@ -15186,7 +15189,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_update(struct rte_eth_dev *dev,
struct rte_flow_action_handle *handle,
const void *update,
@@ -15858,7 +15861,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
"counters are not available");
}
-static int
+int
flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle, void *data,
struct rte_flow_error *error)
@@ -17551,7 +17554,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
* @return
* 0 on success, otherwise negative errno value.
*/
-static int
+int
flow_dv_action_validate(struct rte_eth_dev *dev,
const struct rte_flow_indir_action_conf *conf,
const struct rte_flow_action *action,
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index c3423597bd..29c2e02335 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -62,6 +62,72 @@ flow_hw_rxq_flag_set(struct rte_eth_dev *dev, bool enable)
priv->mark_enabled = enable;
}
+/**
+ * Generate the pattern item flags.
+ * Will be used for shared RSS action.
+ *
+ * @param[in] items
+ * Pointer to the list of items.
+ *
+ * @return
+ * Item flags.
+ */
+static uint64_t
+flow_hw_rss_item_flags_get(const struct rte_flow_item items[])
+{
+ uint64_t item_flags = 0;
+ uint64_t last_item = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) {
+ int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL);
+ int item_type = items->type;
+
+ switch (item_type) {
+ case RTE_FLOW_ITEM_TYPE_IPV4:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV4;
+ break;
+ case RTE_FLOW_ITEM_TYPE_IPV6:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 :
+ MLX5_FLOW_LAYER_OUTER_L3_IPV6;
+ break;
+ case RTE_FLOW_ITEM_TYPE_TCP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP :
+ MLX5_FLOW_LAYER_OUTER_L4_TCP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_UDP:
+ last_item = tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP :
+ MLX5_FLOW_LAYER_OUTER_L4_UDP;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_NVGRE:
+ last_item = MLX5_FLOW_LAYER_GRE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN:
+ last_item = MLX5_FLOW_LAYER_VXLAN;
+ break;
+ case RTE_FLOW_ITEM_TYPE_VXLAN_GPE:
+ last_item = MLX5_FLOW_LAYER_VXLAN_GPE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GENEVE:
+ last_item = MLX5_FLOW_LAYER_GENEVE;
+ break;
+ case RTE_FLOW_ITEM_TYPE_MPLS:
+ last_item = MLX5_FLOW_LAYER_MPLS;
+ break;
+ case RTE_FLOW_ITEM_TYPE_GTP:
+ last_item = MLX5_FLOW_LAYER_GTP;
+ break;
+ default:
+ break;
+ }
+ item_flags |= last_item;
+ }
+ return item_flags;
+}
+
/**
* Register destination table DR jump action.
*
@@ -266,6 +332,96 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append shared RSS action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] idx
+ * Shared RSS index.
+ * @param[in] rss
+ * Pointer to the shared RSS info.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_shared_rss_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint32_t idx,
+ struct mlx5_shared_action_rss *rss)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->shared_rss.level = rss->origin.level;
+ act_data->shared_rss.types = !rss->origin.types ? RTE_ETH_RSS_IP :
+ rss->origin.types;
+ act_data->shared_rss.idx = idx;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
+/**
+ * Translate shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_translate(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct mlx5_hw_actions *acts,
+ uint16_t action_src,
+ uint16_t action_dst)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss || __flow_hw_act_data_shared_rss_append
+ (priv, acts,
+ (enum rte_flow_action_type)MLX5_RTE_FLOW_ACTION_TYPE_RSS,
+ action_src, action_dst, idx, shared_rss))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -316,6 +472,20 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
for (i = 0; !actions_end; actions++, masks++) {
switch (actions->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (!attr->group) {
+ DRV_LOG(ERR, "Indirect action is not supported in root table.");
+ goto err;
+ }
+ if (actions->conf && masks->conf) {
+ if (flow_hw_shared_action_translate
+ (dev, actions, acts, actions - action_start, i))
+ goto err;
+ } else if (__flow_hw_act_data_general_append
+ (priv, acts, actions->type,
+ actions - action_start, i)){
+ goto err;
+ }
+ i++;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -407,6 +577,115 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
"fail to create rte table");
}
+/**
+ * Get shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] act_data
+ * Pointer to the recorded action construct data.
+ * @param[in] item_flags
+ * The matcher itme_flags used for RSS lookup.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_get(struct rte_eth_dev *dev,
+ struct mlx5_action_construct_data *act_data,
+ const uint64_t item_flags,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_flow_rss_desc rss_desc = { 0 };
+ uint64_t hash_fields = 0;
+ uint32_t hrxq_idx = 0;
+ struct mlx5_hrxq *hrxq = NULL;
+ int act_type = act_data->type;
+
+ switch (act_type) {
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ rss_desc.level = act_data->shared_rss.level;
+ rss_desc.types = act_data->shared_rss.types;
+ flow_dv_hashfields_set(item_flags, &rss_desc, &hash_fields);
+ hrxq_idx = flow_dv_action_rss_hrxq_lookup
+ (dev, act_data->shared_rss.idx, hash_fields);
+ if (hrxq_idx)
+ hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
+ hrxq_idx);
+ if (hrxq) {
+ rule_act->action = hrxq->action;
+ return 0;
+ }
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d",
+ act_data->type);
+ break;
+ }
+ return -1;
+}
+
+/**
+ * Construct shared indirect action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] action
+ * Pointer to the shared indirect rte_flow action.
+ * @param[in] table
+ * Pointer to the flow table.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
+ * @param[in] rule_act
+ * Pointer to the shared action's destination rule DR action.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_shared_action_construct(struct rte_eth_dev *dev,
+ const struct rte_flow_action *action,
+ struct rte_flow_template_table *table,
+ const uint8_t it_idx,
+ struct mlx5dr_rule_action *rule_act)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_action_construct_data act_data;
+ struct mlx5_shared_action_rss *shared_rss;
+ uint32_t act_idx = (uint32_t)(uintptr_t)action->conf;
+ uint32_t type = act_idx >> MLX5_INDIRECT_ACTION_TYPE_OFFSET;
+ uint32_t idx = act_idx &
+ ((1u << MLX5_INDIRECT_ACTION_TYPE_OFFSET) - 1);
+ uint64_t item_flags;
+
+ memset(&act_data, 0, sizeof(act_data));
+ switch (type) {
+ case MLX5_INDIRECT_ACTION_TYPE_RSS:
+ act_data.type = MLX5_RTE_FLOW_ACTION_TYPE_RSS;
+ shared_rss = mlx5_ipool_get
+ (priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS], idx);
+ if (!shared_rss)
+ return -1;
+ act_data.shared_rss.idx = idx;
+ act_data.shared_rss.level = shared_rss->origin.level;
+ act_data.shared_rss.types = !shared_rss->origin.types ?
+ RTE_ETH_RSS_IP :
+ shared_rss->origin.types;
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, &act_data, item_flags, rule_act))
+ return -1;
+ break;
+ default:
+ DRV_LOG(WARNING, "Unsupported shared action type:%d", type);
+ break;
+ }
+ return 0;
+}
+
/**
* Construct flow action array.
*
@@ -419,6 +698,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
* Pointer to job descriptor.
* @param[in] hw_acts
* Pointer to translated actions from template.
+ * @param[in] it_idx
+ * Item template index the action template refer to.
* @param[in] actions
* Array of rte_flow action need to be checked.
* @param[in] rule_acts
@@ -432,7 +713,8 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
static __rte_always_inline int
flow_hw_actions_construct(struct rte_eth_dev *dev,
struct mlx5_hw_q_job *job,
- struct mlx5_hw_actions *hw_acts,
+ const struct mlx5_hw_actions *hw_acts,
+ const uint8_t it_idx,
const struct rte_flow_action actions[],
struct mlx5dr_rule_action *rule_acts,
uint32_t *acts_num)
@@ -464,14 +746,19 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
+ uint64_t item_flags;
struct mlx5_hw_jump_action *jump;
struct mlx5_hrxq *hrxq;
action = &actions[act_data->action_src];
MLX5_ASSERT(action->type == RTE_FLOW_ACTION_TYPE_INDIRECT ||
(int)action->type == act_data->type);
- switch (action->type) {
+ switch (act_data->type) {
case RTE_FLOW_ACTION_TYPE_INDIRECT:
+ if (flow_hw_shared_action_construct
+ (dev, action, table, it_idx,
+ &rule_acts[act_data->action_dst]))
+ return -1;
break;
case RTE_FLOW_ACTION_TYPE_VOID:
break;
@@ -504,6 +791,13 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
job->flow->hrxq = hrxq;
job->flow->fate_type = MLX5_FLOW_FATE_QUEUE;
break;
+ case MLX5_RTE_FLOW_ACTION_TYPE_RSS:
+ item_flags = table->its[it_idx]->item_flags;
+ if (flow_hw_shared_action_get
+ (dev, act_data, item_flags,
+ &rule_acts[act_data->action_dst]))
+ return -1;
+ break;
default:
break;
}
@@ -589,8 +883,8 @@ flow_hw_async_flow_create(struct rte_eth_dev *dev,
rule_attr.user_data = job;
hw_acts = &table->ats[action_template_index].acts;
/* Construct the flow action array based on the input actions.*/
- flow_hw_actions_construct(dev, job, hw_acts, actions,
- rule_acts, &acts_num);
+ flow_hw_actions_construct(dev, job, hw_acts, pattern_template_index,
+ actions, rule_acts, &acts_num);
ret = mlx5dr_rule_create(table->matcher,
pattern_template_index, items,
rule_acts, acts_num,
@@ -1239,6 +1533,7 @@ flow_hw_pattern_template_create(struct rte_eth_dev *dev,
"cannot create match template");
return NULL;
}
+ it->item_flags = flow_hw_rss_item_flags_get(items);
__atomic_fetch_add(&it->refcnt, 1, __ATOMIC_RELAXED);
LIST_INSERT_HEAD(&priv->flow_hw_itt, it, next);
return it;
@@ -1687,6 +1982,109 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
priv->nb_queue = 0;
}
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] conf
+ * Indirect action configuration.
+ * @param[in] action
+ * rte_flow action detail.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+flow_hw_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ const struct rte_flow_indir_action_conf *conf,
+ const struct rte_flow_action *action,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_create(dev, conf, action, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be updated.
+ * @param[in] update
+ * Update value.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ const void *update,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_update(dev, handle, update, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ * Which queue to be used..
+ * @param[in] attr
+ * Operation attribute.
+ * @param[in] handle
+ * Action handle to be destroyed.
+ * @param[in] user_data
+ * Pointer to the user_data.
+ * @param[out] error
+ * Pointer to error structure.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+flow_hw_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+ const struct rte_flow_op_attr *attr,
+ struct rte_flow_action_handle *handle,
+ void *user_data,
+ struct rte_flow_error *error)
+{
+ RTE_SET_USED(queue);
+ RTE_SET_USED(attr);
+ RTE_SET_USED(user_data);
+ return flow_dv_action_destroy(dev, handle, error);
+}
+
+
const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.info_get = flow_hw_info_get,
.configure = flow_hw_configure,
@@ -1700,6 +2098,14 @@ const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops = {
.async_flow_destroy = flow_hw_async_flow_destroy,
.pull = flow_hw_pull,
.push = flow_hw_push,
+ .async_action_create = flow_hw_action_handle_create,
+ .async_action_destroy = flow_hw_action_handle_destroy,
+ .async_action_update = flow_hw_action_handle_update,
+ .action_validate = flow_dv_action_validate,
+ .action_create = flow_dv_action_create,
+ .action_destroy = flow_dv_action_destroy,
+ .action_update = flow_dv_action_update,
+ .action_query = flow_dv_action_query,
};
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* [PATCH v4 14/14] net/mlx5: add header reformat action
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (12 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 13/14] net/mlx5: add indirect action Suanming Mou
@ 2022-02-24 13:40 ` Suanming Mou
2022-02-24 21:12 ` [PATCH v4 00/14] net/mlx5: add hardware steering Raslan Darawsheh
14 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 13:40 UTC (permalink / raw)
To: viacheslavo, matan; +Cc: rasland, orika, dev
HW steering header reformat action can work under bulk mode. In
this case, when create the table, bulk size of header reformat
actions will be allocated in low level. Afterwards, when create
flow, just simply specify the action index in the bulk and the
encapsulation data to the action will be enough.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_flow.h | 21 +++
drivers/net/mlx5/mlx5_flow_dv.c | 4 +-
drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
4 files changed, 251 insertions(+), 3 deletions(-)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index df0257ff23..0f825396a2 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {
uint32_t type; /* Job type. */
struct rte_flow_hw *flow; /* Flow attached to the job. */
void *user_data; /* Job user data. */
+ uint8_t *encap_data; /* Encap data. */
};
/* HW steering job descriptor LIFO pool. */
diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h
index 4c224bbf52..09f0d7a75d 100644
--- a/drivers/net/mlx5/mlx5_flow.h
+++ b/drivers/net/mlx5/mlx5_flow.h
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {
uint16_t action_src; /* rte_flow_action src offset. */
uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
union {
+ struct {
+ /* encap src(item) offset. */
+ uint16_t src;
+ /* encap dst data offset. */
+ uint16_t dst;
+ /* encap data len. */
+ uint16_t len;
+ } encap;
struct {
uint64_t types; /* RSS hash types. */
uint32_t level; /* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {
struct mlx5dr_action *hws_action;
};
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+ struct mlx5dr_action *action; /* Action object. */
+ size_t data_size; /* Action metadata size. */
+ uint8_t data[]; /* Action data. */
+};
+
/* The maximum actions support in the flow. */
#define MLX5_HW_MAX_ACTS 16
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions {
LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
struct mlx5_hw_jump_action *jump; /* Jump action. */
struct mlx5_hrxq *tir; /* TIR action. */
+ /* Encap/Decap action. */
+ struct mlx5_hw_encap_decap_action *encap_decap;
+ uint16_t encap_decap_pos; /* Encap/Decap action position. */
uint32_t acts_num:4; /* Total action number. */
uint32_t mark:1; /* Indicate the mark action. */
/* Translated DR action array from action template. */
@@ -2027,4 +2045,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
const struct rte_flow_action_handle *handle,
void *data,
struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+ size_t *size, struct rte_flow_error *error);
#endif /* RTE_PMD_MLX5_FLOW_H_ */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 5686986f5c..313dc64604 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -4034,7 +4034,7 @@ flow_dv_push_vlan_action_resource_register
* @return
* sizeof struct item_type, 0 if void or irrelevant.
*/
-static size_t
+size_t
flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
{
size_t retval;
@@ -4100,7 +4100,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-static int
+int
flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
size_t *size, struct rte_flow_error *error)
{
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 29c2e02335..2c2bd0feef 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -332,6 +332,50 @@ __flow_hw_act_data_general_append(struct mlx5_priv *priv,
return 0;
}
+/**
+ * Append dynamic encap action to the dynamic action list.
+ *
+ * @param[in] priv
+ * Pointer to the port private data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] encap_src
+ * Offset of source encap raw data.
+ * @param[in] encap_dst
+ * Offset of destination encap raw data.
+ * @param[in] len
+ * Length of the data to be updated.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+__flow_hw_act_data_encap_append(struct mlx5_priv *priv,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ uint16_t encap_src,
+ uint16_t encap_dst,
+ uint16_t len)
+{ struct mlx5_action_construct_data *act_data;
+
+ act_data = __flow_hw_act_data_alloc(priv, type, action_src, action_dst);
+ if (!act_data)
+ return -1;
+ act_data->encap.src = encap_src;
+ act_data->encap.dst = encap_dst;
+ act_data->encap.len = len;
+ LIST_INSERT_HEAD(&acts->act_list, act_data, next);
+ return 0;
+}
+
/**
* Append shared RSS action to the dynamic action list.
*
@@ -422,6 +466,53 @@ flow_hw_shared_action_translate(struct rte_eth_dev *dev,
return 0;
}
+/**
+ * Translate encap items to encapsulation list.
+ *
+ * @param[in] dev
+ * Pointer to the rte_eth_dev data structure.
+ * @param[in] acts
+ * Pointer to the template HW steering DR actions.
+ * @param[in] type
+ * Action type.
+ * @param[in] action_src
+ * Offset of source rte flow action.
+ * @param[in] action_dst
+ * Offset of destination DR action.
+ * @param[in] items
+ * Encap item pattern.
+ * @param[in] items_m
+ * Encap item mask indicates which part are constant and dynamic.
+ *
+ * @return
+ * 0 on success, negative value otherwise and rte_errno is set.
+ */
+static __rte_always_inline int
+flow_hw_encap_item_translate(struct rte_eth_dev *dev,
+ struct mlx5_hw_actions *acts,
+ enum rte_flow_action_type type,
+ uint16_t action_src,
+ uint16_t action_dst,
+ const struct rte_flow_item *items,
+ const struct rte_flow_item *items_m)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+ size_t len, total_len = 0;
+ uint32_t i = 0;
+
+ for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++, items_m++, i++) {
+ len = flow_dv_get_item_hdr_len(items->type);
+ if ((!items_m->spec ||
+ memcmp(items_m->spec, items->spec, len)) &&
+ __flow_hw_act_data_encap_append(priv, acts, type,
+ action_src, action_dst, i,
+ total_len, len))
+ return -1;
+ total_len += len;
+ }
+ return 0;
+}
+
/**
* Translate rte_flow actions to DR action.
*
@@ -459,6 +550,12 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
struct rte_flow_action *actions = at->actions;
struct rte_flow_action *action_start = actions;
struct rte_flow_action *masks = at->masks;
+ enum mlx5dr_action_reformat_type refmt_type = 0;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL, *enc_item_m = NULL;
+ uint16_t reformat_pos = MLX5_HW_MAX_ACTS, reformat_src = 0;
+ uint8_t *encap_data = NULL;
+ size_t data_size = 0;
bool actions_end = false;
uint32_t type, i;
int err;
@@ -560,6 +657,56 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
}
i++;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_vxlan_encap *)
+ masks->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ enc_item_m =
+ ((const struct rte_flow_action_nvgre_encap *)
+ actions->conf)->definition;
+ reformat_pos = i++;
+ reformat_src = actions - action_start;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_DECAP:
+ case RTE_FLOW_ACTION_TYPE_NVGRE_DECAP:
+ MLX5_ASSERT(reformat_pos == MLX5_HW_MAX_ACTS);
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ actions->conf;
+ encap_data = raw_encap_data->data;
+ data_size = raw_encap_data->size;
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ refmt_type = data_size <
+ MLX5_ENCAPSULATION_DECISION_SIZE ?
+ MLX5DR_ACTION_REFORMAT_TYPE_TNL_L3_TO_L2 :
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L3;
+ } else {
+ reformat_pos = i++;
+ refmt_type =
+ MLX5DR_ACTION_REFORMAT_TYPE_L2_TO_TNL_L2;
+ }
+ reformat_src = actions - action_start;
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_DECAP:
+ reformat_pos = i++;
+ refmt_type = MLX5DR_ACTION_REFORMAT_TYPE_TNL_L2_TO_L2;
+ break;
case RTE_FLOW_ACTION_TYPE_END:
actions_end = true;
break;
@@ -567,6 +714,45 @@ flow_hw_actions_translate(struct rte_eth_dev *dev,
break;
}
}
+ if (reformat_pos != MLX5_HW_MAX_ACTS) {
+ uint8_t buf[MLX5_ENCAP_MAX_LEN];
+
+ if (enc_item) {
+ MLX5_ASSERT(!encap_data);
+ if (flow_dv_convert_encap_data
+ (enc_item, buf, &data_size, error) ||
+ flow_hw_encap_item_translate
+ (dev, acts, (action_start + reformat_src)->type,
+ reformat_src, reformat_pos,
+ enc_item, enc_item_m))
+ goto err;
+ encap_data = buf;
+ } else if (encap_data && __flow_hw_act_data_encap_append
+ (priv, acts,
+ (action_start + reformat_src)->type,
+ reformat_src, reformat_pos, 0, 0, data_size)) {
+ goto err;
+ }
+ acts->encap_decap = mlx5_malloc(MLX5_MEM_ZERO,
+ sizeof(*acts->encap_decap) + data_size,
+ 0, SOCKET_ID_ANY);
+ if (!acts->encap_decap)
+ goto err;
+ if (data_size) {
+ acts->encap_decap->data_size = data_size;
+ memcpy(acts->encap_decap->data, encap_data, data_size);
+ }
+ acts->encap_decap->action = mlx5dr_action_create_reformat
+ (priv->dr_ctx, refmt_type,
+ data_size, encap_data,
+ rte_log2_u32(table_attr->nb_flows),
+ mlx5_hw_act_flag[!!attr->group][type]);
+ if (!acts->encap_decap->action)
+ goto err;
+ acts->rule_acts[reformat_pos].action =
+ acts->encap_decap->action;
+ acts->encap_decap_pos = reformat_pos;
+ }
acts->acts_num = i;
return 0;
err:
@@ -722,6 +908,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
struct rte_flow_template_table *table = job->flow->table;
struct mlx5_action_construct_data *act_data;
const struct rte_flow_action *action;
+ const struct rte_flow_action_raw_encap *raw_encap_data;
+ const struct rte_flow_item *enc_item = NULL;
+ uint8_t *buf = job->encap_data;
struct rte_flow_attr attr = {
.ingress = 1,
};
@@ -743,6 +932,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
} else {
attr.ingress = 1;
}
+ if (hw_acts->encap_decap && hw_acts->encap_decap->data_size)
+ memcpy(buf, hw_acts->encap_decap->data,
+ hw_acts->encap_decap->data_size);
LIST_FOREACH(act_data, &hw_acts->act_list, next) {
uint32_t jump_group;
uint32_t tag;
@@ -798,10 +990,38 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
&rule_acts[act_data->action_dst]))
return -1;
break;
+ case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP:
+ enc_item = ((const struct rte_flow_action_vxlan_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP:
+ enc_item = ((const struct rte_flow_action_nvgre_encap *)
+ action->conf)->definition;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ enc_item[act_data->encap.src].spec,
+ act_data->encap.len);
+ break;
+ case RTE_FLOW_ACTION_TYPE_RAW_ENCAP:
+ raw_encap_data =
+ (const struct rte_flow_action_raw_encap *)
+ action->conf;
+ rte_memcpy((void *)&buf[act_data->encap.dst],
+ raw_encap_data->data, act_data->encap.len);
+ MLX5_ASSERT(raw_encap_data->size ==
+ act_data->encap.len);
+ break;
default:
break;
}
}
+ if (hw_acts->encap_decap) {
+ rule_acts[hw_acts->encap_decap_pos].reformat.offset =
+ job->flow->idx - 1;
+ rule_acts[hw_acts->encap_decap_pos].reformat.data = buf;
+ }
return 0;
}
@@ -1865,6 +2085,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
mem_size += (sizeof(struct mlx5_hw_q_job *) +
+ sizeof(uint8_t) * MLX5_ENCAP_MAX_LEN +
sizeof(struct mlx5_hw_q_job)) *
queue_attr[0]->size;
}
@@ -1875,6 +2096,8 @@ flow_hw_configure(struct rte_eth_dev *dev,
goto err;
}
for (i = 0; i < nb_queue; i++) {
+ uint8_t *encap = NULL;
+
priv->hw_q[i].job_idx = queue_attr[i]->size;
priv->hw_q[i].size = queue_attr[i]->size;
if (i == 0)
@@ -1885,8 +2108,11 @@ flow_hw_configure(struct rte_eth_dev *dev,
&job[queue_attr[i - 1]->size];
job = (struct mlx5_hw_q_job *)
&priv->hw_q[i].job[queue_attr[i]->size];
- for (j = 0; j < queue_attr[i]->size; j++)
+ encap = (uint8_t *)&job[queue_attr[i]->size];
+ for (j = 0; j < queue_attr[i]->size; j++) {
+ job[j].encap_data = &encap[j * MLX5_ENCAP_MAX_LEN];
priv->hw_q[i].job[j] = &job[j];
+ }
}
dr_ctx_attr.pd = priv->sh->cdev->pd;
dr_ctx_attr.queues = nb_queue;
--
2.25.1
^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: [PATCH v4 00/14] net/mlx5: add hardware steering
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
` (13 preceding siblings ...)
2022-02-24 13:40 ` [PATCH v4 14/14] net/mlx5: add header reformat action Suanming Mou
@ 2022-02-24 21:12 ` Raslan Darawsheh
14 siblings, 0 replies; 62+ messages in thread
From: Raslan Darawsheh @ 2022-02-24 21:12 UTC (permalink / raw)
To: Suanming Mou, Slava Ovsiienko, Matan Azrad; +Cc: Ori Kam, dev
Hi,
> -----Original Message-----
> From: Suanming Mou <suanmingm@nvidia.com>
> Sent: Thursday, February 24, 2022 3:41 PM
> To: Slava Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad
> <matan@nvidia.com>
> Cc: Raslan Darawsheh <rasland@nvidia.com>; Ori Kam <orika@nvidia.com>;
> dev@dpdk.org
> Subject: [PATCH v4 00/14] net/mlx5: add hardware steering
>
> The Connect-X steering is a lookup hardware mechanism that accesses flow
> tables, matches packets to the rules, and performs specified actions.
> Historically, mlx5 PMD implements several software engines to manage steering
> hardware facility:
>
> - FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
> - SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
>
> However, there are still some disadvantages:
>
> - performance is limited, we should invoke firmware either to
> manage the entire flow, or to handle some internal steering objects
>
> - organizing and preparing flow infrastructure (actions, matchers,
> groups, etc.) on the flow inserting is sure to cause slow flow
> insertion
>
> - security, exposing the low-level steering entries directly to the
> userspace may cause security risks
>
> A new hardware WQE based steering operation with codename "HW Steering"
> is going to be introduced to get rid of the security risks. And it will take
> advantage of the recently new introduced async queue-based rte_flow APIs to
> prepare everything in advance to achieve high insertion rate.
>
> In this new HW steering engine, the original SW steering rte_flow API will not be
> supported in the first implementation, only the new async queue-based flow
> operations is going to be supported. A new steering mode parameter for
> dv_flow_en will be introduced and user will be able to engage the new steering
> engine.
>
> ---
>
> v4:
> v3:
> - rebase to the latest version.
>
> v2:
> - New HW steering low-level abstract code added.
> - commit message improvement.
> - add protection for rte_flow and rte_flow_async callbacks.
> - rebase to rte_flow_async v9.
> - fix some rte_flow error not filled bugs.
>
> Suanming Mou (14):
> net/mlx5: introduce hardware steering operation
> net/mlx5: add HW steering low-level abstract code
> net/mlx5: introduce hardware steering enable routine
> net/mlx5: add port flow configuration
> net/mlx5: add pattern template management
> net/mlx5: add action template management
> net/mlx5: add table management
> net/mlx5: add basic flow queue operation
> net/mlx5: add flow flush function
> net/mlx5: add flow jump action
> net/mlx5: add queue and RSS action
> net/mlx5: add mark action
> net/mlx5: add indirect action
> net/mlx5: add header reformat action
>
> doc/guides/nics/mlx5.rst | 19 +-
> doc/guides/rel_notes/release_22_03.rst | 1 +
> drivers/net/mlx5/linux/mlx5_flow_os.h | 1 +
> drivers/net/mlx5/linux/mlx5_os.c | 22 +-
> drivers/net/mlx5/meson.build | 2 +
> drivers/net/mlx5/mlx5.c | 55 +-
> drivers/net/mlx5/mlx5.h | 66 +-
> drivers/net/mlx5/mlx5_devx.c | 10 +
> drivers/net/mlx5/mlx5_dr.c | 383 ++++
> drivers/net/mlx5/mlx5_dr.h | 456 +++++
> drivers/net/mlx5/mlx5_flow.c | 721 ++++++-
> drivers/net/mlx5/mlx5_flow.h | 287 +++
> drivers/net/mlx5/mlx5_flow_dv.c | 186 +-
> drivers/net/mlx5/mlx5_flow_hw.c | 2337 +++++++++++++++++++++++
> drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
> drivers/net/mlx5/mlx5_rx.h | 9 +-
> drivers/net/mlx5/mlx5_rxq.c | 85 +-
> drivers/net/mlx5/windows/mlx5_flow_os.h | 1 +
> 18 files changed, 4468 insertions(+), 180 deletions(-) create mode 100644
> drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h
> create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
>
> --
> 2.25.1
Series applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code
2022-02-24 13:40 ` [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
@ 2022-02-24 22:57 ` Ferruh Yigit
2022-02-24 23:49 ` Suanming Mou
0 siblings, 1 reply; 62+ messages in thread
From: Ferruh Yigit @ 2022-02-24 22:57 UTC (permalink / raw)
To: Suanming Mou, viacheslavo, matan; +Cc: rasland, orika, dev
On 2/24/2022 1:40 PM, Suanming Mou wrote:
> The HW steering low-level implementation will be added later in another
> patch series. To avoid the linkage issues the abstract stub replacement
> is provided currently.
>
> Signed-off-by: Suanming Mou<suanmingm@nvidia.com>
> Acked-by: Viacheslav Ovsiienko<viacheslavo@nvidia.com>
> ---
> drivers/net/mlx5/meson.build | 1 +
> drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++
> drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++
> 3 files changed, 840 insertions(+)
> create mode 100644 drivers/net/mlx5/mlx5_dr.c
> create mode 100644 drivers/net/mlx5/mlx5_dr.h
>
> diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
> index 39a2b8c523..393b2c97ac 100644
> --- a/drivers/net/mlx5/meson.build
> +++ b/drivers/net/mlx5/meson.build
> @@ -14,6 +14,7 @@ sources = files(
> 'mlx5.c',
> 'mlx5_ethdev.c',
> 'mlx5_flow.c',
> + 'mlx5_dr.c',
> 'mlx5_flow_meter.c',
> 'mlx5_flow_dv.c',
> 'mlx5_flow_hw.c',
gives syntax warning [1], will fix while merging.
[1]
./devtools/check-meson.py
Error parsing drivers/net/mlx5/meson.build:16, got some tabulation
Error: Incorrect indent at drivers/net/mlx5/meson.build:17
^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code
2022-02-24 22:57 ` Ferruh Yigit
@ 2022-02-24 23:49 ` Suanming Mou
0 siblings, 0 replies; 62+ messages in thread
From: Suanming Mou @ 2022-02-24 23:49 UTC (permalink / raw)
To: Ferruh Yigit, Slava Ovsiienko, Matan Azrad; +Cc: Raslan Darawsheh, Ori Kam, dev
Hi,
> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Friday, February 25, 2022 6:58 AM
> To: Suanming Mou <suanmingm@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: Raslan Darawsheh <rasland@nvidia.com>; Ori Kam <orika@nvidia.com>;
> dev@dpdk.org
> Subject: Re: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract
> code
>
> On 2/24/2022 1:40 PM, Suanming Mou wrote:
[snip]
> > @@ -14,6 +14,7 @@ sources = files(
> > 'mlx5.c',
> > 'mlx5_ethdev.c',
> > 'mlx5_flow.c',
> > + 'mlx5_dr.c',
> > 'mlx5_flow_meter.c',
> > 'mlx5_flow_dv.c',
> > 'mlx5_flow_hw.c',
>
> gives syntax warning [1], will fix while merging.
Just checked, yes, seems vim auto indent placed tab instead of space here.
Thank you for the fixing.
>
>
> [1]
> ./devtools/check-meson.py
> Error parsing drivers/net/mlx5/meson.build:16, got some tabulation
> Error: Incorrect indent at drivers/net/mlx5/meson.build:17
^ permalink raw reply [flat|nested] 62+ messages in thread
end of thread, other threads:[~2022-02-24 23:49 UTC | newest]
Thread overview: 62+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-10 16:29 [PATCH 00/13] net/mlx5: add hardware steering Suanming Mou
2022-02-10 16:29 ` [PATCH 01/13] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-10 16:29 ` [PATCH 02/13] net/mlx5: introduce hardware steering enable routine Suanming Mou
2022-02-10 16:29 ` [PATCH 03/13] net/mlx5: add port flow configuration Suanming Mou
2022-02-10 16:29 ` [PATCH 04/13] net/mlx5: add pattern template management Suanming Mou
2022-02-10 16:29 ` [PATCH 05/13] net/mlx5: add action " Suanming Mou
2022-02-10 16:29 ` [PATCH 06/13] net/mlx5: add table management Suanming Mou
2022-02-10 16:29 ` [PATCH 07/13] net/mlx5: add basic flow queue operation Suanming Mou
2022-02-10 16:29 ` [PATCH 08/13] net/mlx5: add flow flush function Suanming Mou
2022-02-10 16:29 ` [PATCH 09/13] net/mlx5: add flow jump action Suanming Mou
2022-02-10 16:29 ` [PATCH 10/13] net/mlx5: add queue and RSS action Suanming Mou
2022-02-10 16:29 ` [PATCH 11/13] net/mlx5: add mark action Suanming Mou
2022-02-10 16:29 ` [PATCH 12/13] net/mlx5: add indirect action Suanming Mou
2022-02-10 16:29 ` [PATCH 13/13] net/mlx5: add header reformat action Suanming Mou
2022-02-22 8:51 ` [PATCH v2 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-22 8:51 ` [PATCH v2 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-22 8:51 ` [PATCH v2 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
2022-02-22 8:51 ` [PATCH v2 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
2022-02-22 8:51 ` [PATCH v2 04/14] net/mlx5: add port flow configuration Suanming Mou
2022-02-22 8:51 ` [PATCH v2 05/14] net/mlx5: add pattern template management Suanming Mou
2022-02-22 8:51 ` [PATCH v2 06/14] net/mlx5: add action " Suanming Mou
2022-02-22 8:51 ` [PATCH v2 07/14] net/mlx5: add table management Suanming Mou
2022-02-22 8:51 ` [PATCH v2 08/14] net/mlx5: add basic flow queue operation Suanming Mou
2022-02-22 8:51 ` [PATCH v2 09/14] net/mlx5: add flow flush function Suanming Mou
2022-02-22 8:51 ` [PATCH v2 10/14] net/mlx5: add flow jump action Suanming Mou
2022-02-22 8:51 ` [PATCH v2 11/14] net/mlx5: add queue and RSS action Suanming Mou
2022-02-22 8:51 ` [PATCH v2 12/14] net/mlx5: add mark action Suanming Mou
2022-02-22 8:51 ` [PATCH v2 13/14] net/mlx5: add indirect action Suanming Mou
2022-02-22 8:51 ` [PATCH v2 14/14] net/mlx5: add header reformat action Suanming Mou
2022-02-24 3:10 ` [PATCH v3 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-24 3:10 ` [PATCH v3 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-24 3:10 ` [PATCH v3 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
2022-02-24 3:10 ` [PATCH v3 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
2022-02-24 3:10 ` [PATCH v3 04/14] net/mlx5: add port flow configuration Suanming Mou
2022-02-24 3:10 ` [PATCH v3 05/14] net/mlx5: add pattern template management Suanming Mou
2022-02-24 3:10 ` [PATCH v3 06/14] net/mlx5: add action " Suanming Mou
2022-02-24 3:10 ` [PATCH v3 07/14] net/mlx5: add table management Suanming Mou
2022-02-24 3:10 ` [PATCH v3 08/14] net/mlx5: add basic flow queue operation Suanming Mou
2022-02-24 3:10 ` [PATCH v3 09/14] net/mlx5: add flow flush function Suanming Mou
2022-02-24 3:10 ` [PATCH v3 10/14] net/mlx5: add flow jump action Suanming Mou
2022-02-24 3:10 ` [PATCH v3 11/14] net/mlx5: add queue and RSS action Suanming Mou
2022-02-24 3:10 ` [PATCH v3 12/14] net/mlx5: add mark action Suanming Mou
2022-02-24 3:10 ` [PATCH v3 13/14] net/mlx5: add indirect action Suanming Mou
2022-02-24 3:10 ` [PATCH v3 14/14] net/mlx5: add header reformat action Suanming Mou
2022-02-24 13:40 ` [PATCH v4 00/14] net/mlx5: add hardware steering Suanming Mou
2022-02-24 13:40 ` [PATCH v4 01/14] net/mlx5: introduce hardware steering operation Suanming Mou
2022-02-24 13:40 ` [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code Suanming Mou
2022-02-24 22:57 ` Ferruh Yigit
2022-02-24 23:49 ` Suanming Mou
2022-02-24 13:40 ` [PATCH v4 03/14] net/mlx5: introduce hardware steering enable routine Suanming Mou
2022-02-24 13:40 ` [PATCH v4 04/14] net/mlx5: add port flow configuration Suanming Mou
2022-02-24 13:40 ` [PATCH v4 05/14] net/mlx5: add pattern template management Suanming Mou
2022-02-24 13:40 ` [PATCH v4 06/14] net/mlx5: add action " Suanming Mou
2022-02-24 13:40 ` [PATCH v4 07/14] net/mlx5: add table management Suanming Mou
2022-02-24 13:40 ` [PATCH v4 08/14] net/mlx5: add basic flow queue operation Suanming Mou
2022-02-24 13:40 ` [PATCH v4 09/14] net/mlx5: add flow flush function Suanming Mou
2022-02-24 13:40 ` [PATCH v4 10/14] net/mlx5: add flow jump action Suanming Mou
2022-02-24 13:40 ` [PATCH v4 11/14] net/mlx5: add queue and RSS action Suanming Mou
2022-02-24 13:40 ` [PATCH v4 12/14] net/mlx5: add mark action Suanming Mou
2022-02-24 13:40 ` [PATCH v4 13/14] net/mlx5: add indirect action Suanming Mou
2022-02-24 13:40 ` [PATCH v4 14/14] net/mlx5: add header reformat action Suanming Mou
2022-02-24 21:12 ` [PATCH v4 00/14] net/mlx5: add hardware steering Raslan Darawsheh
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).