[RFC 2/2] ethdev: add API to set process to primary or secondary

[Rongwei Liu <rongweil@nvidia.com>]

Users may want to change the DPDK process to different versions
such as hot upgrade.
There is a strong requirement to simplify the logic and shorten the
traffic downtime as much as possible.

This update introduces new rte_eth process role definitions: primary
or secondary.

The primary role means rules are programmed to HW immediately, and no
behavior changed. This is the default state.
The secondary role means rules are queued in the HW. If no primary roles
alive or back to primary, the rules are effective immediately.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
---
 doc/guides/nics/mlx5.rst   | 10 ++++++
 lib/ethdev/ethdev_driver.h | 63 ++++++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.c    | 20 ++++++++++++
 lib/ethdev/version.map     |  3 ++
 4 files changed, 96 insertions(+)

diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 51f51259e3..ea78e14b80 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -2001,3 +2001,13 @@ where:
 * ``sw_queue_id``: queue index in range [64536, 65535].
   This range is the highest 1000 numbers.
 * ``hw_queue_id``: queue index given by HW in queue creation.
+
+ethdev set process primary or secondary
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+User should only program group 0 (fdb_def_rule_en=0) when ``rte_eth_process_set_primary``
+has been called and set to a secondary role.
+Group 0 is shared across different DPDK processes while the other groups are limited
+to the current process scope.
+The process can't move from primary to secondary role if preceding primary application's
+rules are still present and vice versa.
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index 6a550cfc83..67b53840c8 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -179,6 +179,16 @@ struct rte_eth_dev_data {
 	pthread_mutex_t flow_ops_mutex; /**< rte_flow ops mutex */
 } __rte_cache_aligned;
 
+/**@{@name Different rte_eth role flag definitions which will be used
+ *  when miagrating DPDK to a different version.
+ */
+/*
+ * Traffic coming from NIC domain rules will reach
+ * both primary and secondary processes.
+ */
+#define RTE_ETH_PROCESS_NIC_DUP_WITH_SECONDARY RTE_BIT32(0),
+/**@}*/
+
 /**
  * @internal
  * The pool of *rte_eth_dev* structures. The size of the pool
@@ -1087,6 +1097,22 @@ typedef const uint32_t *(*eth_buffer_split_supported_hdr_ptypes_get_t)(struct rt
  */
 typedef int (*eth_dev_priv_dump_t)(struct rte_eth_dev *dev, FILE *file);
 
+/**
+ * @internal
+ * Set rte_eth process to primary or secondary role.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param active
+ *   Device (role) primary or not (secondary).
+ * @param flag
+ *   Role specific flag.
+ *
+ * @return
+ *   Negative value on error, 0 on success.
+ */
+typedef int (*eth_process_set_primary_t)(struct rte_eth_dev *dev, bool primary, uint32_t flag);
+
 /**
  * @internal Set Rx queue available descriptors threshold.
  * @see rte_eth_rx_avail_thresh_set()
@@ -1403,6 +1429,8 @@ struct eth_dev_ops {
 	eth_cman_config_set_t cman_config_set;
 	/** Retrieve congestion management configuration */
 	eth_cman_config_get_t cman_config_get;
+	/** Set the whole rte_eth process to primary or secondary role. */
+	eth_process_set_primary_t eth_process_set_primary;
 };
 
 /**
@@ -2046,6 +2074,41 @@ struct rte_eth_fdir_conf {
 	struct rte_eth_fdir_flex_conf flex_conf;
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Set the rte_eth process to the primary or secondary role which affects
+ * the flow rules offloading. It doesn't allow multiple processes to be the
+ * same role unless no offload rules are set.
+ * The primary process's flow rules are effective immediately while the secondary
+ * process's rules will be queued in hardware until it becomes primary or no
+ * primary process is alive.
+ * The primary application will always receive traffic while the secondary
+ * application will receive traffic when no matching rules are present from
+ * the primary application.
+ *
+ * The application is primary by default if this API is not called.
+ *
+ * When a process transforms from a secondary to a primary role, all preceding
+ * flow rules which are queued by hardware will be effective immediately.
+ * Before role transition, all the rules set by the primary process should be
+ * flushed first.
+ *
+ * When role flag "RTE_ETH_PROCESS_NIC_DUP_WITH_SECONDARY" is set, NIC domain
+ * flow rules are effective immediately even if a process is secondary.
+ *
+ * @param active
+ *   Process primary (role) or not (secondary).
+ * @param flag
+ *   The role flag.
+ * @return
+ *   - (>=0) Number of rte devices which have been switched successfully.
+ *   - (-EINVAL) if bad parameter.
+ */
+__rte_experimental
+int rte_eth_process_set_primary(bool primary, uint32_t flag);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 5d5e18db1e..251908b4e3 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -6318,6 +6318,26 @@ rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t *ptypes
 	return j;
 }
 
+int rte_eth_process_set_primary(bool primary, uint32_t flag)
+{
+	struct rte_eth_dev *dev;
+	uint16_t port_id;
+	int ret, val;
+
+	ret = 0;
+	RTE_ETH_FOREACH_DEV(port_id) {
+		dev = &rte_eth_devices[port_id];
+		if (*dev->dev_ops->eth_process_set_primary == NULL)
+			return -ENOTSUP;
+		val = (*dev->dev_ops->eth_process_set_primary)(dev, primary, flag);
+
+		if (val < 0)
+			return -EINVAL;
+		ret++;
+	}
+	return ret;
+}
+
 RTE_LOG_REGISTER_DEFAULT(rte_eth_dev_logtype, INFO);
 
 RTE_INIT(ethdev_init_telemetry)
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index 17201fbe0f..1823869e73 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -298,6 +298,9 @@ EXPERIMENTAL {
 	rte_flow_get_q_aged_flows;
 	rte_mtr_meter_policy_get;
 	rte_mtr_meter_profile_get;
+
+	# added in 23.03
+	rte_eth_process_set_primary;
 };
 
 INTERNAL {
-- 
2.27.0