From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 4F522A034C;
	Wed, 21 Dec 2022 10:13:28 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 3F1AB41141;
	Wed, 21 Dec 2022 10:13:28 +0100 (CET)
Received: from mail-vk1-f173.google.com (mail-vk1-f173.google.com
 [209.85.221.173])
 by mails.dpdk.org (Postfix) with ESMTP id 6B94641141
 for <dev@dpdk.org>; Wed, 21 Dec 2022 10:13:26 +0100 (CET)
Received: by mail-vk1-f173.google.com with SMTP id x65so6936655vkg.11
 for <dev@dpdk.org>; Wed, 21 Dec 2022 01:13:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:from:to:cc:subject:date:message-id:reply-to;
 bh=lw4nY1squ7Zr3Xdv0d1CBk3A5ut6zRouVjrkNny3QXc=;
 b=geh/irGWZmpILfEQqOzjruygA9IXTSwN+6xkW+rnyZ+Nwm0xmDt4pkRVEG4DbhYZwG
 KPOclEw9shLh5Cw0melQNJTpYpeB03+1QzsEHbpnKMscmsnPbtHuADwlYLzQY9/DYaeG
 RQPRveUnaV0CUG8CDSndGgNSK7e6N9FKOk9veU7mH3p+Ry7OcbiRoYxx2b2RY/QyfS1N
 CLp9oQzo1cqFF3cvQylhRZhT2wIQkd6f0CPyF5fiF5PXfqIWRGwzwpGynP5DR+eQS2nH
 zx8dAj8Ut6nsLSoN5ZQJyLcURF/l4b826sQED9I/Kb03T3sDDF7eFe97mhH6Hc1jP5hR
 GDKw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
 :reply-to;
 bh=lw4nY1squ7Zr3Xdv0d1CBk3A5ut6zRouVjrkNny3QXc=;
 b=ZgGrpqZ9Gk4SNEniUfeKMPPQJkjXtrBck2lZfC2OvCwE5/eCit6jO1JgHLRO+2zg5v
 AFUz3q2wHKp3MptBDeOOCPzY2cOW8tsqRpd+OyZHultZjFKzXY88baIJEJWJVn82CoH7
 9gf1i+fUW5dXVv73rxTvflrR/G29mV/v2f88HVVDOGJQO4E0pFdViP9/qWWTEadyTrNb
 Y/W1oPXyPD66u1E8nCeBF5vsFI/fD1Cvcewx7z4/yHRXR3UPXXGd9vZqdoqP08RyvEvu
 l1QcxJPiE9cEymcG6YwguDLdxWW6Nx+0G6dmR8Lg3NqjaWWkSpzlaV7PlR07N9T7aLXM
 k5gg==
X-Gm-Message-State: AFqh2kryXrpPDcWkgydw7AWlwqtAwFro3XPcD33U5tDMRY9ord2gZ+zm
 zcGXiTU8MDIwdzpHLrFt66yM4ogP2iJmWUTrbqE=
X-Google-Smtp-Source: AMrXdXulYPPuGVxr8Cgrxa0Sr+u73BNGcJG1mbam4a8waEVSBn5nnYYxtohELQ73nbDBCsp/UvNYJpjac/U10iIbLrg=
X-Received: by 2002:ac5:cb62:0:b0:3d2:1350:bdb0 with SMTP id
 l2-20020ac5cb62000000b003d21350bdb0mr102252vkn.20.1671614004243; Wed, 21 Dec
 2022 01:13:24 -0800 (PST)
MIME-Version: 1.0
References: <20221205215416.7ac53a55@hermes.local>
 <20221221090017.3715030-1-rongweil@nvidia.com>
 <20221221090017.3715030-3-rongweil@nvidia.com>
In-Reply-To: <20221221090017.3715030-3-rongweil@nvidia.com>
From: Jerin Jacob <jerinjacobk@gmail.com>
Date: Wed, 21 Dec 2022 14:42:58 +0530
Message-ID: <CALBAE1PjEzXozMzic3TjuLZ9KOAnPFpuv2XO4Ea6RAsJpKpyug@mail.gmail.com>
Subject: Re: [RFC v3 2/2] ethdev: add API to set process to active or standby
To: Rongwei Liu <rongweil@nvidia.com>
Cc: matan@nvidia.com, viacheslavo@nvidia.com, orika@nvidia.com, 
 thomas@monjalon.net, Ferruh Yigit <ferruh.yigit@amd.com>, 
 Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>, dev@dpdk.org,
 rasland@nvidia.com
Content-Type: text/plain; charset="UTF-8"
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

On Wed, Dec 21, 2022 at 2:31 PM Rongwei Liu <rongweil@nvidia.com> wrote:
>
> Users may want to change the DPDK process to different versions

Different version of DPDK? If there is any ABI change how to support this?

> such as hot upgrade.
> There is a strong requirement to simplify the logic and shorten the
> traffic downtime as much as possible.
>
> This update introduces new rte_eth process role definitions: active
> or standby.
>
> The active role means rules are programmed to HW immediately, and no

Why it has to be specific only to rte_flow rule? If it spedieic to
rte_flow, why it is in rte_eth_process_ name space?

Also, if we are moving the standby, What about the rule whose ABI is
changed between versions?

> behavior changed. This is the default state.
> The standby role means rules are queued in the HW. If no active roles
> alive or back to active, the rules are effective immediately.
>
> Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
> ---
>  doc/guides/nics/mlx5.rst               | 10 ++++
>  doc/guides/rel_notes/release_22_03.rst |  5 ++
>  lib/ethdev/ethdev_driver.h             | 63 ++++++++++++++++++++++++++
>  lib/ethdev/rte_ethdev.c                | 41 +++++++++++++++++
>  lib/ethdev/rte_ethdev.h                |  7 ++-
>  lib/ethdev/version.map                 |  3 ++
>  6 files changed, 128 insertions(+), 1 deletion(-)
>
> diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
> index 51f51259e3..de1fdac0a1 100644
> --- a/doc/guides/nics/mlx5.rst
> +++ b/doc/guides/nics/mlx5.rst
> @@ -2001,3 +2001,13 @@ where:
>  * ``sw_queue_id``: queue index in range [64536, 65535].
>    This range is the highest 1000 numbers.
>  * ``hw_queue_id``: queue index given by HW in queue creation.
> +
> +ethdev set process active or standby
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +User should only program group 0 (fdb_def_rule_en=0) when ``rte_eth_process_set_active``
> +has been called and set to a standby role.
> +Group 0 is shared across different DPDK processes while the other groups are limited
> +to the current process scope.
> +The process can't move from active to standby role if preceding active application's
> +rules are still present and vice versa.
> diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
> index 0923707cb8..6fa48106c4 100644
> --- a/doc/guides/rel_notes/release_22_03.rst
> +++ b/doc/guides/rel_notes/release_22_03.rst
> @@ -207,6 +207,11 @@ API Changes
>  * ethdev: Old public macros and enumeration constants without ``RTE_ETH_`` prefix,
>    which are kept for backward compatibility, are marked as deprecated.
>
> +* ethdev: added a new experimental api:
> +
> +  The new API ``rte_eth_process_set_active()`` was added.
> +  If ``RTE_ETH_CAPA_PROCESS_SET_ROLE`` is not advertised, this new api is not supported.
> +
>  * cryptodev: The asymmetric session handling was modified to use a single
>    mempool object. An API ``rte_cryptodev_asym_session_pool_create`` was added
>    to create a mempool with element size big enough to hold the generic asymmetric
> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
> index 6a550cfc83..3c583bc39d 100644
> --- a/lib/ethdev/ethdev_driver.h
> +++ b/lib/ethdev/ethdev_driver.h
> @@ -179,6 +179,16 @@ struct rte_eth_dev_data {
>         pthread_mutex_t flow_ops_mutex; /**< rte_flow ops mutex */
>  } __rte_cache_aligned;
>
> +/**@{@name Different rte_eth role flag definitions which will be used
> + *  when miagrating DPDK to a different version.
> + */
> +/*
> + * Traffic coming from NIC domain rules will reach
> + * both active and standby processes.
> + */
> +#define RTE_ETH_PROCESS_NIC_DUP_WITH_STANDBY RTE_BIT32(0),
> +/**@}*/
> +
>  /**
>   * @internal
>   * The pool of *rte_eth_dev* structures. The size of the pool
> @@ -1087,6 +1097,22 @@ typedef const uint32_t *(*eth_buffer_split_supported_hdr_ptypes_get_t)(struct rt
>   */
>  typedef int (*eth_dev_priv_dump_t)(struct rte_eth_dev *dev, FILE *file);
>
> +/**
> + * @internal
> + * Set rte_eth process to active or standby role.
> + *
> + * @param dev
> + *   Port (ethdev) handle.
> + * @param active
> + *   Device (role) active or not (standby).
> + * @param flag
> + *   Role specific flag.
> + *
> + * @return
> + *   Negative value on error, 0 on success.
> + */
> +typedef int (*eth_process_set_active_t)(struct rte_eth_dev *dev, bool active, uint32_t flag);
> +
>  /**
>   * @internal Set Rx queue available descriptors threshold.
>   * @see rte_eth_rx_avail_thresh_set()
> @@ -1403,6 +1429,8 @@ struct eth_dev_ops {
>         eth_cman_config_set_t cman_config_set;
>         /** Retrieve congestion management configuration */
>         eth_cman_config_get_t cman_config_get;
> +       /** Set the whole rte_eth process to active or standby role. */
> +       eth_process_set_active_t eth_process_set_active;
>  };
>
>  /**
> @@ -2046,6 +2074,41 @@ struct rte_eth_fdir_conf {
>         struct rte_eth_fdir_flex_conf flex_conf;
>  };
>
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Set the rte_eth process to the active or standby role which affects
> + * the flow rules offloading. It doesn't allow multiple processes to be the
> + * same role unless no offload rules are set.
> + * The active process flow rules are effective immediately while the standby
> + * process rules will be matched (active) when the process becomes active or
> + * when the traffic is not matched by the active process rules.
> + * The active application will always receive traffic while the standby
> + * application will receive traffic when no matching rules are present from
> + * the active application.
> + *
> + * The application is active by default if this API is not called.
> + *
> + * When a process transforms from a standby to a active role, all preceding
> + * flow rules which are queued by hardware will be effective immediately.
> + * Before role transition, all the rules set by the active process should be
> + * flushed first.
> + *
> + * When role flag "RTE_ETH_PROCESS_NIC_DUP_WITH_STANDBY" is set, NIC domain
> + * flow rules are effective immediately even if a process is standby role.
> + *
> + * @param active
> + *   Process active (role) or not (standby).
> + * @param flag
> + *   The role flag.
> + * @return
> + *   - (>=0) Number of rte devices which have been switched successfully.
> + *   - (-EINVAL) if bad parameter.
> + */
> +__rte_experimental
> +int rte_eth_process_set_active(bool active, uint32_t flag);
> +
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
> index 5d5e18db1e..f19da75bfe 100644
> --- a/lib/ethdev/rte_ethdev.c
> +++ b/lib/ethdev/rte_ethdev.c
> @@ -6318,6 +6318,47 @@ rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t *ptypes
>         return j;
>  }
>
> +int rte_eth_process_set_active(bool active, uint32_t flag)
> +{
> +       struct rte_eth_dev_info dev_info = {0};
> +       uint32_t flags[RTE_MAX_ETHPORTS];
> +       struct rte_eth_dev *dev;
> +       uint16_t port_id;
> +       int ret = 0;
> +
> +       /* Check if all devices support. */
> +       RTE_ETH_FOREACH_DEV(port_id) {
> +               dev = &rte_eth_devices[port_id];
> +               if (*dev->dev_ops->dev_infos_get == NULL ||
> +                   *dev->dev_ops->eth_process_set_active == NULL)
> +                       return -ENOTSUP;
> +               if ((*dev->dev_ops->dev_infos_get)(dev, &dev_info))
> +                       return -EINVAL;
> +               if (!(dev_info.dev_capa & RTE_ETH_CAPA_PROCESS_SET_ROLE))
> +                       return -ENOTSUP;
> +       }
> +       RTE_ETH_FOREACH_DEV(port_id) {
> +               dev = &rte_eth_devices[port_id];
> +               if ((*dev->dev_ops->dev_infos_get)(dev, &dev_info))
> +                       goto err;
> +               flags[port_id] = dev_info.eth_process_flag;
> +               if ((*dev->dev_ops->eth_process_set_active)(dev, active, flag) < 0)
> +                       goto err;
> +               ret++;
> +       }
> +       return ret;
> +err:
> +       if (!ret)
> +               return 0;
> +       RTE_ETH_FOREACH_DEV(port_id) {
> +               dev = &rte_eth_devices[port_id];
> +               (*dev->dev_ops->eth_process_set_active)(dev, !active, flags[port_id]);
> +               if (--ret == 0)
> +                       break;
> +       }
> +       return 0;
> +}
> +
>  RTE_LOG_REGISTER_DEFAULT(rte_eth_dev_logtype, INFO);
>
>  RTE_INIT(ethdev_init_telemetry)
> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> index c129ca1eaf..d29f051d6f 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -1606,6 +1606,8 @@ struct rte_eth_conf {
>  #define RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP         RTE_BIT64(3)
>  /** Device supports keeping shared flow objects across restart. */
>  #define RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP RTE_BIT64(4)
> +/**Device supports "rte_eth_process_set_active" callback. */
> +#define RTE_ETH_CAPA_PROCESS_SET_ROLE RTE_BIT64(5)
>  /**@}*/
>
>  /*
> @@ -1777,8 +1779,11 @@ struct rte_eth_dev_info {
>         struct rte_eth_switch_info switch_info;
>         /** Supported error handling mode. */
>         enum rte_eth_err_handle_mode err_handle_mode;
> +       /** Process specific role flag. */
> +       uint32_t eth_process_flag;
>
> -       uint64_t reserved_64s[2]; /**< Reserved for future fields */
> +       uint32_t reserved_32s[1]; /**< Reserved for future fields */
> +       uint64_t reserved_64s[1]; /**< Reserved for future fields */
>         void *reserved_ptrs[2];   /**< Reserved for future fields */
>  };
>
> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
> index 17201fbe0f..a5503f6fde 100644
> --- a/lib/ethdev/version.map
> +++ b/lib/ethdev/version.map
> @@ -298,6 +298,9 @@ EXPERIMENTAL {
>         rte_flow_get_q_aged_flows;
>         rte_mtr_meter_policy_get;
>         rte_mtr_meter_profile_get;
> +
> +       # added in 23.03
> +       rte_eth_process_set_active;
>  };
>
>  INTERNAL {
> --
> 2.27.0
>