From: Rongwei Liu <rongweil@nvidia.com>
To: Jerin Jacob <jerinjacobk@gmail.com>
Cc: Matan Azrad <matan@nvidia.com>,
Slava Ovsiienko <viacheslavo@nvidia.com>,
Ori Kam <orika@nvidia.com>,
"NBU-Contact-Thomas Monjalon (EXTERNAL)" <thomas@monjalon.net>,
Ferruh Yigit <ferruh.yigit@amd.com>,
Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>,
"dev@dpdk.org" <dev@dpdk.org>,
Raslan Darawsheh <rasland@nvidia.com>
Subject: RE: [RFC v3 2/2] ethdev: add API to set process to active or standby
Date: Wed, 21 Dec 2022 09:32:23 +0000 [thread overview]
Message-ID: <BN9PR12MB5273BC5F50F51A82B24DEC4FABEB9@BN9PR12MB5273.namprd12.prod.outlook.com> (raw)
In-Reply-To: <CALBAE1PjEzXozMzic3TjuLZ9KOAnPFpuv2XO4Ea6RAsJpKpyug@mail.gmail.com>
HI Jerin:
BR
Rongwei
> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Wednesday, December 21, 2022 17:13
> To: Rongwei Liu <rongweil@nvidia.com>
> Cc: Matan Azrad <matan@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; NBU-Contact-
> Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; Ferruh Yigit
> <ferruh.yigit@amd.com>; Andrew Rybchenko
> <andrew.rybchenko@oktetlabs.ru>; dev@dpdk.org; Raslan Darawsheh
> <rasland@nvidia.com>
> Subject: Re: [RFC v3 2/2] ethdev: add API to set process to active or standby
>
> External email: Use caution opening links or attachments
>
>
> On Wed, Dec 21, 2022 at 2:31 PM Rongwei Liu <rongweil@nvidia.com> wrote:
> >
> > Users may want to change the DPDK process to different versions
>
> Different version of DPDK? If there is any ABI change how to support this?
>
There is a new member which was introduced into rte_eth_dev_info but it shouldn’t be ABI breaking since using reserved fields.
> > such as hot upgrade.
> > There is a strong requirement to simplify the logic and shorten the
> > traffic downtime as much as possible.
> >
> > This update introduces new rte_eth process role definitions: active or
> > standby.
> >
> > The active role means rules are programmed to HW immediately, and no
>
> Why it has to be specific only to rte_flow rule? If it spedieic to rte_flow, why it
> is in rte_eth_process_ name space?
For now, this design focuses on the flow rule offloading and traffic redirection.
When switching process version, it' important to make sure which application receives and handles the traffic.
The changing should be effective across all probing eth devices, that' why it was put under rte_eth_process_ (for all rte_eth_dev) name space.
>
> Also, if we are moving the standby, What about the rule whose ABI is changed
> between versions?
Like the comments mentioned: " Before role transition, all the rules set by the active process should be flushed first. "
> > behavior changed. This is the default state.
> > The standby role means rules are queued in the HW. If no active roles
> > alive or back to active, the rules are effective immediately.
> >
> > Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
> > ---
> > doc/guides/nics/mlx5.rst | 10 ++++
> > doc/guides/rel_notes/release_22_03.rst | 5 ++
> > lib/ethdev/ethdev_driver.h | 63 ++++++++++++++++++++++++++
> > lib/ethdev/rte_ethdev.c | 41 +++++++++++++++++
> > lib/ethdev/rte_ethdev.h | 7 ++-
> > lib/ethdev/version.map | 3 ++
> > 6 files changed, 128 insertions(+), 1 deletion(-)
> >
> > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index
> > 51f51259e3..de1fdac0a1 100644
> > --- a/doc/guides/nics/mlx5.rst
> > +++ b/doc/guides/nics/mlx5.rst
> > @@ -2001,3 +2001,13 @@ where:
> > * ``sw_queue_id``: queue index in range [64536, 65535].
> > This range is the highest 1000 numbers.
> > * ``hw_queue_id``: queue index given by HW in queue creation.
> > +
> > +ethdev set process active or standby
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +User should only program group 0 (fdb_def_rule_en=0) when
> > +``rte_eth_process_set_active`` has been called and set to a standby role.
> > +Group 0 is shared across different DPDK processes while the other
> > +groups are limited to the current process scope.
> > +The process can't move from active to standby role if preceding
> > +active application's rules are still present and vice versa.
> > diff --git a/doc/guides/rel_notes/release_22_03.rst
> > b/doc/guides/rel_notes/release_22_03.rst
> > index 0923707cb8..6fa48106c4 100644
> > --- a/doc/guides/rel_notes/release_22_03.rst
> > +++ b/doc/guides/rel_notes/release_22_03.rst
> > @@ -207,6 +207,11 @@ API Changes
> > * ethdev: Old public macros and enumeration constants without
> ``RTE_ETH_`` prefix,
> > which are kept for backward compatibility, are marked as deprecated.
> >
> > +* ethdev: added a new experimental api:
> > +
> > + The new API ``rte_eth_process_set_active()`` was added.
> > + If ``RTE_ETH_CAPA_PROCESS_SET_ROLE`` is not advertised, this new api
> is not supported.
> > +
> > * cryptodev: The asymmetric session handling was modified to use a single
> > mempool object. An API ``rte_cryptodev_asym_session_pool_create`` was
> added
> > to create a mempool with element size big enough to hold the
> > generic asymmetric diff --git a/lib/ethdev/ethdev_driver.h
> > b/lib/ethdev/ethdev_driver.h index 6a550cfc83..3c583bc39d 100644
> > --- a/lib/ethdev/ethdev_driver.h
> > +++ b/lib/ethdev/ethdev_driver.h
> > @@ -179,6 +179,16 @@ struct rte_eth_dev_data {
> > pthread_mutex_t flow_ops_mutex; /**< rte_flow ops mutex */ }
> > __rte_cache_aligned;
> >
> > +/**@{@name Different rte_eth role flag definitions which will be used
> > + * when miagrating DPDK to a different version.
> > + */
> > +/*
> > + * Traffic coming from NIC domain rules will reach
> > + * both active and standby processes.
> > + */
> > +#define RTE_ETH_PROCESS_NIC_DUP_WITH_STANDBY RTE_BIT32(0),
> /**@}*/
> > +
> > /**
> > * @internal
> > * The pool of *rte_eth_dev* structures. The size of the pool @@
> > -1087,6 +1097,22 @@ typedef const uint32_t
> *(*eth_buffer_split_supported_hdr_ptypes_get_t)(struct rt
> > */
> > typedef int (*eth_dev_priv_dump_t)(struct rte_eth_dev *dev, FILE
> > *file);
> >
> > +/**
> > + * @internal
> > + * Set rte_eth process to active or standby role.
> > + *
> > + * @param dev
> > + * Port (ethdev) handle.
> > + * @param active
> > + * Device (role) active or not (standby).
> > + * @param flag
> > + * Role specific flag.
> > + *
> > + * @return
> > + * Negative value on error, 0 on success.
> > + */
> > +typedef int (*eth_process_set_active_t)(struct rte_eth_dev *dev, bool
> > +active, uint32_t flag);
> > +
> > /**
> > * @internal Set Rx queue available descriptors threshold.
> > * @see rte_eth_rx_avail_thresh_set() @@ -1403,6 +1429,8 @@ struct
> > eth_dev_ops {
> > eth_cman_config_set_t cman_config_set;
> > /** Retrieve congestion management configuration */
> > eth_cman_config_get_t cman_config_get;
> > + /** Set the whole rte_eth process to active or standby role. */
> > + eth_process_set_active_t eth_process_set_active;
> > };
> >
> > /**
> > @@ -2046,6 +2074,41 @@ struct rte_eth_fdir_conf {
> > struct rte_eth_fdir_flex_conf flex_conf; };
> >
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice
> > + *
> > + * Set the rte_eth process to the active or standby role which
> > +affects
> > + * the flow rules offloading. It doesn't allow multiple processes to
> > +be the
> > + * same role unless no offload rules are set.
> > + * The active process flow rules are effective immediately while the
> > +standby
> > + * process rules will be matched (active) when the process becomes
> > +active or
> > + * when the traffic is not matched by the active process rules.
> > + * The active application will always receive traffic while the
> > +standby
> > + * application will receive traffic when no matching rules are
> > +present from
> > + * the active application.
> > + *
> > + * The application is active by default if this API is not called.
> > + *
> > + * When a process transforms from a standby to a active role, all
> > +preceding
> > + * flow rules which are queued by hardware will be effective immediately.
> > + * Before role transition, all the rules set by the active process
> > +should be
> > + * flushed first.
> > + *
> > + * When role flag "RTE_ETH_PROCESS_NIC_DUP_WITH_STANDBY" is set,
> NIC
> > +domain
> > + * flow rules are effective immediately even if a process is standby role.
> > + *
> > + * @param active
> > + * Process active (role) or not (standby).
> > + * @param flag
> > + * The role flag.
> > + * @return
> > + * - (>=0) Number of rte devices which have been switched successfully.
> > + * - (-EINVAL) if bad parameter.
> > + */
> > +__rte_experimental
> > +int rte_eth_process_set_active(bool active, uint32_t flag);
> > +
> > #ifdef __cplusplus
> > }
> > #endif
> > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index
> > 5d5e18db1e..f19da75bfe 100644
> > --- a/lib/ethdev/rte_ethdev.c
> > +++ b/lib/ethdev/rte_ethdev.c
> > @@ -6318,6 +6318,47 @@
> rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t
> *ptypes
> > return j;
> > }
> >
> > +int rte_eth_process_set_active(bool active, uint32_t flag) {
> > + struct rte_eth_dev_info dev_info = {0};
> > + uint32_t flags[RTE_MAX_ETHPORTS];
> > + struct rte_eth_dev *dev;
> > + uint16_t port_id;
> > + int ret = 0;
> > +
> > + /* Check if all devices support. */
> > + RTE_ETH_FOREACH_DEV(port_id) {
> > + dev = &rte_eth_devices[port_id];
> > + if (*dev->dev_ops->dev_infos_get == NULL ||
> > + *dev->dev_ops->eth_process_set_active == NULL)
> > + return -ENOTSUP;
> > + if ((*dev->dev_ops->dev_infos_get)(dev, &dev_info))
> > + return -EINVAL;
> > + if (!(dev_info.dev_capa & RTE_ETH_CAPA_PROCESS_SET_ROLE))
> > + return -ENOTSUP;
> > + }
> > + RTE_ETH_FOREACH_DEV(port_id) {
> > + dev = &rte_eth_devices[port_id];
> > + if ((*dev->dev_ops->dev_infos_get)(dev, &dev_info))
> > + goto err;
> > + flags[port_id] = dev_info.eth_process_flag;
> > + if ((*dev->dev_ops->eth_process_set_active)(dev, active, flag) < 0)
> > + goto err;
> > + ret++;
> > + }
> > + return ret;
> > +err:
> > + if (!ret)
> > + return 0;
> > + RTE_ETH_FOREACH_DEV(port_id) {
> > + dev = &rte_eth_devices[port_id];
> > + (*dev->dev_ops->eth_process_set_active)(dev, !active,
> flags[port_id]);
> > + if (--ret == 0)
> > + break;
> > + }
> > + return 0;
> > +}
> > +
> > RTE_LOG_REGISTER_DEFAULT(rte_eth_dev_logtype, INFO);
> >
> > RTE_INIT(ethdev_init_telemetry)
> > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index
> > c129ca1eaf..d29f051d6f 100644
> > --- a/lib/ethdev/rte_ethdev.h
> > +++ b/lib/ethdev/rte_ethdev.h
> > @@ -1606,6 +1606,8 @@ struct rte_eth_conf {
> > #define RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP RTE_BIT64(3)
> > /** Device supports keeping shared flow objects across restart. */
> > #define RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP RTE_BIT64(4)
> > +/**Device supports "rte_eth_process_set_active" callback. */ #define
> > +RTE_ETH_CAPA_PROCESS_SET_ROLE RTE_BIT64(5)
> > /**@}*/
> >
> > /*
> > @@ -1777,8 +1779,11 @@ struct rte_eth_dev_info {
> > struct rte_eth_switch_info switch_info;
> > /** Supported error handling mode. */
> > enum rte_eth_err_handle_mode err_handle_mode;
> > + /** Process specific role flag. */
> > + uint32_t eth_process_flag;
> >
> > - uint64_t reserved_64s[2]; /**< Reserved for future fields */
> > + uint32_t reserved_32s[1]; /**< Reserved for future fields */
> > + uint64_t reserved_64s[1]; /**< Reserved for future fields */
> > void *reserved_ptrs[2]; /**< Reserved for future fields */
> > };
> >
> > diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map index
> > 17201fbe0f..a5503f6fde 100644
> > --- a/lib/ethdev/version.map
> > +++ b/lib/ethdev/version.map
> > @@ -298,6 +298,9 @@ EXPERIMENTAL {
> > rte_flow_get_q_aged_flows;
> > rte_mtr_meter_policy_get;
> > rte_mtr_meter_profile_get;
> > +
> > + # added in 23.03
> > + rte_eth_process_set_active;
> > };
> >
> > INTERNAL {
> > --
> > 2.27.0
> >
next prev parent reply other threads:[~2022-12-21 9:32 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-01 8:20 [RFC 0/2] add API to set process to primary or secondary Rongwei Liu
2022-12-01 8:20 ` [RFC 1/2] ethdev: add group description Rongwei Liu
2022-12-01 8:20 ` [RFC 2/2] ethdev: add API to set process to primary or secondary Rongwei Liu
2022-12-01 15:10 ` Stephen Hemminger
2022-12-02 3:27 ` Rongwei Liu
2022-12-05 16:08 ` Stephen Hemminger
2022-12-06 3:47 ` Rongwei Liu
2022-12-06 5:54 ` Stephen Hemminger
2022-12-21 9:00 ` [RFC v3 0/2] add API to set process to active or standby Rongwei Liu
2022-12-21 9:00 ` [RFC v3 1/2] ethdev: add group description Rongwei Liu
2023-01-18 15:44 ` [PATCH v4 0/3] add API for live migration Rongwei Liu
2023-01-18 15:44 ` [PATCH v4 1/3] ethdev: add flow rule group description Rongwei Liu
2023-01-31 11:53 ` Ori Kam
2023-02-06 12:15 ` Rongwei Liu
2023-02-07 2:57 ` [PATCH v5] " Rongwei Liu
2023-02-08 20:28 ` Ferruh Yigit
2023-02-09 2:06 ` Rongwei Liu
2023-02-09 7:32 ` [PATCH v6] " Rongwei Liu
2023-02-09 8:01 ` Ori Kam
2023-02-09 11:26 ` Ferruh Yigit
2023-01-18 15:44 ` [PATCH v4 2/3] ethdev: add standby state for live migration Rongwei Liu
2023-01-31 13:50 ` Ori Kam
2023-01-31 18:14 ` Jerin Jacob
2023-01-31 22:55 ` Thomas Monjalon
2023-02-01 7:32 ` Andrew Rybchenko
2023-02-01 8:31 ` Thomas Monjalon
2023-02-01 8:40 ` Jerin Jacob
2023-02-01 8:46 ` Thomas Monjalon
2023-02-02 10:23 ` Rongwei Liu
2023-02-01 7:52 ` Andrew Rybchenko
2023-02-01 8:27 ` Thomas Monjalon
2023-02-01 8:40 ` Andrew Rybchenko
2023-01-18 15:44 ` [PATCH v4 3/3] ethdev: add standby flags " Rongwei Liu
2023-01-23 13:20 ` Jerin Jacob
2023-01-30 2:47 ` Rongwei Liu
2023-01-30 17:10 ` Jerin Jacob
2023-01-31 2:53 ` Rongwei Liu
2023-01-31 8:45 ` Jerin Jacob
2023-01-31 9:01 ` Rongwei Liu
2023-01-31 14:37 ` Jerin Jacob
2023-01-31 14:45 ` Ori Kam
2023-01-31 17:50 ` Thomas Monjalon
2023-01-31 18:10 ` Jerin Jacob
2022-12-21 9:00 ` [RFC v3 2/2] ethdev: add API to set process to active or standby Rongwei Liu
2022-12-21 9:12 ` Jerin Jacob
2022-12-21 9:32 ` Rongwei Liu [this message]
2022-12-21 10:59 ` Jerin Jacob
2022-12-21 12:05 ` Rongwei Liu
2022-12-21 12:44 ` Jerin Jacob
2022-12-21 12:50 ` Rongwei Liu
2022-12-21 13:12 ` Jerin Jacob
2022-12-21 14:33 ` Rongwei Liu
2022-12-26 16:44 ` Ori Kam
2023-01-15 22:46 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=BN9PR12MB5273BC5F50F51A82B24DEC4FABEB9@BN9PR12MB5273.namprd12.prod.outlook.com \
--to=rongweil@nvidia.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@amd.com \
--cc=jerinjacobk@gmail.com \
--cc=matan@nvidia.com \
--cc=orika@nvidia.com \
--cc=rasland@nvidia.com \
--cc=thomas@monjalon.net \
--cc=viacheslavo@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).