[dpdk-dev] [RFC] ethdev: flow counters batch query

[dpdk-dev] [RFC] ethdev: flow counters batch query

[viacheslavo]

There is a demand to perform query operations on the set of counters
generally belonging to the different RTE flows. The counter queries
is the very effective way for application to know in detail what
is going on into the network and to take some actions basing on this
knowledge. As example, the modern vSwitch implementation supporting
full hardware offload may need a huge number of counters and query
these ones periodically for configuration aging check purposes.

For some devices the counter query can be very expensive operation
and can take a significant amount of time - often it involves the
calls of kernel mode components and performing requests to hardware.
If application wants to query multiple counters, belonging to the
different RTE flows, it has to execute queries in one-by-one fashion
due to the current RTE API implementation that allows to query
counters belonging to the same flow only. The proposed RTE API
extension introduces the compatible and more performance way
to allow applications to query the counter values as a batch.

At the creation the counter can be optionally assigned with the
new introduced 'group_id' identifier (batch identifier). All
counters with the same group_id form the new introduced entity
- 'counter batch'. The group_id specifies the unique batch
within device with given port_id. The same group_id may specify
the different batches on different devices.

The new method 'rte_flow_query_update()' is proposed to perform the
counter batch query from the device as a single call, this eliminates
a much of overhead involved with multiple quieries for the counters
belonging to the different flows. The rte_flow_query_update() fetches
the actual counter values from the device using underlying software
and/or hardware and stores obtained data into the counter objects
within PMD.

The application can get the stored data by invoking the existing
rte_flow_query() method with specifying the new introduced flag
'read_cached'. Such approach is compatible with the current
implementation, improves the performance and requires the minor
changes from applications.

Let's assume we have an array of flows and attached counters.
If application wants to read them it does something like that:

foreach(flow) {                                 // compatible mode
        rte_flow_query(flow, flow_counters[]);  // no read_cached set
}                                               // doing as previously

With counter batch query implemented application can do the following:

                                                // new query mode
rte_flow_query_update(group_id of interest);    // actual query here
foreach(flow) {                                 // as single call
        rte_flow_query(flow, flow_counters[]);  // read_cached flag set
}                                               // read stored data
                                                // no underlying calls

For some devices implementation of rte_flow_query_update() may require
a lot of preparations before performing the actual query. If batch is
permanent and assumed not to be changed frequently the preparations
can be cached internally by implementation. By setting the
RTE_FLOW_QUERY_FLAG_PERMANENT flag application gives the hint to PMD
that batch is assumed to be long-term and allows to optimize the
succesive calls of rte_flow_query_update() for the same group_id
on given device.

If permanent batch is subject to change (occurs an adding or removing
the counter with specified batch id) the PMD should free all
resources, internally allocated for batch query optimization.

If RTE_FLOW_QUERY_FLAG_PERMANENT is not set, rte_flow_query_update()
should free the resources allocated (including ones done in previous
calls, if any) for batch query optimization with given group_id.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 doc/guides/prog_guide/rte_flow.rst | 76 ++++++++++++++++++++++++++++----------
 lib/librte_ethdev/rte_flow.h       | 59 ++++++++++++++++++++++++++++-
 2 files changed, 113 insertions(+), 22 deletions(-)

diff --git a/doc/guides/prog_guide/rte_flow.rst b/doc/guides/prog_guide/rte_flow.rst
index b305a72..84b3b67 100644
--- a/doc/guides/prog_guide/rte_flow.rst
+++ b/doc/guides/prog_guide/rte_flow.rst
@@ -1500,6 +1500,10 @@ action must specify a unique id.
 Counters can be retrieved and reset through ``rte_flow_query()``, see
 ``struct rte_flow_query_count``.
 
+Counters can be assigned with group_id, all counters with matched group_id
+on the same port are grouped into batch and can be queried from device
+using the single call of rte_flow_query_update()
+
 The shared flag indicates whether the counter is unique to the flow rule the
 action is specified with, or whether it is a shared counter.
 
@@ -1515,13 +1519,15 @@ to all ports within that switch domain.
 
 .. table:: COUNT
 
-   +------------+---------------------+
-   | Field      | Value               |
-   +============+=====================+
-   | ``shared`` | shared counter flag |
-   +------------+---------------------+
-   | ``id``     | counter id          |
-   +------------+---------------------+
+   +--------------+----------------------+
+   | Field        | Value                |
+   +==============+======================+
+   | ``shared``   | shared counter flag  |
+   +--------------+----------------------+
+   | ``id``       | counter id           |
+   +--------------+----------------------+
+   | ``group_id`` | batch id             |
+   +--------------+----------------------+
 
 Query structure to retrieve and reset flow rule counters:
 
@@ -1529,19 +1535,21 @@ Query structure to retrieve and reset flow rule counters:
 
 .. table:: COUNT query
 
-   +---------------+-----+-----------------------------------+
-   | Field         | I/O | Value                             |
-   +===============+=====+===================================+
-   | ``reset``     | in  | reset counter after query         |
-   +---------------+-----+-----------------------------------+
-   | ``hits_set``  | out | ``hits`` field is set             |
-   +---------------+-----+-----------------------------------+
-   | ``bytes_set`` | out | ``bytes`` field is set            |
-   +---------------+-----+-----------------------------------+
-   | ``hits``      | out | number of hits for this rule      |
-   +---------------+-----+-----------------------------------+
-   | ``bytes``     | out | number of bytes through this rule |
-   +---------------+-----+-----------------------------------+
+   +-----------------+-----+-------------------------------------+
+   | Field           | I/O | Value                               |
+   +=================+=====+=====================================+
+   | ``reset``       | in  | reset counter after query           |
+   +-----------------+-----+-------------------------------------+
+   | ``hits_set``    | out | ``hits`` field is set               |
+   +-----------------+-----+-------------------------------------+
+   | ``bytes_set``   | out | ``bytes`` field is set              |
+   +-----------------+-----+-------------------------------------+
+   | ``read_cached`` | in  | read cached data instead of device  |
+   +-----------------+-----+-------------------------------------+
+   | ``hits``        | out | number of hits for this rule        |
+   +-----------------+-----+-------------------------------------+
+   | ``bytes``       | out | number of bytes through this rule   |
+   +-----------------+-----+-------------------------------------+
 
 Action: ``RSS``
 ^^^^^^^^^^^^^^^
@@ -2288,6 +2296,34 @@ Return values:
 
 - 0 on success, a negative errno value otherwise and ``rte_errno`` is set.
 
+Batch Query
+~~~~~~~~~~~
+
+Query a batch of existing flow rules.
+
+This function allows retrieving flow-specific data such as counters
+belonging to the different flows on the given port in single batch
+query call.
+
+.. code-block:: c
+
+   int
+   rte_flow_query_update(uint16_t port_id,
+		         uint32_t group_id,
+		         uint32_t flags);
+
+Arguments:
+
+- ``port_id``: port identifier of Ethernet device.
+- ``group_id``: batch to query, specifies the group of actions.
+- ``flags``: can be combination of ``RTE_FLOW_QUERY_FLAG_RESET``
+  and ``RTE_FLOW_QUERY_FLAG_PERMANENT`` values
+
+Return values:
+
+- 0 on success, a negative errno value otherwise and ``rte_errno`` is set.
+
+
 Isolated mode
 -------------
 
diff --git a/lib/librte_ethdev/rte_flow.h b/lib/librte_ethdev/rte_flow.h
index f8ba71c..0cbc8fa 100644
--- a/lib/librte_ethdev/rte_flow.h
+++ b/lib/librte_ethdev/rte_flow.h
@@ -1561,10 +1561,14 @@ struct rte_flow_action_queue {
  * Counters can be retrieved and reset through ``rte_flow_query()``, see
  * ``struct rte_flow_query_count``.
  *
+ * Counters can be assigned with group_id, all counters with matched group_id
+ * on the same port are grouped into batch and can be queried from device
+ * using the single call of rte_flow_query_update()
+ *
  * The shared flag indicates whether the counter is unique to the flow rule the
  * action is specified with, or whether it is a shared counter.
  *
- * For a count action with the shared flag set, then then a global device
+ * For a count action with the shared flag set, then a global device
  * namespace is assumed for the counter id, so that any matched flow rules using
  * a count action with the same counter id on the same port will contribute to
  * that counter.
@@ -1576,6 +1580,7 @@ struct rte_flow_action_count {
 	uint32_t shared:1; /**< Share counter ID with other flow rules. */
 	uint32_t reserved:31; /**< Reserved, must be zero. */
 	uint32_t id; /**< Counter ID. */
+	uint32_t group_id; /**< ID of batch that counter belongs to */
 };
 
 /**
@@ -1587,7 +1592,8 @@ struct rte_flow_query_count {
 	uint32_t reset:1; /**< Reset counters after query [in]. */
 	uint32_t hits_set:1; /**< hits field is set [out]. */
 	uint32_t bytes_set:1; /**< bytes field is set [out]. */
-	uint32_t reserved:29; /**< Reserved, must be zero [in, out]. */
+	uint32_t read_cached:1; /**< read stored data instead of device [in]. */
+	uint32_t reserved:28; /**< Reserved, must be zero [in, out]. */
 	uint64_t hits; /**< Number of hits for this rule [out]. */
 	uint64_t bytes; /**< Number of bytes through this rule [out]. */
 };
@@ -2094,6 +2100,55 @@ struct rte_flow *
 	       struct rte_flow_error *error);
 
 /**
+ * Query a batch of existing flow rules.
+ *
+ * This function allows retrieving flow-specific data such as counters
+ * belonging to the different flows on the given port in single batch
+ * query call.
+ *
+ * \see RTE_FLOW_ACTION_TYPE_COUNT
+ *
+ * @param port_id
+ *   Port identifier of Ethernet device.
+ * @param group_id
+ *   Batch identifier, specifies the group of actions to be queried.
+ * @param flags
+ *   RTE_FLOW_QUERY_FLAG_RESET
+ *   RTE_FLOW_QUERY_FLAG_PERMANENT
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+rte_flow_query_update(uint16_t port_id,
+		      uint32_t group_id,
+		      uint32_t flags);
+
+#define RTE_FLOW_QUERY_FLAG_RESET (1 << 0)
+/**< Reset counters after query */
+/**
+ * For some devices implementation of rte_flow_query_update() may require
+ * a lot of preparations before performing the actual query. If batch is
+ * permanent and assumed not to be changed frequently the preparations
+ * can be cached internally by implementation. By setting the
+ * RTE_FLOW_QUERY_FLAG_PERMANENT flag application gives the hint to PMD
+ * that batch is assumed to be long-term and allows to optimize the
+ * succesive calls of rte_flow_query_update() for the same group_id
+ * on given device.
+ *
+ * If permanent batch is subject to change (occurs an adding or
+ * removing the action with specified batch id) the PMD should free all
+ * resources, internally allocated for batch query optimization
+ *
+ * If RTE_FLOW_QUERY_FLAG_PERMANENT is not set, rte_flow_query_update()
+ * should free the resources (if any) early allocated for batch query
+ * optimization with given group_id.
+ *
+ */
+#define RTE_FLOW_QUERY_FLAG_PERMANENT (1 << 1)
+/**< Assume batch permanent */
+
+/**
  * Restrict ingress traffic to the defined flow rules.
  *
  * Isolated mode guarantees that all ingress traffic comes from defined flow
-- 
1.8.3.1

Re: [dpdk-dev] [RFC] ethdev: flow counters batch query

[Slava Ovsiienko]

Adding relevant maintainers.

> -----Original Message-----
> From: Slava Ovsiienko
> Sent: Monday, August 20, 2018 14:39
> To: dev@dpdk.org
> Cc: Shahaf Shuler <shahafs@mellanox.com>; Slava Ovsiienko
> <viacheslavo@mellanox.com>
> Subject: [RFC] ethdev: flow counters batch query
> 
> There is a demand to perform query operations on the set of counters
> generally belonging to the different RTE flows. The counter queries is the
> very effective way for application to know in detail what is going on into the
> network and to take some actions basing on this knowledge. As example, the
> modern vSwitch implementation supporting full hardware offload may need
> a huge number of counters and query these ones periodically for
> configuration aging check purposes.
> 
> For some devices the counter query can be very expensive operation and
> can take a significant amount of time - often it involves the calls of kernel
> mode components and performing requests to hardware.
> If application wants to query multiple counters, belonging to the different
> RTE flows, it has to execute queries in one-by-one fashion due to the current
> RTE API implementation that allows to query counters belonging to the same
> flow only. The proposed RTE API extension introduces the compatible and
> more performance way to allow applications to query the counter values as a
> batch.
> 
> At the creation the counter can be optionally assigned with the new
> introduced 'group_id' identifier (batch identifier). All counters with the same
> group_id form the new introduced entity
> - 'counter batch'. The group_id specifies the unique batch within device with
> given port_id. The same group_id may specify the different batches on
> different devices.
> 
> The new method 'rte_flow_query_update()' is proposed to perform the
> counter batch query from the device as a single call, this eliminates a much of
> overhead involved with multiple quieries for the counters belonging to the
> different flows. The rte_flow_query_update() fetches the actual counter
> values from the device using underlying software and/or hardware and
> stores obtained data into the counter objects within PMD.
> 
> The application can get the stored data by invoking the existing
> rte_flow_query() method with specifying the new introduced flag
> 'read_cached'. Such approach is compatible with the current
> implementation, improves the performance and requires the minor changes
> from applications.
> 
> Let's assume we have an array of flows and attached counters.
> If application wants to read them it does something like that:
> 
> foreach(flow) {                                 // compatible mode
>         rte_flow_query(flow, flow_counters[]);  // no read_cached set
> }                                               // doing as previously
> 
> With counter batch query implemented application can do the following:
> 
>                                                 // new query mode
> rte_flow_query_update(group_id of interest);    // actual query here
> foreach(flow) {                                 // as single call
>         rte_flow_query(flow, flow_counters[]);  // read_cached flag set
> }                                               // read stored data
>                                                 // no underlying calls
> 
> For some devices implementation of rte_flow_query_update() may require
> a lot of preparations before performing the actual query. If batch is
> permanent and assumed not to be changed frequently the preparations can
> be cached internally by implementation. By setting the
> RTE_FLOW_QUERY_FLAG_PERMANENT flag application gives the hint to
> PMD that batch is assumed to be long-term and allows to optimize the
> succesive calls of rte_flow_query_update() for the same group_id on given
> device.
> 
> If permanent batch is subject to change (occurs an adding or removing the
> counter with specified batch id) the PMD should free all resources, internally
> allocated for batch query optimization.
> 
> If RTE_FLOW_QUERY_FLAG_PERMANENT is not set,
> rte_flow_query_update() should free the resources allocated (including
> ones done in previous calls, if any) for batch query optimization with given
> group_id.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  doc/guides/prog_guide/rte_flow.rst | 76
> ++++++++++++++++++++++++++++----------
>  lib/librte_ethdev/rte_flow.h       | 59 ++++++++++++++++++++++++++++-
>  2 files changed, 113 insertions(+), 22 deletions(-)
> 
> diff --git a/doc/guides/prog_guide/rte_flow.rst
> b/doc/guides/prog_guide/rte_flow.rst
> index b305a72..84b3b67 100644
> --- a/doc/guides/prog_guide/rte_flow.rst
> +++ b/doc/guides/prog_guide/rte_flow.rst
> @@ -1500,6 +1500,10 @@ action must specify a unique id.
>  Counters can be retrieved and reset through ``rte_flow_query()``, see
> ``struct rte_flow_query_count``.
> 
> +Counters can be assigned with group_id, all counters with matched
> +group_id on the same port are grouped into batch and can be queried
> +from device using the single call of rte_flow_query_update()
> +
>  The shared flag indicates whether the counter is unique to the flow rule the
> action is specified with, or whether it is a shared counter.
> 
> @@ -1515,13 +1519,15 @@ to all ports within that switch domain.
> 
>  .. table:: COUNT
> 
> -   +------------+---------------------+
> -   | Field      | Value               |
> -   +============+=====================+
> -   | ``shared`` | shared counter flag |
> -   +------------+---------------------+
> -   | ``id``     | counter id          |
> -   +------------+---------------------+
> +   +--------------+----------------------+
> +   | Field        | Value                |
> +   +==============+======================+
> +   | ``shared``   | shared counter flag  |
> +   +--------------+----------------------+
> +   | ``id``       | counter id           |
> +   +--------------+----------------------+
> +   | ``group_id`` | batch id             |
> +   +--------------+----------------------+
> 
>  Query structure to retrieve and reset flow rule counters:
> 
> @@ -1529,19 +1535,21 @@ Query structure to retrieve and reset flow rule
> counters:
> 
>  .. table:: COUNT query
> 
> -   +---------------+-----+-----------------------------------+
> -   | Field         | I/O | Value                             |
> -
> +===============+=====+===================================
> +
> -   | ``reset``     | in  | reset counter after query         |
> -   +---------------+-----+-----------------------------------+
> -   | ``hits_set``  | out | ``hits`` field is set             |
> -   +---------------+-----+-----------------------------------+
> -   | ``bytes_set`` | out | ``bytes`` field is set            |
> -   +---------------+-----+-----------------------------------+
> -   | ``hits``      | out | number of hits for this rule      |
> -   +---------------+-----+-----------------------------------+
> -   | ``bytes``     | out | number of bytes through this rule |
> -   +---------------+-----+-----------------------------------+
> +   +-----------------+-----+-------------------------------------+
> +   | Field           | I/O | Value                               |
> +
> +=================+=====+=================================
> ====+
> +   | ``reset``       | in  | reset counter after query           |
> +   +-----------------+-----+-------------------------------------+
> +   | ``hits_set``    | out | ``hits`` field is set               |
> +   +-----------------+-----+-------------------------------------+
> +   | ``bytes_set``   | out | ``bytes`` field is set              |
> +   +-----------------+-----+-------------------------------------+
> +   | ``read_cached`` | in  | read cached data instead of device  |
> +   +-----------------+-----+-------------------------------------+
> +   | ``hits``        | out | number of hits for this rule        |
> +   +-----------------+-----+-------------------------------------+
> +   | ``bytes``       | out | number of bytes through this rule   |
> +   +-----------------+-----+-------------------------------------+
> 
>  Action: ``RSS``
>  ^^^^^^^^^^^^^^^
> @@ -2288,6 +2296,34 @@ Return values:
> 
>  - 0 on success, a negative errno value otherwise and ``rte_errno`` is set.
> 
> +Batch Query
> +~~~~~~~~~~~
> +
> +Query a batch of existing flow rules.
> +
> +This function allows retrieving flow-specific data such as counters
> +belonging to the different flows on the given port in single batch
> +query call.
> +
> +.. code-block:: c
> +
> +   int
> +   rte_flow_query_update(uint16_t port_id,
> +		         uint32_t group_id,
> +		         uint32_t flags);
> +
> +Arguments:
> +
> +- ``port_id``: port identifier of Ethernet device.
> +- ``group_id``: batch to query, specifies the group of actions.
> +- ``flags``: can be combination of ``RTE_FLOW_QUERY_FLAG_RESET``
> +  and ``RTE_FLOW_QUERY_FLAG_PERMANENT`` values
> +
> +Return values:
> +
> +- 0 on success, a negative errno value otherwise and ``rte_errno`` is set.
> +
> +
>  Isolated mode
>  -------------
> 
> diff --git a/lib/librte_ethdev/rte_flow.h b/lib/librte_ethdev/rte_flow.h index
> f8ba71c..0cbc8fa 100644
> --- a/lib/librte_ethdev/rte_flow.h
> +++ b/lib/librte_ethdev/rte_flow.h
> @@ -1561,10 +1561,14 @@ struct rte_flow_action_queue {
>   * Counters can be retrieved and reset through ``rte_flow_query()``, see
>   * ``struct rte_flow_query_count``.
>   *
> + * Counters can be assigned with group_id, all counters with matched
> + group_id
> + * on the same port are grouped into batch and can be queried from
> + device
> + * using the single call of rte_flow_query_update()
> + *
>   * The shared flag indicates whether the counter is unique to the flow rule
> the
>   * action is specified with, or whether it is a shared counter.
>   *
> - * For a count action with the shared flag set, then then a global device
> + * For a count action with the shared flag set, then a global device
>   * namespace is assumed for the counter id, so that any matched flow rules
> using
>   * a count action with the same counter id on the same port will contribute
> to
>   * that counter.
> @@ -1576,6 +1580,7 @@ struct rte_flow_action_count {
>  	uint32_t shared:1; /**< Share counter ID with other flow rules. */
>  	uint32_t reserved:31; /**< Reserved, must be zero. */
>  	uint32_t id; /**< Counter ID. */
> +	uint32_t group_id; /**< ID of batch that counter belongs to */
>  };
> 
>  /**
> @@ -1587,7 +1592,8 @@ struct rte_flow_query_count {
>  	uint32_t reset:1; /**< Reset counters after query [in]. */
>  	uint32_t hits_set:1; /**< hits field is set [out]. */
>  	uint32_t bytes_set:1; /**< bytes field is set [out]. */
> -	uint32_t reserved:29; /**< Reserved, must be zero [in, out]. */
> +	uint32_t read_cached:1; /**< read stored data instead of device [in].
> */
> +	uint32_t reserved:28; /**< Reserved, must be zero [in, out]. */
>  	uint64_t hits; /**< Number of hits for this rule [out]. */
>  	uint64_t bytes; /**< Number of bytes through this rule [out]. */  };
> @@ -2094,6 +2100,55 @@ struct rte_flow *
>  	       struct rte_flow_error *error);
> 
>  /**
> + * Query a batch of existing flow rules.
> + *
> + * This function allows retrieving flow-specific data such as counters
> + * belonging to the different flows on the given port in single batch
> + * query call.
> + *
> + * \see RTE_FLOW_ACTION_TYPE_COUNT
> + *
> + * @param port_id
> + *   Port identifier of Ethernet device.
> + * @param group_id
> + *   Batch identifier, specifies the group of actions to be queried.
> + * @param flags
> + *   RTE_FLOW_QUERY_FLAG_RESET
> + *   RTE_FLOW_QUERY_FLAG_PERMANENT
> + *
> + * @return
> + *   0 on success, a negative errno value otherwise and rte_errno is set.
> + */
> +int
> +rte_flow_query_update(uint16_t port_id,
> +		      uint32_t group_id,
> +		      uint32_t flags);
> +
> +#define RTE_FLOW_QUERY_FLAG_RESET (1 << 0) /**< Reset counters after
> +query */
> +/**
> + * For some devices implementation of rte_flow_query_update() may
> +require
> + * a lot of preparations before performing the actual query. If batch
> +is
> + * permanent and assumed not to be changed frequently the preparations
> + * can be cached internally by implementation. By setting the
> + * RTE_FLOW_QUERY_FLAG_PERMANENT flag application gives the hint to
> PMD
> + * that batch is assumed to be long-term and allows to optimize the
> + * succesive calls of rte_flow_query_update() for the same group_id
> + * on given device.
> + *
> + * If permanent batch is subject to change (occurs an adding or
> + * removing the action with specified batch id) the PMD should free all
> + * resources, internally allocated for batch query optimization
> + *
> + * If RTE_FLOW_QUERY_FLAG_PERMANENT is not set,
> rte_flow_query_update()
> + * should free the resources (if any) early allocated for batch query
> + * optimization with given group_id.
> + *
> + */
> +#define RTE_FLOW_QUERY_FLAG_PERMANENT (1 << 1) /**< Assume
> batch
> +permanent */
> +
> +/**
>   * Restrict ingress traffic to the defined flow rules.
>   *
>   * Isolated mode guarantees that all ingress traffic comes from defined flow
> --
> 1.8.3.1