[RFC] ethdev: sharing indirect actions between ports

[RFC] ethdev: sharing indirect actions between ports

[Viacheslav Ovsiienko]

The RTE Flow API implements the concept of shared objects,
known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
An application can create the indirect action of desired
type and configuration with rte_flow_action_handle_create
call and then specify the obtained action handle in multiple
flows.

The initial concept supposes the action handle has strict
attachment to the port it was created on and to be used
exclusively in the flows being installed on the port.

Nowadays the multipath network topologies are quite common,
packets belonging to the same connection might arrive and
be sent over multiple ports, and there is the raising demand
to handle these "spread" connections. To fulfil this demand
it is proposed to extend indirect action sharing across the
multiple ports. This kind of sharing would be extremely useful
for the meters and counters, allowing to manage the single
connection over the multiple ports.

This cross-port object sharing is hard to implement in
generic way merely with software on the upper layers, but
can be provided by the driver over the single hardware
instance, where  multiple ports reside on the same physical
NIC and share the same hardware context.

To allow this action sharing application should specify
the "host port" during flow configuring to claim the intention
to share the indirect actions. All indirect actions reside within
"host port" context and can be shared in flows being installed
on the host port and on all the ports referencing this one.

If sharing between host and port being configured is not supported
the configuration should be rejected with error. There might be
multiple independent (mutual exclusive) sharing domains with
dedicated host and referencing ports.

To manage the shared indirect action any port from sharing domain
can be specified. To share or not the created action is up to
application, no API change is needed.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 lib/ethdev/rte_flow.c |  6 ++++++
 lib/ethdev/rte_flow.h | 11 +++++++++++
 2 files changed, 17 insertions(+)

diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
index 7d0c24366c..692d37925a 100644
--- a/lib/ethdev/rte_flow.c
+++ b/lib/ethdev/rte_flow.c
@@ -1476,6 +1476,12 @@ rte_flow_configure(uint16_t port_id,
 		RTE_FLOW_LOG(ERR, "Port %"PRIu16" queue info is NULL.\n", port_id);
 		return -EINVAL;
 	}
+	if ((port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) &&
+	     !rte_eth_dev_is_valid_port(port_attr->host_port_id)) {
+		return rte_flow_error_set(error, ENODEV,
+					  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+					  NULL, rte_strerror(ENODEV));
+	}
 	if (likely(!!ops->configure)) {
 		ret = ops->configure(dev, port_attr, nb_queue, queue_attr, error);
 		if (ret == 0)
diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index b60987db4b..c784f4ec30 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -4903,6 +4903,13 @@ rte_flow_info_get(uint16_t port_id,
 		  struct rte_flow_queue_info *queue_info,
 		  struct rte_flow_error *error);
 
+/**
+ * Indicate all steering objects should be created on contexts
+ * of the host port, providing indirect object sharing beetween
+ * ports.
+ */
+#define RTE_FLOW_PORT_FLAG_SHARE_INDIRECT RTE_BIT32(0)
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
@@ -4932,6 +4939,10 @@ struct rte_flow_port_attr {
 	 * @see RTE_FLOW_ACTION_TYPE_CONNTRACK
 	 */
 	uint32_t nb_conn_tracks;
+	/**
+	 * Port to base shared objects on.
+	 */
+	uint16_t host_port_id;
 	/**
 	 * Port flags (RTE_FLOW_PORT_FLAG_*).
 	 */
-- 
2.18.1

RE: [RFC] ethdev: sharing indirect actions between ports

[Ori Kam]

Hi Slava,

Looks good to me,
Best,
Ori

> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Wednesday, 28 December 2022 18:55
> 
> The RTE Flow API implements the concept of shared objects,
> known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> An application can create the indirect action of desired
> type and configuration with rte_flow_action_handle_create
> call and then specify the obtained action handle in multiple
> flows.
> 
> The initial concept supposes the action handle has strict
> attachment to the port it was created on and to be used
> exclusively in the flows being installed on the port.
> 
> Nowadays the multipath network topologies are quite common,
> packets belonging to the same connection might arrive and
> be sent over multiple ports, and there is the raising demand
> to handle these "spread" connections. To fulfil this demand
> it is proposed to extend indirect action sharing across the
> multiple ports. This kind of sharing would be extremely useful
> for the meters and counters, allowing to manage the single
> connection over the multiple ports.
> 
> This cross-port object sharing is hard to implement in
> generic way merely with software on the upper layers, but
> can be provided by the driver over the single hardware
> instance, where  multiple ports reside on the same physical
> NIC and share the same hardware context.
> 
> To allow this action sharing application should specify
> the "host port" during flow configuring to claim the intention
> to share the indirect actions. All indirect actions reside within
> "host port" context and can be shared in flows being installed
> on the host port and on all the ports referencing this one.
> 
> If sharing between host and port being configured is not supported
> the configuration should be rejected with error. There might be
> multiple independent (mutual exclusive) sharing domains with
> dedicated host and referencing ports.
> 
> To manage the shared indirect action any port from sharing domain
> can be specified. To share or not the created action is up to
> application, no API change is needed.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---
>  lib/ethdev/rte_flow.c |  6 ++++++
>  lib/ethdev/rte_flow.h | 11 +++++++++++
>  2 files changed, 17 insertions(+)
> 
> diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
> index 7d0c24366c..692d37925a 100644
> --- a/lib/ethdev/rte_flow.c
> +++ b/lib/ethdev/rte_flow.c
> @@ -1476,6 +1476,12 @@ rte_flow_configure(uint16_t port_id,
>  		RTE_FLOW_LOG(ERR, "Port %"PRIu16" queue info is
> NULL.\n", port_id);
>  		return -EINVAL;
>  	}
> +	if ((port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) &&
> +	     !rte_eth_dev_is_valid_port(port_attr->host_port_id)) {
> +		return rte_flow_error_set(error, ENODEV,
> +
> RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
> +					  NULL, rte_strerror(ENODEV));
> +	}
>  	if (likely(!!ops->configure)) {
>  		ret = ops->configure(dev, port_attr, nb_queue, queue_attr,
> error);
>  		if (ret == 0)
> diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
> index b60987db4b..c784f4ec30 100644
> --- a/lib/ethdev/rte_flow.h
> +++ b/lib/ethdev/rte_flow.h
> @@ -4903,6 +4903,13 @@ rte_flow_info_get(uint16_t port_id,
>  		  struct rte_flow_queue_info *queue_info,
>  		  struct rte_flow_error *error);
> 
> +/**
> + * Indicate all steering objects should be created on contexts
> + * of the host port, providing indirect object sharing beetween
> + * ports.
> + */
> +#define RTE_FLOW_PORT_FLAG_SHARE_INDIRECT RTE_BIT32(0)
> +
>  /**
>   * @warning
>   * @b EXPERIMENTAL: this API may change without prior notice.
> @@ -4932,6 +4939,10 @@ struct rte_flow_port_attr {
>  	 * @see RTE_FLOW_ACTION_TYPE_CONNTRACK
>  	 */
>  	uint32_t nb_conn_tracks;
> +	/**
> +	 * Port to base shared objects on.
> +	 */
> +	uint16_t host_port_id;
>  	/**
>  	 * Port flags (RTE_FLOW_PORT_FLAG_*).
>  	 */
> --
> 2.18.1

Re: [RFC] ethdev: sharing indirect actions between ports

[Thomas Monjalon]

28/12/2022 17:54, Viacheslav Ovsiienko:
> The RTE Flow API implements the concept of shared objects,
> known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> An application can create the indirect action of desired
> type and configuration with rte_flow_action_handle_create
> call and then specify the obtained action handle in multiple
> flows.
> 
> The initial concept supposes the action handle has strict
> attachment to the port it was created on and to be used
> exclusively in the flows being installed on the port.
> 
> Nowadays the multipath network topologies are quite common,
> packets belonging to the same connection might arrive and
> be sent over multiple ports, and there is the raising demand
> to handle these "spread" connections. To fulfil this demand
> it is proposed to extend indirect action sharing across the
> multiple ports. This kind of sharing would be extremely useful
> for the meters and counters, allowing to manage the single
> connection over the multiple ports.
> 
> This cross-port object sharing is hard to implement in
> generic way merely with software on the upper layers, but
> can be provided by the driver over the single hardware
> instance, where  multiple ports reside on the same physical
> NIC and share the same hardware context.
> 
> To allow this action sharing application should specify
> the "host port" during flow configuring to claim the intention
> to share the indirect actions. All indirect actions reside within
> "host port" context and can be shared in flows being installed

I don't like the word "host" because it may refer to the host CPU.
Also if I understand well, the application must choose one port
between all ports of the NIC and keep using the same.
I guess we don't want to create a NIC id.
So I would suggest to rename to nic_ref_port or something like that.

> on the host port and on all the ports referencing this one.

Not "all the ports" but only on sibling ports (ports of the same NIC).

> If sharing between host and port being configured is not supported
> the configuration should be rejected with error. There might be
> multiple independent (mutual exclusive) sharing domains with
> dedicated host and referencing ports.
> 
> To manage the shared indirect action any port from sharing domain
> can be specified. To share or not the created action is up to
> application, no API change is needed.
[...]
> +	/**
> +	 * Port to base shared objects on.
> +	 */
> +	uint16_t host_port_id;

Is it a DPDK ethdev port ID?

You should add this feature to the release notes and in rte_flow guide.

PS: please Cc ethdev maintainers.

RE: [RFC] ethdev: sharing indirect actions between ports

[Ori Kam]

Hi

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, 18 January 2023 14:07
> 
> 28/12/2022 17:54, Viacheslav Ovsiienko:
> > The RTE Flow API implements the concept of shared objects,
> > known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> > An application can create the indirect action of desired
> > type and configuration with rte_flow_action_handle_create
> > call and then specify the obtained action handle in multiple
> > flows.
> >
> > The initial concept supposes the action handle has strict
> > attachment to the port it was created on and to be used
> > exclusively in the flows being installed on the port.
> >
> > Nowadays the multipath network topologies are quite common,
> > packets belonging to the same connection might arrive and
> > be sent over multiple ports, and there is the raising demand
> > to handle these "spread" connections. To fulfil this demand
> > it is proposed to extend indirect action sharing across the
> > multiple ports. This kind of sharing would be extremely useful
> > for the meters and counters, allowing to manage the single
> > connection over the multiple ports.
> >
> > This cross-port object sharing is hard to implement in
> > generic way merely with software on the upper layers, but
> > can be provided by the driver over the single hardware
> > instance, where  multiple ports reside on the same physical
> > NIC and share the same hardware context.
> >
> > To allow this action sharing application should specify
> > the "host port" during flow configuring to claim the intention
> > to share the indirect actions. All indirect actions reside within
> > "host port" context and can be shared in flows being installed
> 
> I don't like the word "host" because it may refer to the host CPU.
> Also if I understand well, the application must choose one port
> between all ports of the NIC and keep using the same.
> I guess we don't want to create a NIC id.
> So I would suggest to rename to nic_ref_port or something like that.
> 

I think that host is the correct word since this port hosts all resources for other
ports. (this is also why the host is used in case of CPU 😊)
I don't think it is correct to use bad wording due to the fact that some one else
also uses this word. 
in rte_flow we never talk about host CPU so I don't think this is confusing.

> > on the host port and on all the ports referencing this one.
> 
> Not "all the ports" but only on sibling ports (ports of the same NIC).
> 
> > If sharing between host and port being configured is not supported
> > the configuration should be rejected with error. There might be
> > multiple independent (mutual exclusive) sharing domains with
> > dedicated host and referencing ports.
> >
> > To manage the shared indirect action any port from sharing domain
> > can be specified. To share or not the created action is up to
> > application, no API change is needed.
> [...]
> > +	/**
> > +	 * Port to base shared objects on.
> > +	 */
> > +	uint16_t host_port_id;
> 
> Is it a DPDK ethdev port ID?
> 
> You should add this feature to the release notes and in rte_flow guide.
> 
> PS: please Cc ethdev maintainers.
> 

Re: [RFC] ethdev: sharing indirect actions between ports

[Thomas Monjalon]

18/01/2023 16:17, Ori Kam:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 28/12/2022 17:54, Viacheslav Ovsiienko:
> > > The RTE Flow API implements the concept of shared objects,
> > > known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> > > An application can create the indirect action of desired
> > > type and configuration with rte_flow_action_handle_create
> > > call and then specify the obtained action handle in multiple
> > > flows.
> > >
> > > The initial concept supposes the action handle has strict
> > > attachment to the port it was created on and to be used
> > > exclusively in the flows being installed on the port.
> > >
> > > Nowadays the multipath network topologies are quite common,
> > > packets belonging to the same connection might arrive and
> > > be sent over multiple ports, and there is the raising demand
> > > to handle these "spread" connections. To fulfil this demand
> > > it is proposed to extend indirect action sharing across the
> > > multiple ports. This kind of sharing would be extremely useful
> > > for the meters and counters, allowing to manage the single
> > > connection over the multiple ports.
> > >
> > > This cross-port object sharing is hard to implement in
> > > generic way merely with software on the upper layers, but
> > > can be provided by the driver over the single hardware
> > > instance, where  multiple ports reside on the same physical
> > > NIC and share the same hardware context.
> > >
> > > To allow this action sharing application should specify
> > > the "host port" during flow configuring to claim the intention
> > > to share the indirect actions. All indirect actions reside within
> > > "host port" context and can be shared in flows being installed
> > 
> > I don't like the word "host" because it may refer to the host CPU.
> > Also if I understand well, the application must choose one port
> > between all ports of the NIC and keep using the same.
> > I guess we don't want to create a NIC id.
> > So I would suggest to rename to nic_ref_port or something like that.
> > 
> 
> I think that host is the correct word since this port hosts all resources for other
> ports. (this is also why the host is used in case of CPU 😊)
> I don't think it is correct to use bad wording due to the fact that some one else
> also uses this word.
> in rte_flow we never talk about host CPU so I don't think this is confusing.

The confusion is that we can think of a port on the host.

RE: [RFC] ethdev: sharing indirect actions between ports

[Slava Ovsiienko]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, January 18, 2023 6:22 PM
> To: Slava Ovsiienko <viacheslavo@nvidia.com>; Ori Kam
> <orika@nvidia.com>
> Cc: dev@dpdk.org; Matan Azrad <matan@nvidia.com>; Raslan Darawsheh
> <rasland@nvidia.com>; andrew.rybchenko@oktetlabs.ru;
> ivan.malov@oktetlabs.ru; ferruh.yigit@amd.com
> Subject: Re: [RFC] ethdev: sharing indirect actions between ports
> 
> 18/01/2023 16:17, Ori Kam:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 28/12/2022 17:54, Viacheslav Ovsiienko:
> > > > The RTE Flow API implements the concept of shared objects, known
> > > > as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> > > > An application can create the indirect action of desired type and
> > > > configuration with rte_flow_action_handle_create call and then
> > > > specify the obtained action handle in multiple flows.
> > > >
> > > > The initial concept supposes the action handle has strict
> > > > attachment to the port it was created on and to be used
> > > > exclusively in the flows being installed on the port.
> > > >
> > > > Nowadays the multipath network topologies are quite common,
> > > > packets belonging to the same connection might arrive and be sent
> > > > over multiple ports, and there is the raising demand to handle
> > > > these "spread" connections. To fulfil this demand it is proposed
> > > > to extend indirect action sharing across the multiple ports. This
> > > > kind of sharing would be extremely useful for the meters and
> > > > counters, allowing to manage the single connection over the
> > > > multiple ports.
> > > >
> > > > This cross-port object sharing is hard to implement in generic way
> > > > merely with software on the upper layers, but can be provided by
> > > > the driver over the single hardware instance, where  multiple
> > > > ports reside on the same physical NIC and share the same hardware
> > > > context.
> > > >
> > > > To allow this action sharing application should specify the "host
> > > > port" during flow configuring to claim the intention to share the
> > > > indirect actions. All indirect actions reside within "host port"
> > > > context and can be shared in flows being installed
> > >
> > > I don't like the word "host" because it may refer to the host CPU.
> > > Also if I understand well, the application must choose one port
> > > between all ports of the NIC and keep using the same.
> > > I guess we don't want to create a NIC id.
> > > So I would suggest to rename to nic_ref_port or something like that.
> > >
> >
> > I think that host is the correct word since this port hosts all
> > resources for other ports. (this is also why the host is used in case
> > of CPU 😊)
> > I don't think it is correct to use bad wording due to the fact that
> > some one else also uses this word.
> > in rte_flow we never talk about host CPU so I don't think this is confusing.
> 
> The confusion is that we can think of a port on the host.

In my humble opinion, "_port_id" suffix explicitly specifies what field is and does not leave
too much space for confusion.

"root_port_id"? "base_port_id"?  "container_port_id" ? "mgmnt_port_id" ?
 Looks worse as for me and does not reflect the exact meaning.
As Ori mentioned this is DPDK port ID that embraces all the shared actions.
It plays a host role for them.

With best regards, Slava

> 
> 

Re: [RFC] ethdev: sharing indirect actions between ports

[Andrew Rybchenko]

On 1/18/23 19:37, Slava Ovsiienko wrote:
> 
> 
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Wednesday, January 18, 2023 6:22 PM
>> To: Slava Ovsiienko <viacheslavo@nvidia.com>; Ori Kam
>> <orika@nvidia.com>
>> Cc: dev@dpdk.org; Matan Azrad <matan@nvidia.com>; Raslan Darawsheh
>> <rasland@nvidia.com>; andrew.rybchenko@oktetlabs.ru;
>> ivan.malov@oktetlabs.ru; ferruh.yigit@amd.com
>> Subject: Re: [RFC] ethdev: sharing indirect actions between ports
>>
>> 18/01/2023 16:17, Ori Kam:
>>> From: Thomas Monjalon <thomas@monjalon.net>
>>>> 28/12/2022 17:54, Viacheslav Ovsiienko:
>>>>> The RTE Flow API implements the concept of shared objects, known
>>>>> as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
>>>>> An application can create the indirect action of desired type and
>>>>> configuration with rte_flow_action_handle_create call and then
>>>>> specify the obtained action handle in multiple flows.
>>>>>
>>>>> The initial concept supposes the action handle has strict
>>>>> attachment to the port it was created on and to be used
>>>>> exclusively in the flows being installed on the port.
>>>>>
>>>>> Nowadays the multipath network topologies are quite common,
>>>>> packets belonging to the same connection might arrive and be sent
>>>>> over multiple ports, and there is the raising demand to handle
>>>>> these "spread" connections. To fulfil this demand it is proposed
>>>>> to extend indirect action sharing across the multiple ports. This
>>>>> kind of sharing would be extremely useful for the meters and
>>>>> counters, allowing to manage the single connection over the
>>>>> multiple ports.
>>>>>
>>>>> This cross-port object sharing is hard to implement in generic way
>>>>> merely with software on the upper layers, but can be provided by
>>>>> the driver over the single hardware instance, where  multiple
>>>>> ports reside on the same physical NIC and share the same hardware
>>>>> context.
>>>>>
>>>>> To allow this action sharing application should specify the "host
>>>>> port" during flow configuring to claim the intention to share the
>>>>> indirect actions. All indirect actions reside within "host port"
>>>>> context and can be shared in flows being installed
>>>>
>>>> I don't like the word "host" because it may refer to the host CPU.
>>>> Also if I understand well, the application must choose one port
>>>> between all ports of the NIC and keep using the same.
>>>> I guess we don't want to create a NIC id.
>>>> So I would suggest to rename to nic_ref_port or something like that.
>>>>
>>>
>>> I think that host is the correct word since this port hosts all
>>> resources for other ports. (this is also why the host is used in case
>>> of CPU 😊)
>>> I don't think it is correct to use bad wording due to the fact that
>>> some one else also uses this word.
>>> in rte_flow we never talk about host CPU so I don't think this is confusing.
>>
>> The confusion is that we can think of a port on the host.
> 
> In my humble opinion, "_port_id" suffix explicitly specifies what field is and does not leave
> too much space for confusion.
> 
> "root_port_id"? "base_port_id"?  "container_port_id" ? "mgmnt_port_id" ?
>   Looks worse as for me and does not reflect the exact meaning.
> As Ori mentioned this is DPDK port ID that embraces all the shared actions.
> It plays a host role for them.

Maybe 'owner_port_id' or 'rsrc_port_id' ?

RE: [RFC] ethdev: sharing indirect actions between ports

[Ori Kam]

> -----Original Message-----
> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Sent: Friday, 20 January 2023 14:23
> 
> On 1/18/23 19:37, Slava Ovsiienko wrote:
> >
> >
> >> -----Original Message-----
> >> From: Thomas Monjalon <thomas@monjalon.net>
> >> Sent: Wednesday, January 18, 2023 6:22 PM
> >> To: Slava Ovsiienko <viacheslavo@nvidia.com>; Ori Kam
> >> <orika@nvidia.com>
> >> Cc: dev@dpdk.org; Matan Azrad <matan@nvidia.com>; Raslan
> Darawsheh
> >> <rasland@nvidia.com>; andrew.rybchenko@oktetlabs.ru;
> >> ivan.malov@oktetlabs.ru; ferruh.yigit@amd.com
> >> Subject: Re: [RFC] ethdev: sharing indirect actions between ports
> >>
> >> 18/01/2023 16:17, Ori Kam:
> >>> From: Thomas Monjalon <thomas@monjalon.net>
> >>>> 28/12/2022 17:54, Viacheslav Ovsiienko:
> >>>>> The RTE Flow API implements the concept of shared objects, known
> >>>>> as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> >>>>> An application can create the indirect action of desired type and
> >>>>> configuration with rte_flow_action_handle_create call and then
> >>>>> specify the obtained action handle in multiple flows.
> >>>>>
> >>>>> The initial concept supposes the action handle has strict
> >>>>> attachment to the port it was created on and to be used
> >>>>> exclusively in the flows being installed on the port.
> >>>>>
> >>>>> Nowadays the multipath network topologies are quite common,
> >>>>> packets belonging to the same connection might arrive and be sent
> >>>>> over multiple ports, and there is the raising demand to handle
> >>>>> these "spread" connections. To fulfil this demand it is proposed
> >>>>> to extend indirect action sharing across the multiple ports. This
> >>>>> kind of sharing would be extremely useful for the meters and
> >>>>> counters, allowing to manage the single connection over the
> >>>>> multiple ports.
> >>>>>
> >>>>> This cross-port object sharing is hard to implement in generic way
> >>>>> merely with software on the upper layers, but can be provided by
> >>>>> the driver over the single hardware instance, where  multiple
> >>>>> ports reside on the same physical NIC and share the same hardware
> >>>>> context.
> >>>>>
> >>>>> To allow this action sharing application should specify the "host
> >>>>> port" during flow configuring to claim the intention to share the
> >>>>> indirect actions. All indirect actions reside within "host port"
> >>>>> context and can be shared in flows being installed
> >>>>
> >>>> I don't like the word "host" because it may refer to the host CPU.
> >>>> Also if I understand well, the application must choose one port
> >>>> between all ports of the NIC and keep using the same.
> >>>> I guess we don't want to create a NIC id.
> >>>> So I would suggest to rename to nic_ref_port or something like that.
> >>>>
> >>>
> >>> I think that host is the correct word since this port hosts all
> >>> resources for other ports. (this is also why the host is used in case
> >>> of CPU 😊)
> >>> I don't think it is correct to use bad wording due to the fact that
> >>> some one else also uses this word.
> >>> in rte_flow we never talk about host CPU so I don't think this is
> confusing.
> >>
> >> The confusion is that we can think of a port on the host.
> >
> > In my humble opinion, "_port_id" suffix explicitly specifies what field is and
> does not leave
> > too much space for confusion.
> >
> > "root_port_id"? "base_port_id"?  "container_port_id" ? "mgmnt_port_id"
> ?
> >   Looks worse as for me and does not reflect the exact meaning.
> > As Ori mentioned this is DPDK port ID that embraces all the shared actions.
> > It plays a host role for them.
> 
> Maybe 'owner_port_id' or 'rsrc_port_id' ?
> 
Rsrc?
Owner_port looks O.K but I'm not sure what is the issue with the original suggestion.

Best,
Ori

[PATCH 1/9] ethdev: sharing indirect actions between ports

[Viacheslav Ovsiienko]

The RTE Flow API implements the concept of shared objects,
known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
An application can create the indirect action of desired
type and configuration with rte_flow_action_handle_create
call and then specify the obtained action handle in multiple
flows.

The initial concept supposes the action handle has strict
attachment to the port it was created on and to be used
exclusively in the flows being installed on the port.

Nowadays the multipath network topologies are quite common,
packets belonging to the same connection might arrive and
be sent over multiple ports, and there is the raising demand
to handle these "spread" connections. To fulfil this demand
it is proposed to extend indirect action sharing across the
multiple ports. This kind of sharing would be extremely useful
for the meters and counters, allowing to manage the single
connection over the multiple ports.

This cross-port object sharing is hard to implement in
generic way merely with software on the upper layers, but
can be provided by the driver over the single hardware
instance, where  multiple ports reside on the same physical
NIC and share the same hardware context.

To allow this action sharing application should specify
the "host port" during flow configuring to claim the intention
to share the indirect actions. All indirect actions reside within
"host port" context and can be shared in flows being installed
on the host port and on all the ports referencing this one.

If sharing between host and port being configured is not supported
the configuration should be rejected with error. There might be
multiple independent (mutual exclusive) sharing domains with
dedicated host and referencing ports.

To manage the shared indirect action any port from sharing domain
can be specified. To share or not the created action is up to
application, no API change is needed.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 lib/ethdev/rte_flow.c |  6 ++++++
 lib/ethdev/rte_flow.h | 11 +++++++++++
 2 files changed, 17 insertions(+)

diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
index 7d0c24366c..692d37925a 100644
--- a/lib/ethdev/rte_flow.c
+++ b/lib/ethdev/rte_flow.c
@@ -1476,6 +1476,12 @@ rte_flow_configure(uint16_t port_id,
 		RTE_FLOW_LOG(ERR, "Port %"PRIu16" queue info is NULL.\n", port_id);
 		return -EINVAL;
 	}
+	if ((port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) &&
+	     !rte_eth_dev_is_valid_port(port_attr->host_port_id)) {
+		return rte_flow_error_set(error, ENODEV,
+					  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+					  NULL, rte_strerror(ENODEV));
+	}
 	if (likely(!!ops->configure)) {
 		ret = ops->configure(dev, port_attr, nb_queue, queue_attr, error);
 		if (ret == 0)
diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index b60987db4b..c784f4ec30 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -4903,6 +4903,13 @@ rte_flow_info_get(uint16_t port_id,
 		  struct rte_flow_queue_info *queue_info,
 		  struct rte_flow_error *error);
 
+/**
+ * Indicate all steering objects should be created on contexts
+ * of the host port, providing indirect object sharing beetween
+ * ports.
+ */
+#define RTE_FLOW_PORT_FLAG_SHARE_INDIRECT RTE_BIT32(0)
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
@@ -4932,6 +4939,10 @@ struct rte_flow_port_attr {
 	 * @see RTE_FLOW_ACTION_TYPE_CONNTRACK
 	 */
 	uint32_t nb_conn_tracks;
+	/**
+	 * Port to base shared objects on.
+	 */
+	uint16_t host_port_id;
 	/**
 	 * Port flags (RTE_FLOW_PORT_FLAG_*).
 	 */
-- 
2.18.1

[PATCH 2/9] net/mlx5/hws: Matcher, Free FT from RTC id before set the new value

[Viacheslav Ovsiienko]

From: Erez Shitrit <erezsh@nvidia.com>

While matcher is being connect/disconnect in shared gvmi flow we set the
first ft in the table to point on the first matcher,
The FW is increasing the refcount on the first matcher RTC because of
that no matcher if it is the same RTC that was set before, and when we
will try to release that RTC we will get the following syndrome:
0xaa0093 -   destroy_rtc_object: rtc in use or doesn't exist.

In order to resolve that we clean the current pointed RTC from that ft
and only after that setting it to the new RTC value.

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/hws/mlx5dr_matcher.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c
index d509a2f0e1..32bf3f5886 100644
--- a/drivers/net/mlx5/hws/mlx5dr_matcher.c
+++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c
@@ -334,6 +334,24 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher)
 		return ret;
 	}
 
+	if (!next) {
+		/* ft no longer points to any RTC, drop refcount */
+		ret = mlx5dr_matcher_free_rtc_pointing(tbl->ctx,
+						       tbl->fw_ft_type,
+						       tbl->type,
+						       prev_ft);
+		if (ret) {
+			DR_LOG(ERR, "Failed to reset last RTC refcount");
+			return ret;
+		}
+	}
+
+	ret = mlx5dr_matcher_shared_update_local_ft(tbl);
+	if (ret) {
+		DR_LOG(ERR, "Failed to update local_ft in shared table");
+		return ret;
+	}
+
 	return 0;
 }
 
-- 
2.18.1

[PATCH 3/9] net/mlx5/hws: fix disconnecting matcher

[Viacheslav Ovsiienko]

From: Erez Shitrit <erezsh@nvidia.com>

This patch fixes the matcher disconnection handling, by removing the RTC
references from flow table if the currently removed matcher was the last
one for the given table. As a result RTC in this matcher can be
correctly freed, since there are no dangling references to the RTC.

Fixes: c467608215b2 ("net/mlx5/hws: add matcher object")

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Reviewed-by: Alex Vesker <valex@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/net/mlx5/hws/mlx5dr_matcher.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c
index 32bf3f5886..a0297c9787 100644
--- a/drivers/net/mlx5/hws/mlx5dr_matcher.c
+++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c
@@ -346,12 +346,6 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher)
 		}
 	}
 
-	ret = mlx5dr_matcher_shared_update_local_ft(tbl);
-	if (ret) {
-		DR_LOG(ERR, "Failed to update local_ft in shared table");
-		return ret;
-	}
-
 	return 0;
 }
 
-- 
2.18.1

[PATCH 4/9] common/mlx5: add cross port object sharing capability

[Viacheslav Ovsiienko]

Add query port capabilities to share steering objects
between multiple ports of the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 13 +++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index e3a4927d0f..17128035ec 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -1047,6 +1047,19 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		attr->flow_counter_bulk_log_granularity =
 			MLX5_GET(cmd_hca_cap_2, hcattr,
 				 flow_counter_bulk_log_granularity);
+		rc = MLX5_GET(cmd_hca_cap_2, hcattr,
+			      cross_vhca_object_to_object_supported);
+		attr->cross_vhca =
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_FT) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC);
+		rc = MLX5_GET(cmd_hca_cap_2, hcattr,
+			      allowed_object_for_other_vhca_access);
+		attr->cross_vhca = attr->cross_vhca &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR) &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_FT) &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC);
 	}
 	if (attr->log_min_stride_wqe_sz == 0)
 		attr->log_min_stride_wqe_sz = MLX5_MPRQ_LOG_MIN_STRIDE_WQE_SIZE;
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index c94b9eac06..b65ba569bc 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -288,6 +288,7 @@ struct mlx5_hca_attr {
 	uint32_t alloc_flow_counter_pd:1;
 	uint32_t flow_counter_access_aso:1;
 	uint32_t flow_access_aso_opc_mod:8;
+	uint32_t cross_vhca:1;
 };
 
 /* LAG Context. */
-- 
2.18.1

[PATCH 5/9] net/mlx5: add cross port shared mode for HW steering

[Viacheslav Ovsiienko]

Add host port option for sharing steering objects between
multiple ports of the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/mlx5.c         |  6 +++
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow_hw.c | 78 +++++++++++++++++++++++++++++++--
 drivers/net/mlx5/mlx5_hws_cnt.c | 12 +++++
 4 files changed, 94 insertions(+), 4 deletions(-)

diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b8643cebdd..2eca2cceef 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2013,6 +2013,12 @@ mlx5_dev_close(struct rte_eth_dev *dev)
 	}
 	if (!priv->sh)
 		return 0;
+	if (priv->shared_refcnt) {
+		DRV_LOG(ERR, "port %u is shared host in use (%u)",
+			dev->data->port_id, priv->shared_refcnt);
+		rte_errno = EBUSY;
+		return -EBUSY;
+	}
 	DRV_LOG(DEBUG, "port %u closing device \"%s\"",
 		dev->data->port_id,
 		((priv->sh->cdev->ctx != NULL) ?
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 16b33e1548..525bdd47f7 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1780,6 +1780,8 @@ struct mlx5_priv {
 	struct mlx5_flow_hw_ctrl_rx *hw_ctrl_rx;
 	/**< HW steering templates used to create control flow rules. */
 #endif
+	struct rte_eth_dev *shared_host; /* Host device for HW steering. */
+	uint16_t shared_refcnt; /* HW steering host reference counter. */
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index ce9e0219e1..fc87e687c9 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -5,6 +5,8 @@
 #include <rte_flow.h>
 
 #include <mlx5_malloc.h>
+
+#include "mlx5.h"
 #include "mlx5_defs.h"
 #include "mlx5_flow.h"
 #include "mlx5_rx.h"
@@ -6302,6 +6304,12 @@ flow_hw_ct_pool_create(struct rte_eth_dev *dev,
 	int reg_id;
 	uint32_t flags;
 
+	if (port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Connection tracking is not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return NULL;
+	}
 	pool = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*pool), 0, SOCKET_ID_ANY);
 	if (!pool) {
 		rte_errno = ENOMEM;
@@ -6786,6 +6794,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		  struct rte_flow_error *error)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_priv *host_priv = NULL;
 	struct mlx5dr_context *dr_ctx = NULL;
 	struct mlx5dr_context_attr dr_ctx_attr = {0};
 	struct mlx5_hw_q *hw_q;
@@ -6800,7 +6809,8 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		.free = mlx5_free,
 		.type = "mlx5_hw_action_construct_data",
 	};
-	/* Adds one queue to be used by PMD.
+	/*
+	 * Adds one queue to be used by PMD.
 	 * The last queue will be used by the PMD.
 	 */
 	uint16_t nb_q_updated = 0;
@@ -6919,6 +6929,57 @@ flow_hw_configure(struct rte_eth_dev *dev,
 	dr_ctx_attr.queues = nb_q_updated;
 	/* Queue size should all be the same. Take the first one. */
 	dr_ctx_attr.queue_size = _queue_attr[0]->size;
+	if (port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		struct rte_eth_dev *host_dev = NULL;
+		uint16_t port_id;
+
+		MLX5_ASSERT(rte_eth_dev_is_valid_port(port_attr->host_port_id));
+		if (is_proxy) {
+			DRV_LOG(ERR, "cross vHCA shared mode not supported "
+				     " for E-Switch confgiurations");
+			rte_errno = ENOTSUP;
+			goto err;
+		}
+		MLX5_ETH_FOREACH_DEV(port_id, dev->device) {
+			if (port_id == port_attr->host_port_id) {
+				host_dev = &rte_eth_devices[port_id];
+				break;
+			}
+		}
+		if (!host_dev || host_dev == dev ||
+		    !host_dev->data || !host_dev->data->dev_private) {
+			DRV_LOG(ERR, "Invalid cross vHCA host port %u",
+				port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		host_priv = host_dev->data->dev_private;
+		if (host_priv->sh->cdev->ctx == priv->sh->cdev->ctx) {
+			DRV_LOG(ERR, "Sibling ports %u and %u do not "
+				     "require cross vHCA sharing mode",
+				dev->data->port_id, port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		if (host_priv->shared_host) {
+			DRV_LOG(ERR, "Host port %u is not the sharing base",
+				port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		if (port_attr->nb_counters ||
+		    port_attr->nb_aging_objects ||
+		    port_attr->nb_meters ||
+		    port_attr->nb_conn_tracks) {
+			DRV_LOG(ERR,
+				"Object numbers on guest port must be zeros");
+			rte_errno = EINVAL;
+			goto err;
+		}
+		dr_ctx_attr.shared_ibv_ctx = host_priv->sh->cdev->ctx;
+		priv->shared_host = host_dev;
+		__atomic_fetch_add(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+	}
 	dr_ctx = mlx5dr_context_open(priv->sh->cdev->ctx, &dr_ctx_attr);
 	/* rte_errno has been updated by HWS layer. */
 	if (!dr_ctx)
@@ -6934,7 +6995,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		goto err;
 	}
 	/* Initialize meter library*/
-	if (port_attr->nb_meters)
+	if (port_attr->nb_meters || (host_priv && host_priv->hws_mpool))
 		if (mlx5_flow_meter_init(dev, port_attr->nb_meters, 1, 1, nb_q_updated))
 			goto err;
 	/* Add global actions. */
@@ -6971,7 +7032,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 			goto err;
 		}
 	}
-	if (port_attr->nb_conn_tracks) {
+	if (port_attr->nb_conn_tracks || (host_priv && host_priv->hws_ctpool)) {
 		mem_size = sizeof(struct mlx5_aso_sq) * nb_q_updated +
 			   sizeof(*priv->ct_mng);
 		priv->ct_mng = mlx5_malloc(MLX5_MEM_ZERO, mem_size,
@@ -6985,7 +7046,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 			goto err;
 		priv->sh->ct_aso_en = 1;
 	}
-	if (port_attr->nb_counters) {
+	if (port_attr->nb_counters || (host_priv && host_priv->hws_cpool)) {
 		priv->hws_cpool = mlx5_hws_cnt_pool_create(dev, port_attr,
 							   nb_queue);
 		if (priv->hws_cpool == NULL)
@@ -7054,6 +7115,10 @@ flow_hw_configure(struct rte_eth_dev *dev,
 	}
 	if (_queue_attr)
 		mlx5_free(_queue_attr);
+	if (priv->shared_host) {
+		__atomic_fetch_sub(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+		priv->shared_host = NULL;
+	}
 	/* Do not overwrite the internal errno information. */
 	if (ret)
 		return ret;
@@ -7132,6 +7197,11 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
 	mlx5_free(priv->hw_q);
 	priv->hw_q = NULL;
 	claim_zero(mlx5dr_context_close(priv->dr_ctx));
+	if (priv->shared_host) {
+		struct mlx5_priv *host_priv = priv->shared_host->data->dev_private;
+		__atomic_fetch_sub(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+		priv->shared_host = NULL;
+	}
 	priv->dr_ctx = NULL;
 	priv->nb_queue = 0;
 }
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.c b/drivers/net/mlx5/mlx5_hws_cnt.c
index 51704ef754..afc93821e4 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.c
+++ b/drivers/net/mlx5/mlx5_hws_cnt.c
@@ -618,6 +618,12 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	int ret = 0;
 	size_t sz;
 
+	if (pattr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Counters are not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return NULL;
+	}
 	/* init cnt service if not. */
 	if (priv->sh->cnt_svc == NULL) {
 		ret = mlx5_hws_cnt_svc_init(priv->sh);
@@ -1189,6 +1195,12 @@ mlx5_hws_age_pool_init(struct rte_eth_dev *dev,
 
 	strict_queue = !!(attr->flags & RTE_FLOW_PORT_FLAG_STRICT_QUEUE);
 	MLX5_ASSERT(priv->hws_cpool);
+	if (attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Aging sn not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
 	nb_alloc_cnts = mlx5_hws_cnt_pool_get_size(priv->hws_cpool);
 	if (strict_queue) {
 		rsize = mlx5_hws_aged_out_q_ring_size_get(nb_alloc_cnts,
-- 
2.18.1

[PATCH 6/9] net/mlx5: support counters in cross port shared mode

[Viacheslav Ovsiienko]

In the cross vHCA sharing mode the host counter pool
should be used in counter related routines. The local
port pool is used to store the dedicated DR action
handle, per queue counter caches and query data are
ignored and not allocated on local pool.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/mlx5_flow_hw.c |  12 ++-
 drivers/net/mlx5/mlx5_hws_cnt.c | 163 ++++++++++++++++----------------
 drivers/net/mlx5/mlx5_hws_cnt.h | 109 +++++++++++----------
 3 files changed, 150 insertions(+), 134 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index fc87e687c9..1495bc384e 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -2311,8 +2311,10 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
 				break;
 			/* Fall-through. */
 		case RTE_FLOW_ACTION_TYPE_COUNT:
-			ret = mlx5_hws_cnt_pool_get(priv->hws_cpool, &queue,
-						    &cnt_id, age_idx);
+			ret = mlx5_hws_cnt_pool_get(priv->hws_cpool,
+					(priv->shared_refcnt ||
+					 priv->hws_cpool->cfg.host_cpool) ?
+					NULL : &queue, &cnt_id, age_idx);
 			if (ret != 0)
 				return ret;
 			ret = mlx5_hws_cnt_pool_get_action_offset
@@ -7997,6 +7999,7 @@ static int
 flow_hw_query_counter(const struct rte_eth_dev *dev, uint32_t counter,
 		      void *data, struct rte_flow_error *error)
 {
+	struct mlx5_hws_cnt_pool *hpool;
 	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_hws_cnt *cnt;
 	struct rte_flow_query_count *qc = data;
@@ -8007,8 +8010,9 @@ flow_hw_query_counter(const struct rte_eth_dev *dev, uint32_t counter,
 		return rte_flow_error_set(error, EINVAL,
 				RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
 				"counter are not available");
-	iidx = mlx5_hws_cnt_iidx(priv->hws_cpool, counter);
-	cnt = &priv->hws_cpool->pool[iidx];
+	hpool = mlx5_hws_cnt_host_pool(priv->hws_cpool);
+	iidx = mlx5_hws_cnt_iidx(hpool, counter);
+	cnt = &hpool->pool[iidx];
 	__hws_cnt_query_raw(priv->hws_cpool, counter, &pkts, &bytes);
 	qc->hits_set = 1;
 	qc->bytes_set = 1;
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.c b/drivers/net/mlx5/mlx5_hws_cnt.c
index afc93821e4..12b2cb0b27 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.c
+++ b/drivers/net/mlx5/mlx5_hws_cnt.c
@@ -24,12 +24,8 @@
 static void
 __hws_cnt_id_load(struct mlx5_hws_cnt_pool *cpool)
 {
-	uint32_t preload;
-	uint32_t q_num = cpool->cache->q_num;
 	uint32_t cnt_num = mlx5_hws_cnt_pool_get_size(cpool);
-	cnt_id_t cnt_id;
-	uint32_t qidx, iidx = 0;
-	struct rte_ring *qcache = NULL;
+	uint32_t iidx;
 
 	/*
 	 * Counter ID order is important for tracking the max number of in used
@@ -39,18 +35,9 @@ __hws_cnt_id_load(struct mlx5_hws_cnt_pool *cpool)
 	 * and then the global free list.
 	 * In the end, user fetch the counter from minimal to the maximum.
 	 */
-	preload = RTE_MIN(cpool->cache->preload_sz, cnt_num / q_num);
-	for (qidx = 0; qidx < q_num; qidx++) {
-		for (; iidx < preload * (qidx + 1); iidx++) {
-			cnt_id = mlx5_hws_cnt_id_gen(cpool, iidx);
-			qcache = cpool->cache->qcache[qidx];
-			if (qcache)
-				rte_ring_enqueue_elem(qcache, &cnt_id,
-						sizeof(cnt_id));
-		}
-	}
-	for (; iidx < cnt_num; iidx++) {
-		cnt_id = mlx5_hws_cnt_id_gen(cpool, iidx);
+	for (iidx = 0; iidx < cnt_num; iidx++) {
+		cnt_id_t cnt_id  = mlx5_hws_cnt_id_gen(cpool, iidx);
+
 		rte_ring_enqueue_elem(cpool->free_list, &cnt_id,
 				sizeof(cnt_id));
 	}
@@ -334,7 +321,26 @@ mlx5_hws_cnt_svc(void *opaque)
 	return NULL;
 }
 
-struct mlx5_hws_cnt_pool *
+static void
+mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool * const cntp)
+{
+	uint32_t qidx = 0;
+	if (cntp == NULL)
+		return;
+	rte_ring_free(cntp->free_list);
+	rte_ring_free(cntp->wait_reset_list);
+	rte_ring_free(cntp->reuse_list);
+	if (cntp->cache) {
+		for (qidx = 0; qidx < cntp->cache->q_num; qidx++)
+			rte_ring_free(cntp->cache->qcache[qidx]);
+	}
+	mlx5_free(cntp->cache);
+	mlx5_free(cntp->raw_mng);
+	mlx5_free(cntp->pool);
+	mlx5_free(cntp);
+}
+
+static struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		       const struct mlx5_hws_cnt_pool_cfg *pcfg,
 		       const struct mlx5_hws_cache_param *ccfg)
@@ -352,6 +358,8 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		return NULL;
 
 	cntp->cfg = *pcfg;
+	if (cntp->cfg.host_cpool)
+		return cntp;
 	cntp->cache = mlx5_malloc(MLX5_MEM_ANY | MLX5_MEM_ZERO,
 			sizeof(*cntp->cache) +
 			sizeof(((struct mlx5_hws_cnt_pool_caches *)0)->qcache[0])
@@ -387,8 +395,9 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		goto error;
 	snprintf(mz_name, sizeof(mz_name), "%s_F_RING", pcfg->name);
 	cntp->free_list = rte_ring_create_elem(mz_name, sizeof(cnt_id_t),
-			(uint32_t)cnt_num, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
+				(uint32_t)cnt_num, SOCKET_ID_ANY,
+				RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ |
+				RING_F_EXACT_SZ);
 	if (cntp->free_list == NULL) {
 		DRV_LOG(ERR, "failed to create free list ring");
 		goto error;
@@ -404,7 +413,7 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 	snprintf(mz_name, sizeof(mz_name), "%s_U_RING", pcfg->name);
 	cntp->reuse_list = rte_ring_create_elem(mz_name, sizeof(cnt_id_t),
 			(uint32_t)cnt_num, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
+			RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
 	if (cntp->reuse_list == NULL) {
 		DRV_LOG(ERR, "failed to create reuse list ring");
 		goto error;
@@ -427,25 +436,6 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 	return NULL;
 }
 
-void
-mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool * const cntp)
-{
-	uint32_t qidx = 0;
-	if (cntp == NULL)
-		return;
-	rte_ring_free(cntp->free_list);
-	rte_ring_free(cntp->wait_reset_list);
-	rte_ring_free(cntp->reuse_list);
-	if (cntp->cache) {
-		for (qidx = 0; qidx < cntp->cache->q_num; qidx++)
-			rte_ring_free(cntp->cache->qcache[qidx]);
-	}
-	mlx5_free(cntp->cache);
-	mlx5_free(cntp->raw_mng);
-	mlx5_free(cntp->pool);
-	mlx5_free(cntp);
-}
-
 int
 mlx5_hws_cnt_service_thread_create(struct mlx5_dev_ctx_shared *sh)
 {
@@ -482,7 +472,7 @@ mlx5_hws_cnt_service_thread_destroy(struct mlx5_dev_ctx_shared *sh)
 	sh->cnt_svc->service_thread = 0;
 }
 
-int
+static int
 mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 			    struct mlx5_hws_cnt_pool *cpool)
 {
@@ -494,6 +484,7 @@ mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 	struct mlx5_devx_counter_attr attr = {0};
 	struct mlx5_devx_obj *dcs;
 
+	MLX5_ASSERT(cpool->cfg.host_cpool == NULL);
 	if (hca_attr->flow_counter_bulk_log_max_alloc == 0) {
 		DRV_LOG(ERR, "Fw doesn't support bulk log max alloc");
 		return -1;
@@ -549,7 +540,7 @@ mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 	return -1;
 }
 
-void
+static void
 mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
 			   struct mlx5_hws_cnt_pool *cpool)
 {
@@ -565,22 +556,39 @@ mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
 	}
 }
 
-int
+static void
+mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool)
+{
+	uint32_t idx;
+
+	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
+		struct mlx5_hws_cnt_dcs *dcs = &cpool->dcs_mng.dcs[idx];
+
+		if (dcs->dr_action != NULL) {
+			mlx5dr_action_destroy(dcs->dr_action);
+			dcs->dr_action = NULL;
+		}
+	}
+}
+
+static int
 mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
 		struct mlx5_hws_cnt_pool *cpool)
 {
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint32_t idx;
 	int ret = 0;
-	struct mlx5_hws_cnt_dcs *dcs;
 	uint32_t flags;
 
 	flags = MLX5DR_ACTION_FLAG_HWS_RX | MLX5DR_ACTION_FLAG_HWS_TX;
 	if (priv->sh->config.dv_esw_en && priv->master)
 		flags |= MLX5DR_ACTION_FLAG_HWS_FDB;
-	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
-		dcs = &cpool->dcs_mng.dcs[idx];
+	for (idx = 0; idx < hpool->dcs_mng.batch_total; idx++) {
+		struct mlx5_hws_cnt_dcs *hdcs = &hpool->dcs_mng.dcs[idx];
+		struct mlx5_hws_cnt_dcs *dcs = &cpool->dcs_mng.dcs[idx];
+
 		dcs->dr_action = mlx5dr_action_create_counter(priv->dr_ctx,
-					(struct mlx5dr_devx_obj *)dcs->obj,
+					(struct mlx5dr_devx_obj *)hdcs->obj,
 					flags);
 		if (dcs->dr_action == NULL) {
 			mlx5_hws_cnt_pool_action_destroy(cpool);
@@ -591,21 +599,6 @@ mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
 	return ret;
 }
 
-void
-mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool)
-{
-	uint32_t idx;
-	struct mlx5_hws_cnt_dcs *dcs;
-
-	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
-		dcs = &cpool->dcs_mng.dcs[idx];
-		if (dcs->dr_action != NULL) {
-			mlx5dr_action_destroy(dcs->dr_action);
-			dcs->dr_action = NULL;
-		}
-	}
-}
-
 struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 		const struct rte_flow_port_attr *pattr, uint16_t nb_queue)
@@ -618,11 +611,28 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	int ret = 0;
 	size_t sz;
 
+	mp_name = mlx5_malloc(MLX5_MEM_ZERO, RTE_MEMZONE_NAMESIZE, 0,
+			SOCKET_ID_ANY);
+	if (mp_name == NULL)
+		goto error;
+	snprintf(mp_name, RTE_MEMZONE_NAMESIZE, "MLX5_HWS_CNT_POOL_%u",
+			dev->data->port_id);
+	pcfg.name = mp_name;
+	pcfg.request_num = pattr->nb_counters;
+	pcfg.alloc_factor = HWS_CNT_ALLOC_FACTOR_DEFAULT;
 	if (pattr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
-		DRV_LOG(ERR, "Counters are not supported "
-			     "in cross vHCA sharing mode");
-		rte_errno = ENOTSUP;
-		return NULL;
+		struct mlx5_priv *host_priv =
+				priv->shared_host->data->dev_private;
+		struct mlx5_hws_cnt_pool *chost = host_priv->hws_cpool;
+
+		pcfg.host_cpool = chost;
+		cpool = mlx5_hws_cnt_pool_init(priv->sh, &pcfg, &cparam);
+		if (cpool == NULL)
+			goto error;
+		ret = mlx5_hws_cnt_pool_action_create(priv, cpool);
+		if (ret != 0)
+			goto error;
+		return cpool;
 	}
 	/* init cnt service if not. */
 	if (priv->sh->cnt_svc == NULL) {
@@ -635,15 +645,6 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	cparam.q_num = nb_queue;
 	cparam.threshold = HWS_CNT_CACHE_THRESHOLD_DEFAULT;
 	cparam.size = HWS_CNT_CACHE_SZ_DEFAULT;
-	pcfg.alloc_factor = HWS_CNT_ALLOC_FACTOR_DEFAULT;
-	mp_name = mlx5_malloc(MLX5_MEM_ZERO, RTE_MEMZONE_NAMESIZE, 0,
-			SOCKET_ID_ANY);
-	if (mp_name == NULL)
-		goto error;
-	snprintf(mp_name, RTE_MEMZONE_NAMESIZE, "MLX5_HWS_CNT_POOL_%u",
-			dev->data->port_id);
-	pcfg.name = mp_name;
-	pcfg.request_num = pattr->nb_counters;
 	cpool = mlx5_hws_cnt_pool_init(priv->sh, &pcfg, &cparam);
 	if (cpool == NULL)
 		goto error;
@@ -678,11 +679,15 @@ mlx5_hws_cnt_pool_destroy(struct mlx5_dev_ctx_shared *sh,
 {
 	if (cpool == NULL)
 		return;
-	if (--sh->cnt_svc->refcnt == 0)
-		mlx5_hws_cnt_svc_deinit(sh);
+	if (cpool->cfg.host_cpool == NULL) {
+		if (--sh->cnt_svc->refcnt == 0)
+			mlx5_hws_cnt_svc_deinit(sh);
+	}
 	mlx5_hws_cnt_pool_action_destroy(cpool);
-	mlx5_hws_cnt_pool_dcs_free(sh, cpool);
-	mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+	if (cpool->cfg.host_cpool == NULL) {
+		mlx5_hws_cnt_pool_dcs_free(sh, cpool);
+		mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+	}
 	mlx5_free((void *)cpool->cfg.name);
 	mlx5_hws_cnt_pool_deinit(cpool);
 }
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.h b/drivers/net/mlx5/mlx5_hws_cnt.h
index 030dcead86..d35d083eeb 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.h
+++ b/drivers/net/mlx5/mlx5_hws_cnt.h
@@ -86,6 +86,7 @@ struct mlx5_hws_cnt_pool_cfg {
 	char *name;
 	uint32_t request_num;
 	uint32_t alloc_factor;
+	struct mlx5_hws_cnt_pool *host_cpool;
 };
 
 struct mlx5_hws_cnt_pool_caches {
@@ -148,6 +149,22 @@ struct mlx5_hws_age_param {
 	void *context; /* Flow AGE context. */
 } __rte_packed __rte_cache_aligned;
 
+
+/**
+ * Return the actual counter pool should be used in cross vHCA sharing mode.
+ * as index of raw/cnt pool.
+ *
+ * @param cnt_id
+ *   The external counter id
+ * @return
+ *   Internal index
+ */
+static __always_inline struct mlx5_hws_cnt_pool *
+mlx5_hws_cnt_host_pool(struct mlx5_hws_cnt_pool *cpool)
+{
+	return cpool->cfg.host_cpool ? cpool->cfg.host_cpool : cpool;
+}
+
 /**
  * Translate counter id into internal index (start from 0), which can be used
  * as index of raw/cnt pool.
@@ -160,11 +177,12 @@ struct mlx5_hws_age_param {
 static __rte_always_inline uint32_t
 mlx5_hws_cnt_iidx(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint8_t dcs_idx = cnt_id >> MLX5_HWS_CNT_DCS_IDX_OFFSET;
 	uint32_t offset = cnt_id & MLX5_HWS_CNT_IDX_MASK;
 
 	dcs_idx &= MLX5_HWS_CNT_DCS_IDX_MASK;
-	return (cpool->dcs_mng.dcs[dcs_idx].iidx + offset);
+	return (hpool->dcs_mng.dcs[dcs_idx].iidx + offset);
 }
 
 /**
@@ -191,7 +209,8 @@ mlx5_hws_cnt_id_valid(cnt_id_t cnt_id)
 static __rte_always_inline cnt_id_t
 mlx5_hws_cnt_id_gen(struct mlx5_hws_cnt_pool *cpool, uint32_t iidx)
 {
-	struct mlx5_hws_cnt_dcs_mng *dcs_mng = &cpool->dcs_mng;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	struct mlx5_hws_cnt_dcs_mng *dcs_mng = &hpool->dcs_mng;
 	uint32_t idx;
 	uint32_t offset;
 	cnt_id_t cnt_id;
@@ -212,7 +231,8 @@ static __rte_always_inline void
 __hws_cnt_query_raw(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id,
 		uint64_t *raw_pkts, uint64_t *raw_bytes)
 {
-	struct mlx5_hws_cnt_raw_data_mng *raw_mng = cpool->raw_mng;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	struct mlx5_hws_cnt_raw_data_mng *raw_mng = hpool->raw_mng;
 	struct flow_counter_stats s[2];
 	uint8_t i = 0x1;
 	size_t stat_sz = sizeof(s[0]);
@@ -393,22 +413,23 @@ mlx5_hws_cnt_pool_put(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 		      cnt_id_t *cnt_id)
 {
 	unsigned int ret = 0;
+	struct mlx5_hws_cnt_pool *hpool;
 	struct rte_ring_zc_data zcdc = {0};
 	struct rte_ring_zc_data zcdr = {0};
 	struct rte_ring *qcache = NULL;
 	unsigned int wb_num = 0; /* cache write-back number. */
 	uint32_t iidx;
 
-	iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
-	MLX5_ASSERT(cpool->pool[iidx].in_used);
-	cpool->pool[iidx].in_used = false;
-	cpool->pool[iidx].query_gen_when_free =
-		__atomic_load_n(&cpool->query_gen, __ATOMIC_RELAXED);
-	if (likely(queue != NULL))
-		qcache = cpool->cache->qcache[*queue];
+	hpool = mlx5_hws_cnt_host_pool(cpool);
+	iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
+	hpool->pool[iidx].in_used = false;
+	hpool->pool[iidx].query_gen_when_free =
+		__atomic_load_n(&hpool->query_gen, __ATOMIC_RELAXED);
+	if (likely(queue != NULL) && cpool->cfg.host_cpool == NULL)
+		qcache = hpool->cache->qcache[*queue];
 	if (unlikely(qcache == NULL)) {
-		ret = rte_ring_enqueue_elem(cpool->wait_reset_list, cnt_id,
-					    sizeof(cnt_id_t));
+		ret = rte_ring_enqueue_elem(hpool->wait_reset_list, cnt_id,
+				sizeof(cnt_id_t));
 		MLX5_ASSERT(ret == 0);
 		return;
 	}
@@ -465,9 +486,10 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 	uint32_t iidx, query_gen = 0;
 	cnt_id_t tmp_cid = 0;
 
-	if (likely(queue != NULL))
+	if (likely(queue != NULL && cpool->cfg.host_cpool == NULL))
 		qcache = cpool->cache->qcache[*queue];
 	if (unlikely(qcache == NULL)) {
+		cpool = mlx5_hws_cnt_host_pool(cpool);
 		ret = rte_ring_dequeue_elem(cpool->reuse_list, &tmp_cid,
 				sizeof(cnt_id_t));
 		if (unlikely(ret != 0)) {
@@ -534,7 +556,9 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 static __rte_always_inline unsigned int
 mlx5_hws_cnt_pool_get_size(struct mlx5_hws_cnt_pool *cpool)
 {
-	return rte_ring_get_capacity(cpool->free_list);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+
+	return rte_ring_get_capacity(hpool->free_list);
 }
 
 static __rte_always_inline int
@@ -554,51 +578,56 @@ static __rte_always_inline int
 mlx5_hws_cnt_shared_get(struct mlx5_hws_cnt_pool *cpool, cnt_id_t *cnt_id,
 			uint32_t age_idx)
 {
-	int ret;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint32_t iidx;
+	int ret;
 
-	ret = mlx5_hws_cnt_pool_get(cpool, NULL, cnt_id, age_idx);
+	ret = mlx5_hws_cnt_pool_get(hpool, NULL, cnt_id, age_idx);
 	if (ret != 0)
 		return ret;
-	iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
-	cpool->pool[iidx].share = 1;
+	iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
+	hpool->pool[iidx].share = 1;
 	return 0;
 }
 
 static __rte_always_inline void
 mlx5_hws_cnt_shared_put(struct mlx5_hws_cnt_pool *cpool, cnt_id_t *cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
 
-	cpool->pool[iidx].share = 0;
-	mlx5_hws_cnt_pool_put(cpool, NULL, cnt_id);
+	hpool->pool[iidx].share = 0;
+	mlx5_hws_cnt_pool_put(hpool, NULL, cnt_id);
 }
 
 static __rte_always_inline bool
 mlx5_hws_cnt_is_shared(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	return cpool->pool[iidx].share ? true : false;
+	return hpool->pool[iidx].share ? true : false;
 }
 
 static __rte_always_inline void
 mlx5_hws_cnt_age_set(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id,
 		     uint32_t age_idx)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	MLX5_ASSERT(cpool->pool[iidx].share);
-	cpool->pool[iidx].age_idx = age_idx;
+	MLX5_ASSERT(hpool->pool[iidx].share);
+	hpool->pool[iidx].age_idx = age_idx;
 }
 
 static __rte_always_inline uint32_t
 mlx5_hws_cnt_age_get(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	MLX5_ASSERT(cpool->pool[iidx].share);
-	return cpool->pool[iidx].age_idx;
+	MLX5_ASSERT(hpool->pool[iidx].share);
+	return hpool->pool[iidx].age_idx;
 }
 
 static __rte_always_inline cnt_id_t
@@ -645,34 +674,12 @@ mlx5_hws_age_is_indirect(uint32_t age_idx)
 }
 
 /* init HWS counter pool. */
-struct mlx5_hws_cnt_pool *
-mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
-		       const struct mlx5_hws_cnt_pool_cfg *pcfg,
-		       const struct mlx5_hws_cache_param *ccfg);
-
-void
-mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool *cntp);
-
 int
 mlx5_hws_cnt_service_thread_create(struct mlx5_dev_ctx_shared *sh);
 
 void
 mlx5_hws_cnt_service_thread_destroy(struct mlx5_dev_ctx_shared *sh);
 
-int
-mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
-		struct mlx5_hws_cnt_pool *cpool);
-void
-mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
-		struct mlx5_hws_cnt_pool *cpool);
-
-int
-mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
-		struct mlx5_hws_cnt_pool *cpool);
-
-void
-mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool);
-
 struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 		const struct rte_flow_port_attr *pattr, uint16_t nb_queue);
-- 
2.18.1

[PATCH 7/9] app/testpmd: add host port parameter into flow config

[Viacheslav Ovsiienko]

Host port id parameter is added to "flow configure" command.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index 88108498e0..b88756903b 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -228,6 +228,7 @@ enum index {
 	CONFIG_METERS_NUMBER,
 	CONFIG_CONN_TRACK_NUMBER,
 	CONFIG_FLAGS,
+	CONFIG_HOST_PORT,
 
 	/* Indirect action arguments */
 	INDIRECT_ACTION_CREATE,
@@ -1098,6 +1099,7 @@ static const enum index next_config_attr[] = {
 	CONFIG_METERS_NUMBER,
 	CONFIG_CONN_TRACK_NUMBER,
 	CONFIG_FLAGS,
+	CONFIG_HOST_PORT,
 	END,
 	ZERO,
 };
@@ -2719,6 +2721,14 @@ static const struct token token_list[] = {
 		.args = ARGS(ARGS_ENTRY(struct buffer,
 					args.configure.port_attr.flags)),
 	},
+	[CONFIG_HOST_PORT] = {
+		.name = "host_port",
+		.help = "host port for shared objects",
+		.next = NEXT(next_config_attr,
+			     NEXT_ENTRY(COMMON_UNSIGNED)),
+		.args = ARGS(ARGS_ENTRY(struct buffer,
+					args.configure.port_attr.host_port_id)),
+	},
 	/* Top-level command. */
 	[PATTERN_TEMPLATE] = {
 		.name = "pattern_template",
-- 
2.18.1

[PATCH 8/9] app/testpmd: add shared indirect action support

[Viacheslav Ovsiienko]

The shared indirect action can be shared between ports,
action should be created on single port and the handle
can be used in the templates and flows on multiple ports,
example:

  flow configure 0 queues_number 1 queues_size 64 counters_number 64
  flow configure 1 queues_number 1 queues_size 64 counters_number 0 host_port 0 flags 1

  flow indirect_action 0 create ingress action_id 0 action count / end

  flow actions_template 0 create ingress actions_template_id 8
       template indirect 0 / queue index 0 / end
       mask count / queue index 0 / end

  flow actions_template 1 create ingress actions_template_id 18
       template shared_indirect 0 0 / queue index 0 / end
       mask count / queue index 0 / end

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 53 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 52 insertions(+), 1 deletion(-)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index b88756903b..734959ba9b 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -595,6 +595,8 @@ enum index {
 	ACTION_SAMPLE_INDEX,
 	ACTION_SAMPLE_INDEX_VALUE,
 	ACTION_INDIRECT,
+	ACTION_SHARED_INDIRECT,
+	INDIRECT_ACTION_PORT,
 	INDIRECT_ACTION_ID2PTR,
 	ACTION_MODIFY_FIELD,
 	ACTION_MODIFY_FIELD_OP,
@@ -1882,6 +1884,7 @@ static const enum index next_action[] = {
 	ACTION_AGE_UPDATE,
 	ACTION_SAMPLE,
 	ACTION_INDIRECT,
+	ACTION_SHARED_INDIRECT,
 	ACTION_MODIFY_FIELD,
 	ACTION_CONNTRACK,
 	ACTION_CONNTRACK_UPDATE,
@@ -2387,6 +2390,9 @@ static int parse_ia_destroy(struct context *ctx, const struct token *token,
 static int parse_ia_id2ptr(struct context *ctx, const struct token *token,
 			   const char *str, unsigned int len, void *buf,
 			   unsigned int size);
+static int parse_ia_port(struct context *ctx, const struct token *token,
+			 const char *str, unsigned int len, void *buf,
+			 unsigned int size);
 static int parse_mp(struct context *, const struct token *,
 		    const char *, unsigned int,
 		    void *, unsigned int);
@@ -6374,6 +6380,23 @@ static const struct token token_list[] = {
 		.args = ARGS(ARGS_ENTRY_ARB(0, sizeof(uint32_t))),
 		.call = parse_vc,
 	},
+	[ACTION_SHARED_INDIRECT] = {
+		.name = "shared_indirect",
+		.help = "apply indirect action by id and port",
+		.priv = PRIV_ACTION(INDIRECT, 0),
+		.next = NEXT(NEXT_ENTRY(INDIRECT_ACTION_PORT)),
+		.args = ARGS(ARGS_ENTRY_ARB(0, sizeof(uint32_t)),
+			     ARGS_ENTRY_ARB(0, sizeof(uint32_t))),
+		.call = parse_vc,
+	},
+	[INDIRECT_ACTION_PORT] = {
+		.name = "{indirect_action_port}",
+		.type = "INDIRECT_ACTION_PORT",
+		.help = "indirect action port",
+		.next = NEXT(NEXT_ENTRY(INDIRECT_ACTION_ID2PTR)),
+		.call = parse_ia_port,
+		.comp = comp_none,
+	},
 	[INDIRECT_ACTION_ID2PTR] = {
 		.name = "{action_id}",
 		.type = "INDIRECT_ACTION_ID",
@@ -9788,6 +9811,31 @@ parse_port(struct context *ctx, const struct token *token,
 	return ret;
 }
 
+/** Parse tokens for shared indirect actions. */
+static int
+parse_ia_port(struct context *ctx, const struct token *token,
+	      const char *str, unsigned int len,
+	      void *buf, unsigned int size)
+{
+	struct rte_flow_action *action = ctx->object;
+	uint32_t id;
+	int ret;
+
+	(void)buf;
+	(void)size;
+	ctx->objdata = 0;
+	ctx->object = &id;
+	ctx->objmask = NULL;
+	ret = parse_int(ctx, token, str, len, ctx->object, sizeof(id));
+	ctx->object = action;
+	if (ret != (int)len)
+		return ret;
+	/* set indirect action */
+	if (action)
+		action->conf = (void *)(uintptr_t)id;
+	return ret;
+}
+
 static int
 parse_ia_id2ptr(struct context *ctx, const struct token *token,
 		const char *str, unsigned int len,
@@ -9808,7 +9856,10 @@ parse_ia_id2ptr(struct context *ctx, const struct token *token,
 		return ret;
 	/* set indirect action */
 	if (action) {
-		action->conf = port_action_handle_get_by_id(ctx->port, id);
+		portid_t port_id = ctx->port;
+		if (ctx->prev == INDIRECT_ACTION_PORT)
+			port_id = (portid_t)(uintptr_t)action->conf;
+		action->conf = port_action_handle_get_by_id(port_id, id);
 		ret = (action->conf) ? ret : -1;
 	}
 	return ret;
-- 
2.18.1

[PATCH 9/9] doc: update cross-port indirect shared action

[Viacheslav Ovsiienko]

The patch updates the Release Notes and documentation
for the indirect RTE Flow actions as shared ones between
ports on the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 doc/guides/rel_notes/release_23_03.rst      | 6 ++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst | 4 +++-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index c15f6fbb9f..eb6643bc66 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -55,6 +55,12 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added cross-port indirect action in asynchronous Flow API.**
+
+  * Added ``RTE_FLOW_PORT_FLAG_SHARE_INDIRECT`` flag option to
+    ``rte_flow_configure``.
+  * Added ``host_port_id`` to ``rte_flow_port_attr`` structure.
+
 * **Updated Intel QuickAssist Technology (QAT) crypto driver.**
 
   * Added support for SHA3 224/256/384/512 plain hash in QAT GEN 3.
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 0037506a79..cf6c214197 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -3047,7 +3047,9 @@ for asynchronous flow creation/destruction operations. It is bound to
        [queues_number {number}] [queues_size {size}]
        [counters_number {number}]
        [aging_counters_number {number}]
-       [meters_number {number}] [flags {number}]
+       [host_port {number}]
+       [meters_number {number}]
+       [flags {number}]
 
 If successful, it will show::
 
-- 
2.18.1

[PATCH v2 1/9] ethdev: sharing indirect actions between ports

[Viacheslav Ovsiienko]

The RTE Flow API implements the concept of shared objects,
known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
An application can create the indirect action of desired
type and configuration with rte_flow_action_handle_create
call and then specify the obtained action handle in multiple
flows.

The initial concept supposes the action handle has strict
attachment to the port it was created on and to be used
exclusively in the flows being installed on the port.

Nowadays the multipath network topologies are quite common,
packets belonging to the same connection might arrive and
be sent over multiple ports, and there is the raising demand
to handle these "spread" connections. To fulfil this demand
it is proposed to extend indirect action sharing across the
multiple ports. This kind of sharing would be extremely useful
for the meters and counters, allowing to manage the single
connection over the multiple ports.

This cross-port object sharing is hard to implement in
generic way merely with software on the upper layers, but
can be provided by the driver over the single hardware
instance, where  multiple ports reside on the same physical
NIC and share the same hardware context.

To allow this action sharing application should specify
the "host port" during flow configuring to claim the intention
to share the indirect actions. All indirect actions reside within
"host port" context and can be shared in flows being installed
on the host port and on all the ports referencing this one.

If sharing between host and port being configured is not supported
the configuration should be rejected with error. There might be
multiple independent (mutual exclusive) sharing domains with
dedicated host and referencing ports.

To manage the shared indirect action any port from sharing domain
can be specified. To share or not the created action is up to
application, no API change is needed.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 lib/ethdev/rte_flow.c |  6 ++++++
 lib/ethdev/rte_flow.h | 11 +++++++++++
 2 files changed, 17 insertions(+)

diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
index 7d0c24366c..692d37925a 100644
--- a/lib/ethdev/rte_flow.c
+++ b/lib/ethdev/rte_flow.c
@@ -1476,6 +1476,12 @@ rte_flow_configure(uint16_t port_id,
 		RTE_FLOW_LOG(ERR, "Port %"PRIu16" queue info is NULL.\n", port_id);
 		return -EINVAL;
 	}
+	if ((port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) &&
+	     !rte_eth_dev_is_valid_port(port_attr->host_port_id)) {
+		return rte_flow_error_set(error, ENODEV,
+					  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+					  NULL, rte_strerror(ENODEV));
+	}
 	if (likely(!!ops->configure)) {
 		ret = ops->configure(dev, port_attr, nb_queue, queue_attr, error);
 		if (ret == 0)
diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index b60987db4b..89d2394bd8 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -4903,6 +4903,13 @@ rte_flow_info_get(uint16_t port_id,
 		  struct rte_flow_queue_info *queue_info,
 		  struct rte_flow_error *error);
 
+/**
+ * Indicate all steering objects should be created on contexts
+ * of the host port, providing indirect object sharing between
+ * ports.
+ */
+#define RTE_FLOW_PORT_FLAG_SHARE_INDIRECT RTE_BIT32(0)
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
@@ -4932,6 +4939,10 @@ struct rte_flow_port_attr {
 	 * @see RTE_FLOW_ACTION_TYPE_CONNTRACK
 	 */
 	uint32_t nb_conn_tracks;
+	/**
+	 * Port to base shared objects on.
+	 */
+	uint16_t host_port_id;
 	/**
 	 * Port flags (RTE_FLOW_PORT_FLAG_*).
 	 */
-- 
2.18.1

[PATCH v2 2/9] app/testpmd: add host port parameter into flow config

[Viacheslav Ovsiienko]

Host port id parameter is added to "flow configure" command.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index 88108498e0..b88756903b 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -228,6 +228,7 @@ enum index {
 	CONFIG_METERS_NUMBER,
 	CONFIG_CONN_TRACK_NUMBER,
 	CONFIG_FLAGS,
+	CONFIG_HOST_PORT,
 
 	/* Indirect action arguments */
 	INDIRECT_ACTION_CREATE,
@@ -1098,6 +1099,7 @@ static const enum index next_config_attr[] = {
 	CONFIG_METERS_NUMBER,
 	CONFIG_CONN_TRACK_NUMBER,
 	CONFIG_FLAGS,
+	CONFIG_HOST_PORT,
 	END,
 	ZERO,
 };
@@ -2719,6 +2721,14 @@ static const struct token token_list[] = {
 		.args = ARGS(ARGS_ENTRY(struct buffer,
 					args.configure.port_attr.flags)),
 	},
+	[CONFIG_HOST_PORT] = {
+		.name = "host_port",
+		.help = "host port for shared objects",
+		.next = NEXT(next_config_attr,
+			     NEXT_ENTRY(COMMON_UNSIGNED)),
+		.args = ARGS(ARGS_ENTRY(struct buffer,
+					args.configure.port_attr.host_port_id)),
+	},
 	/* Top-level command. */
 	[PATTERN_TEMPLATE] = {
 		.name = "pattern_template",
-- 
2.18.1

RE: [PATCH v2 2/9] app/testpmd: add host port parameter into flow config

[Ori Kam]

Hi Slava,

> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Tuesday, 7 February 2023 16:02
> 
> Host port id parameter is added to "flow configure" command.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---

Acked-by: Ori Kam <orika@nvidia.com>
Best,
Ori

[PATCH v2 3/9] app/testpmd: add shared indirect action support

[Viacheslav Ovsiienko]

The shared indirect action can be shared between ports,
action should be created on single port and the handle
can be used in the templates and flows on multiple ports,
example:

  flow configure 0 queues_number 1 queues_size 64 counters_number 64
  flow configure 1 queues_number 1 queues_size 64 counters_number 0 \
                   host_port 0 flags 1

  flow indirect_action 0 create ingress action_id 0 action count / end

  flow actions_template 0 create ingress actions_template_id 8
       template indirect 0 / queue index 0 / end
       mask count / queue index 0 / end

  flow actions_template 1 create ingress actions_template_id 18
       template shared_indirect 0 0 / queue index 0 / end
       mask count / queue index 0 / end

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 53 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 52 insertions(+), 1 deletion(-)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index b88756903b..734959ba9b 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -595,6 +595,8 @@ enum index {
 	ACTION_SAMPLE_INDEX,
 	ACTION_SAMPLE_INDEX_VALUE,
 	ACTION_INDIRECT,
+	ACTION_SHARED_INDIRECT,
+	INDIRECT_ACTION_PORT,
 	INDIRECT_ACTION_ID2PTR,
 	ACTION_MODIFY_FIELD,
 	ACTION_MODIFY_FIELD_OP,
@@ -1882,6 +1884,7 @@ static const enum index next_action[] = {
 	ACTION_AGE_UPDATE,
 	ACTION_SAMPLE,
 	ACTION_INDIRECT,
+	ACTION_SHARED_INDIRECT,
 	ACTION_MODIFY_FIELD,
 	ACTION_CONNTRACK,
 	ACTION_CONNTRACK_UPDATE,
@@ -2387,6 +2390,9 @@ static int parse_ia_destroy(struct context *ctx, const struct token *token,
 static int parse_ia_id2ptr(struct context *ctx, const struct token *token,
 			   const char *str, unsigned int len, void *buf,
 			   unsigned int size);
+static int parse_ia_port(struct context *ctx, const struct token *token,
+			 const char *str, unsigned int len, void *buf,
+			 unsigned int size);
 static int parse_mp(struct context *, const struct token *,
 		    const char *, unsigned int,
 		    void *, unsigned int);
@@ -6374,6 +6380,23 @@ static const struct token token_list[] = {
 		.args = ARGS(ARGS_ENTRY_ARB(0, sizeof(uint32_t))),
 		.call = parse_vc,
 	},
+	[ACTION_SHARED_INDIRECT] = {
+		.name = "shared_indirect",
+		.help = "apply indirect action by id and port",
+		.priv = PRIV_ACTION(INDIRECT, 0),
+		.next = NEXT(NEXT_ENTRY(INDIRECT_ACTION_PORT)),
+		.args = ARGS(ARGS_ENTRY_ARB(0, sizeof(uint32_t)),
+			     ARGS_ENTRY_ARB(0, sizeof(uint32_t))),
+		.call = parse_vc,
+	},
+	[INDIRECT_ACTION_PORT] = {
+		.name = "{indirect_action_port}",
+		.type = "INDIRECT_ACTION_PORT",
+		.help = "indirect action port",
+		.next = NEXT(NEXT_ENTRY(INDIRECT_ACTION_ID2PTR)),
+		.call = parse_ia_port,
+		.comp = comp_none,
+	},
 	[INDIRECT_ACTION_ID2PTR] = {
 		.name = "{action_id}",
 		.type = "INDIRECT_ACTION_ID",
@@ -9788,6 +9811,31 @@ parse_port(struct context *ctx, const struct token *token,
 	return ret;
 }
 
+/** Parse tokens for shared indirect actions. */
+static int
+parse_ia_port(struct context *ctx, const struct token *token,
+	      const char *str, unsigned int len,
+	      void *buf, unsigned int size)
+{
+	struct rte_flow_action *action = ctx->object;
+	uint32_t id;
+	int ret;
+
+	(void)buf;
+	(void)size;
+	ctx->objdata = 0;
+	ctx->object = &id;
+	ctx->objmask = NULL;
+	ret = parse_int(ctx, token, str, len, ctx->object, sizeof(id));
+	ctx->object = action;
+	if (ret != (int)len)
+		return ret;
+	/* set indirect action */
+	if (action)
+		action->conf = (void *)(uintptr_t)id;
+	return ret;
+}
+
 static int
 parse_ia_id2ptr(struct context *ctx, const struct token *token,
 		const char *str, unsigned int len,
@@ -9808,7 +9856,10 @@ parse_ia_id2ptr(struct context *ctx, const struct token *token,
 		return ret;
 	/* set indirect action */
 	if (action) {
-		action->conf = port_action_handle_get_by_id(ctx->port, id);
+		portid_t port_id = ctx->port;
+		if (ctx->prev == INDIRECT_ACTION_PORT)
+			port_id = (portid_t)(uintptr_t)action->conf;
+		action->conf = port_action_handle_get_by_id(port_id, id);
 		ret = (action->conf) ? ret : -1;
 	}
 	return ret;
-- 
2.18.1

RE: [PATCH v2 3/9] app/testpmd: add shared indirect action support

[Ori Kam]

Hi Slava,

> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Tuesday, 7 February 2023 16:02
> Subject: [PATCH v2 3/9] app/testpmd: add shared indirect action support
> 
> The shared indirect action can be shared between ports,
> action should be created on single port and the handle
> can be used in the templates and flows on multiple ports,
> example:
> 
>   flow configure 0 queues_number 1 queues_size 64 counters_number 64
>   flow configure 1 queues_number 1 queues_size 64 counters_number 0 \
>                    host_port 0 flags 1
> 
>   flow indirect_action 0 create ingress action_id 0 action count / end
> 
>   flow actions_template 0 create ingress actions_template_id 8
>        template indirect 0 / queue index 0 / end
>        mask count / queue index 0 / end
> 
>   flow actions_template 1 create ingress actions_template_id 18
>        template shared_indirect 0 0 / queue index 0 / end
>        mask count / queue index 0 / end
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---

Acked-by: Ori Kam <orika@nvidia.com>
Best,
Ori

[PATCH v2 4/9] net/mlx5/hws: free FT from RTC id before set the new value

[Viacheslav Ovsiienko]

From: Erez Shitrit <erezsh@nvidia.com>

While matcher is being connect/disconnect in shared gvmi flow we set the
first ft in the table to point on the first matcher,
The FW is increasing the refcount on the first matcher RTC because of
that no matcher if it is the same RTC that was set before, and when we
will try to release that RTC we will get the following syndrome:
0xaa0093 -   destroy_rtc_object: rtc in use or doesn't exist.

In order to resolve that we clean the current pointed RTC from that ft
and only after that setting it to the new RTC value.

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/hws/mlx5dr_matcher.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c
index d509a2f0e1..32bf3f5886 100644
--- a/drivers/net/mlx5/hws/mlx5dr_matcher.c
+++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c
@@ -334,6 +334,24 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher)
 		return ret;
 	}
 
+	if (!next) {
+		/* ft no longer points to any RTC, drop refcount */
+		ret = mlx5dr_matcher_free_rtc_pointing(tbl->ctx,
+						       tbl->fw_ft_type,
+						       tbl->type,
+						       prev_ft);
+		if (ret) {
+			DR_LOG(ERR, "Failed to reset last RTC refcount");
+			return ret;
+		}
+	}
+
+	ret = mlx5dr_matcher_shared_update_local_ft(tbl);
+	if (ret) {
+		DR_LOG(ERR, "Failed to update local_ft in shared table");
+		return ret;
+	}
+
 	return 0;
 }
 
-- 
2.18.1

[PATCH v2 5/9] net/mlx5/hws: fix disconnecting matcher

[Viacheslav Ovsiienko]

From: Erez Shitrit <erezsh@nvidia.com>

This patch fixes the matcher disconnection handling, by removing the RTC
references from flow table if the currently removed matcher was the last
one for the given table. As a result RTC in this matcher can be
correctly freed, since there are no dangling references to the RTC.

Fixes: c467608215b2 ("net/mlx5/hws: add matcher object")
Cc: stable@dpdk.org

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Reviewed-by: Alex Vesker <valex@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/net/mlx5/hws/mlx5dr_matcher.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c
index 32bf3f5886..a0297c9787 100644
--- a/drivers/net/mlx5/hws/mlx5dr_matcher.c
+++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c
@@ -346,12 +346,6 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher)
 		}
 	}
 
-	ret = mlx5dr_matcher_shared_update_local_ft(tbl);
-	if (ret) {
-		DR_LOG(ERR, "Failed to update local_ft in shared table");
-		return ret;
-	}
-
 	return 0;
 }
 
-- 
2.18.1

[PATCH v2 6/9] common/mlx5: add cross port object sharing capability

[Viacheslav Ovsiienko]

Add query port capabilities to share steering objects
between multiple ports of the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 13 +++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index e3a4927d0f..17128035ec 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -1047,6 +1047,19 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		attr->flow_counter_bulk_log_granularity =
 			MLX5_GET(cmd_hca_cap_2, hcattr,
 				 flow_counter_bulk_log_granularity);
+		rc = MLX5_GET(cmd_hca_cap_2, hcattr,
+			      cross_vhca_object_to_object_supported);
+		attr->cross_vhca =
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_FT) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC);
+		rc = MLX5_GET(cmd_hca_cap_2, hcattr,
+			      allowed_object_for_other_vhca_access);
+		attr->cross_vhca = attr->cross_vhca &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR) &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_FT) &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC);
 	}
 	if (attr->log_min_stride_wqe_sz == 0)
 		attr->log_min_stride_wqe_sz = MLX5_MPRQ_LOG_MIN_STRIDE_WQE_SIZE;
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index c94b9eac06..b65ba569bc 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -288,6 +288,7 @@ struct mlx5_hca_attr {
 	uint32_t alloc_flow_counter_pd:1;
 	uint32_t flow_counter_access_aso:1;
 	uint32_t flow_access_aso_opc_mod:8;
+	uint32_t cross_vhca:1;
 };
 
 /* LAG Context. */
-- 
2.18.1

[PATCH v2 7/9] net/mlx5: add cross port shared mode for HW steering

[Viacheslav Ovsiienko]

Add host port option for sharing steering objects between
multiple ports of the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/mlx5.c         |  6 +++
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow_hw.c | 78 +++++++++++++++++++++++++++++++--
 drivers/net/mlx5/mlx5_hws_cnt.c | 12 +++++
 4 files changed, 94 insertions(+), 4 deletions(-)

diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b8643cebdd..2eca2cceef 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2013,6 +2013,12 @@ mlx5_dev_close(struct rte_eth_dev *dev)
 	}
 	if (!priv->sh)
 		return 0;
+	if (priv->shared_refcnt) {
+		DRV_LOG(ERR, "port %u is shared host in use (%u)",
+			dev->data->port_id, priv->shared_refcnt);
+		rte_errno = EBUSY;
+		return -EBUSY;
+	}
 	DRV_LOG(DEBUG, "port %u closing device \"%s\"",
 		dev->data->port_id,
 		((priv->sh->cdev->ctx != NULL) ?
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 16b33e1548..525bdd47f7 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1780,6 +1780,8 @@ struct mlx5_priv {
 	struct mlx5_flow_hw_ctrl_rx *hw_ctrl_rx;
 	/**< HW steering templates used to create control flow rules. */
 #endif
+	struct rte_eth_dev *shared_host; /* Host device for HW steering. */
+	uint16_t shared_refcnt; /* HW steering host reference counter. */
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 20c71ff7f0..59b5d1980c 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -5,6 +5,8 @@
 #include <rte_flow.h>
 
 #include <mlx5_malloc.h>
+
+#include "mlx5.h"
 #include "mlx5_defs.h"
 #include "mlx5_flow.h"
 #include "mlx5_rx.h"
@@ -6301,6 +6303,12 @@ flow_hw_ct_pool_create(struct rte_eth_dev *dev,
 	int reg_id;
 	uint32_t flags;
 
+	if (port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Connection tracking is not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return NULL;
+	}
 	pool = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*pool), 0, SOCKET_ID_ANY);
 	if (!pool) {
 		rte_errno = ENOMEM;
@@ -6785,6 +6793,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		  struct rte_flow_error *error)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_priv *host_priv = NULL;
 	struct mlx5dr_context *dr_ctx = NULL;
 	struct mlx5dr_context_attr dr_ctx_attr = {0};
 	struct mlx5_hw_q *hw_q;
@@ -6799,7 +6808,8 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		.free = mlx5_free,
 		.type = "mlx5_hw_action_construct_data",
 	};
-	/* Adds one queue to be used by PMD.
+	/*
+	 * Adds one queue to be used by PMD.
 	 * The last queue will be used by the PMD.
 	 */
 	uint16_t nb_q_updated = 0;
@@ -6918,6 +6928,57 @@ flow_hw_configure(struct rte_eth_dev *dev,
 	dr_ctx_attr.queues = nb_q_updated;
 	/* Queue size should all be the same. Take the first one. */
 	dr_ctx_attr.queue_size = _queue_attr[0]->size;
+	if (port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		struct rte_eth_dev *host_dev = NULL;
+		uint16_t port_id;
+
+		MLX5_ASSERT(rte_eth_dev_is_valid_port(port_attr->host_port_id));
+		if (is_proxy) {
+			DRV_LOG(ERR, "cross vHCA shared mode not supported "
+				     " for E-Switch confgiurations");
+			rte_errno = ENOTSUP;
+			goto err;
+		}
+		MLX5_ETH_FOREACH_DEV(port_id, dev->device) {
+			if (port_id == port_attr->host_port_id) {
+				host_dev = &rte_eth_devices[port_id];
+				break;
+			}
+		}
+		if (!host_dev || host_dev == dev ||
+		    !host_dev->data || !host_dev->data->dev_private) {
+			DRV_LOG(ERR, "Invalid cross vHCA host port %u",
+				port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		host_priv = host_dev->data->dev_private;
+		if (host_priv->sh->cdev->ctx == priv->sh->cdev->ctx) {
+			DRV_LOG(ERR, "Sibling ports %u and %u do not "
+				     "require cross vHCA sharing mode",
+				dev->data->port_id, port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		if (host_priv->shared_host) {
+			DRV_LOG(ERR, "Host port %u is not the sharing base",
+				port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		if (port_attr->nb_counters ||
+		    port_attr->nb_aging_objects ||
+		    port_attr->nb_meters ||
+		    port_attr->nb_conn_tracks) {
+			DRV_LOG(ERR,
+				"Object numbers on guest port must be zeros");
+			rte_errno = EINVAL;
+			goto err;
+		}
+		dr_ctx_attr.shared_ibv_ctx = host_priv->sh->cdev->ctx;
+		priv->shared_host = host_dev;
+		__atomic_fetch_add(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+	}
 	dr_ctx = mlx5dr_context_open(priv->sh->cdev->ctx, &dr_ctx_attr);
 	/* rte_errno has been updated by HWS layer. */
 	if (!dr_ctx)
@@ -6933,7 +6994,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		goto err;
 	}
 	/* Initialize meter library*/
-	if (port_attr->nb_meters)
+	if (port_attr->nb_meters || (host_priv && host_priv->hws_mpool))
 		if (mlx5_flow_meter_init(dev, port_attr->nb_meters, 1, 1, nb_q_updated))
 			goto err;
 	/* Add global actions. */
@@ -6970,7 +7031,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 			goto err;
 		}
 	}
-	if (port_attr->nb_conn_tracks) {
+	if (port_attr->nb_conn_tracks || (host_priv && host_priv->hws_ctpool)) {
 		mem_size = sizeof(struct mlx5_aso_sq) * nb_q_updated +
 			   sizeof(*priv->ct_mng);
 		priv->ct_mng = mlx5_malloc(MLX5_MEM_ZERO, mem_size,
@@ -6984,7 +7045,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 			goto err;
 		priv->sh->ct_aso_en = 1;
 	}
-	if (port_attr->nb_counters) {
+	if (port_attr->nb_counters || (host_priv && host_priv->hws_cpool)) {
 		priv->hws_cpool = mlx5_hws_cnt_pool_create(dev, port_attr,
 							   nb_queue);
 		if (priv->hws_cpool == NULL)
@@ -7053,6 +7114,10 @@ flow_hw_configure(struct rte_eth_dev *dev,
 	}
 	if (_queue_attr)
 		mlx5_free(_queue_attr);
+	if (priv->shared_host) {
+		__atomic_fetch_sub(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+		priv->shared_host = NULL;
+	}
 	/* Do not overwrite the internal errno information. */
 	if (ret)
 		return ret;
@@ -7131,6 +7196,11 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
 	mlx5_free(priv->hw_q);
 	priv->hw_q = NULL;
 	claim_zero(mlx5dr_context_close(priv->dr_ctx));
+	if (priv->shared_host) {
+		struct mlx5_priv *host_priv = priv->shared_host->data->dev_private;
+		__atomic_fetch_sub(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+		priv->shared_host = NULL;
+	}
 	priv->dr_ctx = NULL;
 	priv->nb_queue = 0;
 }
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.c b/drivers/net/mlx5/mlx5_hws_cnt.c
index 51704ef754..afc93821e4 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.c
+++ b/drivers/net/mlx5/mlx5_hws_cnt.c
@@ -618,6 +618,12 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	int ret = 0;
 	size_t sz;
 
+	if (pattr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Counters are not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return NULL;
+	}
 	/* init cnt service if not. */
 	if (priv->sh->cnt_svc == NULL) {
 		ret = mlx5_hws_cnt_svc_init(priv->sh);
@@ -1189,6 +1195,12 @@ mlx5_hws_age_pool_init(struct rte_eth_dev *dev,
 
 	strict_queue = !!(attr->flags & RTE_FLOW_PORT_FLAG_STRICT_QUEUE);
 	MLX5_ASSERT(priv->hws_cpool);
+	if (attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Aging sn not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
 	nb_alloc_cnts = mlx5_hws_cnt_pool_get_size(priv->hws_cpool);
 	if (strict_queue) {
 		rsize = mlx5_hws_aged_out_q_ring_size_get(nb_alloc_cnts,
-- 
2.18.1

[PATCH v2 8/9] net/mlx5: support counters in cross port shared mode

[Viacheslav Ovsiienko]

In the cross vHCA sharing mode the host counter pool
should be used in counter related routines. The local
port pool is used to store the dedicated DR action
handle, per queue counter caches and query data are
ignored and not allocated on local pool.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/mlx5_flow_hw.c |  12 ++-
 drivers/net/mlx5/mlx5_hws_cnt.c | 163 ++++++++++++++++----------------
 drivers/net/mlx5/mlx5_hws_cnt.h | 109 +++++++++++----------
 3 files changed, 150 insertions(+), 134 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 59b5d1980c..b9405669c8 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -2311,8 +2311,10 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
 				break;
 			/* Fall-through. */
 		case RTE_FLOW_ACTION_TYPE_COUNT:
-			ret = mlx5_hws_cnt_pool_get(priv->hws_cpool, &queue,
-						    &cnt_id, age_idx);
+			ret = mlx5_hws_cnt_pool_get(priv->hws_cpool,
+					(priv->shared_refcnt ||
+					 priv->hws_cpool->cfg.host_cpool) ?
+					NULL : &queue, &cnt_id, age_idx);
 			if (ret != 0)
 				return ret;
 			ret = mlx5_hws_cnt_pool_get_action_offset
@@ -7996,6 +7998,7 @@ static int
 flow_hw_query_counter(const struct rte_eth_dev *dev, uint32_t counter,
 		      void *data, struct rte_flow_error *error)
 {
+	struct mlx5_hws_cnt_pool *hpool;
 	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_hws_cnt *cnt;
 	struct rte_flow_query_count *qc = data;
@@ -8006,8 +8009,9 @@ flow_hw_query_counter(const struct rte_eth_dev *dev, uint32_t counter,
 		return rte_flow_error_set(error, EINVAL,
 				RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
 				"counter are not available");
-	iidx = mlx5_hws_cnt_iidx(priv->hws_cpool, counter);
-	cnt = &priv->hws_cpool->pool[iidx];
+	hpool = mlx5_hws_cnt_host_pool(priv->hws_cpool);
+	iidx = mlx5_hws_cnt_iidx(hpool, counter);
+	cnt = &hpool->pool[iidx];
 	__hws_cnt_query_raw(priv->hws_cpool, counter, &pkts, &bytes);
 	qc->hits_set = 1;
 	qc->bytes_set = 1;
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.c b/drivers/net/mlx5/mlx5_hws_cnt.c
index afc93821e4..12b2cb0b27 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.c
+++ b/drivers/net/mlx5/mlx5_hws_cnt.c
@@ -24,12 +24,8 @@
 static void
 __hws_cnt_id_load(struct mlx5_hws_cnt_pool *cpool)
 {
-	uint32_t preload;
-	uint32_t q_num = cpool->cache->q_num;
 	uint32_t cnt_num = mlx5_hws_cnt_pool_get_size(cpool);
-	cnt_id_t cnt_id;
-	uint32_t qidx, iidx = 0;
-	struct rte_ring *qcache = NULL;
+	uint32_t iidx;
 
 	/*
 	 * Counter ID order is important for tracking the max number of in used
@@ -39,18 +35,9 @@ __hws_cnt_id_load(struct mlx5_hws_cnt_pool *cpool)
 	 * and then the global free list.
 	 * In the end, user fetch the counter from minimal to the maximum.
 	 */
-	preload = RTE_MIN(cpool->cache->preload_sz, cnt_num / q_num);
-	for (qidx = 0; qidx < q_num; qidx++) {
-		for (; iidx < preload * (qidx + 1); iidx++) {
-			cnt_id = mlx5_hws_cnt_id_gen(cpool, iidx);
-			qcache = cpool->cache->qcache[qidx];
-			if (qcache)
-				rte_ring_enqueue_elem(qcache, &cnt_id,
-						sizeof(cnt_id));
-		}
-	}
-	for (; iidx < cnt_num; iidx++) {
-		cnt_id = mlx5_hws_cnt_id_gen(cpool, iidx);
+	for (iidx = 0; iidx < cnt_num; iidx++) {
+		cnt_id_t cnt_id  = mlx5_hws_cnt_id_gen(cpool, iidx);
+
 		rte_ring_enqueue_elem(cpool->free_list, &cnt_id,
 				sizeof(cnt_id));
 	}
@@ -334,7 +321,26 @@ mlx5_hws_cnt_svc(void *opaque)
 	return NULL;
 }
 
-struct mlx5_hws_cnt_pool *
+static void
+mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool * const cntp)
+{
+	uint32_t qidx = 0;
+	if (cntp == NULL)
+		return;
+	rte_ring_free(cntp->free_list);
+	rte_ring_free(cntp->wait_reset_list);
+	rte_ring_free(cntp->reuse_list);
+	if (cntp->cache) {
+		for (qidx = 0; qidx < cntp->cache->q_num; qidx++)
+			rte_ring_free(cntp->cache->qcache[qidx]);
+	}
+	mlx5_free(cntp->cache);
+	mlx5_free(cntp->raw_mng);
+	mlx5_free(cntp->pool);
+	mlx5_free(cntp);
+}
+
+static struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		       const struct mlx5_hws_cnt_pool_cfg *pcfg,
 		       const struct mlx5_hws_cache_param *ccfg)
@@ -352,6 +358,8 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		return NULL;
 
 	cntp->cfg = *pcfg;
+	if (cntp->cfg.host_cpool)
+		return cntp;
 	cntp->cache = mlx5_malloc(MLX5_MEM_ANY | MLX5_MEM_ZERO,
 			sizeof(*cntp->cache) +
 			sizeof(((struct mlx5_hws_cnt_pool_caches *)0)->qcache[0])
@@ -387,8 +395,9 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		goto error;
 	snprintf(mz_name, sizeof(mz_name), "%s_F_RING", pcfg->name);
 	cntp->free_list = rte_ring_create_elem(mz_name, sizeof(cnt_id_t),
-			(uint32_t)cnt_num, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
+				(uint32_t)cnt_num, SOCKET_ID_ANY,
+				RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ |
+				RING_F_EXACT_SZ);
 	if (cntp->free_list == NULL) {
 		DRV_LOG(ERR, "failed to create free list ring");
 		goto error;
@@ -404,7 +413,7 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 	snprintf(mz_name, sizeof(mz_name), "%s_U_RING", pcfg->name);
 	cntp->reuse_list = rte_ring_create_elem(mz_name, sizeof(cnt_id_t),
 			(uint32_t)cnt_num, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
+			RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
 	if (cntp->reuse_list == NULL) {
 		DRV_LOG(ERR, "failed to create reuse list ring");
 		goto error;
@@ -427,25 +436,6 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 	return NULL;
 }
 
-void
-mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool * const cntp)
-{
-	uint32_t qidx = 0;
-	if (cntp == NULL)
-		return;
-	rte_ring_free(cntp->free_list);
-	rte_ring_free(cntp->wait_reset_list);
-	rte_ring_free(cntp->reuse_list);
-	if (cntp->cache) {
-		for (qidx = 0; qidx < cntp->cache->q_num; qidx++)
-			rte_ring_free(cntp->cache->qcache[qidx]);
-	}
-	mlx5_free(cntp->cache);
-	mlx5_free(cntp->raw_mng);
-	mlx5_free(cntp->pool);
-	mlx5_free(cntp);
-}
-
 int
 mlx5_hws_cnt_service_thread_create(struct mlx5_dev_ctx_shared *sh)
 {
@@ -482,7 +472,7 @@ mlx5_hws_cnt_service_thread_destroy(struct mlx5_dev_ctx_shared *sh)
 	sh->cnt_svc->service_thread = 0;
 }
 
-int
+static int
 mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 			    struct mlx5_hws_cnt_pool *cpool)
 {
@@ -494,6 +484,7 @@ mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 	struct mlx5_devx_counter_attr attr = {0};
 	struct mlx5_devx_obj *dcs;
 
+	MLX5_ASSERT(cpool->cfg.host_cpool == NULL);
 	if (hca_attr->flow_counter_bulk_log_max_alloc == 0) {
 		DRV_LOG(ERR, "Fw doesn't support bulk log max alloc");
 		return -1;
@@ -549,7 +540,7 @@ mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 	return -1;
 }
 
-void
+static void
 mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
 			   struct mlx5_hws_cnt_pool *cpool)
 {
@@ -565,22 +556,39 @@ mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
 	}
 }
 
-int
+static void
+mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool)
+{
+	uint32_t idx;
+
+	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
+		struct mlx5_hws_cnt_dcs *dcs = &cpool->dcs_mng.dcs[idx];
+
+		if (dcs->dr_action != NULL) {
+			mlx5dr_action_destroy(dcs->dr_action);
+			dcs->dr_action = NULL;
+		}
+	}
+}
+
+static int
 mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
 		struct mlx5_hws_cnt_pool *cpool)
 {
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint32_t idx;
 	int ret = 0;
-	struct mlx5_hws_cnt_dcs *dcs;
 	uint32_t flags;
 
 	flags = MLX5DR_ACTION_FLAG_HWS_RX | MLX5DR_ACTION_FLAG_HWS_TX;
 	if (priv->sh->config.dv_esw_en && priv->master)
 		flags |= MLX5DR_ACTION_FLAG_HWS_FDB;
-	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
-		dcs = &cpool->dcs_mng.dcs[idx];
+	for (idx = 0; idx < hpool->dcs_mng.batch_total; idx++) {
+		struct mlx5_hws_cnt_dcs *hdcs = &hpool->dcs_mng.dcs[idx];
+		struct mlx5_hws_cnt_dcs *dcs = &cpool->dcs_mng.dcs[idx];
+
 		dcs->dr_action = mlx5dr_action_create_counter(priv->dr_ctx,
-					(struct mlx5dr_devx_obj *)dcs->obj,
+					(struct mlx5dr_devx_obj *)hdcs->obj,
 					flags);
 		if (dcs->dr_action == NULL) {
 			mlx5_hws_cnt_pool_action_destroy(cpool);
@@ -591,21 +599,6 @@ mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
 	return ret;
 }
 
-void
-mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool)
-{
-	uint32_t idx;
-	struct mlx5_hws_cnt_dcs *dcs;
-
-	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
-		dcs = &cpool->dcs_mng.dcs[idx];
-		if (dcs->dr_action != NULL) {
-			mlx5dr_action_destroy(dcs->dr_action);
-			dcs->dr_action = NULL;
-		}
-	}
-}
-
 struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 		const struct rte_flow_port_attr *pattr, uint16_t nb_queue)
@@ -618,11 +611,28 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	int ret = 0;
 	size_t sz;
 
+	mp_name = mlx5_malloc(MLX5_MEM_ZERO, RTE_MEMZONE_NAMESIZE, 0,
+			SOCKET_ID_ANY);
+	if (mp_name == NULL)
+		goto error;
+	snprintf(mp_name, RTE_MEMZONE_NAMESIZE, "MLX5_HWS_CNT_POOL_%u",
+			dev->data->port_id);
+	pcfg.name = mp_name;
+	pcfg.request_num = pattr->nb_counters;
+	pcfg.alloc_factor = HWS_CNT_ALLOC_FACTOR_DEFAULT;
 	if (pattr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
-		DRV_LOG(ERR, "Counters are not supported "
-			     "in cross vHCA sharing mode");
-		rte_errno = ENOTSUP;
-		return NULL;
+		struct mlx5_priv *host_priv =
+				priv->shared_host->data->dev_private;
+		struct mlx5_hws_cnt_pool *chost = host_priv->hws_cpool;
+
+		pcfg.host_cpool = chost;
+		cpool = mlx5_hws_cnt_pool_init(priv->sh, &pcfg, &cparam);
+		if (cpool == NULL)
+			goto error;
+		ret = mlx5_hws_cnt_pool_action_create(priv, cpool);
+		if (ret != 0)
+			goto error;
+		return cpool;
 	}
 	/* init cnt service if not. */
 	if (priv->sh->cnt_svc == NULL) {
@@ -635,15 +645,6 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	cparam.q_num = nb_queue;
 	cparam.threshold = HWS_CNT_CACHE_THRESHOLD_DEFAULT;
 	cparam.size = HWS_CNT_CACHE_SZ_DEFAULT;
-	pcfg.alloc_factor = HWS_CNT_ALLOC_FACTOR_DEFAULT;
-	mp_name = mlx5_malloc(MLX5_MEM_ZERO, RTE_MEMZONE_NAMESIZE, 0,
-			SOCKET_ID_ANY);
-	if (mp_name == NULL)
-		goto error;
-	snprintf(mp_name, RTE_MEMZONE_NAMESIZE, "MLX5_HWS_CNT_POOL_%u",
-			dev->data->port_id);
-	pcfg.name = mp_name;
-	pcfg.request_num = pattr->nb_counters;
 	cpool = mlx5_hws_cnt_pool_init(priv->sh, &pcfg, &cparam);
 	if (cpool == NULL)
 		goto error;
@@ -678,11 +679,15 @@ mlx5_hws_cnt_pool_destroy(struct mlx5_dev_ctx_shared *sh,
 {
 	if (cpool == NULL)
 		return;
-	if (--sh->cnt_svc->refcnt == 0)
-		mlx5_hws_cnt_svc_deinit(sh);
+	if (cpool->cfg.host_cpool == NULL) {
+		if (--sh->cnt_svc->refcnt == 0)
+			mlx5_hws_cnt_svc_deinit(sh);
+	}
 	mlx5_hws_cnt_pool_action_destroy(cpool);
-	mlx5_hws_cnt_pool_dcs_free(sh, cpool);
-	mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+	if (cpool->cfg.host_cpool == NULL) {
+		mlx5_hws_cnt_pool_dcs_free(sh, cpool);
+		mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+	}
 	mlx5_free((void *)cpool->cfg.name);
 	mlx5_hws_cnt_pool_deinit(cpool);
 }
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.h b/drivers/net/mlx5/mlx5_hws_cnt.h
index 030dcead86..d35d083eeb 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.h
+++ b/drivers/net/mlx5/mlx5_hws_cnt.h
@@ -86,6 +86,7 @@ struct mlx5_hws_cnt_pool_cfg {
 	char *name;
 	uint32_t request_num;
 	uint32_t alloc_factor;
+	struct mlx5_hws_cnt_pool *host_cpool;
 };
 
 struct mlx5_hws_cnt_pool_caches {
@@ -148,6 +149,22 @@ struct mlx5_hws_age_param {
 	void *context; /* Flow AGE context. */
 } __rte_packed __rte_cache_aligned;
 
+
+/**
+ * Return the actual counter pool should be used in cross vHCA sharing mode.
+ * as index of raw/cnt pool.
+ *
+ * @param cnt_id
+ *   The external counter id
+ * @return
+ *   Internal index
+ */
+static __always_inline struct mlx5_hws_cnt_pool *
+mlx5_hws_cnt_host_pool(struct mlx5_hws_cnt_pool *cpool)
+{
+	return cpool->cfg.host_cpool ? cpool->cfg.host_cpool : cpool;
+}
+
 /**
  * Translate counter id into internal index (start from 0), which can be used
  * as index of raw/cnt pool.
@@ -160,11 +177,12 @@ struct mlx5_hws_age_param {
 static __rte_always_inline uint32_t
 mlx5_hws_cnt_iidx(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint8_t dcs_idx = cnt_id >> MLX5_HWS_CNT_DCS_IDX_OFFSET;
 	uint32_t offset = cnt_id & MLX5_HWS_CNT_IDX_MASK;
 
 	dcs_idx &= MLX5_HWS_CNT_DCS_IDX_MASK;
-	return (cpool->dcs_mng.dcs[dcs_idx].iidx + offset);
+	return (hpool->dcs_mng.dcs[dcs_idx].iidx + offset);
 }
 
 /**
@@ -191,7 +209,8 @@ mlx5_hws_cnt_id_valid(cnt_id_t cnt_id)
 static __rte_always_inline cnt_id_t
 mlx5_hws_cnt_id_gen(struct mlx5_hws_cnt_pool *cpool, uint32_t iidx)
 {
-	struct mlx5_hws_cnt_dcs_mng *dcs_mng = &cpool->dcs_mng;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	struct mlx5_hws_cnt_dcs_mng *dcs_mng = &hpool->dcs_mng;
 	uint32_t idx;
 	uint32_t offset;
 	cnt_id_t cnt_id;
@@ -212,7 +231,8 @@ static __rte_always_inline void
 __hws_cnt_query_raw(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id,
 		uint64_t *raw_pkts, uint64_t *raw_bytes)
 {
-	struct mlx5_hws_cnt_raw_data_mng *raw_mng = cpool->raw_mng;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	struct mlx5_hws_cnt_raw_data_mng *raw_mng = hpool->raw_mng;
 	struct flow_counter_stats s[2];
 	uint8_t i = 0x1;
 	size_t stat_sz = sizeof(s[0]);
@@ -393,22 +413,23 @@ mlx5_hws_cnt_pool_put(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 		      cnt_id_t *cnt_id)
 {
 	unsigned int ret = 0;
+	struct mlx5_hws_cnt_pool *hpool;
 	struct rte_ring_zc_data zcdc = {0};
 	struct rte_ring_zc_data zcdr = {0};
 	struct rte_ring *qcache = NULL;
 	unsigned int wb_num = 0; /* cache write-back number. */
 	uint32_t iidx;
 
-	iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
-	MLX5_ASSERT(cpool->pool[iidx].in_used);
-	cpool->pool[iidx].in_used = false;
-	cpool->pool[iidx].query_gen_when_free =
-		__atomic_load_n(&cpool->query_gen, __ATOMIC_RELAXED);
-	if (likely(queue != NULL))
-		qcache = cpool->cache->qcache[*queue];
+	hpool = mlx5_hws_cnt_host_pool(cpool);
+	iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
+	hpool->pool[iidx].in_used = false;
+	hpool->pool[iidx].query_gen_when_free =
+		__atomic_load_n(&hpool->query_gen, __ATOMIC_RELAXED);
+	if (likely(queue != NULL) && cpool->cfg.host_cpool == NULL)
+		qcache = hpool->cache->qcache[*queue];
 	if (unlikely(qcache == NULL)) {
-		ret = rte_ring_enqueue_elem(cpool->wait_reset_list, cnt_id,
-					    sizeof(cnt_id_t));
+		ret = rte_ring_enqueue_elem(hpool->wait_reset_list, cnt_id,
+				sizeof(cnt_id_t));
 		MLX5_ASSERT(ret == 0);
 		return;
 	}
@@ -465,9 +486,10 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 	uint32_t iidx, query_gen = 0;
 	cnt_id_t tmp_cid = 0;
 
-	if (likely(queue != NULL))
+	if (likely(queue != NULL && cpool->cfg.host_cpool == NULL))
 		qcache = cpool->cache->qcache[*queue];
 	if (unlikely(qcache == NULL)) {
+		cpool = mlx5_hws_cnt_host_pool(cpool);
 		ret = rte_ring_dequeue_elem(cpool->reuse_list, &tmp_cid,
 				sizeof(cnt_id_t));
 		if (unlikely(ret != 0)) {
@@ -534,7 +556,9 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 static __rte_always_inline unsigned int
 mlx5_hws_cnt_pool_get_size(struct mlx5_hws_cnt_pool *cpool)
 {
-	return rte_ring_get_capacity(cpool->free_list);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+
+	return rte_ring_get_capacity(hpool->free_list);
 }
 
 static __rte_always_inline int
@@ -554,51 +578,56 @@ static __rte_always_inline int
 mlx5_hws_cnt_shared_get(struct mlx5_hws_cnt_pool *cpool, cnt_id_t *cnt_id,
 			uint32_t age_idx)
 {
-	int ret;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint32_t iidx;
+	int ret;
 
-	ret = mlx5_hws_cnt_pool_get(cpool, NULL, cnt_id, age_idx);
+	ret = mlx5_hws_cnt_pool_get(hpool, NULL, cnt_id, age_idx);
 	if (ret != 0)
 		return ret;
-	iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
-	cpool->pool[iidx].share = 1;
+	iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
+	hpool->pool[iidx].share = 1;
 	return 0;
 }
 
 static __rte_always_inline void
 mlx5_hws_cnt_shared_put(struct mlx5_hws_cnt_pool *cpool, cnt_id_t *cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
 
-	cpool->pool[iidx].share = 0;
-	mlx5_hws_cnt_pool_put(cpool, NULL, cnt_id);
+	hpool->pool[iidx].share = 0;
+	mlx5_hws_cnt_pool_put(hpool, NULL, cnt_id);
 }
 
 static __rte_always_inline bool
 mlx5_hws_cnt_is_shared(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	return cpool->pool[iidx].share ? true : false;
+	return hpool->pool[iidx].share ? true : false;
 }
 
 static __rte_always_inline void
 mlx5_hws_cnt_age_set(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id,
 		     uint32_t age_idx)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	MLX5_ASSERT(cpool->pool[iidx].share);
-	cpool->pool[iidx].age_idx = age_idx;
+	MLX5_ASSERT(hpool->pool[iidx].share);
+	hpool->pool[iidx].age_idx = age_idx;
 }
 
 static __rte_always_inline uint32_t
 mlx5_hws_cnt_age_get(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	MLX5_ASSERT(cpool->pool[iidx].share);
-	return cpool->pool[iidx].age_idx;
+	MLX5_ASSERT(hpool->pool[iidx].share);
+	return hpool->pool[iidx].age_idx;
 }
 
 static __rte_always_inline cnt_id_t
@@ -645,34 +674,12 @@ mlx5_hws_age_is_indirect(uint32_t age_idx)
 }
 
 /* init HWS counter pool. */
-struct mlx5_hws_cnt_pool *
-mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
-		       const struct mlx5_hws_cnt_pool_cfg *pcfg,
-		       const struct mlx5_hws_cache_param *ccfg);
-
-void
-mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool *cntp);
-
 int
 mlx5_hws_cnt_service_thread_create(struct mlx5_dev_ctx_shared *sh);
 
 void
 mlx5_hws_cnt_service_thread_destroy(struct mlx5_dev_ctx_shared *sh);
 
-int
-mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
-		struct mlx5_hws_cnt_pool *cpool);
-void
-mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
-		struct mlx5_hws_cnt_pool *cpool);
-
-int
-mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
-		struct mlx5_hws_cnt_pool *cpool);
-
-void
-mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool);
-
 struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 		const struct rte_flow_port_attr *pattr, uint16_t nb_queue);
-- 
2.18.1

[PATCH v2 9/9] doc: update cross-port indirect shared action

[Viacheslav Ovsiienko]

The patch updates the Release Notes and documentation
for the indirect RTE Flow actions as shared ones between
ports on the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 doc/guides/rel_notes/release_23_03.rst      | 6 ++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst | 4 +++-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index c15f6fbb9f..eb6643bc66 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -55,6 +55,12 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added cross-port indirect action in asynchronous Flow API.**
+
+  * Added ``RTE_FLOW_PORT_FLAG_SHARE_INDIRECT`` flag option to
+    ``rte_flow_configure``.
+  * Added ``host_port_id`` to ``rte_flow_port_attr`` structure.
+
 * **Updated Intel QuickAssist Technology (QAT) crypto driver.**
 
   * Added support for SHA3 224/256/384/512 plain hash in QAT GEN 3.
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 0037506a79..cf6c214197 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -3047,7 +3047,9 @@ for asynchronous flow creation/destruction operations. It is bound to
        [queues_number {number}] [queues_size {size}]
        [counters_number {number}]
        [aging_counters_number {number}]
-       [meters_number {number}] [flags {number}]
+       [host_port {number}]
+       [meters_number {number}]
+       [flags {number}]
 
 If successful, it will show::
 
-- 
2.18.1

RE: [PATCH v2 9/9] doc: update cross-port indirect shared action

[Ori Kam]

Hi Slava,

> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Tuesday, 7 February 2023 16:02
> 
> The patch updates the Release Notes and documentation
> for the indirect RTE Flow actions as shared ones between
> ports on the same physical NIC.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---


Acked-by: Ori Kam <orika@nvidia.com>
Best,
Ori

Re: [PATCH v2 9/9] doc: update cross-port indirect shared action

[Ferruh Yigit]

On 2/7/2023 2:02 PM, Viacheslav Ovsiienko wrote:
> The patch updates the Release Notes and documentation
> for the indirect RTE Flow actions as shared ones between
> ports on the same physical NIC.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---
>  doc/guides/rel_notes/release_23_03.rst      | 6 ++++++
>  doc/guides/testpmd_app_ug/testpmd_funcs.rst | 4 +++-
>  2 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
> index c15f6fbb9f..eb6643bc66 100644
> --- a/doc/guides/rel_notes/release_23_03.rst
> +++ b/doc/guides/rel_notes/release_23_03.rst
> @@ -55,6 +55,12 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =======================================================
>  
> +* **Added cross-port indirect action in asynchronous Flow API.**
> +
> +  * Added ``RTE_FLOW_PORT_FLAG_SHARE_INDIRECT`` flag option to
> +    ``rte_flow_configure``.
> +  * Added ``host_port_id`` to ``rte_flow_port_attr`` structure.
> +
>  * **Updated Intel QuickAssist Technology (QAT) crypto driver.**
>  
>    * Added support for SHA3 224/256/384/512 plain hash in QAT GEN 3.
> diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
> index 0037506a79..cf6c214197 100644
> --- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
> +++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
> @@ -3047,7 +3047,9 @@ for asynchronous flow creation/destruction operations. It is bound to
>         [queues_number {number}] [queues_size {size}]
>         [counters_number {number}]
>         [aging_counters_number {number}]
> -       [meters_number {number}] [flags {number}]
> +       [host_port {number}]
> +       [meters_number {number}]
> +       [flags {number}]
>  
>  If successful, it will show::
>  

Can you please distribute the documentation to the patches that actually
adds the documented feature?
So no need to have document only patch.

RE: [PATCH v2 1/9] ethdev: sharing indirect actions between ports

[Ori Kam]

Hi Slava


> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Tuesday, 7 February 2023 16:02
> 
> The RTE Flow API implements the concept of shared objects,
> known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> An application can create the indirect action of desired
> type and configuration with rte_flow_action_handle_create
> call and then specify the obtained action handle in multiple
> flows.
> 
> The initial concept supposes the action handle has strict
> attachment to the port it was created on and to be used
> exclusively in the flows being installed on the port.
> 
> Nowadays the multipath network topologies are quite common,
> packets belonging to the same connection might arrive and
> be sent over multiple ports, and there is the raising demand
> to handle these "spread" connections. To fulfil this demand
> it is proposed to extend indirect action sharing across the
> multiple ports. This kind of sharing would be extremely useful
> for the meters and counters, allowing to manage the single
> connection over the multiple ports.
> 
> This cross-port object sharing is hard to implement in
> generic way merely with software on the upper layers, but
> can be provided by the driver over the single hardware
> instance, where  multiple ports reside on the same physical
> NIC and share the same hardware context.
> 
> To allow this action sharing application should specify
> the "host port" during flow configuring to claim the intention
> to share the indirect actions. All indirect actions reside within
> "host port" context and can be shared in flows being installed
> on the host port and on all the ports referencing this one.
> 
> If sharing between host and port being configured is not supported
> the configuration should be rejected with error. There might be
> multiple independent (mutual exclusive) sharing domains with
> dedicated host and referencing ports.
> 
> To manage the shared indirect action any port from sharing domain
> can be specified. To share or not the created action is up to
> application, no API change is needed.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---

Series-acked-by:  Ori Kam <orika@nvidia.com>

Thanks,
Ori

RE: [PATCH v2 1/9] ethdev: sharing indirect actions between ports

[Ori Kam]

Hi Slava

> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Tuesday, 7 February 2023 16:02
> 
> The RTE Flow API implements the concept of shared objects,
> known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> An application can create the indirect action of desired
> type and configuration with rte_flow_action_handle_create
> call and then specify the obtained action handle in multiple
> flows.
> 
> The initial concept supposes the action handle has strict
> attachment to the port it was created on and to be used
> exclusively in the flows being installed on the port.
> 
> Nowadays the multipath network topologies are quite common,
> packets belonging to the same connection might arrive and
> be sent over multiple ports, and there is the raising demand
> to handle these "spread" connections. To fulfil this demand
> it is proposed to extend indirect action sharing across the
> multiple ports. This kind of sharing would be extremely useful
> for the meters and counters, allowing to manage the single
> connection over the multiple ports.
> 
> This cross-port object sharing is hard to implement in
> generic way merely with software on the upper layers, but
> can be provided by the driver over the single hardware
> instance, where  multiple ports reside on the same physical
> NIC and share the same hardware context.
> 
> To allow this action sharing application should specify
> the "host port" during flow configuring to claim the intention
> to share the indirect actions. All indirect actions reside within
> "host port" context and can be shared in flows being installed
> on the host port and on all the ports referencing this one.
> 
> If sharing between host and port being configured is not supported
> the configuration should be rejected with error. There might be
> multiple independent (mutual exclusive) sharing domains with
> dedicated host and referencing ports.
> 
> To manage the shared indirect action any port from sharing domain
> can be specified. To share or not the created action is up to
> application, no API change is needed.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---

Acked-by: Ori Kam <orika@nvidia.com>
Best,
Ori

Re: [PATCH v2 1/9] ethdev: sharing indirect actions between ports

[Ferruh Yigit]

On 2/7/2023 2:01 PM, Viacheslav Ovsiienko wrote:
> The RTE Flow API implements the concept of shared objects,
> known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> An application can create the indirect action of desired
> type and configuration with rte_flow_action_handle_create
> call and then specify the obtained action handle in multiple
> flows.
> 
> The initial concept supposes the action handle has strict
> attachment to the port it was created on and to be used
> exclusively in the flows being installed on the port.
> 
> Nowadays the multipath network topologies are quite common,
> packets belonging to the same connection might arrive and
> be sent over multiple ports, and there is the raising demand
> to handle these "spread" connections. To fulfil this demand
> it is proposed to extend indirect action sharing across the
> multiple ports. This kind of sharing would be extremely useful
> for the meters and counters, allowing to manage the single
> connection over the multiple ports.
> 
> This cross-port object sharing is hard to implement in
> generic way merely with software on the upper layers, but
> can be provided by the driver over the single hardware
> instance, where  multiple ports reside on the same physical
> NIC and share the same hardware context.
> 
> To allow this action sharing application should specify
> the "host port" during flow configuring to claim the intention
> to share the indirect actions. All indirect actions reside within
> "host port" context and can be shared in flows being installed
> on the host port and on all the ports referencing this one.
> 
> If sharing between host and port being configured is not supported
> the configuration should be rejected with error. There might be
> multiple independent (mutual exclusive) sharing domains with
> dedicated host and referencing ports.
> 
> To manage the shared indirect action any port from sharing domain
> can be specified. To share or not the created action is up to
> application, no API change is needed.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

Hi Viacheslav,

This set doesn't compile, it fails because of some undefined
variable/macro in driver code. By any chance can this patch based some
internal code?

Do you want to separate ethdev and driver patches into different sets,
so ehtdev part can go into next-net and driver part can be merged to mlx
tree afterwards.

RE: [PATCH v2 1/9] ethdev: sharing indirect actions between ports

[Slava Ovsiienko]

Hi, Ferruh

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@amd.com>
> Sent: Friday, February 10, 2023 4:35 PM
> To: Slava Ovsiienko <viacheslavo@nvidia.com>; dev@dpdk.org
> Cc: Ori Kam <orika@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com>;
> Matan Azrad <matan@nvidia.com>; arybchenko@solarflare.com
> Subject: Re: [PATCH v2 1/9] ethdev: sharing indirect actions between ports
> 
> This set doesn't compile, it fails because of some undefined variable/macro in
> driver code. By any chance can this patch based some internal code?
> 
> Do you want to separate ethdev and driver patches into different sets, so ehtdev
> part can go into next-net and driver part can be merged to mlx tree afterwards.

Yes, let's merge  ethdev part now and let me check what Is wrong with the rest
of patch set.

With best regards,
Slava

 

[PATCH v3 0/3] *ethdev: sharing indirect actions between port*

[Viacheslav Ovsiienko]

The RTE Flow API implements the concept of shared objects,
known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
An application can create the indirect action of desired
type and configuration with rte_flow_action_handle_create
call and then specify the obtained action handle in multiple
flows.

The initial concept supposes the action handle has strict
attachment to the port it was created on and to be used
exclusively in the flows being installed on the port.

Nowadays the multipath network topologies are quite common,
packets belonging to the same connection might arrive and
be sent over multiple ports, and there is the raising demand
to handle these "spread" connections. To fulfil this demand
it is proposed to extend indirect action sharing across the
multiple ports. This kind of sharing would be extremely useful
for the meters and counters, allowing to manage the single
connection over the multiple ports.

This cross-port object sharing is hard to implement in
generic way merely with software on the upper layers, but
can be provided by the driver over the single hardware
instance, where  multiple ports reside on the same physical
NIC and share the same hardware context.

To allow this action sharing application should specify
the "host port" during flow configuring to claim the intention
to share the indirect actions. All indirect actions reside within
"host port" context and can be shared in flows being installed
on the host port and on all the ports referencing this one.

If sharing between host and port being configured is not supported
the configuration should be rejected with error. There might be
multiple independent (mutual exclusive) sharing domains with
dedicated host and referencing ports.

To manage the shared indirect action any port from sharing domain
can be specified. To share or not the created action is up to
application, no API change is needed.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

Viacheslav Ovsiienko (3):
  ethdev: sharing indirect actions between ports
  app/testpmd: add host port parameter into flow config
  app/testpmd: add shared indirect action support

 app/test-pmd/cmdline_flow.c                 | 63 ++++++++++++++++++++-
 doc/guides/rel_notes/release_23_03.rst      |  6 ++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  4 +-
 lib/ethdev/rte_flow.c                       |  6 ++
 lib/ethdev/rte_flow.h                       | 11 ++++
 5 files changed, 88 insertions(+), 2 deletions(-)

-- 
2.18.1

---
RFC: https://inbox.dpdk.org/dev/20221228165433.18185-1-viacheslavo@nvidia.com/
v1:  https://inbox.dpdk.org/dev/20230206095229.23027-1-viacheslavo@nvidia.com/
v2:  https://inbox.dpdk.org/dev/20230207140206.29139-1-viacheslavo@nvidia.com/
     minor comilation issues, rebase, typos
v3:  series split to ethdev/testpmd and mlx5 PMD patch to be smoothly applied
     over dpdk-next/main. The mlx5 PMD seris is coming.

[PATCH v3 1/3] ethdev: sharing indirect actions between ports

[Viacheslav Ovsiienko]

The RTE Flow API implements the concept of shared objects,
known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
An application can create the indirect action of desired
type and configuration with rte_flow_action_handle_create
call and then specify the obtained action handle in multiple
flows.

The initial concept supposes the action handle has strict
attachment to the port it was created on and to be used
exclusively in the flows being installed on the port.

Nowadays the multipath network topologies are quite common,
packets belonging to the same connection might arrive and
be sent over multiple ports, and there is the raising demand
to handle these "spread" connections. To fulfil this demand
it is proposed to extend indirect action sharing across the
multiple ports. This kind of sharing would be extremely useful
for the meters and counters, allowing to manage the single
connection over the multiple ports.

This cross-port object sharing is hard to implement in
generic way merely with software on the upper layers, but
can be provided by the driver over the single hardware
instance, where  multiple ports reside on the same physical
NIC and share the same hardware context.

To allow this action sharing application should specify
the "host port" during flow configuring to claim the intention
to share the indirect actions. All indirect actions reside within
"host port" context and can be shared in flows being installed
on the host port and on all the ports referencing this one.

If sharing between host and port being configured is not supported
the configuration should be rejected with error. There might be
multiple independent (mutual exclusive) sharing domains with
dedicated host and referencing ports.

To manage the shared indirect action any port from sharing domain
can be specified. To share or not the created action is up to
application, no API change is needed.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 doc/guides/rel_notes/release_23_03.rst      |  6 ++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  4 +++-
 lib/ethdev/rte_flow.c                       |  6 ++++++
 lib/ethdev/rte_flow.h                       | 11 +++++++++++
 4 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 75c64cafa4..ea12c746e3 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -69,6 +69,12 @@ New Features
 
   * Added modify flex item: ``rte_flow_action_modify_data.flex_handle``.
 
+* **Added cross-port indirect action in asynchronous Flow API.**
+
+  * Added ``RTE_FLOW_PORT_FLAG_SHARE_INDIRECT`` flag option to
+    ``rte_flow_configure``.
+  * Added ``host_port_id`` to ``rte_flow_port_attr`` structure.
+
 * **Updated AMD axgbe driver.**
 
   * Added multi-process support.
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 725062e6b0..c448cf47b3 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -3076,7 +3076,9 @@ for asynchronous flow creation/destruction operations. It is bound to
        [queues_number {number}] [queues_size {size}]
        [counters_number {number}]
        [aging_counters_number {number}]
-       [meters_number {number}] [flags {number}]
+       [host_port {number}]
+       [meters_number {number}]
+       [flags {number}]
 
 If successful, it will show::
 
diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
index 57bec6f5e1..a5680daa19 100644
--- a/lib/ethdev/rte_flow.c
+++ b/lib/ethdev/rte_flow.c
@@ -1597,6 +1597,12 @@ rte_flow_configure(uint16_t port_id,
 		RTE_FLOW_LOG(ERR, "Port %"PRIu16" queue info is NULL.\n", port_id);
 		return -EINVAL;
 	}
+	if ((port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) &&
+	     !rte_eth_dev_is_valid_port(port_attr->host_port_id)) {
+		return rte_flow_error_set(error, ENODEV,
+					  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+					  NULL, rte_strerror(ENODEV));
+	}
 	if (likely(!!ops->configure)) {
 		ret = ops->configure(dev, port_attr, nb_queue, queue_attr, error);
 		if (ret == 0)
diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index f4d8b7d3d5..c78684efb0 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -4992,6 +4992,13 @@ rte_flow_info_get(uint16_t port_id,
 		  struct rte_flow_queue_info *queue_info,
 		  struct rte_flow_error *error);
 
+/**
+ * Indicate all steering objects should be created on contexts
+ * of the host port, providing indirect object sharing between
+ * ports.
+ */
+#define RTE_FLOW_PORT_FLAG_SHARE_INDIRECT RTE_BIT32(0)
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
@@ -5021,6 +5028,10 @@ struct rte_flow_port_attr {
 	 * @see RTE_FLOW_ACTION_TYPE_CONNTRACK
 	 */
 	uint32_t nb_conn_tracks;
+	/**
+	 * Port to base shared objects on.
+	 */
+	uint16_t host_port_id;
 	/**
 	 * Port flags (RTE_FLOW_PORT_FLAG_*).
 	 */
-- 
2.18.1

[PATCH v3 2/3] app/testpmd: add host port parameter into flow config

[Viacheslav Ovsiienko]

Host port id parameter is added to "flow configure" command.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index 3b846ae73f..7778828919 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -230,6 +230,7 @@ enum index {
 	CONFIG_METERS_NUMBER,
 	CONFIG_CONN_TRACK_NUMBER,
 	CONFIG_FLAGS,
+	CONFIG_HOST_PORT,
 
 	/* Indirect action arguments */
 	INDIRECT_ACTION_CREATE,
@@ -1114,6 +1115,7 @@ static const enum index next_config_attr[] = {
 	CONFIG_METERS_NUMBER,
 	CONFIG_CONN_TRACK_NUMBER,
 	CONFIG_FLAGS,
+	CONFIG_HOST_PORT,
 	END,
 	ZERO,
 };
@@ -2767,6 +2769,14 @@ static const struct token token_list[] = {
 		.args = ARGS(ARGS_ENTRY(struct buffer,
 					args.configure.port_attr.flags)),
 	},
+	[CONFIG_HOST_PORT] = {
+		.name = "host_port",
+		.help = "host port for shared objects",
+		.next = NEXT(next_config_attr,
+			     NEXT_ENTRY(COMMON_UNSIGNED)),
+		.args = ARGS(ARGS_ENTRY(struct buffer,
+					args.configure.port_attr.host_port_id)),
+	},
 	/* Top-level command. */
 	[PATTERN_TEMPLATE] = {
 		.name = "pattern_template",
-- 
2.18.1

[PATCH v3 3/3] app/testpmd: add shared indirect action support

[Viacheslav Ovsiienko]

The shared indirect action can be shared between ports,
action should be created on single port and the handle
can be used in the templates and flows on multiple ports,
example:

  flow configure 0 queues_number 1 queues_size 64 counters_number 64
  flow configure 1 queues_number 1 queues_size 64 counters_number 0 \
                   host_port 0 flags 1

  flow indirect_action 0 create ingress action_id 0 action count / end

  flow actions_template 0 create ingress actions_template_id 8
       template indirect 0 / queue index 0 / end
       mask count / queue index 0 / end

  flow actions_template 1 create ingress actions_template_id 18
       template shared_indirect 0 0 / queue index 0 / end
       mask count / queue index 0 / end

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 app/test-pmd/cmdline_flow.c | 53 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 52 insertions(+), 1 deletion(-)

diff --git a/app/test-pmd/cmdline_flow.c b/app/test-pmd/cmdline_flow.c
index 7778828919..f2403622d6 100644
--- a/app/test-pmd/cmdline_flow.c
+++ b/app/test-pmd/cmdline_flow.c
@@ -607,6 +607,8 @@ enum index {
 	ACTION_SAMPLE_INDEX,
 	ACTION_SAMPLE_INDEX_VALUE,
 	ACTION_INDIRECT,
+	ACTION_SHARED_INDIRECT,
+	INDIRECT_ACTION_PORT,
 	INDIRECT_ACTION_ID2PTR,
 	ACTION_MODIFY_FIELD,
 	ACTION_MODIFY_FIELD_OP,
@@ -1926,6 +1928,7 @@ static const enum index next_action[] = {
 	ACTION_AGE_UPDATE,
 	ACTION_SAMPLE,
 	ACTION_INDIRECT,
+	ACTION_SHARED_INDIRECT,
 	ACTION_MODIFY_FIELD,
 	ACTION_CONNTRACK,
 	ACTION_CONNTRACK_UPDATE,
@@ -2435,6 +2438,9 @@ static int parse_ia_destroy(struct context *ctx, const struct token *token,
 static int parse_ia_id2ptr(struct context *ctx, const struct token *token,
 			   const char *str, unsigned int len, void *buf,
 			   unsigned int size);
+static int parse_ia_port(struct context *ctx, const struct token *token,
+			 const char *str, unsigned int len, void *buf,
+			 unsigned int size);
 static int parse_mp(struct context *, const struct token *,
 		    const char *, unsigned int,
 		    void *, unsigned int);
@@ -6522,6 +6528,23 @@ static const struct token token_list[] = {
 		.args = ARGS(ARGS_ENTRY_ARB(0, sizeof(uint32_t))),
 		.call = parse_vc,
 	},
+	[ACTION_SHARED_INDIRECT] = {
+		.name = "shared_indirect",
+		.help = "apply indirect action by id and port",
+		.priv = PRIV_ACTION(INDIRECT, 0),
+		.next = NEXT(NEXT_ENTRY(INDIRECT_ACTION_PORT)),
+		.args = ARGS(ARGS_ENTRY_ARB(0, sizeof(uint32_t)),
+			     ARGS_ENTRY_ARB(0, sizeof(uint32_t))),
+		.call = parse_vc,
+	},
+	[INDIRECT_ACTION_PORT] = {
+		.name = "{indirect_action_port}",
+		.type = "INDIRECT_ACTION_PORT",
+		.help = "indirect action port",
+		.next = NEXT(NEXT_ENTRY(INDIRECT_ACTION_ID2PTR)),
+		.call = parse_ia_port,
+		.comp = comp_none,
+	},
 	[INDIRECT_ACTION_ID2PTR] = {
 		.name = "{action_id}",
 		.type = "INDIRECT_ACTION_ID",
@@ -10006,6 +10029,31 @@ parse_port(struct context *ctx, const struct token *token,
 	return ret;
 }
 
+/** Parse tokens for shared indirect actions. */
+static int
+parse_ia_port(struct context *ctx, const struct token *token,
+	      const char *str, unsigned int len,
+	      void *buf, unsigned int size)
+{
+	struct rte_flow_action *action = ctx->object;
+	uint32_t id;
+	int ret;
+
+	(void)buf;
+	(void)size;
+	ctx->objdata = 0;
+	ctx->object = &id;
+	ctx->objmask = NULL;
+	ret = parse_int(ctx, token, str, len, ctx->object, sizeof(id));
+	ctx->object = action;
+	if (ret != (int)len)
+		return ret;
+	/* set indirect action */
+	if (action)
+		action->conf = (void *)(uintptr_t)id;
+	return ret;
+}
+
 static int
 parse_ia_id2ptr(struct context *ctx, const struct token *token,
 		const char *str, unsigned int len,
@@ -10026,7 +10074,10 @@ parse_ia_id2ptr(struct context *ctx, const struct token *token,
 		return ret;
 	/* set indirect action */
 	if (action) {
-		action->conf = port_action_handle_get_by_id(ctx->port, id);
+		portid_t port_id = ctx->port;
+		if (ctx->prev == INDIRECT_ACTION_PORT)
+			port_id = (portid_t)(uintptr_t)action->conf;
+		action->conf = port_action_handle_get_by_id(port_id, id);
 		ret = (action->conf) ? ret : -1;
 	}
 	return ret;
-- 
2.18.1

Re: [PATCH v3 0/3] *ethdev: sharing indirect actions between port*

[Ferruh Yigit]

On 2/10/2023 3:17 PM, Viacheslav Ovsiienko wrote:
> The RTE Flow API implements the concept of shared objects,
> known as indirect actions (RTE_FLOW_ACTION_TYPE_INDIRECT).
> An application can create the indirect action of desired
> type and configuration with rte_flow_action_handle_create
> call and then specify the obtained action handle in multiple
> flows.
> 
> The initial concept supposes the action handle has strict
> attachment to the port it was created on and to be used
> exclusively in the flows being installed on the port.
> 
> Nowadays the multipath network topologies are quite common,
> packets belonging to the same connection might arrive and
> be sent over multiple ports, and there is the raising demand
> to handle these "spread" connections. To fulfil this demand
> it is proposed to extend indirect action sharing across the
> multiple ports. This kind of sharing would be extremely useful
> for the meters and counters, allowing to manage the single
> connection over the multiple ports.
> 
> This cross-port object sharing is hard to implement in
> generic way merely with software on the upper layers, but
> can be provided by the driver over the single hardware
> instance, where  multiple ports reside on the same physical
> NIC and share the same hardware context.
> 
> To allow this action sharing application should specify
> the "host port" during flow configuring to claim the intention
> to share the indirect actions. All indirect actions reside within
> "host port" context and can be shared in flows being installed
> on the host port and on all the ports referencing this one.
> 
> If sharing between host and port being configured is not supported
> the configuration should be rejected with error. There might be
> multiple independent (mutual exclusive) sharing domains with
> dedicated host and referencing ports.
> 
> To manage the shared indirect action any port from sharing domain
> can be specified. To share or not the created action is up to
> application, no API change is needed.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> 
> Viacheslav Ovsiienko (3):
>   ethdev: sharing indirect actions between ports
>   app/testpmd: add host port parameter into flow config
>   app/testpmd: add shared indirect action support
>
> Acked-by: Ori Kam <orika@nvidia.com>
>

Series applied to dpdk-next-net/main, thanks.

[PATCH v4 0/5] net/mlx5: sharing indirect actions between port

[Viacheslav Ovsiienko]

Adds shared indirect action support to mlx5 PMD.

Erez Shitrit (2):
  net/mlx5/hws: free FT from RTC ID before set the new value
  net/mlx5/hws: fix disconnecting matcher

Viacheslav Ovsiienko (3):
  common/mlx5: add cross port object sharing capability
  net/mlx5: add cross port shared mode for HW steering
  net/mlx5: support counters in cross port shared mode

 drivers/common/mlx5/mlx5_devx_cmds.c  |  13 ++
 drivers/common/mlx5/mlx5_devx_cmds.h  |   1 +
 drivers/net/mlx5/hws/mlx5dr_matcher.c |  12 ++
 drivers/net/mlx5/mlx5.c               |   6 +
 drivers/net/mlx5/mlx5.h               |   2 +
 drivers/net/mlx5/mlx5_flow_hw.c       |  90 ++++++++++++--
 drivers/net/mlx5/mlx5_hws_cnt.c       | 167 ++++++++++++++------------
 drivers/net/mlx5/mlx5_hws_cnt.h       | 109 +++++++++--------
 8 files changed, 266 insertions(+), 134 deletions(-)

-- 
2.18.1

---
RFC: https://inbox.dpdk.org/dev/20221228165433.18185-1-viacheslavo@nvidia.com/
v1:  https://inbox.dpdk.org/dev/20230206095229.23027-1-viacheslavo@nvidia.com/
v2:  https://inbox.dpdk.org/dev/20230207140206.29139-1-viacheslavo@nvidia.com/
     minor comilation issues, rebase, typos
v3:  series split to ethdev/testpmd and mlx5 PMD patch to be smoothly applied
     over dpdk-next/main. The mlx5 PMD seris is coming.
v4:  the remaining part of original series split for mlx5 PMD

[PATCH v4 1/5] net/mlx5/hws: free FT from RTC ID before set the new value

[Viacheslav Ovsiienko]

From: Erez Shitrit <erezsh@nvidia.com>

While matcher is being connect/disconnect in shared gvmi flow we set the
first ft in the table to point on the first matcher,
The FW is increasing the refcount on the first matcher RTC because of
that no matcher if it is the same RTC that was set before, and when we
will try to release that RTC we will get the following syndrome:
0xaa0093 -   destroy_rtc_object: rtc in use or doesn't exist.

In order to resolve that we clean the current pointed RTC from that ft
and only after that setting it to the new RTC value.

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/hws/mlx5dr_matcher.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c
index 5508cfe230..6af493d87a 100644
--- a/drivers/net/mlx5/hws/mlx5dr_matcher.c
+++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c
@@ -334,6 +334,24 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher)
 		return ret;
 	}
 
+	if (!next) {
+		/* ft no longer points to any RTC, drop refcount */
+		ret = mlx5dr_matcher_free_rtc_pointing(tbl->ctx,
+						       tbl->fw_ft_type,
+						       tbl->type,
+						       prev_ft);
+		if (ret) {
+			DR_LOG(ERR, "Failed to reset last RTC refcount");
+			return ret;
+		}
+	}
+
+	ret = mlx5dr_matcher_shared_update_local_ft(tbl);
+	if (ret) {
+		DR_LOG(ERR, "Failed to update local_ft in shared table");
+		return ret;
+	}
+
 	return 0;
 }
 
-- 
2.18.1

[PATCH v4 2/5] net/mlx5/hws: fix disconnecting matcher

[Viacheslav Ovsiienko]

From: Erez Shitrit <erezsh@nvidia.com>

This patch fixes the matcher disconnection handling, by removing the RTC
references from flow table if the currently removed matcher was the last
one for the given table. As a result RTC in this matcher can be
correctly freed, since there are no dangling references to the RTC.

Fixes: c467608215b2 ("net/mlx5/hws: add matcher object")
Cc: stable@dpdk.org

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Reviewed-by: Alex Vesker <valex@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/net/mlx5/hws/mlx5dr_matcher.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c
index 6af493d87a..1fe7ec1bc3 100644
--- a/drivers/net/mlx5/hws/mlx5dr_matcher.c
+++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c
@@ -346,12 +346,6 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher)
 		}
 	}
 
-	ret = mlx5dr_matcher_shared_update_local_ft(tbl);
-	if (ret) {
-		DR_LOG(ERR, "Failed to update local_ft in shared table");
-		return ret;
-	}
-
 	return 0;
 }
 
-- 
2.18.1

[PATCH v4 3/5] common/mlx5: add cross port object sharing capability

[Viacheslav Ovsiienko]

Add query port capabilities to share steering objects
between multiple ports of the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 13 +++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index e3a4927d0f..17128035ec 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -1047,6 +1047,19 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		attr->flow_counter_bulk_log_granularity =
 			MLX5_GET(cmd_hca_cap_2, hcattr,
 				 flow_counter_bulk_log_granularity);
+		rc = MLX5_GET(cmd_hca_cap_2, hcattr,
+			      cross_vhca_object_to_object_supported);
+		attr->cross_vhca =
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_FT) &&
+			(rc & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC);
+		rc = MLX5_GET(cmd_hca_cap_2, hcattr,
+			      allowed_object_for_other_vhca_access);
+		attr->cross_vhca = attr->cross_vhca &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR) &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_FT) &&
+			(rc & MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC);
 	}
 	if (attr->log_min_stride_wqe_sz == 0)
 		attr->log_min_stride_wqe_sz = MLX5_MPRQ_LOG_MIN_STRIDE_WQE_SIZE;
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index c94b9eac06..b65ba569bc 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -288,6 +288,7 @@ struct mlx5_hca_attr {
 	uint32_t alloc_flow_counter_pd:1;
 	uint32_t flow_counter_access_aso:1;
 	uint32_t flow_access_aso_opc_mod:8;
+	uint32_t cross_vhca:1;
 };
 
 /* LAG Context. */
-- 
2.18.1

[PATCH v4 4/5] net/mlx5: add cross port shared mode for HW steering

[Viacheslav Ovsiienko]

Add host port option for sharing steering objects between
multiple ports of the same physical NIC.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/net/mlx5/mlx5.c         |  6 +++
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow_hw.c | 78 +++++++++++++++++++++++++++++++--
 drivers/net/mlx5/mlx5_hws_cnt.c | 12 +++++
 4 files changed, 94 insertions(+), 4 deletions(-)

diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b8643cebdd..2eca2cceef 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2013,6 +2013,12 @@ mlx5_dev_close(struct rte_eth_dev *dev)
 	}
 	if (!priv->sh)
 		return 0;
+	if (priv->shared_refcnt) {
+		DRV_LOG(ERR, "port %u is shared host in use (%u)",
+			dev->data->port_id, priv->shared_refcnt);
+		rte_errno = EBUSY;
+		return -EBUSY;
+	}
 	DRV_LOG(DEBUG, "port %u closing device \"%s\"",
 		dev->data->port_id,
 		((priv->sh->cdev->ctx != NULL) ?
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 16b33e1548..525bdd47f7 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1780,6 +1780,8 @@ struct mlx5_priv {
 	struct mlx5_flow_hw_ctrl_rx *hw_ctrl_rx;
 	/**< HW steering templates used to create control flow rules. */
 #endif
+	struct rte_eth_dev *shared_host; /* Host device for HW steering. */
+	uint16_t shared_refcnt; /* HW steering host reference counter. */
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index aacde224f2..3b9789aa53 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -5,6 +5,8 @@
 #include <rte_flow.h>
 
 #include <mlx5_malloc.h>
+
+#include "mlx5.h"
 #include "mlx5_defs.h"
 #include "mlx5_flow.h"
 #include "mlx5_rx.h"
@@ -6303,6 +6305,12 @@ flow_hw_ct_pool_create(struct rte_eth_dev *dev,
 	int reg_id;
 	uint32_t flags;
 
+	if (port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Connection tracking is not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return NULL;
+	}
 	pool = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*pool), 0, SOCKET_ID_ANY);
 	if (!pool) {
 		rte_errno = ENOMEM;
@@ -6787,6 +6795,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		  struct rte_flow_error *error)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_priv *host_priv = NULL;
 	struct mlx5dr_context *dr_ctx = NULL;
 	struct mlx5dr_context_attr dr_ctx_attr = {0};
 	struct mlx5_hw_q *hw_q;
@@ -6801,7 +6810,8 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		.free = mlx5_free,
 		.type = "mlx5_hw_action_construct_data",
 	};
-	/* Adds one queue to be used by PMD.
+	/*
+	 * Adds one queue to be used by PMD.
 	 * The last queue will be used by the PMD.
 	 */
 	uint16_t nb_q_updated = 0;
@@ -6920,6 +6930,57 @@ flow_hw_configure(struct rte_eth_dev *dev,
 	dr_ctx_attr.queues = nb_q_updated;
 	/* Queue size should all be the same. Take the first one. */
 	dr_ctx_attr.queue_size = _queue_attr[0]->size;
+	if (port_attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		struct rte_eth_dev *host_dev = NULL;
+		uint16_t port_id;
+
+		MLX5_ASSERT(rte_eth_dev_is_valid_port(port_attr->host_port_id));
+		if (is_proxy) {
+			DRV_LOG(ERR, "cross vHCA shared mode not supported "
+				     " for E-Switch confgiurations");
+			rte_errno = ENOTSUP;
+			goto err;
+		}
+		MLX5_ETH_FOREACH_DEV(port_id, dev->device) {
+			if (port_id == port_attr->host_port_id) {
+				host_dev = &rte_eth_devices[port_id];
+				break;
+			}
+		}
+		if (!host_dev || host_dev == dev ||
+		    !host_dev->data || !host_dev->data->dev_private) {
+			DRV_LOG(ERR, "Invalid cross vHCA host port %u",
+				port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		host_priv = host_dev->data->dev_private;
+		if (host_priv->sh->cdev->ctx == priv->sh->cdev->ctx) {
+			DRV_LOG(ERR, "Sibling ports %u and %u do not "
+				     "require cross vHCA sharing mode",
+				dev->data->port_id, port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		if (host_priv->shared_host) {
+			DRV_LOG(ERR, "Host port %u is not the sharing base",
+				port_attr->host_port_id);
+			rte_errno = EINVAL;
+			goto err;
+		}
+		if (port_attr->nb_counters ||
+		    port_attr->nb_aging_objects ||
+		    port_attr->nb_meters ||
+		    port_attr->nb_conn_tracks) {
+			DRV_LOG(ERR,
+				"Object numbers on guest port must be zeros");
+			rte_errno = EINVAL;
+			goto err;
+		}
+		dr_ctx_attr.shared_ibv_ctx = host_priv->sh->cdev->ctx;
+		priv->shared_host = host_dev;
+		__atomic_fetch_add(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+	}
 	dr_ctx = mlx5dr_context_open(priv->sh->cdev->ctx, &dr_ctx_attr);
 	/* rte_errno has been updated by HWS layer. */
 	if (!dr_ctx)
@@ -6935,7 +6996,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 		goto err;
 	}
 	/* Initialize meter library*/
-	if (port_attr->nb_meters)
+	if (port_attr->nb_meters || (host_priv && host_priv->hws_mpool))
 		if (mlx5_flow_meter_init(dev, port_attr->nb_meters, 1, 1, nb_q_updated))
 			goto err;
 	/* Add global actions. */
@@ -6972,7 +7033,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 			goto err;
 		}
 	}
-	if (port_attr->nb_conn_tracks) {
+	if (port_attr->nb_conn_tracks || (host_priv && host_priv->hws_ctpool)) {
 		mem_size = sizeof(struct mlx5_aso_sq) * nb_q_updated +
 			   sizeof(*priv->ct_mng);
 		priv->ct_mng = mlx5_malloc(MLX5_MEM_ZERO, mem_size,
@@ -6986,7 +7047,7 @@ flow_hw_configure(struct rte_eth_dev *dev,
 			goto err;
 		priv->sh->ct_aso_en = 1;
 	}
-	if (port_attr->nb_counters) {
+	if (port_attr->nb_counters || (host_priv && host_priv->hws_cpool)) {
 		priv->hws_cpool = mlx5_hws_cnt_pool_create(dev, port_attr,
 							   nb_queue);
 		if (priv->hws_cpool == NULL)
@@ -7055,6 +7116,10 @@ flow_hw_configure(struct rte_eth_dev *dev,
 	}
 	if (_queue_attr)
 		mlx5_free(_queue_attr);
+	if (priv->shared_host) {
+		__atomic_fetch_sub(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+		priv->shared_host = NULL;
+	}
 	/* Do not overwrite the internal errno information. */
 	if (ret)
 		return ret;
@@ -7133,6 +7198,11 @@ flow_hw_resource_release(struct rte_eth_dev *dev)
 	mlx5_free(priv->hw_q);
 	priv->hw_q = NULL;
 	claim_zero(mlx5dr_context_close(priv->dr_ctx));
+	if (priv->shared_host) {
+		struct mlx5_priv *host_priv = priv->shared_host->data->dev_private;
+		__atomic_fetch_sub(&host_priv->shared_refcnt, 1, __ATOMIC_RELAXED);
+		priv->shared_host = NULL;
+	}
 	priv->dr_ctx = NULL;
 	priv->nb_queue = 0;
 }
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.c b/drivers/net/mlx5/mlx5_hws_cnt.c
index 05cc954903..797844439f 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.c
+++ b/drivers/net/mlx5/mlx5_hws_cnt.c
@@ -619,6 +619,12 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	int ret = 0;
 	size_t sz;
 
+	if (pattr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Counters are not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return NULL;
+	}
 	/* init cnt service if not. */
 	if (priv->sh->cnt_svc == NULL) {
 		ret = mlx5_hws_cnt_svc_init(priv->sh);
@@ -1190,6 +1196,12 @@ mlx5_hws_age_pool_init(struct rte_eth_dev *dev,
 
 	strict_queue = !!(attr->flags & RTE_FLOW_PORT_FLAG_STRICT_QUEUE);
 	MLX5_ASSERT(priv->hws_cpool);
+	if (attr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
+		DRV_LOG(ERR, "Aging sn not supported "
+			     "in cross vHCA sharing mode");
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
 	nb_alloc_cnts = mlx5_hws_cnt_pool_get_size(priv->hws_cpool);
 	if (strict_queue) {
 		rsize = mlx5_hws_aged_out_q_ring_size_get(nb_alloc_cnts,
-- 
2.18.1

[PATCH v4 5/5] net/mlx5: support counters in cross port shared mode

[Viacheslav Ovsiienko]

In the cross vHCA sharing mode the host counter pool
should be used in counter related routines. The local
port pool is used to store the dedicated DR action
handle, per queue counter caches and query data are
ignored and not allocated on local pool.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/net/mlx5/mlx5_flow_hw.c |  12 ++-
 drivers/net/mlx5/mlx5_hws_cnt.c | 163 ++++++++++++++++----------------
 drivers/net/mlx5/mlx5_hws_cnt.h | 109 +++++++++++----------
 3 files changed, 150 insertions(+), 134 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index 3b9789aa53..8ff72871f3 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -2311,8 +2311,10 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
 				break;
 			/* Fall-through. */
 		case RTE_FLOW_ACTION_TYPE_COUNT:
-			ret = mlx5_hws_cnt_pool_get(priv->hws_cpool, &queue,
-						    &cnt_id, age_idx);
+			ret = mlx5_hws_cnt_pool_get(priv->hws_cpool,
+					(priv->shared_refcnt ||
+					 priv->hws_cpool->cfg.host_cpool) ?
+					NULL : &queue, &cnt_id, age_idx);
 			if (ret != 0)
 				return ret;
 			ret = mlx5_hws_cnt_pool_get_action_offset
@@ -7998,6 +8000,7 @@ static int
 flow_hw_query_counter(const struct rte_eth_dev *dev, uint32_t counter,
 		      void *data, struct rte_flow_error *error)
 {
+	struct mlx5_hws_cnt_pool *hpool;
 	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_hws_cnt *cnt;
 	struct rte_flow_query_count *qc = data;
@@ -8008,8 +8011,9 @@ flow_hw_query_counter(const struct rte_eth_dev *dev, uint32_t counter,
 		return rte_flow_error_set(error, EINVAL,
 				RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
 				"counter are not available");
-	iidx = mlx5_hws_cnt_iidx(priv->hws_cpool, counter);
-	cnt = &priv->hws_cpool->pool[iidx];
+	hpool = mlx5_hws_cnt_host_pool(priv->hws_cpool);
+	iidx = mlx5_hws_cnt_iidx(hpool, counter);
+	cnt = &hpool->pool[iidx];
 	__hws_cnt_query_raw(priv->hws_cpool, counter, &pkts, &bytes);
 	qc->hits_set = 1;
 	qc->bytes_set = 1;
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.c b/drivers/net/mlx5/mlx5_hws_cnt.c
index 797844439f..d6a017a757 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.c
+++ b/drivers/net/mlx5/mlx5_hws_cnt.c
@@ -24,12 +24,8 @@
 static void
 __hws_cnt_id_load(struct mlx5_hws_cnt_pool *cpool)
 {
-	uint32_t preload;
-	uint32_t q_num = cpool->cache->q_num;
 	uint32_t cnt_num = mlx5_hws_cnt_pool_get_size(cpool);
-	cnt_id_t cnt_id;
-	uint32_t qidx, iidx = 0;
-	struct rte_ring *qcache = NULL;
+	uint32_t iidx;
 
 	/*
 	 * Counter ID order is important for tracking the max number of in used
@@ -39,18 +35,9 @@ __hws_cnt_id_load(struct mlx5_hws_cnt_pool *cpool)
 	 * and then the global free list.
 	 * In the end, user fetch the counter from minimal to the maximum.
 	 */
-	preload = RTE_MIN(cpool->cache->preload_sz, cnt_num / q_num);
-	for (qidx = 0; qidx < q_num; qidx++) {
-		for (; iidx < preload * (qidx + 1); iidx++) {
-			cnt_id = mlx5_hws_cnt_id_gen(cpool, iidx);
-			qcache = cpool->cache->qcache[qidx];
-			if (qcache)
-				rte_ring_enqueue_elem(qcache, &cnt_id,
-						sizeof(cnt_id));
-		}
-	}
-	for (; iidx < cnt_num; iidx++) {
-		cnt_id = mlx5_hws_cnt_id_gen(cpool, iidx);
+	for (iidx = 0; iidx < cnt_num; iidx++) {
+		cnt_id_t cnt_id  = mlx5_hws_cnt_id_gen(cpool, iidx);
+
 		rte_ring_enqueue_elem(cpool->free_list, &cnt_id,
 				sizeof(cnt_id));
 	}
@@ -334,7 +321,26 @@ mlx5_hws_cnt_svc(void *opaque)
 	return NULL;
 }
 
-struct mlx5_hws_cnt_pool *
+static void
+mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool * const cntp)
+{
+	uint32_t qidx = 0;
+	if (cntp == NULL)
+		return;
+	rte_ring_free(cntp->free_list);
+	rte_ring_free(cntp->wait_reset_list);
+	rte_ring_free(cntp->reuse_list);
+	if (cntp->cache) {
+		for (qidx = 0; qidx < cntp->cache->q_num; qidx++)
+			rte_ring_free(cntp->cache->qcache[qidx]);
+	}
+	mlx5_free(cntp->cache);
+	mlx5_free(cntp->raw_mng);
+	mlx5_free(cntp->pool);
+	mlx5_free(cntp);
+}
+
+static struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		       const struct mlx5_hws_cnt_pool_cfg *pcfg,
 		       const struct mlx5_hws_cache_param *ccfg)
@@ -352,6 +358,8 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		return NULL;
 
 	cntp->cfg = *pcfg;
+	if (cntp->cfg.host_cpool)
+		return cntp;
 	cntp->cache = mlx5_malloc(MLX5_MEM_ANY | MLX5_MEM_ZERO,
 			sizeof(*cntp->cache) +
 			sizeof(((struct mlx5_hws_cnt_pool_caches *)0)->qcache[0])
@@ -387,8 +395,9 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 		goto error;
 	snprintf(mz_name, sizeof(mz_name), "%s_F_RING", pcfg->name);
 	cntp->free_list = rte_ring_create_elem(mz_name, sizeof(cnt_id_t),
-			(uint32_t)cnt_num, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
+				(uint32_t)cnt_num, SOCKET_ID_ANY,
+				RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ |
+				RING_F_EXACT_SZ);
 	if (cntp->free_list == NULL) {
 		DRV_LOG(ERR, "failed to create free list ring");
 		goto error;
@@ -404,7 +413,7 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 	snprintf(mz_name, sizeof(mz_name), "%s_U_RING", pcfg->name);
 	cntp->reuse_list = rte_ring_create_elem(mz_name, sizeof(cnt_id_t),
 			(uint32_t)cnt_num, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
+			RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ | RING_F_EXACT_SZ);
 	if (cntp->reuse_list == NULL) {
 		DRV_LOG(ERR, "failed to create reuse list ring");
 		goto error;
@@ -427,25 +436,6 @@ mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
 	return NULL;
 }
 
-void
-mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool * const cntp)
-{
-	uint32_t qidx = 0;
-	if (cntp == NULL)
-		return;
-	rte_ring_free(cntp->free_list);
-	rte_ring_free(cntp->wait_reset_list);
-	rte_ring_free(cntp->reuse_list);
-	if (cntp->cache) {
-		for (qidx = 0; qidx < cntp->cache->q_num; qidx++)
-			rte_ring_free(cntp->cache->qcache[qidx]);
-	}
-	mlx5_free(cntp->cache);
-	mlx5_free(cntp->raw_mng);
-	mlx5_free(cntp->pool);
-	mlx5_free(cntp);
-}
-
 int
 mlx5_hws_cnt_service_thread_create(struct mlx5_dev_ctx_shared *sh)
 {
@@ -483,7 +473,7 @@ mlx5_hws_cnt_service_thread_destroy(struct mlx5_dev_ctx_shared *sh)
 	sh->cnt_svc->service_thread = 0;
 }
 
-int
+static int
 mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 			    struct mlx5_hws_cnt_pool *cpool)
 {
@@ -495,6 +485,7 @@ mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 	struct mlx5_devx_counter_attr attr = {0};
 	struct mlx5_devx_obj *dcs;
 
+	MLX5_ASSERT(cpool->cfg.host_cpool == NULL);
 	if (hca_attr->flow_counter_bulk_log_max_alloc == 0) {
 		DRV_LOG(ERR, "Fw doesn't support bulk log max alloc");
 		return -1;
@@ -550,7 +541,7 @@ mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
 	return -1;
 }
 
-void
+static void
 mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
 			   struct mlx5_hws_cnt_pool *cpool)
 {
@@ -566,22 +557,39 @@ mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
 	}
 }
 
-int
+static void
+mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool)
+{
+	uint32_t idx;
+
+	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
+		struct mlx5_hws_cnt_dcs *dcs = &cpool->dcs_mng.dcs[idx];
+
+		if (dcs->dr_action != NULL) {
+			mlx5dr_action_destroy(dcs->dr_action);
+			dcs->dr_action = NULL;
+		}
+	}
+}
+
+static int
 mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
 		struct mlx5_hws_cnt_pool *cpool)
 {
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint32_t idx;
 	int ret = 0;
-	struct mlx5_hws_cnt_dcs *dcs;
 	uint32_t flags;
 
 	flags = MLX5DR_ACTION_FLAG_HWS_RX | MLX5DR_ACTION_FLAG_HWS_TX;
 	if (priv->sh->config.dv_esw_en && priv->master)
 		flags |= MLX5DR_ACTION_FLAG_HWS_FDB;
-	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
-		dcs = &cpool->dcs_mng.dcs[idx];
+	for (idx = 0; idx < hpool->dcs_mng.batch_total; idx++) {
+		struct mlx5_hws_cnt_dcs *hdcs = &hpool->dcs_mng.dcs[idx];
+		struct mlx5_hws_cnt_dcs *dcs = &cpool->dcs_mng.dcs[idx];
+
 		dcs->dr_action = mlx5dr_action_create_counter(priv->dr_ctx,
-					(struct mlx5dr_devx_obj *)dcs->obj,
+					(struct mlx5dr_devx_obj *)hdcs->obj,
 					flags);
 		if (dcs->dr_action == NULL) {
 			mlx5_hws_cnt_pool_action_destroy(cpool);
@@ -592,21 +600,6 @@ mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
 	return ret;
 }
 
-void
-mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool)
-{
-	uint32_t idx;
-	struct mlx5_hws_cnt_dcs *dcs;
-
-	for (idx = 0; idx < cpool->dcs_mng.batch_total; idx++) {
-		dcs = &cpool->dcs_mng.dcs[idx];
-		if (dcs->dr_action != NULL) {
-			mlx5dr_action_destroy(dcs->dr_action);
-			dcs->dr_action = NULL;
-		}
-	}
-}
-
 struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 		const struct rte_flow_port_attr *pattr, uint16_t nb_queue)
@@ -619,11 +612,28 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	int ret = 0;
 	size_t sz;
 
+	mp_name = mlx5_malloc(MLX5_MEM_ZERO, RTE_MEMZONE_NAMESIZE, 0,
+			SOCKET_ID_ANY);
+	if (mp_name == NULL)
+		goto error;
+	snprintf(mp_name, RTE_MEMZONE_NAMESIZE, "MLX5_HWS_CNT_POOL_%u",
+			dev->data->port_id);
+	pcfg.name = mp_name;
+	pcfg.request_num = pattr->nb_counters;
+	pcfg.alloc_factor = HWS_CNT_ALLOC_FACTOR_DEFAULT;
 	if (pattr->flags & RTE_FLOW_PORT_FLAG_SHARE_INDIRECT) {
-		DRV_LOG(ERR, "Counters are not supported "
-			     "in cross vHCA sharing mode");
-		rte_errno = ENOTSUP;
-		return NULL;
+		struct mlx5_priv *host_priv =
+				priv->shared_host->data->dev_private;
+		struct mlx5_hws_cnt_pool *chost = host_priv->hws_cpool;
+
+		pcfg.host_cpool = chost;
+		cpool = mlx5_hws_cnt_pool_init(priv->sh, &pcfg, &cparam);
+		if (cpool == NULL)
+			goto error;
+		ret = mlx5_hws_cnt_pool_action_create(priv, cpool);
+		if (ret != 0)
+			goto error;
+		return cpool;
 	}
 	/* init cnt service if not. */
 	if (priv->sh->cnt_svc == NULL) {
@@ -636,15 +646,6 @@ mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 	cparam.q_num = nb_queue;
 	cparam.threshold = HWS_CNT_CACHE_THRESHOLD_DEFAULT;
 	cparam.size = HWS_CNT_CACHE_SZ_DEFAULT;
-	pcfg.alloc_factor = HWS_CNT_ALLOC_FACTOR_DEFAULT;
-	mp_name = mlx5_malloc(MLX5_MEM_ZERO, RTE_MEMZONE_NAMESIZE, 0,
-			SOCKET_ID_ANY);
-	if (mp_name == NULL)
-		goto error;
-	snprintf(mp_name, RTE_MEMZONE_NAMESIZE, "MLX5_HWS_CNT_POOL_%u",
-			dev->data->port_id);
-	pcfg.name = mp_name;
-	pcfg.request_num = pattr->nb_counters;
 	cpool = mlx5_hws_cnt_pool_init(priv->sh, &pcfg, &cparam);
 	if (cpool == NULL)
 		goto error;
@@ -679,11 +680,15 @@ mlx5_hws_cnt_pool_destroy(struct mlx5_dev_ctx_shared *sh,
 {
 	if (cpool == NULL)
 		return;
-	if (--sh->cnt_svc->refcnt == 0)
-		mlx5_hws_cnt_svc_deinit(sh);
+	if (cpool->cfg.host_cpool == NULL) {
+		if (--sh->cnt_svc->refcnt == 0)
+			mlx5_hws_cnt_svc_deinit(sh);
+	}
 	mlx5_hws_cnt_pool_action_destroy(cpool);
-	mlx5_hws_cnt_pool_dcs_free(sh, cpool);
-	mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+	if (cpool->cfg.host_cpool == NULL) {
+		mlx5_hws_cnt_pool_dcs_free(sh, cpool);
+		mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+	}
 	mlx5_free((void *)cpool->cfg.name);
 	mlx5_hws_cnt_pool_deinit(cpool);
 }
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.h b/drivers/net/mlx5/mlx5_hws_cnt.h
index 030dcead86..d35d083eeb 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.h
+++ b/drivers/net/mlx5/mlx5_hws_cnt.h
@@ -86,6 +86,7 @@ struct mlx5_hws_cnt_pool_cfg {
 	char *name;
 	uint32_t request_num;
 	uint32_t alloc_factor;
+	struct mlx5_hws_cnt_pool *host_cpool;
 };
 
 struct mlx5_hws_cnt_pool_caches {
@@ -148,6 +149,22 @@ struct mlx5_hws_age_param {
 	void *context; /* Flow AGE context. */
 } __rte_packed __rte_cache_aligned;
 
+
+/**
+ * Return the actual counter pool should be used in cross vHCA sharing mode.
+ * as index of raw/cnt pool.
+ *
+ * @param cnt_id
+ *   The external counter id
+ * @return
+ *   Internal index
+ */
+static __always_inline struct mlx5_hws_cnt_pool *
+mlx5_hws_cnt_host_pool(struct mlx5_hws_cnt_pool *cpool)
+{
+	return cpool->cfg.host_cpool ? cpool->cfg.host_cpool : cpool;
+}
+
 /**
  * Translate counter id into internal index (start from 0), which can be used
  * as index of raw/cnt pool.
@@ -160,11 +177,12 @@ struct mlx5_hws_age_param {
 static __rte_always_inline uint32_t
 mlx5_hws_cnt_iidx(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint8_t dcs_idx = cnt_id >> MLX5_HWS_CNT_DCS_IDX_OFFSET;
 	uint32_t offset = cnt_id & MLX5_HWS_CNT_IDX_MASK;
 
 	dcs_idx &= MLX5_HWS_CNT_DCS_IDX_MASK;
-	return (cpool->dcs_mng.dcs[dcs_idx].iidx + offset);
+	return (hpool->dcs_mng.dcs[dcs_idx].iidx + offset);
 }
 
 /**
@@ -191,7 +209,8 @@ mlx5_hws_cnt_id_valid(cnt_id_t cnt_id)
 static __rte_always_inline cnt_id_t
 mlx5_hws_cnt_id_gen(struct mlx5_hws_cnt_pool *cpool, uint32_t iidx)
 {
-	struct mlx5_hws_cnt_dcs_mng *dcs_mng = &cpool->dcs_mng;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	struct mlx5_hws_cnt_dcs_mng *dcs_mng = &hpool->dcs_mng;
 	uint32_t idx;
 	uint32_t offset;
 	cnt_id_t cnt_id;
@@ -212,7 +231,8 @@ static __rte_always_inline void
 __hws_cnt_query_raw(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id,
 		uint64_t *raw_pkts, uint64_t *raw_bytes)
 {
-	struct mlx5_hws_cnt_raw_data_mng *raw_mng = cpool->raw_mng;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	struct mlx5_hws_cnt_raw_data_mng *raw_mng = hpool->raw_mng;
 	struct flow_counter_stats s[2];
 	uint8_t i = 0x1;
 	size_t stat_sz = sizeof(s[0]);
@@ -393,22 +413,23 @@ mlx5_hws_cnt_pool_put(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 		      cnt_id_t *cnt_id)
 {
 	unsigned int ret = 0;
+	struct mlx5_hws_cnt_pool *hpool;
 	struct rte_ring_zc_data zcdc = {0};
 	struct rte_ring_zc_data zcdr = {0};
 	struct rte_ring *qcache = NULL;
 	unsigned int wb_num = 0; /* cache write-back number. */
 	uint32_t iidx;
 
-	iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
-	MLX5_ASSERT(cpool->pool[iidx].in_used);
-	cpool->pool[iidx].in_used = false;
-	cpool->pool[iidx].query_gen_when_free =
-		__atomic_load_n(&cpool->query_gen, __ATOMIC_RELAXED);
-	if (likely(queue != NULL))
-		qcache = cpool->cache->qcache[*queue];
+	hpool = mlx5_hws_cnt_host_pool(cpool);
+	iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
+	hpool->pool[iidx].in_used = false;
+	hpool->pool[iidx].query_gen_when_free =
+		__atomic_load_n(&hpool->query_gen, __ATOMIC_RELAXED);
+	if (likely(queue != NULL) && cpool->cfg.host_cpool == NULL)
+		qcache = hpool->cache->qcache[*queue];
 	if (unlikely(qcache == NULL)) {
-		ret = rte_ring_enqueue_elem(cpool->wait_reset_list, cnt_id,
-					    sizeof(cnt_id_t));
+		ret = rte_ring_enqueue_elem(hpool->wait_reset_list, cnt_id,
+				sizeof(cnt_id_t));
 		MLX5_ASSERT(ret == 0);
 		return;
 	}
@@ -465,9 +486,10 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 	uint32_t iidx, query_gen = 0;
 	cnt_id_t tmp_cid = 0;
 
-	if (likely(queue != NULL))
+	if (likely(queue != NULL && cpool->cfg.host_cpool == NULL))
 		qcache = cpool->cache->qcache[*queue];
 	if (unlikely(qcache == NULL)) {
+		cpool = mlx5_hws_cnt_host_pool(cpool);
 		ret = rte_ring_dequeue_elem(cpool->reuse_list, &tmp_cid,
 				sizeof(cnt_id_t));
 		if (unlikely(ret != 0)) {
@@ -534,7 +556,9 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 static __rte_always_inline unsigned int
 mlx5_hws_cnt_pool_get_size(struct mlx5_hws_cnt_pool *cpool)
 {
-	return rte_ring_get_capacity(cpool->free_list);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+
+	return rte_ring_get_capacity(hpool->free_list);
 }
 
 static __rte_always_inline int
@@ -554,51 +578,56 @@ static __rte_always_inline int
 mlx5_hws_cnt_shared_get(struct mlx5_hws_cnt_pool *cpool, cnt_id_t *cnt_id,
 			uint32_t age_idx)
 {
-	int ret;
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
 	uint32_t iidx;
+	int ret;
 
-	ret = mlx5_hws_cnt_pool_get(cpool, NULL, cnt_id, age_idx);
+	ret = mlx5_hws_cnt_pool_get(hpool, NULL, cnt_id, age_idx);
 	if (ret != 0)
 		return ret;
-	iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
-	cpool->pool[iidx].share = 1;
+	iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
+	hpool->pool[iidx].share = 1;
 	return 0;
 }
 
 static __rte_always_inline void
 mlx5_hws_cnt_shared_put(struct mlx5_hws_cnt_pool *cpool, cnt_id_t *cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, *cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, *cnt_id);
 
-	cpool->pool[iidx].share = 0;
-	mlx5_hws_cnt_pool_put(cpool, NULL, cnt_id);
+	hpool->pool[iidx].share = 0;
+	mlx5_hws_cnt_pool_put(hpool, NULL, cnt_id);
 }
 
 static __rte_always_inline bool
 mlx5_hws_cnt_is_shared(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	return cpool->pool[iidx].share ? true : false;
+	return hpool->pool[iidx].share ? true : false;
 }
 
 static __rte_always_inline void
 mlx5_hws_cnt_age_set(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id,
 		     uint32_t age_idx)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	MLX5_ASSERT(cpool->pool[iidx].share);
-	cpool->pool[iidx].age_idx = age_idx;
+	MLX5_ASSERT(hpool->pool[iidx].share);
+	hpool->pool[iidx].age_idx = age_idx;
 }
 
 static __rte_always_inline uint32_t
 mlx5_hws_cnt_age_get(struct mlx5_hws_cnt_pool *cpool, cnt_id_t cnt_id)
 {
-	uint32_t iidx = mlx5_hws_cnt_iidx(cpool, cnt_id);
+	struct mlx5_hws_cnt_pool *hpool = mlx5_hws_cnt_host_pool(cpool);
+	uint32_t iidx = mlx5_hws_cnt_iidx(hpool, cnt_id);
 
-	MLX5_ASSERT(cpool->pool[iidx].share);
-	return cpool->pool[iidx].age_idx;
+	MLX5_ASSERT(hpool->pool[iidx].share);
+	return hpool->pool[iidx].age_idx;
 }
 
 static __rte_always_inline cnt_id_t
@@ -645,34 +674,12 @@ mlx5_hws_age_is_indirect(uint32_t age_idx)
 }
 
 /* init HWS counter pool. */
-struct mlx5_hws_cnt_pool *
-mlx5_hws_cnt_pool_init(struct mlx5_dev_ctx_shared *sh,
-		       const struct mlx5_hws_cnt_pool_cfg *pcfg,
-		       const struct mlx5_hws_cache_param *ccfg);
-
-void
-mlx5_hws_cnt_pool_deinit(struct mlx5_hws_cnt_pool *cntp);
-
 int
 mlx5_hws_cnt_service_thread_create(struct mlx5_dev_ctx_shared *sh);
 
 void
 mlx5_hws_cnt_service_thread_destroy(struct mlx5_dev_ctx_shared *sh);
 
-int
-mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
-		struct mlx5_hws_cnt_pool *cpool);
-void
-mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
-		struct mlx5_hws_cnt_pool *cpool);
-
-int
-mlx5_hws_cnt_pool_action_create(struct mlx5_priv *priv,
-		struct mlx5_hws_cnt_pool *cpool);
-
-void
-mlx5_hws_cnt_pool_action_destroy(struct mlx5_hws_cnt_pool *cpool);
-
 struct mlx5_hws_cnt_pool *
 mlx5_hws_cnt_pool_create(struct rte_eth_dev *dev,
 		const struct rte_flow_port_attr *pattr, uint16_t nb_queue);
-- 
2.18.1

RE: [PATCH v4 0/5] net/mlx5: sharing indirect actions between port

[Raslan Darawsheh]

Hi,

> -----Original Message-----
> From: Slava Ovsiienko <viacheslavo@nvidia.com>
> Sent: Monday, February 13, 2023 3:38 PM
> To: dev@dpdk.org
> Cc: Ori Kam <orika@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com>;
> Matan Azrad <matan@nvidia.com>
> Subject: [PATCH v4 0/5] net/mlx5: sharing indirect actions between port
> 
> Adds shared indirect action support to mlx5 PMD.
> 
> Erez Shitrit (2):
>   net/mlx5/hws: free FT from RTC ID before set the new value
>   net/mlx5/hws: fix disconnecting matcher
> 
> Viacheslav Ovsiienko (3):
>   common/mlx5: add cross port object sharing capability
>   net/mlx5: add cross port shared mode for HW steering
>   net/mlx5: support counters in cross port shared mode
> 
>  drivers/common/mlx5/mlx5_devx_cmds.c  |  13 ++
>  drivers/common/mlx5/mlx5_devx_cmds.h  |   1 +
>  drivers/net/mlx5/hws/mlx5dr_matcher.c |  12 ++
>  drivers/net/mlx5/mlx5.c               |   6 +
>  drivers/net/mlx5/mlx5.h               |   2 +
>  drivers/net/mlx5/mlx5_flow_hw.c       |  90 ++++++++++++--
>  drivers/net/mlx5/mlx5_hws_cnt.c       | 167 ++++++++++++++------------
>  drivers/net/mlx5/mlx5_hws_cnt.h       | 109 +++++++++--------
>  8 files changed, 266 insertions(+), 134 deletions(-)
> 
> --
> 2.18.1
> 
> ---
> RFC: https://inbox.dpdk.org/dev/20221228165433.18185-1-
> viacheslavo@nvidia.com/
> v1:  https://inbox.dpdk.org/dev/20230206095229.23027-1-
> viacheslavo@nvidia.com/
> v2:  https://inbox.dpdk.org/dev/20230207140206.29139-1-
> viacheslavo@nvidia.com/
>      minor comilation issues, rebase, typos
> v3:  series split to ethdev/testpmd and mlx5 PMD patch to be smoothly
> applied
>      over dpdk-next/main. The mlx5 PMD seris is coming.
> v4:  the remaining part of original series split for mlx5 PMD

Series applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh