[dpdk-dev] [PATCH] ethdev: support queue-based priority flow control

[dpdk-dev] [PATCH] ethdev: support queue-based priority flow control

[jerinj]

From: Jerin Jacob <jerinj@marvell.com>

Based on device support and use-case need, there are two different ways
to enable PFC. The first case is the port level PFC configuration, in
this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
configure the PFC, and PFC frames will be generated using based on VLAN
TC value.

The second case is the queue level PFC configuration, in this
case, Any packet field content can be used to steer the packet to the
specific queue using rte_flow or RSS and then use
rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping on each
queue. Based on congestion selected on the specific queue, configured TC
shall be used to generate PFC frames.

Operation of these modes are mutually exclusive, when driver sets
non zero value for rte_eth_dev_info::pfc_queue_tc_max,
application must use queue level PFC configuration via
rte_eth_dev_priority_flow_ctrl_queue_set() API instead of port level
PFC configuration via rte_eth_dev_priority_flow_ctrl_set() API to
realize PFC configuration.

This patch enables the configuration for second case a.k.a queue
based PFC also updates rte_eth_dev_priority_flow_ctrl_set()
implmentaion to adheher to rte_eth_dev_info::pfc_queue_tc_max
handling.

Also updated libabigail.abignore to ignore the update
to reserved fields in rte_eth_dev_info.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
---
 devtools/libabigail.abignore           |  6 ++
 doc/guides/nics/features.rst           |  5 +-
 doc/guides/rel_notes/release_22_03.rst |  4 ++
 lib/ethdev/ethdev_driver.h             |  6 +-
 lib/ethdev/rte_ethdev.c                | 73 ++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h                | 77 +++++++++++++++++++++++++-
 lib/ethdev/version.map                 |  3 +
 7 files changed, 168 insertions(+), 6 deletions(-)

diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index 4b676f317d..e5686115f7 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,9 @@
 ; Ignore generated PMD information strings
 [suppress_variable]
         name_regexp = _pmd_info$
+
+; Ignore fields inserted in place of reserved fields of rte_eth_dev_info
+[suppress_type]
+       name = rte_eth_dev_info
+       has_data_member_inserted_between = {offset_after(switch_info), end}
+
diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 27be2d2576..6bb7337c58 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -379,9 +379,10 @@ Flow control
 Supports configuring link flow control.
 
 * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
-  ``priority_flow_ctrl_set``.
+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_set_t``.
 * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
-  ``rte_eth_dev_priority_flow_ctrl_set()``.
+  ``rte_eth_dev_priority_flow_ctrl_set()``, ``rte_eth_dev_priority_flow_ctrl_queue_set()``.
+* **[provides]   rte_eth_dev_info**: ``pfc_queue_tc_max``.
 
 
 .. _nic_features_rate_limitation:
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 6d99d1eaa9..f336ee2996 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -55,6 +55,10 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added an API to enable queue based priority flow ctrl(PFC).**
+
+  A new API, ``rte_eth_dev_priority_flow_ctrl_queue_set()``, was added.
+
 
 Removed Items
 -------------
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index d95605a355..0cbd434152 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -532,6 +532,9 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
 /** @internal Setup priority flow control parameter on an Ethernet device. */
 typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
 				struct rte_eth_pfc_conf *pfc_conf);
+/** @internal Queue setup for priority flow control parameter on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_set_t)(struct rte_eth_dev *dev,
+		uint16_t queue_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 
 /** @internal Update RSS redirection table on an Ethernet device. */
 typedef int (*reta_update_t)(struct rte_eth_dev *dev,
@@ -1080,7 +1083,8 @@ struct eth_dev_ops {
 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
 	/** Setup priority flow control */
 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
-
+	/** Priority flow control queue setup */
+	priority_flow_ctrl_queue_set_t   priority_flow_ctrl_queue_set;
 	/** Set Unicast Table Array */
 	eth_uc_hash_table_set_t    uc_hash_table_set;
 	/** Set Unicast hash bitmap */
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index a1d475a292..86e0adac24 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -3998,7 +3998,9 @@ int
 rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 				   struct rte_eth_pfc_conf *pfc_conf)
 {
+	struct rte_eth_dev_info dev_info;
 	struct rte_eth_dev *dev;
+	int ret;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
 	dev = &rte_eth_devices[port_id];
@@ -4010,6 +4012,17 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 		return -EINVAL;
 	}
 
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max != 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u driver does not support port level PFC config\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
 	if (pfc_conf->priority > (RTE_ETH_DCB_NUM_USER_PRIORITIES - 1)) {
 		RTE_ETHDEV_LOG(ERR, "Invalid priority, only 0-7 allowed\n");
 		return -EINVAL;
@@ -4022,6 +4035,66 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 	return -ENOTSUP;
 }
 
+int
+rte_eth_dev_priority_flow_ctrl_queue_set(
+	uint16_t port_id, uint16_t queue_id,
+	struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	struct rte_eth_dev_info dev_info;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_conf == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			"Cannot set ethdev port %u queue %d PFC from NULL config\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max == 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u driver does not support PFC TC values\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
+	if (pfc_queue_conf->mode != RTE_ETH_FC_TX_PAUSE &&
+	    queue_id >= dev_info.nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR,
+			"PFC Rx queue not in range(requested: %d configured: %d)\n",
+			queue_id, dev_info.nb_rx_queues);
+		return -EINVAL;
+	}
+
+	if (pfc_queue_conf->mode != RTE_ETH_FC_RX_PAUSE &&
+	    queue_id >= dev_info.nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR,
+			"PFC Tx queue not in range(requested: %d configured: %d)\n",
+			queue_id, dev_info.nb_tx_queues);
+		return -EINVAL;
+	}
+
+	if (pfc_queue_conf->tc >= dev_info.pfc_queue_tc_max) {
+		RTE_ETHDEV_LOG(ERR,
+			"PFC TC not in range(requested: %d max: %d)\n",
+			pfc_queue_conf->tc, dev_info.pfc_queue_tc_max);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_set)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_set)(
+				       dev, queue_id, pfc_queue_conf));
+	return -ENOTSUP;
+}
+
 static int
 eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
 			uint16_t reta_size)
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index fa299c8ad7..b40e54576f 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1395,6 +1395,19 @@ struct rte_eth_pfc_conf {
 	uint8_t priority;          /**< VLAN User Priority. */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to configure Ethernet priority flow control parameter for
+ * ethdev queues.
+ */
+struct rte_eth_pfc_queue_conf {
+	uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) specification */
+	uint16_t pause_time;  /**< Pause quota in the Pause frame */
+	enum rte_eth_fc_mode mode;  /**< Link flow control mode */
+};
+
 /**
  * Tunnel type for device-specific classifier configuration.
  * @see rte_eth_udp_tunnel
@@ -1841,8 +1854,30 @@ struct rte_eth_dev_info {
 	 * embedded managed interconnect/switch.
 	 */
 	struct rte_eth_switch_info switch_info;
-
-	uint64_t reserved_64s[2]; /**< Reserved for future fields */
+	/**
+	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
+	 *
+	 * Based on device support and use-case need, there are two different
+	 * ways to enable PFC. The first case is the port level PFC
+	 * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
+	 * API shall be used to configure the PFC, and PFC frames will be
+	 * generated using based on VLAN TC value.
+	 * The second case is the queue level PFC configuration, in this case,
+	 * Any packet field content can be used to steer the packet to the
+	 * specific queue using rte_flow or RSS and then use
+	 * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
+	 * on each queue. Based on congestion selected on the specific queue,
+	 * configured TC shall be used to generate PFC frames.
+	 *
+	 * When set to non zero value, application must use queue level
+	 * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
+	 * instead of port level PFC configuration via
+	 * rte_eth_dev_priority_flow_ctrl_set() API to realize
+	 * PFC configuration.
+	 */
+	uint8_t pfc_queue_tc_max;
+	uint8_t reserved_8s[7];
+	uint64_t reserved_64s[1]; /**< Reserved for future fields */
 	void *reserved_ptrs[2];   /**< Reserved for future fields */
 };
 
@@ -4109,6 +4144,9 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  * Configure the Ethernet priority flow control under DCB environment
  * for Ethernet device.
  *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority
+ * flow control usage models.
+ *
  * @param port_id
  * The port identifier of the Ethernet device.
  * @param pfc_conf
@@ -4119,10 +4157,43 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  *   - (-ENODEV)  if *port_id* invalid.
  *   - (-EINVAL)  if bad parameter
  *   - (-EIO)     if flow control setup failure or device is removed.
+ *
  */
 int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
-				struct rte_eth_pfc_conf *pfc_conf);
+				       struct rte_eth_pfc_conf *pfc_conf);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Configure the Ethernet priority flow control for a given queue
+ * for Ethernet device.
+ *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority flow control
+ * usage models.
+ *
+ * @note When an ethdev port switches to PFC mode, the unconfigured
+ * queues shall be configured by the driver with default values such as
+ * lower priority value for TC etc.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Rx/Tx queue to apply the PFC configuration.
+ *   @note pfc_queue_conf::mode depicts the queue direction(Rx and/or Tx)
+ * @param pfc_queue_conf
+ *   The pointer to the structure of the priority flow control parameters
+ *   for the queue.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if hardware doesn't support priority flow control mode.
+ *   - (-ENODEV)  if *port_id* invalid.
+ *   - (-EINVAL)  if bad parameter
+ *   - (-EIO)     if flow control setup queue failure
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_set(uint16_t port_id, uint16_t queue_id,
+					     struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 /**
  * Add a MAC address to the set used for filtering incoming packets.
  *
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index c2fb0669a4..8f361ec15a 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -256,6 +256,9 @@ EXPERIMENTAL {
 	rte_flow_flex_item_create;
 	rte_flow_flex_item_release;
 	rte_flow_pick_transfer_proxy;
+
+	# added in 22.03
+	rte_eth_dev_priority_flow_ctrl_queue_set;
 };
 
 INTERNAL {
-- 
2.34.1

Re: [dpdk-dev] [PATCH] ethdev: support queue-based priority flow control

[Stephen Hemminger]

On Sat, 4 Dec 2021 22:54:58 +0530
<jerinj@marvell.com> wrote:

> +	/**
> +	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
> +	 *
> +	 * Based on device support and use-case need, there are two different
> +	 * ways to enable PFC. The first case is the port level PFC
> +	 * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
> +	 * API shall be used to configure the PFC, and PFC frames will be
> +	 * generated using based on VLAN TC value.
> +	 * The second case is the queue level PFC configuration, in this case,
> +	 * Any packet field content can be used to steer the packet to the
> +	 * specific queue using rte_flow or RSS and then use
> +	 * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
> +	 * on each queue. Based on congestion selected on the specific queue,
> +	 * configured TC shall be used to generate PFC frames.
> +	 *
> +	 * When set to non zero value, application must use queue level
> +	 * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
> +	 * instead of port level PFC configuration via
> +	 * rte_eth_dev_priority_flow_ctrl_set() API to realize
> +	 * PFC configuration.
> +	 */
> +	uint8_t pfc_queue_tc_max;
> +	uint8_t reserved_8s[7];
> +	uint64_t reserved_64s[1]; /**< Reserved for future fields */
>  	void *reserved_ptrs[2];   /**< Reserved for future fields */

Not sure you can claim ABI compatibility because the previous versions of DPDK
did not enforce that reserved fields must be zero.  The Linux kernel
learned this when adding flags for new system calls; reserved fields only
work if you enforce that application must set them to zero.

Re: [dpdk-dev] [PATCH] ethdev: support queue-based priority flow control

[Jerin Jacob]

On Sat, Dec 4, 2021 at 11:08 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Sat, 4 Dec 2021 22:54:58 +0530
> <jerinj@marvell.com> wrote:
>
> > +     /**
> > +      * Maximum supported traffic class as per PFC (802.1Qbb) specification.
> > +      *
> > +      * Based on device support and use-case need, there are two different
> > +      * ways to enable PFC. The first case is the port level PFC
> > +      * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
> > +      * API shall be used to configure the PFC, and PFC frames will be
> > +      * generated using based on VLAN TC value.
> > +      * The second case is the queue level PFC configuration, in this case,
> > +      * Any packet field content can be used to steer the packet to the
> > +      * specific queue using rte_flow or RSS and then use
> > +      * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
> > +      * on each queue. Based on congestion selected on the specific queue,
> > +      * configured TC shall be used to generate PFC frames.
> > +      *
> > +      * When set to non zero value, application must use queue level
> > +      * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
> > +      * instead of port level PFC configuration via
> > +      * rte_eth_dev_priority_flow_ctrl_set() API to realize
> > +      * PFC configuration.
> > +      */
> > +     uint8_t pfc_queue_tc_max;
> > +     uint8_t reserved_8s[7];
> > +     uint64_t reserved_64s[1]; /**< Reserved for future fields */
> >       void *reserved_ptrs[2];   /**< Reserved for future fields */
>
> Not sure you can claim ABI compatibility because the previous versions of DPDK
> did not enforce that reserved fields must be zero.  The Linux kernel
> learned this when adding flags for new system calls; reserved fields only
> work if you enforce that application must set them to zero.

In this case it rte_eth_dev_info is an out parameter and implementation of
rte_eth_dev_info_get() already memseting to 0.
Do you still see any other ABI issue?

See rte_eth_dev_info_get()
        /*
         * Init dev_info before port_id check since caller does not have
         * return status and does not know if get is successful or not.
         */
        memset(dev_info, 0, sizeof(struct rte_eth_dev_info));

Re: [dpdk-dev] [PATCH] ethdev: support queue-based priority flow control

[Stephen Hemminger]

On Sun, 5 Dec 2021 12:33:57 +0530
Jerin Jacob <jerinjacobk@gmail.com> wrote:

> On Sat, Dec 4, 2021 at 11:08 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Sat, 4 Dec 2021 22:54:58 +0530
> > <jerinj@marvell.com> wrote:
> >  
> > > +     /**
> > > +      * Maximum supported traffic class as per PFC (802.1Qbb) specification.
> > > +      *
> > > +      * Based on device support and use-case need, there are two different
> > > +      * ways to enable PFC. The first case is the port level PFC
> > > +      * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
> > > +      * API shall be used to configure the PFC, and PFC frames will be
> > > +      * generated using based on VLAN TC value.
> > > +      * The second case is the queue level PFC configuration, in this case,
> > > +      * Any packet field content can be used to steer the packet to the
> > > +      * specific queue using rte_flow or RSS and then use
> > > +      * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
> > > +      * on each queue. Based on congestion selected on the specific queue,
> > > +      * configured TC shall be used to generate PFC frames.
> > > +      *
> > > +      * When set to non zero value, application must use queue level
> > > +      * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
> > > +      * instead of port level PFC configuration via
> > > +      * rte_eth_dev_priority_flow_ctrl_set() API to realize
> > > +      * PFC configuration.
> > > +      */
> > > +     uint8_t pfc_queue_tc_max;
> > > +     uint8_t reserved_8s[7];
> > > +     uint64_t reserved_64s[1]; /**< Reserved for future fields */
> > >       void *reserved_ptrs[2];   /**< Reserved for future fields */  
> >
> > Not sure you can claim ABI compatibility because the previous versions of DPDK
> > did not enforce that reserved fields must be zero.  The Linux kernel
> > learned this when adding flags for new system calls; reserved fields only
> > work if you enforce that application must set them to zero.  
> 
> In this case it rte_eth_dev_info is an out parameter and implementation of
> rte_eth_dev_info_get() already memseting to 0.
> Do you still see any other ABI issue?
> 
> See rte_eth_dev_info_get()
>         /*
>          * Init dev_info before port_id check since caller does not have
>          * return status and does not know if get is successful or not.
>          */
>         memset(dev_info, 0, sizeof(struct rte_eth_dev_info));

The concern was from the misreading comment.  It talks about what application should do.
Could you reword the comment so that it describes what pfc_queue_tc_max is here
and move the flow control set part of the comment to where the API for that is.

Re: [dpdk-dev] [PATCH] ethdev: support queue-based priority flow control

[Jerin Jacob]

On Sun, Dec 5, 2021 at 11:30 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Sun, 5 Dec 2021 12:33:57 +0530
> Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> > On Sat, Dec 4, 2021 at 11:08 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > On Sat, 4 Dec 2021 22:54:58 +0530
> > > <jerinj@marvell.com> wrote:
> > >
> > > > +     /**
> > > > +      * Maximum supported traffic class as per PFC (802.1Qbb) specification.
> > > > +      *
> > > > +      * Based on device support and use-case need, there are two different
> > > > +      * ways to enable PFC. The first case is the port level PFC
> > > > +      * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
> > > > +      * API shall be used to configure the PFC, and PFC frames will be
> > > > +      * generated using based on VLAN TC value.
> > > > +      * The second case is the queue level PFC configuration, in this case,
> > > > +      * Any packet field content can be used to steer the packet to the
> > > > +      * specific queue using rte_flow or RSS and then use
> > > > +      * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
> > > > +      * on each queue. Based on congestion selected on the specific queue,
> > > > +      * configured TC shall be used to generate PFC frames.
> > > > +      *
> > > > +      * When set to non zero value, application must use queue level
> > > > +      * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
> > > > +      * instead of port level PFC configuration via
> > > > +      * rte_eth_dev_priority_flow_ctrl_set() API to realize
> > > > +      * PFC configuration.
> > > > +      */
> > > > +     uint8_t pfc_queue_tc_max;
> > > > +     uint8_t reserved_8s[7];
> > > > +     uint64_t reserved_64s[1]; /**< Reserved for future fields */
> > > >       void *reserved_ptrs[2];   /**< Reserved for future fields */
> > >
> > > Not sure you can claim ABI compatibility because the previous versions of DPDK
> > > did not enforce that reserved fields must be zero.  The Linux kernel
> > > learned this when adding flags for new system calls; reserved fields only
> > > work if you enforce that application must set them to zero.
> >
> > In this case it rte_eth_dev_info is an out parameter and implementation of
> > rte_eth_dev_info_get() already memseting to 0.
> > Do you still see any other ABI issue?
> >
> > See rte_eth_dev_info_get()
> >         /*
> >          * Init dev_info before port_id check since caller does not have
> >          * return status and does not know if get is successful or not.
> >          */
> >         memset(dev_info, 0, sizeof(struct rte_eth_dev_info));
>
> The concern was from the misreading comment.  It talks about what application should do.
> Could you reword the comment so that it describes what pfc_queue_tc_max is here

The comment is at rte_eth_dev_info::pfc_queue_tc_max. So it is implied
that get pararamter.
current comment
---

+ * When set to non zero value, application must use queue level
+ * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
+ * instead of port level PFC configuration via
+ * rte_eth_dev_priority_flow_ctrl_set() API to realize
+ * PFC configuration.

---
Is updating to following help to clarify. If so, I will send v2, if
not, Please suggest.

---
+ * When set to non zero value by the driver, application must use queue level
                                                  ^^^^^^^^^^^
+ * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
+ * instead of port level PFC configuration via
+ * rte_eth_dev_priority_flow_ctrl_set() API to realize
+ * PFC configuration.
---

> and move the flow control set part of the comment to where the API for that is.

The comment is needed for rte_eth_dev_priority_flow_ctrl_set() and
rte_eth_dev_priority_flow_ctrl_queue_set().
Instead of duplicating the comments, I added the comment at
rte_eth_dev_info::pfc_queue_tc_max and
added "@see struct rte_eth_dev_info::pfc_queue_tc_max priority flow
control usage models"
in rte_eth_dev_priority_flow_ctrl_set() and
rte_eth_dev_priority_flow_ctrl_queue_set().

[PATCH v1 1/2] ethdev: support queue-based priority flow control

[skori]

From: Sunil Kumar Kori <skori@marvell.com>

Based on device support and use-case need, there are two different ways
to enable PFC. The first case is the port level PFC configuration, in
this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
configure the PFC, and PFC frames will be generated using based on VLAN
TC value.

The second case is the queue level PFC configuration, in this
case, Any packet field content can be used to steer the packet to the
specific queue using rte_flow or RSS and then use
rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping on each
queue. Based on congestion selected on the specific queue, configured TC
shall be used to generate PFC frames.

Operation of these modes are mutually exclusive, when driver sets
non zero value for rte_eth_dev_info::pfc_queue_tc_max,
application must use queue level PFC configuration via
rte_eth_dev_priority_flow_ctrl_queue_set() API instead of port level
PFC configuration via rte_eth_dev_priority_flow_ctrl_set() API to
realize PFC configuration.

This patch enables the configuration for second case a.k.a queue
based PFC also updates rte_eth_dev_priority_flow_ctrl_set()
implmentaion to adheher to rte_eth_dev_info::pfc_queue_tc_max
handling.

Also updated libabigail.abignore to ignore the update
to reserved fields in rte_eth_dev_info.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
Change-Id: Ibd67e65914cc7c4164c779ba0a59fb807b6e2baa
---
v1:
 - Added queue based PFC config API instead port based

 devtools/libabigail.abignore           |   5 ++
 doc/guides/nics/features.rst           |   5 +-
 doc/guides/rel_notes/release_22_03.rst |   3 +
 lib/ethdev/ethdev_driver.h             |   6 +-
 lib/ethdev/rte_ethdev.c                | 109 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h                |  83 ++++++++++++++++++-
 lib/ethdev/version.map                 |   3 +
 7 files changed, 208 insertions(+), 6 deletions(-)

diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index 4b676f317d..1d29e60c06 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,8 @@
 ; Ignore generated PMD information strings
 [suppress_variable]
         name_regexp = _pmd_info$
+
+; Ignore fields inserted in place of reserved fields of rte_eth_dev_info
+[suppress_type]
+       name = rte_eth_dev_info
+       has_data_member_inserted_between = {offset_after(switch_info), end}
diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 27be2d2576..277a784f4e 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -379,9 +379,10 @@ Flow control
 Supports configuring link flow control.
 
 * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
-  ``priority_flow_ctrl_set``.
+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_set``.
 * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
-  ``rte_eth_dev_priority_flow_ctrl_set()``.
+  ``rte_eth_dev_priority_flow_ctrl_set()``, ``rte_eth_dev_priority_flow_ctrl_queue_set()``.
+* **[provides]   rte_eth_dev_info**: ``pfc_queue_tc_max``.
 
 
 .. _nic_features_rate_limitation:
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 6d99d1eaa9..b75c0356e6 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -55,6 +55,9 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added an API to enable queue based priority flow ctrl(PFC).**
+
+  A new API, ``rte_eth_dev_priority_flow_ctrl_queue_set()``, was added.
 
 Removed Items
 -------------
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index d95605a355..e0bbfe89d7 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -532,6 +532,9 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
 /** @internal Setup priority flow control parameter on an Ethernet device. */
 typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
 				struct rte_eth_pfc_conf *pfc_conf);
+/** @internal Queue setup for priority flow control parameter on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_set_t)(struct rte_eth_dev *dev,
+				struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 
 /** @internal Update RSS redirection table on an Ethernet device. */
 typedef int (*reta_update_t)(struct rte_eth_dev *dev,
@@ -1080,7 +1083,8 @@ struct eth_dev_ops {
 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
 	/** Setup priority flow control */
 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
-
+	/** Priority flow control queue setup */
+	priority_flow_ctrl_queue_set_t   priority_flow_ctrl_queue_set;
 	/** Set Unicast Table Array */
 	eth_uc_hash_table_set_t    uc_hash_table_set;
 	/** Set Unicast hash bitmap */
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index a1d475a292..6def057720 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -3998,7 +3998,9 @@ int
 rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 				   struct rte_eth_pfc_conf *pfc_conf)
 {
+	struct rte_eth_dev_info dev_info;
 	struct rte_eth_dev *dev;
+	int ret;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
 	dev = &rte_eth_devices[port_id];
@@ -4010,6 +4012,17 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 		return -EINVAL;
 	}
 
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max != 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u driver does not support port level PFC config\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
 	if (pfc_conf->priority > (RTE_ETH_DCB_NUM_USER_PRIORITIES - 1)) {
 		RTE_ETHDEV_LOG(ERR, "Invalid priority, only 0-7 allowed\n");
 		return -EINVAL;
@@ -4022,6 +4035,102 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 	return -ENOTSUP;
 }
 
+static inline int
+validate_rx_pause_config(struct rte_eth_dev_info *dev_info,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
+	    (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx pause"
+				       " (requested: %d configured: %d)\n",
+				       pfc_queue_conf->rx_pause.tx_qid,
+				       dev_info->nb_tx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->rx_pause.tc >= dev_info->pfc_queue_tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
+				       " (requested: %d max: %d)\n",
+				       pfc_queue_conf->rx_pause.tc,
+				       dev_info->pfc_queue_tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+validate_tx_pause_config(struct rte_eth_dev_info *dev_info,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
+	     (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx pause"
+				       "(requested: %d configured: %d)\n",
+				       pfc_queue_conf->tx_pause.rx_qid,
+				       dev_info->nb_rx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->tx_pause.tc >= dev_info->pfc_queue_tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
+				       "(requested: %d max: %d)\n",
+				       pfc_queue_conf->tx_pause.tc,
+				       dev_info->pfc_queue_tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_set(
+	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	struct rte_eth_dev_info dev_info;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_conf == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max == 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u does not support PFC TC values\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
+	ret = validate_rx_pause_config(&dev_info, pfc_queue_conf);
+	if (ret != 0)
+		return ret;
+
+	ret = validate_tx_pause_config(&dev_info, pfc_queue_conf);
+	if (ret != 0)
+		return ret;
+
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_set)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_set)(
+				       dev, pfc_queue_conf));
+	return -ENOTSUP;
+}
+
 static int
 eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
 			uint16_t reta_size)
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index fa299c8ad7..d13b6df3b9 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1395,6 +1395,28 @@ struct rte_eth_pfc_conf {
 	uint8_t priority;          /**< VLAN User Priority. */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to configure Ethernet priority flow control parameter for
+ * ethdev queues.
+ */
+struct rte_eth_pfc_queue_conf {
+	enum rte_eth_fc_mode mode;  /**< Link flow control mode */
+
+	struct {
+		uint16_t tx_qid;
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
+
+	struct {
+		uint16_t pause_time; /**< Pause quota in the Pause frame */
+		uint16_t rx_qid;
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
+};
+
 /**
  * Tunnel type for device-specific classifier configuration.
  * @see rte_eth_udp_tunnel
@@ -1841,8 +1863,30 @@ struct rte_eth_dev_info {
 	 * embedded managed interconnect/switch.
 	 */
 	struct rte_eth_switch_info switch_info;
-
-	uint64_t reserved_64s[2]; /**< Reserved for future fields */
+	/**
+	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
+	 *
+	 * Based on device support and use-case need, there are two different
+	 * ways to enable PFC. The first case is the port level PFC
+	 * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
+	 * API shall be used to configure the PFC, and PFC frames will be
+	 * generated using based on VLAN TC value.
+	 * The second case is the queue level PFC configuration, in this case,
+	 * Any packet field content can be used to steer the packet to the
+	 * specific queue using rte_flow or RSS and then use
+	 * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
+	 * on each queue. Based on congestion selected on the specific queue,
+	 * configured TC shall be used to generate PFC frames.
+	 *
+	 * When set to non zero value, application must use queue level
+	 * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
+	 * instead of port level PFC configuration via
+	 * rte_eth_dev_priority_flow_ctrl_set() API to realize
+	 * PFC configuration.
+	 */
+	uint8_t pfc_queue_tc_max;
+	uint8_t reserved_8s[7];
+	uint64_t reserved_64s[1]; /**< Reserved for future fields */
 	void *reserved_ptrs[2];   /**< Reserved for future fields */
 };
 
@@ -4109,6 +4153,9 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  * Configure the Ethernet priority flow control under DCB environment
  * for Ethernet device.
  *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority
+ * flow control usage models.
+ *
  * @param port_id
  * The port identifier of the Ethernet device.
  * @param pfc_conf
@@ -4119,10 +4166,40 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  *   - (-ENODEV)  if *port_id* invalid.
  *   - (-EINVAL)  if bad parameter
  *   - (-EIO)     if flow control setup failure or device is removed.
+ *
  */
 int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
-				struct rte_eth_pfc_conf *pfc_conf);
+				       struct rte_eth_pfc_conf *pfc_conf);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Configure the Ethernet priority flow control for a given queue
+ * for Ethernet device.
+ *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority flow control
+ * usage models.
+ *
+ * @note When an ethdev port switches to PFC mode, the unconfigured
+ * queues shall be configured by the driver with default values such as
+ * lower priority value for TC etc.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_conf
+ *   The pointer to the structure of the priority flow control parameters
+ *   for the queue.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if hardware doesn't support priority flow control mode.
+ *   - (-ENODEV)  if *port_id* invalid.
+ *   - (-EINVAL)  if bad parameter
+ *   - (-EIO)     if flow control setup queue failure
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_set(uint16_t port_id,
+					     struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 /**
  * Add a MAC address to the set used for filtering incoming packets.
  *
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index c2fb0669a4..8f361ec15a 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -256,6 +256,9 @@ EXPERIMENTAL {
 	rte_flow_flex_item_create;
 	rte_flow_flex_item_release;
 	rte_flow_pick_transfer_proxy;
+
+	# added in 22.03
+	rte_eth_dev_priority_flow_ctrl_queue_set;
 };
 
 INTERNAL {
-- 
2.25.1

[PATCH v1 2/2] app/testpmd: add queue based pfc CLI options

[skori]

From: Sunil Kumar Kori <skori@marvell.com>

Patch adds command line options to configure queue based
priority flow control.

- Syntax command is given as below:

set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
	tx <on|off> <rx_qid> <rx_tc> <pause_time>

- Example command to configure queue based priority flow control
  on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
  time 2047

testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047

Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
Change-Id: Ic6d0e3b1ba5f0a98ca7a5b7cafb0044c6c568b81
---
 app/test-pmd/cmdline.c                      | 122 ++++++++++++++++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  22 ++++
 2 files changed, 144 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 6e10afeedd..0240fcf9e7 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -544,6 +544,11 @@ static void cmd_help_long_parsed(void *parsed_result,
 			"    Set the priority flow control parameter on a"
 			" port.\n\n"
 
+			"set pfc_queue_ctrl (port_id) rx (on|off) (tx_qid)"
+			" (tx_tc) tx (on|off) (rx_qid) (rx_tc) (pause_time)\n"
+			"    Set the queue priority flow control parameter on a"
+			" given Rx and Tx queues of a port.\n\n"
+
 			"set stat_qmap (tx|rx) (port_id) (queue_id) (qmapping)\n"
 			"    Set statistics mapping (qmapping 0..15) for RX/TX"
 			" queue on port.\n"
@@ -7690,6 +7695,122 @@ cmdline_parse_inst_t cmd_priority_flow_control_set = {
 	},
 };
 
+struct cmd_queue_priority_flow_ctrl_set_result {
+	cmdline_fixed_string_t set;
+	cmdline_fixed_string_t pfc_queue_ctrl;
+	portid_t port_id;
+	cmdline_fixed_string_t rx;
+	cmdline_fixed_string_t rx_pfc_mode;
+	uint16_t tx_qid;
+	uint8_t  tx_tc;
+	cmdline_fixed_string_t tx;
+	cmdline_fixed_string_t tx_pfc_mode;
+	uint16_t rx_qid;
+	uint8_t  rx_tc;
+	uint16_t pause_time;
+};
+
+static void
+cmd_queue_priority_flow_ctrl_set_parsed(void *parsed_result,
+					__rte_unused struct cmdline *cl,
+					__rte_unused void *data)
+{
+	struct cmd_queue_priority_flow_ctrl_set_result *res = parsed_result;
+	struct rte_eth_pfc_queue_conf pfc_queue_conf;
+	int rx_fc_enable, tx_fc_enable;
+	int ret;
+
+	/*
+	 * Rx on/off, flow control is enabled/disabled on RX side. This can
+	 * indicate the RTE_ETH_FC_TX_PAUSE, Transmit pause frame at the Rx
+	 * side. Tx on/off, flow control is enabled/disabled on TX side. This
+	 * can indicate the RTE_ETH_FC_RX_PAUSE, Respond to the pause frame at
+	 * the Tx side.
+	 */
+	static enum rte_eth_fc_mode rx_tx_onoff_2_mode[2][2] = {
+		{RTE_ETH_FC_NONE, RTE_ETH_FC_TX_PAUSE},
+		{RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_FULL}
+	};
+
+	memset(&pfc_queue_conf, 0, sizeof(struct rte_eth_pfc_queue_conf));
+	rx_fc_enable = (!strncmp(res->rx_pfc_mode, "on", 2)) ? 1 : 0;
+	tx_fc_enable = (!strncmp(res->tx_pfc_mode, "on", 2)) ? 1 : 0;
+	pfc_queue_conf.mode = rx_tx_onoff_2_mode[rx_fc_enable][tx_fc_enable];
+	pfc_queue_conf.rx_pause.tc  = res->tx_tc;
+	pfc_queue_conf.rx_pause.tx_qid = res->tx_qid;
+	pfc_queue_conf.tx_pause.tc  = res->rx_tc;
+	pfc_queue_conf.tx_pause.rx_qid  = res->rx_qid;
+	pfc_queue_conf.tx_pause.pause_time = res->pause_time;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_set(res->port_id,
+						       &pfc_queue_conf);
+	if (ret != 0) {
+		fprintf(stderr,
+			"bad queue priority flow control parameter, rc = %d\n",
+			ret);
+	}
+}
+
+cmdline_parse_token_string_t cmd_q_pfc_set_set =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				set, "set");
+cmdline_parse_token_string_t cmd_q_pfc_set_flow_ctrl =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pfc_queue_ctrl, "pfc_queue_ctrl");
+cmdline_parse_token_num_t cmd_q_pfc_set_portid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				port_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_q_pfc_set_rx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx, "rx");
+cmdline_parse_token_string_t cmd_q_pfc_set_rx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_tc, RTE_UINT8);
+cmdline_parse_token_string_t cmd_q_pfc_set_tx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx, "tx");
+cmdline_parse_token_string_t cmd_q_pfc_set_tx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_tc, RTE_UINT8);
+cmdline_parse_token_num_t cmd_q_pfc_set_pause_time =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pause_time, RTE_UINT16);
+
+cmdline_parse_inst_t cmd_queue_priority_flow_control_set = {
+	.f = cmd_queue_priority_flow_ctrl_set_parsed,
+	.data = NULL,
+	.help_str = "set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> "
+		"tx <on|off> <rx_qid> <rx_tc> <pause_time>: "
+		"Configure the Ethernet queue priority flow control",
+	.tokens = {
+		(void *)&cmd_q_pfc_set_set,
+		(void *)&cmd_q_pfc_set_flow_ctrl,
+		(void *)&cmd_q_pfc_set_portid,
+		(void *)&cmd_q_pfc_set_rx,
+		(void *)&cmd_q_pfc_set_rx_mode,
+		(void *)&cmd_q_pfc_set_tx_qid,
+		(void *)&cmd_q_pfc_set_tx_tc,
+		(void *)&cmd_q_pfc_set_tx,
+		(void *)&cmd_q_pfc_set_tx_mode,
+		(void *)&cmd_q_pfc_set_rx_qid,
+		(void *)&cmd_q_pfc_set_rx_tc,
+		(void *)&cmd_q_pfc_set_pause_time,
+		NULL,
+	},
+};
+
 /* *** RESET CONFIGURATION *** */
 struct cmd_reset_result {
 	cmdline_fixed_string_t reset;
@@ -17765,6 +17886,7 @@ cmdline_parse_ctx_t main_ctx[] = {
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_set_autoneg,
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_show,
 	(cmdline_parse_inst_t *)&cmd_priority_flow_control_set,
+	(cmdline_parse_inst_t *)&cmd_queue_priority_flow_control_set,
 	(cmdline_parse_inst_t *)&cmd_config_dcb,
 	(cmdline_parse_inst_t *)&cmd_read_reg,
 	(cmdline_parse_inst_t *)&cmd_read_reg_bit_field,
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 44228cd7d2..5ea2888adc 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -1561,6 +1561,28 @@ Where:
 
 * ``priority`` (0-7): VLAN User Priority.
 
+set pfc_queue_ctrl
+~~~~~~~~~~~~~~~~~~
+
+Set the priority flow control parameter on a given Rx and Tx queue of a port::
+
+   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
+            tx (on|off) <rx_qid> <rx_tc> <pause_time>
+
+Where:
+
+* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
+  will be paused when PFC frame is received with ``tx_tc`` enabled.
+
+* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
+
+* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
+  frame will be generated with ``tx_tc`` when exceeds the threshold.
+
+* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
+
+* ``pause_time`` (integer): Pause quota filled in the PFC frame.
+
 set stat_qmap
 ~~~~~~~~~~~~~
 
-- 
2.25.1

[PATCH v1 1/2] ethdev: support queue-based priority flow control

[skori]

From: Sunil Kumar Kori <skori@marvell.com>

Based on device support and use-case need, there are two different ways
to enable PFC. The first case is the port level PFC configuration, in
this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
configure the PFC, and PFC frames will be generated using based on VLAN
TC value.

The second case is the queue level PFC configuration, in this
case, Any packet field content can be used to steer the packet to the
specific queue using rte_flow or RSS and then use
rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping on each
queue. Based on congestion selected on the specific queue, configured TC
shall be used to generate PFC frames.

Operation of these modes are mutually exclusive, when driver sets
non zero value for rte_eth_dev_info::pfc_queue_tc_max,
application must use queue level PFC configuration via
rte_eth_dev_priority_flow_ctrl_queue_set() API instead of port level
PFC configuration via rte_eth_dev_priority_flow_ctrl_set() API to
realize PFC configuration.

This patch enables the configuration for second case a.k.a queue
based PFC also updates rte_eth_dev_priority_flow_ctrl_set()
implmentaion to adheher to rte_eth_dev_info::pfc_queue_tc_max
handling.

Also updated libabigail.abignore to ignore the update
to reserved fields in rte_eth_dev_info.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
v1:
 - Added queue based PFC config API instead port based

 devtools/libabigail.abignore           |   5 ++
 doc/guides/nics/features.rst           |   5 +-
 doc/guides/rel_notes/release_22_03.rst |   3 +
 lib/ethdev/ethdev_driver.h             |   6 +-
 lib/ethdev/rte_ethdev.c                | 109 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h                |  83 ++++++++++++++++++-
 lib/ethdev/version.map                 |   3 +
 7 files changed, 208 insertions(+), 6 deletions(-)

diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index 4b676f317d..1d29e60c06 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,8 @@
 ; Ignore generated PMD information strings
 [suppress_variable]
         name_regexp = _pmd_info$
+
+; Ignore fields inserted in place of reserved fields of rte_eth_dev_info
+[suppress_type]
+       name = rte_eth_dev_info
+       has_data_member_inserted_between = {offset_after(switch_info), end}
diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 27be2d2576..277a784f4e 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -379,9 +379,10 @@ Flow control
 Supports configuring link flow control.
 
 * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
-  ``priority_flow_ctrl_set``.
+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_set``.
 * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
-  ``rte_eth_dev_priority_flow_ctrl_set()``.
+  ``rte_eth_dev_priority_flow_ctrl_set()``, ``rte_eth_dev_priority_flow_ctrl_queue_set()``.
+* **[provides]   rte_eth_dev_info**: ``pfc_queue_tc_max``.
 
 
 .. _nic_features_rate_limitation:
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 6d99d1eaa9..b75c0356e6 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -55,6 +55,9 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added an API to enable queue based priority flow ctrl(PFC).**
+
+  A new API, ``rte_eth_dev_priority_flow_ctrl_queue_set()``, was added.
 
 Removed Items
 -------------
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index d95605a355..e0bbfe89d7 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -532,6 +532,9 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
 /** @internal Setup priority flow control parameter on an Ethernet device. */
 typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
 				struct rte_eth_pfc_conf *pfc_conf);
+/** @internal Queue setup for priority flow control parameter on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_set_t)(struct rte_eth_dev *dev,
+				struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 
 /** @internal Update RSS redirection table on an Ethernet device. */
 typedef int (*reta_update_t)(struct rte_eth_dev *dev,
@@ -1080,7 +1083,8 @@ struct eth_dev_ops {
 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
 	/** Setup priority flow control */
 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
-
+	/** Priority flow control queue setup */
+	priority_flow_ctrl_queue_set_t   priority_flow_ctrl_queue_set;
 	/** Set Unicast Table Array */
 	eth_uc_hash_table_set_t    uc_hash_table_set;
 	/** Set Unicast hash bitmap */
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index a1d475a292..6def057720 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -3998,7 +3998,9 @@ int
 rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 				   struct rte_eth_pfc_conf *pfc_conf)
 {
+	struct rte_eth_dev_info dev_info;
 	struct rte_eth_dev *dev;
+	int ret;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
 	dev = &rte_eth_devices[port_id];
@@ -4010,6 +4012,17 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 		return -EINVAL;
 	}
 
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max != 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u driver does not support port level PFC config\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
 	if (pfc_conf->priority > (RTE_ETH_DCB_NUM_USER_PRIORITIES - 1)) {
 		RTE_ETHDEV_LOG(ERR, "Invalid priority, only 0-7 allowed\n");
 		return -EINVAL;
@@ -4022,6 +4035,102 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 	return -ENOTSUP;
 }
 
+static inline int
+validate_rx_pause_config(struct rte_eth_dev_info *dev_info,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
+	    (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx pause"
+				       " (requested: %d configured: %d)\n",
+				       pfc_queue_conf->rx_pause.tx_qid,
+				       dev_info->nb_tx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->rx_pause.tc >= dev_info->pfc_queue_tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
+				       " (requested: %d max: %d)\n",
+				       pfc_queue_conf->rx_pause.tc,
+				       dev_info->pfc_queue_tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+validate_tx_pause_config(struct rte_eth_dev_info *dev_info,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
+	     (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx pause"
+				       "(requested: %d configured: %d)\n",
+				       pfc_queue_conf->tx_pause.rx_qid,
+				       dev_info->nb_rx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->tx_pause.tc >= dev_info->pfc_queue_tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
+				       "(requested: %d max: %d)\n",
+				       pfc_queue_conf->tx_pause.tc,
+				       dev_info->pfc_queue_tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_set(
+	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	struct rte_eth_dev_info dev_info;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_conf == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max == 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u does not support PFC TC values\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
+	ret = validate_rx_pause_config(&dev_info, pfc_queue_conf);
+	if (ret != 0)
+		return ret;
+
+	ret = validate_tx_pause_config(&dev_info, pfc_queue_conf);
+	if (ret != 0)
+		return ret;
+
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_set)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_set)(
+				       dev, pfc_queue_conf));
+	return -ENOTSUP;
+}
+
 static int
 eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
 			uint16_t reta_size)
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index fa299c8ad7..d13b6df3b9 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1395,6 +1395,28 @@ struct rte_eth_pfc_conf {
 	uint8_t priority;          /**< VLAN User Priority. */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to configure Ethernet priority flow control parameter for
+ * ethdev queues.
+ */
+struct rte_eth_pfc_queue_conf {
+	enum rte_eth_fc_mode mode;  /**< Link flow control mode */
+
+	struct {
+		uint16_t tx_qid;
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
+
+	struct {
+		uint16_t pause_time; /**< Pause quota in the Pause frame */
+		uint16_t rx_qid;
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
+};
+
 /**
  * Tunnel type for device-specific classifier configuration.
  * @see rte_eth_udp_tunnel
@@ -1841,8 +1863,30 @@ struct rte_eth_dev_info {
 	 * embedded managed interconnect/switch.
 	 */
 	struct rte_eth_switch_info switch_info;
-
-	uint64_t reserved_64s[2]; /**< Reserved for future fields */
+	/**
+	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
+	 *
+	 * Based on device support and use-case need, there are two different
+	 * ways to enable PFC. The first case is the port level PFC
+	 * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
+	 * API shall be used to configure the PFC, and PFC frames will be
+	 * generated using based on VLAN TC value.
+	 * The second case is the queue level PFC configuration, in this case,
+	 * Any packet field content can be used to steer the packet to the
+	 * specific queue using rte_flow or RSS and then use
+	 * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
+	 * on each queue. Based on congestion selected on the specific queue,
+	 * configured TC shall be used to generate PFC frames.
+	 *
+	 * When set to non zero value, application must use queue level
+	 * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
+	 * instead of port level PFC configuration via
+	 * rte_eth_dev_priority_flow_ctrl_set() API to realize
+	 * PFC configuration.
+	 */
+	uint8_t pfc_queue_tc_max;
+	uint8_t reserved_8s[7];
+	uint64_t reserved_64s[1]; /**< Reserved for future fields */
 	void *reserved_ptrs[2];   /**< Reserved for future fields */
 };
 
@@ -4109,6 +4153,9 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  * Configure the Ethernet priority flow control under DCB environment
  * for Ethernet device.
  *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority
+ * flow control usage models.
+ *
  * @param port_id
  * The port identifier of the Ethernet device.
  * @param pfc_conf
@@ -4119,10 +4166,40 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  *   - (-ENODEV)  if *port_id* invalid.
  *   - (-EINVAL)  if bad parameter
  *   - (-EIO)     if flow control setup failure or device is removed.
+ *
  */
 int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
-				struct rte_eth_pfc_conf *pfc_conf);
+				       struct rte_eth_pfc_conf *pfc_conf);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Configure the Ethernet priority flow control for a given queue
+ * for Ethernet device.
+ *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority flow control
+ * usage models.
+ *
+ * @note When an ethdev port switches to PFC mode, the unconfigured
+ * queues shall be configured by the driver with default values such as
+ * lower priority value for TC etc.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_conf
+ *   The pointer to the structure of the priority flow control parameters
+ *   for the queue.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if hardware doesn't support priority flow control mode.
+ *   - (-ENODEV)  if *port_id* invalid.
+ *   - (-EINVAL)  if bad parameter
+ *   - (-EIO)     if flow control setup queue failure
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_set(uint16_t port_id,
+					     struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 /**
  * Add a MAC address to the set used for filtering incoming packets.
  *
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index c2fb0669a4..8f361ec15a 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -256,6 +256,9 @@ EXPERIMENTAL {
 	rte_flow_flex_item_create;
 	rte_flow_flex_item_release;
 	rte_flow_pick_transfer_proxy;
+
+	# added in 22.03
+	rte_eth_dev_priority_flow_ctrl_queue_set;
 };
 
 INTERNAL {
-- 
2.25.1

[PATCH v1 2/2] app/testpmd: add queue based pfc CLI options

[skori]

From: Sunil Kumar Kori <skori@marvell.com>

Patch adds command line options to configure queue based
priority flow control.

- Syntax command is given as below:

set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
	tx <on|off> <rx_qid> <rx_tc> <pause_time>

- Example command to configure queue based priority flow control
  on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
  time 2047

testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047

Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
 app/test-pmd/cmdline.c                      | 122 ++++++++++++++++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  22 ++++
 2 files changed, 144 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 6e10afeedd..0240fcf9e7 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -544,6 +544,11 @@ static void cmd_help_long_parsed(void *parsed_result,
 			"    Set the priority flow control parameter on a"
 			" port.\n\n"
 
+			"set pfc_queue_ctrl (port_id) rx (on|off) (tx_qid)"
+			" (tx_tc) tx (on|off) (rx_qid) (rx_tc) (pause_time)\n"
+			"    Set the queue priority flow control parameter on a"
+			" given Rx and Tx queues of a port.\n\n"
+
 			"set stat_qmap (tx|rx) (port_id) (queue_id) (qmapping)\n"
 			"    Set statistics mapping (qmapping 0..15) for RX/TX"
 			" queue on port.\n"
@@ -7690,6 +7695,122 @@ cmdline_parse_inst_t cmd_priority_flow_control_set = {
 	},
 };
 
+struct cmd_queue_priority_flow_ctrl_set_result {
+	cmdline_fixed_string_t set;
+	cmdline_fixed_string_t pfc_queue_ctrl;
+	portid_t port_id;
+	cmdline_fixed_string_t rx;
+	cmdline_fixed_string_t rx_pfc_mode;
+	uint16_t tx_qid;
+	uint8_t  tx_tc;
+	cmdline_fixed_string_t tx;
+	cmdline_fixed_string_t tx_pfc_mode;
+	uint16_t rx_qid;
+	uint8_t  rx_tc;
+	uint16_t pause_time;
+};
+
+static void
+cmd_queue_priority_flow_ctrl_set_parsed(void *parsed_result,
+					__rte_unused struct cmdline *cl,
+					__rte_unused void *data)
+{
+	struct cmd_queue_priority_flow_ctrl_set_result *res = parsed_result;
+	struct rte_eth_pfc_queue_conf pfc_queue_conf;
+	int rx_fc_enable, tx_fc_enable;
+	int ret;
+
+	/*
+	 * Rx on/off, flow control is enabled/disabled on RX side. This can
+	 * indicate the RTE_ETH_FC_TX_PAUSE, Transmit pause frame at the Rx
+	 * side. Tx on/off, flow control is enabled/disabled on TX side. This
+	 * can indicate the RTE_ETH_FC_RX_PAUSE, Respond to the pause frame at
+	 * the Tx side.
+	 */
+	static enum rte_eth_fc_mode rx_tx_onoff_2_mode[2][2] = {
+		{RTE_ETH_FC_NONE, RTE_ETH_FC_TX_PAUSE},
+		{RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_FULL}
+	};
+
+	memset(&pfc_queue_conf, 0, sizeof(struct rte_eth_pfc_queue_conf));
+	rx_fc_enable = (!strncmp(res->rx_pfc_mode, "on", 2)) ? 1 : 0;
+	tx_fc_enable = (!strncmp(res->tx_pfc_mode, "on", 2)) ? 1 : 0;
+	pfc_queue_conf.mode = rx_tx_onoff_2_mode[rx_fc_enable][tx_fc_enable];
+	pfc_queue_conf.rx_pause.tc  = res->tx_tc;
+	pfc_queue_conf.rx_pause.tx_qid = res->tx_qid;
+	pfc_queue_conf.tx_pause.tc  = res->rx_tc;
+	pfc_queue_conf.tx_pause.rx_qid  = res->rx_qid;
+	pfc_queue_conf.tx_pause.pause_time = res->pause_time;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_set(res->port_id,
+						       &pfc_queue_conf);
+	if (ret != 0) {
+		fprintf(stderr,
+			"bad queue priority flow control parameter, rc = %d\n",
+			ret);
+	}
+}
+
+cmdline_parse_token_string_t cmd_q_pfc_set_set =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				set, "set");
+cmdline_parse_token_string_t cmd_q_pfc_set_flow_ctrl =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pfc_queue_ctrl, "pfc_queue_ctrl");
+cmdline_parse_token_num_t cmd_q_pfc_set_portid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				port_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_q_pfc_set_rx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx, "rx");
+cmdline_parse_token_string_t cmd_q_pfc_set_rx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_tc, RTE_UINT8);
+cmdline_parse_token_string_t cmd_q_pfc_set_tx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx, "tx");
+cmdline_parse_token_string_t cmd_q_pfc_set_tx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_tc, RTE_UINT8);
+cmdline_parse_token_num_t cmd_q_pfc_set_pause_time =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pause_time, RTE_UINT16);
+
+cmdline_parse_inst_t cmd_queue_priority_flow_control_set = {
+	.f = cmd_queue_priority_flow_ctrl_set_parsed,
+	.data = NULL,
+	.help_str = "set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> "
+		"tx <on|off> <rx_qid> <rx_tc> <pause_time>: "
+		"Configure the Ethernet queue priority flow control",
+	.tokens = {
+		(void *)&cmd_q_pfc_set_set,
+		(void *)&cmd_q_pfc_set_flow_ctrl,
+		(void *)&cmd_q_pfc_set_portid,
+		(void *)&cmd_q_pfc_set_rx,
+		(void *)&cmd_q_pfc_set_rx_mode,
+		(void *)&cmd_q_pfc_set_tx_qid,
+		(void *)&cmd_q_pfc_set_tx_tc,
+		(void *)&cmd_q_pfc_set_tx,
+		(void *)&cmd_q_pfc_set_tx_mode,
+		(void *)&cmd_q_pfc_set_rx_qid,
+		(void *)&cmd_q_pfc_set_rx_tc,
+		(void *)&cmd_q_pfc_set_pause_time,
+		NULL,
+	},
+};
+
 /* *** RESET CONFIGURATION *** */
 struct cmd_reset_result {
 	cmdline_fixed_string_t reset;
@@ -17765,6 +17886,7 @@ cmdline_parse_ctx_t main_ctx[] = {
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_set_autoneg,
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_show,
 	(cmdline_parse_inst_t *)&cmd_priority_flow_control_set,
+	(cmdline_parse_inst_t *)&cmd_queue_priority_flow_control_set,
 	(cmdline_parse_inst_t *)&cmd_config_dcb,
 	(cmdline_parse_inst_t *)&cmd_read_reg,
 	(cmdline_parse_inst_t *)&cmd_read_reg_bit_field,
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 44228cd7d2..5ea2888adc 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -1561,6 +1561,28 @@ Where:
 
 * ``priority`` (0-7): VLAN User Priority.
 
+set pfc_queue_ctrl
+~~~~~~~~~~~~~~~~~~
+
+Set the priority flow control parameter on a given Rx and Tx queue of a port::
+
+   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
+            tx (on|off) <rx_qid> <rx_tc> <pause_time>
+
+Where:
+
+* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
+  will be paused when PFC frame is received with ``tx_tc`` enabled.
+
+* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
+
+* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
+  frame will be generated with ``tx_tc`` when exceeds the threshold.
+
+* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
+
+* ``pause_time`` (integer): Pause quota filled in the PFC frame.
+
 set stat_qmap
 ~~~~~~~~~~~~~
 
-- 
2.25.1

RE: [PATCH v1 1/2] ethdev: support queue-based priority flow control

[Sunil Kumar Kori]

As per discussion with Jerin against the RFC http://patches.dpdk.org/project/dpdk/patch/20211204172458.1904300-1-jerinj@marvell.com/, following change set adds queue based PFC configuration. 
Also corresponding implementation for the API is available at http://patches.dpdk.org/project/dpdk/patch/20220109111130.751933-2-skori@marvell.com/ for Marvell platforms. 

Regards
Sunil Kumar Kori

>-----Original Message-----
>From: skori@marvell.com <skori@marvell.com>
>Sent: Sunday, January 9, 2022 4:29 PM
>To: Ray Kinsella <mdr@ashroe.eu>; Thomas Monjalon
><thomas@monjalon.net>; Ferruh Yigit <ferruh.yigit@intel.com>; Andrew
>Rybchenko <andrew.rybchenko@oktetlabs.ru>
>Cc: dev@dpdk.org; Sunil Kumar Kori <skori@marvell.com>; Jerin Jacob
>Kollanukkaran <jerinj@marvell.com>
>Subject: [PATCH v1 1/2] ethdev: support queue-based priority flow control
>
>From: Sunil Kumar Kori <skori@marvell.com>
>
>Based on device support and use-case need, there are two different ways to
>enable PFC. The first case is the port level PFC configuration, in this case,
>rte_eth_dev_priority_flow_ctrl_set() API shall be used to configure the PFC,
>and PFC frames will be generated using based on VLAN TC value.
>
>The second case is the queue level PFC configuration, in this case, Any packet
>field content can be used to steer the packet to the specific queue using
>rte_flow or RSS and then use
>rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping on each
>queue. Based on congestion selected on the specific queue, configured TC
>shall be used to generate PFC frames.
>
>Operation of these modes are mutually exclusive, when driver sets non zero
>value for rte_eth_dev_info::pfc_queue_tc_max,
>application must use queue level PFC configuration via
>rte_eth_dev_priority_flow_ctrl_queue_set() API instead of port level PFC
>configuration via rte_eth_dev_priority_flow_ctrl_set() API to realize PFC
>configuration.
>
>This patch enables the configuration for second case a.k.a queue based PFC
>also updates rte_eth_dev_priority_flow_ctrl_set()
>implmentaion to adheher to rte_eth_dev_info::pfc_queue_tc_max
>handling.
>
>Also updated libabigail.abignore to ignore the update to reserved fields in
>rte_eth_dev_info.
>
>Signed-off-by: Jerin Jacob <jerinj@marvell.com>
>Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
>---
>v1:
> - Added queue based PFC config API instead port based
>
> devtools/libabigail.abignore           |   5 ++
> doc/guides/nics/features.rst           |   5 +-
> doc/guides/rel_notes/release_22_03.rst |   3 +
> lib/ethdev/ethdev_driver.h             |   6 +-
> lib/ethdev/rte_ethdev.c                | 109 +++++++++++++++++++++++++
> lib/ethdev/rte_ethdev.h                |  83 ++++++++++++++++++-
> lib/ethdev/version.map                 |   3 +
> 7 files changed, 208 insertions(+), 6 deletions(-)
>
>diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore index
>4b676f317d..1d29e60c06 100644
>--- a/devtools/libabigail.abignore
>+++ b/devtools/libabigail.abignore
>@@ -11,3 +11,8 @@
> ; Ignore generated PMD information strings  [suppress_variable]
>         name_regexp = _pmd_info$
>+
>+; Ignore fields inserted in place of reserved fields of
>+rte_eth_dev_info [suppress_type]
>+       name = rte_eth_dev_info
>+       has_data_member_inserted_between = {offset_after(switch_info),
>+end}
>diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst index
>27be2d2576..277a784f4e 100644
>--- a/doc/guides/nics/features.rst
>+++ b/doc/guides/nics/features.rst
>@@ -379,9 +379,10 @@ Flow control
> Supports configuring link flow control.
>
> * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
>-  ``priority_flow_ctrl_set``.
>+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_set``.
> * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``,
>``rte_eth_dev_flow_ctrl_set()``,
>-  ``rte_eth_dev_priority_flow_ctrl_set()``.
>+  ``rte_eth_dev_priority_flow_ctrl_set()``,
>``rte_eth_dev_priority_flow_ctrl_queue_set()``.
>+* **[provides]   rte_eth_dev_info**: ``pfc_queue_tc_max``.
>
>
> .. _nic_features_rate_limitation:
>diff --git a/doc/guides/rel_notes/release_22_03.rst
>b/doc/guides/rel_notes/release_22_03.rst
>index 6d99d1eaa9..b75c0356e6 100644
>--- a/doc/guides/rel_notes/release_22_03.rst
>+++ b/doc/guides/rel_notes/release_22_03.rst
>@@ -55,6 +55,9 @@ New Features
>      Also, make sure to start the actual text at the margin.
>      =======================================================
>
>+* **Added an API to enable queue based priority flow ctrl(PFC).**
>+
>+  A new API, ``rte_eth_dev_priority_flow_ctrl_queue_set()``, was added.
>
> Removed Items
> -------------
>diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h index
>d95605a355..e0bbfe89d7 100644
>--- a/lib/ethdev/ethdev_driver.h
>+++ b/lib/ethdev/ethdev_driver.h
>@@ -532,6 +532,9 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
> /** @internal Setup priority flow control parameter on an Ethernet device. */
>typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
> 				struct rte_eth_pfc_conf *pfc_conf);
>+/** @internal Queue setup for priority flow control parameter on an
>+Ethernet device. */ typedef int (*priority_flow_ctrl_queue_set_t)(struct
>rte_eth_dev *dev,
>+				struct rte_eth_pfc_queue_conf
>*pfc_queue_conf);
>
> /** @internal Update RSS redirection table on an Ethernet device. */  typedef
>int (*reta_update_t)(struct rte_eth_dev *dev, @@ -1080,7 +1083,8 @@ struct
>eth_dev_ops {
> 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
> 	/** Setup priority flow control */
> 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
>-
>+	/** Priority flow control queue setup */
>+	priority_flow_ctrl_queue_set_t   priority_flow_ctrl_queue_set;
> 	/** Set Unicast Table Array */
> 	eth_uc_hash_table_set_t    uc_hash_table_set;
> 	/** Set Unicast hash bitmap */
>diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index
>a1d475a292..6def057720 100644
>--- a/lib/ethdev/rte_ethdev.c
>+++ b/lib/ethdev/rte_ethdev.c
>@@ -3998,7 +3998,9 @@ int
> rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
> 				   struct rte_eth_pfc_conf *pfc_conf)  {
>+	struct rte_eth_dev_info dev_info;
> 	struct rte_eth_dev *dev;
>+	int ret;
>
> 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> 	dev = &rte_eth_devices[port_id];
>@@ -4010,6 +4012,17 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t
>port_id,
> 		return -EINVAL;
> 	}
>
>+	ret = rte_eth_dev_info_get(port_id, &dev_info);
>+	if (ret != 0)
>+		return ret;
>+
>+	if (dev_info.pfc_queue_tc_max != 0) {
>+		RTE_ETHDEV_LOG(ERR,
>+			"Ethdev port %u driver does not support port level PFC
>config\n",
>+			port_id);
>+		return -ENOTSUP;
>+	}
>+
> 	if (pfc_conf->priority > (RTE_ETH_DCB_NUM_USER_PRIORITIES - 1)) {
> 		RTE_ETHDEV_LOG(ERR, "Invalid priority, only 0-7 allowed\n");
> 		return -EINVAL;
>@@ -4022,6 +4035,102 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t
>port_id,
> 	return -ENOTSUP;
> }
>
>+static inline int
>+validate_rx_pause_config(struct rte_eth_dev_info *dev_info,
>+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf) {
>+	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
>+	    (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
>+		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info-
>>nb_tx_queues) {
>+			RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx
>pause"
>+				       " (requested: %d configured: %d)\n",
>+				       pfc_queue_conf->rx_pause.tx_qid,
>+				       dev_info->nb_tx_queues);
>+			return -EINVAL;
>+		}
>+
>+		if (pfc_queue_conf->rx_pause.tc >= dev_info-
>>pfc_queue_tc_max) {
>+			RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
>+				       " (requested: %d max: %d)\n",
>+				       pfc_queue_conf->rx_pause.tc,
>+				       dev_info->pfc_queue_tc_max);
>+			return -EINVAL;
>+		}
>+	}
>+
>+	return 0;
>+}
>+
>+static inline int
>+validate_tx_pause_config(struct rte_eth_dev_info *dev_info,
>+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf) {
>+	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
>+	     (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
>+		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info-
>>nb_rx_queues) {
>+			RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx
>pause"
>+				       "(requested: %d configured: %d)\n",
>+				       pfc_queue_conf->tx_pause.rx_qid,
>+				       dev_info->nb_rx_queues);
>+			return -EINVAL;
>+		}
>+
>+		if (pfc_queue_conf->tx_pause.tc >= dev_info-
>>pfc_queue_tc_max) {
>+			RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
>+				       "(requested: %d max: %d)\n",
>+				       pfc_queue_conf->tx_pause.tc,
>+				       dev_info->pfc_queue_tc_max);
>+			return -EINVAL;
>+		}
>+	}
>+
>+	return 0;
>+}
>+
>+int
>+rte_eth_dev_priority_flow_ctrl_queue_set(
>+	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf) {
>+	struct rte_eth_dev_info dev_info;
>+	struct rte_eth_dev *dev;
>+	int ret;
>+
>+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
>+	dev = &rte_eth_devices[port_id];
>+
>+	if (pfc_queue_conf == NULL) {
>+		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port
>(%u)\n",
>+			       port_id);
>+		return -EINVAL;
>+	}
>+
>+	ret = rte_eth_dev_info_get(port_id, &dev_info);
>+	if (ret != 0)
>+		return ret;
>+
>+	if (dev_info.pfc_queue_tc_max == 0) {
>+		RTE_ETHDEV_LOG(ERR,
>+			"Ethdev port %u does not support PFC TC values\n",
>+			port_id);
>+		return -ENOTSUP;
>+	}
>+
>+	ret = validate_rx_pause_config(&dev_info, pfc_queue_conf);
>+	if (ret != 0)
>+		return ret;
>+
>+	ret = validate_tx_pause_config(&dev_info, pfc_queue_conf);
>+	if (ret != 0)
>+		return ret;
>+
>+
>+	if (*dev->dev_ops->priority_flow_ctrl_queue_set)
>+		return eth_err(port_id,
>+			       (*dev->dev_ops->priority_flow_ctrl_queue_set)(
>+				       dev, pfc_queue_conf));
>+	return -ENOTSUP;
>+}
>+
> static int
> eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
> 			uint16_t reta_size)
>diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index
>fa299c8ad7..d13b6df3b9 100644
>--- a/lib/ethdev/rte_ethdev.h
>+++ b/lib/ethdev/rte_ethdev.h
>@@ -1395,6 +1395,28 @@ struct rte_eth_pfc_conf {
> 	uint8_t priority;          /**< VLAN User Priority. */
> };
>
>+/**
>+ * @warning
>+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
>+notice
>+ *
>+ * A structure used to configure Ethernet priority flow control
>+parameter for
>+ * ethdev queues.
>+ */
>+struct rte_eth_pfc_queue_conf {
>+	enum rte_eth_fc_mode mode;  /**< Link flow control mode */
>+
>+	struct {
>+		uint16_t tx_qid;
>+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
>+	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode ==
>FC_FULL) */
>+
>+	struct {
>+		uint16_t pause_time; /**< Pause quota in the Pause frame */
>+		uint16_t rx_qid;
>+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
>+	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode ==
>FC_FULL) */
>+};
>+
> /**
>  * Tunnel type for device-specific classifier configuration.
>  * @see rte_eth_udp_tunnel
>@@ -1841,8 +1863,30 @@ struct rte_eth_dev_info {
> 	 * embedded managed interconnect/switch.
> 	 */
> 	struct rte_eth_switch_info switch_info;
>-
>-	uint64_t reserved_64s[2]; /**< Reserved for future fields */
>+	/**
>+	 * Maximum supported traffic class as per PFC (802.1Qbb)
>specification.
>+	 *
>+	 * Based on device support and use-case need, there are two different
>+	 * ways to enable PFC. The first case is the port level PFC
>+	 * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
>+	 * API shall be used to configure the PFC, and PFC frames will be
>+	 * generated using based on VLAN TC value.
>+	 * The second case is the queue level PFC configuration, in this case,
>+	 * Any packet field content can be used to steer the packet to the
>+	 * specific queue using rte_flow or RSS and then use
>+	 * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
>+	 * on each queue. Based on congestion selected on the specific queue,
>+	 * configured TC shall be used to generate PFC frames.
>+	 *
>+	 * When set to non zero value, application must use queue level
>+	 * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set()
>API
>+	 * instead of port level PFC configuration via
>+	 * rte_eth_dev_priority_flow_ctrl_set() API to realize
>+	 * PFC configuration.
>+	 */
>+	uint8_t pfc_queue_tc_max;
>+	uint8_t reserved_8s[7];
>+	uint64_t reserved_64s[1]; /**< Reserved for future fields */
> 	void *reserved_ptrs[2];   /**< Reserved for future fields */
> };
>
>@@ -4109,6 +4153,9 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
>  * Configure the Ethernet priority flow control under DCB environment
>  * for Ethernet device.
>  *
>+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority
>+ * flow control usage models.
>+ *
>  * @param port_id
>  * The port identifier of the Ethernet device.
>  * @param pfc_conf
>@@ -4119,10 +4166,40 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
>  *   - (-ENODEV)  if *port_id* invalid.
>  *   - (-EINVAL)  if bad parameter
>  *   - (-EIO)     if flow control setup failure or device is removed.
>+ *
>  */
> int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>-				struct rte_eth_pfc_conf *pfc_conf);
>+				       struct rte_eth_pfc_conf *pfc_conf);
>
>+/**
>+ * @warning
>+ * @b EXPERIMENTAL: this API may change without prior notice.
>+ *
>+ * Configure the Ethernet priority flow control for a given queue
>+ * for Ethernet device.
>+ *
>+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority flow control
>+ * usage models.
>+ *
>+ * @note When an ethdev port switches to PFC mode, the unconfigured
>+ * queues shall be configured by the driver with default values such as
>+ * lower priority value for TC etc.
>+ *
>+ * @param port_id
>+ *   The port identifier of the Ethernet device.
>+ * @param pfc_queue_conf
>+ *   The pointer to the structure of the priority flow control parameters
>+ *   for the queue.
>+ * @return
>+ *   - (0) if successful.
>+ *   - (-ENOTSUP) if hardware doesn't support priority flow control mode.
>+ *   - (-ENODEV)  if *port_id* invalid.
>+ *   - (-EINVAL)  if bad parameter
>+ *   - (-EIO)     if flow control setup queue failure
>+ */
>+__rte_experimental
>+int rte_eth_dev_priority_flow_ctrl_queue_set(uint16_t port_id,
>+					     struct rte_eth_pfc_queue_conf
>*pfc_queue_conf);
> /**
>  * Add a MAC address to the set used for filtering incoming packets.
>  *
>diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map index
>c2fb0669a4..8f361ec15a 100644
>--- a/lib/ethdev/version.map
>+++ b/lib/ethdev/version.map
>@@ -256,6 +256,9 @@ EXPERIMENTAL {
> 	rte_flow_flex_item_create;
> 	rte_flow_flex_item_release;
> 	rte_flow_pick_transfer_proxy;
>+
>+	# added in 22.03
>+	rte_eth_dev_priority_flow_ctrl_queue_set;
> };
>
> INTERNAL {
>--
>2.25.1

[dpdk-dev] [PATCH v2 1/2] ethdev: support queue-based priority flow control

[jerinj]

From: Jerin Jacob <jerinj@marvell.com>

Based on device support and use-case need, there are two different ways
to enable PFC. The first case is the port level PFC configuration, in
this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
configure the PFC, and PFC frames will be generated using based on VLAN
TC value.

The second case is the queue level PFC configuration, in this
case, Any packet field content can be used to steer the packet to the
specific queue using rte_flow or RSS and then use
rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping on each
queue. Based on congestion selected on the specific queue, configured TC
shall be used to generate PFC frames.

Operation of these modes are mutually exclusive, when driver sets
non zero value for rte_eth_dev_info::pfc_queue_tc_max,
application must use queue level PFC configuration via
rte_eth_dev_priority_flow_ctrl_queue_set() API instead of port level
PFC configuration via rte_eth_dev_priority_flow_ctrl_set() API to
realize PFC configuration.

This patch enables the configuration for second case a.k.a queue
based PFC also updates rte_eth_dev_priority_flow_ctrl_set()
implmentaion to adheher to rte_eth_dev_info::pfc_queue_tc_max
handling.

Also updated libabigail.abignore to ignore the update
to reserved fields in rte_eth_dev_info.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---

A driver implemtion based on this API is at
https://patches.dpdk.org/project/dpdk/patch/20220111081831.881374-1-skori@marvell.com/

RFC..v1
 - Added queue based PFC config API instead port based

v1..v2
- Updated libabigail.ignore as
  has_data_member_inserted_between = {offset_of(reserved_64s), end}
- Updated doxygen comments of rte_eth_pfc_queue_conf


 devtools/libabigail.abignore           |   5 ++
 doc/guides/nics/features.rst           |   5 +-
 doc/guides/rel_notes/release_22_03.rst |   3 +
 lib/ethdev/ethdev_driver.h             |   6 +-
 lib/ethdev/rte_ethdev.c                | 109 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h                |  83 ++++++++++++++++++-
 lib/ethdev/version.map                 |   3 +
 7 files changed, 208 insertions(+), 6 deletions(-)

diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index 4b676f317d..3bdecaaef0 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -11,3 +11,8 @@
 ; Ignore generated PMD information strings
 [suppress_variable]
         name_regexp = _pmd_info$
+
+;Ignore fields inserted in place of reserved fields of rte_eth_dev_info
+[suppress_type]
+       name = rte_eth_dev_info
+       has_data_member_inserted_between = {offset_of(reserved_64s), end}
diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 27be2d2576..277a784f4e 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -379,9 +379,10 @@ Flow control
 Supports configuring link flow control.
 
 * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
-  ``priority_flow_ctrl_set``.
+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_set``.
 * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
-  ``rte_eth_dev_priority_flow_ctrl_set()``.
+  ``rte_eth_dev_priority_flow_ctrl_set()``, ``rte_eth_dev_priority_flow_ctrl_queue_set()``.
+* **[provides]   rte_eth_dev_info**: ``pfc_queue_tc_max``.
 
 
 .. _nic_features_rate_limitation:
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 6d99d1eaa9..b75c0356e6 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -55,6 +55,9 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added an API to enable queue based priority flow ctrl(PFC).**
+
+  A new API, ``rte_eth_dev_priority_flow_ctrl_queue_set()``, was added.
 
 Removed Items
 -------------
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index d95605a355..e0bbfe89d7 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -532,6 +532,9 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
 /** @internal Setup priority flow control parameter on an Ethernet device. */
 typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
 				struct rte_eth_pfc_conf *pfc_conf);
+/** @internal Queue setup for priority flow control parameter on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_set_t)(struct rte_eth_dev *dev,
+				struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 
 /** @internal Update RSS redirection table on an Ethernet device. */
 typedef int (*reta_update_t)(struct rte_eth_dev *dev,
@@ -1080,7 +1083,8 @@ struct eth_dev_ops {
 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
 	/** Setup priority flow control */
 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
-
+	/** Priority flow control queue setup */
+	priority_flow_ctrl_queue_set_t   priority_flow_ctrl_queue_set;
 	/** Set Unicast Table Array */
 	eth_uc_hash_table_set_t    uc_hash_table_set;
 	/** Set Unicast hash bitmap */
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index a1d475a292..6def057720 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -3998,7 +3998,9 @@ int
 rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 				   struct rte_eth_pfc_conf *pfc_conf)
 {
+	struct rte_eth_dev_info dev_info;
 	struct rte_eth_dev *dev;
+	int ret;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
 	dev = &rte_eth_devices[port_id];
@@ -4010,6 +4012,17 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 		return -EINVAL;
 	}
 
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max != 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u driver does not support port level PFC config\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
 	if (pfc_conf->priority > (RTE_ETH_DCB_NUM_USER_PRIORITIES - 1)) {
 		RTE_ETHDEV_LOG(ERR, "Invalid priority, only 0-7 allowed\n");
 		return -EINVAL;
@@ -4022,6 +4035,102 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 	return -ENOTSUP;
 }
 
+static inline int
+validate_rx_pause_config(struct rte_eth_dev_info *dev_info,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
+	    (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx pause"
+				       " (requested: %d configured: %d)\n",
+				       pfc_queue_conf->rx_pause.tx_qid,
+				       dev_info->nb_tx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->rx_pause.tc >= dev_info->pfc_queue_tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
+				       " (requested: %d max: %d)\n",
+				       pfc_queue_conf->rx_pause.tc,
+				       dev_info->pfc_queue_tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+validate_tx_pause_config(struct rte_eth_dev_info *dev_info,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
+	     (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx pause"
+				       "(requested: %d configured: %d)\n",
+				       pfc_queue_conf->tx_pause.rx_qid,
+				       dev_info->nb_rx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->tx_pause.tc >= dev_info->pfc_queue_tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
+				       "(requested: %d max: %d)\n",
+				       pfc_queue_conf->tx_pause.tc,
+				       dev_info->pfc_queue_tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_set(
+	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	struct rte_eth_dev_info dev_info;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_conf == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	if (dev_info.pfc_queue_tc_max == 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u does not support PFC TC values\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
+	ret = validate_rx_pause_config(&dev_info, pfc_queue_conf);
+	if (ret != 0)
+		return ret;
+
+	ret = validate_tx_pause_config(&dev_info, pfc_queue_conf);
+	if (ret != 0)
+		return ret;
+
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_set)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_set)(
+				       dev, pfc_queue_conf));
+	return -ENOTSUP;
+}
+
 static int
 eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
 			uint16_t reta_size)
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index fa299c8ad7..33e43f6e1d 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1395,6 +1395,28 @@ struct rte_eth_pfc_conf {
 	uint8_t priority;          /**< VLAN User Priority. */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to configure Ethernet priority flow control parameter for
+ * ethdev queues.
+ */
+struct rte_eth_pfc_queue_conf {
+	enum rte_eth_fc_mode mode;  /**< Link flow control mode */
+
+	struct {
+		uint16_t tx_qid; /**< Tx queue ID */
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
+
+	struct {
+		uint16_t pause_time; /**< Pause quota in the Pause frame */
+		uint16_t rx_qid; /**< Rx queue ID */
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
+};
+
 /**
  * Tunnel type for device-specific classifier configuration.
  * @see rte_eth_udp_tunnel
@@ -1841,8 +1863,30 @@ struct rte_eth_dev_info {
 	 * embedded managed interconnect/switch.
 	 */
 	struct rte_eth_switch_info switch_info;
-
-	uint64_t reserved_64s[2]; /**< Reserved for future fields */
+	/**
+	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
+	 *
+	 * Based on device support and use-case need, there are two different
+	 * ways to enable PFC. The first case is the port level PFC
+	 * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
+	 * API shall be used to configure the PFC, and PFC frames will be
+	 * generated using based on VLAN TC value.
+	 * The second case is the queue level PFC configuration, in this case,
+	 * Any packet field content can be used to steer the packet to the
+	 * specific queue using rte_flow or RSS and then use
+	 * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
+	 * on each queue. Based on congestion selected on the specific queue,
+	 * configured TC shall be used to generate PFC frames.
+	 *
+	 * When set to non zero value, application must use queue level
+	 * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
+	 * instead of port level PFC configuration via
+	 * rte_eth_dev_priority_flow_ctrl_set() API to realize
+	 * PFC configuration.
+	 */
+	uint8_t pfc_queue_tc_max;
+	uint8_t reserved_8s[7];
+	uint64_t reserved_64s[1]; /**< Reserved for future fields */
 	void *reserved_ptrs[2];   /**< Reserved for future fields */
 };
 
@@ -4109,6 +4153,9 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  * Configure the Ethernet priority flow control under DCB environment
  * for Ethernet device.
  *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority
+ * flow control usage models.
+ *
  * @param port_id
  * The port identifier of the Ethernet device.
  * @param pfc_conf
@@ -4119,10 +4166,40 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
  *   - (-ENODEV)  if *port_id* invalid.
  *   - (-EINVAL)  if bad parameter
  *   - (-EIO)     if flow control setup failure or device is removed.
+ *
  */
 int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
-				struct rte_eth_pfc_conf *pfc_conf);
+				       struct rte_eth_pfc_conf *pfc_conf);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Configure the Ethernet priority flow control for a given queue
+ * for Ethernet device.
+ *
+ * @see struct rte_eth_dev_info::pfc_queue_tc_max priority flow control
+ * usage models.
+ *
+ * @note When an ethdev port switches to PFC mode, the unconfigured
+ * queues shall be configured by the driver with default values such as
+ * lower priority value for TC etc.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_conf
+ *   The pointer to the structure of the priority flow control parameters
+ *   for the queue.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if hardware doesn't support priority flow control mode.
+ *   - (-ENODEV)  if *port_id* invalid.
+ *   - (-EINVAL)  if bad parameter
+ *   - (-EIO)     if flow control setup queue failure
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_set(uint16_t port_id,
+					     struct rte_eth_pfc_queue_conf *pfc_queue_conf);
 /**
  * Add a MAC address to the set used for filtering incoming packets.
  *
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index c2fb0669a4..8f361ec15a 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -256,6 +256,9 @@ EXPERIMENTAL {
 	rte_flow_flex_item_create;
 	rte_flow_flex_item_release;
 	rte_flow_pick_transfer_proxy;
+
+	# added in 22.03
+	rte_eth_dev_priority_flow_ctrl_queue_set;
 };
 
 INTERNAL {
-- 
2.34.1

[dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based pfc CLI options

[jerinj]

From: Sunil Kumar Kori <skori@marvell.com>

Patch adds command line options to configure queue based
priority flow control.

- Syntax command is given as below:

set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
	tx <on|off> <rx_qid> <rx_tc> <pause_time>

- Example command to configure queue based priority flow control
  on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
  time 2047

testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047

Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
 app/test-pmd/cmdline.c                      | 122 ++++++++++++++++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  22 ++++
 2 files changed, 144 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index e626b1c7d9..19dbcea4f1 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -544,6 +544,11 @@ static void cmd_help_long_parsed(void *parsed_result,
 			"    Set the priority flow control parameter on a"
 			" port.\n\n"
 
+			"set pfc_queue_ctrl (port_id) rx (on|off) (tx_qid)"
+			" (tx_tc) tx (on|off) (rx_qid) (rx_tc) (pause_time)\n"
+			"    Set the queue priority flow control parameter on a"
+			" given Rx and Tx queues of a port.\n\n"
+
 			"set stat_qmap (tx|rx) (port_id) (queue_id) (qmapping)\n"
 			"    Set statistics mapping (qmapping 0..15) for RX/TX"
 			" queue on port.\n"
@@ -7690,6 +7695,122 @@ cmdline_parse_inst_t cmd_priority_flow_control_set = {
 	},
 };
 
+struct cmd_queue_priority_flow_ctrl_set_result {
+	cmdline_fixed_string_t set;
+	cmdline_fixed_string_t pfc_queue_ctrl;
+	portid_t port_id;
+	cmdline_fixed_string_t rx;
+	cmdline_fixed_string_t rx_pfc_mode;
+	uint16_t tx_qid;
+	uint8_t  tx_tc;
+	cmdline_fixed_string_t tx;
+	cmdline_fixed_string_t tx_pfc_mode;
+	uint16_t rx_qid;
+	uint8_t  rx_tc;
+	uint16_t pause_time;
+};
+
+static void
+cmd_queue_priority_flow_ctrl_set_parsed(void *parsed_result,
+					__rte_unused struct cmdline *cl,
+					__rte_unused void *data)
+{
+	struct cmd_queue_priority_flow_ctrl_set_result *res = parsed_result;
+	struct rte_eth_pfc_queue_conf pfc_queue_conf;
+	int rx_fc_enable, tx_fc_enable;
+	int ret;
+
+	/*
+	 * Rx on/off, flow control is enabled/disabled on RX side. This can
+	 * indicate the RTE_ETH_FC_TX_PAUSE, Transmit pause frame at the Rx
+	 * side. Tx on/off, flow control is enabled/disabled on TX side. This
+	 * can indicate the RTE_ETH_FC_RX_PAUSE, Respond to the pause frame at
+	 * the Tx side.
+	 */
+	static enum rte_eth_fc_mode rx_tx_onoff_2_mode[2][2] = {
+		{RTE_ETH_FC_NONE, RTE_ETH_FC_TX_PAUSE},
+		{RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_FULL}
+	};
+
+	memset(&pfc_queue_conf, 0, sizeof(struct rte_eth_pfc_queue_conf));
+	rx_fc_enable = (!strncmp(res->rx_pfc_mode, "on", 2)) ? 1 : 0;
+	tx_fc_enable = (!strncmp(res->tx_pfc_mode, "on", 2)) ? 1 : 0;
+	pfc_queue_conf.mode = rx_tx_onoff_2_mode[rx_fc_enable][tx_fc_enable];
+	pfc_queue_conf.rx_pause.tc  = res->tx_tc;
+	pfc_queue_conf.rx_pause.tx_qid = res->tx_qid;
+	pfc_queue_conf.tx_pause.tc  = res->rx_tc;
+	pfc_queue_conf.tx_pause.rx_qid  = res->rx_qid;
+	pfc_queue_conf.tx_pause.pause_time = res->pause_time;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_set(res->port_id,
+						       &pfc_queue_conf);
+	if (ret != 0) {
+		fprintf(stderr,
+			"bad queue priority flow control parameter, rc = %d\n",
+			ret);
+	}
+}
+
+cmdline_parse_token_string_t cmd_q_pfc_set_set =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				set, "set");
+cmdline_parse_token_string_t cmd_q_pfc_set_flow_ctrl =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pfc_queue_ctrl, "pfc_queue_ctrl");
+cmdline_parse_token_num_t cmd_q_pfc_set_portid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				port_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_q_pfc_set_rx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx, "rx");
+cmdline_parse_token_string_t cmd_q_pfc_set_rx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_tc, RTE_UINT8);
+cmdline_parse_token_string_t cmd_q_pfc_set_tx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx, "tx");
+cmdline_parse_token_string_t cmd_q_pfc_set_tx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_tc, RTE_UINT8);
+cmdline_parse_token_num_t cmd_q_pfc_set_pause_time =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pause_time, RTE_UINT16);
+
+cmdline_parse_inst_t cmd_queue_priority_flow_control_set = {
+	.f = cmd_queue_priority_flow_ctrl_set_parsed,
+	.data = NULL,
+	.help_str = "set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> "
+		"tx <on|off> <rx_qid> <rx_tc> <pause_time>: "
+		"Configure the Ethernet queue priority flow control",
+	.tokens = {
+		(void *)&cmd_q_pfc_set_set,
+		(void *)&cmd_q_pfc_set_flow_ctrl,
+		(void *)&cmd_q_pfc_set_portid,
+		(void *)&cmd_q_pfc_set_rx,
+		(void *)&cmd_q_pfc_set_rx_mode,
+		(void *)&cmd_q_pfc_set_tx_qid,
+		(void *)&cmd_q_pfc_set_tx_tc,
+		(void *)&cmd_q_pfc_set_tx,
+		(void *)&cmd_q_pfc_set_tx_mode,
+		(void *)&cmd_q_pfc_set_rx_qid,
+		(void *)&cmd_q_pfc_set_rx_tc,
+		(void *)&cmd_q_pfc_set_pause_time,
+		NULL,
+	},
+};
+
 /* *** RESET CONFIGURATION *** */
 struct cmd_reset_result {
 	cmdline_fixed_string_t reset;
@@ -17765,6 +17886,7 @@ cmdline_parse_ctx_t main_ctx[] = {
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_set_autoneg,
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_show,
 	(cmdline_parse_inst_t *)&cmd_priority_flow_control_set,
+	(cmdline_parse_inst_t *)&cmd_queue_priority_flow_control_set,
 	(cmdline_parse_inst_t *)&cmd_config_dcb,
 	(cmdline_parse_inst_t *)&cmd_read_reg,
 	(cmdline_parse_inst_t *)&cmd_read_reg_bit_field,
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 94792d88cc..82333d518e 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -1561,6 +1561,28 @@ Where:
 
 * ``priority`` (0-7): VLAN User Priority.
 
+set pfc_queue_ctrl
+~~~~~~~~~~~~~~~~~~
+
+Set the priority flow control parameter on a given Rx and Tx queue of a port::
+
+   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
+            tx (on|off) <rx_qid> <rx_tc> <pause_time>
+
+Where:
+
+* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
+  will be paused when PFC frame is received with ``tx_tc`` enabled.
+
+* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
+
+* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
+  frame will be generated with ``tx_tc`` when exceeds the threshold.
+
+* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
+
+* ``pause_time`` (integer): Pause quota filled in the PFC frame.
+
 set stat_qmap
 ~~~~~~~~~~~~~
 
-- 
2.34.1

Re: [dpdk-dev] [PATCH v2 1/2] ethdev: support queue-based priority flow control

[Ferruh Yigit]

On 1/13/2022 10:27 AM, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
> 
> Based on device support and use-case need, there are two different ways
> to enable PFC. The first case is the port level PFC configuration, in
> this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
> configure the PFC, and PFC frames will be generated using based on VLAN
> TC value.
> 
> The second case is the queue level PFC configuration, in this
> case, Any packet field content can be used to steer the packet to the
> specific queue using rte_flow or RSS and then use
> rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping on each
> queue. Based on congestion selected on the specific queue, configured TC
> shall be used to generate PFC frames.
> > Operation of these modes are mutually exclusive, when driver sets
> non zero value for rte_eth_dev_info::pfc_queue_tc_max,
> application must use queue level PFC configuration via
> rte_eth_dev_priority_flow_ctrl_queue_set() API instead of port level
> PFC configuration via rte_eth_dev_priority_flow_ctrl_set() API to
> realize PFC configuration.
> 
> This patch enables the configuration for second case a.k.a queue
> based PFC also updates rte_eth_dev_priority_flow_ctrl_set()
> implmentaion to adheher to rte_eth_dev_info::pfc_queue_tc_max
> handling.
> 
> Also updated libabigail.abignore to ignore the update
> to reserved fields in rte_eth_dev_info.
> 
> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> ---
> 
> A driver implemtion based on this API is at
> https://patches.dpdk.org/project/dpdk/patch/20220111081831.881374-1-skori@marvell.com/
> 
> RFC..v1
>   - Added queue based PFC config API instead port based
> 
> v1..v2
> - Updated libabigail.ignore as
>    has_data_member_inserted_between = {offset_of(reserved_64s), end}
> - Updated doxygen comments of rte_eth_pfc_queue_conf
> 
> 
>   devtools/libabigail.abignore           |   5 ++
>   doc/guides/nics/features.rst           |   5 +-
>   doc/guides/rel_notes/release_22_03.rst |   3 +
>   lib/ethdev/ethdev_driver.h             |   6 +-
>   lib/ethdev/rte_ethdev.c                | 109 +++++++++++++++++++++++++
>   lib/ethdev/rte_ethdev.h                |  83 ++++++++++++++++++-
>   lib/ethdev/version.map                 |   3 +
>   7 files changed, 208 insertions(+), 6 deletions(-)
> 
> diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
> index 4b676f317d..3bdecaaef0 100644
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -11,3 +11,8 @@
>   ; Ignore generated PMD information strings
>   [suppress_variable]
>           name_regexp = _pmd_info$
> +
> +;Ignore fields inserted in place of reserved fields of rte_eth_dev_info
> +[suppress_type]
> +       name = rte_eth_dev_info
> +       has_data_member_inserted_between = {offset_of(reserved_64s), end}
> diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> index 27be2d2576..277a784f4e 100644
> --- a/doc/guides/nics/features.rst
> +++ b/doc/guides/nics/features.rst
> @@ -379,9 +379,10 @@ Flow control
>   Supports configuring link flow control.
>   
>   * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
> -  ``priority_flow_ctrl_set``.
> +  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_set``.
>   * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
> -  ``rte_eth_dev_priority_flow_ctrl_set()``.
> +  ``rte_eth_dev_priority_flow_ctrl_set()``, ``rte_eth_dev_priority_flow_ctrl_queue_set()``.
> +* **[provides]   rte_eth_dev_info**: ``pfc_queue_tc_max``.
>   
>   
>   .. _nic_features_rate_limitation:
> diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
> index 6d99d1eaa9..b75c0356e6 100644
> --- a/doc/guides/rel_notes/release_22_03.rst
> +++ b/doc/guides/rel_notes/release_22_03.rst
> @@ -55,6 +55,9 @@ New Features
>        Also, make sure to start the actual text at the margin.
>        =======================================================
>   
> +* **Added an API to enable queue based priority flow ctrl(PFC).**
> +
> +  A new API, ``rte_eth_dev_priority_flow_ctrl_queue_set()``, was added.
>   
>   Removed Items
>   -------------
> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
> index d95605a355..e0bbfe89d7 100644
> --- a/lib/ethdev/ethdev_driver.h
> +++ b/lib/ethdev/ethdev_driver.h
> @@ -532,6 +532,9 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
>   /** @internal Setup priority flow control parameter on an Ethernet device. */
>   typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
>   				struct rte_eth_pfc_conf *pfc_conf);
> +/** @internal Queue setup for priority flow control parameter on an Ethernet device. */
> +typedef int (*priority_flow_ctrl_queue_set_t)(struct rte_eth_dev *dev,
> +				struct rte_eth_pfc_queue_conf *pfc_queue_conf);
>   
>   /** @internal Update RSS redirection table on an Ethernet device. */
>   typedef int (*reta_update_t)(struct rte_eth_dev *dev,
> @@ -1080,7 +1083,8 @@ struct eth_dev_ops {
>   	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
>   	/** Setup priority flow control */
>   	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
> -
> +	/** Priority flow control queue setup */
> +	priority_flow_ctrl_queue_set_t   priority_flow_ctrl_queue_set;
>   	/** Set Unicast Table Array */
>   	eth_uc_hash_table_set_t    uc_hash_table_set;
>   	/** Set Unicast hash bitmap */
> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
> index a1d475a292..6def057720 100644
> --- a/lib/ethdev/rte_ethdev.c
> +++ b/lib/ethdev/rte_ethdev.c
> @@ -3998,7 +3998,9 @@ int
>   rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>   				   struct rte_eth_pfc_conf *pfc_conf)
>   {
> +	struct rte_eth_dev_info dev_info;
>   	struct rte_eth_dev *dev;
> +	int ret;
>   
>   	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
>   	dev = &rte_eth_devices[port_id];
> @@ -4010,6 +4012,17 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>   		return -EINVAL;
>   	}
>   
> +	ret = rte_eth_dev_info_get(port_id, &dev_info);
> +	if (ret != 0)
> +		return ret;
> +
> +	if (dev_info.pfc_queue_tc_max != 0) {
> +		RTE_ETHDEV_LOG(ERR,
> +			"Ethdev port %u driver does not support port level PFC config\n",
> +			port_id);
> +		return -ENOTSUP;
> +	}
> +
>   	if (pfc_conf->priority > (RTE_ETH_DCB_NUM_USER_PRIORITIES - 1)) {
>   		RTE_ETHDEV_LOG(ERR, "Invalid priority, only 0-7 allowed\n");
>   		return -EINVAL;
> @@ -4022,6 +4035,102 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>   	return -ENOTSUP;
>   }
>   
> +static inline int
> +validate_rx_pause_config(struct rte_eth_dev_info *dev_info,
> +			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> +{
> +	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
> +	    (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
> +		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
> +			RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx pause"
> +				       " (requested: %d configured: %d)\n",
> +				       pfc_queue_conf->rx_pause.tx_qid,
> +				       dev_info->nb_tx_queues);
> +			return -EINVAL;
> +		}
> +
> +		if (pfc_queue_conf->rx_pause.tc >= dev_info->pfc_queue_tc_max) {
> +			RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
> +				       " (requested: %d max: %d)\n",
> +				       pfc_queue_conf->rx_pause.tc,
> +				       dev_info->pfc_queue_tc_max);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static inline int
> +validate_tx_pause_config(struct rte_eth_dev_info *dev_info,
> +			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> +{
> +	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
> +	     (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
> +		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
> +			RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx pause"
> +				       "(requested: %d configured: %d)\n",
> +				       pfc_queue_conf->tx_pause.rx_qid,
> +				       dev_info->nb_rx_queues);
> +			return -EINVAL;
> +		}
> +
> +		if (pfc_queue_conf->tx_pause.tc >= dev_info->pfc_queue_tc_max) {
> +			RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
> +				       "(requested: %d max: %d)\n",
> +				       pfc_queue_conf->tx_pause.tc,
> +				       dev_info->pfc_queue_tc_max);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +rte_eth_dev_priority_flow_ctrl_queue_set(
> +	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> +{
> +	struct rte_eth_dev_info dev_info;
> +	struct rte_eth_dev *dev;
> +	int ret;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> +	dev = &rte_eth_devices[port_id];
> +
> +	if (pfc_queue_conf == NULL) {
> +		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
> +			       port_id);
> +		return -EINVAL;
> +	}
> +
> +	ret = rte_eth_dev_info_get(port_id, &dev_info);
> +	if (ret != 0)
> +		return ret;
> +
> +	if (dev_info.pfc_queue_tc_max == 0) {
> +		RTE_ETHDEV_LOG(ERR,
> +			"Ethdev port %u does not support PFC TC values\n",
> +			port_id);
> +		return -ENOTSUP;
> +	}
> +
> +	ret = validate_rx_pause_config(&dev_info, pfc_queue_conf);
> +	if (ret != 0)
> +		return ret;
> +
> +	ret = validate_tx_pause_config(&dev_info, pfc_queue_conf);
> +	if (ret != 0)
> +		return ret;
> +
> +
> +	if (*dev->dev_ops->priority_flow_ctrl_queue_set)
> +		return eth_err(port_id,
> +			       (*dev->dev_ops->priority_flow_ctrl_queue_set)(
> +				       dev, pfc_queue_conf));
> +	return -ENOTSUP;
> +}
> +
>   static int
>   eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
>   			uint16_t reta_size)
> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> index fa299c8ad7..33e43f6e1d 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -1395,6 +1395,28 @@ struct rte_eth_pfc_conf {
>   	uint8_t priority;          /**< VLAN User Priority. */
>   };
>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + *
> + * A structure used to configure Ethernet priority flow control parameter for
> + * ethdev queues.
> + */
> +struct rte_eth_pfc_queue_conf {
> +	enum rte_eth_fc_mode mode;  /**< Link flow control mode */
> +
> +	struct {
> +		uint16_t tx_qid; /**< Tx queue ID */
> +		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
> +	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
> +
> +	struct {
> +		uint16_t pause_time; /**< Pause quota in the Pause frame */
> +		uint16_t rx_qid; /**< Rx queue ID */
> +		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
> +	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
> +};
> +
>   /**
>    * Tunnel type for device-specific classifier configuration.
>    * @see rte_eth_udp_tunnel
> @@ -1841,8 +1863,30 @@ struct rte_eth_dev_info {
>   	 * embedded managed interconnect/switch.
>   	 */
>   	struct rte_eth_switch_info switch_info;
> -
> -	uint64_t reserved_64s[2]; /**< Reserved for future fields */
> +	/**
> +	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
> +	 *
> +	 * Based on device support and use-case need, there are two different
> +	 * ways to enable PFC. The first case is the port level PFC
> +	 * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
> +	 * API shall be used to configure the PFC, and PFC frames will be
> +	 * generated using based on VLAN TC value.
> +	 * The second case is the queue level PFC configuration, in this case,
> +	 * Any packet field content can be used to steer the packet to the
> +	 * specific queue using rte_flow or RSS and then use
> +	 * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
> +	 * on each queue. Based on congestion selected on the specific queue,
> +	 * configured TC shall be used to generate PFC frames.
> +	 *
> +	 * When set to non zero value, application must use queue level
> +	 * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
> +	 * instead of port level PFC configuration via
> +	 * rte_eth_dev_priority_flow_ctrl_set() API to realize
> +	 * PFC configuration.
> +	 */
> +	uint8_t pfc_queue_tc_max;


'rte_eth_dev_info_get()' is one of the APIs that anyone using ethdev needs to use.

Instead of expanding it with less used features, what do you think to have a
specific API to get the 'pfc_queue_tc_max'?

It also can be used by application to detect queue based PFC is supported by
driver or not.
Assume API is 'rte_eth_dev_priority_flow_ctrl_get()', if it returns '-ENOTSUP'
application can know that PMD doesn't support queue based PFC.


> +	uint8_t reserved_8s[7];
> +	uint64_t reserved_64s[1]; /**< Reserved for future fields */
>   	void *reserved_ptrs[2];   /**< Reserved for future fields */
>   };
>   
> @@ -4109,6 +4153,9 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
>    * Configure the Ethernet priority flow control under DCB environment
>    * for Ethernet device.
>    *
> + * @see struct rte_eth_dev_info::pfc_queue_tc_max priority
> + * flow control usage models.
> + *
>    * @param port_id
>    * The port identifier of the Ethernet device.
>    * @param pfc_conf
> @@ -4119,10 +4166,40 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
>    *   - (-ENODEV)  if *port_id* invalid.
>    *   - (-EINVAL)  if bad parameter
>    *   - (-EIO)     if flow control setup failure or device is removed.
> + *
>    */
>   int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
> -				struct rte_eth_pfc_conf *pfc_conf);
> +				       struct rte_eth_pfc_conf *pfc_conf);

Above syntax changes are not needed.

DPDK coding convention is using tabs (mostly two) for multi line function
decleration/definition, please be consistant with usage, this patch has
multiple variations.

>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Configure the Ethernet priority flow control for a given queue
> + * for Ethernet device.
> + *
> + * @see struct rte_eth_dev_info::pfc_queue_tc_max priority flow control
> + * usage models.
> + *
> + * @note When an ethdev port switches to PFC mode, the unconfigured

Doesit mean queue based PFC mode?

> + * queues shall be configured by the driver with default values such as
> + * lower priority value for TC etc.
> + *

I assume there is no way for application to know what the defaults values are,
also not sure if application interested in this.

> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param pfc_queue_conf
> + *   The pointer to the structure of the priority flow control parameters
> + *   for the queue.
> + * @return
> + *   - (0) if successful.
> + *   - (-ENOTSUP) if hardware doesn't support priority flow control mode.
> + *   - (-ENODEV)  if *port_id* invalid.
> + *   - (-EINVAL)  if bad parameter
> + *   - (-EIO)     if flow control setup queue failure
> + */
> +__rte_experimental
> +int rte_eth_dev_priority_flow_ctrl_queue_set(uint16_t port_id,
> +					     struct rte_eth_pfc_queue_conf *pfc_queue_conf);

I wonder if Rx/Tx queue id should be API arguments, to be consistent
with some other APIs, and will it help application that configures queues
in a loop.
But I can see 'rx_pause' or 'tx_pause' (in config) can be valid or not
based on the 'mode', so I understand to have queue ids in the struct.
No strong opinion.

Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based pfc CLI options

[Ferruh Yigit]

On 1/13/2022 10:27 AM, jerinj@marvell.com wrote:
> From: Sunil Kumar Kori <skori@marvell.com>
> 
> Patch adds command line options to configure queue based
> priority flow control.
> 
> - Syntax command is given as below:
> 
> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
> 	tx <on|off> <rx_qid> <rx_tc> <pause_time>
> 

Isn't the order of the paramters odd, it is mixing Rx/Tx config,
what about ordering Rx and Tx paramters?

> - Example command to configure queue based priority flow control
>    on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
>    time 2047
> 
> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
> 
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>

<...>

Re: [dpdk-dev] [PATCH v2 1/2] ethdev: support queue-based priority flow control

[Jerin Jacob]

On Tue, Jan 25, 2022 at 11:05 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 1/13/2022 10:27 AM, jerinj@marvell.com wrote:
> > From: Jerin Jacob <jerinj@marvell.com>
> >
> > Based on device support and use-case need, there are two different ways
> > to enable PFC. The first case is the port level PFC configuration, in
> > this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
> > configure the PFC, and PFC frames will be generated using based on VLAN
> > TC value.
> >
> > The second case is the queue level PFC configuration, in this
> > case, Any packet field content can be used to steer the packet to the
> > specific queue using rte_flow or RSS and then use
> > rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping on each
> > queue. Based on congestion selected on the specific queue, configured TC
> > shall be used to generate PFC frames.
> > > Operation of these modes are mutually exclusive, when driver sets
> > non zero value for rte_eth_dev_info::pfc_queue_tc_max,
> > application must use queue level PFC configuration via
> > rte_eth_dev_priority_flow_ctrl_queue_set() API instead of port level
> > PFC configuration via rte_eth_dev_priority_flow_ctrl_set() API to
> > realize PFC configuration.
> >
> > This patch enables the configuration for second case a.k.a queue
> > based PFC also updates rte_eth_dev_priority_flow_ctrl_set()
> > implmentaion to adheher to rte_eth_dev_info::pfc_queue_tc_max
> > handling.
> >
> > Also updated libabigail.abignore to ignore the update
> > to reserved fields in rte_eth_dev_info.
> >
> > Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> > Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> > ---
> >
> > A driver implemtion based on this API is at
> > https://patches.dpdk.org/project/dpdk/patch/20220111081831.881374-1-skori@marvell.com/
> >
> > RFC..v1
> >   - Added queue based PFC config API instead port based
> >
> > v1..v2

> > +
> >   /**
> >    * Tunnel type for device-specific classifier configuration.
> >    * @see rte_eth_udp_tunnel
> > @@ -1841,8 +1863,30 @@ struct rte_eth_dev_info {
> >        * embedded managed interconnect/switch.
> >        */
> >       struct rte_eth_switch_info switch_info;
> > -
> > -     uint64_t reserved_64s[2]; /**< Reserved for future fields */
> > +     /**
> > +      * Maximum supported traffic class as per PFC (802.1Qbb) specification.
> > +      *
> > +      * Based on device support and use-case need, there are two different
> > +      * ways to enable PFC. The first case is the port level PFC
> > +      * configuration, in this case, rte_eth_dev_priority_flow_ctrl_set()
> > +      * API shall be used to configure the PFC, and PFC frames will be
> > +      * generated using based on VLAN TC value.
> > +      * The second case is the queue level PFC configuration, in this case,
> > +      * Any packet field content can be used to steer the packet to the
> > +      * specific queue using rte_flow or RSS and then use
> > +      * rte_eth_dev_priority_flow_ctrl_queue_set() to set the TC mapping
> > +      * on each queue. Based on congestion selected on the specific queue,
> > +      * configured TC shall be used to generate PFC frames.
> > +      *
> > +      * When set to non zero value, application must use queue level
> > +      * PFC configuration via rte_eth_dev_priority_flow_ctrl_queue_set() API
> > +      * instead of port level PFC configuration via
> > +      * rte_eth_dev_priority_flow_ctrl_set() API to realize
> > +      * PFC configuration.
> > +      */
> > +     uint8_t pfc_queue_tc_max;
>
>
> 'rte_eth_dev_info_get()' is one of the APIs that anyone using ethdev needs to use.
>
> Instead of expanding it with less used features, what do you think to have a
> specific API to get the 'pfc_queue_tc_max'?


OK. rte_eth_dev_info_get() was general theme used in ethdev API. Fine
for new API
specifically for this.

>
> It also can be used by application to detect queue based PFC is supported by
> driver or not.

OK.

> Assume API is 'rte_eth_dev_priority_flow_ctrl_get()', if it returns '-ENOTSUP'
> application can know that PMD doesn't support queue based PFC.

OK.

>
>
> > +     uint8_t reserved_8s[7];
> > +     uint64_t reserved_64s[1]; /**< Reserved for future fields */
> >       void *reserved_ptrs[2];   /**< Reserved for future fields */
> >   };
> >
> > @@ -4109,6 +4153,9 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
> >    * Configure the Ethernet priority flow control under DCB environment
> >    * for Ethernet device.
> >    *
> > + * @see struct rte_eth_dev_info::pfc_queue_tc_max priority
> > + * flow control usage models.
> > + *
> >    * @param port_id
> >    * The port identifier of the Ethernet device.
> >    * @param pfc_conf
> > @@ -4119,10 +4166,40 @@ int rte_eth_dev_flow_ctrl_set(uint16_t port_id,
> >    *   - (-ENODEV)  if *port_id* invalid.
> >    *   - (-EINVAL)  if bad parameter
> >    *   - (-EIO)     if flow control setup failure or device is removed.
> > + *
> >    */
> >   int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
> > -                             struct rte_eth_pfc_conf *pfc_conf);
> > +                                    struct rte_eth_pfc_conf *pfc_conf);
>
> Above syntax changes are not needed.
>
> DPDK coding convention is using tabs (mostly two) for multi line function
> decleration/definition, please be consistant with usage, this patch has
> multiple variations.

Ack.

>
> >
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Configure the Ethernet priority flow control for a given queue
> > + * for Ethernet device.
> > + *
> > + * @see struct rte_eth_dev_info::pfc_queue_tc_max priority flow control
> > + * usage models.
> > + *
> > + * @note When an ethdev port switches to PFC mode, the unconfigured
>
> Doesit mean queue based PFC mode?

Yes.  I will change to PFC mode -> queue-based PFC mode.

>
> > + * queues shall be configured by the driver with default values such as
> > + * lower priority value for TC etc.
> > + *
>
> I assume there is no way for application to know what the defaults values are,
> also not sure if application interested in this.

Yes. I don't think the application cares. But added in the doc to
clarify the state.

>
> > + * @param port_id
> > + *   The port identifier of the Ethernet device.
> > + * @param pfc_queue_conf
> > + *   The pointer to the structure of the priority flow control parameters
> > + *   for the queue.
> > + * @return
> > + *   - (0) if successful.
> > + *   - (-ENOTSUP) if hardware doesn't support priority flow control mode.
> > + *   - (-ENODEV)  if *port_id* invalid.
> > + *   - (-EINVAL)  if bad parameter
> > + *   - (-EIO)     if flow control setup queue failure
> > + */
> > +__rte_experimental
> > +int rte_eth_dev_priority_flow_ctrl_queue_set(uint16_t port_id,
> > +                                          struct rte_eth_pfc_queue_conf *pfc_queue_conf);
>
> I wonder if Rx/Tx queue id should be API arguments, to be consistent
> with some other APIs, and will it help application that configures queues
> in a loop.
> But I can see 'rx_pause' or 'tx_pause' (in config) can be valid or not
> based on the 'mode', so I understand to have queue ids in the struct.
> No strong opinion.

Yes. I keep it as is.

RE: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based pfc CLI options

[Sunil Kumar Kori]

>-----Original Message-----
>From: Ferruh Yigit <ferruh.yigit@intel.com>
>Sent: Tuesday, January 25, 2022 11:07 PM
>To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; dev@dpdk.org; Xiaoyun
>Li <xiaoyun.li@intel.com>; Aman Singh <aman.deep.singh@intel.com>; Yuying
>Zhang <yuying.zhang@intel.com>
>Cc: thomas@monjalon.net; ajit.khaparde@broadcom.com;
>aboyer@pensando.io; andrew.rybchenko@oktetlabs.ru;
>beilei.xing@intel.com; bruce.richardson@intel.com; chas3@att.com;
>chenbo.xia@intel.com; ciara.loftus@intel.com; Devendra Singh Rawat
><dsinghrawat@marvell.com>; ed.czeck@atomicrules.com;
>evgenys@amazon.com; grive@u256.net; g.singh@nxp.com;
>zhouguoyang@huawei.com; haiyue.wang@intel.com; Harman Kalra
><hkalra@marvell.com>; heinrich.kuhn@corigine.com;
>hemant.agrawal@nxp.com; hyonkim@cisco.com; igorch@amazon.com; Igor
>Russkikh <irusskikh@marvell.com>; jgrajcia@cisco.com;
>jasvinder.singh@intel.com; jianwang@trustnetic.com;
>jiawenwu@trustnetic.com; jingjing.wu@intel.com; johndale@cisco.com;
>john.miller@atomicrules.com; linville@tuxdriver.com; keith.wiles@intel.com;
>Kiran Kumar Kokkilagadda <kirankumark@marvell.com>;
>oulijun@huawei.com; Liron Himi <lironh@marvell.com>;
>longli@microsoft.com; mw@semihalf.com; spinler@cesnet.cz;
>matan@nvidia.com; matt.peters@windriver.com;
>maxime.coquelin@redhat.com; mk@semihalf.com; humin29@huawei.com;
>Pradeep Kumar Nalla <pnalla@marvell.com>; Nithin Kumar Dabilpuram
><ndabilpuram@marvell.com>; qiming.yang@intel.com; qi.z.zhang@intel.com;
>Radha Chintakuntla <radhac@marvell.com>; rahul.lakkireddy@chelsio.com;
>Rasesh Mody <rmody@marvell.com>; rosen.xu@intel.com;
>sachin.saxena@oss.nxp.com; Satha Koteswara Rao Kottidi
><skoteshwar@marvell.com>; Shahed Shaikh <shshaikh@marvell.com>;
>shaibran@amazon.com; shepard.siegel@atomicrules.com;
>asomalap@amd.com; somnath.kotur@broadcom.com;
>sthemmin@microsoft.com; steven.webster@windriver.com; Sunil Kumar Kori
><skori@marvell.com>; mtetsuyah@gmail.com; Veerasenareddy Burru
><vburru@marvell.com>; viacheslavo@nvidia.com; xiao.w.wang@intel.com;
>cloud.wangxiaoyun@huawei.com; yisen.zhuang@huawei.com;
>yongwang@vmware.com; xuanziyang2@huawei.com
>Subject: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based
>pfc CLI options
>
>External Email
>
>----------------------------------------------------------------------
>On 1/13/2022 10:27 AM, jerinj@marvell.com wrote:
>> From: Sunil Kumar Kori <skori@marvell.com>
>>
>> Patch adds command line options to configure queue based priority flow
>> control.
>>
>> - Syntax command is given as below:
>>
>> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
>> 	tx <on|off> <rx_qid> <rx_tc> <pause_time>
>>
>
>Isn't the order of the paramters odd, it is mixing Rx/Tx config, what about
>ordering Rx and Tx paramters?
>
It's been kept like this to portray config for rx_pause and tx_pause separately i.e. mode and corresponding config.

>> - Example command to configure queue based priority flow control
>>    on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
>>    time 2047
>>
>> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
>>
>> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
>
><...>

Re: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based pfc CLI options

[Ferruh Yigit]

On 1/27/2022 7:13 AM, Sunil Kumar Kori wrote:
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> Sent: Tuesday, January 25, 2022 11:07 PM
>> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; dev@dpdk.org; Xiaoyun
>> Li <xiaoyun.li@intel.com>; Aman Singh <aman.deep.singh@intel.com>; Yuying
>> Zhang <yuying.zhang@intel.com>
>> Cc: thomas@monjalon.net; ajit.khaparde@broadcom.com;
>> aboyer@pensando.io; andrew.rybchenko@oktetlabs.ru;
>> beilei.xing@intel.com; bruce.richardson@intel.com; chas3@att.com;
>> chenbo.xia@intel.com; ciara.loftus@intel.com; Devendra Singh Rawat
>> <dsinghrawat@marvell.com>; ed.czeck@atomicrules.com;
>> evgenys@amazon.com; grive@u256.net; g.singh@nxp.com;
>> zhouguoyang@huawei.com; haiyue.wang@intel.com; Harman Kalra
>> <hkalra@marvell.com>; heinrich.kuhn@corigine.com;
>> hemant.agrawal@nxp.com; hyonkim@cisco.com; igorch@amazon.com; Igor
>> Russkikh <irusskikh@marvell.com>; jgrajcia@cisco.com;
>> jasvinder.singh@intel.com; jianwang@trustnetic.com;
>> jiawenwu@trustnetic.com; jingjing.wu@intel.com; johndale@cisco.com;
>> john.miller@atomicrules.com; linville@tuxdriver.com; keith.wiles@intel.com;
>> Kiran Kumar Kokkilagadda <kirankumark@marvell.com>;
>> oulijun@huawei.com; Liron Himi <lironh@marvell.com>;
>> longli@microsoft.com; mw@semihalf.com; spinler@cesnet.cz;
>> matan@nvidia.com; matt.peters@windriver.com;
>> maxime.coquelin@redhat.com; mk@semihalf.com; humin29@huawei.com;
>> Pradeep Kumar Nalla <pnalla@marvell.com>; Nithin Kumar Dabilpuram
>> <ndabilpuram@marvell.com>; qiming.yang@intel.com; qi.z.zhang@intel.com;
>> Radha Chintakuntla <radhac@marvell.com>; rahul.lakkireddy@chelsio.com;
>> Rasesh Mody <rmody@marvell.com>; rosen.xu@intel.com;
>> sachin.saxena@oss.nxp.com; Satha Koteswara Rao Kottidi
>> <skoteshwar@marvell.com>; Shahed Shaikh <shshaikh@marvell.com>;
>> shaibran@amazon.com; shepard.siegel@atomicrules.com;
>> asomalap@amd.com; somnath.kotur@broadcom.com;
>> sthemmin@microsoft.com; steven.webster@windriver.com; Sunil Kumar Kori
>> <skori@marvell.com>; mtetsuyah@gmail.com; Veerasenareddy Burru
>> <vburru@marvell.com>; viacheslavo@nvidia.com; xiao.w.wang@intel.com;
>> cloud.wangxiaoyun@huawei.com; yisen.zhuang@huawei.com;
>> yongwang@vmware.com; xuanziyang2@huawei.com
>> Subject: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based
>> pfc CLI options
>>
>> External Email
>>
>> ----------------------------------------------------------------------
>> On 1/13/2022 10:27 AM, jerinj@marvell.com wrote:
>>> From: Sunil Kumar Kori <skori@marvell.com>
>>>
>>> Patch adds command line options to configure queue based priority flow
>>> control.
>>>
>>> - Syntax command is given as below:
>>>
>>> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
>>> 	tx <on|off> <rx_qid> <rx_tc> <pause_time>
>>>
>>
>> Isn't the order of the paramters odd, it is mixing Rx/Tx config, what about
>> ordering Rx and Tx paramters?
>>
> It's been kept like this to portray config for rx_pause and tx_pause separately i.e. mode and corresponding config.
> 

What do you mean 'separately'? You need to provide all arguments anyway, right?

I was thinking first have the Rx arguments, later Tx, like:

rx <on|off> <rx_qid> <rx_tc> tx <on|off> <tx_qid> <tx_tc> <pause_time>

Am I missing something, is there a benefit of what you did in this patch?

>>> - Example command to configure queue based priority flow control
>>>     on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
>>>     time 2047
>>>
>>> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
>>>
>>> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
>>
>> <...>

Re: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based pfc CLI options

[Ajit Khaparde]

On Thu, Jan 27, 2022 at 2:40 AM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 1/27/2022 7:13 AM, Sunil Kumar Kori wrote:
> >
> >> -----Original Message-----
> >> From: Ferruh Yigit <ferruh.yigit@intel.com>
> >> Sent: Tuesday, January 25, 2022 11:07 PM
> >> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; dev@dpdk.org; Xiaoyun
> >> Li <xiaoyun.li@intel.com>; Aman Singh <aman.deep.singh@intel.com>; Yuying
> >> Zhang <yuying.zhang@intel.com>
> >> Cc: thomas@monjalon.net; ajit.khaparde@broadcom.com;
> >> aboyer@pensando.io; andrew.rybchenko@oktetlabs.ru;
> >> beilei.xing@intel.com; bruce.richardson@intel.com; chas3@att.com;
> >> chenbo.xia@intel.com; ciara.loftus@intel.com; Devendra Singh Rawat
> >> <dsinghrawat@marvell.com>; ed.czeck@atomicrules.com;
> >> evgenys@amazon.com; grive@u256.net; g.singh@nxp.com;
> >> zhouguoyang@huawei.com; haiyue.wang@intel.com; Harman Kalra
> >> <hkalra@marvell.com>; heinrich.kuhn@corigine.com;
> >> hemant.agrawal@nxp.com; hyonkim@cisco.com; igorch@amazon.com; Igor
> >> Russkikh <irusskikh@marvell.com>; jgrajcia@cisco.com;
> >> jasvinder.singh@intel.com; jianwang@trustnetic.com;
> >> jiawenwu@trustnetic.com; jingjing.wu@intel.com; johndale@cisco.com;
> >> john.miller@atomicrules.com; linville@tuxdriver.com; keith.wiles@intel.com;
> >> Kiran Kumar Kokkilagadda <kirankumark@marvell.com>;
> >> oulijun@huawei.com; Liron Himi <lironh@marvell.com>;
> >> longli@microsoft.com; mw@semihalf.com; spinler@cesnet.cz;
> >> matan@nvidia.com; matt.peters@windriver.com;
> >> maxime.coquelin@redhat.com; mk@semihalf.com; humin29@huawei.com;
> >> Pradeep Kumar Nalla <pnalla@marvell.com>; Nithin Kumar Dabilpuram
> >> <ndabilpuram@marvell.com>; qiming.yang@intel.com; qi.z.zhang@intel.com;
> >> Radha Chintakuntla <radhac@marvell.com>; rahul.lakkireddy@chelsio.com;
> >> Rasesh Mody <rmody@marvell.com>; rosen.xu@intel.com;
> >> sachin.saxena@oss.nxp.com; Satha Koteswara Rao Kottidi
> >> <skoteshwar@marvell.com>; Shahed Shaikh <shshaikh@marvell.com>;
> >> shaibran@amazon.com; shepard.siegel@atomicrules.com;
> >> asomalap@amd.com; somnath.kotur@broadcom.com;
> >> sthemmin@microsoft.com; steven.webster@windriver.com; Sunil Kumar Kori
> >> <skori@marvell.com>; mtetsuyah@gmail.com; Veerasenareddy Burru
> >> <vburru@marvell.com>; viacheslavo@nvidia.com; xiao.w.wang@intel.com;
> >> cloud.wangxiaoyun@huawei.com; yisen.zhuang@huawei.com;
> >> yongwang@vmware.com; xuanziyang2@huawei.com
> >> Subject: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based
> >> pfc CLI options
> >>
> >> External Email
> >>
> >> ----------------------------------------------------------------------
> >> On 1/13/2022 10:27 AM, jerinj@marvell.com wrote:
> >>> From: Sunil Kumar Kori <skori@marvell.com>
> >>>
> >>> Patch adds command line options to configure queue based priority flow
> >>> control.
> >>>
> >>> - Syntax command is given as below:
> >>>
> >>> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
> >>>     tx <on|off> <rx_qid> <rx_tc> <pause_time>
> >>>
> >>
> >> Isn't the order of the paramters odd, it is mixing Rx/Tx config, what about
> >> ordering Rx and Tx paramters?
> >>
> > It's been kept like this to portray config for rx_pause and tx_pause separately i.e. mode and corresponding config.
> >
>
> What do you mean 'separately'? You need to provide all arguments anyway, right?
>
> I was thinking first have the Rx arguments, later Tx, like:
>
> rx <on|off> <rx_qid> <rx_tc> tx <on|off> <tx_qid> <tx_tc> <pause_time>
I think this grouping is better.

>
> Am I missing something, is there a benefit of what you did in this patch?

>
> >>> - Example command to configure queue based priority flow control
> >>>     on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
> >>>     time 2047
> >>>
> >>> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
> >>>
> >>> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> >>
> >> <...>
>

RE: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based pfc CLI options

[Sunil Kumar Kori]

-----Original Message-----
From: Ajit Khaparde <ajit.khaparde@broadcom.com> 
Sent: Thursday, January 27, 2022 10:27 PM
To: Ferruh Yigit <ferruh.yigit@intel.com>
Cc: Sunil Kumar Kori <skori@marvell.com>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; dev@dpdk.org; Xiaoyun Li <xiaoyun.li@intel.com>; Aman Singh <aman.deep.singh@intel.com>; Yuying Zhang <yuying.zhang@intel.com>; thomas@monjalon.net; aboyer@pensando.io; andrew.rybchenko@oktetlabs.ru; beilei.xing@intel.com; bruce.richardson@intel.com; chas3@att.com; chenbo.xia@intel.com; ciara.loftus@intel.com; Devendra Singh Rawat <dsinghrawat@marvell.com>; ed.czeck@atomicrules.com; evgenys@amazon.com; grive@u256.net; g.singh@nxp.com; zhouguoyang@huawei.com; haiyue.wang@intel.com; Harman Kalra <hkalra@marvell.com>; heinrich.kuhn@corigine.com; hemant.agrawal@nxp.com; hyonkim@cisco.com; igorch@amazon.com; Igor Russkikh <irusskikh@marvell.com>; jgrajcia@cisco.com; jasvinder.singh@intel.com; jianwang@trustnetic.com; jiawenwu@trustnetic.com; jingjing.wu@intel.com; johndale@cisco.com; john.miller@atomicrules.com; linville@tuxdriver.com; keith.wiles@intel.com; Kiran Kumar Kokkilagadda <kirankumark@marvell.com>; oulijun@huawei.com; Liron Himi <lironh@marvell.com>; longli@microsoft.com; mw@semihalf.com; spinler@cesnet.cz; matan@nvidia.com; matt.peters@windriver.com; maxime.coquelin@redhat.com; mk@semihalf.com; humin29@huawei.com; Pradeep Kumar Nalla <pnalla@marvell.com>; Nithin Kumar Dabilpuram <ndabilpuram@marvell.com>; qiming.yang@intel.com; qi.z.zhang@intel.com; Radha Chintakuntla <radhac@marvell.com>; rahul.lakkireddy@chelsio.com; Rasesh Mody <rmody@marvell.com>; rosen.xu@intel.com; sachin.saxena@oss.nxp.com; Satha Koteswara Rao Kottidi <skoteshwar@marvell.com>; Shahed Shaikh <shshaikh@marvell.com>; shaibran@amazon.com; shepard.siegel@atomicrules.com; asomalap@amd.com; somnath.kotur@broadcom.com; sthemmin@microsoft.com; steven.webster@windriver.com; mtetsuyah@gmail.com; Veerasenareddy Burru <vburru@marvell.com>; viacheslavo@nvidia.com; xiao.w.wang@intel.com; cloud.wangxiaoyun@huawei.com; yisen.zhuang@huawei.com; yongwang@vmware.com; xuanziyang2@huawei.com
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue based pfc CLI options

On Thu, Jan 27, 2022 at 2:40 AM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 1/27/2022 7:13 AM, Sunil Kumar Kori wrote:
> >
> >> -----Original Message-----
> >> From: Ferruh Yigit <ferruh.yigit@intel.com>
> >> Sent: Tuesday, January 25, 2022 11:07 PM
> >> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; dev@dpdk.org; 
> >> Xiaoyun Li <xiaoyun.li@intel.com>; Aman Singh 
> >> <aman.deep.singh@intel.com>; Yuying Zhang <yuying.zhang@intel.com>
> >> Cc: thomas@monjalon.net; ajit.khaparde@broadcom.com; 
> >> aboyer@pensando.io; andrew.rybchenko@oktetlabs.ru; 
> >> beilei.xing@intel.com; bruce.richardson@intel.com; chas3@att.com; 
> >> chenbo.xia@intel.com; ciara.loftus@intel.com; Devendra Singh Rawat 
> >> <dsinghrawat@marvell.com>; ed.czeck@atomicrules.com; 
> >> evgenys@amazon.com; grive@u256.net; g.singh@nxp.com; 
> >> zhouguoyang@huawei.com; haiyue.wang@intel.com; Harman Kalra 
> >> <hkalra@marvell.com>; heinrich.kuhn@corigine.com; 
> >> hemant.agrawal@nxp.com; hyonkim@cisco.com; igorch@amazon.com; Igor 
> >> Russkikh <irusskikh@marvell.com>; jgrajcia@cisco.com; 
> >> jasvinder.singh@intel.com; jianwang@trustnetic.com; 
> >> jiawenwu@trustnetic.com; jingjing.wu@intel.com; johndale@cisco.com; 
> >> john.miller@atomicrules.com; linville@tuxdriver.com; 
> >> keith.wiles@intel.com; Kiran Kumar Kokkilagadda 
> >> <kirankumark@marvell.com>; oulijun@huawei.com; Liron Himi 
> >> <lironh@marvell.com>; longli@microsoft.com; mw@semihalf.com; 
> >> spinler@cesnet.cz; matan@nvidia.com; matt.peters@windriver.com; 
> >> maxime.coquelin@redhat.com; mk@semihalf.com; humin29@huawei.com; 
> >> Pradeep Kumar Nalla <pnalla@marvell.com>; Nithin Kumar Dabilpuram 
> >> <ndabilpuram@marvell.com>; qiming.yang@intel.com; 
> >> qi.z.zhang@intel.com; Radha Chintakuntla <radhac@marvell.com>; 
> >> rahul.lakkireddy@chelsio.com; Rasesh Mody <rmody@marvell.com>; 
> >> rosen.xu@intel.com; sachin.saxena@oss.nxp.com; Satha Koteswara Rao 
> >> Kottidi <skoteshwar@marvell.com>; Shahed Shaikh 
> >> <shshaikh@marvell.com>; shaibran@amazon.com; 
> >> shepard.siegel@atomicrules.com; asomalap@amd.com; 
> >> somnath.kotur@broadcom.com; sthemmin@microsoft.com; 
> >> steven.webster@windriver.com; Sunil Kumar Kori <skori@marvell.com>; 
> >> mtetsuyah@gmail.com; Veerasenareddy Burru <vburru@marvell.com>; 
> >> viacheslavo@nvidia.com; xiao.w.wang@intel.com; 
> >> cloud.wangxiaoyun@huawei.com; yisen.zhuang@huawei.com; 
> >> yongwang@vmware.com; xuanziyang2@huawei.com
> >> Subject: [EXT] Re: [dpdk-dev] [PATCH v2 2/2] app/testpmd: add queue 
> >> based pfc CLI options
> >>
> >> External Email
> >>
> >> -------------------------------------------------------------------
> >> --- On 1/13/2022 10:27 AM, jerinj@marvell.com wrote:
> >>> From: Sunil Kumar Kori <skori@marvell.com>
> >>>
> >>> Patch adds command line options to configure queue based priority 
> >>> flow control.
> >>>
> >>> - Syntax command is given as below:
> >>>
> >>> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
> >>>     tx <on|off> <rx_qid> <rx_tc> <pause_time>
> >>>
> >>
> >> Isn't the order of the paramters odd, it is mixing Rx/Tx config, 
> >> what about ordering Rx and Tx paramters?
> >>
> > It's been kept like this to portray config for rx_pause and tx_pause separately i.e. mode and corresponding config.
> >
>
> What do you mean 'separately'? You need to provide all arguments anyway, right?
>
> I was thinking first have the Rx arguments, later Tx, like:
>
> rx <on|off> <rx_qid> <rx_tc> tx <on|off> <tx_qid> <tx_tc> <pause_time>
I think this grouping is better.

>
> Am I missing something, is there a benefit of what you did in this patch?

Mentioned syntax takes input as per below config structure:
struct rte_eth_pfc_queue_conf {
        enum rte_eth_fc_mode mode; /**< Link flow control mode */

        struct {
                uint16_t tx_qid; /**< Tx queue ID */
                uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
        } rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */

        struct {
                uint16_t pause_time; /**< Pause quota in the Pause frame */
                uint16_t rx_qid;     /**< Rx queue ID */
                uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
        } tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
};

First part represent first structure information and later part represent second structure information.
For the next version, I am keeping this syntax as it is and will be fixing some ethdev changes.

>
> >>> - Example command to configure queue based priority flow control
> >>>     on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
> >>>     time 2047
> >>>
> >>> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
> >>>
> >>> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> >>
> >> <...>
>

[dpdk-dev] [PATCH v3 1/2] ethdev: support queue-based priority flow control

[jerinj]

From: Jerin Jacob <jerinj@marvell.com>

Based on device support and use-case need, there are two different ways
to enable PFC. The first case is the port level PFC configuration, in
this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
configure the PFC, and PFC frames will be generated using based on VLAN
TC value.

The second case is the queue level PFC configuration, in this
case, Any packet field content can be used to steer the packet to the
specific queue using rte_flow or RSS and then use
rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
TC mapping on each queue.
Based on congestion selected on the specific queue, configured TC
shall be used to generate PFC frames.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---

v2..v1:
- Introduce rte_eth_dev_priority_flow_ctrl_queue_info_get() to
avoid updates to rte_eth_dev_info
- Removed devtools/libabigail.abignore changes
- Address the comment from Ferruh in
http://patches.dpdk.org/project/dpdk/patch/20220113102718.3167282-1-jerinj@marvell.com/

 doc/guides/nics/features.rst           |   7 +-
 doc/guides/rel_notes/release_22_03.rst |   6 ++
 lib/ethdev/ethdev_driver.h             |  12 ++-
 lib/ethdev/rte_ethdev.c                | 132 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h                |  89 +++++++++++++++++
 lib/ethdev/version.map                 |   4 +
 6 files changed, 247 insertions(+), 3 deletions(-)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 27be2d2576..1cacdc883a 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -379,9 +379,12 @@ Flow control
 Supports configuring link flow control.
 
 * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
-  ``priority_flow_ctrl_set``.
+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_info_get``,
+  ``priority_flow_ctrl_queue_configure``
 * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
-  ``rte_eth_dev_priority_flow_ctrl_set()``.
+  ``rte_eth_dev_priority_flow_ctrl_set()``,
+  ``rte_eth_dev_priority_flow_ctrl_queue_info_get()``,
+  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``.
 
 
 .. _nic_features_rate_limitation:
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 3bc0630c7c..e988c104e8 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -69,6 +69,12 @@ New Features
 
   The new API ``rte_event_eth_rx_adapter_event_port_get()`` was added.
 
+* **Added an API to enable queue based priority flow ctrl(PFC).**
+
+  New APIs, ``rte_eth_dev_priority_flow_ctrl_queue_info_get()`` and
+  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``, was added.
+
+
 
 Removed Items
 -------------
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index d95605a355..320a364766 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -533,6 +533,13 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
 typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
 				struct rte_eth_pfc_conf *pfc_conf);
 
+/** @internal Get info for queue based PFC on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_info_get_t)(
+	struct rte_eth_dev *dev, struct rte_eth_pfc_queue_info *pfc_queue_info);
+/** @internal Configure queue based PFC parameter on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_config_t)(
+	struct rte_eth_dev *dev, struct rte_eth_pfc_queue_conf *pfc_queue_conf);
+
 /** @internal Update RSS redirection table on an Ethernet device. */
 typedef int (*reta_update_t)(struct rte_eth_dev *dev,
 			     struct rte_eth_rss_reta_entry64 *reta_conf,
@@ -1080,7 +1087,10 @@ struct eth_dev_ops {
 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
 	/** Setup priority flow control */
 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
-
+	/** Priority flow control queue info get */
+	priority_flow_ctrl_queue_info_get_t priority_flow_ctrl_queue_info_get;
+	/** Priority flow control queue configure */
+	priority_flow_ctrl_queue_config_t priority_flow_ctrl_queue_config;
 	/** Set Unicast Table Array */
 	eth_uc_hash_table_set_t    uc_hash_table_set;
 	/** Set Unicast hash bitmap */
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index a1d475a292..2ce38cd2c5 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -4022,6 +4022,138 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 	return -ENOTSUP;
 }
 
+static inline int
+validate_rx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
+	    (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx pause"
+				       " (requested: %d configured: %d)\n",
+				       pfc_queue_conf->rx_pause.tx_qid,
+				       dev_info->nb_tx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->rx_pause.tc >= tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
+				       " (requested: %d max: %d)\n",
+				       pfc_queue_conf->rx_pause.tc, tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+validate_tx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
+	     (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
+			RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx pause"
+				       "(requested: %d configured: %d)\n",
+				       pfc_queue_conf->tx_pause.rx_qid,
+				       dev_info->nb_rx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->tx_pause.tc >= tc_max) {
+			RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
+				       "(requested: %d max: %d)\n",
+				       pfc_queue_conf->tx_pause.tc, tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_info_get(
+	uint16_t port_id, struct rte_eth_pfc_queue_info *pfc_queue_info)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_info == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC info param is NULL for port (%u)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_info_get)(
+				       dev, pfc_queue_info));
+	return -ENOTSUP;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_configure(
+	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	struct rte_eth_pfc_queue_info pfc_info;
+	struct rte_eth_dev_info dev_info;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_conf == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_info_get(port_id, &pfc_info);
+	if (ret != 0)
+		return ret;
+
+	if (pfc_info.capa == 0) {
+		RTE_ETHDEV_LOG(ERR, "Ethdev port %u does not support PFC\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
+	if (pfc_info.tc_max == 0) {
+		RTE_ETHDEV_LOG(ERR,
+			"Ethdev port %u does not support PFC TC values\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
+	if (pfc_info.capa & RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE) {
+		ret = validate_rx_pause_config(&dev_info, pfc_info.tc_max,
+					       pfc_queue_conf);
+		if (ret != 0)
+			return ret;
+	}
+
+	if (pfc_info.capa & RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE) {
+		ret = validate_tx_pause_config(&dev_info, pfc_info.tc_max,
+					       pfc_queue_conf);
+		if (ret != 0)
+			return ret;
+	}
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_config)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_config)(
+				       dev, pfc_queue_conf));
+	return -ENOTSUP;
+}
+
 static int
 eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
 			uint16_t reta_size)
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index fa299c8ad7..383ad1cdd7 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1395,6 +1395,48 @@ struct rte_eth_pfc_conf {
 	uint8_t priority;          /**< VLAN User Priority. */
 };
 
+/** Device supports Rx pause for queue based PFC. */
+#define RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE RTE_BIT64(0)
+/** Device supports Tx pause for queue based PFC. */
+#define RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE RTE_BIT64(1)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to retrieve information of queue based PFC.
+ */
+struct rte_eth_pfc_queue_info {
+	/**
+	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
+	 */
+	uint8_t tc_max;
+	/** PFC queue capabilities (RTE_ETH_PFC_QUEUE_CAPA_). */
+	uint64_t capa;
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to configure Ethernet priority flow control parameter for
+ * ethdev queues.
+ */
+struct rte_eth_pfc_queue_conf {
+	enum rte_eth_fc_mode mode; /**< Link flow control mode */
+
+	struct {
+		uint16_t tx_qid; /**< Tx queue ID */
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
+
+	struct {
+		uint16_t pause_time; /**< Pause quota in the Pause frame */
+		uint16_t rx_qid;     /**< Rx queue ID */
+		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
+};
+
 /**
  * Tunnel type for device-specific classifier configuration.
  * @see rte_eth_udp_tunnel
@@ -4144,6 +4186,53 @@ int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
 				uint32_t pool);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Retrieve the information for queue based PFC.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_info
+ *   A pointer to a structure of type *rte_eth_pfc_queue_info* to be filled with
+ *   the information about queue based PFC.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if support for priority_flow_ctrl_queue_info_get does not exist.
+ *   - (-ENODEV) if *port_id* invalid.
+ *   - (-EINVAL) if bad parameter.
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
+				   struct rte_eth_pfc_queue_info *pfc_queue_info);
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Configure the queue based priority flow control for a given queue
+ * for Ethernet device.
+ *
+ * @note When an ethdev port switches to queue based PFC mode, the
+ * unconfigured queues shall be configured by the driver with
+ * default values such as lower priority value for TC etc.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_conf
+ *   The pointer to the structure of the priority flow control parameters
+ *   for the queue.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if hardware doesn't support queue based PFC mode.
+ *   - (-ENODEV)  if *port_id* invalid.
+ *   - (-EINVAL)  if bad parameter
+ *   - (-EIO)     if flow control setup queue failure
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
+			      struct rte_eth_pfc_queue_conf *pfc_queue_conf);
+
 /**
  * Remove a MAC address from the internal array of addresses.
  *
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index c2fb0669a4..49523ebc45 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -256,6 +256,10 @@ EXPERIMENTAL {
 	rte_flow_flex_item_create;
 	rte_flow_flex_item_release;
 	rte_flow_pick_transfer_proxy;
+
+	# added in 22.03
+	rte_eth_dev_priority_flow_ctrl_queue_configure;
+	rte_eth_dev_priority_flow_ctrl_queue_info_get;
 };
 
 INTERNAL {
-- 
2.35.1

[dpdk-dev] [PATCH v3 2/2] app/testpmd: add queue based pfc CLI options

[jerinj]

From: Sunil Kumar Kori <skori@marvell.com>

Patch adds command line options to configure queue based
priority flow control.

- Syntax command is given as below:

set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
	tx <on|off> <rx_qid> <rx_tc> <pause_time>

- Example command to configure queue based priority flow control
  on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
  time 2047

testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047

Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
v2..v1
- Sync up the implementation to use new APIs

 app/test-pmd/cmdline.c                      | 122 ++++++++++++++++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  22 ++++
 2 files changed, 144 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index e626b1c7d9..1af0321af0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -544,6 +544,11 @@ static void cmd_help_long_parsed(void *parsed_result,
 			"    Set the priority flow control parameter on a"
 			" port.\n\n"
 
+			"set pfc_queue_ctrl (port_id) rx (on|off) (tx_qid)"
+			" (tx_tc) tx (on|off) (rx_qid) (rx_tc) (pause_time)\n"
+			"    Set the queue priority flow control parameter on a"
+			" given Rx and Tx queues of a port.\n\n"
+
 			"set stat_qmap (tx|rx) (port_id) (queue_id) (qmapping)\n"
 			"    Set statistics mapping (qmapping 0..15) for RX/TX"
 			" queue on port.\n"
@@ -7690,6 +7695,122 @@ cmdline_parse_inst_t cmd_priority_flow_control_set = {
 	},
 };
 
+struct cmd_queue_priority_flow_ctrl_set_result {
+	cmdline_fixed_string_t set;
+	cmdline_fixed_string_t pfc_queue_ctrl;
+	portid_t port_id;
+	cmdline_fixed_string_t rx;
+	cmdline_fixed_string_t rx_pfc_mode;
+	uint16_t tx_qid;
+	uint8_t  tx_tc;
+	cmdline_fixed_string_t tx;
+	cmdline_fixed_string_t tx_pfc_mode;
+	uint16_t rx_qid;
+	uint8_t  rx_tc;
+	uint16_t pause_time;
+};
+
+static void
+cmd_queue_priority_flow_ctrl_set_parsed(void *parsed_result,
+					__rte_unused struct cmdline *cl,
+					__rte_unused void *data)
+{
+	struct cmd_queue_priority_flow_ctrl_set_result *res = parsed_result;
+	struct rte_eth_pfc_queue_conf pfc_queue_conf;
+	int rx_fc_enable, tx_fc_enable;
+	int ret;
+
+	/*
+	 * Rx on/off, flow control is enabled/disabled on RX side. This can
+	 * indicate the RTE_ETH_FC_TX_PAUSE, Transmit pause frame at the Rx
+	 * side. Tx on/off, flow control is enabled/disabled on TX side. This
+	 * can indicate the RTE_ETH_FC_RX_PAUSE, Respond to the pause frame at
+	 * the Tx side.
+	 */
+	static enum rte_eth_fc_mode rx_tx_onoff_2_mode[2][2] = {
+		{RTE_ETH_FC_NONE, RTE_ETH_FC_TX_PAUSE},
+		{RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_FULL}
+	};
+
+	memset(&pfc_queue_conf, 0, sizeof(struct rte_eth_pfc_queue_conf));
+	rx_fc_enable = (!strncmp(res->rx_pfc_mode, "on", 2)) ? 1 : 0;
+	tx_fc_enable = (!strncmp(res->tx_pfc_mode, "on", 2)) ? 1 : 0;
+	pfc_queue_conf.mode = rx_tx_onoff_2_mode[rx_fc_enable][tx_fc_enable];
+	pfc_queue_conf.rx_pause.tc  = res->tx_tc;
+	pfc_queue_conf.rx_pause.tx_qid = res->tx_qid;
+	pfc_queue_conf.tx_pause.tc  = res->rx_tc;
+	pfc_queue_conf.tx_pause.rx_qid  = res->rx_qid;
+	pfc_queue_conf.tx_pause.pause_time = res->pause_time;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_configure(res->port_id,
+							     &pfc_queue_conf);
+	if (ret != 0) {
+		fprintf(stderr,
+			"bad queue priority flow control parameter, rc = %d\n",
+			ret);
+	}
+}
+
+cmdline_parse_token_string_t cmd_q_pfc_set_set =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				set, "set");
+cmdline_parse_token_string_t cmd_q_pfc_set_flow_ctrl =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pfc_queue_ctrl, "pfc_queue_ctrl");
+cmdline_parse_token_num_t cmd_q_pfc_set_portid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				port_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_q_pfc_set_rx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx, "rx");
+cmdline_parse_token_string_t cmd_q_pfc_set_rx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_tc, RTE_UINT8);
+cmdline_parse_token_string_t cmd_q_pfc_set_tx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx, "tx");
+cmdline_parse_token_string_t cmd_q_pfc_set_tx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_tc, RTE_UINT8);
+cmdline_parse_token_num_t cmd_q_pfc_set_pause_time =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pause_time, RTE_UINT16);
+
+cmdline_parse_inst_t cmd_queue_priority_flow_control_set = {
+	.f = cmd_queue_priority_flow_ctrl_set_parsed,
+	.data = NULL,
+	.help_str = "set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> "
+		"tx <on|off> <rx_qid> <rx_tc> <pause_time>: "
+		"Configure the Ethernet queue priority flow control",
+	.tokens = {
+		(void *)&cmd_q_pfc_set_set,
+		(void *)&cmd_q_pfc_set_flow_ctrl,
+		(void *)&cmd_q_pfc_set_portid,
+		(void *)&cmd_q_pfc_set_rx,
+		(void *)&cmd_q_pfc_set_rx_mode,
+		(void *)&cmd_q_pfc_set_tx_qid,
+		(void *)&cmd_q_pfc_set_tx_tc,
+		(void *)&cmd_q_pfc_set_tx,
+		(void *)&cmd_q_pfc_set_tx_mode,
+		(void *)&cmd_q_pfc_set_rx_qid,
+		(void *)&cmd_q_pfc_set_rx_tc,
+		(void *)&cmd_q_pfc_set_pause_time,
+		NULL,
+	},
+};
+
 /* *** RESET CONFIGURATION *** */
 struct cmd_reset_result {
 	cmdline_fixed_string_t reset;
@@ -17765,6 +17886,7 @@ cmdline_parse_ctx_t main_ctx[] = {
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_set_autoneg,
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_show,
 	(cmdline_parse_inst_t *)&cmd_priority_flow_control_set,
+	(cmdline_parse_inst_t *)&cmd_queue_priority_flow_control_set,
 	(cmdline_parse_inst_t *)&cmd_config_dcb,
 	(cmdline_parse_inst_t *)&cmd_read_reg,
 	(cmdline_parse_inst_t *)&cmd_read_reg_bit_field,
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 94792d88cc..82333d518e 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -1561,6 +1561,28 @@ Where:
 
 * ``priority`` (0-7): VLAN User Priority.
 
+set pfc_queue_ctrl
+~~~~~~~~~~~~~~~~~~
+
+Set the priority flow control parameter on a given Rx and Tx queue of a port::
+
+   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
+            tx (on|off) <rx_qid> <rx_tc> <pause_time>
+
+Where:
+
+* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
+  will be paused when PFC frame is received with ``tx_tc`` enabled.
+
+* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
+
+* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
+  frame will be generated with ``tx_tc`` when exceeds the threshold.
+
+* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
+
+* ``pause_time`` (integer): Pause quota filled in the PFC frame.
+
 set stat_qmap
 ~~~~~~~~~~~~~
 
-- 
2.35.1

Re: [dpdk-dev] [PATCH v3 1/2] ethdev: support queue-based priority flow control

[Ferruh Yigit]

On 1/31/2022 6:08 PM, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
> 
> Based on device support and use-case need, there are two different ways
> to enable PFC. The first case is the port level PFC configuration, in
> this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
> configure the PFC, and PFC frames will be generated using based on VLAN
> TC value.
> 
> The second case is the queue level PFC configuration, in this
> case, Any packet field content can be used to steer the packet to the
> specific queue using rte_flow or RSS and then use
> rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
> TC mapping on each queue.
> Based on congestion selected on the specific queue, configured TC
> shall be used to generate PFC frames.
> 

Hi Jerin, Sunil,

Please find below minor comments, mostly syntax issues.

> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> ---
> 
> v2..v1:
> - Introduce rte_eth_dev_priority_flow_ctrl_queue_info_get() to
> avoid updates to rte_eth_dev_info
> - Removed devtools/libabigail.abignore changes
> - Address the comment from Ferruh in
> http://patches.dpdk.org/project/dpdk/patch/20220113102718.3167282-1-jerinj@marvell.com/
> 
>   doc/guides/nics/features.rst           |   7 +-
>   doc/guides/rel_notes/release_22_03.rst |   6 ++
>   lib/ethdev/ethdev_driver.h             |  12 ++-
>   lib/ethdev/rte_ethdev.c                | 132 +++++++++++++++++++++++++
>   lib/ethdev/rte_ethdev.h                |  89 +++++++++++++++++
>   lib/ethdev/version.map                 |   4 +
>   6 files changed, 247 insertions(+), 3 deletions(-)
> 
> diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> index 27be2d2576..1cacdc883a 100644
> --- a/doc/guides/nics/features.rst
> +++ b/doc/guides/nics/features.rst
> @@ -379,9 +379,12 @@ Flow control
>   Supports configuring link flow control.
>   
>   * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
> -  ``priority_flow_ctrl_set``.
> +  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_info_get``,
> +  ``priority_flow_ctrl_queue_configure``
>   * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
> -  ``rte_eth_dev_priority_flow_ctrl_set()``.
> +  ``rte_eth_dev_priority_flow_ctrl_set()``,
> +  ``rte_eth_dev_priority_flow_ctrl_queue_info_get()``,
> +  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``.
>   
>   
>   .. _nic_features_rate_limitation:
> diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
> index 3bc0630c7c..e988c104e8 100644
> --- a/doc/guides/rel_notes/release_22_03.rst
> +++ b/doc/guides/rel_notes/release_22_03.rst
> @@ -69,6 +69,12 @@ New Features
>   
>     The new API ``rte_event_eth_rx_adapter_event_port_get()`` was added.
>   
> +* **Added an API to enable queue based priority flow ctrl(PFC).**
> +
> +  New APIs, ``rte_eth_dev_priority_flow_ctrl_queue_info_get()`` and
> +  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``, was added.
> +
> +
>  


Can you please move this update before ethdev driver updates.
And no need double empty lines.
  
>   Removed Items
>   -------------
> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
> index d95605a355..320a364766 100644
> --- a/lib/ethdev/ethdev_driver.h
> +++ b/lib/ethdev/ethdev_driver.h
> @@ -533,6 +533,13 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
>   typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
>   				struct rte_eth_pfc_conf *pfc_conf);
>   
> +/** @internal Get info for queue based PFC on an Ethernet device. */
> +typedef int (*priority_flow_ctrl_queue_info_get_t)(
> +	struct rte_eth_dev *dev, struct rte_eth_pfc_queue_info *pfc_queue_info);
> +/** @internal Configure queue based PFC parameter on an Ethernet device. */
> +typedef int (*priority_flow_ctrl_queue_config_t)(
> +	struct rte_eth_dev *dev, struct rte_eth_pfc_queue_conf *pfc_queue_conf);
> +

Instead of ending line with opening parantesis '(', can you break the line after
first argument, like:

typedef int (*priority_flow_ctrl_queue_config_t)(struct rte_eth_dev *dev,
				struct rte_eth_pfc_queue_conf *pfc_queue_conf);

Same for all instances.

>   /** @internal Update RSS redirection table on an Ethernet device. */
>   typedef int (*reta_update_t)(struct rte_eth_dev *dev,
>   			     struct rte_eth_rss_reta_entry64 *reta_conf,
> @@ -1080,7 +1087,10 @@ struct eth_dev_ops {
>   	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
>   	/** Setup priority flow control */
>   	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
> -
> +	/** Priority flow control queue info get */
> +	priority_flow_ctrl_queue_info_get_t priority_flow_ctrl_queue_info_get;
> +	/** Priority flow control queue configure */
> +	priority_flow_ctrl_queue_config_t priority_flow_ctrl_queue_config;

Can you please keep empty line before next (hash) group?

>   	/** Set Unicast Table Array */
>   	eth_uc_hash_table_set_t    uc_hash_table_set;
>   	/** Set Unicast hash bitmap */
> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
> index a1d475a292..2ce38cd2c5 100644
> --- a/lib/ethdev/rte_ethdev.c
> +++ b/lib/ethdev/rte_ethdev.c
> @@ -4022,6 +4022,138 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>   	return -ENOTSUP;
>   }
>   
> +static inline int

Not sure if there is a value to ask function to be 'inline', this is in control
path, only static can be enough.

> +validate_rx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
> +			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> +{
> +	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
> +	    (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {

We don't allign to paranthesis in dpdk coding convenion [1], it should be as:

if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
		(pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
	if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
		...
         }
}



[1]
Altough I am aware many instances sneaked in, still I think better to follow
the convention.

> +		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
> +			RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx pause"
> +				       " (requested: %d configured: %d)\n",
> +				       pfc_queue_conf->rx_pause.tx_qid,
> +				       dev_info->nb_tx_queues);


Should log mention that this is related to the "priority flow Rx queue control"?

> +			return -EINVAL;
> +		}
> +
> +		if (pfc_queue_conf->rx_pause.tc >= tc_max) {

Should we document somewhere that 'tc_max' value itself is an invalid value?

> +			RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
> +				       " (requested: %d max: %d)\n",
> +				       pfc_queue_conf->rx_pause.tc, tc_max);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static inline int
> +validate_tx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
> +			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> +{
> +	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
> +	     (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
> +		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
> +			RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx pause"
> +				       "(requested: %d configured: %d)\n",
> +				       pfc_queue_conf->tx_pause.rx_qid,
> +				       dev_info->nb_rx_queues);
> +			return -EINVAL;
> +		}
> +
> +		if (pfc_queue_conf->tx_pause.tc >= tc_max) {
> +			RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
> +				       "(requested: %d max: %d)\n",
> +				       pfc_queue_conf->tx_pause.tc, tc_max);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +rte_eth_dev_priority_flow_ctrl_queue_info_get(
> +	uint16_t port_id, struct rte_eth_pfc_queue_info *pfc_queue_info)
> +{
> +	struct rte_eth_dev *dev;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> +	dev = &rte_eth_devices[port_id];
> +
> +	if (pfc_queue_info == NULL) {
> +		RTE_ETHDEV_LOG(ERR, "PFC info param is NULL for port (%u)\n",
> +			       port_id);
> +		return -EINVAL;
> +	}
> +
> +	if (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
> +		return eth_err(port_id,
> +			       (*dev->dev_ops->priority_flow_ctrl_queue_info_get)(
> +				       dev, pfc_queue_info));
> +	return -ENOTSUP;
> +}
> +
> +int
> +rte_eth_dev_priority_flow_ctrl_queue_configure(
> +	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> +{
> +	struct rte_eth_pfc_queue_info pfc_info;
> +	struct rte_eth_dev_info dev_info;
> +	struct rte_eth_dev *dev;
> +	int ret;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> +	dev = &rte_eth_devices[port_id];
> +
> +	if (pfc_queue_conf == NULL) {
> +		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
> +			       port_id);
> +		return -EINVAL;
> +	}
> +
> +	ret = rte_eth_dev_info_get(port_id, &dev_info);
> +	if (ret != 0)
> +		return ret;
> +
> +	ret = rte_eth_dev_priority_flow_ctrl_queue_info_get(port_id, &pfc_info);
> +	if (ret != 0)
> +		return ret;
> +
> +	if (pfc_info.capa == 0) {
> +		RTE_ETHDEV_LOG(ERR, "Ethdev port %u does not support PFC\n",
> +			port_id);
> +		return -ENOTSUP;
> +	}
> +
> +	if (pfc_info.tc_max == 0) {
> +		RTE_ETHDEV_LOG(ERR,
> +			"Ethdev port %u does not support PFC TC values\n",
> +			port_id);
> +		return -ENOTSUP;
> +	}
> +
> +	if (pfc_info.capa & RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE) {
> +		ret = validate_rx_pause_config(&dev_info, pfc_info.tc_max,
> +					       pfc_queue_conf);

There is capablilty flags for RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE and RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE
also there is config flags RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_TX_PAUSE and RTE_ETH_FC_FULL


What should happen if driver only support RX_PAUSE but app config request only
TX_PAUSE?
As far as can see with current code it pass the validation, but should it?


> +		if (ret != 0)
> +			return ret;
> +	}
> +
> +	if (pfc_info.capa & RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE) {
> +		ret = validate_tx_pause_config(&dev_info, pfc_info.tc_max,
> +					       pfc_queue_conf);

syntax, please don't align to paranthesis

> +		if (ret != 0)
> +			return ret;
> +	}
> +
> +	if (*dev->dev_ops->priority_flow_ctrl_queue_config)
> +		return eth_err(port_id,
> +			       (*dev->dev_ops->priority_flow_ctrl_queue_config)(
> +				       dev, pfc_queue_conf));
> +	return -ENOTSUP;
> +}
> +
>   static int
>   eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
>   			uint16_t reta_size)
> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> index fa299c8ad7..383ad1cdd7 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -1395,6 +1395,48 @@ struct rte_eth_pfc_conf {
>   	uint8_t priority;          /**< VLAN User Priority. */
>   };
>   
> +/** Device supports Rx pause for queue based PFC. */
> +#define RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE RTE_BIT64(0)
> +/** Device supports Tx pause for queue based PFC. */
> +#define RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE RTE_BIT64(1)
> +

There is already control flow mode enum 'enum rte_eth_fc_mode', those
enum items use FC as abrivation (RTE_ETH_FC_RX_PAUSE), above ones use PFC,
should they use same prefix 'RTE_ETH_FC_' for consistency?

And overall, what for struct and functins too, is the correct abreviation
'pfc' or 'fc', since old code has 'fc' as far as I see?

> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + *
> + * A structure used to retrieve information of queue based PFC.
> + */
> +struct rte_eth_pfc_queue_info {
> +	/**
> +	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.

Will it be redundant to say valid value should be bigger than 0?

> +	 */
> +	uint8_t tc_max;
> +	/** PFC queue capabilities (RTE_ETH_PFC_QUEUE_CAPA_). */

Can move doxygen comments to same line if they fit to 80 char limit:
	uint64_t capa; /**< PFC queue capabilities (RTE_ETH_PFC_QUEUE_CAPA_). */

> +	uint64_t capa;
> +};
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + *
> + * A structure used to configure Ethernet priority flow control parameter for
> + * ethdev queues.
> + */
> +struct rte_eth_pfc_queue_conf {
> +	enum rte_eth_fc_mode mode; /**< Link flow control mode */
> +
> +	struct {
> +		uint16_t tx_qid; /**< Tx queue ID */

'tx_qid' within 'rx_pause' struct, this seems done intentionally but just to double
check, can you please describe here the intendent usage?

> +		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
> +	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
> +
> +	struct {
> +		uint16_t pause_time; /**< Pause quota in the Pause frame */
> +		uint16_t rx_qid;     /**< Rx queue ID */
> +		uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
> +	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
> +};
> +
>   /**
>    * Tunnel type for device-specific classifier configuration.
>    * @see rte_eth_udp_tunnel
> @@ -4144,6 +4186,53 @@ int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>   int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
>   				uint32_t pool);
>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Retrieve the information for queue based PFC.
> + *
> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param pfc_queue_info
> + *   A pointer to a structure of type *rte_eth_pfc_queue_info* to be filled with
> + *   the information about queue based PFC.
> + * @return
> + *   - (0) if successful.
> + *   - (-ENOTSUP) if support for priority_flow_ctrl_queue_info_get does not exist.
> + *   - (-ENODEV) if *port_id* invalid.
> + *   - (-EINVAL) if bad parameter.
> + */
> +__rte_experimental
> +int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
> +				   struct rte_eth_pfc_queue_info *pfc_queue_info);
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Configure the queue based priority flow control for a given queue
> + * for Ethernet device.
> + *
> + * @note When an ethdev port switches to queue based PFC mode, the
> + * unconfigured queues shall be configured by the driver with
> + * default values such as lower priority value for TC etc.
> + *

May be good to document it has dependency to 'rte_eth_dev_info_get()' API?

> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param pfc_queue_conf
> + *   The pointer to the structure of the priority flow control parameters
> + *   for the queue.
> + * @return
> + *   - (0) if successful.
> + *   - (-ENOTSUP) if hardware doesn't support queue based PFC mode.
> + *   - (-ENODEV)  if *port_id* invalid.
> + *   - (-EINVAL)  if bad parameter
> + *   - (-EIO)     if flow control setup queue failure
> + */
> +__rte_experimental
> +int rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
> +			      struct rte_eth_pfc_queue_conf *pfc_queue_conf);
> +
>   /**
>    * Remove a MAC address from the internal array of addresses.
>    *
> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
> index c2fb0669a4..49523ebc45 100644
> --- a/lib/ethdev/version.map
> +++ b/lib/ethdev/version.map
> @@ -256,6 +256,10 @@ EXPERIMENTAL {
>   	rte_flow_flex_item_create;
>   	rte_flow_flex_item_release;
>   	rte_flow_pick_transfer_proxy;
> +
> +	# added in 22.03
> +	rte_eth_dev_priority_flow_ctrl_queue_configure;
> +	rte_eth_dev_priority_flow_ctrl_queue_info_get;
>   };
>   
>   INTERNAL {

Re: [dpdk-dev] [PATCH v3 2/2] app/testpmd: add queue based pfc CLI options

[Ferruh Yigit]

On 1/31/2022 6:08 PM, jerinj@marvell.com wrote:
> From: Sunil Kumar Kori <skori@marvell.com>
> 
> Patch adds command line options to configure queue based
> priority flow control.
> 
> - Syntax command is given as below:
> 
> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
> 	tx <on|off> <rx_qid> <rx_tc> <pause_time>
> 

Ahh I see the order is related to the configure struct, where tx_qid is
part of rx_pause struct.
Can you please explaing this items selection in the struct?
Most probaly it will clarify the above usage.

> - Example command to configure queue based priority flow control
>    on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
>    time 2047
> 
> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
> 
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>

<...>

>   
> +set pfc_queue_ctrl
> +~~~~~~~~~~~~~~~~~~
> +
> +Set the priority flow control parameter on a given Rx and Tx queue of a port::
> +
> +   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
> +            tx (on|off) <rx_qid> <rx_tc> <pause_time>
> +
> +Where:
> +
> +* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
> +  will be paused when PFC frame is received with ``tx_tc`` enabled.
> +
> +* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
> +
> +* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
> +  frame will be generated with ``tx_tc`` when exceeds the threshold.
> +
> +* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
> +
> +* ``pause_time`` (integer): Pause quota filled in the PFC frame.
> +

'pause_time' is related to the TX pause configuration, right? At least in the
config struct it only exists for Tx. If so can you please document this?

RE: [EXT] Re: [dpdk-dev] [PATCH v3 2/2] app/testpmd: add queue based pfc CLI options

[Sunil Kumar Kori]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Thursday, February 3, 2022 9:36 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; dev@dpdk.org; Xiaoyun
> Li <xiaoyun.li@intel.com>; Aman Singh <aman.deep.singh@intel.com>;
> Yuying Zhang <yuying.zhang@intel.com>
> Cc: thomas@monjalon.net; ajit.khaparde@broadcom.com;
> aboyer@pensando.io; andrew.rybchenko@oktetlabs.ru;
> beilei.xing@intel.com; bruce.richardson@intel.com; chas3@att.com;
> chenbo.xia@intel.com; ciara.loftus@intel.com; Devendra Singh Rawat
> <dsinghrawat@marvell.com>; ed.czeck@atomicrules.com;
> evgenys@amazon.com; grive@u256.net; g.singh@nxp.com;
> zhouguoyang@huawei.com; haiyue.wang@intel.com; Harman Kalra
> <hkalra@marvell.com>; heinrich.kuhn@corigine.com;
> hemant.agrawal@nxp.com; hyonkim@cisco.com; igorch@amazon.com; Igor
> Russkikh <irusskikh@marvell.com>; jgrajcia@cisco.com;
> jasvinder.singh@intel.com; jianwang@trustnetic.com;
> jiawenwu@trustnetic.com; jingjing.wu@intel.com; johndale@cisco.com;
> john.miller@atomicrules.com; linville@tuxdriver.com;
> keith.wiles@intel.com; Kiran Kumar Kokkilagadda
> <kirankumark@marvell.com>; oulijun@huawei.com; Liron Himi
> <lironh@marvell.com>; longli@microsoft.com; mw@semihalf.com;
> spinler@cesnet.cz; matan@nvidia.com; matt.peters@windriver.com;
> maxime.coquelin@redhat.com; mk@semihalf.com; humin29@huawei.com;
> Pradeep Kumar Nalla <pnalla@marvell.com>; Nithin Kumar Dabilpuram
> <ndabilpuram@marvell.com>; qiming.yang@intel.com;
> qi.z.zhang@intel.com; Radha Chintakuntla <radhac@marvell.com>;
> rahul.lakkireddy@chelsio.com; Rasesh Mody <rmody@marvell.com>;
> rosen.xu@intel.com; sachin.saxena@oss.nxp.com; Satha Koteswara Rao
> Kottidi <skoteshwar@marvell.com>; Shahed Shaikh
> <shshaikh@marvell.com>; shaibran@amazon.com;
> shepard.siegel@atomicrules.com; asomalap@amd.com;
> somnath.kotur@broadcom.com; sthemmin@microsoft.com;
> steven.webster@windriver.com; Sunil Kumar Kori <skori@marvell.com>;
> mtetsuyah@gmail.com; Veerasenareddy Burru <vburru@marvell.com>;
> viacheslavo@nvidia.com; xiao.w.wang@intel.com;
> cloud.wangxiaoyun@huawei.com; yisen.zhuang@huawei.com;
> yongwang@vmware.com; xuanziyang2@huawei.com
> Subject: [EXT] Re: [dpdk-dev] [PATCH v3 2/2] app/testpmd: add queue based
> pfc CLI options
> 
> External Email
> 
> ----------------------------------------------------------------------
> On 1/31/2022 6:08 PM, jerinj@marvell.com wrote:
> > From: Sunil Kumar Kori <skori@marvell.com>
> >
> > Patch adds command line options to configure queue based priority flow
> > control.
> >
> > - Syntax command is given as below:
> >
> > set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
> > 	tx <on|off> <rx_qid> <rx_tc> <pause_time>
> >
> 
> Ahh I see the order is related to the configure struct, where tx_qid is part of
> rx_pause struct.
> Can you please explaing this items selection in the struct?
> Most probaly it will clarify the above usage.
> 
Ack.

> > - Example command to configure queue based priority flow control
> >    on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
> >    time 2047
> >
> > testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
> >
> > Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> 
> <...>
> 
> >
> > +set pfc_queue_ctrl
> > +~~~~~~~~~~~~~~~~~~
> > +
> > +Set the priority flow control parameter on a given Rx and Tx queue of a
> port::
> > +
> > +   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
> > +            tx (on|off) <rx_qid> <rx_tc> <pause_time>
> > +
> > +Where:
> > +
> > +* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied
> > +and traffic
> > +  will be paused when PFC frame is received with ``tx_tc`` enabled.
> > +
> > +* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
> > +
> > +* ``rx_qid`` (integer): Rx qid for which threshold will be applied
> > +and PFC
> > +  frame will be generated with ``tx_tc`` when exceeds the threshold.
> > +
> > +* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be
> paused.
> > +
> > +* ``pause_time`` (integer): Pause quota filled in the PFC frame.
> > +
> 
> 'pause_time' is related to the TX pause configuration, right? At least in the
> config struct it only exists for Tx. If so can you please document this?
>
Ack.
 

Re: [dpdk-dev] [PATCH v3 1/2] ethdev: support queue-based priority flow control

[Jerin Jacob]

)


On Thu, Feb 3, 2022 at 9:31 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 1/31/2022 6:08 PM, jerinj@marvell.com wrote:
> > From: Jerin Jacob <jerinj@marvell.com>
> >
> > Based on device support and use-case need, there are two different ways
> > to enable PFC. The first case is the port level PFC configuration, in
> > this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
> > configure the PFC, and PFC frames will be generated using based on VLAN
> > TC value.
> >
> > The second case is the queue level PFC configuration, in this
> > case, Any packet field content can be used to steer the packet to the
> > specific queue using rte_flow or RSS and then use
> > rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
> > TC mapping on each queue.
> > Based on congestion selected on the specific queue, configured TC
> > shall be used to generate PFC frames.
> >
>
> Hi Jerin, Sunil,
>
> Please find below minor comments, mostly syntax issues.
>
> > Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> > Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> > ---
> >
> > v2..v1:
> > - Introduce rte_eth_dev_priority_flow_ctrl_queue_info_get() to
> > avoid updates to rte_eth_dev_info
> > - Removed devtools/libabigail.abignore changes
> > - Address the comment from Ferruh in
> > http://patches.dpdk.org/project/dpdk/patch/20220113102718.3167282-1-jerinj@marvell.com/
> >
> >   doc/guides/nics/features.rst           |   7 +-
> >   doc/guides/rel_notes/release_22_03.rst |   6 ++
> >   lib/ethdev/ethdev_driver.h             |  12 ++-
> >   lib/ethdev/rte_ethdev.c                | 132 +++++++++++++++++++++++++
> >   lib/ethdev/rte_ethdev.h                |  89 +++++++++++++++++
> >   lib/ethdev/version.map                 |   4 +
> >   6 files changed, 247 insertions(+), 3 deletions(-)
> >
> > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > index 27be2d2576..1cacdc883a 100644
> > --- a/doc/guides/nics/features.rst
> > +++ b/doc/guides/nics/features.rst
> > @@ -379,9 +379,12 @@ Flow control
> >   Supports configuring link flow control.
> >
> >   * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
> > -  ``priority_flow_ctrl_set``.
> > +  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_info_get``,
> > +  ``priority_flow_ctrl_queue_configure``
> >   * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
> > -  ``rte_eth_dev_priority_flow_ctrl_set()``.
> > +  ``rte_eth_dev_priority_flow_ctrl_set()``,
> > +  ``rte_eth_dev_priority_flow_ctrl_queue_info_get()``,
> > +  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``.
> >
> >
> >   .. _nic_features_rate_limitation:
> > diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
> > index 3bc0630c7c..e988c104e8 100644
> > --- a/doc/guides/rel_notes/release_22_03.rst
> > +++ b/doc/guides/rel_notes/release_22_03.rst
> > @@ -69,6 +69,12 @@ New Features
> >
> >     The new API ``rte_event_eth_rx_adapter_event_port_get()`` was added.
> >
> > +* **Added an API to enable queue based priority flow ctrl(PFC).**
> > +
> > +  New APIs, ``rte_eth_dev_priority_flow_ctrl_queue_info_get()`` and
> > +  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``, was added.
> > +
> > +
> >
>
>
> Can you please move this update before ethdev driver updates.
> And no need double empty lines.


Ack


>
> >   Removed Items
> >   -------------
> > diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
> > index d95605a355..320a364766 100644
> > --- a/lib/ethdev/ethdev_driver.h
> > +++ b/lib/ethdev/ethdev_driver.h
> > @@ -533,6 +533,13 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
> >   typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
> >                               struct rte_eth_pfc_conf *pfc_conf);
> >
> > +/** @internal Get info for queue based PFC on an Ethernet device. */
> > +typedef int (*priority_flow_ctrl_queue_info_get_t)(
> > +     struct rte_eth_dev *dev, struct rte_eth_pfc_queue_info *pfc_queue_info);
> > +/** @internal Configure queue based PFC parameter on an Ethernet device. */
> > +typedef int (*priority_flow_ctrl_queue_config_t)(
> > +     struct rte_eth_dev *dev, struct rte_eth_pfc_queue_conf *pfc_queue_conf);
> > +
>
> Instead of ending line with opening parantesis '(', can you break the line after
> first argument, like:
>
> typedef int (*priority_flow_ctrl_queue_config_t)(struct rte_eth_dev *dev,
>                                 struct rte_eth_pfc_queue_conf *pfc_queue_conf);


Ack

>
> Same for all instances.
>
> >   /** @internal Update RSS redirection table on an Ethernet device. */
> >   typedef int (*reta_update_t)(struct rte_eth_dev *dev,
> >                            struct rte_eth_rss_reta_entry64 *reta_conf,
> > @@ -1080,7 +1087,10 @@ struct eth_dev_ops {
> >       flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
> >       /** Setup priority flow control */
> >       priority_flow_ctrl_set_t   priority_flow_ctrl_set;
> > -
> > +     /** Priority flow control queue info get */
> > +     priority_flow_ctrl_queue_info_get_t priority_flow_ctrl_queue_info_get;
> > +     /** Priority flow control queue configure */
> > +     priority_flow_ctrl_queue_config_t priority_flow_ctrl_queue_config;
>
> Can you please keep empty line before next (hash) group?


Ack

>
> >       /** Set Unicast Table Array */
> >       eth_uc_hash_table_set_t    uc_hash_table_set;
> >       /** Set Unicast hash bitmap */
> > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
> > index a1d475a292..2ce38cd2c5 100644
> > --- a/lib/ethdev/rte_ethdev.c
> > +++ b/lib/ethdev/rte_ethdev.c
> > @@ -4022,6 +4022,138 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
> >       return -ENOTSUP;
> >   }
> >
> > +static inline int
>
> Not sure if there is a value to ask function to be 'inline', this is in control
> path, only static can be enough.

Ack

>
> > +validate_rx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
> > +                      struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> > +{
> > +     if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
> > +         (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
>
> We don't allign to paranthesis in dpdk coding convenion [1], it should be as:
>
> if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
>                 (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
>         if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
>                 ...
>          }
> }

Ack

>
>
> [1]
> Altough I am aware many instances sneaked in, still I think better to follow
> the convention.
>
> > +             if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
> > +                     RTE_ETHDEV_LOG(ERR, "Tx queue not in range for Rx pause"
> > +                                    " (requested: %d configured: %d)\n",
> > +                                    pfc_queue_conf->rx_pause.tx_qid,
> > +                                    dev_info->nb_tx_queues);
>
>
> Should log mention that this is related to the "priority flow Rx queue control"?

Prepended "PFC" in the log message


>
> > +                     return -EINVAL;
> > +             }
> > +
> > +             if (pfc_queue_conf->rx_pause.tc >= tc_max) {
>
> Should we document somewhere that 'tc_max' value itself is an invalid value?

Updated the documentation to following

        struct {
                uint16_t tx_qid; /**< Tx queue ID */
-               uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+               uint8_t tc;
+               /**< Traffic class as per PFC (802.1Qbb) spec. The value must be
+                * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
+                */


>
> > +                     RTE_ETHDEV_LOG(ERR, "TC not in range for Rx pause"
> > +                                    " (requested: %d max: %d)\n",
> > +                                    pfc_queue_conf->rx_pause.tc, tc_max);
> > +                     return -EINVAL;
> > +             }
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +static inline int
> > +validate_tx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
> > +                      struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> > +{
> > +     if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
> > +          (pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
> > +             if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
> > +                     RTE_ETHDEV_LOG(ERR, "Rx queue not in range for Tx pause"
> > +                                    "(requested: %d configured: %d)\n",
> > +                                    pfc_queue_conf->tx_pause.rx_qid,
> > +                                    dev_info->nb_rx_queues);
> > +                     return -EINVAL;
> > +             }
> > +
> > +             if (pfc_queue_conf->tx_pause.tc >= tc_max) {
> > +                     RTE_ETHDEV_LOG(ERR, "TC not in range for Tx pause"
> > +                                    "(requested: %d max: %d)\n",
> > +                                    pfc_queue_conf->tx_pause.tc, tc_max);
> > +                     return -EINVAL;
> > +             }
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +int
> > +rte_eth_dev_priority_flow_ctrl_queue_info_get(
> > +     uint16_t port_id, struct rte_eth_pfc_queue_info *pfc_queue_info)
> > +{
> > +     struct rte_eth_dev *dev;
> > +
> > +     RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> > +     dev = &rte_eth_devices[port_id];
> > +
> > +     if (pfc_queue_info == NULL) {
> > +             RTE_ETHDEV_LOG(ERR, "PFC info param is NULL for port (%u)\n",
> > +                            port_id);
> > +             return -EINVAL;
> > +     }
> > +
> > +     if (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
> > +             return eth_err(port_id,
> > +                            (*dev->dev_ops->priority_flow_ctrl_queue_info_get)(
> > +                                    dev, pfc_queue_info));
> > +     return -ENOTSUP;
> > +}
> > +
> > +int
> > +rte_eth_dev_priority_flow_ctrl_queue_configure(
> > +     uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> > +{
> > +     struct rte_eth_pfc_queue_info pfc_info;
> > +     struct rte_eth_dev_info dev_info;
> > +     struct rte_eth_dev *dev;
> > +     int ret;
> > +
> > +     RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> > +     dev = &rte_eth_devices[port_id];
> > +
> > +     if (pfc_queue_conf == NULL) {
> > +             RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
> > +                            port_id);
> > +             return -EINVAL;
> > +     }
> > +
> > +     ret = rte_eth_dev_info_get(port_id, &dev_info);
> > +     if (ret != 0)
> > +             return ret;
> > +
> > +     ret = rte_eth_dev_priority_flow_ctrl_queue_info_get(port_id, &pfc_info);
> > +     if (ret != 0)
> > +             return ret;
> > +
> > +     if (pfc_info.capa == 0) {
> > +             RTE_ETHDEV_LOG(ERR, "Ethdev port %u does not support PFC\n",
> > +                     port_id);
> > +             return -ENOTSUP;
> > +     }
> > +
> > +     if (pfc_info.tc_max == 0) {
> > +             RTE_ETHDEV_LOG(ERR,
> > +                     "Ethdev port %u does not support PFC TC values\n",
> > +                     port_id);
> > +             return -ENOTSUP;
> > +     }
> > +
> > +     if (pfc_info.capa & RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE) {
> > +             ret = validate_rx_pause_config(&dev_info, pfc_info.tc_max,
> > +                                            pfc_queue_conf);
>
> There is capablilty flags for RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE and RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE
> also there is config flags RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_TX_PAUSE and RTE_ETH_FC_FULL

Removed that and reused rte_eth_fc_mode

-/** Device supports Rx pause for queue based PFC. */
-#define RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE RTE_BIT64(0)
-/** Device supports Tx pause for queue based PFC. */
-#define RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE RTE_BIT64(1)
-
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
@@ -1424,8 +1419,8 @@ struct rte_eth_pfc_queue_info {
         * Maximum supported traffic class as per PFC (802.1Qbb) specification.
         */
        uint8_t tc_max;
-       /** PFC queue capabilities (RTE_ETH_PFC_QUEUE_CAPA_). */
-       uint64_t capa;
+       /** PFC queue mode capabilities. */
+       enum rte_eth_fc_mode mode_capa;
 };



>
>
> What should happen if driver only support RX_PAUSE but app config request only
> TX_PAUSE?
> As far as can see with current code it pass the validation, but should it?

Yes. Good catch. Added

        /* Check requested mode supported or not */
        if (pfc_info.mode_capa == RTE_ETH_FC_RX_PAUSE &&
                        pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) {
                RTE_ETHDEV_LOG(ERR, "PFC Tx pause unsupported for port (%d)\n",
                               port_id);
                return -EINVAL;
        }

        if (pfc_info.mode_capa == RTE_ETH_FC_TX_PAUSE &&
                        pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) {
                RTE_ETHDEV_LOG(ERR, "PFC Rx pause unsupported for port (%d)\n",
                               port_id);
                return -EINVAL;
        }


>
>
> > +             if (ret != 0)
> > +                     return ret;
> > +     }
> > +
> > +     if (pfc_info.capa & RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE) {
> > +             ret = validate_tx_pause_config(&dev_info, pfc_info.tc_max,
> > +                                            pfc_queue_conf);
>
> syntax, please don't align to paranthesis
>
> > +             if (ret != 0)
> > +                     return ret;
> > +     }
> > +
> > +     if (*dev->dev_ops->priority_flow_ctrl_queue_config)
> > +             return eth_err(port_id,
> > +                            (*dev->dev_ops->priority_flow_ctrl_queue_config)(
> > +                                    dev, pfc_queue_conf));
> > +     return -ENOTSUP;
> > +}
> > +
> >   static int
> >   eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
> >                       uint16_t reta_size)
> > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> > index fa299c8ad7..383ad1cdd7 100644
> > --- a/lib/ethdev/rte_ethdev.h
> > +++ b/lib/ethdev/rte_ethdev.h
> > @@ -1395,6 +1395,48 @@ struct rte_eth_pfc_conf {
> >       uint8_t priority;          /**< VLAN User Priority. */
> >   };
> >
> > +/** Device supports Rx pause for queue based PFC. */
> > +#define RTE_ETH_PFC_QUEUE_CAPA_RX_PAUSE RTE_BIT64(0)
> > +/** Device supports Tx pause for queue based PFC. */
> > +#define RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE RTE_BIT64(1)
> > +
>
> There is already control flow mode enum 'enum rte_eth_fc_mode', those
> enum items use FC as abrivation (RTE_ETH_FC_RX_PAUSE), above ones use PFC,
> should they use same prefix 'RTE_ETH_FC_' for consistency?
>
> And overall, what for struct and functins too, is the correct abreviation
> 'pfc' or 'fc', since old code has 'fc' as far as I see?

Removed RTE_ETH_PFC_QUEUE_CAPA_TX_PAUSE and reused enum rte_eth_fc_mode
>
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> > + *
> > + * A structure used to retrieve information of queue based PFC.
> > + */
> > +struct rte_eth_pfc_queue_info {
> > +     /**
> > +      * Maximum supported traffic class as per PFC (802.1Qbb) specification.
>
> Will it be redundant to say valid value should be bigger than 0?

Updated the document as:
@@ -1440,13 +1435,19 @@ struct rte_eth_pfc_queue_conf {

        struct {
                uint16_t tx_qid; /**< Tx queue ID */
-               uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
+               uint8_t tc;
+               /**< Traffic class as per PFC (802.1Qbb) spec. The value must be
+                * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
+                */



>
> > +      */
> > +     uint8_t tc_max;
> > +     /** PFC queue capabilities (RTE_ETH_PFC_QUEUE_CAPA_). */
>
> Can move doxygen comments to same line if they fit to 80 char limit:
>         uint64_t capa; /**< PFC queue capabilities (RTE_ETH_PFC_QUEUE_CAPA_). */
>
> > +     uint64_t capa;
> > +};
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> > + *
> > + * A structure used to configure Ethernet priority flow control parameter for
> > + * ethdev queues.
> > + */
> > +struct rte_eth_pfc_queue_conf {
> > +     enum rte_eth_fc_mode mode; /**< Link flow control mode */
> > +
> > +     struct {
> > +             uint16_t tx_qid; /**< Tx queue ID */
>
> 'tx_qid' within 'rx_pause' struct, this seems done intentionally but just to double
> check, can you please describe here the intendent usage?


Clarified the doc as

@@ -1429,6 +1429,16 @@ struct rte_eth_pfc_queue_info {
  *
  * A structure used to configure Ethernet priority flow control parameter for
  * ethdev queues.
+ *
+ * rte_eth_pfc_queue_conf::rx_pause structure shall used to configure given
+ * tx_qid with corresponding tc. When ethdev device receives PFC frame with
+ * rte_eth_pfc_queue_conf::rx_pause::tc, traffic will be paused on
+ * rte_eth_pfc_queue_conf::rx_pause::tx_qid for that tc.
+ *
+ * rte_eth_pfc_queue_conf::tx_pause structure shall used to configure given
+ * rx_qid. When rx_qid is congested, PFC frames are generated with
+ * rte_eth_pfc_queue_conf::rx_pause::tc and
+ * rte_eth_pfc_queue_conf::rx_pause::pause_time to the peer.

>
> > +             uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
> > +     } rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
> > +
> > +     struct {
> > +             uint16_t pause_time; /**< Pause quota in the Pause frame */
> > +             uint16_t rx_qid;     /**< Rx queue ID */
> > +             uint8_t tc; /**< Traffic class as per PFC (802.1Qbb) spec */
> > +     } tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
> > +};
> > +
> >   /**
> >    * Tunnel type for device-specific classifier configuration.
> >    * @see rte_eth_udp_tunnel
> > @@ -4144,6 +4186,53 @@ int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
> >   int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
> >                               uint32_t pool);
> >
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Retrieve the information for queue based PFC.
> > + *
> > + * @param port_id
> > + *   The port identifier of the Ethernet device.
> > + * @param pfc_queue_info
> > + *   A pointer to a structure of type *rte_eth_pfc_queue_info* to be filled with
> > + *   the information about queue based PFC.
> > + * @return
> > + *   - (0) if successful.
> > + *   - (-ENOTSUP) if support for priority_flow_ctrl_queue_info_get does not exist.
> > + *   - (-ENODEV) if *port_id* invalid.
> > + *   - (-EINVAL) if bad parameter.
> > + */
> > +__rte_experimental
> > +int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
> > +                                struct rte_eth_pfc_queue_info *pfc_queue_info);
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Configure the queue based priority flow control for a given queue
> > + * for Ethernet device.
> > + *
> > + * @note When an ethdev port switches to queue based PFC mode, the
> > + * unconfigured queues shall be configured by the driver with
> > + * default values such as lower priority value for TC etc.
> > + *
>
> May be good to document it has dependency to 'rte_eth_dev_info_get()' API?

There is no dependency to rte_eth_dev_info_get() in the spec.

>
> > + * @param port_id
> > + *   The port identifier of the Ethernet device.
> > + * @param pfc_queue_conf
> > + *   The pointer to the structure of the priority flow control parameters
> > + *   for the queue.
> > + * @return
> > + *   - (0) if successful.
> > + *   - (-ENOTSUP) if hardware doesn't support queue based PFC mode.
> > + *   - (-ENODEV)  if *port_id* invalid.
> > + *   - (-EINVAL)  if bad parameter
> > + *   - (-EIO)     if flow control setup queue failure
> > + */
> > +__rte_experimental
> > +int rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
> > +                           struct rte_eth_pfc_queue_conf *pfc_queue_conf);
> > +
> >   /**
> >    * Remove a MAC address from the internal array of addresses.
> >    *
> > diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
> > index c2fb0669a4..49523ebc45 100644
> > --- a/lib/ethdev/version.map
> > +++ b/lib/ethdev/version.map
> > @@ -256,6 +256,10 @@ EXPERIMENTAL {
> >       rte_flow_flex_item_create;
> >       rte_flow_flex_item_release;
> >       rte_flow_pick_transfer_proxy;
> > +
> > +     # added in 22.03
> > +     rte_eth_dev_priority_flow_ctrl_queue_configure;
> > +     rte_eth_dev_priority_flow_ctrl_queue_info_get;
> >   };
> >
> >   INTERNAL {
>

[dpdk-dev] [PATCH v4 1/2] ethdev: support queue-based priority flow control

[jerinj]

From: Jerin Jacob <jerinj@marvell.com>

Based on device support and use-case need, there are two different ways
to enable PFC. The first case is the port level PFC configuration, in
this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
configure the PFC, and PFC frames will be generated using based on VLAN
TC value.

The second case is the queue level PFC configuration, in this
case, Any packet field content can be used to steer the packet to the
specific queue using rte_flow or RSS and then use
rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
TC mapping on each queue.
Based on congestion selected on the specific queue, configured TC
shall be used to generate PFC frames.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---

v4..v3:

- Remove RTE_ETH_PFC_QUEUE_CAPA_* and replace with enum rte_eth_fc_mode mode_capa
- More documentaion
- Address the comment from Ferruh in
http://patches.dpdk.org/project/dpdk/patch/20220131180859.2662034-1-jerinj@marvell.com/

v3..v1:

- Introduce rte_eth_dev_priority_flow_ctrl_queue_info_get() to
avoid updates to rte_eth_dev_info

- Removed devtools/libabigail.abignore changes
- Address the comment from Ferruh in
http://patches.dpdk.org/project/dpdk/patch/20220113102718.3167282-1-jerinj@marvell.com/


 doc/guides/nics/features.rst           |   7 +-
 doc/guides/rel_notes/release_22_03.rst |   5 +
 lib/ethdev/ethdev_driver.h             |  11 ++
 lib/ethdev/rte_ethdev.c                | 139 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h                | 100 ++++++++++++++++++
 lib/ethdev/version.map                 |   4 +
 6 files changed, 264 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 27be2d2576..1cacdc883a 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -379,9 +379,12 @@ Flow control
 Supports configuring link flow control.
 
 * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
-  ``priority_flow_ctrl_set``.
+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_info_get``,
+  ``priority_flow_ctrl_queue_configure``
 * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
-  ``rte_eth_dev_priority_flow_ctrl_set()``.
+  ``rte_eth_dev_priority_flow_ctrl_set()``,
+  ``rte_eth_dev_priority_flow_ctrl_queue_info_get()``,
+  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``.
 
 
 .. _nic_features_rate_limitation:
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 746f50e84f..2ec6f53efe 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added an API to enable queue based priority flow ctrl(PFC).**
+
+  New APIs, ``rte_eth_dev_priority_flow_ctrl_queue_info_get()`` and
+  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``, was added.
+
 * **Updated Cisco enic driver.**
 
   * Added rte_flow support for matching GENEVE packets.
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index d95605a355..d4d396468c 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -533,6 +533,13 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
 typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
 				struct rte_eth_pfc_conf *pfc_conf);
 
+/** @internal Get info for queue based PFC on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_info_get_t)(struct rte_eth_dev *dev,
+			struct rte_eth_pfc_queue_info *pfc_queue_info);
+/** @internal Configure queue based PFC parameter on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_config_t)(struct rte_eth_dev *dev,
+			struct rte_eth_pfc_queue_conf *pfc_queue_conf);
+
 /** @internal Update RSS redirection table on an Ethernet device. */
 typedef int (*reta_update_t)(struct rte_eth_dev *dev,
 			     struct rte_eth_rss_reta_entry64 *reta_conf,
@@ -1080,6 +1087,10 @@ struct eth_dev_ops {
 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
 	/** Setup priority flow control */
 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
+	/** Priority flow control queue info get */
+	priority_flow_ctrl_queue_info_get_t priority_flow_ctrl_queue_info_get;
+	/** Priority flow control queue configure */
+	priority_flow_ctrl_queue_config_t priority_flow_ctrl_queue_config;
 
 	/** Set Unicast Table Array */
 	eth_uc_hash_table_set_t    uc_hash_table_set;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 29e21ad580..882f231e42 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -4022,6 +4022,145 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 	return -ENOTSUP;
 }
 
+static int
+validate_rx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
+			(pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
+			RTE_ETHDEV_LOG(ERR, "PFC Tx queue not in range for Rx pause requested:%d configured:%d\n",
+				       pfc_queue_conf->rx_pause.tx_qid,
+				       dev_info->nb_tx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->rx_pause.tc >= tc_max) {
+			RTE_ETHDEV_LOG(ERR, "PFC TC not in range for Rx pause requested:%d max:%d\n",
+				       pfc_queue_conf->rx_pause.tc, tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+validate_tx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
+			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
+			(pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
+			RTE_ETHDEV_LOG(ERR, "PFC Rx queue not in range for Tx pause requested:%d configured:%d\n",
+				       pfc_queue_conf->tx_pause.rx_qid,
+				       dev_info->nb_rx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->tx_pause.tc >= tc_max) {
+			RTE_ETHDEV_LOG(ERR, "PFC TC not in range for Tx pause requested:%d max:%d\n",
+				       pfc_queue_conf->tx_pause.tc, tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_info_get(
+	uint16_t port_id, struct rte_eth_pfc_queue_info *pfc_queue_info)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_info == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC info param is NULL for port (%u)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
+		return eth_err(port_id, (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
+					 (dev, pfc_queue_info));
+	return -ENOTSUP;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_configure(
+	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	struct rte_eth_pfc_queue_info pfc_info;
+	struct rte_eth_dev_info dev_info;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_conf == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_info_get(port_id, &pfc_info);
+	if (ret != 0)
+		return ret;
+
+	if (pfc_info.tc_max == 0) {
+		RTE_ETHDEV_LOG(ERR, "Ethdev port %u does not support PFC TC values\n",
+			       port_id);
+		return -ENOTSUP;
+	}
+
+	/* Check requested mode supported or not */
+	if (pfc_info.mode_capa == RTE_ETH_FC_RX_PAUSE &&
+			pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) {
+		RTE_ETHDEV_LOG(ERR, "PFC Tx pause unsupported for port (%d)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	if (pfc_info.mode_capa == RTE_ETH_FC_TX_PAUSE &&
+			pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) {
+		RTE_ETHDEV_LOG(ERR, "PFC Rx pause unsupported for port (%d)\n",
+			       port_id);
+		return -EINVAL;
+	}
+
+	/* Validate Rx pause parameters */
+	if (pfc_info.mode_capa == RTE_ETH_FC_FULL ||
+			pfc_info.mode_capa == RTE_ETH_FC_RX_PAUSE) {
+		ret = validate_rx_pause_config(&dev_info, pfc_info.tc_max,
+					       pfc_queue_conf);
+		if (ret != 0)
+			return ret;
+	}
+
+	/* Validate Tx pause parameters */
+	if (pfc_info.mode_capa == RTE_ETH_FC_FULL ||
+			pfc_info.mode_capa == RTE_ETH_FC_TX_PAUSE) {
+		ret = validate_tx_pause_config(&dev_info, pfc_info.tc_max,
+					       pfc_queue_conf);
+		if (ret != 0)
+			return ret;
+	}
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_config)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_config)(
+				       dev, pfc_queue_conf));
+	return -ENOTSUP;
+}
+
 static int
 eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
 			uint16_t reta_size)
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 147cc1ced3..c526c8fc0a 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1408,6 +1408,59 @@ struct rte_eth_pfc_conf {
 	uint8_t priority;          /**< VLAN User Priority. */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to retrieve information of queue based PFC.
+ */
+struct rte_eth_pfc_queue_info {
+	/**
+	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
+	 */
+	uint8_t tc_max;
+	/** PFC queue mode capabilities. */
+	enum rte_eth_fc_mode mode_capa;
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to configure Ethernet priority flow control parameter for
+ * ethdev queues.
+ *
+ * rte_eth_pfc_queue_conf::rx_pause structure shall used to configure given
+ * tx_qid with corresponding tc. When ethdev device receives PFC frame with
+ * rte_eth_pfc_queue_conf::rx_pause::tc, traffic will be paused on
+ * rte_eth_pfc_queue_conf::rx_pause::tx_qid for that tc.
+ *
+ * rte_eth_pfc_queue_conf::tx_pause structure shall used to configure given
+ * rx_qid. When rx_qid is congested, PFC frames are generated with
+ * rte_eth_pfc_queue_conf::rx_pause::tc and
+ * rte_eth_pfc_queue_conf::rx_pause::pause_time to the peer.
+ */
+struct rte_eth_pfc_queue_conf {
+	enum rte_eth_fc_mode mode; /**< Link flow control mode */
+
+	struct {
+		uint16_t tx_qid; /**< Tx queue ID */
+		uint8_t tc;
+		/**< Traffic class as per PFC (802.1Qbb) spec. The value must be
+		 * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
+		 */
+	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
+
+	struct {
+		uint16_t pause_time; /**< Pause quota in the Pause frame */
+		uint16_t rx_qid;     /**< Rx queue ID */
+		uint8_t tc;
+		/**< Traffic class as per PFC (802.1Qbb) spec. The value must be
+		 * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
+		 */
+	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
+};
+
 /**
  * Tunnel type for device-specific classifier configuration.
  * @see rte_eth_udp_tunnel
@@ -4158,6 +4211,53 @@ int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
 				uint32_t pool);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Retrieve the information for queue based PFC.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_info
+ *   A pointer to a structure of type *rte_eth_pfc_queue_info* to be filled with
+ *   the information about queue based PFC.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if support for priority_flow_ctrl_queue_info_get does not exist.
+ *   - (-ENODEV) if *port_id* invalid.
+ *   - (-EINVAL) if bad parameter.
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
+				   struct rte_eth_pfc_queue_info *pfc_queue_info);
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Configure the queue based priority flow control for a given queue
+ * for Ethernet device.
+ *
+ * @note When an ethdev port switches to queue based PFC mode, the
+ * unconfigured queues shall be configured by the driver with
+ * default values such as lower priority value for TC etc.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_conf
+ *   The pointer to the structure of the priority flow control parameters
+ *   for the queue.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if hardware doesn't support queue based PFC mode.
+ *   - (-ENODEV)  if *port_id* invalid.
+ *   - (-EINVAL)  if bad parameter
+ *   - (-EIO)     if flow control setup queue failure
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
+			      struct rte_eth_pfc_queue_conf *pfc_queue_conf);
+
 /**
  * Remove a MAC address from the internal array of addresses.
  *
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index c2fb0669a4..49523ebc45 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -256,6 +256,10 @@ EXPERIMENTAL {
 	rte_flow_flex_item_create;
 	rte_flow_flex_item_release;
 	rte_flow_pick_transfer_proxy;
+
+	# added in 22.03
+	rte_eth_dev_priority_flow_ctrl_queue_configure;
+	rte_eth_dev_priority_flow_ctrl_queue_info_get;
 };
 
 INTERNAL {
-- 
2.35.1

[dpdk-dev] [PATCH v4 2/2] app/testpmd: add queue based pfc CLI options

[jerinj]

From: Sunil Kumar Kori <skori@marvell.com>

Patch adds command line options to configure queue based
priority flow control.

- Syntax command is given as below:

set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
	tx <on|off> <rx_qid> <rx_tc> <pause_time>

- Example command to configure queue based priority flow control
  on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
  time 2047

testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047

Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
v4..v3:
- fix documentation to explain CLI syntax.
v3..v1:
- Sync up the implementation to use new APIs

 app/test-pmd/cmdline.c                      | 122 ++++++++++++++++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  23 ++++
 2 files changed, 145 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index e626b1c7d9..1af0321af0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -544,6 +544,11 @@ static void cmd_help_long_parsed(void *parsed_result,
 			"    Set the priority flow control parameter on a"
 			" port.\n\n"
 
+			"set pfc_queue_ctrl (port_id) rx (on|off) (tx_qid)"
+			" (tx_tc) tx (on|off) (rx_qid) (rx_tc) (pause_time)\n"
+			"    Set the queue priority flow control parameter on a"
+			" given Rx and Tx queues of a port.\n\n"
+
 			"set stat_qmap (tx|rx) (port_id) (queue_id) (qmapping)\n"
 			"    Set statistics mapping (qmapping 0..15) for RX/TX"
 			" queue on port.\n"
@@ -7690,6 +7695,122 @@ cmdline_parse_inst_t cmd_priority_flow_control_set = {
 	},
 };
 
+struct cmd_queue_priority_flow_ctrl_set_result {
+	cmdline_fixed_string_t set;
+	cmdline_fixed_string_t pfc_queue_ctrl;
+	portid_t port_id;
+	cmdline_fixed_string_t rx;
+	cmdline_fixed_string_t rx_pfc_mode;
+	uint16_t tx_qid;
+	uint8_t  tx_tc;
+	cmdline_fixed_string_t tx;
+	cmdline_fixed_string_t tx_pfc_mode;
+	uint16_t rx_qid;
+	uint8_t  rx_tc;
+	uint16_t pause_time;
+};
+
+static void
+cmd_queue_priority_flow_ctrl_set_parsed(void *parsed_result,
+					__rte_unused struct cmdline *cl,
+					__rte_unused void *data)
+{
+	struct cmd_queue_priority_flow_ctrl_set_result *res = parsed_result;
+	struct rte_eth_pfc_queue_conf pfc_queue_conf;
+	int rx_fc_enable, tx_fc_enable;
+	int ret;
+
+	/*
+	 * Rx on/off, flow control is enabled/disabled on RX side. This can
+	 * indicate the RTE_ETH_FC_TX_PAUSE, Transmit pause frame at the Rx
+	 * side. Tx on/off, flow control is enabled/disabled on TX side. This
+	 * can indicate the RTE_ETH_FC_RX_PAUSE, Respond to the pause frame at
+	 * the Tx side.
+	 */
+	static enum rte_eth_fc_mode rx_tx_onoff_2_mode[2][2] = {
+		{RTE_ETH_FC_NONE, RTE_ETH_FC_TX_PAUSE},
+		{RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_FULL}
+	};
+
+	memset(&pfc_queue_conf, 0, sizeof(struct rte_eth_pfc_queue_conf));
+	rx_fc_enable = (!strncmp(res->rx_pfc_mode, "on", 2)) ? 1 : 0;
+	tx_fc_enable = (!strncmp(res->tx_pfc_mode, "on", 2)) ? 1 : 0;
+	pfc_queue_conf.mode = rx_tx_onoff_2_mode[rx_fc_enable][tx_fc_enable];
+	pfc_queue_conf.rx_pause.tc  = res->tx_tc;
+	pfc_queue_conf.rx_pause.tx_qid = res->tx_qid;
+	pfc_queue_conf.tx_pause.tc  = res->rx_tc;
+	pfc_queue_conf.tx_pause.rx_qid  = res->rx_qid;
+	pfc_queue_conf.tx_pause.pause_time = res->pause_time;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_configure(res->port_id,
+							     &pfc_queue_conf);
+	if (ret != 0) {
+		fprintf(stderr,
+			"bad queue priority flow control parameter, rc = %d\n",
+			ret);
+	}
+}
+
+cmdline_parse_token_string_t cmd_q_pfc_set_set =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				set, "set");
+cmdline_parse_token_string_t cmd_q_pfc_set_flow_ctrl =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pfc_queue_ctrl, "pfc_queue_ctrl");
+cmdline_parse_token_num_t cmd_q_pfc_set_portid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				port_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_q_pfc_set_rx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx, "rx");
+cmdline_parse_token_string_t cmd_q_pfc_set_rx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_tc, RTE_UINT8);
+cmdline_parse_token_string_t cmd_q_pfc_set_tx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx, "tx");
+cmdline_parse_token_string_t cmd_q_pfc_set_tx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_tc, RTE_UINT8);
+cmdline_parse_token_num_t cmd_q_pfc_set_pause_time =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pause_time, RTE_UINT16);
+
+cmdline_parse_inst_t cmd_queue_priority_flow_control_set = {
+	.f = cmd_queue_priority_flow_ctrl_set_parsed,
+	.data = NULL,
+	.help_str = "set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> "
+		"tx <on|off> <rx_qid> <rx_tc> <pause_time>: "
+		"Configure the Ethernet queue priority flow control",
+	.tokens = {
+		(void *)&cmd_q_pfc_set_set,
+		(void *)&cmd_q_pfc_set_flow_ctrl,
+		(void *)&cmd_q_pfc_set_portid,
+		(void *)&cmd_q_pfc_set_rx,
+		(void *)&cmd_q_pfc_set_rx_mode,
+		(void *)&cmd_q_pfc_set_tx_qid,
+		(void *)&cmd_q_pfc_set_tx_tc,
+		(void *)&cmd_q_pfc_set_tx,
+		(void *)&cmd_q_pfc_set_tx_mode,
+		(void *)&cmd_q_pfc_set_rx_qid,
+		(void *)&cmd_q_pfc_set_rx_tc,
+		(void *)&cmd_q_pfc_set_pause_time,
+		NULL,
+	},
+};
+
 /* *** RESET CONFIGURATION *** */
 struct cmd_reset_result {
 	cmdline_fixed_string_t reset;
@@ -17765,6 +17886,7 @@ cmdline_parse_ctx_t main_ctx[] = {
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_set_autoneg,
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_show,
 	(cmdline_parse_inst_t *)&cmd_priority_flow_control_set,
+	(cmdline_parse_inst_t *)&cmd_queue_priority_flow_control_set,
 	(cmdline_parse_inst_t *)&cmd_config_dcb,
 	(cmdline_parse_inst_t *)&cmd_read_reg,
 	(cmdline_parse_inst_t *)&cmd_read_reg_bit_field,
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 94792d88cc..44b582a486 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -1561,6 +1561,29 @@ Where:
 
 * ``priority`` (0-7): VLAN User Priority.
 
+set pfc_queue_ctrl
+~~~~~~~~~~~~~~~~~~
+
+Set the priority flow control parameter on a given Rx and Tx queue of a port::
+
+   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
+            tx (on|off) <rx_qid> <rx_tc> <pause_time>
+
+Where:
+
+* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
+  will be paused when PFC frame is received with ``tx_tc`` enabled.
+
+* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
+
+* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
+  frame will be generated with ``tx_tc`` when exceeds the threshold.
+
+* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
+
+* ``pause_time`` (integer): Pause quanta filled in the PFC frame for which
+  interval, remote Tx will be paused. Valid only if Tx pause is on.
+
 set stat_qmap
 ~~~~~~~~~~~~~
 
-- 
2.35.1

Re: [dpdk-dev] [PATCH v4 1/2] ethdev: support queue-based priority flow control

[Ferruh Yigit]

On 2/7/2022 1:55 PM, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
> 
> Based on device support and use-case need, there are two different ways
> to enable PFC. The first case is the port level PFC configuration, in
> this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
> configure the PFC, and PFC frames will be generated using based on VLAN
> TC value.
> 
> The second case is the queue level PFC configuration, in this
> case, Any packet field content can be used to steer the packet to the
> specific queue using rte_flow or RSS and then use
> rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
> TC mapping on each queue.
> Based on congestion selected on the specific queue, configured TC
> shall be used to generate PFC frames.
> 

I put only some syntax comments, can you please check below?

patch lgtm except syntax issues.

> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> ---
> 
> v4..v3:
> 
> - Remove RTE_ETH_PFC_QUEUE_CAPA_* and replace with enum rte_eth_fc_mode mode_capa
> - More documentaion
> - Address the comment from Ferruh in
> http://patches.dpdk.org/project/dpdk/patch/20220131180859.2662034-1-jerinj@marvell.com/
> 
> v3..v1:
> 
> - Introduce rte_eth_dev_priority_flow_ctrl_queue_info_get() to
> avoid updates to rte_eth_dev_info
> 
> - Removed devtools/libabigail.abignore changes
> - Address the comment from Ferruh in
> http://patches.dpdk.org/project/dpdk/patch/20220113102718.3167282-1-jerinj@marvell.com/
> 
> 
>   doc/guides/nics/features.rst           |   7 +-
>   doc/guides/rel_notes/release_22_03.rst |   5 +
>   lib/ethdev/ethdev_driver.h             |  11 ++
>   lib/ethdev/rte_ethdev.c                | 139 +++++++++++++++++++++++++
>   lib/ethdev/rte_ethdev.h                | 100 ++++++++++++++++++
>   lib/ethdev/version.map                 |   4 +
>   6 files changed, 264 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> index 27be2d2576..1cacdc883a 100644
> --- a/doc/guides/nics/features.rst
> +++ b/doc/guides/nics/features.rst
> @@ -379,9 +379,12 @@ Flow control
>   Supports configuring link flow control.
>   
>   * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
> -  ``priority_flow_ctrl_set``.
> +  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_info_get``,
> +  ``priority_flow_ctrl_queue_configure``
>   * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
> -  ``rte_eth_dev_priority_flow_ctrl_set()``.
> +  ``rte_eth_dev_priority_flow_ctrl_set()``,
> +  ``rte_eth_dev_priority_flow_ctrl_queue_info_get()``,
> +  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``.
>   
>   
>   .. _nic_features_rate_limitation:
> diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
> index 746f50e84f..2ec6f53efe 100644
> --- a/doc/guides/rel_notes/release_22_03.rst
> +++ b/doc/guides/rel_notes/release_22_03.rst
> @@ -55,6 +55,11 @@ New Features
>        Also, make sure to start the actual text at the margin.
>        =======================================================
>   
> +* **Added an API to enable queue based priority flow ctrl(PFC).**
> +
> +  New APIs, ``rte_eth_dev_priority_flow_ctrl_queue_info_get()`` and
> +  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``, was added.
> +
>   * **Updated Cisco enic driver.**
>   
>     * Added rte_flow support for matching GENEVE packets.
> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
> index d95605a355..d4d396468c 100644
> --- a/lib/ethdev/ethdev_driver.h
> +++ b/lib/ethdev/ethdev_driver.h
> @@ -533,6 +533,13 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
>   typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
>   				struct rte_eth_pfc_conf *pfc_conf);
>   
> +/** @internal Get info for queue based PFC on an Ethernet device. */
> +typedef int (*priority_flow_ctrl_queue_info_get_t)(struct rte_eth_dev *dev,
> +			struct rte_eth_pfc_queue_info *pfc_queue_info);
> +/** @internal Configure queue based PFC parameter on an Ethernet device. */
> +typedef int (*priority_flow_ctrl_queue_config_t)(struct rte_eth_dev *dev,
> +			struct rte_eth_pfc_queue_conf *pfc_queue_conf);
> +
>   /** @internal Update RSS redirection table on an Ethernet device. */
>   typedef int (*reta_update_t)(struct rte_eth_dev *dev,
>   			     struct rte_eth_rss_reta_entry64 *reta_conf,
> @@ -1080,6 +1087,10 @@ struct eth_dev_ops {
>   	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
>   	/** Setup priority flow control */
>   	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
> +	/** Priority flow control queue info get */
> +	priority_flow_ctrl_queue_info_get_t priority_flow_ctrl_queue_info_get;
> +	/** Priority flow control queue configure */
> +	priority_flow_ctrl_queue_config_t priority_flow_ctrl_queue_config;
>   
>   	/** Set Unicast Table Array */
>   	eth_uc_hash_table_set_t    uc_hash_table_set;
> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
> index 29e21ad580..882f231e42 100644
> --- a/lib/ethdev/rte_ethdev.c
> +++ b/lib/ethdev/rte_ethdev.c
> @@ -4022,6 +4022,145 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>   	return -ENOTSUP;
>   }
>   
> +static int
> +validate_rx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
> +			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)

Instead of alligning to the '(', can you just put two tabs?

> +{
> +	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
> +			(pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
> +		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
> +			RTE_ETHDEV_LOG(ERR, "PFC Tx queue not in range for Rx pause requested:%d configured:%d\n",
> +				       pfc_queue_conf->rx_pause.tx_qid,

Same here, no need to allign to upper line.
Also can you please break the line after 'ERR,' to reduce the line length:
RTE_ETHDEV_LOG(ERR,
	"PFC Tx queue not in range for Rx pause requested:%d configured:%d\n",
	....

> +				       dev_info->nb_tx_queues);
> +			return -EINVAL;
> +		}
> +
> +		if (pfc_queue_conf->rx_pause.tc >= tc_max) {
> +			RTE_ETHDEV_LOG(ERR, "PFC TC not in range for Rx pause requested:%d max:%d\n",
> +				       pfc_queue_conf->rx_pause.tc, tc_max);

same here

> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +validate_tx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
> +			 struct rte_eth_pfc_queue_conf *pfc_queue_conf)

ditto

> +{
> +	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
> +			(pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
> +		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
> +			RTE_ETHDEV_LOG(ERR, "PFC Rx queue not in range for Tx pause requested:%d configured:%d\n",
> +				       pfc_queue_conf->tx_pause.rx_qid,
> +				       dev_info->nb_rx_queues);

ditto

> +			return -EINVAL;
> +		}
> +
> +		if (pfc_queue_conf->tx_pause.tc >= tc_max) {
> +			RTE_ETHDEV_LOG(ERR, "PFC TC not in range for Tx pause requested:%d max:%d\n",
> +				       pfc_queue_conf->tx_pause.tc, tc_max);

ditto

> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +rte_eth_dev_priority_flow_ctrl_queue_info_get(

Please don't end the line with '(', can you move the 'uint16_t port_id' to this line?

> +	uint16_t port_id, struct rte_eth_pfc_queue_info *pfc_queue_info)
> +{
> +	struct rte_eth_dev *dev;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> +	dev = &rte_eth_devices[port_id];
> +
> +	if (pfc_queue_info == NULL) {
> +		RTE_ETHDEV_LOG(ERR, "PFC info param is NULL for port (%u)\n",
> +			       port_id);

no align

> +		return -EINVAL;
> +	}
> +
> +	if (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
> +		return eth_err(port_id, (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
> +					 (dev, pfc_queue_info));

ditto

> +	return -ENOTSUP;
> +}
> +
> +int
> +rte_eth_dev_priority_flow_ctrl_queue_configure(

no line ending with '(' please

> +	uint16_t port_id, struct rte_eth_pfc_queue_conf *pfc_queue_conf)
> +{
> +	struct rte_eth_pfc_queue_info pfc_info;
> +	struct rte_eth_dev_info dev_info;
> +	struct rte_eth_dev *dev;
> +	int ret;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> +	dev = &rte_eth_devices[port_id];
> +
> +	if (pfc_queue_conf == NULL) {
> +		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
> +			       port_id);

no align

> +		return -EINVAL;
> +	}
> +
> +	ret = rte_eth_dev_info_get(port_id, &dev_info);
> +	if (ret != 0)
> +		return ret;
> +
> +	ret = rte_eth_dev_priority_flow_ctrl_queue_info_get(port_id, &pfc_info);
> +	if (ret != 0)
> +		return ret;
> +
> +	if (pfc_info.tc_max == 0) {
> +		RTE_ETHDEV_LOG(ERR, "Ethdev port %u does not support PFC TC values\n",
> +			       port_id);

ditto

> +		return -ENOTSUP;
> +	}
> +
> +	/* Check requested mode supported or not */
> +	if (pfc_info.mode_capa == RTE_ETH_FC_RX_PAUSE &&
> +			pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) {
> +		RTE_ETHDEV_LOG(ERR, "PFC Tx pause unsupported for port (%d)\n",
> +			       port_id);

ditto

> +		return -EINVAL;
> +	}
> +
> +	if (pfc_info.mode_capa == RTE_ETH_FC_TX_PAUSE &&
> +			pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) {
> +		RTE_ETHDEV_LOG(ERR, "PFC Rx pause unsupported for port (%d)\n",
> +			       port_id);

ditto

> +		return -EINVAL;
> +	}
> +
> +	/* Validate Rx pause parameters */
> +	if (pfc_info.mode_capa == RTE_ETH_FC_FULL ||
> +			pfc_info.mode_capa == RTE_ETH_FC_RX_PAUSE) {
> +		ret = validate_rx_pause_config(&dev_info, pfc_info.tc_max,
> +					       pfc_queue_conf);

ditto

> +		if (ret != 0)
> +			return ret;
> +	}
> +
> +	/* Validate Tx pause parameters */
> +	if (pfc_info.mode_capa == RTE_ETH_FC_FULL ||
> +			pfc_info.mode_capa == RTE_ETH_FC_TX_PAUSE) {
> +		ret = validate_tx_pause_config(&dev_info, pfc_info.tc_max,
> +					       pfc_queue_conf);

ditto

> +		if (ret != 0)
> +			return ret;
> +	}
> +
> +	if (*dev->dev_ops->priority_flow_ctrl_queue_config)
> +		return eth_err(port_id,
> +			       (*dev->dev_ops->priority_flow_ctrl_queue_config)(

ditto

> +				       dev, pfc_queue_conf));
> +	return -ENOTSUP;
> +}
> +
>   static int
>   eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
>   			uint16_t reta_size)
> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> index 147cc1ced3..c526c8fc0a 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -1408,6 +1408,59 @@ struct rte_eth_pfc_conf {
>   	uint8_t priority;          /**< VLAN User Priority. */
>   };
>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + *
> + * A structure used to retrieve information of queue based PFC.
> + */
> +struct rte_eth_pfc_queue_info {
> +	/**
> +	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
> +	 */
> +	uint8_t tc_max;
> +	/** PFC queue mode capabilities. */
> +	enum rte_eth_fc_mode mode_capa;
> +};
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + *
> + * A structure used to configure Ethernet priority flow control parameter for
> + * ethdev queues.
> + *
> + * rte_eth_pfc_queue_conf::rx_pause structure shall used to configure given
> + * tx_qid with corresponding tc. When ethdev device receives PFC frame with
> + * rte_eth_pfc_queue_conf::rx_pause::tc, traffic will be paused on
> + * rte_eth_pfc_queue_conf::rx_pause::tx_qid for that tc.
> + *
> + * rte_eth_pfc_queue_conf::tx_pause structure shall used to configure given
> + * rx_qid. When rx_qid is congested, PFC frames are generated with
> + * rte_eth_pfc_queue_conf::rx_pause::tc and
> + * rte_eth_pfc_queue_conf::rx_pause::pause_time to the peer.
> + */
> +struct rte_eth_pfc_queue_conf {
> +	enum rte_eth_fc_mode mode; /**< Link flow control mode */
> +
> +	struct {
> +		uint16_t tx_qid; /**< Tx queue ID */
> +		uint8_t tc;
> +		/**< Traffic class as per PFC (802.1Qbb) spec. The value must be
> +		 * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
> +		 */

Can you please move multi line doxygen comments above the variable?

> +	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
> +
> +	struct {
> +		uint16_t pause_time; /**< Pause quota in the Pause frame */
> +		uint16_t rx_qid;     /**< Rx queue ID */
> +		uint8_t tc;
> +		/**< Traffic class as per PFC (802.1Qbb) spec. The value must be
> +		 * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
> +		 */

ditto

> +	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
> +};
> +
>   /**
>    * Tunnel type for device-specific classifier configuration.
>    * @see rte_eth_udp_tunnel
> @@ -4158,6 +4211,53 @@ int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
>   int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
>   				uint32_t pool);
>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Retrieve the information for queue based PFC.
> + *
> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param pfc_queue_info
> + *   A pointer to a structure of type *rte_eth_pfc_queue_info* to be filled with
> + *   the information about queue based PFC.
> + * @return
> + *   - (0) if successful.
> + *   - (-ENOTSUP) if support for priority_flow_ctrl_queue_info_get does not exist.
> + *   - (-ENODEV) if *port_id* invalid.
> + *   - (-EINVAL) if bad parameter.
> + */
> +__rte_experimental
> +int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
> +				   struct rte_eth_pfc_queue_info *pfc_queue_info);

2/3 tabs is OK for the next line

Can you please put an empty line between functions?

> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Configure the queue based priority flow control for a given queue
> + * for Ethernet device.
> + *
> + * @note When an ethdev port switches to queue based PFC mode, the
> + * unconfigured queues shall be configured by the driver with
> + * default values such as lower priority value for TC etc.
> + *
> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param pfc_queue_conf
> + *   The pointer to the structure of the priority flow control parameters
> + *   for the queue.
> + * @return
> + *   - (0) if successful.
> + *   - (-ENOTSUP) if hardware doesn't support queue based PFC mode.
> + *   - (-ENODEV)  if *port_id* invalid.
> + *   - (-EINVAL)  if bad parameter
> + *   - (-EIO)     if flow control setup queue failure
> + */
> +__rte_experimental
> +int rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
> +			      struct rte_eth_pfc_queue_conf *pfc_queue_conf);
> +

ditto

>   /**
>    * Remove a MAC address from the internal array of addresses.
>    *
> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
> index c2fb0669a4..49523ebc45 100644
> --- a/lib/ethdev/version.map
> +++ b/lib/ethdev/version.map
> @@ -256,6 +256,10 @@ EXPERIMENTAL {
>   	rte_flow_flex_item_create;
>   	rte_flow_flex_item_release;
>   	rte_flow_pick_transfer_proxy;
> +
> +	# added in 22.03
> +	rte_eth_dev_priority_flow_ctrl_queue_configure;
> +	rte_eth_dev_priority_flow_ctrl_queue_info_get;
>   };
>   
>   INTERNAL {

Re: [dpdk-dev] [PATCH v4 2/2] app/testpmd: add queue based pfc CLI options

[Ferruh Yigit]

On 2/7/2022 1:55 PM, jerinj@marvell.com wrote:
> From: Sunil Kumar Kori<skori@marvell.com>
> 
> Patch adds command line options to configure queue based
> priority flow control.
> 
> - Syntax command is given as below:
> 
> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
> 	tx <on|off> <rx_qid> <rx_tc> <pause_time>
> 
> - Example command to configure queue based priority flow control
>    on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
>    time 2047
> 
> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
> 
> Signed-off-by: Sunil Kumar Kori<skori@marvell.com>

lgtm, thanks

Re: [dpdk-dev] [PATCH v4 2/2] app/testpmd: add queue based pfc CLI options

[Ajit Khaparde]

[-- Attachment #1: Type: text/plain, Size: 3037 bytes --]

On Mon, Feb 7, 2022 at 5:54 AM <jerinj@marvell.com> wrote:
>
> From: Sunil Kumar Kori <skori@marvell.com>
>
> Patch adds command line options to configure queue based
> priority flow control.
>
> - Syntax command is given as below:
>
> set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
>         tx <on|off> <rx_qid> <rx_tc> <pause_time>
>
> - Example command to configure queue based priority flow control
>   on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
>   time 2047
>
> testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047
>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>

Couple of minor nits. Otherwise this looks good.
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

> ---
> v4..v3:
> - fix documentation to explain CLI syntax.
> v3..v1:
> - Sync up the implementation to use new APIs
>
>  app/test-pmd/cmdline.c                      | 122 ++++++++++++++++++++
>  doc/guides/testpmd_app_ug/testpmd_funcs.rst |  23 ++++
>  2 files changed, 145 insertions(+)
>
::snip::

> +
> +static void
> +cmd_queue_priority_flow_ctrl_set_parsed(void *parsed_result,
> +                                       __rte_unused struct cmdline *cl,
> +                                       __rte_unused void *data)
> +{
> +       struct cmd_queue_priority_flow_ctrl_set_result *res = parsed_result;
> +       struct rte_eth_pfc_queue_conf pfc_queue_conf;
> +       int rx_fc_enable, tx_fc_enable;
> +       int ret;
> +
> +       /*
> +        * Rx on/off, flow control is enabled/disabled on RX side. This can
on the RX side.

> +        * indicate the RTE_ETH_FC_TX_PAUSE, Transmit pause frame at the Rx
> +        * side. Tx on/off, flow control is enabled/disabled on TX side. This
> +        * can indicate the RTE_ETH_FC_RX_PAUSE, Respond to the pause frame at
> +        * the Tx side.
> +        */
> +       static enum rte_eth_fc_mode rx_tx_onoff_2_mode[2][2] = {
> +               {RTE_ETH_FC_NONE, RTE_ETH_FC_TX_PAUSE},
> +               {RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_FULL}
> +       };
> +
::snip:

>
> +set pfc_queue_ctrl
> +~~~~~~~~~~~~~~~~~~
> +
> +Set the priority flow control parameter on a given Rx and Tx queue of a port::
> +
> +   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
> +            tx (on|off) <rx_qid> <rx_tc> <pause_time>
> +
> +Where:
> +
> +* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
> +  will be paused when PFC frame is received with ``tx_tc`` enabled.
> +
> +* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
> +
> +* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
> +  frame will be generated with ``tx_tc`` when exceeds the threshold.
when the threshold exceeds.

> +
> +* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
> +
> +* ``pause_time`` (integer): Pause quanta filled in the PFC frame for which
> +  interval, remote Tx will be paused. Valid only if Tx pause is on.
> +
>  set stat_qmap
>  ~~~~~~~~~~~~~
>
> --
> 2.35.1
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

Re: [dpdk-dev] [PATCH v4 1/2] ethdev: support queue-based priority flow control

[Ajit Khaparde]

[-- Attachment #1: Type: text/plain, Size: 3656 bytes --]

On Mon, Feb 7, 2022 at 5:54 AM <jerinj@marvell.com> wrote:
>
> From: Jerin Jacob <jerinj@marvell.com>
>
> Based on device support and use-case need, there are two different ways
> to enable PFC. The first case is the port level PFC configuration, in
> this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
> configure the PFC, and PFC frames will be generated using based on VLAN
> TC value.
>
> The second case is the queue level PFC configuration, in this
> case, Any packet field content can be used to steer the packet to the
> specific queue using rte_flow or RSS and then use
> rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
> TC mapping on each queue.
> Based on congestion selected on the specific queue, configured TC
> shall be used to generate PFC frames.
>
> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
Couple of nits inline.

> ---
>
> v4..v3:
>
> - Remove RTE_ETH_PFC_QUEUE_CAPA_* and replace with enum rte_eth_fc_mode mode_capa
> - More documentaion
> - Address the comment from Ferruh in
> http://patches.dpdk.org/project/dpdk/patch/20220131180859.2662034-1-jerinj@marvell.com/
>
> v3..v1:
>
> - Introduce rte_eth_dev_priority_flow_ctrl_queue_info_get() to
> avoid updates to rte_eth_dev_info
>
> - Removed devtools/libabigail.abignore changes
> - Address the comment from Ferruh in
> http://patches.dpdk.org/project/dpdk/patch/20220113102718.3167282-1-jerinj@marvell.com/
>
>
>  doc/guides/nics/features.rst           |   7 +-
>  doc/guides/rel_notes/release_22_03.rst |   5 +
>  lib/ethdev/ethdev_driver.h             |  11 ++
>  lib/ethdev/rte_ethdev.c                | 139 +++++++++++++++++++++++++
>  lib/ethdev/rte_ethdev.h                | 100 ++++++++++++++++++
>  lib/ethdev/version.map                 |   4 +
>  6 files changed, 264 insertions(+), 2 deletions(-)
>
::snip::
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + *
> + * A structure used to configure Ethernet priority flow control parameter for
s/parameter/parameters

> + * ethdev queues.
> + *
> + * rte_eth_pfc_queue_conf::rx_pause structure shall used to configure given
"shall be used"

> + * tx_qid with corresponding tc. When ethdev device receives PFC frame with
> + * rte_eth_pfc_queue_conf::rx_pause::tc, traffic will be paused on
> + * rte_eth_pfc_queue_conf::rx_pause::tx_qid for that tc.
> + *
> + * rte_eth_pfc_queue_conf::tx_pause structure shall used to configure given
"shall be used"

> + * rx_qid. When rx_qid is congested, PFC frames are generated with
> + * rte_eth_pfc_queue_conf::rx_pause::tc and
> + * rte_eth_pfc_queue_conf::rx_pause::pause_time to the peer.
> + */
> +struct rte_eth_pfc_queue_conf {
> +       enum rte_eth_fc_mode mode; /**< Link flow control mode */
> +
> +       struct {
> +               uint16_t tx_qid; /**< Tx queue ID */
> +               uint8_t tc;
> +               /**< Traffic class as per PFC (802.1Qbb) spec. The value must be
> +                * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
> +                */
> +       } rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
> +
> +       struct {
> +               uint16_t pause_time; /**< Pause quota in the Pause frame */
> +               uint16_t rx_qid;     /**< Rx queue ID */
> +               uint8_t tc;
> +               /**< Traffic class as per PFC (802.1Qbb) spec. The value must be
> +                * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
> +                */
> +       } tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
::snip::

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

[dpdk-dev] [PATCH v5 1/2] ethdev: support queue-based priority flow control

[jerinj]

From: Jerin Jacob <jerinj@marvell.com>

Based on device support and use-case need, there are two different ways
to enable PFC. The first case is the port level PFC configuration, in
this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
configure the PFC, and PFC frames will be generated using based on VLAN
TC value.

The second case is the queue level PFC configuration, in this
case, Any packet field content can be used to steer the packet to the
specific queue using rte_flow or RSS and then use
rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
TC mapping on each queue.
Based on congestion selected on the specific queue, configured TC
shall be used to generate PFC frames.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---

v5..v4:

- Address style issues(Ferruh)
http://patches.dpdk.org/project/dpdk/patch/20220207135511.3012285-1-jerinj@marvell.com/
- Address doxgen comment issues(Ajit)
http://patches.dpdk.org/project/dpdk/patch/20220207135511.3012285-1-jerinj@marvell.com/

v4..v3:

- Remove RTE_ETH_PFC_QUEUE_CAPA_* and replace with enum rte_eth_fc_mode mode_capa
- More documentaion
- Address the comment from Ferruh in
http://patches.dpdk.org/project/dpdk/patch/20220131180859.2662034-1-jerinj@marvell.com/

v3..v1:

- Introduce rte_eth_dev_priority_flow_ctrl_queue_info_get() to
avoid updates to rte_eth_dev_info

- Removed devtools/libabigail.abignore changes
- Address the comment from Ferruh in
http://patches.dpdk.org/project/dpdk/patch/20220113102718.3167282-1-jerinj@marvell.com/

 doc/guides/nics/features.rst           |   7 +-
 doc/guides/rel_notes/release_22_03.rst |   5 +
 lib/ethdev/ethdev_driver.h             |  11 ++
 lib/ethdev/rte_ethdev.c                | 143 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h                | 102 ++++++++++++++++++
 lib/ethdev/version.map                 |   4 +
 6 files changed, 270 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 27be2d2576..1cacdc883a 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -379,9 +379,12 @@ Flow control
 Supports configuring link flow control.
 
 * **[implements] eth_dev_ops**: ``flow_ctrl_get``, ``flow_ctrl_set``,
-  ``priority_flow_ctrl_set``.
+  ``priority_flow_ctrl_set``, ``priority_flow_ctrl_queue_info_get``,
+  ``priority_flow_ctrl_queue_configure``
 * **[related]    API**: ``rte_eth_dev_flow_ctrl_get()``, ``rte_eth_dev_flow_ctrl_set()``,
-  ``rte_eth_dev_priority_flow_ctrl_set()``.
+  ``rte_eth_dev_priority_flow_ctrl_set()``,
+  ``rte_eth_dev_priority_flow_ctrl_queue_info_get()``,
+  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``.
 
 
 .. _nic_features_rate_limitation:
diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
index 746f50e84f..2ec6f53efe 100644
--- a/doc/guides/rel_notes/release_22_03.rst
+++ b/doc/guides/rel_notes/release_22_03.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added an API to enable queue based priority flow ctrl(PFC).**
+
+  New APIs, ``rte_eth_dev_priority_flow_ctrl_queue_info_get()`` and
+  ``rte_eth_dev_priority_flow_ctrl_queue_configure()``, was added.
+
 * **Updated Cisco enic driver.**
 
   * Added rte_flow support for matching GENEVE packets.
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index d95605a355..d4d396468c 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -533,6 +533,13 @@ typedef int (*flow_ctrl_set_t)(struct rte_eth_dev *dev,
 typedef int (*priority_flow_ctrl_set_t)(struct rte_eth_dev *dev,
 				struct rte_eth_pfc_conf *pfc_conf);
 
+/** @internal Get info for queue based PFC on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_info_get_t)(struct rte_eth_dev *dev,
+			struct rte_eth_pfc_queue_info *pfc_queue_info);
+/** @internal Configure queue based PFC parameter on an Ethernet device. */
+typedef int (*priority_flow_ctrl_queue_config_t)(struct rte_eth_dev *dev,
+			struct rte_eth_pfc_queue_conf *pfc_queue_conf);
+
 /** @internal Update RSS redirection table on an Ethernet device. */
 typedef int (*reta_update_t)(struct rte_eth_dev *dev,
 			     struct rte_eth_rss_reta_entry64 *reta_conf,
@@ -1080,6 +1087,10 @@ struct eth_dev_ops {
 	flow_ctrl_set_t            flow_ctrl_set; /**< Setup flow control */
 	/** Setup priority flow control */
 	priority_flow_ctrl_set_t   priority_flow_ctrl_set;
+	/** Priority flow control queue info get */
+	priority_flow_ctrl_queue_info_get_t priority_flow_ctrl_queue_info_get;
+	/** Priority flow control queue configure */
+	priority_flow_ctrl_queue_config_t priority_flow_ctrl_queue_config;
 
 	/** Set Unicast Table Array */
 	eth_uc_hash_table_set_t    uc_hash_table_set;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 29e21ad580..34d2b61097 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -4022,6 +4022,149 @@ rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 	return -ENOTSUP;
 }
 
+static int
+validate_rx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
+		struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) ||
+			(pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->rx_pause.tx_qid >= dev_info->nb_tx_queues) {
+			RTE_ETHDEV_LOG(ERR,
+				"PFC Tx queue not in range for Rx pause requested:%d configured:%d\n",
+				pfc_queue_conf->rx_pause.tx_qid,
+				dev_info->nb_tx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->rx_pause.tc >= tc_max) {
+			RTE_ETHDEV_LOG(ERR,
+				"PFC TC not in range for Rx pause requested:%d max:%d\n",
+				pfc_queue_conf->rx_pause.tc, tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+validate_tx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
+		struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	if ((pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) ||
+			(pfc_queue_conf->mode == RTE_ETH_FC_FULL)) {
+		if (pfc_queue_conf->tx_pause.rx_qid >= dev_info->nb_rx_queues) {
+			RTE_ETHDEV_LOG(ERR,
+				"PFC Rx queue not in range for Tx pause requested:%d configured:%d\n",
+				pfc_queue_conf->tx_pause.rx_qid,
+				dev_info->nb_rx_queues);
+			return -EINVAL;
+		}
+
+		if (pfc_queue_conf->tx_pause.tc >= tc_max) {
+			RTE_ETHDEV_LOG(ERR,
+				"PFC TC not in range for Tx pause requested:%d max:%d\n",
+				pfc_queue_conf->tx_pause.tc, tc_max);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
+		struct rte_eth_pfc_queue_info *pfc_queue_info)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_info == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC info param is NULL for port (%u)\n",
+			port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
+		return eth_err(port_id, (*dev->dev_ops->priority_flow_ctrl_queue_info_get)
+			(dev, pfc_queue_info));
+	return -ENOTSUP;
+}
+
+int
+rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
+		struct rte_eth_pfc_queue_conf *pfc_queue_conf)
+{
+	struct rte_eth_pfc_queue_info pfc_info;
+	struct rte_eth_dev_info dev_info;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (pfc_queue_conf == NULL) {
+		RTE_ETHDEV_LOG(ERR, "PFC parameters are NULL for port (%u)\n",
+			port_id);
+		return -EINVAL;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret != 0)
+		return ret;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_info_get(port_id, &pfc_info);
+	if (ret != 0)
+		return ret;
+
+	if (pfc_info.tc_max == 0) {
+		RTE_ETHDEV_LOG(ERR, "Ethdev port %u does not support PFC TC values\n",
+			port_id);
+		return -ENOTSUP;
+	}
+
+	/* Check requested mode supported or not */
+	if (pfc_info.mode_capa == RTE_ETH_FC_RX_PAUSE &&
+			pfc_queue_conf->mode == RTE_ETH_FC_TX_PAUSE) {
+		RTE_ETHDEV_LOG(ERR, "PFC Tx pause unsupported for port (%d)\n",
+			port_id);
+		return -EINVAL;
+	}
+
+	if (pfc_info.mode_capa == RTE_ETH_FC_TX_PAUSE &&
+			pfc_queue_conf->mode == RTE_ETH_FC_RX_PAUSE) {
+		RTE_ETHDEV_LOG(ERR, "PFC Rx pause unsupported for port (%d)\n",
+			port_id);
+		return -EINVAL;
+	}
+
+	/* Validate Rx pause parameters */
+	if (pfc_info.mode_capa == RTE_ETH_FC_FULL ||
+			pfc_info.mode_capa == RTE_ETH_FC_RX_PAUSE) {
+		ret = validate_rx_pause_config(&dev_info, pfc_info.tc_max,
+				pfc_queue_conf);
+		if (ret != 0)
+			return ret;
+	}
+
+	/* Validate Tx pause parameters */
+	if (pfc_info.mode_capa == RTE_ETH_FC_FULL ||
+			pfc_info.mode_capa == RTE_ETH_FC_TX_PAUSE) {
+		ret = validate_tx_pause_config(&dev_info, pfc_info.tc_max,
+				pfc_queue_conf);
+		if (ret != 0)
+			return ret;
+	}
+
+	if (*dev->dev_ops->priority_flow_ctrl_queue_config)
+		return eth_err(port_id,
+			       (*dev->dev_ops->priority_flow_ctrl_queue_config)(
+				dev, pfc_queue_conf));
+	return -ENOTSUP;
+}
+
 static int
 eth_check_reta_mask(struct rte_eth_rss_reta_entry64 *reta_conf,
 			uint16_t reta_size)
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 147cc1ced3..2660e4f374 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1408,6 +1408,59 @@ struct rte_eth_pfc_conf {
 	uint8_t priority;          /**< VLAN User Priority. */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to retrieve information of queue based PFC.
+ */
+struct rte_eth_pfc_queue_info {
+	/**
+	 * Maximum supported traffic class as per PFC (802.1Qbb) specification.
+	 */
+	uint8_t tc_max;
+	/** PFC queue mode capabilities. */
+	enum rte_eth_fc_mode mode_capa;
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * A structure used to configure Ethernet priority flow control parameters for
+ * ethdev queues.
+ *
+ * rte_eth_pfc_queue_conf::rx_pause structure shall be used to configure given
+ * tx_qid with corresponding tc. When ethdev device receives PFC frame with
+ * rte_eth_pfc_queue_conf::rx_pause::tc, traffic will be paused on
+ * rte_eth_pfc_queue_conf::rx_pause::tx_qid for that tc.
+ *
+ * rte_eth_pfc_queue_conf::tx_pause structure shall be used to configure given
+ * rx_qid. When rx_qid is congested, PFC frames are generated with
+ * rte_eth_pfc_queue_conf::rx_pause::tc and
+ * rte_eth_pfc_queue_conf::rx_pause::pause_time to the peer.
+ */
+struct rte_eth_pfc_queue_conf {
+	enum rte_eth_fc_mode mode; /**< Link flow control mode */
+
+	struct {
+		uint16_t tx_qid; /**< Tx queue ID */
+		/** Traffic class as per PFC (802.1Qbb) spec. The value must be
+		 * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
+		 */
+		uint8_t tc;
+	} rx_pause; /* Valid when (mode == FC_RX_PAUSE || mode == FC_FULL) */
+
+	struct {
+		uint16_t pause_time; /**< Pause quota in the Pause frame */
+		uint16_t rx_qid;     /**< Rx queue ID */
+		/** Traffic class as per PFC (802.1Qbb) spec. The value must be
+		 * in the range [0, rte_eth_pfc_queue_info::tx_max - 1]
+		 */
+		uint8_t tc;
+	} tx_pause; /* Valid when (mode == FC_TX_PAUSE || mode == FC_FULL) */
+};
+
 /**
  * Tunnel type for device-specific classifier configuration.
  * @see rte_eth_udp_tunnel
@@ -4158,6 +4211,55 @@ int rte_eth_dev_priority_flow_ctrl_set(uint16_t port_id,
 int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
 				uint32_t pool);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Retrieve the information for queue based PFC.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_info
+ *   A pointer to a structure of type *rte_eth_pfc_queue_info* to be filled with
+ *   the information about queue based PFC.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if support for priority_flow_ctrl_queue_info_get does not exist.
+ *   - (-ENODEV) if *port_id* invalid.
+ *   - (-EINVAL) if bad parameter.
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
+		struct rte_eth_pfc_queue_info *pfc_queue_info);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Configure the queue based priority flow control for a given queue
+ * for Ethernet device.
+ *
+ * @note When an ethdev port switches to queue based PFC mode, the
+ * unconfigured queues shall be configured by the driver with
+ * default values such as lower priority value for TC etc.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param pfc_queue_conf
+ *   The pointer to the structure of the priority flow control parameters
+ *   for the queue.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENOTSUP) if hardware doesn't support queue based PFC mode.
+ *   - (-ENODEV)  if *port_id* invalid.
+ *   - (-EINVAL)  if bad parameter
+ *   - (-EIO)     if flow control setup queue failure
+ */
+__rte_experimental
+int rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
+		struct rte_eth_pfc_queue_conf *pfc_queue_conf);
+
+
 /**
  * Remove a MAC address from the internal array of addresses.
  *
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index c2fb0669a4..49523ebc45 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -256,6 +256,10 @@ EXPERIMENTAL {
 	rte_flow_flex_item_create;
 	rte_flow_flex_item_release;
 	rte_flow_pick_transfer_proxy;
+
+	# added in 22.03
+	rte_eth_dev_priority_flow_ctrl_queue_configure;
+	rte_eth_dev_priority_flow_ctrl_queue_info_get;
 };
 
 INTERNAL {
-- 
2.35.1

[dpdk-dev] [PATCH v5 2/2] app/testpmd: add queue based pfc CLI options

[jerinj]

From: Sunil Kumar Kori <skori@marvell.com>

Patch adds command line options to configure queue based
priority flow control.

- Syntax command is given as below:

set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> \
	tx <on|off> <rx_qid> <rx_tc> <pause_time>

- Example command to configure queue based priority flow control
  on rx and tx side for port 0, Rx queue 0, Tx queue 0 with pause
  time 2047

testpmd> set pfc_queue_ctrl 0 rx on 0 0 tx on 0 0 2047

Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
v5..v4:
- No change
v4..v3:
- fix documentation to explain CLI syntax.
v3..v1:
- Sync up the implementation to use new APIs

 app/test-pmd/cmdline.c                      | 122 ++++++++++++++++++++
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  23 ++++
 2 files changed, 145 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index e626b1c7d9..1af0321af0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -544,6 +544,11 @@ static void cmd_help_long_parsed(void *parsed_result,
 			"    Set the priority flow control parameter on a"
 			" port.\n\n"
 
+			"set pfc_queue_ctrl (port_id) rx (on|off) (tx_qid)"
+			" (tx_tc) tx (on|off) (rx_qid) (rx_tc) (pause_time)\n"
+			"    Set the queue priority flow control parameter on a"
+			" given Rx and Tx queues of a port.\n\n"
+
 			"set stat_qmap (tx|rx) (port_id) (queue_id) (qmapping)\n"
 			"    Set statistics mapping (qmapping 0..15) for RX/TX"
 			" queue on port.\n"
@@ -7690,6 +7695,122 @@ cmdline_parse_inst_t cmd_priority_flow_control_set = {
 	},
 };
 
+struct cmd_queue_priority_flow_ctrl_set_result {
+	cmdline_fixed_string_t set;
+	cmdline_fixed_string_t pfc_queue_ctrl;
+	portid_t port_id;
+	cmdline_fixed_string_t rx;
+	cmdline_fixed_string_t rx_pfc_mode;
+	uint16_t tx_qid;
+	uint8_t  tx_tc;
+	cmdline_fixed_string_t tx;
+	cmdline_fixed_string_t tx_pfc_mode;
+	uint16_t rx_qid;
+	uint8_t  rx_tc;
+	uint16_t pause_time;
+};
+
+static void
+cmd_queue_priority_flow_ctrl_set_parsed(void *parsed_result,
+					__rte_unused struct cmdline *cl,
+					__rte_unused void *data)
+{
+	struct cmd_queue_priority_flow_ctrl_set_result *res = parsed_result;
+	struct rte_eth_pfc_queue_conf pfc_queue_conf;
+	int rx_fc_enable, tx_fc_enable;
+	int ret;
+
+	/*
+	 * Rx on/off, flow control is enabled/disabled on RX side. This can
+	 * indicate the RTE_ETH_FC_TX_PAUSE, Transmit pause frame at the Rx
+	 * side. Tx on/off, flow control is enabled/disabled on TX side. This
+	 * can indicate the RTE_ETH_FC_RX_PAUSE, Respond to the pause frame at
+	 * the Tx side.
+	 */
+	static enum rte_eth_fc_mode rx_tx_onoff_2_mode[2][2] = {
+		{RTE_ETH_FC_NONE, RTE_ETH_FC_TX_PAUSE},
+		{RTE_ETH_FC_RX_PAUSE, RTE_ETH_FC_FULL}
+	};
+
+	memset(&pfc_queue_conf, 0, sizeof(struct rte_eth_pfc_queue_conf));
+	rx_fc_enable = (!strncmp(res->rx_pfc_mode, "on", 2)) ? 1 : 0;
+	tx_fc_enable = (!strncmp(res->tx_pfc_mode, "on", 2)) ? 1 : 0;
+	pfc_queue_conf.mode = rx_tx_onoff_2_mode[rx_fc_enable][tx_fc_enable];
+	pfc_queue_conf.rx_pause.tc  = res->tx_tc;
+	pfc_queue_conf.rx_pause.tx_qid = res->tx_qid;
+	pfc_queue_conf.tx_pause.tc  = res->rx_tc;
+	pfc_queue_conf.tx_pause.rx_qid  = res->rx_qid;
+	pfc_queue_conf.tx_pause.pause_time = res->pause_time;
+
+	ret = rte_eth_dev_priority_flow_ctrl_queue_configure(res->port_id,
+							     &pfc_queue_conf);
+	if (ret != 0) {
+		fprintf(stderr,
+			"bad queue priority flow control parameter, rc = %d\n",
+			ret);
+	}
+}
+
+cmdline_parse_token_string_t cmd_q_pfc_set_set =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				set, "set");
+cmdline_parse_token_string_t cmd_q_pfc_set_flow_ctrl =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pfc_queue_ctrl, "pfc_queue_ctrl");
+cmdline_parse_token_num_t cmd_q_pfc_set_portid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				port_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_q_pfc_set_rx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx, "rx");
+cmdline_parse_token_string_t cmd_q_pfc_set_rx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_tx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_tc, RTE_UINT8);
+cmdline_parse_token_string_t cmd_q_pfc_set_tx =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx, "tx");
+cmdline_parse_token_string_t cmd_q_pfc_set_tx_mode =
+	TOKEN_STRING_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				tx_pfc_mode, "on#off");
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_qid =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_qid, RTE_UINT16);
+cmdline_parse_token_num_t cmd_q_pfc_set_rx_tc =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				rx_tc, RTE_UINT8);
+cmdline_parse_token_num_t cmd_q_pfc_set_pause_time =
+	TOKEN_NUM_INITIALIZER(struct cmd_queue_priority_flow_ctrl_set_result,
+				pause_time, RTE_UINT16);
+
+cmdline_parse_inst_t cmd_queue_priority_flow_control_set = {
+	.f = cmd_queue_priority_flow_ctrl_set_parsed,
+	.data = NULL,
+	.help_str = "set pfc_queue_ctrl <port_id> rx <on|off> <tx_qid> <tx_tc> "
+		"tx <on|off> <rx_qid> <rx_tc> <pause_time>: "
+		"Configure the Ethernet queue priority flow control",
+	.tokens = {
+		(void *)&cmd_q_pfc_set_set,
+		(void *)&cmd_q_pfc_set_flow_ctrl,
+		(void *)&cmd_q_pfc_set_portid,
+		(void *)&cmd_q_pfc_set_rx,
+		(void *)&cmd_q_pfc_set_rx_mode,
+		(void *)&cmd_q_pfc_set_tx_qid,
+		(void *)&cmd_q_pfc_set_tx_tc,
+		(void *)&cmd_q_pfc_set_tx,
+		(void *)&cmd_q_pfc_set_tx_mode,
+		(void *)&cmd_q_pfc_set_rx_qid,
+		(void *)&cmd_q_pfc_set_rx_tc,
+		(void *)&cmd_q_pfc_set_pause_time,
+		NULL,
+	},
+};
+
 /* *** RESET CONFIGURATION *** */
 struct cmd_reset_result {
 	cmdline_fixed_string_t reset;
@@ -17765,6 +17886,7 @@ cmdline_parse_ctx_t main_ctx[] = {
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_set_autoneg,
 	(cmdline_parse_inst_t *)&cmd_link_flow_control_show,
 	(cmdline_parse_inst_t *)&cmd_priority_flow_control_set,
+	(cmdline_parse_inst_t *)&cmd_queue_priority_flow_control_set,
 	(cmdline_parse_inst_t *)&cmd_config_dcb,
 	(cmdline_parse_inst_t *)&cmd_read_reg,
 	(cmdline_parse_inst_t *)&cmd_read_reg_bit_field,
diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
index 94792d88cc..44b582a486 100644
--- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst
+++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst
@@ -1561,6 +1561,29 @@ Where:
 
 * ``priority`` (0-7): VLAN User Priority.
 
+set pfc_queue_ctrl
+~~~~~~~~~~~~~~~~~~
+
+Set the priority flow control parameter on a given Rx and Tx queue of a port::
+
+   testpmd> set pfc_queue_ctrl <port_id> rx (on|off) <tx_qid> <tx_tc> \
+            tx (on|off) <rx_qid> <rx_tc> <pause_time>
+
+Where:
+
+* ``tx_qid`` (integer): Tx qid for which ``tx_tc`` will be applied and traffic
+  will be paused when PFC frame is received with ``tx_tc`` enabled.
+
+* ``tx_tc`` (0-15): TC for which traffic is to be paused for xmit.
+
+* ``rx_qid`` (integer): Rx qid for which threshold will be applied and PFC
+  frame will be generated with ``tx_tc`` when exceeds the threshold.
+
+* ``rx_tc`` (0-15): TC filled in PFC frame for which remote Tx is to be paused.
+
+* ``pause_time`` (integer): Pause quanta filled in the PFC frame for which
+  interval, remote Tx will be paused. Valid only if Tx pause is on.
+
 set stat_qmap
 ~~~~~~~~~~~~~
 
-- 
2.35.1

Re: [dpdk-dev] [PATCH v5 1/2] ethdev: support queue-based priority flow control

[Ferruh Yigit]

On 2/8/2022 8:50 AM, jerinj@marvell.com wrote:
> From: Jerin Jacob <jerinj@marvell.com>
> 
> Based on device support and use-case need, there are two different ways
> to enable PFC. The first case is the port level PFC configuration, in
> this case, rte_eth_dev_priority_flow_ctrl_set() API shall be used to
> configure the PFC, and PFC frames will be generated using based on VLAN
> TC value.
> 
> The second case is the queue level PFC configuration, in this
> case, Any packet field content can be used to steer the packet to the
> specific queue using rte_flow or RSS and then use
> rte_eth_dev_priority_flow_ctrl_queue_configure() to configure the
> TC mapping on each queue.
> Based on congestion selected on the specific queue, configured TC
> shall be used to generate PFC frames.
> 
> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>

For series,
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>

Applied to dpdk-next-net/main, thanks.