DPDK patches and discussions
 help / color / mirror / Atom feed
From: Bing Zhao <bingz@nvidia.com>
To: thomas@monjalon.net, orika@nvidia.com, ferruh.yigit@intel.com,
	arybchenko@solarflare.com
Cc: dev@dpdk.org
Subject: [dpdk-dev] [RFC PATCH v2 3/4] ethdev: add hairpin bind APIs
Date: Sun, 13 Sep 2020 23:48:59 +0800	[thread overview]
Message-ID: <1600012140-70151-4-git-send-email-bingz@nvidia.com> (raw)
In-Reply-To: <1600012140-70151-1-git-send-email-bingz@nvidia.com>

In single port hairpin mode, all the hairpin TX and RX queues belong
to the same device. After the queues are set up properly, there is
no other dependency between the TX queue and its RX peer queue. The
binding process that connected the TX and RX queues together from
hardware level will be done automatically during the device start
procedure. Everything required for binding will be configured and
initialized before.
But in two ports hairpin mode, there will be some cross-dependences
between two different ports. Usually, the ports will be initialized
serially by the master thread but not in parallel. The earlier port
will not be able to enable the bind if the following peer port is
not configured done with HW resources. What's more, if one port is
detached / attached dynamically, it would introduce more trouble
for the hairpin binding.
To overcome these, new APIs for binding and unbinding are added.
During startup, only the hairpin TX and RX peer queues will be set
up. Nothing will be done when starting the device if the queues are
without auto bind attribute. Only after the required ports pair
started, the rte_eth_hairpin_bind() API can be called to bind the
all TX queues of the egress port to the RX queues of the peer port.
The connection between the egress and ingress ports pair will be
established.
rte_eth_hairpin_unbind() API could be used to disconnect the egress
and the peer ingress ports. This should only be called before the
device is closed if needed.

Signed-off-by: Bing Zhao <bingz@nvidia.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 100 ++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev.h        |  51 +++++++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h |  52 ++++++++++++++++++
 3 files changed, 203 insertions(+)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 066751f..8a3dc73 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -2175,6 +2175,106 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
 	return eth_err(port_id, ret);
 }
 
+int
+rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
+{
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev *rdev;
+	uint16_t p;
+	uint16_t rp;
+	int ret = 0;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(tx_port, -EINVAL);
+	dev = &rte_eth_devices[tx_port];
+	if (!dev->data->dev_started) {
+		RTE_ETHDEV_LOG(ERR, "TX port %d is not started", tx_port);
+		return -EBUSY;
+	}
+
+	if (rx_port == RTE_MAX_ETHPORTS) {
+		RTE_ETH_FOREACH_DEV(p) {
+			rdev = &rte_eth_devices[p];
+			if (!rdev->data->dev_started) {
+				RTE_ETHDEV_LOG(ERR,
+					       "RX port %d is not started", p);
+				ret = -EBUSY;
+				goto error;
+			}
+			ret = (*dev->dev_ops->hairpin_bind)(dev, p);
+			if (ret) {
+				RTE_ETHDEV_LOG(ERR, "Failed to bind hairpin TX "
+					       "%d to RX %d", tx_port, p);
+				goto error;
+			}
+		}
+	} else {
+		RTE_ETH_VALID_PORTID_OR_ERR_RET(rx_port, -EINVAL);
+		rdev = &rte_eth_devices[rx_port];
+		if (!rdev->data->dev_started) {
+			RTE_ETHDEV_LOG(ERR,
+				       "RX port %d is not started", rx_port);
+			return -EBUSY;
+		}
+		ret = (*dev->dev_ops->hairpin_bind)(dev, rx_port);
+		if (ret)
+			RTE_ETHDEV_LOG(ERR, "Failed to bind hairpin TX %d "
+				       "to RX %d", tx_port, rx_port);
+	}
+
+	return ret;
+
+error:
+	RTE_ETH_FOREACH_DEV(rp) {
+		if (rp < p)
+			(*dev->dev_ops->hairpin_unbind)(dev, rp);
+	}
+	return ret;
+}
+
+int
+rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port)
+{
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev *rdev;
+	uint16_t p;
+	uint16_t rp;
+	int ret = 0;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(tx_port, -EINVAL);
+	dev = &rte_eth_devices[tx_port];
+	if (!dev->data->dev_started) {
+		RTE_ETHDEV_LOG(ERR, "TX port %d is stopped", tx_port);
+		return -EBUSY;
+	}
+
+	if (rx_port == RTE_MAX_ETHPORTS) {
+		RTE_ETH_FOREACH_DEV(p) {
+			rdev = &rte_eth_devices[p];
+			if (!rdev->data->dev_started) {
+				RTE_ETHDEV_LOG(ERR, "RX port %d is stopped", p);
+				ret = -EBUSY;
+				break;
+			}
+			ret = (*dev->dev_ops->hairpin_unbind)(dev, p);
+			if (ret) {
+				RTE_ETHDEV_LOG(ERR, "Failed to unbind hairpin "
+					       "TX %d from RX %d", tx_port, p);
+				break;
+			}
+		}
+	} else {
+		RTE_ETH_VALID_PORTID_OR_ERR_RET(rx_port, -EINVAL);
+		rdev = &rte_eth_devices[rx_port];
+		if (!rdev->data->dev_started) {
+			RTE_ETHDEV_LOG(ERR, "RX port %d is stopped", rx_port);
+			return -EBUSY;
+		}
+		ret = (*dev->dev_ops->hairpin_unbind)(dev, rx_port);
+	}
+
+	return ret;
+}
+
 void
 rte_eth_tx_buffer_drop_callback(struct rte_mbuf **pkts, uint16_t unsent,
 		void *userdata __rte_unused)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index e7733d8..fb217b4 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -2101,6 +2101,57 @@ int rte_eth_tx_hairpin_queue_setup
 	 const struct rte_eth_hairpin_conf *conf);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Bind all hairpin TX queues of one port to the RX queues of the peer port.
+ * Only allowed after all hairpin queues are configured properly and the
+ * devices of TX and peer RX are in started state.
+ *
+ * @param tx_port
+ *   The TX port identifier of the Ethernet device.
+ * @param rx_port
+ *   The peer RX port identifier of the Ethernet device.
+ *   RTE_MAX_ETHPORTS is allowed for traversal of all devices.
+ *   RX port ID could have the same value with TX port ID.
+ *
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EBUSY) if device is not in started state.
+ *   - (-ENOTSUP) if hardware doesn't support.
+ *   - Others detailed errors from PMD drivers.
+ */
+__rte_experimental
+int rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Unbind all hairpin TX queues of one port from the RX queues of the peer port.
+ * This should be called before closing the TX or RX devices (optional). After
+ * unbind the hairpin ports pair, it is allowed to bind them again.
+ * Changing queues configuration should be after stopping a device.
+ *
+ * @param tx_port
+ *   The TX port identifier of the Ethernet device.
+ * @param rx_port
+ *   The peer RX port identifier of the Ethernet device.
+ *   RTE_MAX_ETHPORTS is allowed for traversal of all devices.
+ *   RX port ID could have the same value with TX port ID.
+ *
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EBUSY) if device is not in started state.
+ *   - (-ENOTSUP) if hardware doesn't support.
+ *   - Others detailed errors from PMD drivers.
+ */
+__rte_experimental
+int rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port);
+
+/**
  * Return the NUMA socket to which an Ethernet device is connected
  *
  * @param port_id
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index 23cc1e0..b4efad6 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -575,6 +575,54 @@ typedef int (*eth_tx_hairpin_queue_setup_t)
 	 const struct rte_eth_hairpin_conf *hairpin_conf);
 
 /**
+ * @internal
+ * Bind all hairpin TX queues of one port to the RX queues of the peer port.
+ *
+ * @param dev
+ *   ethdev handle of port.
+ * @param rx_port
+ *   the peer RX port.
+ *
+ * @return
+ *   Negative errno value on error, 0 on success.
+ *
+ * @retval 0
+ *   Success, bind successfully.
+ * @retval -ENOTSUP
+ *   Bind API is not supported.
+ * @retval -EINVAL
+ *   One of the parameters is invalid.
+ * @retval -EBUSY
+ *   Device is not started.
+ */
+typedef int (*eth_hairpin_bind)(struct rte_eth_dev *dev,
+				uint16_t rx_port);
+
+/**
+ * @internal
+ * Unbind all hairpin TX queues of one port from the RX queues of the peer port.
+ *
+ * @param dev
+ *   ethdev handle of port.
+ * @param rx_port
+ *   the peer RX port.
+ *
+ * @return
+ *   Negative errno value on error, 0 on success.
+ *
+ * @retval 0
+ *   Success, bind successfully.
+ * @retval -ENOTSUP
+ *   Bind API is not supported.
+ * @retval -EINVAL
+ *   One of the parameters is invalid.
+ * @retval -EBUSY
+ *   Device is already stopped.
+ */
+typedef int (*eth_hairpin_unbind)(struct rte_eth_dev *dev,
+				  uint16_t rx_port);
+
+/**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
 struct eth_dev_ops {
@@ -713,6 +761,10 @@ struct eth_dev_ops {
 	/**< Set up device RX hairpin queue. */
 	eth_tx_hairpin_queue_setup_t tx_hairpin_queue_setup;
 	/**< Set up device TX hairpin queue. */
+	eth_hairpin_bind hairpin_bind;
+	/**< Bind all hairpin TX queues of device to the peer port RX queues. */
+	eth_hairpin_unbind hairpin_unbind;
+	/**< Unbind all hairpin TX queues from the peer port RX queues. */
 };
 
 /**
-- 
2.5.5


  parent reply	other threads:[~2020-09-13 15:49 UTC|newest]

Thread overview: 81+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-11  4:51 [dpdk-dev] [RFC] introduce support for hairpin between two ports Bing Zhao
2020-09-13 15:48 ` [dpdk-dev] [RFC PATCH v2 0/4] " Bing Zhao
2020-09-13 15:48   ` [dpdk-dev] [RFC PATCH v2 1/4] ethdev: add support for flow item transmit queue Bing Zhao
2020-09-13 15:48   ` [dpdk-dev] [RFC PATCH v2 2/4] testpmd: add item transmit queue in flow CLI Bing Zhao
2020-09-13 15:48   ` Bing Zhao [this message]
2020-09-13 15:49   ` [dpdk-dev] [RFC PATCH v2 4/4] ethdev: add new attributes to hairpin queues config Bing Zhao
2020-10-01  0:25   ` [dpdk-dev] [PATCH 0/4] introduce support for hairpin between two ports Bing Zhao
2020-10-01  0:25     ` [dpdk-dev] [PATCH 1/4] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-04  9:20       ` Ori Kam
2020-10-07 11:21         ` Bing Zhao
2020-10-07 11:42           ` Ori Kam
2020-10-01  0:26     ` [dpdk-dev] [PATCH 2/4] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-04  9:22       ` Ori Kam
2020-10-07 11:32         ` Bing Zhao
2020-10-01  0:26     ` [dpdk-dev] [PATCH 3/4] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-04  9:34       ` Ori Kam
2020-10-07 11:34         ` Bing Zhao
2020-10-01  0:26     ` [dpdk-dev] [PATCH 4/4] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-04  9:39       ` Ori Kam
2020-10-07 11:36         ` Bing Zhao
2020-10-04  9:45     ` [dpdk-dev] [PATCH 0/4] introduce support for hairpin between two ports Ori Kam
2020-10-08  8:51     ` [dpdk-dev] [PATCH v2 0/6] " Bing Zhao
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 1/6] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-08  9:07         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 2/6] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-08  9:23         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 3/6] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-08  9:40         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 4/6] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-08  9:44         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 5/6] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-08  9:45         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 6/6] doc: update for two ports hairpin mode Bing Zhao
2020-10-08  9:47         ` Ori Kam
2020-10-08 12:05       ` [dpdk-dev] [PATCH v3 0/6] introduce support for hairpin between two ports Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 1/6] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-14 14:35           ` Thomas Monjalon
2020-10-15  2:56             ` Bing Zhao
2020-10-15  7:31               ` Thomas Monjalon
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 2/6] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-12 21:37           ` Thomas Monjalon
2020-10-13 12:29             ` Bing Zhao
2020-10-13 12:41               ` Thomas Monjalon
2020-10-13 13:21                 ` Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 3/6] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-08 12:31           ` Ori Kam
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 4/6] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 5/6] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 6/6] doc: update for two ports hairpin mode Bing Zhao
2020-10-12 21:30           ` Thomas Monjalon
2020-10-13  1:13             ` Bing Zhao
2020-10-13  6:37               ` Thomas Monjalon
2020-10-13  6:40                 ` Bing Zhao
2020-10-13 16:19         ` [dpdk-dev] [PATCH v4 0/5] introduce support for hairpin between two ports Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 1/5] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-14 14:43             ` Thomas Monjalon
2020-10-15  2:59               ` Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 2/5] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 3/5] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-14 15:02             ` Thomas Monjalon
2020-10-15  4:03               ` Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 4/5] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 5/5] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-15  5:35     ` [dpdk-dev] [PATCH v5 0/5] introduce support for hairpin between two ports Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 1/5] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-15 10:34         ` Thomas Monjalon
2020-10-15 11:39           ` Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 2/5] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-15 10:46         ` Thomas Monjalon
2020-10-15 13:45           ` Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 3/5] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 4/5] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 5/5] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-15 13:08     ` [dpdk-dev] [PATCH v6 0/5] introduce support for hairpin between two ports Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 1/5] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 2/5] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 3/5] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 4/5] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 5/5] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-15 23:03       ` [dpdk-dev] [PATCH v6 0/5] introduce support for hairpin between two ports Ferruh Yigit
2020-10-16  1:34         ` Bing Zhao

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1600012140-70151-4-git-send-email-bingz@nvidia.com \
    --to=bingz@nvidia.com \
    --cc=arybchenko@solarflare.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=orika@nvidia.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).