DPDK patches and discussions
 help / color / mirror / Atom feed
From: Ori Kam <orika@nvidia.com>
To: Bing Zhao <bingz@nvidia.com>,
	NBU-Contact-Thomas Monjalon <thomas@monjalon.net>,
	"ferruh.yigit@intel.com" <ferruh.yigit@intel.com>,
	"arybchenko@solarflare.com" <arybchenko@solarflare.com>,
	"mdr@ashroe.eu" <mdr@ashroe.eu>,
	"nhorman@tuxdriver.com" <nhorman@tuxdriver.com>,
	"bernard.iremonger@intel.com" <bernard.iremonger@intel.com>,
	"beilei.xing@intel.com" <beilei.xing@intel.com>,
	"wenzhuo.lu@intel.com" <wenzhuo.lu@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH 1/4] ethdev: add hairpin bind and unbind APIs
Date: Sun, 4 Oct 2020 09:20:10 +0000	[thread overview]
Message-ID: <MN2PR12MB428657D6959A2F3B0719D58CD60F0@MN2PR12MB4286.namprd12.prod.outlook.com> (raw)
In-Reply-To: <1601511962-21532-2-git-send-email-bingz@nvidia.com>

Hi Bing,

PSB,

Thanks,
Ori
> -----Original Message-----
> From: Bing Zhao <bingz@nvidia.com>
> Sent: Thursday, October 1, 2020 3:26 AM
> Cc: dev@dpdk.org
> Subject: [PATCH 1/4] ethdev: add hairpin bind and unbind APIs
> 
> In single port hairpin mode, all the hairpin TX and RX queues belong
> to the same device. After the queues are set up properly, there is
> no other dependency between the TX queue and its RX peer queue. The
> binding process that connected the TX and RX queues together from
> hardware level will be done automatically during the device start
> procedure. Everything required is configured and initialized already
> for the binding process.
> 
> But in two ports hairpin mode, there will be some cross-dependences
> between two different ports. Usually, the ports will be initialized
> serially by the main thread but not in parallel. The earlier port
> will not be able to enable the bind if the following peer port is
> not yet configured with HW resources. What's more, if one port is
> detached / attached dynamically, it would introduce more trouble
> for the hairpin binding.
> 
> To overcome these, new APIs for binding and unbinding are added.
> During startup, only the hairpin TX and RX peer queues will be set
> up. Nothing will be done when starting the device if the queues are
> without auto-bind attribute. Only after the required ports pair
> started, the `rte_eth_hairpin_bind()` API can be called to bind the
> all TX queues of the egress port to the RX queues of the peer port.
> Then the connection between the egress and ingress ports pair will
> be established.
> 
> The `rte_eth_hairpin_unbind()` API could be used to disconnect the
> egress and the peer ingress ports. This should only be called before
> the device is closed if needed. When doing the clean up, all the
> egress and ingress pairs related to a single port should be taken
> into consideration.
> 
> Signed-off-by: Bing Zhao <bingz@nvidia.com>
> ---
>  lib/librte_ethdev/rte_ethdev.c           | 107
> +++++++++++++++++++++++++++++++
>  lib/librte_ethdev/rte_ethdev.h           |  51 +++++++++++++++
>  lib/librte_ethdev/rte_ethdev_driver.h    |  52 +++++++++++++++
>  lib/librte_ethdev/rte_ethdev_version.map |   2 +
>  4 files changed, 212 insertions(+)
> 
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index dfe5c1b..72f567b 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c
> @@ -2175,6 +2175,113 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id,
> uint16_t tx_queue_id,
>  	return eth_err(port_id, ret);
>  }
> 
> +int
> +rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
> +{
> +	struct rte_eth_dev *dev;
> +	struct rte_eth_dev *rdev;
> +	uint16_t p;
> +	uint16_t rp;
> +	int ret = 0;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(tx_port, -EINVAL);
> +	dev = &rte_eth_devices[tx_port];
> +	if (!dev->data->dev_started) {
> +		RTE_ETHDEV_LOG(ERR, "TX port %d is not started", tx_port);
> +		return -EBUSY;
> +	}
> +
> +	/*
> +	 * If the all the ports probed belong to two or more separate NICs, it
> +	 * is recommended that each pair is bound independently but not in the
> +	 * loop to bind all ports.
> +	 */

I don't understand your comment. 

> +	if (rx_port == RTE_MAX_ETHPORTS) {

I think maybe this should be done in the tx queue. Since if the bind don't need some port why do
we care if it is started?
So either add a new function to get all peer ports from the tx port, or move this logic to the 
Target PMD.

> +		RTE_ETH_FOREACH_DEV(p) {
> +			rdev = &rte_eth_devices[p];
> +			if (!rdev->data->dev_started) {
> +				RTE_ETHDEV_LOG(ERR,
> +					       "RX port %d is not started", p);
> +				ret = -EBUSY;
> +				goto unbind;
> +			}
> +			ret = (*dev->dev_ops->hairpin_bind)(dev, p);
> +			if (ret) {
> +				RTE_ETHDEV_LOG(ERR, "Failed to bind hairpin
> TX "
> +					       "%d to RX %d", tx_port, p);
> +				goto unbind;
> +			}
> +		}
> +	} else {
> +		RTE_ETH_VALID_PORTID_OR_ERR_RET(rx_port, -EINVAL);
> +		rdev = &rte_eth_devices[rx_port];
> +		if (!rdev->data->dev_started) {
> +			RTE_ETHDEV_LOG(ERR,
> +				       "RX port %d is not started", rx_port);
> +			return -EBUSY;
> +		}
> +		ret = (*dev->dev_ops->hairpin_bind)(dev, rx_port);
> +		if (ret)
> +			RTE_ETHDEV_LOG(ERR, "Failed to bind hairpin TX %d "
> +				       "to RX %d", tx_port, rx_port);
> +	}
> +
> +	return ret;
> +
> +unbind:
> +	/* Roll back the previous binding process. */
> +	RTE_ETH_FOREACH_DEV(rp) {
> +		if (rp < p)
> +			(*dev->dev_ops->hairpin_unbind)(dev, rp);
> +		else
> +			break;
> +	}
> +	return ret;
> +}
> +
> +int
> +rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port)
> +{
> +	struct rte_eth_dev *dev;
> +	struct rte_eth_dev *rdev;
> +	uint16_t p;
> +	int ret = 0;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(tx_port, -EINVAL);
> +	dev = &rte_eth_devices[tx_port];
> +	if (!dev->data->dev_started) {
> +		RTE_ETHDEV_LOG(ERR, "TX port %d is stopped", tx_port);
> +		return -EBUSY;
> +	}
> +
> +	if (rx_port == RTE_MAX_ETHPORTS) {
> +		RTE_ETH_FOREACH_DEV(p) {
> +			rdev = &rte_eth_devices[p];
> +			if (!rdev->data->dev_started) {

This condition should never be true.
First see my comment above about the list of devices, second port should fail to
stop if it is bounded.

> +				RTE_ETHDEV_LOG(ERR, "RX port %d is
> stopped", p);
> +				ret = -EBUSY;
> +				break;
> +			}
> +			ret = (*dev->dev_ops->hairpin_unbind)(dev, p);
> +			if (ret) {
> +				RTE_ETHDEV_LOG(ERR, "Failed to unbind
> hairpin "
> +					       "TX %d from RX %d", tx_port, p);
> +				break;
> +			}
> +		}
> +	} else {
> +		RTE_ETH_VALID_PORTID_OR_ERR_RET(rx_port, -EINVAL);
> +		rdev = &rte_eth_devices[rx_port];
> +		if (!rdev->data->dev_started) {
> +			RTE_ETHDEV_LOG(ERR, "RX port %d is stopped",
> rx_port);
> +			return -EBUSY;
> +		}
> +		ret = (*dev->dev_ops->hairpin_unbind)(dev, rx_port);
> +	}
> +
> +	return ret;
> +}
> +
>  void
>  rte_eth_tx_buffer_drop_callback(struct rte_mbuf **pkts, uint16_t unsent,
>  		void *userdata __rte_unused)
> diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
> index 645a186..c3fb684 100644
> --- a/lib/librte_ethdev/rte_ethdev.h
> +++ b/lib/librte_ethdev/rte_ethdev.h
> @@ -2133,6 +2133,57 @@ int rte_eth_tx_hairpin_queue_setup
>  	 const struct rte_eth_hairpin_conf *conf);
> 
>  /**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior
> notice
> + *
> + * Bind all hairpin TX queues of one port to the RX queues of the peer port.
> + * It is only allowed to call this API after all hairpin queues are configured
> + * properly and the devices of TX and peer RX are in started state.
> + *
> + * @param tx_port
> + *   The TX port identifier of the Ethernet device.
> + * @param rx_port
> + *   The peer RX port identifier of the Ethernet device.
> + *   RTE_MAX_ETHPORTS is allowed for the traversal of all devices.
> + *   RX port ID could have the same value with TX port ID.
> + *
> + * @return
> + *   - (0) if successful.
> + *   - (-EINVAL) if bad parameter.
> + *   - (-EBUSY) if device is not in started state.
> + *   - (-ENOTSUP) if hardware doesn't support.
> + *   - Others detailed errors from PMD drivers.
> + */
> +__rte_experimental
> +int rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change, or be removed, without prior
> notice
> + *
> + * Unbind all hairpin TX queues of one port from the RX queues of the peer
> port.
> + * This should be called before closing the TX or RX devices (optional). After
> + * unbind the hairpin ports pair, it is allowed to bind them again.
> + * Changing queues configuration should be after stopping the device.
> + *
> + * @param tx_port
> + *   The TX port identifier of the Ethernet device.
> + * @param rx_port
> + *   The peer RX port identifier of the Ethernet device.
> + *   RTE_MAX_ETHPORTS is allowed for traversal of all devices.
> + *   RX port ID could have the same value with TX port ID.
> + *
> + * @return
> + *   - (0) if successful.
> + *   - (-EINVAL) if bad parameter.
> + *   - (-EBUSY) if device is in stopped state.
> + *   - (-ENOTSUP) if hardware doesn't support.
> + *   - Others detailed errors from PMD drivers.
> + */
> +__rte_experimental
> +int rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port);
> +
> +/**
>   * Return the NUMA socket to which an Ethernet device is connected
>   *
>   * @param port_id
> diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> b/lib/librte_ethdev/rte_ethdev_driver.h
> index 04ac8e9..910433f 100644
> --- a/lib/librte_ethdev/rte_ethdev_driver.h
> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> @@ -575,6 +575,54 @@ typedef int (*eth_tx_hairpin_queue_setup_t)
>  	 const struct rte_eth_hairpin_conf *hairpin_conf);
> 
>  /**
> + * @internal
> + * Bind all hairpin TX queues of one port to the RX queues of the peer port.
> + *
> + * @param dev
> + *   ethdev handle of port.
> + * @param rx_port
> + *   the peer RX port.
> + *
> + * @return
> + *   Negative errno value on error, 0 on success.
> + *
> + * @retval 0
> + *   Success, bind successfully.
> + * @retval -ENOTSUP
> + *   Bind API is not supported.
> + * @retval -EINVAL
> + *   One of the parameters is invalid.
> + * @retval -EBUSY
> + *   Device is not started.
> + */
> +typedef int (*eth_hairpin_bind_t)(struct rte_eth_dev *dev,
> +				uint16_t rx_port);
> +
> +/**
> + * @internal
> + * Unbind all hairpin TX queues of one port from the RX queues of the peer
> port.
> + *
> + * @param dev
> + *   ethdev handle of port.
> + * @param rx_port
> + *   the peer RX port.
> + *
> + * @return
> + *   Negative errno value on error, 0 on success.
> + *
> + * @retval 0
> + *   Success, bind successfully.
> + * @retval -ENOTSUP
> + *   Bind API is not supported.
> + * @retval -EINVAL
> + *   One of the parameters is invalid.
> + * @retval -EBUSY
> + *   Device is already stopped.
> + */
> +typedef int (*eth_hairpin_unbind_t)(struct rte_eth_dev *dev,
> +				  uint16_t rx_port);
> +
> +/**
>   * @internal A structure containing the functions exported by an Ethernet
> driver.
>   */
>  struct eth_dev_ops {
> @@ -713,6 +761,10 @@ struct eth_dev_ops {
>  	/**< Set up device RX hairpin queue. */
>  	eth_tx_hairpin_queue_setup_t tx_hairpin_queue_setup;
>  	/**< Set up device TX hairpin queue. */
> +	eth_hairpin_bind_t hairpin_bind;
> +	/**< Bind all hairpin TX queues of device to the peer port RX queues. */
> +	eth_hairpin_unbind_t hairpin_unbind;
> +	/**< Unbind all hairpin TX queues from the peer port RX queues. */
>  };
> 
>  /**
> diff --git a/lib/librte_ethdev/rte_ethdev_version.map
> b/lib/librte_ethdev/rte_ethdev_version.map
> index c95ef51..18efe4e 100644
> --- a/lib/librte_ethdev/rte_ethdev_version.map
> +++ b/lib/librte_ethdev/rte_ethdev_version.map
> @@ -227,6 +227,8 @@ EXPERIMENTAL {
>  	rte_tm_wred_profile_delete;
> 
>  	# added in 20.11
> +	rte_eth_hairpin_bind;
> +	rte_eth_hairpin_unbind;
>  	rte_eth_link_speed_to_str;
>  	rte_eth_link_to_str;
>  };
> --
> 2.5.5


  reply	other threads:[~2020-10-04  9:20 UTC|newest]

Thread overview: 81+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-11  4:51 [dpdk-dev] [RFC] introduce support for hairpin between two ports Bing Zhao
2020-09-13 15:48 ` [dpdk-dev] [RFC PATCH v2 0/4] " Bing Zhao
2020-09-13 15:48   ` [dpdk-dev] [RFC PATCH v2 1/4] ethdev: add support for flow item transmit queue Bing Zhao
2020-09-13 15:48   ` [dpdk-dev] [RFC PATCH v2 2/4] testpmd: add item transmit queue in flow CLI Bing Zhao
2020-09-13 15:48   ` [dpdk-dev] [RFC PATCH v2 3/4] ethdev: add hairpin bind APIs Bing Zhao
2020-09-13 15:49   ` [dpdk-dev] [RFC PATCH v2 4/4] ethdev: add new attributes to hairpin queues config Bing Zhao
2020-10-01  0:25   ` [dpdk-dev] [PATCH 0/4] introduce support for hairpin between two ports Bing Zhao
2020-10-01  0:25     ` [dpdk-dev] [PATCH 1/4] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-04  9:20       ` Ori Kam [this message]
2020-10-07 11:21         ` Bing Zhao
2020-10-07 11:42           ` Ori Kam
2020-10-01  0:26     ` [dpdk-dev] [PATCH 2/4] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-04  9:22       ` Ori Kam
2020-10-07 11:32         ` Bing Zhao
2020-10-01  0:26     ` [dpdk-dev] [PATCH 3/4] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-04  9:34       ` Ori Kam
2020-10-07 11:34         ` Bing Zhao
2020-10-01  0:26     ` [dpdk-dev] [PATCH 4/4] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-04  9:39       ` Ori Kam
2020-10-07 11:36         ` Bing Zhao
2020-10-04  9:45     ` [dpdk-dev] [PATCH 0/4] introduce support for hairpin between two ports Ori Kam
2020-10-08  8:51     ` [dpdk-dev] [PATCH v2 0/6] " Bing Zhao
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 1/6] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-08  9:07         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 2/6] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-08  9:23         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 3/6] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-08  9:40         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 4/6] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-08  9:44         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 5/6] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-08  9:45         ` Ori Kam
2020-10-08  8:51       ` [dpdk-dev] [PATCH v2 6/6] doc: update for two ports hairpin mode Bing Zhao
2020-10-08  9:47         ` Ori Kam
2020-10-08 12:05       ` [dpdk-dev] [PATCH v3 0/6] introduce support for hairpin between two ports Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 1/6] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-14 14:35           ` Thomas Monjalon
2020-10-15  2:56             ` Bing Zhao
2020-10-15  7:31               ` Thomas Monjalon
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 2/6] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-12 21:37           ` Thomas Monjalon
2020-10-13 12:29             ` Bing Zhao
2020-10-13 12:41               ` Thomas Monjalon
2020-10-13 13:21                 ` Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 3/6] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-08 12:31           ` Ori Kam
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 4/6] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 5/6] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-08 12:05         ` [dpdk-dev] [PATCH v3 6/6] doc: update for two ports hairpin mode Bing Zhao
2020-10-12 21:30           ` Thomas Monjalon
2020-10-13  1:13             ` Bing Zhao
2020-10-13  6:37               ` Thomas Monjalon
2020-10-13  6:40                 ` Bing Zhao
2020-10-13 16:19         ` [dpdk-dev] [PATCH v4 0/5] introduce support for hairpin between two ports Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 1/5] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-14 14:43             ` Thomas Monjalon
2020-10-15  2:59               ` Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 2/5] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 3/5] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-14 15:02             ` Thomas Monjalon
2020-10-15  4:03               ` Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 4/5] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-13 16:19           ` [dpdk-dev] [PATCH v4 5/5] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-15  5:35     ` [dpdk-dev] [PATCH v5 0/5] introduce support for hairpin between two ports Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 1/5] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-15 10:34         ` Thomas Monjalon
2020-10-15 11:39           ` Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 2/5] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-15 10:46         ` Thomas Monjalon
2020-10-15 13:45           ` Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 3/5] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 4/5] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-15  5:35       ` [dpdk-dev] [PATCH v5 5/5] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-15 13:08     ` [dpdk-dev] [PATCH v6 0/5] introduce support for hairpin between two ports Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 1/5] ethdev: add hairpin bind and unbind APIs Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 2/5] ethdev: add new attributes to hairpin config Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 3/5] ethdev: add API to get hairpin peer ports list Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 4/5] ethdev: add APIs for hairpin queue operation Bing Zhao
2020-10-15 13:08       ` [dpdk-dev] [PATCH v6 5/5] app/testpmd: change hairpin queues setup Bing Zhao
2020-10-15 23:03       ` [dpdk-dev] [PATCH v6 0/5] introduce support for hairpin between two ports Ferruh Yigit
2020-10-16  1:34         ` Bing Zhao

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=MN2PR12MB428657D6959A2F3B0719D58CD60F0@MN2PR12MB4286.namprd12.prod.outlook.com \
    --to=orika@nvidia.com \
    --cc=arybchenko@solarflare.com \
    --cc=beilei.xing@intel.com \
    --cc=bernard.iremonger@intel.com \
    --cc=bingz@nvidia.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=mdr@ashroe.eu \
    --cc=nhorman@tuxdriver.com \
    --cc=thomas@monjalon.net \
    --cc=wenzhuo.lu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).