Re: [dpdk-dev] [PATCH 3/4] ethdev: add APIs for hairpin queue operation

[Bing Zhao <bingz@nvidia.com>]

Hi Ori,

> -----Original Message-----
> From: Ori Kam <orika@nvidia.com>
> Sent: Sunday, October 4, 2020 5:35 PM
> To: Bing Zhao <bingz@nvidia.com>; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>; ferruh.yigit@intel.com;
> arybchenko@solarflare.com; mdr@ashroe.eu; nhorman@tuxdriver.com;
> bernard.iremonger@intel.com; beilei.xing@intel.com;
> wenzhuo.lu@intel.com
> Cc: dev@dpdk.org
> Subject: RE: [PATCH 3/4] ethdev: add APIs for hairpin queue
> operation
> 
> Hi Bing,
> 
> PSB
> Best,
> Ori
> 
> > -----Original Message-----
> > From: Bing Zhao <bingz@nvidia.com>
> > Sent: Thursday, October 1, 2020 3:26 AM
> > Subject: [PATCH 3/4] ethdev: add APIs for hairpin queue operation
> >
> > Every hairpin queue pair should be configured properly and the
> > connection between TX and RX queues should be established, before
> > hairpin function works. In single port hairpin mode, the queues of
> > each pair belong to the same device. It is easy to get the
> hardware
> > and software information of each queue and configure the hairpin
> > connection with such information. In two ports hairpin mode, it is
> not
> > easy or inappropriate to access one queue's information from
> another
> > device.
> >
> > Since hairpin is configured per queue pair, three new APIs are
> > introduced and they are internal for the PMD using.
> >
> > The peer update API helps to pass one queue's information to the
> peer
> > queue and get the peer's information back for the next step.
> > The peer bind API configures the current queue with the peer's
> > information. For each hairpin queue pair, this API may need to be
> > called twice to configure the TX, RX queues separately.
> > The peer unbind API resets the current queue configuraion and
> state to
> > disconnect it from the peer queue. Also, it may need to be called
> > twice to disconnect TX, RX queues from each other.
> >
> > Some parameter of the above APIs might not be mandatory, and it
> > depends on the PMD implementation.
> >
> > The structure of `rte_hairpin_peer_info` is only a declaration and
> the
> > actual members will be defined in each PMD when being used.
> >
> > Signed-off-by: Bing Zhao <bingz@nvidia.com>
> > ---
> >  lib/librte_ethdev/rte_ethdev.c           |  55 ++++++++++++++++
> >  lib/librte_ethdev/rte_ethdev_driver.h    | 108
> > +++++++++++++++++++++++++++++++
> >  lib/librte_ethdev/rte_ethdev_version.map |   3 +
> >  3 files changed, 166 insertions(+)
> >
> > diff --git a/lib/librte_ethdev/rte_ethdev.c
> > b/lib/librte_ethdev/rte_ethdev.c index 72f567b..4bfc26e 100644
> > --- a/lib/librte_ethdev/rte_ethdev.c
> > +++ b/lib/librte_ethdev/rte_ethdev.c
> > @@ -5515,6 +5515,61 @@ handle_port_link_status(const char *cmd
> > __rte_unused,
> >  	return 0;
> >  }
> >
> > +int
> > +rte_eth_hairpin_queue_peer_update(uint16_t peer_port, uint16_t
> > peer_queue,
> > +				  struct rte_hairpin_peer_info *cur_info,
> > +				  struct rte_hairpin_peer_info *peer_info,
> > +				  bool direction)
> > +{
> > +	struct rte_eth_dev *dev;
> > +
> > +	/* Current queue information is not mandatory. */
> > +	if (peer_info == NULL)
> > +		return -EINVAL;
> > +
> > +	/* No need to check the validity again. */
> > +	dev = &rte_eth_devices[peer_port];
> > +	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops-
> > >hairpin_queue_peer_update,
> > +				-ENOTSUP);
> > +
> > +	return (*dev->dev_ops->hairpin_queue_peer_update)(dev,
> > peer_queue,
> > +					cur_info, peer_info, direction); }
> > +
> > +int
> > +rte_eth_hairpin_queue_peer_bind(uint16_t cur_port, uint16_t
> cur_queue,
> > +				struct rte_hairpin_peer_info *peer_info,
> > +				bool direction)
> > +{
> > +	struct rte_eth_dev *dev;
> > +
> > +	if (peer_info == NULL)
> > +		return -EINVAL;
> > +
> > +	/* No need to check the validity again. */
> > +	dev = &rte_eth_devices[cur_port];
> > +	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops-
> > >hairpin_queue_peer_bind,
> > +				-ENOTSUP);
> > +
> > +	return (*dev->dev_ops->hairpin_queue_peer_bind)(dev, cur_queue,
> > +							peer_info, direction);
> > +}
> > +
> > +int
> > +rte_eth_hairpin_queue_peer_unbind(uint16_t cur_port, uint16_t
> cur_queue,
> > +				  bool direction)
> > +{
> > +	struct rte_eth_dev *dev;
> > +
> > +	/* No need to check the validity again. */
> > +	dev = &rte_eth_devices[cur_port];
> > +	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops-
> > >hairpin_queue_peer_bind,
> > +				-ENOTSUP);
> > +
> > +	return (*dev->dev_ops->hairpin_queue_peer_unbind)(dev,
> cur_queue,
> > +							  direction);
> > +}
> > +
> >  RTE_LOG_REGISTER(rte_eth_dev_logtype, lib.ethdev, INFO);
> >
> >  RTE_INIT(ethdev_init_telemetry)
> > diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> > b/lib/librte_ethdev/rte_ethdev_driver.h
> > index 910433f..d759c58 100644
> > --- a/lib/librte_ethdev/rte_ethdev_driver.h
> > +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> > @@ -21,6 +21,9 @@
> >  extern "C" {
> >  #endif
> >
> > +/**< @internal Declaration of the hairpin peer queue information
> structure.
> > */
> > +struct rte_hairpin_peer_info;
> > +
> >  /*
> >   * Definitions of all functions exported by an Ethernet driver
> through the
> >   * generic structure of type *eth_dev_ops* supplied in the
> > *rte_eth_dev* @@ -622,6 +625,21 @@ typedef int
> > (*eth_hairpin_bind_t)(struct rte_eth_dev *dev,  typedef int
> > (*eth_hairpin_unbind_t)(struct rte_eth_dev *dev,
> >  				  uint16_t rx_port);
> >
> > +typedef int (*eth_hairpin_queue_peer_update_t)
> > +	(struct rte_eth_dev *dev, uint16_t peer_queue,
> > +	 struct rte_hairpin_peer_info *current_info,
> > +	 struct rte_hairpin_peer_info *peer_info, bool direction);
> /**<
> > +@internal Update and fetch peer queue information. */
> > +
> > +typedef int (*eth_hairpin_queue_peer_bind_t)
> > +	(struct rte_eth_dev *dev, uint16_t cur_queue,
> > +	 struct rte_hairpin_peer_info *peer_info, bool direction);
> /**<
> > +@internal Bind peer queue to the current queue with fetched
> > information. */
> > +
> > +typedef int (*eth_hairpin_queue_peer_unbind_t)
> > +	(struct rte_eth_dev *dev, uint16_t cur_queue, bool direction);
> /**<
> > +@internal Unbind peer queue from the current queue. */
> > +
> >  /**
> >   * @internal A structure containing the functions exported by an
> > Ethernet driver.
> >   */
> > @@ -765,6 +783,9 @@ struct eth_dev_ops {
> >  	/**< Bind all hairpin TX queues of device to the peer port RX
> queues. */
> >  	eth_hairpin_unbind_t hairpin_unbind;
> >  	/**< Unbind all hairpin TX queues from the peer port RX queues.
> */
> > +	eth_hairpin_queue_peer_update_t hairpin_queue_peer_update;
> > +	eth_hairpin_queue_peer_bind_t hairpin_queue_peer_bind;
> > +	eth_hairpin_queue_peer_unbind_t hairpin_queue_peer_unbind;
> >  };
> >
> >  /**
> > @@ -1120,6 +1141,93 @@ __rte_internal
> >  int
> >  rte_eth_dev_destroy(struct rte_eth_dev *ethdev, ethdev_uninit_t
> > ethdev_uninit);
> >
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change, or be removed, without
> prior
> > notice
> > + *
> > + * @internal
> > + * Pass the current hairpin queue HW and/or HW information to the
> > + peer
> > queue
> 
> I don't think you need experimental tag when using internal.

OK, I will remove it.

> 
> > + * and fetch back the information of the peer queue.
> > + *
> > + * @param peer_port
> > + *  Peer port identifier of the Ethernet device.
> > + * @param peer_queue
> > + *  Peer queue index of the port.
> > + * @param cur_info
> > + *  Pointer to the current information structure.
> > + * @param peer_info
> > + *  Pointer to the peer information, output.
> > + * @param direction
> > + *  Direction to pass the information.
> > + *  true - pass TX queue information and get peer RX queue
> > +information
> > + *  false - pass RX queue information and get peer TX queue
> > +information
> > + *
> > + * @return
> > + *  Negative errno value on error, 0 on success.
> > + */
> > +__rte_internal
> > +int
> > +rte_eth_hairpin_queue_peer_update(uint16_t peer_port, uint16_t
> > peer_queue,
> > +				  struct rte_hairpin_peer_info *cur_info,
> > +				  struct rte_hairpin_peer_info *peer_info,
> > +				  bool direction);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change, or be removed, without
> prior
> > notice
> > + *
> > + * @internal
> > + * Configure current hairpin queue with the peer information
> fetched
> > + to
> > create
> 
> Same comment as above.

I will remove it.

> 
> > + * the connection (bind) with peer queue in the specified
> direction.
> > + * This function might need to be called twice to fully create
> the connection.
> > + *
> > + * @param cur_port
> > + *  Current port identifier of the Ethernet device.
> > + * @param cur_queue
> > + *  Current queue index of the port.
> > + * @param peer_info
> > + *  Pointer to the peer information, input.
> > + * @param direction
> > + *  Direction to create the connection.
> > + *  true - bind current TX queue to peer RX queue
> > + *  false - bind current RX queue to peer TX queue
> > + *
> > + * @return
> > + *  Negative errno value on error, 0 on success.
> > + */
> > +__rte_internal
> > +int
> > +rte_eth_hairpin_queue_peer_bind(uint16_t cur_port, uint16_t
> cur_queue,
> > +				struct rte_hairpin_peer_info *peer_info,
> > +				bool direction);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change, or be removed, without
> prior
> > notice
> > + *
> > + * @internal
> > + * Reset the current queue state and configuration to disconnect
> > +(unbind) it
> > + * from the peer queue.
> > + * This function might need to be called twice to disconnect each
> other.
> > + *
> > + * @param cur_port
> > + *  Current port identifier of the Ethernet device.
> > + * @param cur_queue
> > + *  Current queue index of the port.
> > + * @param direction
> > + *  Direction to create the connection.
> > + *  true - unbind current TX queue from peer RX queue
> > + *  false - unbind current RX queue from peer TX queue
> > + *
> > + * @return
> > + *  Negative errno value on error, 0 on success.
> > + */
> > +__rte_internal
> > +int
> > +rte_eth_hairpin_queue_peer_unbind(uint16_t cur_port, uint16_t
> cur_queue,
> > +				  bool direction);
> > +
> > +
> >  #ifdef __cplusplus
> >  }
> >  #endif
> > diff --git a/lib/librte_ethdev/rte_ethdev_version.map
> > b/lib/librte_ethdev/rte_ethdev_version.map
> > index 18efe4e..d05cd97 100644
> > --- a/lib/librte_ethdev/rte_ethdev_version.map
> > +++ b/lib/librte_ethdev/rte_ethdev_version.map
> > @@ -250,6 +250,9 @@ INTERNAL {
> >  	rte_eth_devargs_parse;
> >  	rte_eth_dma_zone_free;
> >  	rte_eth_dma_zone_reserve;
> > +	rte_eth_hairpin_queue_peer_bind;
> > +	rte_eth_hairpin_queue_peer_unbind;
> > +	rte_eth_hairpin_queue_peer_update;
> >  	rte_eth_switch_domain_alloc;
> >  	rte_eth_switch_domain_free;
> >  	rte_flow_expand_rss;
> > --
> > 2.5.5

Thanks