[PATCH] ethdev: prohibit polling of a stopped queue

[PATCH] ethdev: prohibit polling of a stopped queue

[Dmitry Kozlyuk]

Whether it is allowed to call Rx/Tx functions for a stopped queue
was undocumented. Some PMDs make this behavior a no-op
either by explicitly checking the queue state
or by the way how their routines are implemented or HW works.

No-op behavior may be convenient for application developers.
But it also means that pollers of stopped queues
would go all the way down to PMD Rx/Tx routines, wasting cycles.
Some PMDs would do a check for the queue state on data path,
even though it may never be needed for a particular application.
Also, use cases for stopping queues or starting them deferred
do not logically require polling stopped queues.

Use case 1: a secondary that was polling the queue has crashed,
the primary is doing a recovery to free all mbufs.
By definition the queue to be restarted is not polled.

Use case 2: deferred queue start or queue reconfiguration.
The polling thread must be synchronized anyway,
because queue start and stop are non-atomic.

Prohibit calling Rx/Tx functions on stopped queues.

Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
This patch is was originally a part of the series:
http://patchwork.dpdk.org/project/dpdk/patch/20220307125351.697936-3-dkozlyuk@nvidia.com/
The discussion there is summarized in the commit message.

 lib/ethdev/rte_ethdev.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 04cff8ee10..435720a84e 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -74,7 +74,7 @@
  * rte_eth_rx_queue_setup()), it must call rte_eth_dev_stop() first to stop the
  * device and then do the reconfiguration before calling rte_eth_dev_start()
  * again. The transmit and receive functions should not be invoked when the
- * device is stopped.
+ * device is stopped or when the queue is stopped (for that queue).
  *
  * Please note that some configuration is not stored between calls to
  * rte_eth_dev_stop()/rte_eth_dev_start(). The following configuration will
-- 
2.25.1

Re: [PATCH] ethdev: prohibit polling of a stopped queue

[Tyler Retzlaff]

On Mon, Apr 11, 2022 at 12:35:50AM +0300, Dmitry Kozlyuk wrote:
> Whether it is allowed to call Rx/Tx functions for a stopped queue
> was undocumented. Some PMDs make this behavior a no-op
> either by explicitly checking the queue state
> or by the way how their routines are implemented or HW works.
> 
> No-op behavior may be convenient for application developers.
> But it also means that pollers of stopped queues
> would go all the way down to PMD Rx/Tx routines, wasting cycles.
> Some PMDs would do a check for the queue state on data path,
> even though it may never be needed for a particular application.
> Also, use cases for stopping queues or starting them deferred
> do not logically require polling stopped queues.
> 
> Use case 1: a secondary that was polling the queue has crashed,
> the primary is doing a recovery to free all mbufs.
> By definition the queue to be restarted is not polled.
> 
> Use case 2: deferred queue start or queue reconfiguration.
> The polling thread must be synchronized anyway,
> because queue start and stop are non-atomic.
> 
> Prohibit calling Rx/Tx functions on stopped queues.
> 
> Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>

Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

Re: [PATCH] ethdev: prohibit polling of a stopped queue

[Thomas Monjalon]

10/04/2022 23:35, Dmitry Kozlyuk:
> Whether it is allowed to call Rx/Tx functions for a stopped queue
> was undocumented. Some PMDs make this behavior a no-op
> either by explicitly checking the queue state
> or by the way how their routines are implemented or HW works.
> 
> No-op behavior may be convenient for application developers.
> But it also means that pollers of stopped queues
> would go all the way down to PMD Rx/Tx routines, wasting cycles.
> Some PMDs would do a check for the queue state on data path,
> even though it may never be needed for a particular application.
> Also, use cases for stopping queues or starting them deferred
> do not logically require polling stopped queues.
> 
> Use case 1: a secondary that was polling the queue has crashed,
> the primary is doing a recovery to free all mbufs.
> By definition the queue to be restarted is not polled.
> 
> Use case 2: deferred queue start or queue reconfiguration.
> The polling thread must be synchronized anyway,
> because queue start and stop are non-atomic.
> 
> Prohibit calling Rx/Tx functions on stopped queues.
> 
> Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---
> This patch is was originally a part of the series:
> http://patchwork.dpdk.org/project/dpdk/patch/20220307125351.697936-3-dkozlyuk@nvidia.com/
> The discussion there is summarized in the commit message.
[...]
>   * rte_eth_rx_queue_setup()), it must call rte_eth_dev_stop() first to stop the
>   * device and then do the reconfiguration before calling rte_eth_dev_start()
>   * again. The transmit and receive functions should not be invoked when the
> - * device is stopped.
> + * device is stopped or when the queue is stopped (for that queue).

I think we can make it simpler:

The transmit and receive functions should not be invoked when the device
or the queue is stopped.

Re: [PATCH] ethdev: prohibit polling of a stopped queue

[Thomas Monjalon]

25/04/2022 10:30, Thomas Monjalon:
> 10/04/2022 23:35, Dmitry Kozlyuk:
> > Whether it is allowed to call Rx/Tx functions for a stopped queue
> > was undocumented. Some PMDs make this behavior a no-op
> > either by explicitly checking the queue state
> > or by the way how their routines are implemented or HW works.
> > 
> > No-op behavior may be convenient for application developers.
> > But it also means that pollers of stopped queues
> > would go all the way down to PMD Rx/Tx routines, wasting cycles.
> > Some PMDs would do a check for the queue state on data path,
> > even though it may never be needed for a particular application.
> > Also, use cases for stopping queues or starting them deferred
> > do not logically require polling stopped queues.
> > 
> > Use case 1: a secondary that was polling the queue has crashed,
> > the primary is doing a recovery to free all mbufs.
> > By definition the queue to be restarted is not polled.
> > 
> > Use case 2: deferred queue start or queue reconfiguration.
> > The polling thread must be synchronized anyway,
> > because queue start and stop are non-atomic.
> > 
> > Prohibit calling Rx/Tx functions on stopped queues.
> > 
> > Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
> > Cc: stable@dpdk.org
> > 
> > Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> > ---
> > This patch is was originally a part of the series:
> > http://patchwork.dpdk.org/project/dpdk/patch/20220307125351.697936-3-dkozlyuk@nvidia.com/
> > The discussion there is summarized in the commit message.
> [...]
> >   * rte_eth_rx_queue_setup()), it must call rte_eth_dev_stop() first to stop the
> >   * device and then do the reconfiguration before calling rte_eth_dev_start()
> >   * again. The transmit and receive functions should not be invoked when the
> > - * device is stopped.
> > + * device is stopped or when the queue is stopped (for that queue).
> 
> I think we can make it simpler:
> 
> The transmit and receive functions should not be invoked when the device
> or the queue is stopped.

No comment after a month.
The patch is applied in next-net with the suggested rewording.