[PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[Stephen Hemminger]

Need to warn users of DPDK spinlocks from non-pinned threads.
This is similar wording to Linux documentation in pthread_spin_init.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 5f0748fba1c0..45d3de8d84f6 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -797,6 +797,16 @@ Known Issues
 
   The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
 
++ locking
+
+  If a pthread, that is not pinned to an lcore acquires a lock such as a
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
+  then there is a possibility of large application delays.
+  The problem is that if a thread is scheduled off the CPU while it holds
+  a lock, then other threads will waste time spinning on the lock until
+  the lock holder is once more rescheduled and releases the lock.
+
+
 cgroup control
 ~~~~~~~~~~~~~~
 
-- 
2.35.1

Re: [PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[Tyler Retzlaff]

On Fri, Jun 10, 2022 at 08:28:19AM -0700, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
> 
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

> ---
>  doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>  
>    The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>  
> ++ locking
> +
> +  If a pthread, that is not pinned to an lcore acquires a lock such as a
> +  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> +  then there is a possibility of large application delays.
> +  The problem is that if a thread is scheduled off the CPU while it holds
> +  a lock, then other threads will waste time spinning on the lock until
> +  the lock holder is once more rescheduled and releases the lock.
> +
> +
>  cgroup control
>  ~~~~~~~~~~~~~~
>  
> -- 
> 2.35.1

Re: [PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[Tyler Retzlaff]

On Fri, Jun 10, 2022 at 08:28:19AM -0700, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
> 
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>  doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>  
>    The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>  
> ++ locking
> +
> +  If a pthread, that is not pinned to an lcore acquires a lock such as a

nit: suggest not using term pthread but instead just say thread as not
to imply a specific platform/implementation.

> +  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> +  then there is a possibility of large application delays.
> +  The problem is that if a thread is scheduled off the CPU while it holds
> +  a lock, then other threads will waste time spinning on the lock until

'until the lock holder' -> 'until the thread holding the lock'

but i'm not really fussed, just a suggestion.

> +  the lock holder is once more rescheduled and releases the lock.
> +
> +
>  cgroup control
>  ~~~~~~~~~~~~~~
>  
> -- 
> 2.35.1

Re: [PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[Stephen Hemminger]

On Fri, 10 Jun 2022 16:48:15 -0700
Tyler Retzlaff <roretzla@linux.microsoft.com> wrote:

> >  
> > ++ locking
> > +
> > +  If a pthread, that is not pinned to an lcore acquires a lock such as a  
> 
> nit: suggest not using term pthread but instead just say thread as not
> to imply a specific platform/implementation.
> 
> > +  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> > +  then there is a possibility of large application delays.
> > +  The problem is that if a thread is scheduled off the CPU while it holds
> > +  a lock, then other threads will waste time spinning on the lock until  
> 
> 'until the lock holder' -> 'until the thread holding the lock'
> 
> but i'm not really fussed, just a suggestion.

Sure, that wording was from existing pthread_spin_init() man page

Re: [PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[fengchengwen]

On 2022/6/10 23:28, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
> 
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>  doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>  
>    The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>  
> ++ locking
> +
> +  If a pthread, that is not pinned to an lcore acquires a lock such as a
> +  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)

Some APIs inherently use rte_spinlock, just like rte_malloc/rte_eal_alarm_set,
Because DPDK API mainly use rte_spinlock to support thread-safty.

Suggest declare DPDK API mainly use rte_spinlock to support thread-safty, so
if the caller thread is not pinned to an lcore may encount a possibility of
large application delays.

> +  then there is a possibility of large application delays.
> +  The problem is that if a thread is scheduled off the CPU while it holds
> +  a lock, then other threads will waste time spinning on the lock until
> +  the lock holder is once more rescheduled and releases the lock.
> +
> +
>  cgroup control
>  ~~~~~~~~~~~~~~
>  
> 

Re: [PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[Stephen Hemminger]

On Sat, 11 Jun 2022 09:55:00 +0800
fengchengwen <fengchengwen@huawei.com> wrote:

> On 2022/6/10 23:28, Stephen Hemminger wrote:
> > Need to warn users of DPDK spinlocks from non-pinned threads.
> > This is similar wording to Linux documentation in pthread_spin_init.
> > 
> > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > ---
> >  doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
> >  1 file changed, 10 insertions(+)
> > 
> > diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> > index 5f0748fba1c0..45d3de8d84f6 100644
> > --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> > +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> > @@ -797,6 +797,16 @@ Known Issues
> >  
> >    The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
> >  
> > ++ locking
> > +
> > +  If a pthread, that is not pinned to an lcore acquires a lock such as a
> > +  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)  
> 
> Some APIs inherently use rte_spinlock, just like rte_malloc/rte_eal_alarm_set,
> Because DPDK API mainly use rte_spinlock to support thread-safty.
> 
> Suggest declare DPDK API mainly use rte_spinlock to support thread-safty, so
> if the caller thread is not pinned to an lcore may encount a possibility of
> large application delays.

I copied text from pthread_spinlock man page. The same caveats apply to
pthread_spinlocks as DPDK; therefore using same wording seemed appropriate.

But it is worth mentioning that other API's may depend on spinlocks internally.

Re: [PATCH] doc/eal: add caveat about spinlocks from non-pinned threads

[Mattias Rönnblom]

On 2022-06-10 17:28, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
> 
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>   doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>   
>     The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>   
> ++ locking
> +

Isn't this problem more general than locks? The use of any 
non-preemption safe data structures potentially causes such delays. 
Regular DPDK rings for sure. The lock-less stack? The hash library?

Both actual and open-coded spinlocks internal to the APIs are also very 
common.

> +  If a pthread, that is not pinned to an lcore acquires a lock such as a
> +  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> +  then there is a possibility of large application delays.

Pinning or not doesn't matter. What matters is if the thread is 
preempted and thus is prevented from making progress for a long time.

> +  The problem is that if a thread is scheduled off the CPU while it holds
> +  a lock, then other threads will waste time spinning on the lock until
> +  the lock holder is once more rescheduled and releases the lock.
> +
> +
>   cgroup control
>   ~~~~~~~~~~~~~~
>