RTE lock

RTE lock

[Mattias Rönnblom]

Shouldn't we have a DPDK-native mutex API, rather than using direct 
POSIX mutex lock calls?

There are two reasons for this, as I see it
1) more cleanly support non-POSIX operating system (i.e., Microsoft 
Windows).
2) to discourage mechanical use of spinlocks in places where a regular 
mutex lock is more appropriate.

I think (and hope) DPDK developers will tend to pick DPDK-native rather 
than other APIs as their first choice.

For locks, they go for spinlocks, even in control (non-fast 
path/non-packet processing) code paths (e.g., calls made by the typical 
non-EAL thread).

Using spinlocks to synchronize threads that may be preempted aren't 
great idea.

Re: RTE lock

[Stephen Hemminger]

On Tue, 5 Mar 2024 21:18:20 +0100
Mattias Rönnblom <hofors@lysator.liu.se> wrote:

> Shouldn't we have a DPDK-native mutex API, rather than using direct 
> POSIX mutex lock calls?
> 
> There are two reasons for this, as I see it
> 1) more cleanly support non-POSIX operating system (i.e., Microsoft 
> Windows).
> 2) to discourage mechanical use of spinlocks in places where a regular 
> mutex lock is more appropriate.
> 
> I think (and hope) DPDK developers will tend to pick DPDK-native rather 
> than other APIs as their first choice.
> 
> For locks, they go for spinlocks, even in control (non-fast 
> path/non-packet processing) code paths (e.g., calls made by the typical 
> non-EAL thread).
> 
> Using spinlocks to synchronize threads that may be preempted aren't 
> great idea.

Maybe a wrapper, but actually implementing mutex with flock is complicated.

Re: RTE lock

[Tyler Retzlaff]

On Tue, Mar 05, 2024 at 09:18:20PM +0100, Mattias Rönnblom wrote:
> Shouldn't we have a DPDK-native mutex API, rather than using direct
> POSIX mutex lock calls?

David raised this a while back and the consensus is yes. I admit it's
been on my radar for a long time for the obvious reasons you list below
but with other work hasn't been a priority (yet).

> 
> There are two reasons for this, as I see it
> 1) more cleanly support non-POSIX operating system (i.e., Microsoft
> Windows).
> 2) to discourage mechanical use of spinlocks in places where a
> regular mutex lock is more appropriate.
> 
> I think (and hope) DPDK developers will tend to pick DPDK-native
> rather than other APIs as their first choice.

I spent some time evaluating C11 mutex but it really didn't strike me as
being fit for purpose so I think DPDK-native is probably the only way to
go. If behind the scenes particular locks relied on something standard
for Windows perhaps it could be hidden as an implementation detail.

> 
> For locks, they go for spinlocks, even in control (non-fast
> path/non-packet processing) code paths (e.g., calls made by the
> typical non-EAL thread).
> 
> Using spinlocks to synchronize threads that may be preempted aren't
> great idea.

If you're thinking of looking into this i'd be happy to see it solved.

ty

Re: RTE lock

[Mattias Rönnblom]

On 2024-03-05 22:02, Tyler Retzlaff wrote:
> On Tue, Mar 05, 2024 at 09:18:20PM +0100, Mattias Rönnblom wrote:
>> Shouldn't we have a DPDK-native mutex API, rather than using direct
>> POSIX mutex lock calls?
> 
> David raised this a while back and the consensus is yes. I admit it's
> been on my radar for a long time for the obvious reasons you list below
> but with other work hasn't been a priority (yet).
> 
>>
>> There are two reasons for this, as I see it
>> 1) more cleanly support non-POSIX operating system (i.e., Microsoft
>> Windows).
>> 2) to discourage mechanical use of spinlocks in places where a
>> regular mutex lock is more appropriate.
>>
>> I think (and hope) DPDK developers will tend to pick DPDK-native
>> rather than other APIs as their first choice.
> 
> I spent some time evaluating C11 mutex but it really didn't strike me as
> being fit for purpose so I think DPDK-native is probably the only way to
> go. If behind the scenes particular locks relied on something standard
> for Windows perhaps it could be hidden as an implementation detail.
> 

What was it you were missing?

I'm not familiar with C11 mtx_*()

>>
>> For locks, they go for spinlocks, even in control (non-fast
>> path/non-packet processing) code paths (e.g., calls made by the
>> typical non-EAL thread).
>>
>> Using spinlocks to synchronize threads that may be preempted aren't
>> great idea.
> 
> If you're thinking of looking into this i'd be happy to see it solved.
> 

I have no immediate plans, but this might be something I'll do in the 
future.

> ty

Re: RTE lock

[Stephen Hemminger]

On Thu, 7 Mar 2024 20:50:26 +0100
Mattias Rönnblom <hofors@lysator.liu.se> wrote:

> On 2024-03-05 22:02, Tyler Retzlaff wrote:
> > On Tue, Mar 05, 2024 at 09:18:20PM +0100, Mattias Rönnblom wrote:  
> >> Shouldn't we have a DPDK-native mutex API, rather than using direct
> >> POSIX mutex lock calls?  
> > 
> > David raised this a while back and the consensus is yes. I admit it's
> > been on my radar for a long time for the obvious reasons you list below
> > but with other work hasn't been a priority (yet).
> >   
> >>
> >> There are two reasons for this, as I see it
> >> 1) more cleanly support non-POSIX operating system (i.e., Microsoft
> >> Windows).
> >> 2) to discourage mechanical use of spinlocks in places where a
> >> regular mutex lock is more appropriate.
> >>
> >> I think (and hope) DPDK developers will tend to pick DPDK-native
> >> rather than other APIs as their first choice.  
> > 
> > I spent some time evaluating C11 mutex but it really didn't strike me as
> > being fit for purpose so I think DPDK-native is probably the only way to
> > go. If behind the scenes particular locks relied on something standard
> > for Windows perhaps it could be hidden as an implementation detail.
> >   
> 
> What was it you were missing?
> 
> I'm not familiar with C11 mtx_*()
> 
> >>
> >> For locks, they go for spinlocks, even in control (non-fast
> >> path/non-packet processing) code paths (e.g., calls made by the
> >> typical non-EAL thread).
> >>
> >> Using spinlocks to synchronize threads that may be preempted aren't
> >> great idea.  
> > 
> > If you're thinking of looking into this i'd be happy to see it solved.
> >   
> 
> I have no immediate plans, but this might be something I'll do in the 
> future.
> 
> > ty  

For Linux, what would be good is a simplified version of what pthread_mutex
does. The current code for pthread_mutex in glibc has a lot of indirection
and complexity which definately slows down the fast uncontended path.
Ideally, an inline version using futex.

Glibc has so much indirection because it supports so many mutex variants
and operating systems like mach and hurd...

Re: RTE lock

[Mattias Rönnblom]

On 2024-03-07 21:27, Stephen Hemminger wrote:
> On Thu, 7 Mar 2024 20:50:26 +0100
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> 
>> On 2024-03-05 22:02, Tyler Retzlaff wrote:
>>> On Tue, Mar 05, 2024 at 09:18:20PM +0100, Mattias Rönnblom wrote:
>>>> Shouldn't we have a DPDK-native mutex API, rather than using direct
>>>> POSIX mutex lock calls?
>>>
>>> David raised this a while back and the consensus is yes. I admit it's
>>> been on my radar for a long time for the obvious reasons you list below
>>> but with other work hasn't been a priority (yet).
>>>    
>>>>
>>>> There are two reasons for this, as I see it
>>>> 1) more cleanly support non-POSIX operating system (i.e., Microsoft
>>>> Windows).
>>>> 2) to discourage mechanical use of spinlocks in places where a
>>>> regular mutex lock is more appropriate.
>>>>
>>>> I think (and hope) DPDK developers will tend to pick DPDK-native
>>>> rather than other APIs as their first choice.
>>>
>>> I spent some time evaluating C11 mutex but it really didn't strike me as
>>> being fit for purpose so I think DPDK-native is probably the only way to
>>> go. If behind the scenes particular locks relied on something standard
>>> for Windows perhaps it could be hidden as an implementation detail.
>>>    
>>
>> What was it you were missing?
>>
>> I'm not familiar with C11 mtx_*()
>>
>>>>
>>>> For locks, they go for spinlocks, even in control (non-fast
>>>> path/non-packet processing) code paths (e.g., calls made by the
>>>> typical non-EAL thread).
>>>>
>>>> Using spinlocks to synchronize threads that may be preempted aren't
>>>> great idea.
>>>
>>> If you're thinking of looking into this i'd be happy to see it solved.
>>>    
>>
>> I have no immediate plans, but this might be something I'll do in the
>> future.
>>
>>> ty
> 
> For Linux, what would be good is a simplified version of what pthread_mutex
> does. The current code for pthread_mutex in glibc has a lot of indirection
> and complexity which definately slows down the fast uncontended path.
> Ideally, an inline version using futex.
> 
> Glibc has so much indirection because it supports so many mutex variants
> and operating systems like mach and hurd...


The use case RTE lock would address could be
A) Synchronizing different control plane threads, or data plane (EAL) 
threads when they haven't yet started processing packets.
B) Synchronizing control plane and data plane threads during 
steady-state operation.

It was A) I had in mind when I started this thread, and for this case, a 
little extra Hurd-induced lock overhead is not a huge deal. RTE lock 
here could be tiny wrapper, although some thought need to go into 
getting the API right obviously.

With direct futex() calls one could attempt to address B as well. EAL 
threads could opt to spin, and control plane threads could be put to 
sleep (i.e., futex()). The uncontended and "trylock" paths could be very 
efficient. Etc.

RE: RTE lock

[Morten Brørup]

> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> Sent: Tuesday, 5 March 2024 21.18
> 
> Shouldn't we have a DPDK-native mutex API, rather than using direct
> POSIX mutex lock calls?
> 
> There are two reasons for this, as I see it
> 1) more cleanly support non-POSIX operating system (i.e., Microsoft
> Windows).
> 2) to discourage mechanical use of spinlocks in places where a regular
> mutex lock is more appropriate.
> 
> I think (and hope) DPDK developers will tend to pick DPDK-native rather
> than other APIs as their first choice.
> 
> For locks, they go for spinlocks, even in control (non-fast
> path/non-packet processing) code paths (e.g., calls made by the typical
> non-EAL thread).
> 
> Using spinlocks to synchronize threads that may be preempted aren't
> great idea.

The DPDK does interact with the control plane in ways that require locks, so adding preemption-friendly synchronization APIs seems appropriate.
And although I often rant about EAL bloat, I suppose this would belong into the EAL.

If you add any APIs with a timeout parameter, make sure to use struct timespec, like pthread_mutex_timedlock(). If a synchronization function is exclusively used for the control plane, millisecond resolution would probably suffice, but if it might be used by the data plane, millisecond level delays are intolerable. This was overlooked when adding rte_epoll_wait(), which was implemented using epoll_wait() instead of epoll_pwait2(), so you have to use the O/S specific timerfd_create/settime() functions for high-res timeout - and then the whole point of this EAL function goes out the window.

For reference, DPDK offers many types of locks [1]:
atomic, mcslock, pflock, rwlock, seqcount, seqlock, spinlock, ticketlock, RCU

[1] https://doc.dpdk.org/api/