EAL threads

EAL threads

[Mattias Rönnblom]

I have some lcore-related questions:

The documentation make use of the term "non-EAL thread". Is a non-EAL 
thread the same thing as a non-lcore thread? I.e., are there EAL threads 
that are not lcore threads?

I also have a question related to rte_register_thread(): shouldn't the 
documentation say the user is assumed to pin the calling thread to some 
core? That is the expectation, correct?

Re: EAL threads

[Bruce Richardson]

On Wed, Sep 21, 2022 at 10:39:15AM +0200, Mattias Rönnblom wrote:
> I have some lcore-related questions:
> 
> The documentation make use of the term "non-EAL thread". Is a non-EAL thread
> the same thing as a non-lcore thread? I.e., are there EAL threads that are
> not lcore threads?

Yes, there are some threads created by EAL which are not lcore threads.
These are generally for background tasks, such as interrupts or telemetry,
and are given a coremask to be kept away from the dataplane lcore threads.
Therefore, I think you are right to try and get clear terminology for the
threads. In most cases, I think non-EAL thread is referring to threads
created by the app itself, but it may in some cases refer to the 'non-lcore
threads' as you call them.

Personally, I would suggest terms like:
* dataplane threads
* background eal threads
* app threads

for clarity, since I think the term EAL thread is ambiguous. However,
you or others might have been ideas for terms.

> 
> I also have a question related to rte_register_thread(): shouldn't the
> documentation say the user is assumed to pin the calling thread to some
> core? That is the expectation, correct?

Probably, but it's not mandatory.

RE: EAL threads

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Wednesday, 21 September 2022 11.06
> 
> On Wed, Sep 21, 2022 at 10:39:15AM +0200, Mattias Rönnblom wrote:
> > I have some lcore-related questions:
> >
> > The documentation make use of the term "non-EAL thread". Is a non-EAL
> thread
> > the same thing as a non-lcore thread? I.e., are there EAL threads
> that are
> > not lcore threads?
> 
> Yes, there are some threads created by EAL which are not lcore threads.
> These are generally for background tasks, such as interrupts or
> telemetry,
> and are given a coremask to be kept away from the dataplane lcore
> threads.
> Therefore, I think you are right to try and get clear terminology for
> the
> threads. In most cases, I think non-EAL thread is referring to threads
> created by the app itself, but it may in some cases refer to the 'non-
> lcore
> threads' as you call them.
> 
> Personally, I would suggest terms like:
> * dataplane threads
> * background eal threads
> * app threads

Assuming that "app threads" are not performing any dataplane packet processing, I think "control plane threads" might be a better name. Consider this: The purpose of a DPDK application is to perform dataplane tasks, so "app threads" might easily be confused with "dataplane threads".

> 
> for clarity, since I think the term EAL thread is ambiguous. However,
> you or others might have been ideas for terms.
> 
> >
> > I also have a question related to rte_register_thread(): shouldn't
> the
> > documentation say the user is assumed to pin the calling thread to
> some
> > core? That is the expectation, correct?
> 
> Probably, but it's not mandatory.

For this terminology, I think both pinning and isolation of lcores should be considered.

Unfortunately, the confusion is much broader...

Most of the DPDK documentation is based on the architecture before the introduction of various mechanisms for cooperative time sharing of lcores, such as Service Cores and Graphs.

This also goes for the names of DPDK types, functions and macros. E.g. RTE_PER_LCORE [1] actually means per thread, regardless which lcore runs the thread (perhaps even multiple lcores may be scheduled from time to time to run the thread). And why aren't there similar RTE_PER_SEVICE and RTE_PER_NODE macros for Services and Graph nodes...

It would be nice settling on an expanded set of well defined terms, and cleaning up the documentation (and type/function/macro names) accordingly.

[1]: https://doc.dpdk.org/api/rte__per__lcore_8h.html

Re: EAL threads

[Mattias Rönnblom]

On 2022-09-21 11:05, Bruce Richardson wrote:
> On Wed, Sep 21, 2022 at 10:39:15AM +0200, Mattias Rönnblom wrote:
>> I have some lcore-related questions:
>>
>> The documentation make use of the term "non-EAL thread". Is a non-EAL thread
>> the same thing as a non-lcore thread? I.e., are there EAL threads that are
>> not lcore threads?
> 
> Yes, there are some threads created by EAL which are not lcore threads.
> These are generally for background tasks, such as interrupts or telemetry,
> and are given a coremask to be kept away from the dataplane lcore threads.
> Therefore, I think you are right to try and get clear terminology for the
> threads. In most cases, I think non-EAL thread is referring to threads
> created by the app itself, but it may in some cases refer to the 'non-lcore
> threads' as you call them.
> 
> Personally, I would suggest terms like:
> * dataplane threads
> * background eal threads
> * app threads
> 
> for clarity, since I think the term EAL thread is ambiguous. However,
> you or others might have been ideas for terms.
> 

It's confusing to have something called an EAL thread to have a lcore id 
and generally being referred to as "a lcore", when it is in fact not 
necessarily tied to a logical core (in the generic hardware sense of the 
word).

What's important to various DPDK code (PMDs, libraries) is
a) If the calling thread has a lcore id, and thus can be trusted to have 
its own, per-lcore data structure.
b) Whether or not the thread may be (involuntarily) preempted, in 
particular by other threads which it might be 
communicating/synchronizing-with.

As of now, an EAL thread qualifies for a), and usually, but not always, 
for b) (not in a strict sense, but it's rarely interrupted and if so, 
only by threads or ISRs outside the DPDK process).

A lot of things in DPDK assumes b) from EAL threads. Any library or PMD 
that uses spinlocks, the default rings, etc. There might be more subtle 
issues as well, I'm guessing. Just take the new busyness-patch set. If a 
thread which is in a busy state it's being interrupted, it will be 
considered busy even when it's not running, but some other thread is. If 
this other thread is also an EAL thread, that does work and is 
considered busy, the telemetry data might be wildly inaccurate and even 
suggest that the system spends more CPU cycles than are actually 
available. DSW port load measurement, using the same scheme, will also 
fall into this trap, when being used by preemptable EAL threads.

Thanks for your answer.

>>
>> I also have a question related to rte_register_thread(): shouldn't the
>> documentation say the user is assumed to pin the calling thread to some
>> core? That is the expectation, correct?
> 
> Probably, but it's not mandatory.