[RFC] eal: per-thread constructors/destructors

[RFC] eal: per-thread constructors/destructors

[Morten Brørup]

This RFC introduces per-thread constructors/destructors:
* Per-thread constructors are functions called as the first thing from newly created threads. This can be used to initialize variables in thread-local storage, variables depending on the (tid_t) thread id, and to call setup functions that must be called from the thread itself.
* Per-thread destructors are functions called (from the thread) as the last thing when a thread ends.

At this time, I am seeking feedback on the concept and the proposed limitations.

Processes have __attribute__(constructor/destructor) to set up functions to be called before main(). Nothing similar exists for threads, so we have to design it ourselves.

The proposed per-thread constructors/destructors should not apply to all threads - only to threads created through the DPDK threads API. Agree?

DPDK has the RTE_INIT()/RTE_FINI() macros for adding process constructors/destructors at build time, so I propose a similar API, i.e. adding RTE_THREAD_INIT() and RTE_THREAD_FINI() macros to build a list of per-thread constructors and destructors at build time.

Two functions that call the list of per-thread constructors respectively destructors will be introduced.

The per-thread constructor function will be called from the newly created threads within DPDK:
1. EAL/SERVICE threads: In eal_thread_loop() before the loop.
2. CTRL threads: In ctrl_thread_init() before start_routine().
3. Non-EAL threads: In rte_thread_register().

Are any thread types missing in the list above?


The per-thread destructor function will also be registered by the per-thread constructor, using the POSIX pthread_cleanup_push() function.


Examples of where per-thread constructors are useful:
PRNG state initialization [1]: http://inbox.dpdk.org/dev/2a5121c7-369f-afde-0898-d45a5b368c3a@ericsson.com/
PMU event setup [2]: http://inbox.dpdk.org/dev/a059c403-8c6c-79c3-c3e9-a8c1815f5a14@ericsson.com/T/#m3d29fbc301059f007425ac148a4113e66d2ef983


Please also refer to the discussion [3] about lcore init callbacks for further thoughts. Especially the de-facto inability to establish constructors/destructors at runtime.

[3]: http://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35D875B6@smartserver.smartshare.dk/T/#m23b9a930a4050dc6b0305d3653754bd152c09ab7


Med venlig hilsen / Kind regards,
-Morten Brørup

Re: [RFC] eal: per-thread constructors/destructors

[Mattias Rönnblom]

On 2022-12-18 11:01, Morten Brørup wrote:
> This RFC introduces per-thread constructors/destructors:
> * Per-thread constructors are functions called as the first thing from newly created threads. This can be used to initialize variables in thread-local storage, variables depending on the (tid_t) thread id, and to call setup functions that must be called from the thread itself.
> * Per-thread destructors are functions called (from the thread) as the last thing when a thread ends.
> 
> At this time, I am seeking feedback on the concept and the proposed limitations.
> 

This seems like a useful mechanism, that would simplify things for 
libraries and PMDs that have dynamically allocated per-thread resources. 
It's unclear to me, how many such modules there are.

> Processes have __attribute__(constructor/destructor) to set up functions to be called before main(). Nothing similar exists for threads, so we have to design it ourselves.
> 
> The proposed per-thread constructors/destructors should not apply to all threads - only to threads created through the DPDK threads API. Agree?
>  > DPDK has the RTE_INIT()/RTE_FINI() macros for adding process 
constructors/destructors at build time, so I propose a similar API, i.e. 
adding RTE_THREAD_INIT() and RTE_THREAD_FINI() macros to build a list of 
per-thread constructors and destructors at build time.
> 
> Two functions that call the list of per-thread constructors respectively destructors will be introduced.
> 
> The per-thread constructor function will be called from the newly created threads within DPDK:
> 1. EAL/SERVICE threads: In eal_thread_loop() before the loop.
> 2. CTRL threads: In ctrl_thread_init() before start_routine().
> 3. Non-EAL threads: In rte_thread_register().
> 
> Are any thread types missing in the list above?
> 

Maybe all unregistered non-EAL threads should be excluded, including 
such created via rte_ctrl_thread_create().

In that case, you would have:
1. EAL threads (both the main and the worker lcores, and both lcores 
employed as service and non-service lcores).
2. Registered non-EAL threads (created using rte_ctrl_thread_create() or 
any other means, but having in common having called rte_thread_register()).

Another thing which is unclear to me is if libraries/PMDs tend to want 
to differentiate between EAL threads and registered non-EAL threads. EAL 
threads are generally required to be safe from preemption and also 
generally do fast path/packet processing kind of work, while what 
registered non-EAL threads tend to do (and thus what kind of API access 
they require) is more open.

Another thing to consider is how this mechanism should work in regards 
to scaling up/down a DPDK application. When an EAL thread is taken out 
of use, should the finalize function be called? It might complicate 
things, but on the other hand would allow libraries and PMDs to free 
resources no longer required.

> 
> The per-thread destructor function will also be registered by the per-thread constructor, using the POSIX pthread_cleanup_push() function.
> 
> 
> Examples of where per-thread constructors are useful:
> PRNG state initialization [1]: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-fc4c610a30810b5d&q=1&e=d8580fcf-a6bb-4e2b-aec4-a772629ece08&u=http%3A%2F%2Finbox.dpdk.org%2Fdev%2F2a5121c7-369f-afde-0898-d45a5b368c3a%40ericsson.com%2F
> PMU event setup [2]: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-9d4f6edbd54a1ac0&q=1&e=d8580fcf-a6bb-4e2b-aec4-a772629ece08&u=http%3A%2F%2Finbox.dpdk.org%2Fdev%2Fa059c403-8c6c-79c3-c3e9-a8c1815f5a14%40ericsson.com%2FT%2F%23m3d29fbc301059f007425ac148a4113e66d2ef983
> 
> 
> Please also refer to the discussion [3] about lcore init callbacks for further thoughts. Especially the de-facto inability to establish constructors/destructors at runtime.
> 
> [3]: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-94c23fe30fb99087&q=1&e=d8580fcf-a6bb-4e2b-aec4-a772629ece08&u=http%3A%2F%2Finbox.dpdk.org%2Fdev%2F98CBD80474FA8B44BF855DF32C47DC35D875B6%40smartserver.smartshare.dk%2FT%2F%23m23b9a930a4050dc6b0305d3653754bd152c09ab7
> 
> 
> Med venlig hilsen / Kind regards,
> -Morten Brørup

Re: [RFC] eal: per-thread constructors/destructors

[David Marchand]

On Sun, Dec 18, 2022 at 11:01 AM Morten Brørup <mb@smartsharesystems.com> wrote:
>
> This RFC introduces per-thread constructors/destructors:
> * Per-thread constructors are functions called as the first thing from newly created threads. This can be used to initialize variables in thread-local storage, variables depending on the (tid_t) thread id, and to call setup functions that must be called from the thread itself.
> * Per-thread destructors are functions called (from the thread) as the last thing when a thread ends.
>
> At this time, I am seeking feedback on the concept and the proposed limitations.
>
> Processes have __attribute__(constructor/destructor) to set up functions to be called before main(). Nothing similar exists for threads, so we have to design it ourselves.
>
> The proposed per-thread constructors/destructors should not apply to all threads - only to threads created through the DPDK threads API. Agree?
>
> DPDK has the RTE_INIT()/RTE_FINI() macros for adding process constructors/destructors at build time, so I propose a similar API, i.e. adding RTE_THREAD_INIT() and RTE_THREAD_FINI() macros to build a list of per-thread constructors and destructors at build time.
>
> Two functions that call the list of per-thread constructors respectively destructors will be introduced.
>
> The per-thread constructor function will be called from the newly created threads within DPDK:
> 1. EAL/SERVICE threads: In eal_thread_loop() before the loop.
> 2. CTRL threads: In ctrl_thread_init() before start_routine().
> 3. Non-EAL threads: In rte_thread_register().
>
> Are any thread types missing in the list above?

I noticed a few drivers doing stuff behind EAL back.. you might want
to grep pthread_create.
They should probably be converted to control threads, or if there is a
reason, it could give more inputs on this topic.


>
>
> The per-thread destructor function will also be registered by the per-thread constructor, using the POSIX pthread_cleanup_push() function.

What about Windows?
Cc: Tyler.


>
>
> Examples of where per-thread constructors are useful:
> PRNG state initialization [1]: http://inbox.dpdk.org/dev/2a5121c7-369f-afde-0898-d45a5b368c3a@ericsson.com/
> PMU event setup [2]: http://inbox.dpdk.org/dev/a059c403-8c6c-79c3-c3e9-a8c1815f5a14@ericsson.com/T/#m3d29fbc301059f007425ac148a4113e66d2ef983
>
>
> Please also refer to the discussion [3] about lcore init callbacks for further thoughts. Especially the de-facto inability to establish constructors/destructors at runtime.
>
> [3]: http://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35D875B6@smartserver.smartshare.dk/T/#m23b9a930a4050dc6b0305d3653754bd152c09ab7



-- 
David Marchand

Re: [RFC] eal: per-thread constructors/destructors

[Tyler Retzlaff]

On Wed, Dec 21, 2022 at 05:12:54PM +0100, David Marchand wrote:
> On Sun, Dec 18, 2022 at 11:01 AM Morten Brørup <mb@smartsharesystems.com> wrote:
> >
> > This RFC introduces per-thread constructors/destructors:
> > * Per-thread constructors are functions called as the first thing from newly created threads. This can be used to initialize variables in thread-local storage, variables depending on the (tid_t) thread id, and to call setup functions that must be called from the thread itself.
> > * Per-thread destructors are functions called (from the thread) as the last thing when a thread ends.
> >
> > At this time, I am seeking feedback on the concept and the proposed limitations.
> >
> > Processes have __attribute__(constructor/destructor) to set up functions to be called before main(). Nothing similar exists for threads, so we have to design it ourselves.
> >
> > The proposed per-thread constructors/destructors should not apply to all threads - only to threads created through the DPDK threads API. Agree?

i'm not sure what threads you mean here, but i guess you mean only
threads created with rte_thread_create? so this excludes the ``main''
thread and any threads created by the application via any other api that
causes a thread to be created? - seems reasonable.

> >
> > DPDK has the RTE_INIT()/RTE_FINI() macros for adding process constructors/destructors at build time, so I propose a similar API, i.e. adding RTE_THREAD_INIT() and RTE_THREAD_FINI() macros to build a list of per-thread constructors and destructors at build time.
> >
> > Two functions that call the list of per-thread constructors respectively destructors will be introduced.
> >
> > The per-thread constructor function will be called from the newly created threads within DPDK:
> > 1. EAL/SERVICE threads: In eal_thread_loop() before the loop.
> > 2. CTRL threads: In ctrl_thread_init() before start_routine().
> > 3. Non-EAL threads: In rte_thread_register().
> >
> > Are any thread types missing in the list above?
> 
> I noticed a few drivers doing stuff behind EAL back.. you might want
> to grep pthread_create.
> They should probably be converted to control threads, or if there is a
> reason, it could give more inputs on this topic.

yes, there are drivers that probably pre-date the rte_ctrl_thread_create
api that ought to be converted to using rte_ctrl_thread_create or
ideally (if we can get more review / merged) the series where i have
added the rte_control_thread_create api that operates with rte_thread_t.

> 
> 
> >
> >
> > The per-thread destructor function will also be registered by the per-thread constructor, using the POSIX pthread_cleanup_push() function.
> 
> What about Windows?
> Cc: Tyler.

yes, i had noticed this rfc and haven't had time to research properly to
respond. in principal it seems like something that would be useful.

in general if it is going to be part of eal we need to understand if
the functionality can be feasibly provided for all supported platforms
otherwise it doesn't belong in the abstraction layer.

so long as it can be consistently provided on windows i certainly have
no objection and if a windows implementation and working unit test is
present i'm happy to prioritize review and testing to facilitate the
work.

i appreciate any assistance given that moves the windows port closer
to parity with linux/freebsd.

thanks!

ty

Re: [RFC] eal: per-thread constructors/destructors

[Mattias Rönnblom]

On 2022-12-21 17:12, David Marchand wrote:
> On Sun, Dec 18, 2022 at 11:01 AM Morten Brørup <mb@smartsharesystems.com> wrote:
>>
>> This RFC introduces per-thread constructors/destructors:
>> * Per-thread constructors are functions called as the first thing from newly created threads. This can be used to initialize variables in thread-local storage, variables depending on the (tid_t) thread id, and to call setup functions that must be called from the thread itself.
>> * Per-thread destructors are functions called (from the thread) as the last thing when a thread ends.
>>
>> At this time, I am seeking feedback on the concept and the proposed limitations.
>>
>> Processes have __attribute__(constructor/destructor) to set up functions to be called before main(). Nothing similar exists for threads, so we have to design it ourselves.
>>
>> The proposed per-thread constructors/destructors should not apply to all threads - only to threads created through the DPDK threads API. Agree?
>>
>> DPDK has the RTE_INIT()/RTE_FINI() macros for adding process constructors/destructors at build time, so I propose a similar API, i.e. adding RTE_THREAD_INIT() and RTE_THREAD_FINI() macros to build a list of per-thread constructors and destructors at build time.
>>
>> Two functions that call the list of per-thread constructors respectively destructors will be introduced.
>>
>> The per-thread constructor function will be called from the newly created threads within DPDK:
>> 1. EAL/SERVICE threads: In eal_thread_loop() before the loop.
>> 2. CTRL threads: In ctrl_thread_init() before start_routine().
>> 3. Non-EAL threads: In rte_thread_register().
>>
>> Are any thread types missing in the list above?
> 
> I noticed a few drivers doing stuff behind EAL back.. you might want
> to grep pthread_create.
> They should probably be converted to control threads, or if there is a
> reason, it could give more inputs on this topic.
> 
> 
>>
>>
>> The per-thread destructor function will also be registered by the per-thread constructor, using the POSIX pthread_cleanup_push() function.
> 
> What about Windows?

I don't think there's a need to involve pthread_cleanup_push(). For 
DPDK-created threads, EAL could run the destructor in a wrapper 
function. The main lcore's fini callbacks could be run in rte_eal_cleanup().

> Cc: Tyler.
> 
>  >>
>>
>> Examples of where per-thread constructors are useful:
>> PRNG state initialization [1]: http://inbox.dpdk.org/dev/2a5121c7-369f-afde-0898-d45a5b368c3a@ericsson.com/
>> PMU event setup [2]: http://inbox.dpdk.org/dev/a059c403-8c6c-79c3-c3e9-a8c1815f5a14@ericsson.com/T/#m3d29fbc301059f007425ac148a4113e66d2ef983
>>
>>
>> Please also refer to the discussion [3] about lcore init callbacks for further thoughts. Especially the de-facto inability to establish constructors/destructors at runtime.
>>
>> [3]: http://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35D875B6@smartserver.smartshare.dk/T/#m23b9a930a4050dc6b0305d3653754bd152c09ab7
> 
> 
>