Re: [RFC] eal: per-thread constructors/destructors

["Mattias Rönnblom" <mattias.ronnblom@ericsson.com>]

On 2022-12-18 11:01, Morten Brørup wrote:
> This RFC introduces per-thread constructors/destructors:
> * Per-thread constructors are functions called as the first thing from newly created threads. This can be used to initialize variables in thread-local storage, variables depending on the (tid_t) thread id, and to call setup functions that must be called from the thread itself.
> * Per-thread destructors are functions called (from the thread) as the last thing when a thread ends.
> 
> At this time, I am seeking feedback on the concept and the proposed limitations.
> 

This seems like a useful mechanism, that would simplify things for 
libraries and PMDs that have dynamically allocated per-thread resources. 
It's unclear to me, how many such modules there are.

> Processes have __attribute__(constructor/destructor) to set up functions to be called before main(). Nothing similar exists for threads, so we have to design it ourselves.
> 
> The proposed per-thread constructors/destructors should not apply to all threads - only to threads created through the DPDK threads API. Agree?
>  > DPDK has the RTE_INIT()/RTE_FINI() macros for adding process 
constructors/destructors at build time, so I propose a similar API, i.e. 
adding RTE_THREAD_INIT() and RTE_THREAD_FINI() macros to build a list of 
per-thread constructors and destructors at build time.
> 
> Two functions that call the list of per-thread constructors respectively destructors will be introduced.
> 
> The per-thread constructor function will be called from the newly created threads within DPDK:
> 1. EAL/SERVICE threads: In eal_thread_loop() before the loop.
> 2. CTRL threads: In ctrl_thread_init() before start_routine().
> 3. Non-EAL threads: In rte_thread_register().
> 
> Are any thread types missing in the list above?
> 

Maybe all unregistered non-EAL threads should be excluded, including 
such created via rte_ctrl_thread_create().

In that case, you would have:
1. EAL threads (both the main and the worker lcores, and both lcores 
employed as service and non-service lcores).
2. Registered non-EAL threads (created using rte_ctrl_thread_create() or 
any other means, but having in common having called rte_thread_register()).

Another thing which is unclear to me is if libraries/PMDs tend to want 
to differentiate between EAL threads and registered non-EAL threads. EAL 
threads are generally required to be safe from preemption and also 
generally do fast path/packet processing kind of work, while what 
registered non-EAL threads tend to do (and thus what kind of API access 
they require) is more open.

Another thing to consider is how this mechanism should work in regards 
to scaling up/down a DPDK application. When an EAL thread is taken out 
of use, should the finalize function be called? It might complicate 
things, but on the other hand would allow libraries and PMDs to free 
resources no longer required.

> 
> The per-thread destructor function will also be registered by the per-thread constructor, using the POSIX pthread_cleanup_push() function.
> 
> 
> Examples of where per-thread constructors are useful:
> PRNG state initialization [1]: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-fc4c610a30810b5d&q=1&e=d8580fcf-a6bb-4e2b-aec4-a772629ece08&u=http%3A%2F%2Finbox.dpdk.org%2Fdev%2F2a5121c7-369f-afde-0898-d45a5b368c3a%40ericsson.com%2F
> PMU event setup [2]: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-9d4f6edbd54a1ac0&q=1&e=d8580fcf-a6bb-4e2b-aec4-a772629ece08&u=http%3A%2F%2Finbox.dpdk.org%2Fdev%2Fa059c403-8c6c-79c3-c3e9-a8c1815f5a14%40ericsson.com%2FT%2F%23m3d29fbc301059f007425ac148a4113e66d2ef983
> 
> 
> Please also refer to the discussion [3] about lcore init callbacks for further thoughts. Especially the de-facto inability to establish constructors/destructors at runtime.
> 
> [3]: https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-94c23fe30fb99087&q=1&e=d8580fcf-a6bb-4e2b-aec4-a772629ece08&u=http%3A%2F%2Finbox.dpdk.org%2Fdev%2F98CBD80474FA8B44BF855DF32C47DC35D875B6%40smartserver.smartshare.dk%2FT%2F%23m23b9a930a4050dc6b0305d3653754bd152c09ab7
> 
> 
> Med venlig hilsen / Kind regards,
> -Morten Brørup