Hi. In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made private, and with it the possibility to look up the thread id of the lcore worker threads disappears. One use case is an application with a monitoring function (on some control plane thread), which uses the thread ids to make sure the worker threads gets the CPU runtime they should, and thus is able to detect stalls. Is there some other way of finding out the thread_id of a lcore worker thread? All I can think of are hacks like using a temporary service function for service cores, in combination with requiring launched application threads also to store their thread id in some global structure (index by lcore_id). Is there some cleaner way? If not, would adding something like a rte_lcore_thread_id() function make sense? Regards, Mattias
> -----Original Message----- > From: dev <dev-bounces@dpdk.org> On Behalf Of Mattias Rönnblom > Sent: Tuesday, July 14, 2020 1:04 PM > To: dpdk-dev <dev@dpdk.org> > Cc: Tomasz Piatkowski <tomasz.piatkowski@ericsson.com> > Subject: [dpdk-dev] Retrieving lcore worker thread id > > Hi. Hey, > In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made private, > and with it the possibility to look up the thread id of the lcore worker > threads disappears. > > One use case is an application with a monitoring function (on some > control plane thread), which uses the thread ids to make sure the worker > threads gets the CPU runtime they should, and thus is able to detect > stalls. > > Is there some other way of finding out the thread_id of a lcore worker > thread? All I can think of are hacks like using a temporary service > function for service cores, in combination with requiring launched > application threads also to store their thread id in some global > structure (index by lcore_id). -1 for the service cores idea. I like the creative solution thinking, but not as a long-term solution. > Is there some cleaner way? If not, would adding something like a > rte_lcore_thread_id() function make sense? I think given your use-case description above, this makes sense. I guess it comes down to a discussion around if thread_id should be exposed at all - but given you have a use I'd argue yes it should. Doing so in a cross-platform way might be a bit tricky though. > Regards, > Mattias Regards, -Harry
<snip> > > > > Hi. > > Hey, > > > In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made > > private, and with it the possibility to look up the thread id of the > > lcore worker threads disappears. > > > > One use case is an application with a monitoring function (on some > > control plane thread), which uses the thread ids to make sure the > > worker threads gets the CPU runtime they should, and thus is able to > > detect stalls. This sounds similar to 'keep alive' functionality. > > > > Is there some other way of finding out the thread_id of a lcore worker > > thread? All I can think of are hacks like using a temporary service > > function for service cores, in combination with requiring launched > > application threads also to store their thread id in some global > > structure (index by lcore_id). > > -1 for the service cores idea. I like the creative solution thinking, but not as a > long-term solution. > > > Is there some cleaner way? If not, would adding something like a > > rte_lcore_thread_id() function make sense? I guess here you mean the OS provided thread ID. Are there OS calls that provide the CPU runtime? > > I think given your use-case description above, this makes sense. > I guess it comes down to a discussion around if thread_id should be exposed > at all - but given you have a use I'd argue yes it should. > Doing so in a cross-platform way might be a bit tricky though. > > > Regards, > > Mattias > > Regards, -Harry
On Tue, 14 Jul 2020 18:59:59 +0000
Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
> <snip>
>
> > >
> > > Hi.
> >
> > Hey,
> >
> > > In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made
> > > private, and with it the possibility to look up the thread id of the
> > > lcore worker threads disappears.
> > >
> > > One use case is an application with a monitoring function (on some
> > > control plane thread), which uses the thread ids to make sure the
> > > worker threads gets the CPU runtime they should, and thus is able to
> > > detect stalls.
> This sounds similar to 'keep alive' functionality.
>
> > >
> > > Is there some other way of finding out the thread_id of a lcore worker
> > > thread? All I can think of are hacks like using a temporary service
> > > function for service cores, in combination with requiring launched
> > > application threads also to store their thread id in some global
> > > structure (index by lcore_id).
> >
> > -1 for the service cores idea. I like the creative solution thinking, but not as a
> > long-term solution.
> >
> > > Is there some cleaner way? If not, would adding something like a
> > > rte_lcore_thread_id() function make sense?
> I guess here you mean the OS provided thread ID. Are there OS calls that provide the CPU runtime?
This might be difficult sinc thread id in Linux/glibc is intentionally and opaque value.
According to Posix the only valid way to look at it is to use return value from
pthread_create() and pthread_self().
On 2020-07-14 22:51, Stephen Hemminger wrote:
> On Tue, 14 Jul 2020 18:59:59 +0000
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
>
>> <snip>
>>
>>>> Hi.
>>> Hey,
>>>
>>>> In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made
>>>> private, and with it the possibility to look up the thread id of the
>>>> lcore worker threads disappears.
>>>>
>>>> One use case is an application with a monitoring function (on some
>>>> control plane thread), which uses the thread ids to make sure the
>>>> worker threads gets the CPU runtime they should, and thus is able to
>>>> detect stalls.
>> This sounds similar to 'keep alive' functionality.
>>
>>>> Is there some other way of finding out the thread_id of a lcore worker
>>>> thread? All I can think of are hacks like using a temporary service
>>>> function for service cores, in combination with requiring launched
>>>> application threads also to store their thread id in some global
>>>> structure (index by lcore_id).
>>> -1 for the service cores idea. I like the creative solution thinking, but not as a
>>> long-term solution.
>>>
>>>> Is there some cleaner way? If not, would adding something like a
>>>> rte_lcore_thread_id() function make sense?
>> I guess here you mean the OS provided thread ID. Are there OS calls that provide the CPU runtime?
> This might be difficult sinc thread id in Linux/glibc is intentionally and opaque value.
> According to Posix the only valid way to look at it is to use return value from
> pthread_create() and pthread_self().
>
The rte_lcore_thread_id() would return this value, which could
subsequently be used in the application, calling pthread_getcpuclockid()
and clock_gettime() to retrieve the run time for the lcore worker
thread. No need to break the opacity in this case, although the Linux
thread id (i.e. the result of a gettid()) would be useful in case you
would want to dig around in /proc for other scheduler statistics.
Regards,
Mattias
On 2020-07-14 15:08, Van Haaren, Harry wrote:
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Mattias Rönnblom
>> Sent: Tuesday, July 14, 2020 1:04 PM
>> To: dpdk-dev <dev@dpdk.org>
>> Cc: Tomasz Piatkowski <tomasz.piatkowski@ericsson.com>
>> Subject: [dpdk-dev] Retrieving lcore worker thread id
>>
>> Hi.
> Hey,
>
>> In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made private,
>> and with it the possibility to look up the thread id of the lcore worker
>> threads disappears.
>>
>> One use case is an application with a monitoring function (on some
>> control plane thread), which uses the thread ids to make sure the worker
>> threads gets the CPU runtime they should, and thus is able to detect
>> stalls.
>>
>> Is there some other way of finding out the thread_id of a lcore worker
>> thread? All I can think of are hacks like using a temporary service
>> function for service cores, in combination with requiring launched
>> application threads also to store their thread id in some global
>> structure (index by lcore_id).
> -1 for the service cores idea. I like the creative solution thinking,
> but not as a long-term solution.
>
>> Is there some cleaner way? If not, would adding something like a
>> rte_lcore_thread_id() function make sense?
> I think given your use-case description above, this makes sense.
> I guess it comes down to a discussion around if thread_id should
> be exposed at all - but given you have a use I'd argue yes it should.
> Doing so in a cross-platform way might be a bit tricky though.
>
With the Windows port ongoing, I guess that's a valid concern.
<rte_lcore.h> already is a heavy user of pthread ids though. Maybe you
would leave that change to a later patch, which would typedef the whole
thing into something DPDK specific and portable.
Regards,
Mattias
On Wed, 15 Jul 2020 10:17:09 +0000
Mattias Rönnblom <mattias.ronnblom@ericsson.com> wrote:
> On 2020-07-14 22:51, Stephen Hemminger wrote:
> > On Tue, 14 Jul 2020 18:59:59 +0000
> > Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
> >
> >> <snip>
> >>
> >>>> Hi.
> >>> Hey,
> >>>
> >>>> In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made
> >>>> private, and with it the possibility to look up the thread id of the
> >>>> lcore worker threads disappears.
> >>>>
> >>>> One use case is an application with a monitoring function (on some
> >>>> control plane thread), which uses the thread ids to make sure the
> >>>> worker threads gets the CPU runtime they should, and thus is able to
> >>>> detect stalls.
> >> This sounds similar to 'keep alive' functionality.
> >>
> >>>> Is there some other way of finding out the thread_id of a lcore worker
> >>>> thread? All I can think of are hacks like using a temporary service
> >>>> function for service cores, in combination with requiring launched
> >>>> application threads also to store their thread id in some global
> >>>> structure (index by lcore_id).
> >>> -1 for the service cores idea. I like the creative solution thinking, but not as a
> >>> long-term solution.
> >>>
> >>>> Is there some cleaner way? If not, would adding something like a
> >>>> rte_lcore_thread_id() function make sense?
> >> I guess here you mean the OS provided thread ID. Are there OS calls that provide the CPU runtime?
> > This might be difficult sinc thread id in Linux/glibc is intentionally and opaque value.
> > According to Posix the only valid way to look at it is to use return value from
> > pthread_create() and pthread_self().
> >
>
> The rte_lcore_thread_id() would return this value, which could
> subsequently be used in the application, calling pthread_getcpuclockid()
> and clock_gettime() to retrieve the run time for the lcore worker
> thread. No need to break the opacity in this case, although the Linux
> thread id (i.e. the result of a gettid()) would be useful in case you
> would want to dig around in /proc for other scheduler statistics.
>
>
> Regards,
>
> Mattias
>
The issue is glibc doesn't want to allow gettid()
there is no wrapper, the only way to get it is using syscall()
On 2020-07-15 18:21, Stephen Hemminger wrote:
> On Wed, 15 Jul 2020 10:17:09 +0000
> Mattias Rönnblom <mattias.ronnblom@ericsson.com> wrote:
>
>> On 2020-07-14 22:51, Stephen Hemminger wrote:
>>> On Tue, 14 Jul 2020 18:59:59 +0000
>>> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
>>>
>>>> <snip>
>>>>
>>>>>> Hi.
>>>>> Hey,
>>>>>
>>>>>> In DPDK 19.11, the lcore_config struct of <rte_lcore.h> is made
>>>>>> private, and with it the possibility to look up the thread id of the
>>>>>> lcore worker threads disappears.
>>>>>>
>>>>>> One use case is an application with a monitoring function (on some
>>>>>> control plane thread), which uses the thread ids to make sure the
>>>>>> worker threads gets the CPU runtime they should, and thus is able to
>>>>>> detect stalls.
>>>> This sounds similar to 'keep alive' functionality.
>>>>
>>>>>> Is there some other way of finding out the thread_id of a lcore worker
>>>>>> thread? All I can think of are hacks like using a temporary service
>>>>>> function for service cores, in combination with requiring launched
>>>>>> application threads also to store their thread id in some global
>>>>>> structure (index by lcore_id).
>>>>> -1 for the service cores idea. I like the creative solution thinking, but not as a
>>>>> long-term solution.
>>>>>
>>>>>> Is there some cleaner way? If not, would adding something like a
>>>>>> rte_lcore_thread_id() function make sense?
>>>> I guess here you mean the OS provided thread ID. Are there OS calls that provide the CPU runtime?
>>> This might be difficult sinc thread id in Linux/glibc is intentionally and opaque value.
>>> According to Posix the only valid way to look at it is to use return value from
>>> pthread_create() and pthread_self().
>>>
>> The rte_lcore_thread_id() would return this value, which could
>> subsequently be used in the application, calling pthread_getcpuclockid()
>> and clock_gettime() to retrieve the run time for the lcore worker
>> thread. No need to break the opacity in this case, although the Linux
>> thread id (i.e. the result of a gettid()) would be useful in case you
>> would want to dig around in /proc for other scheduler statistics.
>>
>>
>> Regards,
>>
>> Mattias
>>
> The issue is glibc doesn't want to allow gettid()
> there is no wrapper, the only way to get it is using syscall()
There was certainly a lot of reluctance to add it, but since glibc 2.30,
there is a syscall wrapper in place.
Introduce function in <rte_lcore.h> to retrieve a lcore worker thread's pthread id. This change allows applications to do things like changing the name of a lcore thread, or detect if it's getting interrupted (e.g. getting preempted by another thread). Signed-off-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com> --- lib/librte_eal/common/eal_common_thread.c | 8 ++++++++ lib/librte_eal/include/rte_lcore.h | 14 ++++++++++++++ lib/librte_eal/rte_eal_version.map | 3 +++ 3 files changed, 25 insertions(+) diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c index 73a055902..6233aef11 100644 --- a/lib/librte_eal/common/eal_common_thread.c +++ b/lib/librte_eal/common/eal_common_thread.c @@ -294,3 +294,11 @@ rte_thread_unregister(void) RTE_LOG(DEBUG, EAL, "Unregistered non-EAL thread (was lcore %u).\n", lcore_id); } + +int rte_lcore_thread_id(unsigned int lcore_id, pthread_t *id) +{ + if (lcore_id >= RTE_MAX_LCORE) + return -1; + *id = lcore_config[lcore_id].thread_id; + return 0; +} diff --git a/lib/librte_eal/include/rte_lcore.h b/lib/librte_eal/include/rte_lcore.h index b8b64a625..dae8d11ca 100644 --- a/lib/librte_eal/include/rte_lcore.h +++ b/lib/librte_eal/include/rte_lcore.h @@ -363,6 +363,20 @@ int rte_thread_set_affinity(rte_cpuset_t *cpusetp); */ void rte_thread_get_affinity(rte_cpuset_t *cpusetp); +/** + * Get the thread id of a lcore worker thread. + * + * @param lcore_id + * The identifier of the lcore, which MUST be between 0 and RTE_MAX_LCORE-1. + * @param[out] id + * A pointer that will be filled in with the lcore worker thread's POSIX + * thread id, on success. + * @return + * On success, return 0; otherwise return a negative value. + */ +__rte_experimental +int rte_lcore_thread_id(unsigned int lcore_id, pthread_t *id); + /** * Set thread names. * diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map index bf0c17c23..571aacd17 100644 --- a/lib/librte_eal/rte_eal_version.map +++ b/lib/librte_eal/rte_eal_version.map @@ -403,6 +403,9 @@ EXPERIMENTAL { rte_mp_disable; rte_thread_register; rte_thread_unregister; + + # added in 20.11 + rte_lcore_thread_id; }; INTERNAL { -- 2.25.1
On Mon, Aug 10, 2020 at 11:37 AM Mattias Rönnblom
<mattias.ronnblom@ericsson.com> wrote:
>
> Introduce function in <rte_lcore.h> to retrieve a lcore worker
> thread's pthread id.
>
> This change allows applications to do things like changing the name of
> a lcore thread, or detect if it's getting interrupted (e.g. getting
> preempted by another thread).
>
> Signed-off-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
I can see no feedback.
I am a bit reluctant at giving access to this info which is not really
abstracted across OS..
Is there no other way to achieve what you wanted?
--
David Marchand
On 2020-10-20 15:38, David Marchand wrote:
> On Mon, Aug 10, 2020 at 11:37 AM Mattias Rönnblom
> <mattias.ronnblom@ericsson.com> wrote:
>> Introduce function in <rte_lcore.h> to retrieve a lcore worker
>> thread's pthread id.
>>
>> This change allows applications to do things like changing the name of
>> a lcore thread, or detect if it's getting interrupted (e.g. getting
>> preempted by another thread).
>>
>> Signed-off-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
> I can see no feedback.
> I am a bit reluctant at giving access to this info which is not really
> abstracted across OS..
> Is there no other way to achieve what you wanted?
>
>
The fact that POSIX threading, or at least such thread identifiers, are
used internally is already visible in the lcore API.
What we are using now is a little hack with a service started on each
core, which calls pthead_self(), stores the result, and turns itself off.