From: Thomas Monjalon <thomas@monjalon.net>
To: Tyler Retzlaff <roretzla@linux.microsoft.com>
Cc: David Marchand <david.marchand@redhat.com>, dev <dev@dpdk.org>,
Bruce Richardson <bruce.richardson@intel.com>,
Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>,
Narcisa Ana Maria Vasile <navasile@linux.microsoft.com>,
Dmitry Malloy <dmitrym@microsoft.com>,
Pallavi Kadam <pallavi.kadam@intel.com>
Subject: Re: [PATCH] eal: factorize lcore main loop
Date: Fri, 25 Mar 2022 15:58:40 +0100 [thread overview]
Message-ID: <2027001.KlZ2vcFHjT@thomas> (raw)
In-Reply-To: <20220325121126.GA6378@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
25/03/2022 13:11, Tyler Retzlaff:
> On Thu, Mar 24, 2022 at 03:44:23PM +0100, David Marchand wrote:
> > On Thu, Mar 24, 2022 at 9:31 AM Tyler Retzlaff
> > <roretzla@linux.microsoft.com> wrote:
> > > > diff --git a/lib/eal/common/eal_common_thread.c b/lib/eal/common/eal_common_thread.c
> > > > index 684bea166c..256de91abc 100644
> > > > --- a/lib/eal/common/eal_common_thread.c
> > > > +++ b/lib/eal/common/eal_common_thread.c
> > > > @@ -9,6 +9,7 @@
> > > > #include <assert.h>
> > > > #include <string.h>
> > > >
> > > > +#include <rte_eal_trace.h>
> > > > #include <rte_errno.h>
> > > > #include <rte_lcore.h>
> > > > #include <rte_log.h>
> > > > @@ -163,6 +164,77 @@ __rte_thread_uninit(void)
> > > > RTE_PER_LCORE(_lcore_id) = LCORE_ID_ANY;
> > > > }
> > > >
> > > > +/* main loop of threads */
> > > > +__rte_noreturn void *
> > > > +eal_thread_loop(__rte_unused void *arg)
> > > > +{
> > > > + char cpuset[RTE_CPU_AFFINITY_STR_LEN];
> > > > + pthread_t thread_id = pthread_self();
> > > > + unsigned int lcore_id;
> > > > + int ret;
> > > > +
> > > > + /* retrieve our lcore_id from the configuration structure */
> > > > + RTE_LCORE_FOREACH_WORKER(lcore_id) {
> > > > + if (thread_id == lcore_config[lcore_id].thread_id)
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >
> > > i can see that in practice this isn't a problem since the linux
> > > implementation of pthread_create(3) stores to pthread_t *thread before
> > > executing start_routine.
> > >
> > > but strictly speaking i don't think the pthread_create api contractually
> > > guarantees that the thread id is stored before start_routine runs. so this
> > > is relying on an internal implementation detail.
> > >
> > > https://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_create.html
> > >
> > > "Upon successful completion, pthread_create() shall store the ID of the
> > > created thread in the location referenced by thread."
> > >
> > > https://man7.org/linux/man-pages/man3/pthread_create.3.html
> > >
> > > "Before returning, a successful call to pthread_create() stores
> > > the ID of the new thread in the buffer pointed to by thread; this
> > > identifier is used to refer to the thread in subsequent calls to
> > > other pthreads functions."
> > >
> > > it doesn't really say when it does this in relation to start_routine running.
> > > depends how hair splitty you want to be about it. but since you're revamping
> > > the code you might be interested in addressing it.
> >
> > I had wondered about this part too in the past.
> >
> > I don't see a reason to keep this loop (even considering baremetal,
> > since this code is within the linux implementation of EAL).
> > And this comment seems a good reason to cleanup the code (like simply
> > pass lcore_id via arg).
> >
> > Something like:
> >
> > Author: David Marchand <david.marchand@redhat.com>
> > Date: Thu Mar 24 11:29:46 2022 +0100
> >
> > eal: cleanup lcore hand-over from main thread
> >
> > As noted by Tyler, there is nothing in the pthread API that strictly
> > guarantees that the new thread won't start running eal_thread_loop
> > before pthread_create writes to &lcore_config[xx].thread_id.
> >
> > Rather than rely on thread id, the main thread can directly pass the
> > worker thread lcore.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >
> > diff --git a/lib/eal/common/eal_common_thread.c
> > b/lib/eal/common/eal_common_thread.c
> > index 256de91abc..962b7e9ac4 100644
> > --- a/lib/eal/common/eal_common_thread.c
> > +++ b/lib/eal/common/eal_common_thread.c
> > @@ -166,26 +166,17 @@ __rte_thread_uninit(void)
> >
> > /* main loop of threads */
> > __rte_noreturn void *
> > -eal_thread_loop(__rte_unused void *arg)
> > +eal_thread_loop(void *arg)
> > {
> > + unsigned int lcore_id = (uintptr_t)arg;
> > char cpuset[RTE_CPU_AFFINITY_STR_LEN];
> > - pthread_t thread_id = pthread_self();
> > - unsigned int lcore_id;
> > int ret;
> >
> > - /* retrieve our lcore_id from the configuration structure */
> > - RTE_LCORE_FOREACH_WORKER(lcore_id) {
> > - if (thread_id == lcore_config[lcore_id].thread_id)
> > - break;
> > - }
> > - if (lcore_id == RTE_MAX_LCORE)
> > - rte_panic("cannot retrieve lcore id\n");
> > -
> > __rte_thread_init(lcore_id, &lcore_config[lcore_id].cpuset);
> >
> > ret = eal_thread_dump_current_affinity(cpuset, sizeof(cpuset));
> > RTE_LOG(DEBUG, EAL, "lcore %u is ready (tid=%zx;cpuset=[%s%s])\n",
> > - lcore_id, (uintptr_t)thread_id, cpuset,
> > + lcore_id, (uintptr_t)pthread_self(), cpuset,
> > ret == 0 ? "" : "...");
> >
> > rte_eal_trace_thread_lcore_ready(lcore_id, cpuset);
> > diff --git a/lib/eal/common/eal_thread.h b/lib/eal/common/eal_thread.h
> > index b08dcf34b5..0fde33e70c 100644
> > --- a/lib/eal/common/eal_thread.h
> > +++ b/lib/eal/common/eal_thread.h
> > @@ -11,7 +11,7 @@
> > * basic loop of thread, called for each thread by eal_init().
> > *
> > * @param arg
> > - * opaque pointer
> > + * The lcore_id (passed as an integer) of this worker thread.
> > */
> > __rte_noreturn void *eal_thread_loop(void *arg);
> >
> > diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
> > index 80bc3d25e0..a6b20960f2 100644
> > --- a/lib/eal/freebsd/eal.c
> > +++ b/lib/eal/freebsd/eal.c
> > @@ -810,7 +810,7 @@ rte_eal_init(int argc, char **argv)
> >
> > /* create a thread for each lcore */
> > ret = pthread_create(&lcore_config[i].thread_id, NULL,
> > - eal_thread_loop, NULL);
> > + eal_thread_loop, (void *)(uintptr_t)i);
> > if (ret != 0)
> > rte_panic("Cannot create thread\n");
> >
> > diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
> > index 8a405d1d59..1ef263434a 100644
> > --- a/lib/eal/linux/eal.c
> > +++ b/lib/eal/linux/eal.c
> > @@ -1145,7 +1145,7 @@ rte_eal_init(int argc, char **argv)
> >
> > /* create a thread for each lcore */
> > ret = pthread_create(&lcore_config[i].thread_id, NULL,
> > - eal_thread_loop, NULL);
> > + eal_thread_loop, (void *)(uintptr_t)i);
> > if (ret != 0)
> > rte_panic("Cannot create thread\n");
> >
> > diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
> > index ca3c41aaa7..1874f9f6d7 100644
> > --- a/lib/eal/windows/eal.c
> > +++ b/lib/eal/windows/eal.c
> > @@ -420,7 +420,7 @@ rte_eal_init(int argc, char **argv)
> > lcore_config[i].state = WAIT;
> >
> > /* create a thread for each lcore */
> > - if (eal_thread_create(&lcore_config[i].thread_id) != 0)
> > + if (eal_thread_create(&lcore_config[i].thread_id, i) != 0)
> > rte_panic("Cannot create thread\n");
> > ret = pthread_setaffinity_np(lcore_config[i].thread_id,
> > sizeof(rte_cpuset_t), &lcore_config[i].cpuset);
> > diff --git a/lib/eal/windows/eal_thread.c b/lib/eal/windows/eal_thread.c
> > index de1c0078a5..704781a83c 100644
> > --- a/lib/eal/windows/eal_thread.c
> > +++ b/lib/eal/windows/eal_thread.c
> > @@ -71,13 +71,14 @@ eal_thread_ack_command(void)
> >
> > /* function to create threads */
> > int
> > -eal_thread_create(pthread_t *thread)
> > +eal_thread_create(pthread_t *thread, unsigned int lcore_id)
> > {
> > HANDLE th;
> >
> > th = CreateThread(NULL, 0,
> > (LPTHREAD_START_ROUTINE)(ULONG_PTR)eal_thread_loop,
> > - NULL, 0, (LPDWORD)thread);
> > + (LPVOID)(uintptr_t)lcore_id, 0,
> > + (LPDWORD)thread);
> > if (!th)
> > return -1;
> >
> >
> >
> > But seeing how this code has been there from day 1, I would not
> > request a backport.
>
> this looks better to me it ends up being a bit less code and it solves
> the problem in a general fashion for platforms including windows.
>
> on windows the implementation does run the start_routine before assigning
> thread which was addressed with this patch. (still not merged)
> http://patchwork.dpdk.org/project/dpdk/list/?series=22094
>
> it's likely your patch will be merged before mine so when that happens
> i'll just quietly abandon mine. however if some desire exists for a
> backport the simpler patch i provided could be used.
Your patch could be merged now that we start a new cycle.
What do you prefer? Is David's solution better?
In this case, should we reject your patch?
next prev parent reply other threads:[~2022-03-25 14:58 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-23 9:30 David Marchand
2022-03-23 12:01 ` Morten Brørup
2022-03-24 8:31 ` Tyler Retzlaff
2022-03-24 14:44 ` David Marchand
2022-03-25 12:11 ` Tyler Retzlaff
2022-03-25 14:58 ` Thomas Monjalon [this message]
2022-03-25 15:09 ` David Marchand
2022-03-25 16:38 ` Tyler Retzlaff
2022-03-25 12:23 ` Tyler Retzlaff
2022-04-01 8:44 ` [PATCH v2 1/2] eal: cleanup lcore ID hand-over David Marchand
2022-04-01 8:44 ` [PATCH v2 2/2] eal: factorize lcore main loop David Marchand
2022-04-05 7:05 ` David Marchand
2022-04-05 16:34 ` [PATCH v3 1/2] eal: cleanup lcore ID hand-over David Marchand
2022-04-05 16:34 ` [PATCH v3 2/2] eal: factorize lcore main loop David Marchand
2022-04-14 11:48 ` David Marchand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2027001.KlZ2vcFHjT@thomas \
--to=thomas@monjalon.net \
--cc=bruce.richardson@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=dmitry.kozliuk@gmail.com \
--cc=dmitrym@microsoft.com \
--cc=navasile@linux.microsoft.com \
--cc=pallavi.kadam@intel.com \
--cc=roretzla@linux.microsoft.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).