Re: [dpdk-dev] [RFC] service: stop lcore threads before 'finalize'

[David Marchand <david.marchand@redhat.com>]

On Fri, Feb 21, 2020 at 1:28 PM Van Haaren, Harry
<harry.van.haaren@intel.com> wrote:
>
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Thursday, February 20, 2020 1:25 PM
> > To: Van Haaren, Harry <harry.van.haaren@intel.com>
> > Cc: Aaron Conole <aconole@redhat.com>; dev <dev@dpdk.org>
> > Subject: Re: [RFC] service: stop lcore threads before 'finalize'
> >
> > On Mon, Feb 10, 2020 at 3:16 PM Van Haaren, Harry
> > <harry.van.haaren@intel.com> wrote:
> > > > > We need a fix for this issue.
> > > >
> > > > +1
> > >
> > > > > Interestingly, Stephen patch that joins all pthreads at
> > > > > rte_eal_cleanup [1] makes this issue disappear.
> > > > > So my understanding is that we are missing a api (well, I could not
> > > > > find a way) to synchronously stop service lcores.
> > > >
> > > > Maybe we can take that patch as a fix.  I hate to see this segfault
> > > > in the field.  I need to figure out what I missed in my cleanup
> > > > (probably missed a synchronization point).
> > >
> > > I haven't easily reproduced this yet - so I'll investigate a way to
> > > reproduce with close to 100% rate, then we can identify the root cause
> > > and actually get a clean fix. If you have pointers to reproduce easily,
> > > please let me know.
> > >
> >
> > ping.
> > I want a fix in 20.05, or I will start considering how to drop this thing.
>
> Hi David,
>
> I have been attempting to reproduce, unfortunately without success.
>
> Attempted you suggested meson test approach (thanks for suggesting!), but
> I haven't had a segfault with that approach (yet, and its done a lot of iterations..)

I reproduced it on the first try, just now.
Travis catches it every once in a while (look at the ovsrobot).

For the reproduction, this is on my laptop (core i7-8650U), baremetal,
no fancy stuff.
FWIW, the cores are ruled by the "powersave" governor.
I can see the frequency oscillates between 3.5GHz and 3.7Ghz while the
max frequency is 4.2GHz.

Travis runs virtual machines with 2 cores, and there must be quite
some overprovisioning on those servers.
We can expect some cycles being stolen or at least something happening
on the various cores.


>
> I've made the service-cores unit tests delay before exit, in an attempt
> to have them access previously rte_free()-ed memory, no luck to reproduce.

Ok, let's forget about the segfault, what do you think of the
backtrace I caught?
A service lcore thread is still in the service loop.
The master thread of the application is in the libc exiting code.

This is what I get in all crashes.


>
> Thinking perhaps we need it on exit, I've also POCed a unit test that leaves
> service cores active on exit on purpose, to try have them poll after exit,
> still no luck.
>
> Simplifying the problem, and using hello-world sample app with a rte_eal_cleaup()
> call at the end also doesn't easily aggravate the problem.
>
> From code inspection, I agree there is an issue. It seems like a call to
> rte_service_lcore_reset_all() from rte_service_finalize() is enough...
> But without reproducing it is hard to have good confidence in a fix.

You promised a doc update on the services API.
Thanks.


--
David Marchand