Re: [dpdk-dev] eventdev: method for finding out unlink status

["Van Haaren, Harry" <harry.van.haaren@intel.com>]

> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Friday, August 10, 2018 3:52 PM
> To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com>
> Cc: Van Haaren, Harry <harry.van.haaren@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] eventdev: method for finding out unlink status
> 
> -----Original Message-----
> > Date: Fri, 10 Aug 2018 14:24:02 +0000
> > From: "Elo, Matias (Nokia - FI/Espoo)" <matias.elo@nokia.com>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > CC: "Van Haaren, Harry" <harry.van.haaren@intel.com>, "dev@dpdk.org"
> >  <dev@dpdk.org>
> > Subject: Re: [dpdk-dev] eventdev: method for finding out unlink status
> > x-mailer: Apple Mail (2.3445.9.1)
> >
> >
> > >
> > > # Other than that, I am still not able to understand, why not
> > > application wait until rte_event_port_unlink() returns.
> >
> > Making rte_event_port_unlink() blocking would be troublesome if one
> doesn’t care
> > about unlink completion. E.g. doing dynamic load balancing.
> 
> By making it as blocking(i.e the rte_event_port_unlink() returns when
> unlink() completed) forcing everyone to care about unlink completion.
> Right?

I'm not sure I understand the issue here.
Is anybody suggesting to make unlink() blocking?

For certain PMDs, perhaps it must be a synchronous handled unlink().
For other PMDs (eg event/sw) there are multiple threads involved,
so it must be async. Hence, APIs should be async to avoid blocking the caller.

With an async API, if you don't want the async behaviuor, it is
easy to build the sync version: call it in a loop, optionally with a delay().


> > > # What in real word use case, application can, do other than waiting
> > > to complete rte_event_port_unlink(). If we try to put some logic in
> like,
> > >
> > > while (rte_event_port_unlink_in_progress(dev, port) > 0){
> > >       do_something();
> > > }
> > >
> > > The do_something() will not be called in some platform at all.
> > >
> > > # Any idea on what will be the real world use case, where
> rte_event_port_unlink() called in fastpath?
> >
> > In our application this could be used for example to pause scheduling of
> new events while
> > working on an “expensive” event to minimise delays. It is also needed when
> destroying
> > queues, though calling this fast path is debatable (our application
> enables creating /
> > destroying queues at runtime).
> 
> If I understand it correctly, Your current issue is, SW driver is
> not waiting for to complete the unlink() operation so that in your
> application you are seeing some abnormalities.

To be more specific:

The issue is that the application cannot reliably know when the unlink()
has completed. As such, the application doesn't know when it can put
a core to sleep instead of busy polling.

Waiting for unlink() to return does not give this info - the
scheduler core may schedule more events until it "acks" the unlink.

Checking that dequeue() == 0 events isn't reliable either, the buffering
in the PMD could hide events.

It seems logical to add an API that allows the user to query the status
of the PMDs unlink() progress. This provides the application with the
information that it needs to reliably determine when a worker is no
longer being scheduled events of a specific type, at which point it
can put its worker to sleep.


> > These are perhaps not the best examples but I would be very cautious to
> make a function
> > blocking if there is even a small probability that it could be called from
> the fast path.
> 
> Let assume even if it is called in fastpath, what else, we can really do
> other that calling rte_pause() in loop. realistically? after issuing
> unlink() operation.

A CPU core could be polling two queues:

for_each(queue) {
  if (unlinks_in_progress() != 0)
    continue;
  // poll queue and handle packets
}

or we could have the unlinks in progress flush buffers:

if (unlinks_in_progress() != 0) {
  tx_flush_packets();
}

I don't think that making unlinks_in_progress() blocking is acceptable,
it feels like too strong a limitation.

If a PMD doesn't support unlinking, we can have the eventdev.c layer
return an error code (or zero..?) to avoid burden on PMDs that don't care.


I've detailed why I think overloading the return value of unlink()
(eg to -EBUSY) isn't a good solution here:
http://mails.dpdk.org/archives/dev/2018-August/109550.html