From: "Liang, Ma" <liang.j.ma@intel.com>
To: "Van Haaren, Harry" <harry.van.haaren@intel.com>
Cc: "jerin.jacob@caviumnetworks.com" <jerin.jacob@caviumnetworks.com>,
"hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>,
"Eads, Gage" <gage.eads@intel.com>,
"Richardson, Bruce" <bruce.richardson@intel.com>,
"santosh.shukla@caviumnetworks.com"
<santosh.shukla@caviumnetworks.com>,
"nipun.gupta@nxp.com" <nipun.gupta@nxp.com>,
"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v4 1/2] eventdev: add device stop flush callback
Date: Fri, 30 Mar 2018 10:54:14 +0100 [thread overview]
Message-ID: <20180330095413.GA21722@sivswdev01.ir.intel.com> (raw)
In-Reply-To: <E923DB57A917B54B9182A2E928D00FA65E018D38@IRSMSX101.ger.corp.intel.com>
On 29 Mar 04:02, Van Haaren, Harry wrote:
> (+Liang Ma for OPDL maintainer)
>
> Ping to maintainers, is the below suggestion acceptable for your PMDs?
>
> Summary of suggestion:
> - After event_dev_stop() dequeue() is no longer allowed on any thread
> - All events in the system (queues, ports, intermediary buffers) will be passed to a user-supplied callback for cleaning up events.
+1, this should work for OPDL
>
>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Van Haaren, Harry
> > Sent: Tuesday, March 27, 2018 9:20 AM
> > To: Eads, Gage <gage.eads@intel.com>; jerin.jacob@caviumnetworks.com;
> > hemant.agrawal@nxp.com
> > Cc: Richardson, Bruce <bruce.richardson@intel.com>;
> > santosh.shukla@caviumnetworks.com; nipun.gupta@nxp.com; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v4 1/2] eventdev: add device stop flush
> > callback
> >
> > > From: Eads, Gage
> > > Sent: Monday, March 26, 2018 10:59 PM
> > > To: Van Haaren, Harry <harry.van.haaren@intel.com>; dev@dpdk.org
> > > Cc: jerin.jacob@caviumnetworks.com; hemant.agrawal@nxp.com; Richardson,
> > Bruce
> > > <bruce.richardson@intel.com>; santosh.shukla@caviumnetworks.com;
> > > nipun.gupta@nxp.com
> > > Subject: RE: [PATCH v4 1/2] eventdev: add device stop flush callback
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Van Haaren, Harry
> > > > Sent: Friday, March 23, 2018 11:57 AM
> > > > To: Eads, Gage <gage.eads@intel.com>; dev@dpdk.org
> > > > Cc: jerin.jacob@caviumnetworks.com; hemant.agrawal@nxp.com; Richardson,
> > > > Bruce <bruce.richardson@intel.com>; santosh.shukla@caviumnetworks.com;
> > > > nipun.gupta@nxp.com
> > > > Subject: RE: [PATCH v4 1/2] eventdev: add device stop flush callback
> > > >
> > > > > From: Eads, Gage
> > > > > Sent: Tuesday, March 20, 2018 2:13 PM
> > > > > To: dev@dpdk.org
> > > > > Cc: jerin.jacob@caviumnetworks.com; Van Haaren, Harry
> > > > > <harry.van.haaren@intel.com>; hemant.agrawal@nxp.com; Richardson,
> > > > > Bruce <bruce.richardson@intel.com>; santosh.shukla@caviumnetworks.com;
> > > > > nipun.gupta@nxp.com
> > > > > Subject: [PATCH v4 1/2] eventdev: add device stop flush callback
> > > > >
> > > > > When an event device is stopped, it drains all event queues. These
> > > > > events may contain pointers, so to prevent memory leaks eventdev now
> > > > > supports a user-provided flush callback that is called during the
> > queue
> > > drain
> > > > process.
> > > > > This callback is stored in process memory, so the callback must be
> > > > > registered by any process that may call rte_event_dev_stop().
> > > > >
> > > > > This commit also clarifies the behavior of rte_event_dev_stop().
> > > > >
> > > > > This follows this mailing list discussion:
> > > > > http://dpdk.org/ml/archives/dev/2018-January/087484.html
> > > > >
> > > > > Signed-off-by: Gage Eads <gage.eads@intel.com>
> > > >
> > > > <snip most of the code - looks good!>
> > > >
> > > > > /**
> > > > > - * Stop an event device. The device can be restarted with a call to
> > > > > - * rte_event_dev_start()
> > > > > + * Stop an event device.
> > > > > + *
> > > > > + * This function causes all queued events to be drained. While
> > > > > + draining
> > > > > events
> > > > > + * out of the device, this function calls the user-provided flush
> > > > > + callback
> > > > > + * (if one was registered) once per event.
> > > > > + *
> > > > > + * This function does not drain events from event ports; the
> > > > > + application is
> > > > > + * responsible for flushing events from all ports before stopping the
> > > > > device.
> > > >
> > > >
> > > > Question about how an application is expected to correctly cleanup all
> > the
> > > > events here. Note in particular the last part: "application is
> > responsible
> > > for
> > > > flushing events from all ports **BEFORE** stopping the device".
> > > >
> > > > Given the event device is still running, how can the application be sure
> > it
> > > has
> > > > flushed all the events (from the dequeue side in particular)?
> > > >
> > >
> > > Appreciate the feedback -- good points all around.
> > >
> > > I was expecting that the application would unlink queues from the ports,
> > and
> > > then dequeue until each port has no events. However, there are PMDs for
> > which
> > > runtime port link/unlink is not supported, so I see that this is not a
> > viable
> > > approach. Plus, this adds the application burden that you describe below.
> >
> > +1.
> >
> > > >
> > > > In order to drain all events from the ports, I was expecting the
> > following:
> > > >
> > > > // stop scheduling new events to worker cores
> > > > rte_event_dev_stop()
> > > > ---> callback gets called for each event
> > > >
> > > > // to dequeue events from each port, and app cleans them up?
> > > > FOR_EACH_PORT( rte_event_dev_dequeue(..., port_id, ...) )
> > > >
> > > >
> > > > I'd like to avoid the dequeue-each-port() approach in application, as it
> > > adds extra
> > > > burden to clean up correctly...
> > >
> > > Agreed, but for a different reason: that approach means we'd have to
> > change
> > > the documented eventdev behavior. rte_eventdev.h states that the
> > "schedule,
> > > enqueue and dequeue functions should not be invoked when the device is
> > > stopped," and this patch reiterates that in the rte_event_dev_stop()
> > > documentation ("Threads that continue to enqueue/dequeue while the device
> > is
> > > stopped, or being stopped, will result in undefined behavior"). Since a
> > PMD's
> > > stop cleanup code could just be repeated calls to a PMD's dequeue code,
> > > allowing applications to dequeue simultaneously could be troublesome.
> >
> > All +1 too, good point about the header stating it is undefined behavior.
> >
> >
> > > > What if we say that dequeue() returns zero after stop() (leaving events
> > > possibly
> > > > in the port-dequeue side SW buffers), and these events which were about
> > to
> > > be
> > > > dequeued by the worker core are also passed to the dev_stop_flush
> > callback?
> > >
> > > I'd prefer to have dequeue-while-stopped be unsupported, so we don't need
> > an
> > > additional check or synchronization in the datapath, but passing the
> > events in
> > > a port to the callback should work (for the sw PMD, at least). How does
> > that
> > > sound?
> >
> >
> > That's fine with me, both from design point of view, and SW PMD.
> >
> > @HW PMD maintainers, would the above approach work for you?
> >
> >
> >
>
next prev parent reply other threads:[~2018-03-30 9:54 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-03-05 23:01 [dpdk-dev] [PATCH " Gage Eads
2018-03-05 23:01 ` [dpdk-dev] [PATCH 2/2] event/sw: support " Gage Eads
2018-03-08 23:10 ` [dpdk-dev] [PATCH v2 1/2] eventdev: add " Gage Eads
2018-03-08 23:10 ` [dpdk-dev] [PATCH v2 2/2] event/sw: support " Gage Eads
2018-03-12 6:25 ` [dpdk-dev] [PATCH v2 1/2] eventdev: add " Jerin Jacob
2018-03-12 14:30 ` Eads, Gage
2018-03-12 14:38 ` Jerin Jacob
2018-03-15 4:12 ` [dpdk-dev] [PATCH v3 " Gage Eads
2018-03-15 4:12 ` [dpdk-dev] [PATCH v3 2/2] event/sw: support " Gage Eads
2018-03-20 7:44 ` [dpdk-dev] [PATCH v3 1/2] eventdev: add " Jerin Jacob
2018-03-20 14:11 ` Eads, Gage
2018-03-20 14:13 ` [dpdk-dev] [PATCH v4 " Gage Eads
2018-03-20 14:13 ` [dpdk-dev] [PATCH v4 2/2] event/sw: support " Gage Eads
2018-03-23 17:05 ` Van Haaren, Harry
2018-03-26 22:01 ` Eads, Gage
2018-04-02 8:03 ` Jerin Jacob
2018-04-02 15:50 ` Eads, Gage
2018-04-02 17:08 ` Jerin Jacob
2018-03-23 16:57 ` [dpdk-dev] [PATCH v4 1/2] eventdev: add " Van Haaren, Harry
2018-03-26 21:59 ` Eads, Gage
2018-03-27 8:20 ` Van Haaren, Harry
2018-03-29 11:02 ` Van Haaren, Harry
2018-03-29 18:34 ` Jerin Jacob
2018-03-30 9:54 ` Liang, Ma [this message]
2018-04-02 8:01 ` Jerin Jacob
2018-04-02 18:03 ` [dpdk-dev] [PATCH v5] " Gage Eads
2018-04-03 1:26 ` Jerin Jacob
2018-04-03 1:31 ` Jerin Jacob
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180330095413.GA21722@sivswdev01.ir.intel.com \
--to=liang.j.ma@intel.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=gage.eads@intel.com \
--cc=harry.van.haaren@intel.com \
--cc=hemant.agrawal@nxp.com \
--cc=jerin.jacob@caviumnetworks.com \
--cc=nipun.gupta@nxp.com \
--cc=santosh.shukla@caviumnetworks.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).