DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Eads, Gage" <gage.eads@intel.com>
To: "Van Haaren, Harry" <harry.van.haaren@intel.com>,
	"dev@dpdk.org" <dev@dpdk.org>
Cc: "jerin.jacob@caviumnetworks.com" <jerin.jacob@caviumnetworks.com>,
	"hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>,
	"Richardson, Bruce" <bruce.richardson@intel.com>,
	"santosh.shukla@caviumnetworks.com"
	<santosh.shukla@caviumnetworks.com>,
	"nipun.gupta@nxp.com" <nipun.gupta@nxp.com>
Subject: Re: [dpdk-dev] [PATCH v4 1/2] eventdev: add device stop flush callback
Date: Mon, 26 Mar 2018 21:59:25 +0000	[thread overview]
Message-ID: <9184057F7FC11744A2107296B6B8EB1E3FA88B42@FMSMSX108.amr.corp.intel.com> (raw)
In-Reply-To: <E923DB57A917B54B9182A2E928D00FA65E01640A@IRSMSX101.ger.corp.intel.com>



> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Friday, March 23, 2018 11:57 AM
> To: Eads, Gage <gage.eads@intel.com>; dev@dpdk.org
> Cc: jerin.jacob@caviumnetworks.com; hemant.agrawal@nxp.com; Richardson,
> Bruce <bruce.richardson@intel.com>; santosh.shukla@caviumnetworks.com;
> nipun.gupta@nxp.com
> Subject: RE: [PATCH v4 1/2] eventdev: add device stop flush callback
> 
> > From: Eads, Gage
> > Sent: Tuesday, March 20, 2018 2:13 PM
> > To: dev@dpdk.org
> > Cc: jerin.jacob@caviumnetworks.com; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; hemant.agrawal@nxp.com; Richardson,
> > Bruce <bruce.richardson@intel.com>; santosh.shukla@caviumnetworks.com;
> > nipun.gupta@nxp.com
> > Subject: [PATCH v4 1/2] eventdev: add device stop flush callback
> >
> > When an event device is stopped, it drains all event queues. These
> > events may contain pointers, so to prevent memory leaks eventdev now
> > supports a user-provided flush callback that is called during the queue drain
> process.
> > This callback is stored in process memory, so the callback must be
> > registered by any process that may call rte_event_dev_stop().
> >
> > This commit also clarifies the behavior of rte_event_dev_stop().
> >
> > This follows this mailing list discussion:
> > http://dpdk.org/ml/archives/dev/2018-January/087484.html
> >
> > Signed-off-by: Gage Eads <gage.eads@intel.com>
> 
> <snip most of the code - looks good!>
> 
> >  /**
> > - * Stop an event device. The device can be restarted with a call to
> > - * rte_event_dev_start()
> > + * Stop an event device.
> > + *
> > + * This function causes all queued events to be drained. While
> > + draining
> > events
> > + * out of the device, this function calls the user-provided flush
> > + callback
> > + * (if one was registered) once per event.
> > + *
> > + * This function does not drain events from event ports; the
> > + application is
> > + * responsible for flushing events from all ports before stopping the
> > device.
> 
> 
> Question about how an application is expected to correctly cleanup all the
> events here. Note in particular the last part: "application is responsible for
> flushing events from all ports **BEFORE** stopping the device".
> 
> Given the event device is still running, how can the application be sure it has
> flushed all the events (from the dequeue side in particular)?
> 

Appreciate the feedback -- good points all around.

I was expecting that the application would unlink queues from the ports, and then dequeue until each port has no events. However, there are PMDs for which runtime port link/unlink is not supported, so I see that this is not a viable approach. Plus, this adds the application burden that you describe below.

> 
> In order to drain all events from the ports, I was expecting the following:
> 
> // stop scheduling new events to worker cores
> rte_event_dev_stop()
> ---> callback gets called for each event
> 
> // to dequeue events from each port, and app cleans them up?
> FOR_EACH_PORT( rte_event_dev_dequeue(..., port_id, ...) )
> 
> 
> I'd like to avoid the dequeue-each-port() approach in application, as it adds extra
> burden to clean up correctly...

Agreed, but for a different reason: that approach means we'd have to change the documented eventdev behavior. rte_eventdev.h states that the "schedule, enqueue and dequeue functions should not be invoked when the device is stopped," and this patch reiterates that in the rte_event_dev_stop() documentation ("Threads that continue to enqueue/dequeue while the device is stopped, or being stopped, will result in undefined behavior"). Since a PMD's stop cleanup code could just be repeated calls to a PMD's dequeue code, allowing applications to dequeue simultaneously could be troublesome.

> 
> What if we say that dequeue() returns zero after stop() (leaving events possibly
> in the port-dequeue side SW buffers), and these events which were about to be
> dequeued by the worker core are also passed to the dev_stop_flush callback?

I'd prefer to have dequeue-while-stopped be unsupported, so we don't need an additional check or synchronization in the datapath, but passing the events in a port to the callback should work (for the sw PMD, at least). How does that sound?

Thanks,
Gage

  reply	other threads:[~2018-03-26 21:59 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-05 23:01 [dpdk-dev] [PATCH " Gage Eads
2018-03-05 23:01 ` [dpdk-dev] [PATCH 2/2] event/sw: support " Gage Eads
2018-03-08 23:10 ` [dpdk-dev] [PATCH v2 1/2] eventdev: add " Gage Eads
2018-03-08 23:10   ` [dpdk-dev] [PATCH v2 2/2] event/sw: support " Gage Eads
2018-03-12  6:25   ` [dpdk-dev] [PATCH v2 1/2] eventdev: add " Jerin Jacob
2018-03-12 14:30     ` Eads, Gage
2018-03-12 14:38       ` Jerin Jacob
2018-03-15  4:12   ` [dpdk-dev] [PATCH v3 " Gage Eads
2018-03-15  4:12     ` [dpdk-dev] [PATCH v3 2/2] event/sw: support " Gage Eads
2018-03-20  7:44     ` [dpdk-dev] [PATCH v3 1/2] eventdev: add " Jerin Jacob
2018-03-20 14:11       ` Eads, Gage
2018-03-20 14:13     ` [dpdk-dev] [PATCH v4 " Gage Eads
2018-03-20 14:13       ` [dpdk-dev] [PATCH v4 2/2] event/sw: support " Gage Eads
2018-03-23 17:05         ` Van Haaren, Harry
2018-03-26 22:01           ` Eads, Gage
2018-04-02  8:03             ` Jerin Jacob
2018-04-02 15:50               ` Eads, Gage
2018-04-02 17:08                 ` Jerin Jacob
2018-03-23 16:57       ` [dpdk-dev] [PATCH v4 1/2] eventdev: add " Van Haaren, Harry
2018-03-26 21:59         ` Eads, Gage [this message]
2018-03-27  8:20           ` Van Haaren, Harry
2018-03-29 11:02             ` Van Haaren, Harry
2018-03-29 18:34               ` Jerin Jacob
2018-03-30  9:54               ` Liang, Ma
2018-04-02  8:01       ` Jerin Jacob
2018-04-02 18:03       ` [dpdk-dev] [PATCH v5] " Gage Eads
2018-04-03  1:26         ` Jerin Jacob
2018-04-03  1:31           ` Jerin Jacob

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9184057F7FC11744A2107296B6B8EB1E3FA88B42@FMSMSX108.amr.corp.intel.com \
    --to=gage.eads@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    --cc=harry.van.haaren@intel.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=jerin.jacob@caviumnetworks.com \
    --cc=nipun.gupta@nxp.com \
    --cc=santosh.shukla@caviumnetworks.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).