From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id D93895F18 for ; Mon, 26 Mar 2018 23:59:30 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 26 Mar 2018 14:59:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,365,1517904000"; d="scan'208";a="38324015" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga003.jf.intel.com with ESMTP; 26 Mar 2018 14:59:27 -0700 Received: from fmsmsx121.amr.corp.intel.com (10.18.125.36) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.319.2; Mon, 26 Mar 2018 14:59:27 -0700 Received: from fmsmsx108.amr.corp.intel.com ([169.254.9.102]) by fmsmsx121.amr.corp.intel.com ([169.254.6.214]) with mapi id 14.03.0319.002; Mon, 26 Mar 2018 14:59:27 -0700 From: "Eads, Gage" To: "Van Haaren, Harry" , "dev@dpdk.org" CC: "jerin.jacob@caviumnetworks.com" , "hemant.agrawal@nxp.com" , "Richardson, Bruce" , "santosh.shukla@caviumnetworks.com" , "nipun.gupta@nxp.com" Thread-Topic: [PATCH v4 1/2] eventdev: add device stop flush callback Thread-Index: AQHTwFWqdT39rqZI0EybXoXPZCJB8qPeCboggAULi6A= Date: Mon, 26 Mar 2018 21:59:25 +0000 Message-ID: <9184057F7FC11744A2107296B6B8EB1E3FA88B42@FMSMSX108.amr.corp.intel.com> References: <1521087130-20244-1-git-send-email-gage.eads@intel.com> <1521555187-25710-1-git-send-email-gage.eads@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMjc5NzI4MjQtYjgzOC00OGJhLTgxYzItMDA0NWZmYjM5MDYwIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IldnbmorcjhDRVduUEMrVXYzOVVzM2hqWGxKcVdvMnVIRWFPUDF4RVVTYmM9In0= x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [10.1.200.108] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v4 1/2] eventdev: add device stop flush callback X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2018 21:59:31 -0000 > -----Original Message----- > From: Van Haaren, Harry > Sent: Friday, March 23, 2018 11:57 AM > To: Eads, Gage ; dev@dpdk.org > Cc: jerin.jacob@caviumnetworks.com; hemant.agrawal@nxp.com; Richardson, > Bruce ; santosh.shukla@caviumnetworks.com; > nipun.gupta@nxp.com > Subject: RE: [PATCH v4 1/2] eventdev: add device stop flush callback >=20 > > From: Eads, Gage > > Sent: Tuesday, March 20, 2018 2:13 PM > > To: dev@dpdk.org > > Cc: jerin.jacob@caviumnetworks.com; Van Haaren, Harry > > ; hemant.agrawal@nxp.com; Richardson, > > Bruce ; santosh.shukla@caviumnetworks.com; > > nipun.gupta@nxp.com > > Subject: [PATCH v4 1/2] eventdev: add device stop flush callback > > > > When an event device is stopped, it drains all event queues. These > > events may contain pointers, so to prevent memory leaks eventdev now > > supports a user-provided flush callback that is called during the queue= drain > process. > > This callback is stored in process memory, so the callback must be > > registered by any process that may call rte_event_dev_stop(). > > > > This commit also clarifies the behavior of rte_event_dev_stop(). > > > > This follows this mailing list discussion: > > http://dpdk.org/ml/archives/dev/2018-January/087484.html > > > > Signed-off-by: Gage Eads >=20 > >=20 > > /** > > - * Stop an event device. The device can be restarted with a call to > > - * rte_event_dev_start() > > + * Stop an event device. > > + * > > + * This function causes all queued events to be drained. While > > + draining > > events > > + * out of the device, this function calls the user-provided flush > > + callback > > + * (if one was registered) once per event. > > + * > > + * This function does not drain events from event ports; the > > + application is > > + * responsible for flushing events from all ports before stopping the > > device. >=20 >=20 > Question about how an application is expected to correctly cleanup all th= e > events here. Note in particular the last part: "application is responsibl= e for > flushing events from all ports **BEFORE** stopping the device". >=20 > Given the event device is still running, how can the application be sure = it has > flushed all the events (from the dequeue side in particular)? >=20 Appreciate the feedback -- good points all around. I was expecting that the application would unlink queues from the ports, an= d then dequeue until each port has no events. However, there are PMDs for w= hich runtime port link/unlink is not supported, so I see that this is not a= viable approach. Plus, this adds the application burden that you describe = below. >=20 > In order to drain all events from the ports, I was expecting the followin= g: >=20 > // stop scheduling new events to worker cores > rte_event_dev_stop() > ---> callback gets called for each event >=20 > // to dequeue events from each port, and app cleans them up? > FOR_EACH_PORT( rte_event_dev_dequeue(..., port_id, ...) ) >=20 >=20 > I'd like to avoid the dequeue-each-port() approach in application, as it = adds extra > burden to clean up correctly... Agreed, but for a different reason: that approach means we'd have to change= the documented eventdev behavior. rte_eventdev.h states that the "schedule= , enqueue and dequeue functions should not be invoked when the device is st= opped," and this patch reiterates that in the rte_event_dev_stop() document= ation ("Threads that continue to enqueue/dequeue while the device is stoppe= d, or being stopped, will result in undefined behavior"). Since a PMD's sto= p cleanup code could just be repeated calls to a PMD's dequeue code, allowi= ng applications to dequeue simultaneously could be troublesome. >=20 > What if we say that dequeue() returns zero after stop() (leaving events p= ossibly > in the port-dequeue side SW buffers), and these events which were about t= o be > dequeued by the worker core are also passed to the dev_stop_flush callbac= k? I'd prefer to have dequeue-while-stopped be unsupported, so we don't need a= n additional check or synchronization in the datapath, but passing the even= ts in a port to the callback should work (for the sw PMD, at least). How do= es that sound? Thanks, Gage