From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 9070D4F93 for ; Mon, 30 Jul 2018 11:38:23 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Jul 2018 02:38:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,422,1526367600"; d="scan'208";a="76685332" Received: from irsmsx106.ger.corp.intel.com ([163.33.3.31]) by fmsmga001.fm.intel.com with ESMTP; 30 Jul 2018 02:38:19 -0700 Received: from irsmsx156.ger.corp.intel.com (10.108.20.68) by IRSMSX106.ger.corp.intel.com (163.33.3.31) with Microsoft SMTP Server (TLS) id 14.3.319.2; Mon, 30 Jul 2018 10:38:02 +0100 Received: from irsmsx102.ger.corp.intel.com ([169.254.2.110]) by IRSMSX156.ger.corp.intel.com ([169.254.3.205]) with mapi id 14.03.0319.002; Mon, 30 Jul 2018 10:38:02 +0100 From: "Van Haaren, Harry" To: Jerin Jacob , "Elo, Matias (Nokia - FI/Espoo)" CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] eventdev: method for finding out unlink status Thread-Index: AQHUJ9Aasi5EGf77JEeq3+fzpPRPXKSnVJsAgAAXXYCAAAM+gIAAETgQ Date: Mon, 30 Jul 2018 09:38:01 +0000 Message-ID: References: <20180730075408.GA14117@jerin> <80CC5C07-0D73-4F86-9F93-0AB78DEF2BFD@nokia.com> <20180730092921.GA22242@jerin> In-Reply-To: <20180730092921.GA22242@jerin> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNDM0NGY5OWMtYzBiMy00YTJlLWE5ZmQtZTUzYTU3ZDY3MjQ3IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiT09FclZ0bnYzMEl6TjBqTEtwcVwvZk9EdHBlRlJWUmVBcDhHR244XC9Bc3hFcE0ra0dWVW5IdDI5QW9hMHE2K00rIn0= x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] eventdev: method for finding out unlink status X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 09:38:24 -0000 > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > Sent: Monday, July 30, 2018 10:29 AM > To: Elo, Matias (Nokia - FI/Espoo) > Cc: dev@dpdk.org; Van Haaren, Harry > Subject: Re: [dpdk-dev] eventdev: method for finding out unlink status >=20 > -----Original Message----- > > Date: Mon, 30 Jul 2018 09:17:47 +0000 > > From: "Elo, Matias (Nokia - FI/Espoo)" > > To: Jerin Jacob > > CC: "dev@dpdk.org" , "Van Haaren, Harry" > > > > Subject: Re: [dpdk-dev] eventdev: method for finding out unlink status > > x-mailer: Apple Mail (2.3445.9.1) > > > > > > >> > > >> In bug report https://bugs.dpdk.org/show_bug.cgi?id=3D60 we have bee= n > discussing > > >> issues related to events ending up in wrong ports after calling > > >> rte_event_port_unlink(). In addition of finding few bugs we have > identified a > > >> need for a new API call (or documentation extension) for an applicat= ion > to be > > > > > > From HW perspective, documentation extension should be enough. adding > > > "there may be pre-scheduled events and the application is responsible= to > process them" > > > on unlink(). Since dequeue() has which queue it is dequeue-ed from, t= he > > > application can allays make action based on that(i.e, Is the event > > > post/pre to unlink) > > > > At least in case of SW eventdev the problem is how the application can = know > that > > it has processed all pre-scheduled events. E.g. dequeue may return noth= ing > but since > > the scheduler is running as a separate process events may still end up = to > the unlinked > > port asynchronously. >=20 > Can't we do, dequeue() in loop to get all the events from port. If > dequeue returns with zero event then ports are drained up. Right? Nope - because the scheduler might not have performed and "Acked" the unlink(), and internally it has *just* scheduled an event, but it wasn't available in the dequeue ring yet. Aka, its racy behavior - and we need a way to retrieve this "Unlink Ack" from the scheduler (which runs in another thread in event/sw). > > >> able to find out when an unlink() call has finished and no new event= s are > > >> scheduled anymore to the particular event port. This is required e.g= . > when doing > > >> clean-up after an application thread stops processing events. > > > > > > If thread stopping then it better to call dev_stop(). At least in HW > > > implementation, > > > > For an application doing dynamic load balancing stopping the whole even= tdev > is not an > > option. >=20 > OK. Makes sense. Doing unlink() and link() in fastpath is not a > problem. Correct > Changing core assignment to event port is problem without stop(). I > guess, you > application or general would be OK with that constraint. I don't think that the eventdev API requires 1:1 Lcore / Port mapping, so r= eally a PMD should be able to handle any thread calling any port. The event/sw PMD allows any thread to call dequeue/enqueue any port, so long as it is not being accessed by another thread. > > > A given event port assigned to a new lcore other than > > > it previous one then we need to do some clean up at port level. > > > > In my case I'm mapping an event port per thread statically (basically > thread_id =3D=3D port_id), > > so this shouldn't be an issue. This is the common case - but I don't think we should demand it. There is a valid scale-down model which just polls *all* ports using a single lcore, instead of unlink() of multiple ports. For this "runtime scale down" use-case the missing information is being able to identify when an unlink is complete. After that (and ensuring the port buffer is empty) the application can be guaranteed that there are no more events going to be sent to that port, and the application can take the worker lcore out of its polling-loop and put it to sleep. As mentioned before, I think an "unlinks_in_progress()" function is perhaps the easiest way to achieve this functionality, as it allows relatively simp= le tracking of unlinks() using an atomic counter in sw. (Implementation detail= s become complex when we have a separate core running event/sw, separate core= s polling, and a control-plane thread calling unlink...)=20 I think the end result we're hoping for is something like pseudo code below= , (keep in mind that the event/sw has a service-core thread running it, so no application code there): int worker_poll =3D 1; worker() { while(worker_poll) { // eventdev_dequeue_burst() etc } go_to_sleep(1); } control_plane_scale_down() { unlink(evdev, worker, queue_id); while(unlinks_in_progress(evdev) > 0) usleep(100); =20 /* here we know that the unlink is complete. * so we can now stop the worker from polling */ worker_poll =3D 0; } Hope my pseudo-code makes pseudo-sense :) -Harry