From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id C6AFE5938 for ; Mon, 16 Nov 2015 18:19:29 +0100 (CET) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga102.fm.intel.com with ESMTP; 16 Nov 2015 09:19:29 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,303,1444719600"; d="scan'208";a="601211660" Received: from irsmsx102.ger.corp.intel.com ([163.33.3.155]) by FMSMGA003.fm.intel.com with ESMTP; 16 Nov 2015 09:19:27 -0800 Received: from irsmsx156.ger.corp.intel.com (10.108.20.68) by IRSMSX102.ger.corp.intel.com (163.33.3.155) with Microsoft SMTP Server (TLS) id 14.3.248.2; Mon, 16 Nov 2015 17:19:26 +0000 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.203]) by IRSMSX156.ger.corp.intel.com ([169.254.3.110]) with mapi id 14.03.0248.002; Mon, 16 Nov 2015 17:19:26 +0000 From: "Ananyev, Konstantin" To: Stephen Hemminger , Thomas Monjalon Thread-Topic: [dpdk-dev] Recent changes related to interrupt thread Thread-Index: AQHRIGrgFV00nvxRM0qhKtV5eYD5HJ6eqiwAgAA3RACAAAH5QA== Date: Mon, 16 Nov 2015 17:19:25 +0000 Message-ID: <2601191342CEEE43887BDE71AB97725836AC961E@irsmsx105.ger.corp.intel.com> References: <20151116123200.GA2667@scalar.blr.asicdesigners.com> <8192567.2fdTdH6sjP@xps13> <20151116090630.0f0a9b27@samsung9> In-Reply-To: <20151116090630.0f0a9b27@samsung9> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.182] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" , Felix Marti , Nirranjan Kirubaharan , Kumar Sanghvi Subject: Re: [dpdk-dev] Recent changes related to interrupt thread X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 17:19:30 -0000 > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger > Sent: Monday, November 16, 2015 5:07 PM > To: Thomas Monjalon > Cc: dev@dpdk.org; Nirranjan Kirubaharan; Felix Marti; Kumar Sanghvi > Subject: Re: [dpdk-dev] Recent changes related to interrupt thread >=20 > On Mon, 16 Nov 2015 14:48:42 +0100 > Thomas Monjalon wrote: >=20 > > Hi, > > > > 2015-11-16 18:02, Rahul Lakkireddy: > > > Hi, > > > > > > I notice that the following changeset: > > > > > > Fixes: fd6949c55c9a ("eal: fix io permission for virtio interrupt > > > handler") > > > > > > has moved the initialization of the interrupt thread to after the mas= ter > > > lcore has been initialized. However, this causes the interrupt threa= d > > > to _inherit_ the affinity of the master lcore. Hence, this seems to > > > make all interrupts to be handled by _only_ the master lcore. Because > > > of this change, it seems that now alarm interrupts would also be hand= led > > > by master lcore only, IIUC. > > > > > > We are seeing a performance regression for cxgbe PMD after this commi= t > > > since, cxgbe PMD relies on alarm to periodically transmit pending > > > coalesced packets. > > > > > > Also, this perf degradation is only seen if there's a queue allocated > > > on the master lcore, such as in l3fwd app. If the master lcore has > > > been skipped, then no degradation in perf is seen since only the alar= m > > > will run on the master lcore. > > > > > > So, is the change done to make all interrupts, including alarm > > > interrupts, be handled by _only_ the master lcore intended? > > > > No it was not intended. The idea was to inherit settings (iopl) from > > the device initialization into the interrupt thread. > > Though a DPDK driver is not really supposed to rely on interrupt perfor= mance. > > So having interrupts managed on any core was more or less a side effect= . > > > > > BTW, I have tried setting the affinity to all cpus instead in > > > eal_intr_init() and this seems to restore the perf back. Perhaps it's > > > better to move the master lcore initialization to after the interrupt > > > thread has been initialized as well? Thoughts? > > > > Yes, i think it's possible. > > We can also imagine a command line option to set the interrupt affinity > > with a default which mimics the old behaviour. > > > > In order to make this conversation clearer, and for later references, > > below is the DPDK init call tree: > > >=20 > With the new interrupt mode, the interrupt thread needs some rework anywa= y. > Ideally, there would be multiple interrupt threads, one per core; > then use SMP affinity to align the MSI-x interrupt for the device queue > to run on the core that is processing that queue. >=20 > This would require new API's to do SMP affinity, wrapper around /proc/irq > and an API to tell DPDK which lcore is being to process a RX (and TX) > queue. There is no one to one mapping between lcore and device queue. Any lcore can do RX/TX on the device queue. Of course it is preferable to do it from the core on the same socket, but n= ot required. You can even have multiple threads RX/TX from/to the same queue - as long as you provide some sync mechanism between them. Konstantin=20 >=20 >=20