From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f169.google.com (mail-io0-f169.google.com [209.85.223.169]) by dpdk.org (Postfix) with ESMTP id E796C378E for ; Mon, 16 Nov 2015 18:40:22 +0100 (CET) Received: by iouu10 with SMTP id u10so161103570iou.0 for ; Mon, 16 Nov 2015 09:40:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber_org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=HUD4ecPh9y82oH6oN6+/1SlHCit6/pYKXuv5mdVDkJw=; b=FMg9Ty67pQp4lxqV2aaUrOLJKRjSDOL51/Rrz2oa3FPNsxfh+fL7sIhj3PuXJGeoRq GDrf0yqQybb8pEn5V6oGh9e32llm+PIt+7zpj64PEzFxr1Ka+qNASVCF+ecSgjQ4CuT9 SalngBxVoLJY+oLXOSbBV8YxQfiInJ16jWBtiwdW9T0ZawaSl1+VUzjglr4An0tE1X70 P+i++03q3xGnjR2QfMFtCRl4Zb6d4nt5QFehkgM+d1tOWs8dG1RRZU+RINExsJhMDLv0 eCsrdflCWGU26YqE4czh3fA2nR6lZUesnuLGG2XPkR5vY47gdOF5oaqqVSkE1uYGuGBJ cEUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=HUD4ecPh9y82oH6oN6+/1SlHCit6/pYKXuv5mdVDkJw=; b=FY6uf39Bav5EaTFGl0vrFNCAfMg4BLAQayGXMYOHk1troANilWbXmfiz1FPxXeoyV0 GOg0JtimqHqfsb0Ya+CFh3aHk0PIRK0VVJ6Sc/qsYJz2NbyzH61fibfGEWxsHA2hu8cC xPAmbm2kjI0CSh3NrxdDXdDNYzy6nyMEZIUE8sIwydea/KbkH8RQS40aamCzvV8x5SiY /b/LAz/IeFFyRNKXj+bpx3JwTfKk8Q4B2FQbo6Mg8G6hiGBjnGvmcXmx0vdG9pIX1eJQ zenjqdH5or/7dV2viSbFagb88D5zRGV+afs6wLExRxlgh3EeHfLBdlD7O6OdIaRDPcoi ZItQ== X-Gm-Message-State: ALoCoQmoy8zd4BxNuJNVUmNxt0UjG4OhbmtezQRLZCOWakGUvKnU96bs0uwK65Krh2mQ7WcEN6oQ MIME-Version: 1.0 X-Received: by 10.107.8.34 with SMTP id 34mr41511023ioi.122.1447695622387; Mon, 16 Nov 2015 09:40:22 -0800 (PST) Received: by 10.64.123.164 with HTTP; Mon, 16 Nov 2015 09:40:22 -0800 (PST) In-Reply-To: <2601191342CEEE43887BDE71AB97725836AC961E@irsmsx105.ger.corp.intel.com> References: <20151116123200.GA2667@scalar.blr.asicdesigners.com> <8192567.2fdTdH6sjP@xps13> <20151116090630.0f0a9b27@samsung9> <2601191342CEEE43887BDE71AB97725836AC961E@irsmsx105.ger.corp.intel.com> Date: Mon, 16 Nov 2015 09:40:22 -0800 Message-ID: From: Stephen Hemminger To: "Ananyev, Konstantin" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" , Felix Marti , Nirranjan Kirubaharan , Kumar Sanghvi Subject: Re: [dpdk-dev] Recent changes related to interrupt thread X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 17:40:23 -0000 I was thinking of something like: rte_intr_affinity(portid, queueid, lcoreid) And per-lcore interrupt threads. On Mon, Nov 16, 2015 at 9:19 AM, Ananyev, Konstantin < konstantin.ananyev@intel.com> wrote: > > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger > > Sent: Monday, November 16, 2015 5:07 PM > > To: Thomas Monjalon > > Cc: dev@dpdk.org; Nirranjan Kirubaharan; Felix Marti; Kumar Sanghvi > > Subject: Re: [dpdk-dev] Recent changes related to interrupt thread > > > > On Mon, 16 Nov 2015 14:48:42 +0100 > > Thomas Monjalon wrote: > > > > > Hi, > > > > > > 2015-11-16 18:02, Rahul Lakkireddy: > > > > Hi, > > > > > > > > I notice that the following changeset: > > > > > > > > Fixes: fd6949c55c9a ("eal: fix io permission for virtio interrupt > > > > handler") > > > > > > > > has moved the initialization of the interrupt thread to after the > master > > > > lcore has been initialized. However, this causes the interrupt > thread > > > > to _inherit_ the affinity of the master lcore. Hence, this seems to > > > > make all interrupts to be handled by _only_ the master lcore. Because > > > > of this change, it seems that now alarm interrupts would also be > handled > > > > by master lcore only, IIUC. > > > > > > > > We are seeing a performance regression for cxgbe PMD after this > commit > > > > since, cxgbe PMD relies on alarm to periodically transmit pending > > > > coalesced packets. > > > > > > > > Also, this perf degradation is only seen if there's a queue allocated > > > > on the master lcore, such as in l3fwd app. If the master lcore has > > > > been skipped, then no degradation in perf is seen since only the > alarm > > > > will run on the master lcore. > > > > > > > > So, is the change done to make all interrupts, including alarm > > > > interrupts, be handled by _only_ the master lcore intended? > > > > > > No it was not intended. The idea was to inherit settings (iopl) from > > > the device initialization into the interrupt thread. > > > Though a DPDK driver is not really supposed to rely on interrupt > performance. > > > So having interrupts managed on any core was more or less a side > effect. > > > > > > > BTW, I have tried setting the affinity to all cpus instead in > > > > eal_intr_init() and this seems to restore the perf back. Perhaps it's > > > > better to move the master lcore initialization to after the interrupt > > > > thread has been initialized as well? Thoughts? > > > > > > Yes, i think it's possible. > > > We can also imagine a command line option to set the interrupt affinity > > > with a default which mimics the old behaviour. > > > > > > In order to make this conversation clearer, and for later references, > > > below is the DPDK init call tree: > > > > > > > With the new interrupt mode, the interrupt thread needs some rework > anyway. > > Ideally, there would be multiple interrupt threads, one per core; > > then use SMP affinity to align the MSI-x interrupt for the device queue > > to run on the core that is processing that queue. > > > > This would require new API's to do SMP affinity, wrapper around /proc/irq > > and an API to tell DPDK which lcore is being to process a RX (and TX) > > queue. > > There is no one to one mapping between lcore and device queue. > Any lcore can do RX/TX on the device queue. > Of course it is preferable to do it from the core on the same socket, but > not required. > You can even have multiple threads RX/TX from/to the same queue - > as long as you provide some sync mechanism between them. > Konstantin > > > > > > >