From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) by dpdk.org (Postfix) with ESMTP id 9DE4012A8 for ; Mon, 16 Nov 2015 14:49:59 +0100 (CET) Received: by wmec201 with SMTP id c201so177346759wme.0 for ; Mon, 16 Nov 2015 05:49:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind_com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:organization:user-agent :in-reply-to:references:mime-version:content-transfer-encoding :content-type; bh=jusiK020m4v4KInYl15lAJbeBcgbVfVidgzi3pnukR4=; b=PJq+BfUBZttdkiIYA2v5GCzbeThQXMk73ayjWikXurhVxb3Q4D2s5DYCNevDVifwz4 pJCDhPhVXzYpS9I+KMwULlDs16MDiHVQPabWh62c2ZmmqN6LjW91SSyu67H9Jv/5nGcW q8BPkKu6RJZ5CRpv8aRNIzYRBv8eu0hQUU1O3CDDPjLLEuE5mH2BvXgEHTMELcjfj0oh PDLTHeL5JA4xwxi0EqUMSyI1tUdflaKtcaK3tFe7wZ4p/W1jLzPPdaimZ80oTituiCHj kz6GysPo1cg2cAaahQcrt0xXG9dqvTXhRE2gtLOUCPFUIGdENQH8bzJ6MKlPhW5VHtWT 1tiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:organization :user-agent:in-reply-to:references:mime-version :content-transfer-encoding:content-type; bh=jusiK020m4v4KInYl15lAJbeBcgbVfVidgzi3pnukR4=; b=D7kUiGI7JF4gWbfDd99KKWZPBtLj5Hg+rcJn1wnXMksQfwf86va8gah9IoWID8vLB3 f7clKjVGHQ6IBOuFOHLSmgIxmL5AdNKeqD5RBFmPdeW7QDuElr338aWFKLW/yX5Yltz5 7/ctj/hBGx+RglvZWISJH7D35a+YzdX1QZWLvBnSc92D/vGYP0NEgkB/sCR0PO7+Czs4 uhMUVMR0yuk+m3+5ixy1Xy5qfx6wshf046iBQ4g/yV67RoteKOrSgWUBhOeZQ49gG4V6 eyuDVU4+Aj1UAvGW37sSG48iPpmriFXoAWX0gNr0Vm2PzQ/isYmSJIY61qW7C4YhXgJV pqlw== X-Gm-Message-State: ALoCoQksEzKR+yKsYeuXC6kCXA/LereqfZmNri33fX9UKQR+QWaiet3BHOuJrzTMK3bwz0+Y7UhQ X-Received: by 10.194.116.167 with SMTP id jx7mr37197373wjb.105.1447681797382; Mon, 16 Nov 2015 05:49:57 -0800 (PST) Received: from xps13.localnet (136-92-190-109.dsl.ovh.fr. [109.190.92.136]) by smtp.gmail.com with ESMTPSA id q141sm18723608wmg.3.2015.11.16.05.49.56 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 16 Nov 2015 05:49:56 -0800 (PST) From: Thomas Monjalon To: Rahul Lakkireddy Date: Mon, 16 Nov 2015 14:48:42 +0100 Message-ID: <8192567.2fdTdH6sjP@xps13> Organization: 6WIND User-Agent: KMail/4.14.10 (Linux/4.1.6-1-ARCH; KDE/4.14.11; x86_64; ; ) In-Reply-To: <20151116123200.GA2667@scalar.blr.asicdesigners.com> References: <20151116123200.GA2667@scalar.blr.asicdesigners.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Cc: dev@dpdk.org, Nirranjan Kirubaharan , Felix Marti , Kumar Sanghvi Subject: Re: [dpdk-dev] Recent changes related to interrupt thread X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 13:49:59 -0000 Hi, 2015-11-16 18:02, Rahul Lakkireddy: > Hi, > > I notice that the following changeset: > > Fixes: fd6949c55c9a ("eal: fix io permission for virtio interrupt > handler") > > has moved the initialization of the interrupt thread to after the master > lcore has been initialized. However, this causes the interrupt thread > to _inherit_ the affinity of the master lcore. Hence, this seems to > make all interrupts to be handled by _only_ the master lcore. Because > of this change, it seems that now alarm interrupts would also be handled > by master lcore only, IIUC. > > We are seeing a performance regression for cxgbe PMD after this commit > since, cxgbe PMD relies on alarm to periodically transmit pending > coalesced packets. > > Also, this perf degradation is only seen if there's a queue allocated > on the master lcore, such as in l3fwd app. If the master lcore has > been skipped, then no degradation in perf is seen since only the alarm > will run on the master lcore. > > So, is the change done to make all interrupts, including alarm > interrupts, be handled by _only_ the master lcore intended? No it was not intended. The idea was to inherit settings (iopl) from the device initialization into the interrupt thread. Though a DPDK driver is not really supposed to rely on interrupt performance. So having interrupts managed on any core was more or less a side effect. > BTW, I have tried setting the affinity to all cpus instead in > eal_intr_init() and this seems to restore the perf back. Perhaps it's > better to move the master lcore initialization to after the interrupt > thread has been initialized as well? Thoughts? Yes, i think it's possible. We can also imagine a command line option to set the interrupt affinity with a default which mimics the old behaviour. In order to make this conversation clearer, and for later references, below is the DPDK init call tree: start driver constructor (if .a) rte_eal_driver_register main rte_eal_init eal_parse_args rte_eal_pci_init rte_eal_memory_init eal_plugins_init dlopen driver constructor (if .so) rte_eal_driver_register eal_thread_init_master eal_thread_set_affinity rte_eal_dev_init driver->init PMD init rte_eth_driver_register rte_eal_intr_init pthread_create eal_intr_thread_main eal_intr_handle_interrupts pthread_create rte_eal_pci_probe driver->devinit rte_eth_dev_init rte_eth_dev_allocate eth_drv->eth_dev_init