From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <stephen@networkplumber.org>
Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com
[209.85.220.54]) by dpdk.org (Postfix) with ESMTP id 2E76137AF
for <dev@dpdk.org>; Mon, 16 Nov 2015 18:06:56 +0100 (CET)
Received: by padhx2 with SMTP id hx2so180686879pad.1
for <dev@dpdk.org>; Mon, 16 Nov 2015 09:06:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=networkplumber_org.20150623.gappssmtp.com; s=20150623;
h=date:from:to:cc:subject:message-id:in-reply-to:references
:mime-version:content-type:content-transfer-encoding;
bh=JUixa4SzJt1fQOm7qPigPd3wdOxkGD7k1moM/lUZuWg=;
b=1Jpeaxu0P10ZEAIdN48sP2MVUqVL6nKzhFU0u+abvNVTMQE8NBEfwN/w1/2zKV1svG
fSenBI9+hdTdQUl4nVqdwIkfNxPCIp6yEgMl7CVlYOe1W6jTuLqKlUzMWOjIMO8l0FQn
HiIOHbS5s0kzRr1EKBqcosGIL0XNeJ5xs24YtXSg2F3iwvGw1iY6ljWI5S8pDokSdgzI
hqT4R5NHZkEjN35DgxYH48KeS/e9j2859tsjBK65fZRRxdj59uwS1UvPIdUIzs4kU2Js
0LbUeVLEhbCAnYlUj/ql6ElnbPiQ4m8GUcQXdra77fN02Yda5oe/Ssy6fvuB5jm7tIxH
2Flg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
:references:mime-version:content-type:content-transfer-encoding;
bh=JUixa4SzJt1fQOm7qPigPd3wdOxkGD7k1moM/lUZuWg=;
b=c5mt5cAccU7AYP44mbjwrkvoi9K9vZ/Tu9nFyNKl+utGcGC1ofL6JhIEcKoBJM/dMj
mjpfbMDy63lMfMaLmO327oHxNn2kHg9cQ0irqR0LNfsNpX8sn3F1zJSWjOZ8t4dQ1Pab
PWXhZ4MwzfmRgHyaLcgSFwqazSXBk3w2QHFcblBbHLUZZWMWNFstS8b7PozfHt0tJYqT
jP8sHwKD5bx29mJkIaQyhme8qkZByn9cYuzIgZYbTfTspD8YGudBIdnRB9k+MXBIaZT7
/bg9k1zYd0gbTOcmBuiYZAlVpgTSbCFGyj4oMy1ARgGfLBA1vyNyFTY2r4tNppXVTjNJ
+7hw==
X-Gm-Message-State: ALoCoQk/RgsLpTIAALTybiOY3dy8Z2kY7DD668GAuWzbFKBnqdBpmi6WOzIP/1WLes4Oq1XeEGqs
X-Received: by 10.68.182.130 with SMTP id ee2mr55058870pbc.156.1447693615324;
Mon, 16 Nov 2015 09:06:55 -0800 (PST)
Received: from samsung9 (50-206-118-3-static.hfc.comcastbusiness.net.
[50.206.118.3])
by smtp.gmail.com with ESMTPSA id ez1sm37749506pab.6.2015.11.16.09.06.54
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Mon, 16 Nov 2015 09:06:55 -0800 (PST)
Date: Mon, 16 Nov 2015 09:06:30 -0800
From: Stephen Hemminger <stephen@networkplumber.org>
To: Thomas Monjalon <thomas.monjalon@6wind.com>
Message-ID: <20151116090630.0f0a9b27@samsung9>
In-Reply-To: <8192567.2fdTdH6sjP@xps13>
References: <20151116123200.GA2667@scalar.blr.asicdesigners.com>
<8192567.2fdTdH6sjP@xps13>
X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Cc: dev@dpdk.org, Nirranjan Kirubaharan <nirranjan@chelsio.com>,
Felix Marti <felix@chelsio.com>, Kumar Sanghvi <kumaras@chelsio.com>
Subject: Re: [dpdk-dev] Recent changes related to interrupt thread
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 16 Nov 2015 17:06:56 -0000
On Mon, 16 Nov 2015 14:48:42 +0100
Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
> Hi,
>
> 2015-11-16 18:02, Rahul Lakkireddy:
> > Hi,
> >
> > I notice that the following changeset:
> >
> > Fixes: fd6949c55c9a ("eal: fix io permission for virtio interrupt
> > handler")
> >
> > has moved the initialization of the interrupt thread to after the master
> > lcore has been initialized. However, this causes the interrupt thread
> > to _inherit_ the affinity of the master lcore. Hence, this seems to
> > make all interrupts to be handled by _only_ the master lcore. Because
> > of this change, it seems that now alarm interrupts would also be handled
> > by master lcore only, IIUC.
> >
> > We are seeing a performance regression for cxgbe PMD after this commit
> > since, cxgbe PMD relies on alarm to periodically transmit pending
> > coalesced packets.
> >
> > Also, this perf degradation is only seen if there's a queue allocated
> > on the master lcore, such as in l3fwd app. If the master lcore has
> > been skipped, then no degradation in perf is seen since only the alarm
> > will run on the master lcore.
> >
> > So, is the change done to make all interrupts, including alarm
> > interrupts, be handled by _only_ the master lcore intended?
>
> No it was not intended. The idea was to inherit settings (iopl) from
> the device initialization into the interrupt thread.
> Though a DPDK driver is not really supposed to rely on interrupt performance.
> So having interrupts managed on any core was more or less a side effect.
>
> > BTW, I have tried setting the affinity to all cpus instead in
> > eal_intr_init() and this seems to restore the perf back. Perhaps it's
> > better to move the master lcore initialization to after the interrupt
> > thread has been initialized as well? Thoughts?
>
> Yes, i think it's possible.
> We can also imagine a command line option to set the interrupt affinity
> with a default which mimics the old behaviour.
>
> In order to make this conversation clearer, and for later references,
> below is the DPDK init call tree:
>
With the new interrupt mode, the interrupt thread needs some rework anyway.
Ideally, there would be multiple interrupt threads, one per core;
then use SMP affinity to align the MSI-x interrupt for the device queue
to run on the core that is processing that queue.
This would require new API's to do SMP affinity, wrapper around /proc/irq
and an API to tell DPDK which lcore is being to process a RX (and TX)
queue.