Re: [dpdk-dev] Recent changes related to interrupt thread

[Stephen Hemminger <stephen@networkplumber.org>]

On Mon, 16 Nov 2015 14:48:42 +0100
Thomas Monjalon <thomas.monjalon@6wind.com> wrote:

> Hi,
> 
> 2015-11-16 18:02, Rahul Lakkireddy:
> > Hi,
> > 
> > I notice that the following changeset:
> > 
> > Fixes: fd6949c55c9a ("eal: fix io permission for virtio interrupt
> > handler")
> > 
> > has moved the initialization of the interrupt thread to after the master
> > lcore has been initialized.  However, this causes the interrupt thread
> > to _inherit_ the affinity of the master lcore. Hence, this seems to
> > make all interrupts to be handled by _only_ the master lcore. Because
> > of this change, it seems that now alarm interrupts would also be handled
> > by master lcore only, IIUC.
> > 
> > We are seeing a performance regression for cxgbe PMD after this commit
> > since, cxgbe PMD relies on alarm to periodically transmit pending
> > coalesced packets.
> > 
> > Also, this perf degradation is only seen if there's a queue allocated
> > on the master lcore, such as in l3fwd app.  If the master lcore has
> > been skipped, then no degradation in perf is seen since only the alarm
> > will run on the master lcore.
> > 
> > So, is the change done to make all interrupts, including alarm
> > interrupts, be handled by _only_ the master lcore intended?
> 
> No it was not intended. The idea was to inherit settings (iopl) from
> the device initialization into the interrupt thread.
> Though a DPDK driver is not really supposed to rely on interrupt performance.
> So having interrupts managed on any core was more or less a side effect.
> 
> > BTW, I have tried setting the affinity to all cpus instead in
> > eal_intr_init() and this seems to restore the perf back. Perhaps it's
> > better to move the master lcore initialization to after the interrupt
> > thread has been initialized as well? Thoughts?
> 
> Yes, i think it's possible.
> We can also imagine a command line option to set the interrupt affinity
> with a default which mimics the old behaviour.
> 
> In order to make this conversation clearer, and for later references,
> below is the DPDK init call tree:
> 

With the new interrupt mode, the interrupt thread needs some rework anyway.
Ideally, there would be multiple interrupt threads, one per core;
then use SMP affinity to align the MSI-x interrupt for the device queue
to run on the core that is processing that queue.

This would require new API's to do SMP affinity, wrapper around /proc/irq
and an API to tell DPDK which lcore is being to process a RX (and TX)
queue.