Re: [dpdk-dev] overcommitting CPUs

["Zhou, Danny" <danny.zhou@intel.com>]

When I have time maybe next month, will submit my initial RFC patch to enable interrupt and polling mode switching for 10G NIC with sample. Welcome anybody in the community to further optimize it.

I think it will be very hard to have a generic DPDK-NAPI API to fit all the use cases, at least thresholds in the algorithm need to be adjusted on a case by case basis.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze
> Sent: Wednesday, August 27, 2014 4:41 PM
> To: Patel, Rashmin N
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] overcommitting CPUs
> 
> IMHO adding "Interrupt Mode" to dpdk is important as this can open
> DPDK to a larger public of consumers, I can easily imagine someone
> trying to find user space networking  solution (And deciding against
> verbs - RDMA) for the obvious reasons and not needing deterministic
> latency.
> 
> A few thoughts:
> 
> Deterministic Latency: Its a fiction in a sence that  this something
> you will be able to see only in a small controlled environment. As
> network latencies in Data Centres(DC) are dominated by switch queuing
> (One good reference is http://fastpass.mit.edu that Vincent shared a
> few days back).
> 
> Virtual environments: In virtual environments this is especially
> interesting as the NIC driver(Hypervisor) is working in IRQ mode which
> unless the Interrupts are pinned to different cpus then the VM will
> have a disruptive effect on the VM's performance. Moving to interrupt
> mode mode in paravirtualised environments makes sense as in any
> environment that is not carefully crafted you should not expect any
> deterministic guaranties and would opt for a simpler programming model
> - like interrupt mode.
> 
> NAPI: With 10G NICs Most CPUs poll rate is faster then the NIC message
> rate resulting in 1:1 napi_poll callback to IRQ ratio this is true
> even with small packets. In some cases where the CPU is working slower
> - for example when intel_iommu=on,strict is set , you can actually see
> a performance inversion where the "slower" CPU can reach higher B/W
> because the slowdown makes NAPI work with the kernel effectively
> moving to polling mode.
> 
> I think that a smarter DPDK-NAPI is important, but it is a next step
> IFF the interrupt mode is adopted.
> 
> On Wed, Aug 27, 2014 at 8:48 AM, Patel, Rashmin N
> <rashmin.n.patel@intel.com> wrote:
> > You're right and I've felt the same harder part of determinism with other hypervisors' soft switch solutions as well. I think it's worth
> thinking about.
> >
> > Thanks,
> > Rashmin
> >
> > On Aug 26, 2014 9:15 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> > The way to handle switch between out of poll mode is to use IRQ coalescing
> > parameters.
> > You want to hold off IRQ until there are a couple packets or a short delay.
> > Going out of poll mode
> > is harder to determine.
> >
> >
> > On Tue, Aug 26, 2014 at 9:59 AM, Zhou, Danny <danny.zhou@intel.com> wrote:
> >
> >>
> >> > -----Original Message-----
> >> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger
> >> > Sent: Wednesday, August 27, 2014 12:39 AM
> >> > To: Michael Marchetti
> >> > Cc: dev@dpdk.org
> >> > Subject: Re: [dpdk-dev] overcommitting CPUs
> >> >
> >> > On Tue, 26 Aug 2014 16:27:14 +0000
> >> > "Michael  Marchetti" <mmarchetti@sandvine.com> wrote:
> >> >
> >> > > Hi, has there been any consideration to introduce a non-spinning
> >> network driver (interrupt based), for the purpose of overcommitting
> >> > CPUs in a virtualized environment?  This would obviously have reduced
> >> high-end performance but would allow for increased guest
> >> > density (sharing of physical CPUs) on a host.
> >> > >
> >> > > I am interested in adding support for this kind of operation, is there
> >> any interest in the community?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Mike.
> >> >
> >> > Better to implement a NAPI like algorithm that adapts from poll to
> >> interrupt.
> >>
> >> Agreed, but DPDK is currently pure poll-mode based, so unlike the NAPI'
> >> simple algorithm, the new heuristic algorithm should not switch from
> >> poll-mode to interrupt-mode immediately once there is no packet in the
> >> recent poll. Otherwise, mode switching will be too frequent which brings
> >> serious negative performance impact to DPDK.
> >>