Re: [dpdk-dev] Beyond DPDK 2.0

[Stephen Hemminger <stephen@networkplumber.org>]

On Fri, 8 May 2015 09:31:34 -0400
Neil Horman <nhorman@tuxdriver.com> wrote:

> On Fri, May 08, 2015 at 12:26:39PM +0200, Hobywan Kenoby wrote:
> > 
> > > Sounds like you want something like libc, but DPDK is a system like a user
> > > space OS more then it is a collection of functions that are independent
> > > like strlen, strcpy, memcpy, printf or ... Some parts of DPDK are
> > > independent and can be used as you suggest, but the real performance
> > > sections are tied together.
> > > 
> > > >> Regards,
> > > >> ++Keith
> > 
> > This is indeed quite a statement. DPDK is not just a
> > bunch of NIC drivers, but "a user space OS" (DPDK 1.0 had a baremetal
> > boot: why did it disappeared?).
> > 
> >  
> > 
> > Why Linux or Windows do not integrate DPDK concepts to
> > catch up performance wise? Is it something so deep like the "Big
> > Kernel Lock" that took so many years to get rid of?
> > 
> Some optimizations are being looked at in the kernel (more deeply ingrained use
> of accelerators/offloads like cam management/flow steering, checksum & encap
> offloads, tx batching, etc)  Those are features which the kernel can
> opportunistically take advantage of in a hardware agnostic fashion.
> 
> Some optimizations simply aren't worth the effort to take into a general purpose
> OS that seeks to support multiple arches.  Many of the DPDK optimizations
> utilize instruction families like AVX or SSE, which, while potentially useful in
> some situations have equal potential to be catastrophic to non network-i/o bound
> workloads. 
> >  
> > 
> > My assumption is that all current kernels have been
> > built with one implicit hypothesis: the memory is much faster than cpu. This is
> Thats not entirely true.  Or more to the point, its not true in any way thats
> relevant to a comparison between DPDK and the Linux network stack.  Linux is as
> careful with its cache management as DPDK is (arguably more so, as it has to
> juggle multiple workloads instead of the single purpose workload that DPDK is
> designed for).  The difference is that Linux often has to ignore some
> performance improvements because it has the additional responsibiilty of
> providing secruity and isolation to multiple processes on multiple
> architectures.
> 
> > the opposite today. DPDK internal structure has been adapted to the new
> > paradigm where the TLBs, the memory bandwidth are the scarce resources to manage. So I
> > guess Linux and Windows will not be able to integrate DPDK concepts for
> > performance anytime soon, if ever...
> > 
> This is the case with every bit of software.  Memory bandwidth is always a
> scarce resoruce to manage.  The difference is that general purpose operating
> systems consider protection/layering/isolation to be of equal or greater
> importance than performance.  Tradeoffs have to be made.  Linux in general
> strives to isolate hardware from applications both functionally and physically
> so as to ensure that there is minimal risk in one process adversely affecting
> the other.  The tradeoff is that the Linux device model can't just do anything
> it wants to improve performance.
> 
> Converserly, DPDK is all about performance.  Up until recently (and likely still
> somewhat in the future), you have to rebuild your application every time you
> move to a new version of DPDK, because the API fluctuated with every release to
> eek out additional performance.  The DPDK can optimize using vectorized x86
> instructions and other cpu specific features througout because it is in the
> position to only worry about a very narrow field of architectures.
> 
> >  
> > 
> > Reading the list carefully, I expect disk block PMDs
> > (and block framework?) to come next.
> > 
> >  
> > 
> > Beyond DPDK 2.0: is it time to accept the fact that
> > DPDK community is actually paving the way to the next generation lightweight,
> > high performance, para-virtualized OS? Is it a DPDK task? Another project ?
> > Should we rename DPDK to PVDK?

The difference is DPDK doesn't care about being general purpose:
  - scheduler, that is the applications problem
  - locking, the application must be bound to cpus or do its own locking
  - buffer management, up to the application.
  - memory protection (haha)

Any operating system provides an abstraction that makes programming easier.
If you strip away those abstractions, then sure things go faster but it is
less safe and harder.

Linux is about providing safe abstraction. If you want an OS that doesn't
do that, look to Cloudius or the other DIY environments like DPDK.

This is not a new concept. Oracle and other DBMS vendors have been asking
for the OS to get out of the way for years, but then customers find that
things like filesystems are convenient necessities.