Re: [dpdk-dev] bifurcated driver

["Zhou, Danny" <danny.zhou@intel.com>]

I roughly read libibverbs related code and relevant infiniband/rdma documents, and found though 
many concepts in libibverbs looks similar to bifurcated driver, but there are still lots of differences as 
illustrated below based on my understanding: 

1) Queue pair defined in RDMA specification are abstract concept, where the queue pairs term used in 
  bifurcated driver are rx/tx queue pairs in the NIC.
2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver (no NIC control), while libibverbs
  as a user space library rather than driver offloads certain operations to kernel driver and NIC by invoking 
  "verbs" APIs.
3) Libibverbs invokes infiniband specific system calls to allow user/kernel space communication based on 
  "verbs" defined in infiniband/RDMA spec, while bifurcated driver build on top of af_packet module 
  and new socket options to do things like hw queue split-off , map certain pages on I/O space to user space 
  operations, etc.
4) There is a specific embedded MMU unit in Infiniband/RDMA to provides memory protection, while
  bifurcated driver uses IOMMU rather than NIC to provide memory protection.

IMHO, libibverbs and corresponding kernel modules/drivers are specifically designed and implemented for 
direct access to RDMA hardware from userspace, and it highly depends on "verbs" related system calls 
supported by infiniband/rdma mechanism in kernel, rather than netdev mechanism that bifurcated driver 
solution depends on. 

> -----Original Message-----
> From: Vincent JARDIN [mailto:vincent.jardin@6wind.com]
> Sent: Thursday, November 06, 2014 9:31 AM
> To: Zhou, Danny
> Cc: Thomas Monjalon; dev@dpdk.org; Fastabend, John R; Or Gerlitz
> Subject: Re: [dpdk-dev] bifurcated driver
> 
> +Or
> 
> On 05/11/2014 23:48, Zhou, Danny wrote:
> > Hi Thomas,
> >
> > Thanks for sharing the links to ibverbs, I will take a close look at it and compare it to bifurcated driver. My take
> > after a rough review is that idea is very much similar, but bifurcated driver implementation is generic for any
> > Ethernet device based on existing af_packet mechanism, with extension of exchanging the messages between
> > user space and kernel space driver.
> >
> > I have an internal document to summary the pros and cons of below solutions, except for ibvers, but
> > will be adding it shortly.
> >
> > - igb_uio
> > - uio_pci_generic
> > - VFIO
> > - bifurcated driver
> >
> > Short answers to your questions:
> >> 	- upstream status
> > Adding IOMMU based memory protection and generic descriptor description support now, into version 2
> > kernel patches.
> >
> >> 	- usable with kernel netdev
> > af_packet based, and relevant patchset will be submitted to netdev for sure.
> >
> >> 	- usable in a vm
> > No, it does no coexist with SRIOV for number of reasons. but if you pass-through a PF to a VM, it works perfect.
> >
> >> 	- usable for Ethernet
> > It could work with all Ethernet NICs, as flow director is available and NIC driver support new net_ops to split off
> > queue pairs for user space.
> >
> >> 	- hardware requirements
> > No specific hardware requirements. All mainstream NICs have multiple qpairs and flow director support.
> >
> >> 	- security protection
> > Leverage IOMMU to provide memory protection on Intel platform. Other archs provide similar memory protection
> > mechanism, so we only use arch-agnostic DMA memory allocation APIs in kernel to support memory protection.
> >
> >> 	- performance
> > DPDK native performance on user space queues, as long as drop_en is enabled to avoid head-of-line blocking.
> >
> > -Danny
> >
> >> -----Original Message-----
> >> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> >> Sent: Wednesday, November 05, 2014 9:01 PM
> >> To: Zhou, Danny
> >> Cc: dev@dpdk.org; Fastabend, John R
> >> Subject: Re: [dpdk-dev] bifurcated driver
> >>
> >> Hi Danny,
> >>
> >> 2014-10-31 17:36, O'driscoll, Tim:
> >>> Bifurcated Driver (Danny.Zhou@intel.com)
> >>
> >> Thanks for the presentation of bifurcated driver during the community call.
> >> I asked if you looked at ibverbs and you wanted a link to check.
> >> The kernel module is here:
> >> 	http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
> >> The userspace library:
> >> 	http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
> >>
> >> Extract from Kconfig:
> >> "
> >> config INFINIBAND_USER_ACCESS
> >> 	tristate "InfiniBand userspace access (verbs and CM)"
> >> 	select ANON_INODES
> >> 	---help---
> >> 	  Userspace InfiniBand access support.  This enables the
> >> 	  kernel side of userspace verbs and the userspace
> >> 	  communication manager (CM).  This allows userspace processes
> >> 	  to set up connections and directly access InfiniBand
> >> 	  hardware for fast-path operations.  You will also need
> >> 	  libibverbs, libibcm and a hardware driver library from
> >> 	  <http://www.openfabrics.org/git/>.
> >> "
> >>
> >> It seems to be close to the bifurcated driver needs.
> >> Not sure if it can solve the security issues if there is no dedicated MMU
> >> in the NIC.
> >>
> >> I feel we should sum up pros and cons of
> >> 	- igb_uio
> >> 	- uio_pci_generic
> >> 	- VFIO
> >> 	- ibverbs
> >> 	- bifurcated driver
> >> I suggest to consider these criterias:
> >> 	- upstream status
> >> 	- usable with kernel netdev
> >> 	- usable in a vm
> >> 	- usable for ethernet
> >> 	- hardware requirements
> >> 	- security protection
> >> 	- performance
> >>
> >> --
> >> Thomas