Re: [dpdk-dev] bifurcated driver

[Alex Markuze <alex@weka.io>]

Danny sums up the issue perfectly IMHO.
While both verbs and DPDK aim to provide generic user space networking, the
similarities end there.
verbs and RDMA HW are closely coupled and behave differently then standard
eth nics and are not related to netdev mechanisms.

Or, welcome to this discussion.

Those interested can read the IB spec's (+1K pages) available from
openfabrics*.
*https://www.openfabrics.org/index.php




On Thu, Nov 6, 2014 at 6:45 AM, Zhou, Danny <danny.zhou@intel.com> wrote:

> I roughly read libibverbs related code and relevant infiniband/rdma
> documents, and found though
> many concepts in libibverbs looks similar to bifurcated driver, but there
> are still lots of differences as
> illustrated below based on my understanding:
>
> 1) Queue pair defined in RDMA specification are abstract concept, where
> the queue pairs term used in
>   bifurcated driver are rx/tx queue pairs in the NIC.
> 2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver
> (no NIC control), while libibverbs
>   as a user space library rather than driver offloads certain operations
> to kernel driver and NIC by invoking
>   "verbs" APIs.
> 3) Libibverbs invokes infiniband specific system calls to allow
> user/kernel space communication based on
>   "verbs" defined in infiniband/RDMA spec, while bifurcated driver build
> on top of af_packet module
>   and new socket options to do things like hw queue split-off , map
> certain pages on I/O space to user space
>   operations, etc.
> 4) There is a specific embedded MMU unit in Infiniband/RDMA to provides
> memory protection, while
>   bifurcated driver uses IOMMU rather than NIC to provide memory
> protection.
>
> IMHO, libibverbs and corresponding kernel modules/drivers are specifically
> designed and implemented for
> direct access to RDMA hardware from userspace, and it highly depends on
> "verbs" related system calls
> supported by infiniband/rdma mechanism in kernel, rather than netdev
> mechanism that bifurcated driver
> solution depends on.
>
> > -----Original Message-----
> > From: Vincent JARDIN [mailto:vincent.jardin@6wind.com]
> > Sent: Thursday, November 06, 2014 9:31 AM
> > To: Zhou, Danny
> > Cc: Thomas Monjalon; dev@dpdk.org; Fastabend, John R; Or Gerlitz
> > Subject: Re: [dpdk-dev] bifurcated driver
> >
> > +Or
> >
> > On 05/11/2014 23:48, Zhou, Danny wrote:
> > > Hi Thomas,
> > >
> > > Thanks for sharing the links to ibverbs, I will take a close look at
> it and compare it to bifurcated driver. My take
> > > after a rough review is that idea is very much similar, but bifurcated
> driver implementation is generic for any
> > > Ethernet device based on existing af_packet mechanism, with extension
> of exchanging the messages between
> > > user space and kernel space driver.
> > >
> > > I have an internal document to summary the pros and cons of below
> solutions, except for ibvers, but
> > > will be adding it shortly.
> > >
> > > - igb_uio
> > > - uio_pci_generic
> > > - VFIO
> > > - bifurcated driver
> > >
> > > Short answers to your questions:
> > >>    - upstream status
> > > Adding IOMMU based memory protection and generic descriptor
> description support now, into version 2
> > > kernel patches.
> > >
> > >>    - usable with kernel netdev
> > > af_packet based, and relevant patchset will be submitted to netdev for
> sure.
> > >
> > >>    - usable in a vm
> > > No, it does no coexist with SRIOV for number of reasons. but if you
> pass-through a PF to a VM, it works perfect.
> > >
> > >>    - usable for Ethernet
> > > It could work with all Ethernet NICs, as flow director is available
> and NIC driver support new net_ops to split off
> > > queue pairs for user space.
> > >
> > >>    - hardware requirements
> > > No specific hardware requirements. All mainstream NICs have multiple
> qpairs and flow director support.
> > >
> > >>    - security protection
> > > Leverage IOMMU to provide memory protection on Intel platform. Other
> archs provide similar memory protection
> > > mechanism, so we only use arch-agnostic DMA memory allocation APIs in
> kernel to support memory protection.
> > >
> > >>    - performance
> > > DPDK native performance on user space queues, as long as drop_en is
> enabled to avoid head-of-line blocking.
> > >
> > > -Danny
> > >
> > >> -----Original Message-----
> > >> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > >> Sent: Wednesday, November 05, 2014 9:01 PM
> > >> To: Zhou, Danny
> > >> Cc: dev@dpdk.org; Fastabend, John R
> > >> Subject: Re: [dpdk-dev] bifurcated driver
> > >>
> > >> Hi Danny,
> > >>
> > >> 2014-10-31 17:36, O'driscoll, Tim:
> > >>> Bifurcated Driver (Danny.Zhou@intel.com)
> > >>
> > >> Thanks for the presentation of bifurcated driver during the community
> call.
> > >> I asked if you looked at ibverbs and you wanted a link to check.
> > >> The kernel module is here:
> > >>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
> > >> The userspace library:
> > >>    http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
> > >>
> > >> Extract from Kconfig:
> > >> "
> > >> config INFINIBAND_USER_ACCESS
> > >>    tristate "InfiniBand userspace access (verbs and CM)"
> > >>    select ANON_INODES
> > >>    ---help---
> > >>      Userspace InfiniBand access support.  This enables the
> > >>      kernel side of userspace verbs and the userspace
> > >>      communication manager (CM).  This allows userspace processes
> > >>      to set up connections and directly access InfiniBand
> > >>      hardware for fast-path operations.  You will also need
> > >>      libibverbs, libibcm and a hardware driver library from
> > >>      <http://www.openfabrics.org/git/>.
> > >> "
> > >>
> > >> It seems to be close to the bifurcated driver needs.
> > >> Not sure if it can solve the security issues if there is no dedicated
> MMU
> > >> in the NIC.
> > >>
> > >> I feel we should sum up pros and cons of
> > >>    - igb_uio
> > >>    - uio_pci_generic
> > >>    - VFIO
> > >>    - ibverbs
> > >>    - bifurcated driver
> > >> I suggest to consider these criterias:
> > >>    - upstream status
> > >>    - usable with kernel netdev
> > >>    - usable in a vm
> > >>    - usable for ethernet
> > >>    - hardware requirements
> > >>    - security protection
> > >>    - performance
> > >>
> > >> --
> > >> Thomas
>
>