From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id CCD572E89 for ; Thu, 6 Nov 2014 05:36:27 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga101.fm.intel.com with ESMTP; 05 Nov 2014 20:45:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,323,1413270000"; d="scan'208";a="617982794" Received: from pgsmsx103.gar.corp.intel.com ([10.221.44.82]) by fmsmga001.fm.intel.com with ESMTP; 05 Nov 2014 20:45:11 -0800 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by PGSMSX103.gar.corp.intel.com (10.221.44.82) with Microsoft SMTP Server (TLS) id 14.3.195.1; Thu, 6 Nov 2014 12:45:11 +0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.174]) by shsmsx102.ccr.corp.intel.com ([169.254.2.156]) with mapi id 14.03.0195.001; Thu, 6 Nov 2014 12:45:10 +0800 From: "Zhou, Danny" To: Thomas Monjalon Thread-Topic: [dpdk-dev] bifurcated driver Thread-Index: AQHP+Pi+EFacz72k3UuzHEADinhAXJxSnTYg//+tQICAAKyAgA== Date: Thu, 6 Nov 2014 04:45:09 +0000 Message-ID: References: <26FA93C7ED1EAA44AB77D62FBE1D27BA54C42C29@IRSMSX102.ger.corp.intel.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA54C4620C@IRSMSX102.ger.corp.intel.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA54C46444@IRSMSX102.ger.corp.intel.com> <4163036.7ZLDrAu3iM@xps13> <545ACF39.5000507@6wind.com> In-Reply-To: <545ACF39.5000507@6wind.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" , "Fastabend, John R" , Or Gerlitz Subject: Re: [dpdk-dev] bifurcated driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 04:36:29 -0000 I roughly read libibverbs related code and relevant infiniband/rdma documen= ts, and found though=20 many concepts in libibverbs looks similar to bifurcated driver, but there a= re still lots of differences as=20 illustrated below based on my understanding:=20 1) Queue pair defined in RDMA specification are abstract concept, where the= queue pairs term used in=20 bifurcated driver are rx/tx queue pairs in the NIC. 2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver (= no NIC control), while libibverbs as a user space library rather than driver offloads certain operations to= kernel driver and NIC by invoking=20 "verbs" APIs. 3) Libibverbs invokes infiniband specific system calls to allow user/kernel= space communication based on=20 "verbs" defined in infiniband/RDMA spec, while bifurcated driver build on= top of af_packet module=20 and new socket options to do things like hw queue split-off , map certain= pages on I/O space to user space=20 operations, etc. 4) There is a specific embedded MMU unit in Infiniband/RDMA to provides mem= ory protection, while bifurcated driver uses IOMMU rather than NIC to provide memory protection= . IMHO, libibverbs and corresponding kernel modules/drivers are specifically = designed and implemented for=20 direct access to RDMA hardware from userspace, and it highly depends on "ve= rbs" related system calls=20 supported by infiniband/rdma mechanism in kernel, rather than netdev mechan= ism that bifurcated driver=20 solution depends on.=20 > -----Original Message----- > From: Vincent JARDIN [mailto:vincent.jardin@6wind.com] > Sent: Thursday, November 06, 2014 9:31 AM > To: Zhou, Danny > Cc: Thomas Monjalon; dev@dpdk.org; Fastabend, John R; Or Gerlitz > Subject: Re: [dpdk-dev] bifurcated driver >=20 > +Or >=20 > On 05/11/2014 23:48, Zhou, Danny wrote: > > Hi Thomas, > > > > Thanks for sharing the links to ibverbs, I will take a close look at it= and compare it to bifurcated driver. My take > > after a rough review is that idea is very much similar, but bifurcated = driver implementation is generic for any > > Ethernet device based on existing af_packet mechanism, with extension o= f exchanging the messages between > > user space and kernel space driver. > > > > I have an internal document to summary the pros and cons of below solut= ions, except for ibvers, but > > will be adding it shortly. > > > > - igb_uio > > - uio_pci_generic > > - VFIO > > - bifurcated driver > > > > Short answers to your questions: > >> - upstream status > > Adding IOMMU based memory protection and generic descriptor description= support now, into version 2 > > kernel patches. > > > >> - usable with kernel netdev > > af_packet based, and relevant patchset will be submitted to netdev for = sure. > > > >> - usable in a vm > > No, it does no coexist with SRIOV for number of reasons. but if you pas= s-through a PF to a VM, it works perfect. > > > >> - usable for Ethernet > > It could work with all Ethernet NICs, as flow director is available and= NIC driver support new net_ops to split off > > queue pairs for user space. > > > >> - hardware requirements > > No specific hardware requirements. All mainstream NICs have multiple qp= airs and flow director support. > > > >> - security protection > > Leverage IOMMU to provide memory protection on Intel platform. Other ar= chs provide similar memory protection > > mechanism, so we only use arch-agnostic DMA memory allocation APIs in k= ernel to support memory protection. > > > >> - performance > > DPDK native performance on user space queues, as long as drop_en is ena= bled to avoid head-of-line blocking. > > > > -Danny > > > >> -----Original Message----- > >> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] > >> Sent: Wednesday, November 05, 2014 9:01 PM > >> To: Zhou, Danny > >> Cc: dev@dpdk.org; Fastabend, John R > >> Subject: Re: [dpdk-dev] bifurcated driver > >> > >> Hi Danny, > >> > >> 2014-10-31 17:36, O'driscoll, Tim: > >>> Bifurcated Driver (Danny.Zhou@intel.com) > >> > >> Thanks for the presentation of bifurcated driver during the community = call. > >> I asked if you looked at ibverbs and you wanted a link to check. > >> The kernel module is here: > >> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/d= rivers/infiniband/core > >> The userspace library: > >> http://git.kernel.org/cgit/libs/infiniband/libibverbs.git > >> > >> Extract from Kconfig: > >> " > >> config INFINIBAND_USER_ACCESS > >> tristate "InfiniBand userspace access (verbs and CM)" > >> select ANON_INODES > >> ---help--- > >> Userspace InfiniBand access support. This enables the > >> kernel side of userspace verbs and the userspace > >> communication manager (CM). This allows userspace processes > >> to set up connections and directly access InfiniBand > >> hardware for fast-path operations. You will also need > >> libibverbs, libibcm and a hardware driver library from > >> . > >> " > >> > >> It seems to be close to the bifurcated driver needs. > >> Not sure if it can solve the security issues if there is no dedicated = MMU > >> in the NIC. > >> > >> I feel we should sum up pros and cons of > >> - igb_uio > >> - uio_pci_generic > >> - VFIO > >> - ibverbs > >> - bifurcated driver > >> I suggest to consider these criterias: > >> - upstream status > >> - usable with kernel netdev > >> - usable in a vm > >> - usable for ethernet > >> - hardware requirements > >> - security protection > >> - performance > >> > >> -- > >> Thomas