From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by dpdk.org (Postfix) with ESMTP id 142005947 for ; Thu, 6 Nov 2014 10:01:12 +0100 (CET) Received: by mail-wi0-f174.google.com with SMTP id d1so803632wiv.13 for ; Thu, 06 Nov 2014 01:10:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:reply-to:organization :user-agent:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=WM+FUUvXc/E7uVc7AyASbUzhRyO6TH/e3YHYdjP+rUI=; b=aahSCiGoBFxyv84MR6z7LvBjgm7yGRIxiIfjBywbjiCQySDfjJEjlGO8NpyOzkr2vb JZ9BwDPpB+K0iHnQNDGOC5MSMK9LUwnbTwYMBuFEo27OSCxjp5uMs31Wnq/ImryK1mLM XtWgxBOpZ2MWh9CH66dpd1toXrwUho4+wzhvVqMkeUPZoS9IRdF9TzJ4v0k9Iggn179r JOoi3bh9FEX8myv+oDlr4/NP0KK1vU7zs8tUuJSVbwrrJTbi7RaWv+14KjxkqWi0YeUz 11SVMpsERi0hTVwr5lGbshseA8KOH4oXEEFxyRmKwIoaPFrFCa16xVoCjM10yhNlSXq7 bJGQ== X-Gm-Message-State: ALoCoQmC5I2NzSgjra78P3uqmJdOLycB3g8T5n7yIZm/3AjTCO57IU2iVBYLszppjX9lYI5YBTBn X-Received: by 10.180.102.65 with SMTP id fm1mr12722315wib.16.1415265038626; Thu, 06 Nov 2014 01:10:38 -0800 (PST) Received: from ?IPv6:2a01:e35:8b63:dc30:18a2:ac37:3e1d:167c? ([2a01:e35:8b63:dc30:18a2:ac37:3e1d:167c]) by mx.google.com with ESMTPSA id p1sm7013832wjy.22.2014.11.06.01.10.34 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Nov 2014 01:10:37 -0800 (PST) Message-ID: <545B3B09.10403@6wind.com> Date: Thu, 06 Nov 2014 10:10:33 +0100 From: Nicolas Dichtel Organization: 6WIND User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Alex Markuze , "Zhou, Danny" References: <26FA93C7ED1EAA44AB77D62FBE1D27BA54C42C29@IRSMSX102.ger.corp.intel.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA54C4620C@IRSMSX102.ger.corp.intel.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA54C46444@IRSMSX102.ger.corp.intel.com> <4163036.7ZLDrAu3iM@xps13> <545ACF39.5000507@6wind.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Cc: "dev@dpdk.org" , "Fastabend, John R" , Or Gerlitz , netdev Subject: Re: [dpdk-dev] bifurcated driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: nicolas.dichtel@6wind.com List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 09:01:12 -0000 Also CC netdev, this thread may interest network folks. Le 06/11/2014 09:13, Alex Markuze a écrit : > Danny sums up the issue perfectly IMHO. > While both verbs and DPDK aim to provide generic user space networking, the > similarities end there. > verbs and RDMA HW are closely coupled and behave differently then standard > eth nics and are not related to netdev mechanisms. > > Or, welcome to this discussion. > > Those interested can read the IB spec's (+1K pages) available from > openfabrics*. > *https://www.openfabrics.org/index.php > > > > > On Thu, Nov 6, 2014 at 6:45 AM, Zhou, Danny wrote: > >> I roughly read libibverbs related code and relevant infiniband/rdma >> documents, and found though >> many concepts in libibverbs looks similar to bifurcated driver, but there >> are still lots of differences as >> illustrated below based on my understanding: >> >> 1) Queue pair defined in RDMA specification are abstract concept, where >> the queue pairs term used in >> bifurcated driver are rx/tx queue pairs in the NIC. >> 2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver >> (no NIC control), while libibverbs >> as a user space library rather than driver offloads certain operations >> to kernel driver and NIC by invoking >> "verbs" APIs. >> 3) Libibverbs invokes infiniband specific system calls to allow >> user/kernel space communication based on >> "verbs" defined in infiniband/RDMA spec, while bifurcated driver build >> on top of af_packet module >> and new socket options to do things like hw queue split-off , map >> certain pages on I/O space to user space >> operations, etc. >> 4) There is a specific embedded MMU unit in Infiniband/RDMA to provides >> memory protection, while >> bifurcated driver uses IOMMU rather than NIC to provide memory >> protection. >> >> IMHO, libibverbs and corresponding kernel modules/drivers are specifically >> designed and implemented for >> direct access to RDMA hardware from userspace, and it highly depends on >> "verbs" related system calls >> supported by infiniband/rdma mechanism in kernel, rather than netdev >> mechanism that bifurcated driver >> solution depends on. >> >>> -----Original Message----- >>> From: Vincent JARDIN [mailto:vincent.jardin@6wind.com] >>> Sent: Thursday, November 06, 2014 9:31 AM >>> To: Zhou, Danny >>> Cc: Thomas Monjalon; dev@dpdk.org; Fastabend, John R; Or Gerlitz >>> Subject: Re: [dpdk-dev] bifurcated driver >>> >>> +Or >>> >>> On 05/11/2014 23:48, Zhou, Danny wrote: >>>> Hi Thomas, >>>> >>>> Thanks for sharing the links to ibverbs, I will take a close look at >> it and compare it to bifurcated driver. My take >>>> after a rough review is that idea is very much similar, but bifurcated >> driver implementation is generic for any >>>> Ethernet device based on existing af_packet mechanism, with extension >> of exchanging the messages between >>>> user space and kernel space driver. >>>> >>>> I have an internal document to summary the pros and cons of below >> solutions, except for ibvers, but >>>> will be adding it shortly. >>>> >>>> - igb_uio >>>> - uio_pci_generic >>>> - VFIO >>>> - bifurcated driver >>>> >>>> Short answers to your questions: >>>>> - upstream status >>>> Adding IOMMU based memory protection and generic descriptor >> description support now, into version 2 >>>> kernel patches. >>>> >>>>> - usable with kernel netdev >>>> af_packet based, and relevant patchset will be submitted to netdev for >> sure. >>>> >>>>> - usable in a vm >>>> No, it does no coexist with SRIOV for number of reasons. but if you >> pass-through a PF to a VM, it works perfect. >>>> >>>>> - usable for Ethernet >>>> It could work with all Ethernet NICs, as flow director is available >> and NIC driver support new net_ops to split off >>>> queue pairs for user space. >>>> >>>>> - hardware requirements >>>> No specific hardware requirements. All mainstream NICs have multiple >> qpairs and flow director support. >>>> >>>>> - security protection >>>> Leverage IOMMU to provide memory protection on Intel platform. Other >> archs provide similar memory protection >>>> mechanism, so we only use arch-agnostic DMA memory allocation APIs in >> kernel to support memory protection. >>>> >>>>> - performance >>>> DPDK native performance on user space queues, as long as drop_en is >> enabled to avoid head-of-line blocking. >>>> >>>> -Danny >>>> >>>>> -----Original Message----- >>>>> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] >>>>> Sent: Wednesday, November 05, 2014 9:01 PM >>>>> To: Zhou, Danny >>>>> Cc: dev@dpdk.org; Fastabend, John R >>>>> Subject: Re: [dpdk-dev] bifurcated driver >>>>> >>>>> Hi Danny, >>>>> >>>>> 2014-10-31 17:36, O'driscoll, Tim: >>>>>> Bifurcated Driver (Danny.Zhou@intel.com) >>>>> >>>>> Thanks for the presentation of bifurcated driver during the community >> call. >>>>> I asked if you looked at ibverbs and you wanted a link to check. >>>>> The kernel module is here: >>>>> >> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core >>>>> The userspace library: >>>>> http://git.kernel.org/cgit/libs/infiniband/libibverbs.git >>>>> >>>>> Extract from Kconfig: >>>>> " >>>>> config INFINIBAND_USER_ACCESS >>>>> tristate "InfiniBand userspace access (verbs and CM)" >>>>> select ANON_INODES >>>>> ---help--- >>>>> Userspace InfiniBand access support. This enables the >>>>> kernel side of userspace verbs and the userspace >>>>> communication manager (CM). This allows userspace processes >>>>> to set up connections and directly access InfiniBand >>>>> hardware for fast-path operations. You will also need >>>>> libibverbs, libibcm and a hardware driver library from >>>>> . >>>>> " >>>>> >>>>> It seems to be close to the bifurcated driver needs. >>>>> Not sure if it can solve the security issues if there is no dedicated >> MMU >>>>> in the NIC. >>>>> >>>>> I feel we should sum up pros and cons of >>>>> - igb_uio >>>>> - uio_pci_generic >>>>> - VFIO >>>>> - ibverbs >>>>> - bifurcated driver >>>>> I suggest to consider these criterias: >>>>> - upstream status >>>>> - usable with kernel netdev >>>>> - usable in a vm >>>>> - usable for ethernet >>>>> - hardware requirements >>>>> - security protection >>>>> - performance >>>>> >>>>> -- >>>>> Thomas >> >>