From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <stephen@networkplumber.org>
Received: from mail-pf0-f180.google.com (mail-pf0-f180.google.com
[209.85.192.180]) by dpdk.org (Postfix) with ESMTP id 787F48D3A
for <dev@dpdk.org>; Fri, 18 Dec 2015 00:29:02 +0100 (CET)
Received: by mail-pf0-f180.google.com with SMTP id v86so38376176pfa.2
for <dev@dpdk.org>; Thu, 17 Dec 2015 15:29:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=networkplumber-org.20150623.gappssmtp.com; s=20150623;
h=date:from:to:cc:subject:message-id:in-reply-to:references
:mime-version:content-type:content-transfer-encoding;
bh=ky2rbqwGbeaSV/bsBeNiMlHfPKPMt6Ghn+s42z+GzYs=;
b=pOApNIzzP4q/ixXP+JHfILEO0yPeZZE84LfjtZVvf58r5v4ESFjD/XhiPBX1AEZ4Sm
151njCV4k0y/SjYHjDmK31WsywhoQnG1hbACE6AY1oUWtbwg8kotjbO9MGHTSVXl7vv8
ckdpaUPkiAEvj85hqJ/9N73i1gqYjhKaZQPLr0mhKA5tk1+SzuFzupMOLwcUKF+7xxEr
A48dG9p9bstuqohwfOgUVY/ga/W9SmI67FWKLlgFiNoQRpR7FekwYc3V8Vn3rjecEyun
jrCWGE+OatAmiQNoyHgk1SapHKsN7/GB3ulAGa4LYw3ZZ4yPkNxoy7Lb/1eJRNBCJM0C
wJlQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
:references:mime-version:content-type:content-transfer-encoding;
bh=ky2rbqwGbeaSV/bsBeNiMlHfPKPMt6Ghn+s42z+GzYs=;
b=DZv+Pn0VoIPQ7wGdQKn/yGOkPXYcTgRfyNLkJv3OqHxWUujWLVY2bVDFtRD5q+IDyW
r8gGwOV/dDGFZP8tEMfogkV8ew6ovgnQGGEl2X4JTzV1fQ3OTytPhvnPwSMc50Um3Sp3
4v6xaFrA09CfVoFJfCKvCmNoPObZKgay8yoL2eZWRLcU+vcKrk29rB6YnaLhqRC+l+0J
lxeTaKARVro3XgGqXEPJREsuFgaGOli+wN18Ak0x4suaHiTiGs/kzehIFn2hyAZYTMDX
MvrAAyHQbmkWtpK/FXLMEi8xu3MDCgXn0myYRXnAjRWVCty1/lbuMGJ0zVjP+XJVvCy8
m+5Q==
X-Gm-Message-State: ALoCoQnhR46wGrmdzz7oxSObF7MkWhgdhVOsGM5apn1a2+g7hLY+TeBQLno+7O86aq2ma4i59gJev9O7QT1THoq4paBpKKF48A==
X-Received: by 10.98.70.144 with SMTP id o16mr700937pfi.32.1450394941785;
Thu, 17 Dec 2015 15:29:01 -0800 (PST)
Received: from xeon-e3 (static-50-53-82-155.bvtn.or.frontiernet.net.
[50.53.82.155])
by smtp.gmail.com with ESMTPSA id sg4sm17919980pac.48.2015.12.17.15.29.01
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Thu, 17 Dec 2015 15:29:01 -0800 (PST)
Date: Thu, 17 Dec 2015 15:29:09 -0800
From: Stephen Hemminger <stephen@networkplumber.org>
To: Jan Viktorin <viktorin@rehivetech.com>
Message-ID: <20151217152909.4c7cbaff@xeon-e3>
In-Reply-To: <20151217203816.1bc6bd2c@jvn>
References: <60420822.AbcfvjLZCk@xps13> <17700135.Qc9aIsHGGP@xps13>
<C6ECDF3AB251BE4894318F4E45123697820B2C77@IRSMSX109.ger.corp.intel.com>
<2580440.9rYeIQLTBH@xps13> <20151217203816.1bc6bd2c@jvn>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] VFIO no-iommu
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 17 Dec 2015 23:29:02 -0000
On Thu, 17 Dec 2015 20:38:16 +0100
Jan Viktorin <viktorin@rehivetech.com> wrote:
> On Thu, 17 Dec 2015 11:09:23 +0100
> Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
>
> > Hi,
> >
> > 2015-12-17 09:52, Burakov, Anatoly:
> > > > > > > On Tue, Dec 15, 2015 at 09:53:18AM -0700, Alex Williamson wrote:
> > > > > > > So it works. Is it acceptable? Useful? Sufficiently complete?
> > > > > > > Does it imply deprecating the uio interface? I believe the
> > > > > > > feature that started this discussion was support for MSI/X
> > > > > > > interrupts so that VFs can support some kind of interrupt (uio
> > > > > > > only supports INTx since it doesn't allow DMA). Implementing that
> > > > > > > would be the ultimate test of whether this provides dpdk with not
> > > > > > > only a more consistent interface, but the feature dpdk wants
> > > > > > > that's missing in uio. Thanks,
> > > > >
> > > > > Ferruh has done a great job so far testing Alex's patch, very few changes
> > > > from DPDK side seem to be required as far as existing functionality goes (not
> > > > sure about VF interrupts mentioned by Alex). However, one thing that
> > > > concerns me is usability. While it is true that no-IOMMU mode in VFIO would
> > > > mean uio interfaces could be deprecated in time, the no-iommu mode is way
> > > > more hassle than using igb_uio/uio_pci_generic because it will require a
> > > > kernel recompile as opposed to simply compiling and insmod'ding an out-of-
> > > > tree driver. So, in essence, if you don't want an IOMMU, it's becoming that
> > > > much harder to use DPDK. Would that be something DPDK is willing to live
> > > > with in the absence of uio interfaces?
> > > >
> > > > Excuse me if I missed something obvious.
> > > > Why a kernel compilation is needed?
> > >
> > > Well, not really full kernel compilation, but in the default configuration, VFIO driver would not support NOIOMMU mode. I.e. it's not compiled by default. Support for no-iommu should be enabled in kernel config and compiled in. So, whoever is going to use DPDK with VFIO-no-iommu will have to download kernel tree and recompile the VFIO module and install it. That's obviously way more hassle than simply compiling an out-of-tree driver that's already included and works with an out-of-the-box kernel.
> >
> > The "out-of-the-box kernel" is configured by your distribution.
> > So we don't know yet what will be their choice.
> > If the distribution supports DPDK, it should be enabled.
>
> I have a question as I am not involved in all possible DPDK
> configurations, platforms, etc. and not yet very involved in vfio. What
> are the devices which do not have IOMMU? If I have, say, DPDK 2.3 with
> vfio-noiommu, which platforms (or computer systems) I am targeting?
>
> Would it be an Intel-based system? Would it be PPC8, ARM?
>
> If it is ARMv7... I would say that the fact I have to explicitly enable
> the no-IOMMU feature and rebuild the kernel (or whatever) is just OK. As
> for such systems, it is common to have a quite customized OS. Well,
> the big distributions are able to run on those devices, that's true...
> However, in such case, the users are usually skilled enough to take
> care of having their own special Linux kernel.
>
> So, is the fact the distributions would not support the no-IOMMU setup
> in their default configuration really an issue? Will some very common
> Intel/DPDK-based box need this?
>
> Regards
> Jan
So far:
* broken hardware (many systems including those from Dell) do not provide
working IOMMU because of Bios bugs etc.
* Linux guest in VMware/KVM/Hyper-V. There is no IOMMU emulation in most
of these systems.
* Older smaller systems (ie Atom) may not have IOMMU