From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id B31595A51 for ; Wed, 16 Dec 2015 18:11:55 +0100 (CET) Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id AD1A98A14E; Wed, 16 Dec 2015 17:11:53 +0000 (UTC) Received: from t450s.home (ovpn-113-219.phx2.redhat.com [10.3.113.219]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id tBGHBqT2022812; Wed, 16 Dec 2015 12:11:53 -0500 Message-ID: <1450285912.2674.22.camel@redhat.com> From: Alex Williamson To: "Burakov, Anatoly" , "Yigit, Ferruh" Date: Wed, 16 Dec 2015 10:11:52 -0700 In-Reply-To: References: <60420822.AbcfvjLZCk@xps13> <566B4A50.9090607@6wind.com> <1449874953.20509.6.camel@redhat.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA6747CE55@IRSMSX108.ger.corp.intel.com> <1450198398.6042.32.camel@redhat.com> <20151216040408.GA18363@sivlogin002.ir.intel.com> <1450240711.2674.11.camel@redhat.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] VFIO no-iommu X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Dec 2015 17:11:55 -0000 On Wed, 2015-12-16 at 08:35 +0000, Burakov, Anatoly wrote: > Hi Alex, > > > On Wed, 2015-12-16 at 04:04 +0000, Ferruh Yigit wrote: > > > On Tue, Dec 15, 2015 at 09:53:18AM -0700, Alex Williamson wrote: > > > I tested the DPDK (HEAD of master) with the patch, with help of > > > Anatoly, and DPDK works in no-iommu environment with a little > > > modification. > > > > > > Basically the only modification is adapt new group naming > > > (noiommu-$) > > > and > > > > Sorry, forgot to mention that one.  The intention with the modified > > group > > name is that I want to be very certain that a user intending to > > only support > > properly iommu isolated devices doesn't accidentally need to deal > > with these > > no-iommu mode devices. > > > > > disable dma mapping (VFIO_IOMMU_MAP_DMA) > > > > > > Also I need to disable VFIO_CHECK_EXTENSION ioctl, because in > > > vfio > > > module, > > > container->noiommu is not set before doing a > > > vfio_group_set_container() > > > and vfio_for_each_iommu_driver selects wrong driver. > > > > Running CHECK_EXTENSION on a container without the group attached > > is > > only going to tell you what extensions vfio is capable of, not > > necessarily what > > extensions are available to you with that group.  Is this just a > > general dpdk- > > vfio ordering bug? > > Yes, that is how VFIO was implemented in DPDK. I was under the > impression that checking extension before assigning devices was the > correct way to do things, so as to not to try anything we know would > fail anyway. Does this imply that CHECK_EXTENSION needs to be called > on both container and groups (or just on groups)? Hmm, in Documentation/vfio.txt we do give the following algorithm:         if (ioctl(container, VFIO_GET_API_VERSION) != VFIO_API_VERSION)                 /* Unknown API version */         if (!ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU))                 /* Doesn't support the IOMMU driver we want. */         ... That's just going to query each iommu driver and we can't yet say whether the group the user attaches to the container later will actually support that extension until we try to do it, that would come at VFIO_SET_IOMMU.  So is it perhaps a vfio bug that we're not advertising no-iommu until the group is attached?  After all, we are capable of it with just an empty container, just like we are with type1, but we're going to fail SET_IOMMU for the wrong combination.  This is exactly the sort of thing that makes me glad we reverted it without feedback from a working user driver.  Thanks, Alex