From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <sshukla@mvista.com> Received: from mail-pa0-f50.google.com (mail-pa0-f50.google.com [209.85.220.50]) by dpdk.org (Postfix) with ESMTP id 01D1F5A6F for <dev@dpdk.org>; Thu, 31 Dec 2015 15:27:44 +0100 (CET) Received: by mail-pa0-f50.google.com with SMTP id cy9so152870920pac.0 for <dev@dpdk.org>; Thu, 31 Dec 2015 06:27:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mvista-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=KfiQNk9penCsmCHTeiLujVnqOmJhKWu7Zqj1XZgVqXA=; b=wmDT2so7WIl7Ix+e7PqzKGllOT+AZMcSPXaLD0L+YTiVnsOB9qwQsMjcL//wTEkHPD RreS11zcPWp+9fVtUsaMkBSCiwtkhO5DV1j8qSOljjbzpyUa8u1IOAioBbthwHco0xAI qQj0Q7O2P3Pol1BRTgyjnti/zIQBu2JW1LHGJRHIB1dYwV57/t4NZ9Al0/VW+jeOB/Fw VX3xG5LifiTsZpzXK+vtHlq8WCee/jHhEbpxa2TK/ZL0msoGXA79vOxkuA0jyh8PG00D OFfuRx5oCMTr5C9EUgqhu8+9sbe0B4X+TLNljxnHXtSV0jkSbOWPxRQt6mpgnvo69hkS BtyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=KfiQNk9penCsmCHTeiLujVnqOmJhKWu7Zqj1XZgVqXA=; b=kaBAU2bXLXpMl2ctxwz7Da4FfA/aavkxJgSgIXN0zH2dxvMy10iG9y7QXVLaXAaIwR eEJIY2LJVMLgQ5FvOJH5PFeQsDXzwAxaXV7TFRSGS/EFZLFqjLW57c1K5ZDly7o12yCm xjLiOaAolXzMUl6G8jJP+Kf+lP+AqWmRLCJ+Zi6XhCti6Ky9ONWypBXHxyKqPmi9Sa1t yDasEyenXk5zouD+3TSEChWFkCkN9h8l7WkKt119SHQOF9CE+DvD3RSeI8b5nRQ5v710 6d6Bp3amgN/PSh7y219K2eW55FanPNebHpu7gdnaXOwaC8Q3Wb9+IDq/HpyxQ/sUSsRM Dhag== X-Gm-Message-State: ALoCoQn/8SAHBgeM/tzR31OcPibHptgWWNwOaek6wMsBj1KDU9sB7Gfpe4kYXEMHcLo9IIyFOQ3WcD403BNWqef9qtZkYRC3iAR/8irv7q151ZWlLYQLViQ= MIME-Version: 1.0 X-Received: by 10.66.163.231 with SMTP id yl7mr98726243pab.141.1451572064184; Thu, 31 Dec 2015 06:27:44 -0800 (PST) Received: by 10.66.196.81 with HTTP; Thu, 31 Dec 2015 06:27:44 -0800 (PST) In-Reply-To: <CAAyOgsZUrdWsCEuescF+CF5TsrhNH5-soQJ9ay_zbKiFu_t9xQ@mail.gmail.com> References: <CAAyOgsaMO7V1K8Yxh=Zf-E4iodDevMFG+rRBPgZXBysca9JopA@mail.gmail.com> <2241331.HNmyzf8foi@xps13> <CAAyOgsad_+fD5yNB83UG_JwBhYhJLWoLHCiH0XQb3pxb3AEcdg@mail.gmail.com> <2979402.yeVYlcCDUH@xps13> <CAAyOgsYp8mn5d+bbFxo34AWSxb21kMfv0h_wg7r=mH1NPxHEYQ@mail.gmail.com> <20151218053053.GL29571@yliu-dev.sh.intel.com> <CAAyOgsbRPMhHo5=-WuRkZzrQfmQQ9ZDHcJbHOPDpLfHJk1nfVA@mail.gmail.com> <20151218082139.GC18863@yliu-dev.sh.intel.com> <CAAyOgsbBMd8jsuAiiAuvwS=K84s3TgvCuZS9aWHu7iPZnEPXrQ@mail.gmail.com> <CAAyOgsbTDDCSQ+Qkn5tOpGNc3m1Y6BFH5hLwQ_Fuk3hYM7KAaQ@mail.gmail.com> <C6ECDF3AB251BE4894318F4E45123697820BB086@IRSMSX109.ger.corp.intel.com> <CAAyOgsZg6W4LZTCw0b864KiDDcot71b0AuZz6J3Zmmr_asmG3w@mail.gmail.com> <1451397878.18084.4.camel@redhat.com> <CAAyOgsZUrdWsCEuescF+CF5TsrhNH5-soQJ9ay_zbKiFu_t9xQ@mail.gmail.com> Date: Thu, 31 Dec 2015 19:57:44 +0530 Message-ID: <CAAyOgsaN3S=Jy2GXb9omJs0CJFbjy+1vYBw6Pv70b0Zxxk7+gA@mail.gmail.com> From: Santosh Shukla <sshukla@mvista.com> To: Alex Williamson <alex.williamson@redhat.com> Content-Type: text/plain; charset=UTF-8 Cc: "dev@dpdk.org" <dev@dpdk.org> Subject: Re: [dpdk-dev] [PATCH] eal: map io resources for non x86 architectures X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 31 Dec 2015 14:27:45 -0000 On Tue, Dec 29, 2015 at 8:21 PM, Santosh Shukla <sshukla@mvista.com> wrote: > On Tue, Dec 29, 2015 at 7:34 PM, Alex Williamson > <alex.williamson@redhat.com> wrote: >> On Tue, 2015-12-29 at 16:17 +0530, Santosh Shukla wrote: >>> On Tue, Dec 29, 2015 at 3:26 PM, Burakov, Anatoly >>> <anatoly.burakov@intel.com> wrote: >>> > Hi Santosh, >>> > >>> > > On Fri, Dec 18, 2015 at 6:25 PM, Santosh Shukla <sshukla@mvista.c >>> > > om> >>> > > wrote: >>> > > > On Fri, Dec 18, 2015 at 1:51 PM, Yuanhan Liu >>> > > > <yuanhan.liu@linux.intel.com> wrote: >>> > > > > On Fri, Dec 18, 2015 at 01:24:41PM +0530, Santosh Shukla >>> > > > > wrote: >>> > > > > > > > I guess we have done enough evaluation / investigation >>> > > > > > > > that >>> > > > > > > > suggest - so to map iopci region to userspace in arch >>> > > > > > > > agnostic-way - >>> > > > > > > > >>> > > > > > > > # either we need to modify kernel >>> > > > > > > > - Make sure all the non-x86 arch to >>> > > > > > > > support >>> > > > > > > > mapping for iopci region (i.e. pci_mmap_page_range). I >>> > > > > > > > don;t >>> > > > > > > > think its a correct approach though. >>> > > > > > > > or >>> > > > > > > > - include /dev/ioport char-mem device >>> > > > > > > > file who >>> > > > > > > > could do more than byte operation, Note that this >>> > > > > > > > implementation >>> > > > > > > > does not exist in kernel. I could send an RFC to lkml. >>> > > > > > > >>> > > > > > > Maybe you could propose the two to lkml, to get some >>> > > > > > > feedbacks >>> > > > > > > from those kernel/ARM gurus? Please cc me if you do so. >>> > > > > > > >>> > > > > > >>> > > > > > The latter one I already shared old lkml thread, Pl. >>> > > > > > revisit my v1 >>> > > > > > 0/6 patch [1] and in that refer [2]. >>> > > > > >>> > > > > Oops, sorry, a bit busy, that I didn't look at it carefully. >>> > > > > My bad, >>> > > > > anyway. >>> > > > > >>> > > > > > Josh has already proposed to lkml but for some reason >>> > > > > > thread didn't >>> > > > > > went far. I can restart that discussion giving dpdk use- >>> > > > > > case as an >>> > > > > > example/ requirement. >>> > > > > >>> > > > > I had a quick go through of the discussion. Both hpa and Arnd >>> > > > > seem to >>> > > > > be fine with the ioctl interface on /dev/port. Have you tried >>> > > > > that? >>> > > > > And if you want to restart it, ioctl might be a better option >>> > > > > than >>> > > > > /dev/ioport, judging from the discussion. >>> > > > > >>> > > > >>> > > > I tried legacy patch and re-writing with ioctl-way; doing >>> > > > changes in >>> > > > dpdk port as well in kernel, had to test on quite other hw not >>> > > > only >>> > > > arm64 though! so it will take time for me, I am travelling >>> > > > tomorrow so >>> > > > bit delayed, We'll post patch to lkml and share dpdk-virtio >>> > > > feedback >>> > > > perhaps by Monday. >>> > > > >>> > > >>> > > I posted a query about /dev/ioports approach in lkml thread [1], >>> > > and Arnd >>> > > suggested to use vfio framework but it looks like vfio too does >>> > > not map >>> > > ioresource_io region. Same communicated to Arnd and waiting for >>> > > his reply. >>> > > >>> > > In mean time I like to ask general question; >>> > > - Has anyone tried vfio/non-iommu approach for virtio pmd driver? >>> > > If not >>> > > then Is there any plan? Someone planning to take up. >>> > > [1] https://lkml.org/lkml/2015/12/23/145 >>> > >>> > I have submitted a patch to support no-iommu mode, but I'm not >>> > aware of anyone trying VFIO-noiommu at all. That's probably >>> > expected since it's Christmas/New Year in a lot of places, and >>> > everyone is on a break. >>> > >>> > That said, I'm not sure I completely understand what is it that >>> > you're asking about. The code you're referring to (in vfio_pci.c, >>> > line 854 as of kernel 4.3) is checking if a PCI BAR is available >>> > for IO (hence checking if the IORESOURCE_MEM >>> >>> >>> Thanks for reply! You comment might help to move this discuss to next >>> level. >>> >>> Look at kernel/resource.c, it exports two symbol ioport_resource and >>> iomem_resource and sets appropriate flag type i.e.. IORESOURCE_IO and >>> IORESOURCE_MEM. In virtio-net case; it creates both pci region i.e.. >>> _io bar and _mem bar. dpdk virtio pmd driver (<= 0.95 virtio spec) >>> uses pci _io bar region for device initialization as virtio headers >>> are locate at pci _io bar region. Since it uses pci _iobar region so >>> likely it update pci_resource.[index].flag = IORESOURCE_IO. and vfio >>> mmap function wont handle ioresource_io (i guess). And that is why I >>> asked same to lkml thread. >>> >>> >>> bit is set). There isn't any "ioresource_mem" region as far as VFIO >>> is >>> concerned, there are only BARs, ROM, VGA and PCI >>> config regions (linux/vfio.h, line 314 as of kernel 4.3). So if >>> you're >>> missing some PCI regions for VFIO to map, they would first need to be >>> added to VFIO kernel implementation before they can be used by DPDK. >>> That is, unless I'm misunderstanding something :) >>> > >>> >>> Agree. As mentioned above, I guess ioresource_io pci bar >>> implementation missing in vfio kernel? but before adding code support >>> in kernel I would like to hear from experts example, You, Alex! >>> (looping alex) >> >> MMIO and I/O port space are simply regions as far as VFIO is concerned. >> The userspace API supports the concept of I/O port regions that can be >> mmap'd by userspace, but the implementation does not since I/O port >> regions cannot be mmap'd on x86. Someone needs to propose some code >> for vfio-pci to support it. Thanks, >> > > I will work on this and post an RFC soon to LKML, That RFC targetted > to map IOport region for non-x86 platform in particular. Thanks for > confirming the vfio behaviour. > Some update: Patches used for setup - - Used Alex(s) vfio-noiommu linux-next patch - then used Anatoly(s) vfio-noiommu dpdk patch [1] - Added ioport pci bar map code support in kernel - Did similar change at dpdk(s) eal_pci_vfio.c file I could able to run testpmd on virtio-net dpdk's pmd driver.. I am cleaning the patches / changes in kernel and dpdk, soon posting the v3 patchset.. Thanks for the input! [1] http://dpdk.org/dev/patchwork/patch/9619/ >> Alex