From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f180.google.com (mail-pf0-f180.google.com [209.85.192.180]) by dpdk.org (Postfix) with ESMTP id A2286DE6 for ; Wed, 9 Dec 2015 19:59:31 +0100 (CET) Received: by pfbg73 with SMTP id g73so35012283pfb.1 for ; Wed, 09 Dec 2015 10:59:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mvista-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=fIpxBvYtmmS/gwbF8qbkIPG7pdv0HxQnyi9ly5l21Es=; b=dnZqbZ6L6/SQ5vwhVJejrorx7MJGtlCPFhbxi6oXrMBzt1Ww7Wf57xjyVwssro+gOV HwXkapsH6vBsVGdknh16g+8tmw+K8H0pvDzOkrPORK1btVPosCWl3AyZ16/2aJHjhDYj qgzy06GqhUd4rEKwT4k92rm4ctvvFOpK1AU54r66gTEDFiF+d9PyXLIOYCaVLj+p/sUI 8fsZiOARUzpNB5DgBB030qy9LNvfs/4lXFKLXS+xiOGOHxD2X+h+jhRhb1O6lKIoKU7V PXHfdQWTJeOkRL0xLsCjmKDhMrl4P6g6lyQysMY+827/wqvXvHrsgQ6nqCH5qQjsmPuE tk5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=fIpxBvYtmmS/gwbF8qbkIPG7pdv0HxQnyi9ly5l21Es=; b=lZs1Tbb5myst2DVVzgB7hbzOH8PeHGwzox1OaM3z+DWZWDtApBFkY+nfqhuYibK+4F gO2ptLNpPFLZsH2jUjsX60vhNvPO3kVW/NgXXUXoA88qLTGrwFXFiHvhUiE1H8d7MYVG QGt9zBpimpQ0oMwEJkYjkby49eSpIn98GS5rHYtYs69ER+lHW4kmquee9qtHEzzF+tL0 nVX3YXeu3kIBMUgrhvYobuLWlY8BbPMUsAdWr2hn3zb0xcbdBgvHdgqxzpa3NeiyUxmy EJYuNb0eRaQQadjis9CNfdQNh3brCExsykAADcglVzpTi7sANhDytEspYtH/vHAZBe2N 3yUQ== X-Gm-Message-State: ALoCoQmZsAHRhNWnd4xwJRshFrBe8XiHLPJVkuKsvAeRgScBzOBW7QUychsJ3pQgozcbzXxuKi9Stw7EQ8TII+8t78yF8ksd+lfkV/G2SCB3Iq+44pqefcA= MIME-Version: 1.0 X-Received: by 10.98.15.91 with SMTP id x88mr491439pfi.144.1449687570960; Wed, 09 Dec 2015 10:59:30 -0800 (PST) Received: by 10.66.13.233 with HTTP; Wed, 9 Dec 2015 10:59:30 -0800 (PST) In-Reply-To: References: <1449250519-28372-1-git-send-email-sshukla@mvista.com> <1449250519-28372-7-git-send-email-sshukla@mvista.com> <20151207090833.03f65d63@xeon-e3> Date: Thu, 10 Dec 2015 00:29:30 +0530 Message-ID: From: Santosh Shukla To: Stephen Hemminger Content-Type: text/plain; charset=UTF-8 Cc: dev@dpdk.org, Rizwan Ansari Subject: Re: [dpdk-dev] [PATCH 6/6] virtio: arm/arm64: memory mapped IO support in pmd driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Dec 2015 18:59:32 -0000 On Tue, Dec 8, 2015 at 6:23 PM, Santosh Shukla wrote: > > > On Mon, Dec 7, 2015 at 10:38 PM, Stephen Hemminger > wrote: >> >> On Fri, 4 Dec 2015 23:05:19 +0530 >> Santosh Shukla wrote: >> >> > >> > +#ifdef RTE_EXEC_ENV_LINUXAPP >> > +/* start address of first pci_iobar slot (user-space virtual-addres) */ >> > +void *ioport_map; >> > +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64) >> > + >> > +#include >> > +#define DEV_NAME "/dev/igb_ioport" >> > + >> > +/* Keeping pci_ioport_size = 4k. >> > + * So maximum mmaped pci_iobar supported = >> > + * (ioport_size/pci_dev->mem_resource[0].len) >> > + * >> > + * note: kernel could allow maximum 32 virtio-net-pci interface, that >> > mean >> > + * maximum 32 PCI_IOBAR(s) where each PCI_IOBAR_LEN=0x20, so >> > virtio_map_ioport() >> > + * func by theory gonna support 4k/0x20 ==> 128 PCI_IOBAR(s), more than >> > + * max-virtio-net-pci interface. >> > + */ >> > +#define PAGE_SIZE 4096 >> > +#define PCI_IOPORT_SIZE PAGE_SIZE >> > +#define PCI_IOPORT_MAX 128 /* 4k / 0x20 */ >> > + >> > +int ioport_map_cnt; >> > +#endif /* ARM, ARM64 */ >> > +#endif /* RTE_EXEC_ENV_LINUXAPP */ >> >> These variables should be static. >> > > (Sorry for delayed follow, Was travelling..) > Right, > >> >> Also, it is should be possible to extract the I/O bar stuff in a generic >> way through sysfs >> and not depend on a character device. The long term goal for DPDK >> acceptance is to >> eliminate (or at least reduce to a minumum) any requirement for special >> kernel drivers. > > > I agree. Existing implementation does read pci_iobar for start address and > size, But for non-x86 arch, we need someway to map pci_iobar and thats-why > thought of adding device file for a purpose, as archs like arm lack iopl() > privileged io syscall support, However iopl() too quite native driver design > assumption. > > I have few idea in my mind such that - Right now I am updating ioport_mapped > addr {kernel-virtual-addr-io-memory} to /sys/bus/pci/pci_bus_xxxx/xxx/map > field, instead of mapping their, I'll try to map to uio's pci interface and > then use existing pci_map_resource() api to mmap kernel-virtual-io-address > to user-space-virtual-ioaddr. We'll come back on this. > Spent sometime digging dpdk's uio/pci source code, Intent was to map pci ioport region via uio-way. In order to achieve that I tried to hack the virtio-net-pci pmd driver. Right now in virtio-net-pci case, It creates two sysfs entry for pci bars: resource0 /1. Resource0; is ioport region Resource1; is iomem region. By appending a RTE_PCI_DRV_NEED_MAPPING flag to drv_flag and passing hw->io_base = resource1 type pci.mem_resource[slot].addr; where slot =1. Resource1 is IORESOURCE_MEM type so uio/pci driver able to mmap. That way I could get the valid user-space virtual address. However this hack did not worked for me because at qemu side: virtio-pxe.rom has virtio_headers located at ioport pci region and guest driver writing at iomem region, that's why driver init failed. Note that default driver doesn't use resource1 memory. This made me think that either I had to add dependent code in kernel or something similar proposed in this patch. It is because: - uio driver and dependent user-space pci api's in dpdk mmaps IORESOURCE_MEM types address only {refer igbuio_setup_bars() and in particular function pci_parse_sysfs_resource()}. - That mmap in userspace backed by arch specific api pci_mmap_page_range() in kernel. - pci_mmap_page_range() does not support mmaping to IO_RESOURCE_IO type memory. - Having said that, we need routine or a way to to map pci_iobar region from kernel virtual-address to user-space virtual address. In x86 case; iopl() exports the ioport address to userspace. But for non-x86 case, we need special device file to mmap ioport kernel address space to user space. Also I grepped dpdk source code little more and noticed that xen dom0 implementation is quite similar too. They are too using /dev/dom0_mem special device file. IMO, there is approach-2 to achieve desired which I mentioned in cover-letter too. By adding /dev/ioport device file support in kernel, so that user could do byte/word/halfword rd/wr, rightnow kernel support only byte long rd/wr (i.e.. /dev/port file driver/char/mem.c). In that way; We care to do file open/rd/wr/close for /dev/ioport. No low level extra arch level api introduced in this patch series. Pretty clean code. But then we do have kernel device file dependency which is a cons. I personally like this approach as we are not dependant on kernel side driver, all desired dependency is in dpdk source base. I am going to send out approach-2 patch series, perhaps in this week for comment. Pl. let me know if you have better approach in mind, Otherwise I am planning to work on V2 patch revision, incorporating other review comment.