From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f44.google.com (mail-pa0-f44.google.com [209.85.220.44]) by dpdk.org (Postfix) with ESMTP id 6215D58D4 for ; Wed, 9 Dec 2015 20:19:09 +0100 (CET) Received: by pacej9 with SMTP id ej9so34268142pac.2 for ; Wed, 09 Dec 2015 11:19:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mvista-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hwE1xCYhv8URXww5zjgorB12ch8AZFHUx7PkeL0c9d4=; b=YHsaouTqG7H4pci95Y6fxiBWH0/fx10pFYJ9oUdy/b5ygXUkb/+IbD80apwzPgytGi znvC6RPtElNN7ZvYxaMthExwI3zKrDPFqdwzRP6+Zf79ybQ+S+P3GAhbcrGvtY4wwmwB 6nx0stnqFWYHM+KheYUFKMzxNJ9O5+1f2yHzpNBEY2ejeBHY1vrMcVTVay/FnMX7/B1P 8vf3v+EM+tg58jjYyGUqonbjONJrlNC6V52YhtRjx7qsOGsTrHL6faYJDu4JF0Eb6o71 B9yH+INE7TYQPcXyauLMEigM4gPcENcr3+CIRNzhOnkrJj8CiEonzWBAVgvQqkR5+RbX qISg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=hwE1xCYhv8URXww5zjgorB12ch8AZFHUx7PkeL0c9d4=; b=fbHRFp86529U/FsCZDgct/WmdbVUgFu12/PKtqkNXs0NNjwgyoM6VKlA1vVJpv+cFk C5j8VmzEfN2h8D1z0cSmL4PB3CemQwpy2eEWQtohFixBrSI9Bi6WZdz/F57stAVDRpS+ zT4EsEjY4g04O/DAQviCOPJwOGsA4SzRuDB+kw8bDOgeFmkrdMS36h4eWqtE+QP6YCBt EIecXFmxFxD6DLi337tNVU0vG57dSZxF7r4mACLsYJQRiI4C7wOw2/+Ffq/LTqXBsLaj ZprdzfyVbWzhRbgSypoDqdjRhtrCuq7TWFehSj9h+hU7dPiQfIhRMB+C6qKXVkbQc+V+ Mlkw== X-Gm-Message-State: ALoCoQn6rhRqgf0/0IvpXva825zd2g5v76qOpCeNO9TA2BdZ6f6GLAShvnDWC+i0lFzsKghhPrmR51GQd4W+waYXBprj2t7zcvNi6WPP4y1hYOSMgLv8Mmg= MIME-Version: 1.0 X-Received: by 10.66.140.79 with SMTP id re15mr10028206pab.127.1449688748396; Wed, 09 Dec 2015 11:19:08 -0800 (PST) Received: by 10.66.13.233 with HTTP; Wed, 9 Dec 2015 11:19:08 -0800 (PST) In-Reply-To: <20151209110406.0740cfb9@xeon-e3> References: <1449250519-28372-1-git-send-email-sshukla@mvista.com> <1449250519-28372-7-git-send-email-sshukla@mvista.com> <20151207090833.03f65d63@xeon-e3> <20151209110406.0740cfb9@xeon-e3> Date: Thu, 10 Dec 2015 00:49:08 +0530 Message-ID: From: Santosh Shukla To: Stephen Hemminger Content-Type: text/plain; charset=UTF-8 Cc: dev@dpdk.org, Rizwan Ansari Subject: Re: [dpdk-dev] [PATCH 6/6] virtio: arm/arm64: memory mapped IO support in pmd driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Dec 2015 19:19:09 -0000 On Thu, Dec 10, 2015 at 12:34 AM, Stephen Hemminger wrote: > On Thu, 10 Dec 2015 00:29:30 +0530 > Santosh Shukla wrote: > >> On Tue, Dec 8, 2015 at 6:23 PM, Santosh Shukla wrote: >> > >> > >> > On Mon, Dec 7, 2015 at 10:38 PM, Stephen Hemminger >> > wrote: >> >> >> >> On Fri, 4 Dec 2015 23:05:19 +0530 >> >> Santosh Shukla wrote: >> >> >> >> > >> >> > +#ifdef RTE_EXEC_ENV_LINUXAPP >> >> > +/* start address of first pci_iobar slot (user-space virtual-addres) */ >> >> > +void *ioport_map; >> >> > +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64) >> >> > + >> >> > +#include >> >> > +#define DEV_NAME "/dev/igb_ioport" >> >> > + >> >> > +/* Keeping pci_ioport_size = 4k. >> >> > + * So maximum mmaped pci_iobar supported = >> >> > + * (ioport_size/pci_dev->mem_resource[0].len) >> >> > + * >> >> > + * note: kernel could allow maximum 32 virtio-net-pci interface, that >> >> > mean >> >> > + * maximum 32 PCI_IOBAR(s) where each PCI_IOBAR_LEN=0x20, so >> >> > virtio_map_ioport() >> >> > + * func by theory gonna support 4k/0x20 ==> 128 PCI_IOBAR(s), more than >> >> > + * max-virtio-net-pci interface. >> >> > + */ >> >> > +#define PAGE_SIZE 4096 >> >> > +#define PCI_IOPORT_SIZE PAGE_SIZE >> >> > +#define PCI_IOPORT_MAX 128 /* 4k / 0x20 */ >> >> > + >> >> > +int ioport_map_cnt; >> >> > +#endif /* ARM, ARM64 */ >> >> > +#endif /* RTE_EXEC_ENV_LINUXAPP */ >> >> >> >> These variables should be static. >> >> >> > >> > (Sorry for delayed follow, Was travelling..) >> > Right, >> > >> >> >> >> Also, it is should be possible to extract the I/O bar stuff in a generic >> >> way through sysfs >> >> and not depend on a character device. The long term goal for DPDK >> >> acceptance is to >> >> eliminate (or at least reduce to a minumum) any requirement for special >> >> kernel drivers. >> > >> > >> > I agree. Existing implementation does read pci_iobar for start address and >> > size, But for non-x86 arch, we need someway to map pci_iobar and thats-why >> > thought of adding device file for a purpose, as archs like arm lack iopl() >> > privileged io syscall support, However iopl() too quite native driver design >> > assumption. >> > >> > I have few idea in my mind such that - Right now I am updating ioport_mapped >> > addr {kernel-virtual-addr-io-memory} to /sys/bus/pci/pci_bus_xxxx/xxx/map >> > field, instead of mapping their, I'll try to map to uio's pci interface and >> > then use existing pci_map_resource() api to mmap kernel-virtual-io-address >> > to user-space-virtual-ioaddr. We'll come back on this. >> > >> >> >> Spent sometime digging dpdk's uio/pci source code, Intent was to map >> pci ioport region via uio-way. In order to achieve that I tried to >> hack the virtio-net-pci pmd driver. Right now in virtio-net-pci case, >> It creates two sysfs entry for pci bars: resource0 /1. >> >> Resource0; is ioport region >> Resource1; is iomem region. >> >> By appending a RTE_PCI_DRV_NEED_MAPPING flag to drv_flag and passing >> hw->io_base = resource1 type pci.mem_resource[slot].addr; where slot >> =1. Resource1 is IORESOURCE_MEM type so uio/pci driver able to mmap. >> That way I could get the valid user-space virtual address. However >> this hack did not worked for me because at qemu side: virtio-pxe.rom >> has virtio_headers located at ioport pci region and guest driver >> writing at iomem region, that's why driver init failed. Note that >> default driver doesn't use resource1 memory. >> >> This made me think that either I had to add dependent code in kernel >> or something similar proposed in this patch. >> It is because: >> - uio driver and dependent user-space pci api's in dpdk mmaps >> IORESOURCE_MEM types address only {refer igbuio_setup_bars() and in >> particular function pci_parse_sysfs_resource()}. >> - That mmap in userspace backed by arch specific api >> pci_mmap_page_range() in kernel. >> - pci_mmap_page_range() does not support mmaping to IO_RESOURCE_IO type memory. >> - Having said that, we need routine or a way to to map pci_iobar >> region from kernel virtual-address to user-space virtual address. > > There a couple of gotcha's with this. It turns out the iomem region > is not mappable on some platforms. I think GCE was one of them. afaik, In linux kernel if arch supports pci_mmap_page_range then iomem region should map, right? I am confused by reading your reply, also I am not aware of GCE? which platform is GCE, please suggest.