From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f180.google.com (mail-pf0-f180.google.com [209.85.192.180]) by dpdk.org (Postfix) with ESMTP id BA242374C for ; Wed, 9 Dec 2015 20:57:18 +0100 (CET) Received: by pfdd184 with SMTP id d184so34911945pfd.3 for ; Wed, 09 Dec 2015 11:57:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; bh=rdgpQd9YLQuVTesZp0oWxwN+PzJF/U5r9sCs07M+KxE=; b=yRBP+igFE92UOReipGcKw7rdweEQEohlHL2G07ksWtpWO4++xJSuwR+nmqr1dnPoiT oPfTti5FRCOybsnsXJm2qiRyvGtASt+OX+sVR1yFyQc3VK9h/LtZndhskwtjRTiNa0zk P527pWLzIqzMgKy7+PvTK0SeBLoTAZQWjT1iaeXMH+4rrPjmiQL8NzSKySp8YX1e23t1 0a1ehEh14vp4CRD1pXRA8bFb7H9BiZ/25Cg2jQD//CnkVC+P0yK+Q5XHza8EkHPOK65f 47zsAJOPL2Q22RdJd6uVkS8TVMNZV0nkT8XdoJTvOMxwtLbs4CgbuMPImlqz87YkZU7L PV6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-type:content-transfer-encoding; bh=rdgpQd9YLQuVTesZp0oWxwN+PzJF/U5r9sCs07M+KxE=; b=cyXZhYpJDrbYdOHnK2TQ5h8qpfDdcunxKVeMXpYqk+mUMcCREdwDqZSbnmGXMVRb0x xeqty9IYL1RmwCkMSgY36Xhe7QywoZAfenKrDpRLpDU2nzKZuHuA2esF/Gg4QGgvcSDU ZoJLqRUUBMHw9YrPcAT5FDJvZql6f0TI+2LtRzgIrk/+aoQtmNhaso9yJzJsstMA/BDJ 6F8zO69bGVX1D2mk/sYOX1Wg0go0SsS8G6Oq6N1mDnLIYC/dght6d1sFW+jIm6L1tF9N r+DwThtWF1zqnb+VQdVqgU60jvig0TR4eUPheAfzG6GAd45/AOfCXsAJViTe3zGwjk3+ KRUQ== X-Gm-Message-State: ALoCoQkr/KwCrpM9bG2AxbJNOvObMQHAa8I1HNsBZEaPybfgxeOMyJfW4anXZUfAOFh3UdUm3pl2/txGCa4OAtMRkrdNY0ozfg== X-Received: by 10.98.15.68 with SMTP id x65mr887389pfi.146.1449691038068; Wed, 09 Dec 2015 11:57:18 -0800 (PST) Received: from xeon-e3 (static-50-53-82-155.bvtn.or.frontiernet.net. [50.53.82.155]) by smtp.gmail.com with ESMTPSA id rz10sm13684182pac.29.2015.12.09.11.57.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Dec 2015 11:57:17 -0800 (PST) Date: Wed, 9 Dec 2015 11:57:25 -0800 From: Stephen Hemminger To: Santosh Shukla Message-ID: <20151209115725.0954d5c3@xeon-e3> In-Reply-To: References: <1449250519-28372-1-git-send-email-sshukla@mvista.com> <1449250519-28372-7-git-send-email-sshukla@mvista.com> <20151207090833.03f65d63@xeon-e3> <20151209110406.0740cfb9@xeon-e3> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: dev@dpdk.org, Rizwan Ansari Subject: Re: [dpdk-dev] [PATCH 6/6] virtio: arm/arm64: memory mapped IO support in pmd driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Dec 2015 19:57:19 -0000 On Thu, 10 Dec 2015 00:49:08 +0530 Santosh Shukla wrote: > On Thu, Dec 10, 2015 at 12:34 AM, Stephen Hemminger > wrote: > > On Thu, 10 Dec 2015 00:29:30 +0530 > > Santosh Shukla wrote: > > > >> On Tue, Dec 8, 2015 at 6:23 PM, Santosh Shukla wrote: > >> > > >> > > >> > On Mon, Dec 7, 2015 at 10:38 PM, Stephen Hemminger > >> > wrote: > >> >> > >> >> On Fri, 4 Dec 2015 23:05:19 +0530 > >> >> Santosh Shukla wrote: > >> >> > >> >> > > >> >> > +#ifdef RTE_EXEC_ENV_LINUXAPP > >> >> > +/* start address of first pci_iobar slot (user-space virtual-addres) */ > >> >> > +void *ioport_map; > >> >> > +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64) > >> >> > + > >> >> > +#include > >> >> > +#define DEV_NAME "/dev/igb_ioport" > >> >> > + > >> >> > +/* Keeping pci_ioport_size = 4k. > >> >> > + * So maximum mmaped pci_iobar supported = > >> >> > + * (ioport_size/pci_dev->mem_resource[0].len) > >> >> > + * > >> >> > + * note: kernel could allow maximum 32 virtio-net-pci interface, that > >> >> > mean > >> >> > + * maximum 32 PCI_IOBAR(s) where each PCI_IOBAR_LEN=0x20, so > >> >> > virtio_map_ioport() > >> >> > + * func by theory gonna support 4k/0x20 ==> 128 PCI_IOBAR(s), more than > >> >> > + * max-virtio-net-pci interface. > >> >> > + */ > >> >> > +#define PAGE_SIZE 4096 > >> >> > +#define PCI_IOPORT_SIZE PAGE_SIZE > >> >> > +#define PCI_IOPORT_MAX 128 /* 4k / 0x20 */ > >> >> > + > >> >> > +int ioport_map_cnt; > >> >> > +#endif /* ARM, ARM64 */ > >> >> > +#endif /* RTE_EXEC_ENV_LINUXAPP */ > >> >> > >> >> These variables should be static. > >> >> > >> > > >> > (Sorry for delayed follow, Was travelling..) > >> > Right, > >> > > >> >> > >> >> Also, it is should be possible to extract the I/O bar stuff in a generic > >> >> way through sysfs > >> >> and not depend on a character device. The long term goal for DPDK > >> >> acceptance is to > >> >> eliminate (or at least reduce to a minumum) any requirement for special > >> >> kernel drivers. > >> > > >> > > >> > I agree. Existing implementation does read pci_iobar for start address and > >> > size, But for non-x86 arch, we need someway to map pci_iobar and thats-why > >> > thought of adding device file for a purpose, as archs like arm lack iopl() > >> > privileged io syscall support, However iopl() too quite native driver design > >> > assumption. > >> > > >> > I have few idea in my mind such that - Right now I am updating ioport_mapped > >> > addr {kernel-virtual-addr-io-memory} to /sys/bus/pci/pci_bus_xxxx/xxx/map > >> > field, instead of mapping their, I'll try to map to uio's pci interface and > >> > then use existing pci_map_resource() api to mmap kernel-virtual-io-address > >> > to user-space-virtual-ioaddr. We'll come back on this. > >> > > >> > >> > >> Spent sometime digging dpdk's uio/pci source code, Intent was to map > >> pci ioport region via uio-way. In order to achieve that I tried to > >> hack the virtio-net-pci pmd driver. Right now in virtio-net-pci case, > >> It creates two sysfs entry for pci bars: resource0 /1. > >> > >> Resource0; is ioport region > >> Resource1; is iomem region. > >> > >> By appending a RTE_PCI_DRV_NEED_MAPPING flag to drv_flag and passing > >> hw->io_base = resource1 type pci.mem_resource[slot].addr; where slot > >> =1. Resource1 is IORESOURCE_MEM type so uio/pci driver able to mmap. > >> That way I could get the valid user-space virtual address. However > >> this hack did not worked for me because at qemu side: virtio-pxe.rom > >> has virtio_headers located at ioport pci region and guest driver > >> writing at iomem region, that's why driver init failed. Note that > >> default driver doesn't use resource1 memory. > >> > >> This made me think that either I had to add dependent code in kernel > >> or something similar proposed in this patch. > >> It is because: > >> - uio driver and dependent user-space pci api's in dpdk mmaps > >> IORESOURCE_MEM types address only {refer igbuio_setup_bars() and in > >> particular function pci_parse_sysfs_resource()}. > >> - That mmap in userspace backed by arch specific api > >> pci_mmap_page_range() in kernel. > >> - pci_mmap_page_range() does not support mmaping to IO_RESOURCE_IO type memory. > >> - Having said that, we need routine or a way to to map pci_iobar > >> region from kernel virtual-address to user-space virtual address. > > > > There a couple of gotcha's with this. It turns out the iomem region > > is not mappable on some platforms. I think GCE was one of them. > > afaik, In linux kernel if arch supports pci_mmap_page_range then iomem > region should map, right? I am confused by reading your reply, also I > am not aware of GCE? which platform is GCE, please suggest. I think it was Google Compute Environment that reported an memory region which was huge and not accessible, they have there own vhost.