From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f51.google.com (mail-pa0-f51.google.com [209.85.220.51]) by dpdk.org (Postfix) with ESMTP id AB557559C for ; Wed, 9 Dec 2015 20:03:58 +0100 (CET) Received: by pacwq6 with SMTP id wq6so33879508pac.1 for ; Wed, 09 Dec 2015 11:03:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; bh=iJluFBiS62c9cGm3fZv9FaxX1RkjJzmdue1oDAT/aJg=; b=dGPbtEVGBSzoqRqSiU+pCq8v/S43TjMYePBtQgPf/epLa8/zAzAtittMzPDW33MPZE dLp/P7WgPBbFjFqEONI4kTBeovBaCFePJhJRdZ7j8ofz8B1Wj51xGSnuR+rXZ0iuE5k8 iMOANAXIuAP7BkIwARRAyCeGi0hYXUp/o4szW0UFpi/5DMqDs/hQGkAvIAJYmyrvYWSx WwGxon7NEFJryBUL5qOrRf7el9PxDp6cvQvYGxaqYSoHgZ4FCpNsRn4rb3C6grDicTCH bnUFvwxUkPwf1b9VqpgZb05oVp13rNDo7gP+jpnx9u11Hv4QLlbTYlinsdcptJtAQ+H0 LyAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-type:content-transfer-encoding; bh=iJluFBiS62c9cGm3fZv9FaxX1RkjJzmdue1oDAT/aJg=; b=kBcQ0ZUhuddigDiui6pBzRWIBh02lvK5PJkWiuQ9Hkzo96KVs+Hv5Y4WcL65rHN0zT z73wBm5tQP2ETXU1cCnMjew15sZl8R3OdFJLkQFRO0Y+Ojbw53+XEHZJMTlFRwUmSZwk pzHaGRs3IRK4TbEwFMYSFW/t8hU8wLKEYdgyI+NbuhajdBxuJotKaCSXK3Z6VJggKyLK dRMWF0j7CgPPlayAO8e0h/0LgH64QMG3Wo9QJeXqTqHC4KQBoIONMf+HytlWyE9eaE8Q FwwZP60WORL4pggJ6WvNuAGPm9oXoAj7npApQOvACZSwAR0RTpqrhGD0beR+8hVVs2Rl kK+Q== X-Gm-Message-State: ALoCoQnHTvzJlBtiw6rqX4syDzfnoNjzeA9RtFpAGO9jb2nqPRSI6p9skJmQL6jxPwNW76Z4zmOu2idxPIO15Ygq9TiPAWbOcw== X-Received: by 10.66.156.106 with SMTP id wd10mr10271106pab.150.1449687837942; Wed, 09 Dec 2015 11:03:57 -0800 (PST) Received: from xeon-e3 (static-50-53-82-155.bvtn.or.frontiernet.net. [50.53.82.155]) by smtp.gmail.com with ESMTPSA id r5sm13283928pfi.73.2015.12.09.11.03.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Dec 2015 11:03:57 -0800 (PST) Date: Wed, 9 Dec 2015 11:04:06 -0800 From: Stephen Hemminger To: Santosh Shukla Message-ID: <20151209110406.0740cfb9@xeon-e3> In-Reply-To: References: <1449250519-28372-1-git-send-email-sshukla@mvista.com> <1449250519-28372-7-git-send-email-sshukla@mvista.com> <20151207090833.03f65d63@xeon-e3> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: dev@dpdk.org, Rizwan Ansari Subject: Re: [dpdk-dev] [PATCH 6/6] virtio: arm/arm64: memory mapped IO support in pmd driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Dec 2015 19:03:58 -0000 On Thu, 10 Dec 2015 00:29:30 +0530 Santosh Shukla wrote: > On Tue, Dec 8, 2015 at 6:23 PM, Santosh Shukla wrote: > > > > > > On Mon, Dec 7, 2015 at 10:38 PM, Stephen Hemminger > > wrote: > >> > >> On Fri, 4 Dec 2015 23:05:19 +0530 > >> Santosh Shukla wrote: > >> > >> > > >> > +#ifdef RTE_EXEC_ENV_LINUXAPP > >> > +/* start address of first pci_iobar slot (user-space virtual-addres) */ > >> > +void *ioport_map; > >> > +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64) > >> > + > >> > +#include > >> > +#define DEV_NAME "/dev/igb_ioport" > >> > + > >> > +/* Keeping pci_ioport_size = 4k. > >> > + * So maximum mmaped pci_iobar supported = > >> > + * (ioport_size/pci_dev->mem_resource[0].len) > >> > + * > >> > + * note: kernel could allow maximum 32 virtio-net-pci interface, that > >> > mean > >> > + * maximum 32 PCI_IOBAR(s) where each PCI_IOBAR_LEN=0x20, so > >> > virtio_map_ioport() > >> > + * func by theory gonna support 4k/0x20 ==> 128 PCI_IOBAR(s), more than > >> > + * max-virtio-net-pci interface. > >> > + */ > >> > +#define PAGE_SIZE 4096 > >> > +#define PCI_IOPORT_SIZE PAGE_SIZE > >> > +#define PCI_IOPORT_MAX 128 /* 4k / 0x20 */ > >> > + > >> > +int ioport_map_cnt; > >> > +#endif /* ARM, ARM64 */ > >> > +#endif /* RTE_EXEC_ENV_LINUXAPP */ > >> > >> These variables should be static. > >> > > > > (Sorry for delayed follow, Was travelling..) > > Right, > > > >> > >> Also, it is should be possible to extract the I/O bar stuff in a generic > >> way through sysfs > >> and not depend on a character device. The long term goal for DPDK > >> acceptance is to > >> eliminate (or at least reduce to a minumum) any requirement for special > >> kernel drivers. > > > > > > I agree. Existing implementation does read pci_iobar for start address and > > size, But for non-x86 arch, we need someway to map pci_iobar and thats-why > > thought of adding device file for a purpose, as archs like arm lack iopl() > > privileged io syscall support, However iopl() too quite native driver design > > assumption. > > > > I have few idea in my mind such that - Right now I am updating ioport_mapped > > addr {kernel-virtual-addr-io-memory} to /sys/bus/pci/pci_bus_xxxx/xxx/map > > field, instead of mapping their, I'll try to map to uio's pci interface and > > then use existing pci_map_resource() api to mmap kernel-virtual-io-address > > to user-space-virtual-ioaddr. We'll come back on this. > > > > > Spent sometime digging dpdk's uio/pci source code, Intent was to map > pci ioport region via uio-way. In order to achieve that I tried to > hack the virtio-net-pci pmd driver. Right now in virtio-net-pci case, > It creates two sysfs entry for pci bars: resource0 /1. > > Resource0; is ioport region > Resource1; is iomem region. > > By appending a RTE_PCI_DRV_NEED_MAPPING flag to drv_flag and passing > hw->io_base = resource1 type pci.mem_resource[slot].addr; where slot > =1. Resource1 is IORESOURCE_MEM type so uio/pci driver able to mmap. > That way I could get the valid user-space virtual address. However > this hack did not worked for me because at qemu side: virtio-pxe.rom > has virtio_headers located at ioport pci region and guest driver > writing at iomem region, that's why driver init failed. Note that > default driver doesn't use resource1 memory. > > This made me think that either I had to add dependent code in kernel > or something similar proposed in this patch. > It is because: > - uio driver and dependent user-space pci api's in dpdk mmaps > IORESOURCE_MEM types address only {refer igbuio_setup_bars() and in > particular function pci_parse_sysfs_resource()}. > - That mmap in userspace backed by arch specific api > pci_mmap_page_range() in kernel. > - pci_mmap_page_range() does not support mmaping to IO_RESOURCE_IO type memory. > - Having said that, we need routine or a way to to map pci_iobar > region from kernel virtual-address to user-space virtual address. There a couple of gotcha's with this. It turns out the iomem region is not mappable on some platforms. I think GCE was one of them.