From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <sshukla@mvista.com>
Received: from mail-pa0-f44.google.com (mail-pa0-f44.google.com
[209.85.220.44]) by dpdk.org (Postfix) with ESMTP id 6215D58D4
for <dev@dpdk.org>; Wed, 9 Dec 2015 20:19:09 +0100 (CET)
Received: by pacej9 with SMTP id ej9so34268142pac.2
for <dev@dpdk.org>; Wed, 09 Dec 2015 11:19:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=mvista-com.20150623.gappssmtp.com; s=20150623;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:cc:content-type;
bh=hwE1xCYhv8URXww5zjgorB12ch8AZFHUx7PkeL0c9d4=;
b=YHsaouTqG7H4pci95Y6fxiBWH0/fx10pFYJ9oUdy/b5ygXUkb/+IbD80apwzPgytGi
znvC6RPtElNN7ZvYxaMthExwI3zKrDPFqdwzRP6+Zf79ybQ+S+P3GAhbcrGvtY4wwmwB
6nx0stnqFWYHM+KheYUFKMzxNJ9O5+1f2yHzpNBEY2ejeBHY1vrMcVTVay/FnMX7/B1P
8vf3v+EM+tg58jjYyGUqonbjONJrlNC6V52YhtRjx7qsOGsTrHL6faYJDu4JF0Eb6o71
B9yH+INE7TYQPcXyauLMEigM4gPcENcr3+CIRNzhOnkrJj8CiEonzWBAVgvQqkR5+RbX
qISg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:mime-version:in-reply-to:references:date
:message-id:subject:from:to:cc:content-type;
bh=hwE1xCYhv8URXww5zjgorB12ch8AZFHUx7PkeL0c9d4=;
b=fbHRFp86529U/FsCZDgct/WmdbVUgFu12/PKtqkNXs0NNjwgyoM6VKlA1vVJpv+cFk
C5j8VmzEfN2h8D1z0cSmL4PB3CemQwpy2eEWQtohFixBrSI9Bi6WZdz/F57stAVDRpS+
zT4EsEjY4g04O/DAQviCOPJwOGsA4SzRuDB+kw8bDOgeFmkrdMS36h4eWqtE+QP6YCBt
EIecXFmxFxD6DLi337tNVU0vG57dSZxF7r4mACLsYJQRiI4C7wOw2/+Ffq/LTqXBsLaj
ZprdzfyVbWzhRbgSypoDqdjRhtrCuq7TWFehSj9h+hU7dPiQfIhRMB+C6qKXVkbQc+V+
Mlkw==
X-Gm-Message-State: ALoCoQn6rhRqgf0/0IvpXva825zd2g5v76qOpCeNO9TA2BdZ6f6GLAShvnDWC+i0lFzsKghhPrmR51GQd4W+waYXBprj2t7zcvNi6WPP4y1hYOSMgLv8Mmg=
MIME-Version: 1.0
X-Received: by 10.66.140.79 with SMTP id re15mr10028206pab.127.1449688748396;
Wed, 09 Dec 2015 11:19:08 -0800 (PST)
Received: by 10.66.13.233 with HTTP; Wed, 9 Dec 2015 11:19:08 -0800 (PST)
In-Reply-To: <20151209110406.0740cfb9@xeon-e3>
References: <1449250519-28372-1-git-send-email-sshukla@mvista.com>
<1449250519-28372-7-git-send-email-sshukla@mvista.com>
<20151207090833.03f65d63@xeon-e3>
<CAAyOgsbR81AYyg8Sv3SXx7mNqO=WaCnMn9zO=cd6oMzFdShjWw@mail.gmail.com>
<CAAyOgsaHLMRWa_659ocH9t5T+D0hgWn_P2ka0rn=a_ahZeD61w@mail.gmail.com>
<20151209110406.0740cfb9@xeon-e3>
Date: Thu, 10 Dec 2015 00:49:08 +0530
Message-ID: <CAAyOgsY_DS2bwDzh0jC3+cN6P2osO-nzYSJPNkGtyTzFuiYKgg@mail.gmail.com>
From: Santosh Shukla <sshukla@mvista.com>
To: Stephen Hemminger <stephen@networkplumber.org>
Content-Type: text/plain; charset=UTF-8
Cc: dev@dpdk.org, Rizwan Ansari <ransari@mvista.com>
Subject: Re: [dpdk-dev] [PATCH 6/6] virtio: arm/arm64: memory mapped IO
support in pmd driver
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Dec 2015 19:19:09 -0000
On Thu, Dec 10, 2015 at 12:34 AM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Thu, 10 Dec 2015 00:29:30 +0530
> Santosh Shukla <sshukla@mvista.com> wrote:
>
>> On Tue, Dec 8, 2015 at 6:23 PM, Santosh Shukla <sshukla@mvista.com> wrote:
>> >
>> >
>> > On Mon, Dec 7, 2015 at 10:38 PM, Stephen Hemminger
>> > <stephen@networkplumber.org> wrote:
>> >>
>> >> On Fri, 4 Dec 2015 23:05:19 +0530
>> >> Santosh Shukla <sshukla@mvista.com> wrote:
>> >>
>> >> >
>> >> > +#ifdef RTE_EXEC_ENV_LINUXAPP
>> >> > +/* start address of first pci_iobar slot (user-space virtual-addres) */
>> >> > +void *ioport_map;
>> >> > +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64)
>> >> > +
>> >> > +#include <sys/mman.h>
>> >> > +#define DEV_NAME "/dev/igb_ioport"
>> >> > +
>> >> > +/* Keeping pci_ioport_size = 4k.
>> >> > + * So maximum mmaped pci_iobar supported =
>> >> > + * (ioport_size/pci_dev->mem_resource[0].len)
>> >> > + *
>> >> > + * note: kernel could allow maximum 32 virtio-net-pci interface, that
>> >> > mean
>> >> > + * maximum 32 PCI_IOBAR(s) where each PCI_IOBAR_LEN=0x20, so
>> >> > virtio_map_ioport()
>> >> > + * func by theory gonna support 4k/0x20 ==> 128 PCI_IOBAR(s), more than
>> >> > + * max-virtio-net-pci interface.
>> >> > + */
>> >> > +#define PAGE_SIZE 4096
>> >> > +#define PCI_IOPORT_SIZE PAGE_SIZE
>> >> > +#define PCI_IOPORT_MAX 128 /* 4k / 0x20 */
>> >> > +
>> >> > +int ioport_map_cnt;
>> >> > +#endif /* ARM, ARM64 */
>> >> > +#endif /* RTE_EXEC_ENV_LINUXAPP */
>> >>
>> >> These variables should be static.
>> >>
>> >
>> > (Sorry for delayed follow, Was travelling..)
>> > Right,
>> >
>> >>
>> >> Also, it is should be possible to extract the I/O bar stuff in a generic
>> >> way through sysfs
>> >> and not depend on a character device. The long term goal for DPDK
>> >> acceptance is to
>> >> eliminate (or at least reduce to a minumum) any requirement for special
>> >> kernel drivers.
>> >
>> >
>> > I agree. Existing implementation does read pci_iobar for start address and
>> > size, But for non-x86 arch, we need someway to map pci_iobar and thats-why
>> > thought of adding device file for a purpose, as archs like arm lack iopl()
>> > privileged io syscall support, However iopl() too quite native driver design
>> > assumption.
>> >
>> > I have few idea in my mind such that - Right now I am updating ioport_mapped
>> > addr {kernel-virtual-addr-io-memory} to /sys/bus/pci/pci_bus_xxxx/xxx/map
>> > field, instead of mapping their, I'll try to map to uio's pci interface and
>> > then use existing pci_map_resource() api to mmap kernel-virtual-io-address
>> > to user-space-virtual-ioaddr. We'll come back on this.
>> >
>>
>>
>> Spent sometime digging dpdk's uio/pci source code, Intent was to map
>> pci ioport region via uio-way. In order to achieve that I tried to
>> hack the virtio-net-pci pmd driver. Right now in virtio-net-pci case,
>> It creates two sysfs entry for pci bars: resource0 /1.
>>
>> Resource0; is ioport region
>> Resource1; is iomem region.
>>
>> By appending a RTE_PCI_DRV_NEED_MAPPING flag to drv_flag and passing
>> hw->io_base = resource1 type pci.mem_resource[slot].addr; where slot
>> =1. Resource1 is IORESOURCE_MEM type so uio/pci driver able to mmap.
>> That way I could get the valid user-space virtual address. However
>> this hack did not worked for me because at qemu side: virtio-pxe.rom
>> has virtio_headers located at ioport pci region and guest driver
>> writing at iomem region, that's why driver init failed. Note that
>> default driver doesn't use resource1 memory.
>>
>> This made me think that either I had to add dependent code in kernel
>> or something similar proposed in this patch.
>> It is because:
>> - uio driver and dependent user-space pci api's in dpdk mmaps
>> IORESOURCE_MEM types address only {refer igbuio_setup_bars() and in
>> particular function pci_parse_sysfs_resource()}.
>> - That mmap in userspace backed by arch specific api
>> pci_mmap_page_range() in kernel.
>> - pci_mmap_page_range() does not support mmaping to IO_RESOURCE_IO type memory.
>> - Having said that, we need routine or a way to to map pci_iobar
>> region from kernel virtual-address to user-space virtual address.
>
> There a couple of gotcha's with this. It turns out the iomem region
> is not mappable on some platforms. I think GCE was one of them.
afaik, In linux kernel if arch supports pci_mmap_page_range then iomem
region should map, right? I am confused by reading your reply, also I
am not aware of GCE? which platform is GCE, please suggest.