From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 70CFDA04DF;
	Wed, 21 Oct 2020 19:24:32 +0200 (CEST)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id 1F51972E2;
	Wed, 21 Oct 2020 19:24:31 +0200 (CEST)
Received: from mga03.intel.com (mga03.intel.com [134.134.136.65])
 by dpdk.org (Postfix) with ESMTP id 318E872E0
 for <dev@dpdk.org>; Wed, 21 Oct 2020 19:24:29 +0200 (CEST)
IronPort-SDR: iyJxRdI8JmCM/ZHGOys9a6MfyTjm2p623HDfkIY8OVFDG+ZXO04lOh81DGC9Z0ed9bGlJDHw3h
 fvs3DEA/MA9A==
X-IronPort-AV: E=McAfee;i="6000,8403,9780"; a="167497478"
X-IronPort-AV: E=Sophos;i="5.77,401,1596524400"; d="scan'208";a="167497478"
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from orsmga005.jf.intel.com ([10.7.209.41])
 by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
 21 Oct 2020 10:24:27 -0700
IronPort-SDR: cMGuOIT8a5U0gSDwTFCbUGmp4To5d5WCBHhaq9PINDCGZCrGWxUdo9sn2nN9rNKW1U3eqJQ2zv
 iOfd77RHiLAQ==
X-IronPort-AV: E=Sophos;i="5.77,401,1596524400"; d="scan'208";a="533615542"
Received: from fyigit-mobl1.ger.corp.intel.com (HELO [10.251.86.101])
 ([10.251.86.101])
 by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
 21 Oct 2020 10:24:25 -0700
To: =?UTF-8?B?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?=
 <huawei.xhw@alibaba-inc.com>, Maxime Coquelin <maxime.coquelin@redhat.com>
Cc: dev@dpdk.org, anatoly.burakov@intel.com, david.marchand@redhat.com,
 grive@u256.net, zhihong.wang@intel.com, chenbo.xia@intel.com
References: <68ecd941-9c56-4de7-fae2-2ad15bdfd81a@alibaba-inc.com>
 <1602578499-13975-1-git-send-email-huawei.xhw@alibaba-inc.com>
 <1602578499-13975-2-git-send-email-huawei.xhw@alibaba-inc.com>
 <2298e6a6-2ca8-0f68-6742-1b7d5dde97a7@intel.com>
 <5d26db1a-6970-6e3a-c2e5-8dc540922028@alibaba-inc.com>
From: Ferruh Yigit <ferruh.yigit@intel.com>
Message-ID: <e23712f1-7dbb-ce3c-2694-5648eb8369ae@intel.com>
Date: Wed, 21 Oct 2020 18:24:21 +0100
MIME-Version: 1.0
In-Reply-To: <5d26db1a-6970-6e3a-c2e5-8dc540922028@alibaba-inc.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Subject: Re: [dpdk-dev] [PATCH v4] pci: support both PIO and MMIO BAR for
 legacy virtio on x86
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

On 10/21/2020 1:32 PM, 谢华伟(此时此刻) wrote:
> 
> On 2020/10/21 19:49, Ferruh Yigit wrote:
>> On 10/13/2020 9:41 AM, 谢华伟(此时此刻) wrote:
>>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>>
>>> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
>>> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
>>>
>>> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
>>> different type of BAR in the similar way.
>>>
>>> In previous implementation, with igb_uio we get PIO address from igb_uio
>>> sysfs entry; with uio_pci_generic, we get PIO address from
>>> /proc/ioports.
>>> For PIO/MMIO RW, there is different path for different drivers and arch.
>>> For VFIO, PIO/MMIO RW is through syscall, which has big performance
>>> issue.
>>> On X86, it assumes only PIO is supported.
>>>
>>> All of the above is too much twisted.
>>> This patch unifies the way to get both PIO and MMIO address for different driver
>>> and arch, all from standard resource attr under pci sysfs.
>>>
>>
>> As mentined above this patch does multiple things.
>>
>> The main target is, as far as I understand, you have a legacy virtio device 
>> which supports "memory-mapped I/O" and "port-mapped I/O", but virtio logic 
>> forces legacy devices to use the PIO but you want to be able to use the MMIO 
>> with this device.
> yes.
>>
>> The solution below is adding MMIO support in the PIO funciton, and distinguish 
>> MMIO or PIO based on their address check.
> Yes, kernel does this in the similar way.
>>
>>
>> Instead of this, can't this be resolved in the virtio side, like if the legacy 
>> device supports MMIO (detect this somehow) use the MMIO istead of hacking PIO 
>> mapping to support MMIO?
> 
> Get your concern.
> 
> 1>
> 
> If we move, I think we should move all those PCI codes into virtio side, not 
> just the mmio part.
> 
> Without my patch, those PCI codes are virtio-pci device specific, not generic.
> 
> With this patch, those pci ioport map/rw code could also be used for other 
> devices if they support both PIO and MMIO.
> 

I was not suggesting moving any code into virtio, but within 'vtpci_init()' what 
happens when "hw->modern = 1;" is set?
And if this is set for your device, will it work without change?

> Every option is ok. Hope i make myself clear.
> 
> 2>  I don't think this is hacking. for rte_pci_ioport_map/read/write, if ioport 
> could be both PIO and MMIO, then everything is reasonable.
> 
> Take how kernel does port map for example:
> 
>      vp_dev->ioaddr = pci_iomap(pci_dev, 0, 0);
> 
> Here io doesn't mean PIO only. It could also be MMIO. Kernel then uses 
> ioread/write to access PIO/MMIO port.
> 
> Actually we are pretty much the same in the interface.
> 
> I think this patch extends rather then hacks the ioport interface to support MMIO.
> 
>>
>> I have other concerns, specially mergin VFIO mapping too, but lets clarify 
>> above first.
> 
> vfio doesn't affect other driver but only virtio.
> 

Why it doesn't affect other drivers, can't there be other driver using PIO?

> igb_uio, uio_pci_generic and vfio-pci all uses the same way to map/rw ioport.
> 

For vfio, code changes 'pci_vfio_ioport_read()' to the direct address read, 
first I don't know if this is always safe, and my question why there is a 
syscall introduced at first place if you can read from address directly?

Is your device works as expected when vfio-pci kernel module used? Since it is 
not suffering from PIO limitation, right?


And I wonder if the patch can be done as three patches to simply it, as:
1) Combine 'RTE_PCI_KDRV_IGB_UIO' & 'RTE_PCI_KDRV_UIO_GENERIC' (remove 
pci_ioport_map)
2) Update 'pci_uio_ioport_map()' to add memory map support (and update 
read/write functions according)
3) Combine vfio & uio

>>
>> Thanks,
>> ferruh
>>
>>
>>
>>> We distinguish PIO and MMIO by their address like how kernel does. It is ugly 
>>> but works.
>>>
>>> Signed-off-by: huawei.xhw <huawei.xhw@alibaba-inc.com>

<...>