Re: [dpdk-dev] [PATCH] pci: support both PIO and MMIO BAR for legacy virtio on x86

["谢华伟(此时此刻）" <huawei.xhw@alibaba-inc.com>]

On 2020/9/25 2:02, Ferruh Yigit wrote:
> On 9/15/2020 4:21 PM, 谢华伟(此时此刻） wrote:
>> Hi Ferruh:
>> Legacy virtio only supports PIO BAR resource. As we need to create 
>> lots of virtio devices and PIO resource on x86 is very limited, we 
>> expose MMIO BAR.
>> Kernel support both PIO and MMIO BAR for legacy virtio device. This 
>> patch deals with two cases in the similar way.
>>
>
> Thanks Huawei,
>
> I was about to ask the actual motivation for the patch, better to put 
> this explenation into the commit log.
>

OK.


> If legacy virtio device only supports PIO BAR, how exposing MMIO BAR 
> helps? Or if there is MMIO BAR available in the device, why not 
> mapping them instead of using PIO?
> Most probably I am missing something but can you please help to 
> understand.
>
DPDK virtio-pmd for legacy device only looks for PIO resource, which 
doesn't work if we expose MMIO bar.

virtio-pci kernel driver doesn't have this requirement. It gets IO 
address either from PIO port or mapped bar. It uses address range to 
distinguish PIO bar and MMIO bar.

That is what this patch fixes.


> Also below code unifies vfio & uio cases for PIO into same function. 
> Are you using igb_uio or both? I am not sure about unifiying the vfio 
> case, what do you think not touching that case in this patch?

All virtio PMD need is to get bar address from pci resouce attribute 
under sysfs like what kernel driver does. UIO/IGB_UIO/VFIO driver path 
for virtio bar mapping is too complicated.

Actually i think we shouldn't put those mapping functions into PCI 
layer, because they are only for virtio legacy device, not generic. I 
don't remember if it is me who put them there.

Besides, VFIO path uses vfio fd to read/write IO bar which has 
performance issue. Syscall is needed to notify virtio backend.


I see IGB_UIO is to be removed in this release.  UIO_PCI_GENERIC doesn't 
support msix. If we expose MMIO bar, we have to use VFIO driver before 
uio_pci_generic kernel driver is enhanced. So i have to fix vfio path.

>
> There are a few more comments inline, please check.
>
> Btw, the patch doesn't apply cleanly, can you please send a new 
> version on top of latest head of repo?
>
>
I will check if we recently applied some pci related patches.

>
>> On 2020/9/15 16:18, 谢华伟(此时此刻） wrote:
>>>  From d0138f24037d8df14cac04c2c24831e4b5d27b8c Mon Sep 17 00:00:00 2001
>>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>> Date: Mon, 14 Sep 2020 23:44:56 +0800
>>> Subject: [PATCH] pci:  support both PIO and MMIO BAR for legacy 
>>> virtio on x86
>>>
>>> In previous implementation, with igb_uio we get PIO address from 
>>> igb_uio
>>> sysfs entry; with uio_pci_generic, we get PIO address from
>>> /proc/ioports.
>>> For PIO/MMIO RW, there is different path for different drivers and 
>>> arch.
>>> For VFIO, PIO/MMIO RW is through syscall, which has big performance
>>> issue.
>>> On X86, it assumes only PIO is supported.
>>>
>>> This is too much twisted.
>>> This patch unifies the way to get both PIO and MMIO address for 
>>> different driver
>>> and arch, all from standard resource attr under pci sysfs.
>>>
>>> We distinguish PIO and MMIO by their address. It is ugly but works.
>>>
>>> Signed-off-by: huawei.xhw <huawei.xhw@alibaba-inc.com>
>
> <...>
>
>>> --- a/drivers/bus/pci/linux/pci_uio.c
>>> +++ b/drivers/bus/pci/linux/pci_uio.c
>>> @@ -372,52 +372,82 @@
>>>   pci_uio_ioport_map(struct rte_pci_device *dev, int bar,
>>>              struct rte_pci_ioport *p)
>>>   {
>
> There are two 'pci_uio_ioport_map()', one for x86 and other for rest. 
> This one is for x86, since read/write functions for multiple arch 
> combined together, should non-x86 version of this function updated or 
> removed completely?
>
non x86 version has weird behavior. It requires bar with  IO attribute 
but does the mmap. Besides, it adds mmap address with physical address.

Fortunately, all other devices on non x86 platform seems to use MMIO bar.
> <...>
>
>>> +    if (flags & IORESOURCE_IO) {
>>> +        iobar = 1;
>>> +        base = (unsigned long)phys_addr;
>>> +        RTE_LOG(INFO, EAL, "%s(): PIO BAR %08lx detected\n", 
>>> __func__, base);
>>> +    } else if (flags & IORESOURCE_MEM) {
>>> +        iobar = 0;
>>> +        base = (unsigned long)dev->mem_resource[bar].addr;
>
> Isn't the 'mem_resource[bar].addr' value set only when it is mapped? 
> For this case I guess it will be 0x0. If not who maps it first?

Yes, it is set only when it is MMIO bar which gets mapped. So when it is 
MMIO bar, we use the already maped address in else path.

>
>>> +        RTE_LOG(INFO, EAL, "%s(): MMIO BAR %08lx detected\n", 
>>> __func__, base);
>>> +    } else {
>>> +        RTE_LOG(ERR, EAL, "%s(): unknown BAR type\n", __func__);
>>> +        goto error;
>>> +    }
>>> +
>>> +    if (iobar && rte_eal_iopl_init() != 0) {
>>> +        RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions 
>>> for PCI device %s\n",
>>> +            __func__, dev->name);
>>> +        goto error;
>>>       }
>>> -    /* ensure we don't get anything funny here, read/write will 
>>> cast to
>>> -     * uin16_t */
>>> -    if (start > UINT16_MAX)
>>> -        return -1;
>>>       /* FIXME only for primary process ? */
>>>       if (dev->intr_handle.type == RTE_INTR_HANDLE_UNKNOWN) {
>>> +        uio_num = pci_get_uio_dev(dev, dirname, sizeof(dirname), 0);
>>> +        if (uio_num < 0)
>>> +            goto error;
>
> How this will work with vfio?

In original implementation, only uio fixes intr handle. I could add UIO 
driver check to keep the same behavior.