[dpdk-dev] 答复: [PATCH v2] bus/pci: resolve multiple NICs address conflicts

["Wangyu (Eric)" <seven.wangyu@huawei.com>]

Hi, Anatoly

Thank you for advices. This problem will happen in both VFIO and UIO, I will modify both according to your advices and test them.

I did some tests with mmap() on my system, when I provided address not page-aligned, mmap() could return page-aligned address too, but the code will return fault because mmap() return address was not equal with address I provided(problem occurs in pci_uio_map_secondary()).


-----邮件原件-----
发件人: Burakov, Anatoly [mailto:anatoly.burakov@intel.com] 
发送时间: 2019年11月6日 19:15
收件人: David Marchand <david.marchand@redhat.com>; Wangyu (Eric) <seven.wangyu@huawei.com>
抄送: dengxiaofeng <dengxiaofeng@huawei.com>; dev@dpdk.org; ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q) <humin29@huawei.com>; Liyuan (Larry) <Larry.T@huawei.com>
主题: Re: [dpdk-dev] [PATCH v2] bus/pci: resolve multiple NICs address conflicts

On 06-Nov-19 7:37 AM, David Marchand wrote:
> On Wed, Nov 6, 2019 at 7:16 AM Wangyu (Eric) <seven.wangyu@huawei.com> wrote:
>>
>>
>> In 64K pagesize system, DPDK will read the size NIC need in uio/uio1/maps/map1/size,  when the size small than pagesize(e.g.,82599 is 16K), dev->mem_resource[i].len will be 16K, but the mmap function applies for at least 1 page size, which is 64K.
>> Then second NIC mmap, start address is first NIC address + 16K, which already used by first NIC.
> 
> Do you see this issue with vfio?
> 
> 
>> So if change the size to first NIC address + 64K, problem solved.
> 
> You are hacking a description of the device resources to workaround a problem.
> This patch is a no go for me.
> 
> Maybe there is something to do with the hint passed to mmap in uio case.
> Adding Anatoly to the loop.
> 
> 

I did a quick code inspection for VFIO and UIO. We do the same thing in both, so both code paths can be for all intents and purposes considered equivalent.

To reserve mappings for addresses, we start at some arbitrary address (find_max_va_end()), and start mapping from there. Then, we do an mmap() *and overwrite* whatever address we expected to get, and then the next address is (current.addr + current.len).

The mmap() is called without MAP_FIXED, so we get an address the kernel feels comfortable for us to get. Meaning, even if the initial address hint was not page-aligned, the return value from mmap() will be page-aligned. It seems to me that your platform/kernel does not do that, and allows mmap() to return page-unaligned addresses. I would strongly suggest checking the mmap() return address on your platform (in either UIO or VFIO - they both do it about the same way).

We could work around that by doing (next_addr = RTE_PTR_ALIGN(current.addr + current.len, pagesize)), but to me it seems like a bug in your kernel/mmap() implementation. This is an easy fix though, and i'm sure we can put in a workaround like i described.

--
Thanks,
Anatoly