[dpdk-dev] 答复: 答复: [PATCH v2] bus/pci: resolve multiple NICs address conflicts

["Wangyu (Eric)" <seven.wangyu@huawei.com>]

Sorry, I didn't explain it clearly, and I will explain this problem step by step.

Precondition, we have a 64K page size system and two 82599 NICs. The memory required for each NIC is as follows:
Map0 : size = 0x0000000000400000
Map1 : size = 0x0000000000004000

1. Primary process start, process mmap() first NIC map0, mmap()'s input address is 0x8202000000, and output address is 0x8202000000, size is 0x0000000000400000, next_addr is 0x8202400000, mmap() executed correctly.

2. Primary mmap() first NIC map1, mmap()'s input address is 0x8202400000, and output address is 0x8202400000, size is 0x0000000000004000, next_addr is 0x8202404000, now mmap() applied from 0x8202400000 to 0x8202410000 actually(because page size is 64K), and next_addr is 0x8202404000, but mmap() executed correctly.

3. Primary mmap() second NIC map0, mmap()'s input address is 0x8202404000, but it's conflict, so output address is 0xffffbcdc0000(system assigned), size is 0x0000000000400000, next_addr is 0xffffbd1c0000, now the address is abnormal, and mmap() executed correctly.

4. Primary mmap() second NIC map1, mmap()'s input address is 0xffffbd1c0000, and output address is 0xffffbcdb0000 (system assigned), size is 0x0000000000004000, now the address is abnormal, and mmap() executed correctly.

5. Secondary process start, process mmap() first NIC map0, it's normal.

6. Secondary process mmap() first NIC map1, it's normal.

7. Secondary process mmap() second NIC map0, mmap()'s input address is 0xffffbcdc0000, but it's conflict on secondary process, so we get another address, but secondary will check if the input address is equal with output address, it's not equal, so secondary will exit with " Cannot mmap device resource file %s to address: %p ".


Now I use (next_addr = RTE_PTR_ALIGN(current.addr + current.len, pagesize)) to solve the problem, and it worked. If it is right, I will submit a patch later.

By the way, I made a mistake, the problem won't happen on VFIO, because VFIO don't apply for 16K memory, only apply for 4M size(map0).But I think VFIO also needs to be modified.


-----邮件原件-----
发件人: Burakov, Anatoly [mailto:anatoly.burakov@intel.com] 
发送时间: 2019年11月7日 20:25
收件人: Wangyu (Eric) <seven.wangyu@huawei.com>; David Marchand <david.marchand@redhat.com>
抄送: dev@dpdk.org; ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q) <humin29@huawei.com>; Liyuan (Larry) <Larry.T@huawei.com>; dengxiaofeng <dengxiaofeng@huawei.com>
主题: Re: 答复: [dpdk-dev] [PATCH v2] bus/pci: resolve multiple NICs address conflicts

On 07-Nov-19 5:44 AM, Wangyu (Eric) wrote:
> Hi, Anatoly
> 
> Thank you for advices. This problem will happen in both VFIO and UIO, I will modify both according to your advices and test them.
> 
> I did some tests with mmap() on my system, when I provided address not page-aligned, mmap() could return page-aligned address too, but the code will return fault because mmap() return address was not equal with address I provided(problem occurs in pci_uio_map_secondary()).
> 

I still don't understand how do you get addresses aligned on a 16K boundary with 64K page size.

The mapping process is as follows:

0) start with max_va_end, or with previous addres + previous len
1) reserve virtual area with mmap() (accepts any return address)
2) map the BAR with MAP_FIXED (checks return address, but should work because we already have that area reserved)

The error you're referring to would've happened at step 2 (MAP_FIXED with unaligned addresses will cause the mmap() to fail), but at that point we already have a valid virtual area for the bar. If you get a 16K-aligned page address for the BAR, you get it on step 1, not step 2.

So, if, by your own admission, your mmap() implementation does return a 64K-aligned address... What exactly is the issue then? How does your BAR end up with an invalid address?

--
Thanks,
Anatoly