[dpdk-dev] 答复: [PATCH v3] bus/pci: resolve multiple NICs address conflicts

[dpdk-dev] 答复: [PATCH v3] bus/pci: resolve multiple NICs address conflicts

[Wangyu (Eric)]

Thank you for your advice, it's helpful.

I will follow your advice to write a new one.

-----邮件原件-----
发件人: Gavin Hu (Arm Technology China) [mailto:Gavin.Hu@arm.com]
发送时间: 2019年11月12日 15:02
收件人: Wangyu (Eric) <seven.wangyu@huawei.com>; dev@dpdk.org
抄送: ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q) <humin29@huawei.com>; dengxiaofeng <dengxiaofeng@huawei.com>; Liyuan (Larry) <Larry.T@huawei.com>; nd <nd@arm.com>
主题: RE: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts

Hi Eric,
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Wangyu (Eric)
> Sent: Tuesday, November 12, 2019 10:23 AM
> To: dev@dpdk.org
> Cc: ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q) 
> <humin29@huawei.com>; dengxiaofeng <dengxiaofeng@huawei.com>; Liyuan
> (Larry) <Larry.T@huawei.com>
> Subject: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address 
> conflicts
Change the headline to "bus/pci: fix mapping issue when page size is greater than bar"?
> NIC address conflicts on 64K pagesize when using multiple NICs,
> 
> as system will mmap 64K pagesize for NIC,
> 
> but dev->mem_resource[i].len is 16K.
I would like the problem is described as follows and followed by analysis and fix.
When integrating NICs that request memory spaces less than a page size, the problem hits as follows:
" Cannot mmap device resource file %s to address: %p "

The root cause is the PCI memory space is mapped at a page size granule, without considering the page size to calculate the next address for future mapping is wrong.

The fix is to calculate the next address for future mappings using the real mapping size.
/Gavin
> 
> Signed-off-by: beard-627 <dengxiaofeng@huawei.com>
> Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
> Acked-by: Wei Hu <xavier.huwei@huawei.com>
> Acked-by: Min Hu <humin29@huawei.com>
> ---
>  drivers/bus/pci/linux/pci_uio.c  | 2 ++ 
> drivers/bus/pci/linux/pci_vfio.c | 3 +++
>  2 files changed, 5 insertions(+)
> 
> diff --git a/drivers/bus/pci/linux/pci_uio.c 
> b/drivers/bus/pci/linux/pci_uio.c index 6dca05a..097dc19 100644
> --- a/drivers/bus/pci/linux/pci_uio.c
> +++ b/drivers/bus/pci/linux/pci_uio.c
> @@ -351,6 +351,8 @@
>  	pci_map_addr = RTE_PTR_ADD(mapaddr,
>  			(size_t)dev->mem_resource[res_idx].len);
> 
> +	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> sysconf(_SC_PAGE_SIZE));
Here "pci_map_addr" is, in effect, exactly the next_addr we talked about in V2, for use of next mapping,  I vote for this fix.
/Gavin
>  	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
>  	maps[map_idx].size = dev->mem_resource[res_idx].len;
>  	maps[map_idx].addr = mapaddr;
> diff --git a/drivers/bus/pci/linux/pci_vfio.c
> b/drivers/bus/pci/linux/pci_vfio.c
> index b8faa23..64cd84a 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -750,6 +750,9 @@
>  		bar_addr = pci_map_addr;
>  		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
> 
> +		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> +					sysconf(_SC_PAGE_SIZE));
> +
>  		maps[i].addr = bar_addr;
>  		maps[i].offset = reg->offset;
>  		maps[i].size = reg->size;
> --
> 1.8.3.1

[dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts

[Wangyu (Eric)]

NIC address conflicts on 64K pagesize when using multiple NICs,

as system will mmap 64K pagesize for NIC,

but dev->mem_resource[i].len is 16K.

Signed-off-by: beard-627 <dengxiaofeng@huawei.com>
Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
Acked-by: Wei Hu <xavier.huwei@huawei.com>
Acked-by: Min Hu <humin29@huawei.com>
---
 drivers/bus/pci/linux/pci_uio.c  | 2 ++  drivers/bus/pci/linux/pci_vfio.c | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c index 6dca05a..097dc19 100644
--- a/drivers/bus/pci/linux/pci_uio.c
+++ b/drivers/bus/pci/linux/pci_uio.c
@@ -351,6 +351,8 @@
 	pci_map_addr = RTE_PTR_ADD(mapaddr,
 			(size_t)dev->mem_resource[res_idx].len);
 
+	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr, sysconf(_SC_PAGE_SIZE));
+
 	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
 	maps[map_idx].size = dev->mem_resource[res_idx].len;
 	maps[map_idx].addr = mapaddr;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index b8faa23..64cd84a 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -750,6 +750,9 @@
 		bar_addr = pci_map_addr;
 		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
 
+		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
+					sysconf(_SC_PAGE_SIZE));
+
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg->offset;
 		maps[i].size = reg->size;
--
1.8.3.1

Re: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts

[Gavin Hu (Arm Technology China)]

Hi Eric,
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Wangyu (Eric)
> Sent: Tuesday, November 12, 2019 10:23 AM
> To: dev@dpdk.org
> Cc: ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q)
> <humin29@huawei.com>; dengxiaofeng <dengxiaofeng@huawei.com>;
> Liyuan (Larry) <Larry.T@huawei.com>
> Subject: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts
Change the headline to "bus/pci: fix mapping issue when page size is greater than bar"?
> NIC address conflicts on 64K pagesize when using multiple NICs,
> 
> as system will mmap 64K pagesize for NIC,
> 
> but dev->mem_resource[i].len is 16K.
I would like the problem is described as follows and followed by analysis and fix.
When integrating NICs that request memory spaces less than a page size, the problem hits as follows:
" Cannot mmap device resource file %s to address: %p "

The root cause is the PCI memory space is mapped at a page size granule, without considering the page size to calculate the next address for future mapping is wrong.

The fix is to calculate the next address for future mappings using the real mapping size.
/Gavin
> 
> Signed-off-by: beard-627 <dengxiaofeng@huawei.com>
> Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
> Acked-by: Wei Hu <xavier.huwei@huawei.com>
> Acked-by: Min Hu <humin29@huawei.com>
> ---
>  drivers/bus/pci/linux/pci_uio.c  | 2 ++  drivers/bus/pci/linux/pci_vfio.c | 3 +++
>  2 files changed, 5 insertions(+)
> 
> diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c
> index 6dca05a..097dc19 100644
> --- a/drivers/bus/pci/linux/pci_uio.c
> +++ b/drivers/bus/pci/linux/pci_uio.c
> @@ -351,6 +351,8 @@
>  	pci_map_addr = RTE_PTR_ADD(mapaddr,
>  			(size_t)dev->mem_resource[res_idx].len);
> 
> +	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> sysconf(_SC_PAGE_SIZE));
Here "pci_map_addr" is, in effect, exactly the next_addr we talked about in V2, for use of next mapping,  I vote for this fix.
/Gavin
>  	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
>  	maps[map_idx].size = dev->mem_resource[res_idx].len;
>  	maps[map_idx].addr = mapaddr;
> diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> index b8faa23..64cd84a 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -750,6 +750,9 @@
>  		bar_addr = pci_map_addr;
>  		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
> 
> +		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> +					sysconf(_SC_PAGE_SIZE));
> +
>  		maps[i].addr = bar_addr;
>  		maps[i].offset = reg->offset;
>  		maps[i].size = reg->size;
> --
> 1.8.3.1

Re: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts

[Burakov, Anatoly]

On 12-Nov-19 2:22 AM, Wangyu (Eric) wrote:
> NIC address conflicts on 64K pagesize when using multiple NICs,
> 
> as system will mmap 64K pagesize for NIC,
> 
> but dev->mem_resource[i].len is 16K.

Suggested rewording:

===
bus/pci: align next mapping address on page boundary

Currently, the next address picked by PCI mapping infrastructure may be 
page-unaligned due to BAR length being smaller than page size. This 
leads to a situation where the requested map address is invalid, 
resulting in mmap() call returning an arbitrary address, which will 
later interfere with device BAR mapping in secondary processes.

Fix it by always aligning the next requested address on page boundary.
===

Also, probably missing Fixes: tag and Cc: stable.

> 
> Signed-off-by: beard-627 <dengxiaofeng@huawei.com>

That's an unconventional name :)

> Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
> Acked-by: Wei Hu <xavier.huwei@huawei.com>
> Acked-by: Min Hu <humin29@huawei.com>
> ---
>   drivers/bus/pci/linux/pci_uio.c  | 2 ++  drivers/bus/pci/linux/pci_vfio.c | 3 +++
>   2 files changed, 5 insertions(+)
> 
> diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c index 6dca05a..097dc19 100644
> --- a/drivers/bus/pci/linux/pci_uio.c
> +++ b/drivers/bus/pci/linux/pci_uio.c
> @@ -351,6 +351,8 @@
>   	pci_map_addr = RTE_PTR_ADD(mapaddr,
>   			(size_t)dev->mem_resource[res_idx].len);
>   
> +	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr, sysconf(_SC_PAGE_SIZE));
> +
>   	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
>   	maps[map_idx].size = dev->mem_resource[res_idx].len;
>   	maps[map_idx].addr = mapaddr;
> diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> index b8faa23..64cd84a 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -750,6 +750,9 @@
>   		bar_addr = pci_map_addr;
>   		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
>   
> +		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> +					sysconf(_SC_PAGE_SIZE));
> +
>   		maps[i].addr = bar_addr;
>   		maps[i].offset = reg->offset;
>   		maps[i].size = reg->size;
> --
> 1.8.3.1
> 
> 

As for contents of this patch (as long as the above is addressed):

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly