DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] 答复:  [PATCH v3] bus/pci: resolve multiple NICs address conflicts
       [not found] <D3165BB52137B64493C4E18DC69F10D62C8F4B4F@dggemm521-mbx.china.huawei.com>
@ 2019-11-13  3:11 ` Wangyu (Eric)
  0 siblings, 0 replies; 4+ messages in thread
From: Wangyu (Eric) @ 2019-11-13  3:11 UTC (permalink / raw)
  To: Gavin Hu (Arm Technology China), dev
  Cc: ferruh.yigit, Linuxarm, humin (Q), Liyuan (Larry), nd, dengxiaofeng


Thank you for your advice, it's helpful.

I will follow your advice to write a new one.

-----邮件原件-----
发件人: Gavin Hu (Arm Technology China) [mailto:Gavin.Hu@arm.com]
发送时间: 2019年11月12日 15:02
收件人: Wangyu (Eric) <seven.wangyu@huawei.com>; dev@dpdk.org
抄送: ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q) <humin29@huawei.com>; dengxiaofeng <dengxiaofeng@huawei.com>; Liyuan (Larry) <Larry.T@huawei.com>; nd <nd@arm.com>
主题: RE: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts

Hi Eric,
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Wangyu (Eric)
> Sent: Tuesday, November 12, 2019 10:23 AM
> To: dev@dpdk.org
> Cc: ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q) 
> <humin29@huawei.com>; dengxiaofeng <dengxiaofeng@huawei.com>; Liyuan
> (Larry) <Larry.T@huawei.com>
> Subject: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address 
> conflicts
Change the headline to "bus/pci: fix mapping issue when page size is greater than bar"?
> NIC address conflicts on 64K pagesize when using multiple NICs,
> 
> as system will mmap 64K pagesize for NIC,
> 
> but dev->mem_resource[i].len is 16K.
I would like the problem is described as follows and followed by analysis and fix.
When integrating NICs that request memory spaces less than a page size, the problem hits as follows:
" Cannot mmap device resource file %s to address: %p "

The root cause is the PCI memory space is mapped at a page size granule, without considering the page size to calculate the next address for future mapping is wrong.

The fix is to calculate the next address for future mappings using the real mapping size.
/Gavin
> 
> Signed-off-by: beard-627 <dengxiaofeng@huawei.com>
> Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
> Acked-by: Wei Hu <xavier.huwei@huawei.com>
> Acked-by: Min Hu <humin29@huawei.com>
> ---
>  drivers/bus/pci/linux/pci_uio.c  | 2 ++ 
> drivers/bus/pci/linux/pci_vfio.c | 3 +++
>  2 files changed, 5 insertions(+)
> 
> diff --git a/drivers/bus/pci/linux/pci_uio.c 
> b/drivers/bus/pci/linux/pci_uio.c index 6dca05a..097dc19 100644
> --- a/drivers/bus/pci/linux/pci_uio.c
> +++ b/drivers/bus/pci/linux/pci_uio.c
> @@ -351,6 +351,8 @@
>  	pci_map_addr = RTE_PTR_ADD(mapaddr,
>  			(size_t)dev->mem_resource[res_idx].len);
> 
> +	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> sysconf(_SC_PAGE_SIZE));
Here "pci_map_addr" is, in effect, exactly the next_addr we talked about in V2, for use of next mapping,  I vote for this fix.
/Gavin
>  	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
>  	maps[map_idx].size = dev->mem_resource[res_idx].len;
>  	maps[map_idx].addr = mapaddr;
> diff --git a/drivers/bus/pci/linux/pci_vfio.c
> b/drivers/bus/pci/linux/pci_vfio.c
> index b8faa23..64cd84a 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -750,6 +750,9 @@
>  		bar_addr = pci_map_addr;
>  		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
> 
> +		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> +					sysconf(_SC_PAGE_SIZE));
> +
>  		maps[i].addr = bar_addr;
>  		maps[i].offset = reg->offset;
>  		maps[i].size = reg->size;
> --
> 1.8.3.1


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts
  2019-11-12  2:22 ` [dpdk-dev] " Wangyu (Eric)
  2019-11-12  7:01   ` Gavin Hu (Arm Technology China)
@ 2019-11-12 12:29   ` Burakov, Anatoly
  1 sibling, 0 replies; 4+ messages in thread
From: Burakov, Anatoly @ 2019-11-12 12:29 UTC (permalink / raw)
  To: Wangyu (Eric), dev
  Cc: ferruh.yigit, Linuxarm, humin (Q), dengxiaofeng, Liyuan (Larry)

On 12-Nov-19 2:22 AM, Wangyu (Eric) wrote:
> NIC address conflicts on 64K pagesize when using multiple NICs,
> 
> as system will mmap 64K pagesize for NIC,
> 
> but dev->mem_resource[i].len is 16K.

Suggested rewording:

===
bus/pci: align next mapping address on page boundary

Currently, the next address picked by PCI mapping infrastructure may be 
page-unaligned due to BAR length being smaller than page size. This 
leads to a situation where the requested map address is invalid, 
resulting in mmap() call returning an arbitrary address, which will 
later interfere with device BAR mapping in secondary processes.

Fix it by always aligning the next requested address on page boundary.
===

Also, probably missing Fixes: tag and Cc: stable.

> 
> Signed-off-by: beard-627 <dengxiaofeng@huawei.com>

That's an unconventional name :)

> Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
> Acked-by: Wei Hu <xavier.huwei@huawei.com>
> Acked-by: Min Hu <humin29@huawei.com>
> ---
>   drivers/bus/pci/linux/pci_uio.c  | 2 ++  drivers/bus/pci/linux/pci_vfio.c | 3 +++
>   2 files changed, 5 insertions(+)
> 
> diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c index 6dca05a..097dc19 100644
> --- a/drivers/bus/pci/linux/pci_uio.c
> +++ b/drivers/bus/pci/linux/pci_uio.c
> @@ -351,6 +351,8 @@
>   	pci_map_addr = RTE_PTR_ADD(mapaddr,
>   			(size_t)dev->mem_resource[res_idx].len);
>   
> +	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr, sysconf(_SC_PAGE_SIZE));
> +
>   	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
>   	maps[map_idx].size = dev->mem_resource[res_idx].len;
>   	maps[map_idx].addr = mapaddr;
> diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> index b8faa23..64cd84a 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -750,6 +750,9 @@
>   		bar_addr = pci_map_addr;
>   		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
>   
> +		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> +					sysconf(_SC_PAGE_SIZE));
> +
>   		maps[i].addr = bar_addr;
>   		maps[i].offset = reg->offset;
>   		maps[i].size = reg->size;
> --
> 1.8.3.1
> 
> 

As for contents of this patch (as long as the above is addressed):

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts
  2019-11-12  2:22 ` [dpdk-dev] " Wangyu (Eric)
@ 2019-11-12  7:01   ` Gavin Hu (Arm Technology China)
  2019-11-12 12:29   ` Burakov, Anatoly
  1 sibling, 0 replies; 4+ messages in thread
From: Gavin Hu (Arm Technology China) @ 2019-11-12  7:01 UTC (permalink / raw)
  To: Wangyu (Eric), dev
  Cc: ferruh.yigit, Linuxarm, humin (Q), dengxiaofeng, Liyuan (Larry), nd

Hi Eric,
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Wangyu (Eric)
> Sent: Tuesday, November 12, 2019 10:23 AM
> To: dev@dpdk.org
> Cc: ferruh.yigit@intel.com; Linuxarm <linuxarm@huawei.com>; humin (Q)
> <humin29@huawei.com>; dengxiaofeng <dengxiaofeng@huawei.com>;
> Liyuan (Larry) <Larry.T@huawei.com>
> Subject: [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts
Change the headline to "bus/pci: fix mapping issue when page size is greater than bar"?
> NIC address conflicts on 64K pagesize when using multiple NICs,
> 
> as system will mmap 64K pagesize for NIC,
> 
> but dev->mem_resource[i].len is 16K.
I would like the problem is described as follows and followed by analysis and fix.
When integrating NICs that request memory spaces less than a page size, the problem hits as follows:
" Cannot mmap device resource file %s to address: %p "

The root cause is the PCI memory space is mapped at a page size granule, without considering the page size to calculate the next address for future mapping is wrong.

The fix is to calculate the next address for future mappings using the real mapping size.
/Gavin
> 
> Signed-off-by: beard-627 <dengxiaofeng@huawei.com>
> Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
> Acked-by: Wei Hu <xavier.huwei@huawei.com>
> Acked-by: Min Hu <humin29@huawei.com>
> ---
>  drivers/bus/pci/linux/pci_uio.c  | 2 ++  drivers/bus/pci/linux/pci_vfio.c | 3 +++
>  2 files changed, 5 insertions(+)
> 
> diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c
> index 6dca05a..097dc19 100644
> --- a/drivers/bus/pci/linux/pci_uio.c
> +++ b/drivers/bus/pci/linux/pci_uio.c
> @@ -351,6 +351,8 @@
>  	pci_map_addr = RTE_PTR_ADD(mapaddr,
>  			(size_t)dev->mem_resource[res_idx].len);
> 
> +	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> sysconf(_SC_PAGE_SIZE));
Here "pci_map_addr" is, in effect, exactly the next_addr we talked about in V2, for use of next mapping,  I vote for this fix.
/Gavin
>  	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
>  	maps[map_idx].size = dev->mem_resource[res_idx].len;
>  	maps[map_idx].addr = mapaddr;
> diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> index b8faa23..64cd84a 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -750,6 +750,9 @@
>  		bar_addr = pci_map_addr;
>  		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
> 
> +		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
> +					sysconf(_SC_PAGE_SIZE));
> +
>  		maps[i].addr = bar_addr;
>  		maps[i].offset = reg->offset;
>  		maps[i].size = reg->size;
> --
> 1.8.3.1


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [dpdk-dev] [PATCH v3] bus/pci: resolve multiple NICs address conflicts
       [not found] <1573521821-21173-1-git-send-email-dengxiaofeng@huawei.com>
@ 2019-11-12  2:22 ` Wangyu (Eric)
  2019-11-12  7:01   ` Gavin Hu (Arm Technology China)
  2019-11-12 12:29   ` Burakov, Anatoly
  0 siblings, 2 replies; 4+ messages in thread
From: Wangyu (Eric) @ 2019-11-12  2:22 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Linuxarm, humin (Q), dengxiaofeng, Liyuan (Larry)

NIC address conflicts on 64K pagesize when using multiple NICs,

as system will mmap 64K pagesize for NIC,

but dev->mem_resource[i].len is 16K.

Signed-off-by: beard-627 <dengxiaofeng@huawei.com>
Signed-off-by: Wangyu (Eric) <seven.wangyu@huawei.com>
Acked-by: Wei Hu <xavier.huwei@huawei.com>
Acked-by: Min Hu <humin29@huawei.com>
---
 drivers/bus/pci/linux/pci_uio.c  | 2 ++  drivers/bus/pci/linux/pci_vfio.c | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c index 6dca05a..097dc19 100644
--- a/drivers/bus/pci/linux/pci_uio.c
+++ b/drivers/bus/pci/linux/pci_uio.c
@@ -351,6 +351,8 @@
 	pci_map_addr = RTE_PTR_ADD(mapaddr,
 			(size_t)dev->mem_resource[res_idx].len);
 
+	pci_map_addr = RTE_PTR_ALIGN(pci_map_addr, sysconf(_SC_PAGE_SIZE));
+
 	maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr;
 	maps[map_idx].size = dev->mem_resource[res_idx].len;
 	maps[map_idx].addr = mapaddr;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index b8faa23..64cd84a 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -750,6 +750,9 @@
 		bar_addr = pci_map_addr;
 		pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg->size);
 
+		pci_map_addr = RTE_PTR_ALIGN(pci_map_addr,
+					sysconf(_SC_PAGE_SIZE));
+
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg->offset;
 		maps[i].size = reg->size;
--
1.8.3.1


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2019-11-13  3:12 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <D3165BB52137B64493C4E18DC69F10D62C8F4B4F@dggemm521-mbx.china.huawei.com>
2019-11-13  3:11 ` [dpdk-dev] 答复: [PATCH v3] bus/pci: resolve multiple NICs address conflicts Wangyu (Eric)
     [not found] <1573521821-21173-1-git-send-email-dengxiaofeng@huawei.com>
2019-11-12  2:22 ` [dpdk-dev] " Wangyu (Eric)
2019-11-12  7:01   ` Gavin Hu (Arm Technology China)
2019-11-12 12:29   ` Burakov, Anatoly

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).