[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct rte_pci_addr *loc = &dev->addr;
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		requested_addr = (uint8_t *) internal_config.base_virtaddr 
+			+ internal_config.memory;
+	}
 
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

I have a slight problems with this patch.

The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.

Also, 

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct rte_pci_addr *loc = &dev->addr;
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		requested_addr = (uint8_t *) internal_config.base_virtaddr 
+			+ internal_config.memory;
+	}
 
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
--
1.9.1

[dpdk-dev] 答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 6452 bytes --]

I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov@intel.com>发送时间：2014年11月5日(星期三) 23:10收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org <dev@dpdk.org>主　题：RE: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the	base_virtaddr is configured.
I have a slight problems with this patch.

The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.

Also, 

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct rte_pci_addr *loc = &dev->addr;
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		requested_addr = (uint8_t *) internal_config.base_virtaddr 
+			+ internal_config.memory;
+	}
 
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
--
1.9.1
From jbshaw@ecsmtp.pdx.intel.com  Wed Nov  5 16:40:40 2014
Return-Path: <jbshaw@ecsmtp.pdx.intel.com>
Received: from mga14.intel.com (mga14.intel.com [192.55.52.115])
 by dpdk.org (Postfix) with ESMTP id 8D43A7F2D
 for <dev@dpdk.org>; Wed,  5 Nov 2014 16:40:39 +0100 (CET)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
 by fmsmga103.fm.intel.com with ESMTP; 05 Nov 2014 07:42:53 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,320,1413270000"; d="scan'208";a="617616041"
Received: from plxs0284.pdx.intel.com ([10.25.97.128])
 by fmsmga001.fm.intel.com with ESMTP; 05 Nov 2014 07:48:20 -0800
Received: from plxv1143.pdx.intel.com (plxv1143.pdx.intel.com [10.25.98.50])
 by plxs0284.pdx.intel.com with ESMTP id sA5FmKfG027858;
 Wed, 5 Nov 2014 07:48:20 -0800
Received: from plxv1143.pdx.intel.com (localhost [127.0.0.1])
 by plxv1143.pdx.intel.com with ESMTP id sA5FmKC5015874;
 Wed, 5 Nov 2014 07:48:20 -0800
Received: (from jbshaw@localhost)
	by plxv1143.pdx.intel.com with œ id sA5FmJB0015821;
	Wed, 5 Nov 2014 07:48:20 -0800
Date: Wed, 5 Nov 2014 07:48:19 -0800
From: Jeff Shaw <jeffrey.b.shaw@intel.com>
To: Gyumin <hkm73560@gmail.com>
Message-ID: <20141105154819.GA25599@plxv1143.pdx.intel.com>
References: <545972BF.9080100@gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <545972BF.9080100@gmail.com>
User-Agent: Mutt/1.4.2.3i
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Intel 82599 tx_conf setting
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 05 Nov 2014 15:40:40 -0000

On Wed, Nov 05, 2014 at 09:43:43AM +0900, Gyumin wrote:
> Hi
>
> I've read the Intel 82599 official manual and I found that optimal
> PTHRESH is the tx descriptor buffer size - N (N is CPU cache line
> divided by 16).

This is sometimes true, but not always.  I believe you are referring
to section "7.2.3.4.1 Transmit Descriptor Fetch and Write-back Settings"
in the datasheet.  You'll see the PTHRESH, HTHRESH, and WTHRESH parameters
should be tuned to for your workload. You should try a few combinations
of parameters (starting with the defaults) to see which is really optimal
for your application.

> 1. I guess the size of the tx descriptor buffer is 128. Isn't it right?
>    Where is the size of the tx descriptor buffer in the official manual?

The wording in the manual may be a bit confusing. You will see the manual
refers to the "on-chip descriptor buffer size".  This is where the NIC
stores descriptors which were fetched from the actual descriptor ring in
host memory.  Section "7.2.3.3 Transmit Descriptor Ring" states that the
size of the on-chip descriptor buffer size per queue is 40.

>
> 2. What it means that the TX_PTHRESH6 in the testpmd.c?
>    If the size of tx descriptor buffer is 128 then optimal thresholds
> to minimize latency are pthresh=4(cache line / 16), hthresh=0 and
> wthresh=0. Is there something I missed?

Since the on-chip descriptor buffer size is 40, it is clear that we have
chosen reasonable defaults since 40 minus 4 is 36. I recommend you test
a few different values to see how these parameters impact the performance
characteristics of your workload.

>
>
> Thanks.
You're welcome.

-Jeff

Re: [dpdk-dev] 答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Hi Liang

Yes it is a problem. Even if it was carefully selected by user, nothing stops the DPDK application from mapping something into where you’re trying to map your UIO devices. Plus, this changes the default behavior where a wrong base-virtaddr leads to a failure to initialize, rather than simply using a different address (remember that pci_map_resource fails if it cannot map the resource at the exact address you requested).

A very crude way of finding out where hugepages end would be to walk the hugepage memory (walk through memsegs and note the maximum start addr + length of that memseg).

Could you perhaps explain what is the problem that you’re trying to solve with this? I can’t think of a situation where the location of UIO maps would matter, to be honest.

Thanks,
Anatoly

From: XU Liang [mailto:liang.xu@cinfotech.cn]
Sent: Wednesday, November 5, 2014 3:49 PM
To: Burakov, Anatoly; dev@dpdk.org
Subject: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>

The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?
------------------------------------------------------------------
发件人：Burakov, Anatoly <anatoly.burakov@intel.com<mailto:anatoly.burakov@intel.com>>
发送时间：2014年11月5日(星期三) 23:10
收件人：徐亮 <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>，dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
主　题：RE: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

I have a slight problems with this patch.

The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.

Also,

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct rte_pci_addr *loc = &dev->addr;
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ static void * requested_addr = NULL;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ requested_addr = (uint8_t *) internal_config.base_virtaddr
+ + internal_config.memory;
+ }

dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}

if (fail) {
--
1.9.1

[dpdk-dev] 答复：答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

I have a multiple processes application. When start a secondary process, we got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".The secondary process link difference shared libraries, so the address 0x7ffff7fba000 is used.
------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov@intel.com>发送时间：2014年11月5日(星期三) 23:59收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org <dev@dpdk.org>主　题：RE: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the	base_virtaddr is configured.



font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0cm;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
*.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
size: 612.0pt 792.0pt;margin: 72.0pt 72.0pt 72.0pt 72.0pt;div.WordSection1 {page: WordSection1;}




Hi Liang
 
Yes it is a problem. Even if it was carefully selected by user, nothing stops the DPDK application from mapping something into where you’re trying to map your
 UIO devices. Plus, this changes the default behavior where a wrong base-virtaddr leads to a failure to initialize, rather than simply using a different address (remember that pci_map_resource fails if it cannot map the resource at the exact address you requested).
 
A very crude way of finding out where hugepages end would be to walk the hugepage memory (walk through memsegs and note the maximum start addr + length of that
 memseg).
 
Could you perhaps explain what is the problem that you’re trying to solve with this? I can’t think of a situation where the location of UIO maps would matter,
 to be honest.
 
Thanks,
Anatoly
 
From: XU Liang [mailto:liang.xu@cinfotech.cn]


Sent: Wednesday, November 5, 2014 3:49 PM

To: Burakov, Anatoly; dev@dpdk.org

Subject: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
 


I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>


 


The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?



------------------------------------------------------------------


发件人：Burakov, Anatoly <anatoly.burakov@intel.com>


发送时间：2014年11月5日(星期三)
 23:10


收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org
 <dev@dpdk.org>


主　题：RE:
 [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.


 

I have a slight problems with this patch.



The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will
 get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever
 reasons.



Also, 



Thanks,

Anatoly



-----Original Message-----

From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu

Sent: Wednesday, November 5, 2014 1:25 PM

To: dev@dpdk.org

Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.



---

lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-

1 file changed, 8 insertions(+), 1 deletion(-)



diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

index 7e62266..bc7ed3a 100644

--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)

struct rte_pci_addr *loc = &dev->addr;

struct mapped_pci_resource *uio_res;

struct pci_map *maps;

+ static void * requested_addr = NULL;

+ if (internal_config.base_virtaddr && NULL == requested_addr) {

+ requested_addr = (uint8_t *) internal_config.base_virtaddr 

+ + internal_config.memory;

+ }



dev->intr_handle.fd = -1;

dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)

if (maps[j].addr != NULL)

fail = 1;

else {

- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,

+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,

(size_t)maps[j].size);

if (mapaddr == NULL)

fail = 1;

+ else if (NULL != requested_addr)

+ requested_addr = (uint8_t *)mapaddr + maps[j].size;

}



if (fail) {

--

1.9.1



From anatoly.burakov@intel.com  Wed Nov  5 17:12:47 2014
Return-Path: <anatoly.burakov@intel.com>
Received: from mga11.intel.com (mga11.intel.com [192.55.52.93])
 by dpdk.org (Postfix) with ESMTP id 4B7BF7F29
 for <dev@dpdk.org>; Wed,  5 Nov 2014 17:12:46 +0100 (CET)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
 by fmsmga102.fm.intel.com with ESMTP; 05 Nov 2014 08:21:43 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,320,1413270000";
 d="scan'208,217";a="617637666"
Received: from irsmsx102.ger.corp.intel.com ([163.33.3.155])
 by fmsmga001.fm.intel.com with ESMTP; 05 Nov 2014 08:19:11 -0800
Received: from irsmsx155.ger.corp.intel.com (163.33.192.3) by
 IRSMSX102.ger.corp.intel.com (163.33.3.155) with Microsoft SMTP Server (TLS)
 id 14.3.195.1; Wed, 5 Nov 2014 16:19:08 +0000
Received: from irsmsx109.ger.corp.intel.com ([169.254.13.101]) by
 IRSMSX155.ger.corp.intel.com ([169.254.14.70]) with mapi id 14.03.0195.001;
 Wed, 5 Nov 2014 16:19:07 +0000
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: XU Liang <liang.xu@cinfotech.cn>, "dev@dpdk.org" <dev@dpdk.org>
Thread-Topic: =?utf-8?B?562U5aSN77ya562U5aSN77yaW2RwZGstZGV2XSBbUEFUQ0hdIGVhbDogbWFw? =?utf-8?B?IHVpbyByZXNvdXJjZXMgYWZ0ZXIgaHVnZXBhZ2VzIHdoZW4gdGhlCWJhc2Vf? =?utf-8?Q?virtaddr_is_configured.?Thread-Index: AQHP+PwmmmVWcq3WK0uPxBQDYbZnPZxSLpdIgAAGJuCAAABnkA=Date: Wed, 5 Nov 2014 16:19:07 +0000
Message-ID: <C6ECDF3AB251BE4894318F4E4512369780C071C8@IRSMSX109.ger.corp.intel.com>
References: <1415193919-17361-1-git-send-email-liang.xu@cinfotech.cn>,
 C6ECDF3AB251BE4894318F4E4512369780C070FA@IRSMSX109.ger.corp.intel.com
 <01d6ff37-3473-43af-aff3-1183d4c4768a@cinfotech.cn>,
 C6ECDF3AB251BE4894318F4E4512369780C07183@IRSMSX109.ger.corp.intel.com
 <7600981e-fc3a-454b-a269-51dd3b9d535b@cinfotech.cn>
In-Reply-To: <7600981e-fc3a-454b-a269-51dd3b9d535b@cinfotech.cn>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [163.33.239.182]
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Subject: Re: [dpdk-dev]
 =?utf-8?b?562U5aSN77ya562U5aSN77yaIFtQQVRDSF0gZWFsOiBt? =?utf-8?q?ap_uio_resources_after_hugepages_when_the	base_virtaddr_is_? =?utf-8?q?configured.?X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 05 Nov 2014 16:12:47 -0000

QWgsIHRoYXQgbWFrZXMgc2Vuc2UuIFNvIHlvdeKAmXJlIGFjdHVhbGx5IGhpdHRpbmcgdGhlIHZl
cnkgaXNzdWUgSeKAmW0gY29uY2VybmVkIGFib3V0IHdoZW4gbWFwcGluZyBwdXJlbHkgd2l0aCBi
YXNlLXZpcnRhZGRyIChhIGxpYnJhcnkgaXMgbWFwcGVkIGludG8gd2hlcmUgeW914oCZcmUgdHJ5
aW5nIHRvIG1hcCB5b3VyIFVJTyByZXNvdXJjZXMpLg0KDQpXZWxsLCBhcyBJIHNhaWQsIHlvdSBj
YW4gdHJ5IGFuZCB3YWxrIHRoZSBtZW1zZWdzIGFuZCB3b3JrIG91dCB0aGUgYmlnZ2VzdCBlbmQt
YWRkcmVzcyBvZiBodWdlcGFnZSBtZW1vcnkuIFRoYXTigJlzIHRoZSBlYXNpZXN0IHdheSBJIGNh
biB0aGluayBvZi4NCg0KVGhhbmtzLA0KQW5hdG9seQ0KDQpGcm9tOiBYVSBMaWFuZyBbbWFpbHRv
OmxpYW5nLnh1QGNpbmZvdGVjaC5jbl0NClNlbnQ6IFdlZG5lc2RheSwgTm92ZW1iZXIgNSwgMjAx
NCA0OjEwIFBNDQpUbzogQnVyYWtvdiwgQW5hdG9seTsgZGV2QGRwZGsub3JnDQpTdWJqZWN0OiDn
rZTlpI3vvJrnrZTlpI3vvJpbZHBkay1kZXZdIFtQQVRDSF0gZWFsOiBtYXAgdWlvIHJlc291cmNl
cyBhZnRlciBodWdlcGFnZXMgd2hlbiB0aGUgYmFzZV92aXJ0YWRkciBpcyBjb25maWd1cmVkLg0K
DQpJIGhhdmUgYSBtdWx0aXBsZSBwcm9jZXNzZXMgYXBwbGljYXRpb24uIFdoZW4gc3RhcnQgYSBz
ZWNvbmRhcnkgcHJvY2Vzcywgd2UgZ290IGVycm9yIG1lc3NhZ2UgIkVBTDogcGNpX21hcF9yZXNv
dXJjZSgpOiBjYW5ub3QgbW1hcCgxMSwgMHg3ZmZmZjdmYmEwMDAsIDB4MjAwMDAsIDB4MCk6IEJh
ZCBmaWxlIGRlc2NyaXB0b3IgKDB4N2ZmZmY3ZmI5MDAwKSIuDQoNClRoZSBzZWNvbmRhcnkgcHJv
Y2VzcyBsaW5rIGRpZmZlcmVuY2Ugc2hhcmVkIGxpYnJhcmllcywgc28gdGhlIGFkZHJlc3MgMHg3
ZmZmZjdmYmEwMDAgaXMgdXNlZC4NCg0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQrlj5Hku7bkurrvvJpCdXJha292LCBB
bmF0b2x5IDxhbmF0b2x5LmJ1cmFrb3ZAaW50ZWwuY29tPG1haWx0bzphbmF0b2x5LmJ1cmFrb3ZA
aW50ZWwuY29tPj4NCuWPkemAgeaXtumXtO+8mjIwMTTlubQxMeaciDXml6Uo5pif5pyf5LiJKSAy
Mzo1OQ0K5pS25Lu25Lq677ya5b6Q5LquIDxsaWFuZy54dUBjaW5mb3RlY2guY248bWFpbHRvOmxp
YW5nLnh1QGNpbmZvdGVjaC5jbj4+77yMZGV2QGRwZGsub3JnPG1haWx0bzpkZXZAZHBkay5vcmc+
IDxkZXZAZHBkay5vcmc8bWFpbHRvOmRldkBkcGRrLm9yZz4+DQrkuLvjgIDpopjvvJpSRTog562U
5aSN77yaW2RwZGstZGV2XSBbUEFUQ0hdIGVhbDogbWFwIHVpbyByZXNvdXJjZXMgYWZ0ZXIgaHVn
ZXBhZ2VzIHdoZW4gdGhlIGJhc2VfdmlydGFkZHIgaXMgY29uZmlndXJlZC4NCg0KSGkgTGlhbmcN
Cg0KWWVzIGl0IGlzIGEgcHJvYmxlbS4gRXZlbiBpZiBpdCB3YXMgY2FyZWZ1bGx5IHNlbGVjdGVk
IGJ5IHVzZXIsIG5vdGhpbmcgc3RvcHMgdGhlIERQREsgYXBwbGljYXRpb24gZnJvbSBtYXBwaW5n
IHNvbWV0aGluZyBpbnRvIHdoZXJlIHlvdeKAmXJlIHRyeWluZyB0byBtYXAgeW91ciBVSU8gZGV2
aWNlcy4gUGx1cywgdGhpcyBjaGFuZ2VzIHRoZSBkZWZhdWx0IGJlaGF2aW9yIHdoZXJlIGEgd3Jv
bmcgYmFzZS12aXJ0YWRkciBsZWFkcyB0byBhIGZhaWx1cmUgdG8gaW5pdGlhbGl6ZSwgcmF0aGVy
IHRoYW4gc2ltcGx5IHVzaW5nIGEgZGlmZmVyZW50IGFkZHJlc3MgKHJlbWVtYmVyIHRoYXQgcGNp
X21hcF9yZXNvdXJjZSBmYWlscyBpZiBpdCBjYW5ub3QgbWFwIHRoZSByZXNvdXJjZSBhdCB0aGUg
ZXhhY3QgYWRkcmVzcyB5b3UgcmVxdWVzdGVkKS4NCg0KQSB2ZXJ5IGNydWRlIHdheSBvZiBmaW5k
aW5nIG91dCB3aGVyZSBodWdlcGFnZXMgZW5kIHdvdWxkIGJlIHRvIHdhbGsgdGhlIGh1Z2VwYWdl
IG1lbW9yeSAod2FsayB0aHJvdWdoIG1lbXNlZ3MgYW5kIG5vdGUgdGhlIG1heGltdW0gc3RhcnQg
YWRkciArIGxlbmd0aCBvZiB0aGF0IG1lbXNlZykuDQoNCkNvdWxkIHlvdSBwZXJoYXBzIGV4cGxh
aW4gd2hhdCBpcyB0aGUgcHJvYmxlbSB0aGF0IHlvdeKAmXJlIHRyeWluZyB0byBzb2x2ZSB3aXRo
IHRoaXM/IEkgY2Fu4oCZdCB0aGluayBvZiBhIHNpdHVhdGlvbiB3aGVyZSB0aGUgbG9jYXRpb24g
b2YgVUlPIG1hcHMgd291bGQgbWF0dGVyLCB0byBiZSBob25lc3QuDQoNClRoYW5rcywNCkFuYXRv
bHkNCg0KRnJvbTogWFUgTGlhbmcgW21haWx0bzpsaWFuZy54dUBjaW5mb3RlY2guY25dDQpTZW50
OiBXZWRuZXNkYXksIE5vdmVtYmVyIDUsIDIwMTQgMzo0OSBQTQ0KVG86IEJ1cmFrb3YsIEFuYXRv
bHk7IGRldkBkcGRrLm9yZzxtYWlsdG86ZGV2QGRwZGsub3JnPg0KU3ViamVjdDog562U5aSN77ya
W2RwZGstZGV2XSBbUEFUQ0hdIGVhbDogbWFwIHVpbyByZXNvdXJjZXMgYWZ0ZXIgaHVnZXBhZ2Vz
IHdoZW4gdGhlIGJhc2VfdmlydGFkZHIgaXMgY29uZmlndXJlZC4NCg0KSSB0aGluayB0aGUgYmFz
ZV92aXJ0YWRkIHdpbGwgYmUgY2FyZWZ1bGx5IHNlbGVjdGVkIGJ5IHVzZXIgd2hlbiB0aGV5IG5l
ZWQgaXQuIFNvIG1heWJlIGl0J3Mgbm90IGEgcmVhbCBwcm9ibGVtLiAgOj4NCg0KVGhlIHJlYWwg
cmVhc29uIGlzIEkgY2FuJ3QgZmluZCBhIGVhc3kgd2F5IHRvIGdldCB0aGUgZW5kIGFkZHJlc3Mg
b2YgaHVnZXBhZ2VzLiBDYW4geW91IGdpdmUgbWUgc29tZSBzdWdnZXN0aW9ucyA/DQotLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0NCuWPkeS7tuS6uu+8mkJ1cmFrb3YsIEFuYXRvbHkgPGFuYXRvbHkuYnVyYWtvdkBpbnRlbC5j
b208bWFpbHRvOmFuYXRvbHkuYnVyYWtvdkBpbnRlbC5jb20+Pg0K5Y+R6YCB5pe26Ze077yaMjAx
NOW5tDEx5pyINeaXpSjmmJ/mnJ/kuIkpIDIzOjEwDQrmlLbku7bkurrvvJrlvpDkuq4gPGxpYW5n
Lnh1QGNpbmZvdGVjaC5jbjxtYWlsdG86bGlhbmcueHVAY2luZm90ZWNoLmNuPj7vvIxkZXZAZHBk
ay5vcmc8bWFpbHRvOmRldkBkcGRrLm9yZz4gPGRldkBkcGRrLm9yZzxtYWlsdG86ZGV2QGRwZGsu
b3JnPj4NCuS4u+OAgOmimO+8mlJFOiBbZHBkay1kZXZdIFtQQVRDSF0gZWFsOiBtYXAgdWlvIHJl
c291cmNlcyBhZnRlciBodWdlcGFnZXMgd2hlbiB0aGUgYmFzZV92aXJ0YWRkciBpcyBjb25maWd1
cmVkLg0KDQpJIGhhdmUgYSBzbGlnaHQgcHJvYmxlbXMgd2l0aCB0aGlzIHBhdGNoLg0KDQpUaGUg
YmFzZV92aXJ0YWRkciBkb2Vzbid0IG5lY2Vzc2FyaWx5IGNvcnJlc3BvbmQgdG8gYW4gYWRkcmVz
cyB0aGF0IGV2ZXJ5dGhpbmcgZ2V0cyBtYXBwZWQgdG8uIEl0J3MgYSAiaGludCIgb2Ygc29ydHMs
IHRoYXQgbWF5IG9yIG1heSBub3QgYmUgdGFrZW4gaW50byBhY2NvdW50IGJ5IG1tYXAuIFRoZXJl
Zm9yZSB3ZSBjYW4ndCBzaW1wbHkgYXNzdW1lIHRoYXQgaWYgd2UgcmVxdWVzdGVkIGEgYmFzZS12
aXJ0YWRkciwgZXZlcnl0aGluZyB3aWxsIGdldCBtYXBwZWQgYXQgZXhhY3RseSB0aGF0IGFkZHJl
c3MuIFdlIGFsc28gY2FuJ3QgYXNzdW1lIHRoYXQgaHVnZXBhZ2VzIHdpbGwgYmUgb3JkZXJlZCBv
bmUgYWZ0ZXIgdGhlIG90aGVyIGFuZCBvY2N1cHkgbmVhdGx5IGFsbCB0aGUgY29udGlndW91cyB2
aXJ0dWFsIG1lbW9yeSBiZXR3ZWVuIGJhc2VfdmlydGFkZHIgYW5kIGJhc2VfdmlydGFkZHIgKyBp
bnRlcm5hbF9jb25maWcubWVtb3J5IC0gdGhlcmUgbWF5IGJlIGhvbGVzLCBmb3Igd2hhdGV2ZXIg
cmVhc29ucy4NCg0KQWxzbywNCg0KVGhhbmtzLA0KQW5hdG9seQ0KDQotLS0tLU9yaWdpbmFsIE1l
c3NhZ2UtLS0tLQ0KRnJvbTogZGV2IFttYWlsdG86ZGV2LWJvdW5jZXNAZHBkay5vcmddIE9uIEJl
aGFsZiBPZiBseHUNClNlbnQ6IFdlZG5lc2RheSwgTm92ZW1iZXIgNSwgMjAxNCAxOjI1IFBNDQpU
bzogZGV2QGRwZGsub3JnPG1haWx0bzpkZXZAZHBkay5vcmc+DQpTdWJqZWN0OiBbZHBkay1kZXZd
IFtQQVRDSF0gZWFsOiBtYXAgdWlvIHJlc291cmNlcyBhZnRlciBodWdlcGFnZXMgd2hlbiB0aGUg
YmFzZV92aXJ0YWRkciBpcyBjb25maWd1cmVkLg0KDQotLS0NCmxpYi9saWJydGVfZWFsL2xpbnV4
YXBwL2VhbC9lYWxfcGNpX3Vpby5jIHwgOSArKysrKysrKy0NCjEgZmlsZSBjaGFuZ2VkLCA4IGlu
c2VydGlvbnMoKyksIDEgZGVsZXRpb24oLSkNCg0KZGlmZiAtLWdpdCBhL2xpYi9saWJydGVfZWFs
L2xpbnV4YXBwL2VhbC9lYWxfcGNpX3Vpby5jIGIvbGliL2xpYnJ0ZV9lYWwvbGludXhhcHAvZWFs
L2VhbF9wY2lfdWlvLmMNCmluZGV4IDdlNjIyNjYuLmJjN2VkM2EgMTAwNjQ0DQotLS0gYS9saWIv
bGlicnRlX2VhbC9saW51eGFwcC9lYWwvZWFsX3BjaV91aW8uYw0KKysrIGIvbGliL2xpYnJ0ZV9l
YWwvbGludXhhcHAvZWFsL2VhbF9wY2lfdWlvLmMNCkBAIC0yODksNiArMjg5LDExIEBAIHBjaV91
aW9fbWFwX3Jlc291cmNlKHN0cnVjdCBydGVfcGNpX2RldmljZSAqZGV2KQ0Kc3RydWN0IHJ0ZV9w
Y2lfYWRkciAqbG9jID0gJmRldi0+YWRkcjsNCnN0cnVjdCBtYXBwZWRfcGNpX3Jlc291cmNlICp1
aW9fcmVzOw0Kc3RydWN0IHBjaV9tYXAgKm1hcHM7DQorIHN0YXRpYyB2b2lkICogcmVxdWVzdGVk
X2FkZHIgPSBOVUxMOw0KKyBpZiAoaW50ZXJuYWxfY29uZmlnLmJhc2VfdmlydGFkZHIgJiYgTlVM
TCA9PSByZXF1ZXN0ZWRfYWRkcikgew0KKyByZXF1ZXN0ZWRfYWRkciA9ICh1aW50OF90ICopIGlu
dGVybmFsX2NvbmZpZy5iYXNlX3ZpcnRhZGRyDQorICsgaW50ZXJuYWxfY29uZmlnLm1lbW9yeTsN
CisgfQ0KDQpkZXYtPmludHJfaGFuZGxlLmZkID0gLTE7DQpkZXYtPmludHJfaGFuZGxlLnR5cGUg
PSBSVEVfSU5UUl9IQU5ETEVfVU5LTk9XTjsgQEAgLTM3MSwxMCArMzc2LDEyIEBAIHBjaV91aW9f
bWFwX3Jlc291cmNlKHN0cnVjdCBydGVfcGNpX2RldmljZSAqZGV2KQ0KaWYgKG1hcHNbal0uYWRk
ciAhPSBOVUxMKQ0KZmFpbCA9IDE7DQplbHNlIHsNCi0gbWFwYWRkciA9IHBjaV9tYXBfcmVzb3Vy
Y2UoTlVMTCwgZmQsIChvZmZfdClvZmZzZXQsDQorIG1hcGFkZHIgPSBwY2lfbWFwX3Jlc291cmNl
KHJlcXVlc3RlZF9hZGRyLCBmZCwgKG9mZl90KW9mZnNldCwNCihzaXplX3QpbWFwc1tqXS5zaXpl
KTsNCmlmIChtYXBhZGRyID09IE5VTEwpDQpmYWlsID0gMTsNCisgZWxzZSBpZiAoTlVMTCAhPSBy
ZXF1ZXN0ZWRfYWRkcikNCisgcmVxdWVzdGVkX2FkZHIgPSAodWludDhfdCAqKW1hcGFkZHIgKyBt
YXBzW2pdLnNpemU7DQp9DQoNCmlmIChmYWlsKSB7DQotLQ0KMS45LjENCg=

Re: [dpdk-dev] 答复：答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Qiu, Michael]

On 11/6/2014 12:12 AM, XU Liang wrote:
> I have a multiple processes application. When start a secondary process, we got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".The secondary process link difference shared libraries, so the address 0x7ffff7fba000 is used.

This is a known issue, and still not be solved yet.  The root cause is
exactly clear, that should be try to map an address have already used in
new process.

BTW, you should learn how to make a patch, like commit log,
signed-off-by, etc.

Thanks,
Michael

> ------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov@intel.com>发送时间：2014年11月5日(星期三) 23:59收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org <dev@dpdk.org>主　题：RE: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the	base_virtaddr is configured.
>
>
>
> font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0cm;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
> a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
> a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
> span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
> *.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
> size: 612.0pt 792.0pt;margin: 72.0pt 72.0pt 72.0pt 72.0pt;div.WordSection1 {page: WordSection1;}
>
>
>
>
> Hi Liang
>  
> Yes it is a problem. Even if it was carefully selected by user, nothing stops the DPDK application from mapping something into where you’re trying to map your
>  UIO devices. Plus, this changes the default behavior where a wrong base-virtaddr leads to a failure to initialize, rather than simply using a different address (remember that pci_map_resource fails if it cannot map the resource at the exact address you requested).
>  
> A very crude way of finding out where hugepages end would be to walk the hugepage memory (walk through memsegs and note the maximum start addr + length of that
>  memseg).
>  
> Could you perhaps explain what is the problem that you’re trying to solve with this? I can’t think of a situation where the location of UIO maps would matter,
>  to be honest.
>  
> Thanks,
> Anatoly
>  
> From: XU Liang [mailto:liang.xu@cinfotech.cn]
>
>
> Sent: Wednesday, November 5, 2014 3:49 PM
>
> To: Burakov, Anatoly; dev@dpdk.org
>
> Subject: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
>  
>
>
> I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>
>
>
>  
>
>
> The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?
>
>
>
> ------------------------------------------------------------------
>
>
> 发件人：Burakov, Anatoly <anatoly.burakov@intel.com>
>
>
> 发送时间：2014年11月5日(星期三)
>  23:10
>
>
> 收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org
>  <dev@dpdk.org>
>
>
> 主　题：RE:
>  [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
>
>
>  
>
> I have a slight problems with this patch.
>
>
>
> The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will
>  get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever
>  reasons.
>
>
>
> Also, 
>
>
>
> Thanks,
>
> Anatoly
>
>
>
> -----Original Message-----
>
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
>
> Sent: Wednesday, November 5, 2014 1:25 PM
>
> To: dev@dpdk.org
>
> Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
>
>
>
> ---
>
> lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
>
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
>
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
>
> index 7e62266..bc7ed3a 100644
>
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
>
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
>
> @@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>
> struct rte_pci_addr *loc = &dev->addr;
>
> struct mapped_pci_resource *uio_res;
>
> struct pci_map *maps;
>
> + static void * requested_addr = NULL;
>
> + if (internal_config.base_virtaddr && NULL == requested_addr) {
>
> + requested_addr = (uint8_t *) internal_config.base_virtaddr 
>
> + + internal_config.memory;
>
> + }
>
>
>
> dev->intr_handle.fd = -1;
>
> dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>
> if (maps[j].addr != NULL)
>
> fail = 1;
>
> else {
>
> - mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
>
> + mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
>
> (size_t)maps[j].size);
>
> if (mapaddr == NULL)
>
> fail = 1;
>
> + else if (NULL != requested_addr)
>
> + requested_addr = (uint8_t *)mapaddr + maps[j].size;
>
> }
>
>
>
> if (fail) {
>
> --
>
> 1.9.1
>
>
>
>

Re: [dpdk-dev] 答复：答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

> This is a known issue, and still not be solved yet.  The root cause is exactly
> clear, that should be try to map an address have already used in new
> process.
> 
> BTW, you should learn how to make a patch, like commit log, signed-off-by,
> etc.

Hi Michael,

As far as I know, the patch that fixes this issue was integrated yesterday (thanks Thomas!), and Liang Xu verified it to be working.

Thanks,
Anatoly

[dpdk-dev] [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = (void *)(last->addr_64 + last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [dpdk-dev] [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Few nitpicks.

Static variables are always initialized to 0, so "= NULL" isn't necessary, a declaration will suffice. Also, we have a macro RTE_PTR_ADD to add numbers to pointers, I think it would be better to use those. Otherwise, looks fine to me.

I still feel uneasy about depending on nothing being mapped directly after hugepages (perhaps we could do mmap(bar_size) before trying pci_map_resource, and increment requested_addr until we find a free spot?), but I imagine this case would be quite rare, so probably it's not worth the added kludge.

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Thursday, November 6, 2014 2:12 PM
To: dev@dpdk.org
Cc: Burakov, Anatoly
Subject: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = (void *)(last->addr_64 + last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

[dpdk-dev] 答复：[PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

[徐亮]

When user configure base_virtaddr, we should believe they can take care it.
In my case, I always check /proc/xxxx/maps to find a huge free address space, such as 0x20 0000 0000, to map all the hugepages and uio resource. 
------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov@intel.com>发送时间：2014年11月6日(星期四) 22:29收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org <dev@dpdk.org>主　题：RE: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
Few nitpicks.

Static variables are always initialized to 0, so "= NULL" isn't necessary, a declaration will suffice. Also, we have a macro RTE_PTR_ADD to add numbers to pointers, I think it would be better to use those. Otherwise, looks fine to me.

I still feel uneasy about depending on nothing being mapped directly after hugepages (perhaps we could do mmap(bar_size) before trying pci_map_resource, and increment requested_addr until we find a free spot?), but I imagine this case would be quite rare, so probably it's not worth the added kludge.

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Thursday, November 6, 2014 2:12 PM
To: dev@dpdk.org
Cc: Burakov, Anatoly
Subject: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = (void *)(last->addr_64 + last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1
From anatoly.burakov@intel.com  Thu Nov  6 15:54:34 2014
Return-Path: <anatoly.burakov@intel.com>
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
 by dpdk.org (Postfix) with ESMTP id 23B417E75
 for <dev@dpdk.org>; Thu,  6 Nov 2014 15:54:31 +0100 (CET)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
 by fmsmga101.fm.intel.com with ESMTP; 06 Nov 2014 07:02:45 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,326,1413270000"; d="scan'208";a="618245383"
Received: from irsmsx101.ger.corp.intel.com ([163.33.3.153])
 by fmsmga001.fm.intel.com with ESMTP; 06 Nov 2014 06:59:25 -0800
Received: from irsmsx107.ger.corp.intel.com (163.33.3.99) by
 IRSMSX101.ger.corp.intel.com (163.33.3.153) with Microsoft SMTP Server (TLS)
 id 14.3.195.1; Thu, 6 Nov 2014 14:58:10 +0000
Received: from irsmsx109.ger.corp.intel.com ([169.254.13.101]) by
 IRSMSX107.ger.corp.intel.com ([169.254.10.154]) with mapi id 14.03.0195.001;
 Thu, 6 Nov 2014 14:58:09 +0000
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: lxu <liang.xu@cinfotech.cn>, "dev@dpdk.org" <dev@dpdk.org>
Thread-Topic: [PATCH v3] eal: map uio resources after hugepages when the
 base_virtaddr is configured.
Thread-Index: AQHP+dCl0ziZwg+l7UqDsOao12IOWZxTsFEg
Date: Thu, 6 Nov 2014 14:58:09 +0000
Message-ID: <C6ECDF3AB251BE4894318F4E4512369780C075E8@IRSMSX109.ger.corp.intel.com>
References: <1415193919-17361-1-git-send-email-liang.xu@cinfotech.cn>
 <1415285223-7662-1-git-send-email-liang.xu@cinfotech.cn>
In-Reply-To: <1415285223-7662-1-git-send-email-liang.xu@cinfotech.cn>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [163.33.239.181]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: [dpdk-dev] [PATCH v3] eal: map uio resources after hugepages
 when the base_virtaddr is configured.
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 06 Nov 2014 14:54:34 -0000

+					requested_addr = (uint8_t *)mapaddr + maps[j].size;

RTE_PTR_ADD?

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Thursday, November 6, 2014 2:47 PM
To: dev@dpdk.org
Cc: Burakov, Anatoly
Subject: [PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..3a218d0 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
--
1.9.1

[dpdk-dev] [PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..3a218d0 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [dpdk-dev] [PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.

[De Lara Guarch, Pablo]

Include at least signoff.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
> Sent: Thursday, November 06, 2014 2:47 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v3] eal: map uio resources after hugepages
> when the base_virtaddr is configured.
> 
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29
> ++++++++++++++++++++++++++++-
>  1 file changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..3a218d0 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char
> *dstbuf,
>  	return uio_num;
>  }
> 
> +static inline const struct rte_memseg *
> +get_physmem_last(void)
> +{
> +	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg * last = seg;
> +	unsigned i = 0;
> +
> +	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if(seg->addr > last->addr)
> +		 	last = seg;
> +
> +	}
> +	return last;
> +}
> +
>  /* map the PCI resource of a PCI device in virtual memory */
>  int
>  pci_uio_map_resource(struct rte_pci_device *dev)
> @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  	struct mapped_pci_resource *uio_res;
>  	struct pci_map *maps;
> 
> +	/* map uio resource into user required virtual address */
> +	static void * requested_addr;
> +	if (internal_config.base_virtaddr && NULL == requested_addr) {
> +		const struct rte_memseg * last = get_physmem_last();
> +		requested_addr = RTE_PTR_ADD(last->addr, last->len);
> +	}
> +
>  	dev->intr_handle.fd = -1;
>  	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
> 
> @@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  			if (maps[j].addr != NULL)
>  				fail = 1;
>  			else {
> -				mapaddr = pci_map_resource(NULL, fd,
> (off_t)offset,
> +				mapaddr =
> pci_map_resource(requested_addr, fd, (off_t)offset,
>  						(size_t)maps[j].size);
>  				if (mapaddr == NULL)
>  					fail = 1;
> +				else if (NULL != requested_addr)
> +					requested_addr = (uint8_t
> *)mapaddr + maps[j].size;
>  			}
> 
>  			if (fail) {
> --
> 1.9.1

[dpdk-dev] [PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [dpdk-dev] [PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Thomas Monjalon]

No explanation -> I won't review it.
No signed-off-by -> I'll ignore it.

Maybe your patch is useful, maybe not. I cannot know.

-- 
Thomas

[dpdk-dev] [PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

[dpdk-dev] [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

Sorry, I'm learning the right way to send a patch by git. 
I have a multiple processes application. When start the secondary process, I got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".
The secondary process links a lot of additional shared libraries, so the address 0x7ffff7fba000 had already be used.
I had fixed similar hugepages mmap problems by base_virtaddr. So I believe the uio resource should be mapped into base_virtaddr at this situation.
This patch try to fix it.


Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [dpdk-dev] [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

The explanation of the patch should be generic and impartial, i.e. when this and this happens, it results in such and such problem, and this patch fixes it by doing this and that. In other words, this will appear in the git history, so whoever is reading the commit log will be able to figure out what this patch does and why it's been applied.

Thomas, do we need to do similar changes to VFIO code, to keep consistency? Also, do we really need for this to depend on --base-virtaddr? Why not do it unconditionally, i.e. map PCI resources right after hugepages in memory?

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Thursday, November 6, 2014 3:32 PM
To: dev@dpdk.org
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

Sorry, I'm learning the right way to send a patch by git. 
I have a multiple processes application. When start the secondary process, I got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".
The secondary process links a lot of additional shared libraries, so the address 0x7ffff7fba000 had already be used.
I had fixed similar hugepages mmap problems by base_virtaddr. So I believe the uio resource should be mapped into base_virtaddr at this situation.
This patch try to fix it.


Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
--
1.9.1

Re: [dpdk-dev] [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Thomas Monjalon]

2014-11-06 15:41, Burakov, Anatoly:
> Thomas, do we need to do similar changes to VFIO code, to keep consistency?
> Also, do we really need for this to depend on --base-virtaddr? Why not do it
> unconditionally, i.e. map PCI resources right after hugepages in memory?

I don't really like the secondary process mechanism at all.
So I won't give good advice here ;)
But I feel this option --base-virtaddr should be improved or removed.

-- 
Thomas

Re: [dpdk-dev] [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Well, removing --base-virtaddr is not what I'm asking about.

The issue at hand here is that, given our secondary process mechanism (that you don't like :-) ), some stuff may be attempted to be mapped into space a secondary process may already have mapped something else into (some libraries, for example). This issue was originally discovered by OVDK, so we added a --base-virtaddr option to try and map hugepages at exact virtual address, rather than wherever mmap decides to do so on its own.

The issue encountered by Liang (the author of the patch) is similar, only it's not the hugepages are mapped into the occupied space, but rather the PCI resources (which are mapped with NULL by default, so can be mapped anywhere). Therefore he suggested a patch that maps the PCI resources into a space just after the last hugepage when --base-virtaddr is provided. I'm not sure we need the dependence on --base-virtaddr though, it can probably be done unconditionally. If you have no opinion on the matter, we can leave this detail of the patch as it is, then.

Also, I would suspect that if we are to modify where UIO resources are mapped, VFIO code should be modified the same way to avoid inconsistency between the two.

Thanks,
Anatoly

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Thursday, November 6, 2014 3:58 PM
To: Burakov, Anatoly
Cc: lxu; dev@dpdk.org; De Lara Guarch, Pablo
Subject: Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

2014-11-06 15:41, Burakov, Anatoly:
> Thomas, do we need to do similar changes to VFIO code, to keep consistency?
> Also, do we really need for this to depend on --base-virtaddr? Why not 
> do it unconditionally, i.e. map PCI resources right after hugepages in memory?

I don't really like the secondary process mechanism at all.
So I won't give good advice here ;)
But I feel this option --base-virtaddr should be improved or removed.

--
Thomas

Re: [dpdk-dev] [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Bruce Richardson]

On Thu, Nov 06, 2014 at 04:10:15PM +0000, Burakov, Anatoly wrote:
> Well, removing --base-virtaddr is not what I'm asking about.
> 
> The issue at hand here is that, given our secondary process mechanism (that you don't like :-) ), some stuff may be attempted to be mapped into space a secondary process may already have mapped something else into (some libraries, for example). This issue was originally discovered by OVDK, so we added a --base-virtaddr option to try and map hugepages at exact virtual address, rather than wherever mmap decides to do so on its own.
> 
> The issue encountered by Liang (the author of the patch) is similar, only it's not the hugepages are mapped into the occupied space, but rather the PCI resources (which are mapped with NULL by default, so can be mapped anywhere). Therefore he suggested a patch that maps the PCI resources into a space just after the last hugepage when --base-virtaddr is provided. I'm not sure we need the dependence on --base-virtaddr though, it can probably be done unconditionally. If you have no opinion on the matter, we can leave this detail of the patch as it is, then.
> 
> Also, I would suspect that if we are to modify where UIO resources are mapped, VFIO code should be modified the same way to avoid inconsistency between the two.
> 

I find nothing wrong with your logic, Anatoly, it makes sense to me. :-)

I'm curious, however, as to what Thomas has in mind for how we might improve
the base-virtaddr flag.

/Bruce

> Thanks,
> Anatoly
> 
> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
> Sent: Thursday, November 6, 2014 3:58 PM
> To: Burakov, Anatoly
> Cc: lxu; dev@dpdk.org; De Lara Guarch, Pablo
> Subject: Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.
> 
> 2014-11-06 15:41, Burakov, Anatoly:
> > Thomas, do we need to do similar changes to VFIO code, to keep consistency?
> > Also, do we really need for this to depend on --base-virtaddr? Why not 
> > do it unconditionally, i.e. map PCI resources right after hugepages in memory?
> 
> I don't really like the secondary process mechanism at all.
> So I won't give good advice here ;)
> But I feel this option --base-virtaddr should be improved or removed.
> 
> --
> Thomas

[dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.

This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Bruce Richardson]

On Fri, Nov 07, 2014 at 04:01:24PM +0800, lxu wrote:
> A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
> 
> This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
> 
> Signed-off-by: lxu <liang.xu@cinfotech.cn>
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
>  1 file changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..a2c9ab6 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
>  	return uio_num;
>  }
>  
> +static inline const struct rte_memseg *
> +get_physmem_last(void)
> +{
> +	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg * last = seg;
> +	unsigned i = 0;
> +
> +	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if(seg->addr > last->addr)
> +		 	last = seg;
> +
> +	}
> +	return last;
> +}
> +
>  /* map the PCI resource of a PCI device in virtual memory */
>  int
>  pci_uio_map_resource(struct rte_pci_device *dev)
> @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  	struct mapped_pci_resource *uio_res;
>  	struct pci_map *maps;
>  
> +	/* map uio resource into user required virtual address */
> +	static void * requested_addr;
> +	if (internal_config.base_virtaddr && NULL == requested_addr) {
> +		const struct rte_memseg * last = get_physmem_last();
> +		requested_addr = RTE_PTR_ADD(last->addr, last->len);
> +	}
> +

Could you perhaps take anatoly's suggestion and modify this patch so the checking
for a space right after the memsegs does not just depend on base-virtaddr being
set. Unless it causes other problems, there is no reason this code should not
always be used.

/Bruce

Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

The commit message looks fine to me, but VFIO code needs to be adjusted the same way.

Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).

My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).

My apologies for not thinking about any of this during the first 6 iterations of the patch :(

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Friday, November 7, 2014 8:01 AM
To: dev@dpdk.org
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.

This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
--
1.9.1

Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.
 If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.
------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov@intel.com>Time:2014 Nov 7 (Fri) 17 : 47To:徐亮 <liang.xu@cinfotech.cn>, dev@dpdk.org <dev@dpdk.org>Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
The commit message looks fine to me, but VFIO code needs to be adjusted the same way.

Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).

My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).

My apologies for not thinking about any of this during the first 6 iterations of the patch :(

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Friday, November 7, 2014 8:01 AM
To: dev@dpdk.org
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.

This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
--
1.9.1
From anatoly.burakov@intel.com  Fri Nov  7 10:53:50 2014
Return-Path: <anatoly.burakov@intel.com>
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
 by dpdk.org (Postfix) with ESMTP id 2EF227EC4
 for <dev@dpdk.org>; Fri,  7 Nov 2014 10:53:45 +0100 (CET)
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
 by fmsmga101.fm.intel.com with ESMTP; 07 Nov 2014 02:02:55 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,331,1413270000";
 d="scan'208,217";a="628228403"
Received: from irsmsx104.ger.corp.intel.com ([163.33.3.159])
 by fmsmga002.fm.intel.com with ESMTP; 07 Nov 2014 02:02:40 -0800
Received: from irsmsx107.ger.corp.intel.com (163.33.3.99) by
 IRSMSX104.ger.corp.intel.com (163.33.3.159) with Microsoft SMTP Server (TLS)
 id 14.3.195.1; Fri, 7 Nov 2014 10:00:58 +0000
Received: from irsmsx109.ger.corp.intel.com ([169.254.13.101]) by
 IRSMSX107.ger.corp.intel.com ([169.254.10.154]) with mapi id 14.03.0195.001;
 Fri, 7 Nov 2014 10:00:57 +0000
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: XU Liang <liang.xu@cinfotech.cn>, "dev@dpdk.org" <dev@dpdk.org>
Thread-Topic: [PATCH v6] eal: map uio resources after hugepages when the
 base_virtaddr is configured.
Thread-Index: AQHP+mEdCoJJM6uhEkSpotX1GZcOhJxU7g3YgAAALIADate: Fri, 7 Nov 2014 10:00:57 +0000
Message-ID: <C6ECDF3AB251BE4894318F4E4512369780C07848@IRSMSX109.ger.corp.intel.com>
References: <1415193919-17361-1-git-send-email-liang.xu@cinfotech.cn>
 <1415347284-15468-1-git-send-email-liang.xu@cinfotech.cn>,
 C6ECDF3AB251BE4894318F4E4512369780C07822@IRSMSX109.ger.corp.intel.com
 <2c2ad9a4-1c04-4641-9667-249617ae2c56@cinfotech.cn>
In-Reply-To: <2c2ad9a4-1c04-4641-9667-249617ae2c56@cinfotech.cn>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [163.33.239.180]
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Subject: Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages
 when the base_virtaddr is configured.
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Nov 2014 09:53:50 -0000

QWgsIHllcywgc29ycnkgYWJvdXQgdGhhdCwgaGF2ZW7igJl0IHF1aXRlIHdva2UgdXAgeWV0IOKY
uSBZb3XigJlyZSByaWdodC4gU28gaXTigJlzIHJlbW92aW5nIHRoZSBkZXBlbmRlbmN5IG9uIC0t
YmFzZS12aXJ0YWRkciwgbW92aW5nIHRoZSBmdW5jdGlvbiB0byBlYWxfcGNpLmMgIGFuZCBhZGRp
bmcgc2ltaWxhciBWRklPIGNvZGUgdGhhdOKAmXMgbGVmdCB0aGVuLg0KDQpUaGFua3MsDQpBbmF0
b2x5DQoNCkZyb206IFhVIExpYW5nIFttYWlsdG86bGlhbmcueHVAY2luZm90ZWNoLmNuXQ0KU2Vu
dDogRnJpZGF5LCBOb3ZlbWJlciA3LCAyMDE0IDk6NTcgQU0NClRvOiBCdXJha292LCBBbmF0b2x5
OyBkZXZAZHBkay5vcmcNCkNjOiB0aG9tYXMubW9uamFsb25ANndpbmQuY29tOyBEZSBMYXJhIEd1
YXJjaCwgUGFibG8NClN1YmplY3Q6IFJlOiBbUEFUQ0ggdjZdIGVhbDogbWFwIHVpbyByZXNvdXJj
ZXMgYWZ0ZXIgaHVnZXBhZ2VzIHdoZW4gdGhlIGJhc2VfdmlydGFkZHIgaXMgY29uZmlndXJlZC4N
Cg0KSG93IHRvIGZpbmQgdGhlIG1heGltdW0gZW5kIHZpcnR1YWwgYWRkcmVzcyA/IEknbSBub3Qg
dGhlIERQREsgZXhwZXJ0LCBidXQgSSB3aWxsIHRyeSB0byBkbyBteSBiZXN0Lg0KDQpJZiB0aGUg
c2VnbWVudHMgaXNuJ3Qgb3ZlcmxhcCwgImlmKHNlZy0+YWRkciA+IGxhc3QtPmFkZHIpIGxhc3Qg
PSBzZWc7IiBhbHJlYWR5IGZpbmQgdGhlIHNlZ21lbnQgd2l0aCBtYXhpbXVtIGVuZCB2aXJ0dWFs
IGFkZHJlc3MuDQoNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KRnJvbTpCdXJha292LCBBbmF0b2x5IDxhbmF0b2x5LmJ1
cmFrb3ZAaW50ZWwuY29tPG1haWx0bzphbmF0b2x5LmJ1cmFrb3ZAaW50ZWwuY29tPj4NClRpbWU6
MjAxNCBOb3YgNyAoRnJpKSAxNyA6IDQ3DQpUbzrlvpDkuq4gPGxpYW5nLnh1QGNpbmZvdGVjaC5j
bjxtYWlsdG86bGlhbmcueHVAY2luZm90ZWNoLmNuPj4sIGRldkBkcGRrLm9yZzxtYWlsdG86ZGV2
QGRwZGsub3JnPiA8ZGV2QGRwZGsub3JnPG1haWx0bzpkZXZAZHBkay5vcmc+Pg0KQ2M6dGhvbWFz
Lm1vbmphbG9uQDZ3aW5kLmNvbSA8dGhvbWFzLm1vbmphbG9uQDZ3aW5kLmNvbTxtYWlsdG86dGhv
bWFzLm1vbmphbG9uQDZ3aW5kLmNvbT4+LCBEZSBMYXJhIEd1YXJjaCwgUGFibG8gPHBhYmxvLmRl
LmxhcmEuZ3VhcmNoQGludGVsLmNvbTxtYWlsdG86cGFibG8uZGUubGFyYS5ndWFyY2hAaW50ZWwu
Y29tPj4NClN1YmplY3Q6UkU6IFtQQVRDSCB2Nl0gZWFsOiBtYXAgdWlvIHJlc291cmNlcyBhZnRl
ciBodWdlcGFnZXMgd2hlbiB0aGUgYmFzZV92aXJ0YWRkciBpcyBjb25maWd1cmVkLg0KDQpUaGUg
Y29tbWl0IG1lc3NhZ2UgbG9va3MgZmluZSB0byBtZSwgYnV0IFZGSU8gY29kZSBuZWVkcyB0byBi
ZSBhZGp1c3RlZCB0aGUgc2FtZSB3YXkuDQoNCkFsc28sIG5vdyB0aGF0IEkgdGhpbmsgb2YgaXQs
IHlvdSBjYW4ndCBzaW1wbHkgYXNzdW1lIHRoYXQgd2hhdGV2ZXIgbGFzdCBtZW1zZWcgeW91IGhh
dmUgaGFzIHRoZSBsYXRlc3QgdmlydHVhbCBhZGRyZXNzLiBXaGVuIElWU0hNRU0gaXMgaW5pdGlh
bGl6ZWQsIGl0IHRvbyByZXNlcnZlcyBzb21lIHNwYWNlIGluIHRoZSB2aXJ0dWFsIG1lbW9yeSwg
d2hpY2ggY2FuIGJlIGhpZ2hlciB0aGFuIHRoZSBsYXN0IGh1Z2VwYWdlLCBidXQgbm90IGJlIHRo
ZSBsYXN0IGh1Z2VwYWdlIChiZWNhdXNlIElWU0hNRU0gbWVtb3J5IGlzIGZpcnN0IHRvIGJlIHJl
c2VydmVkLCBiZWZvcmUgdGhlIG1haW4gbWVtb3J5KS4NCg0KTXkgYWR2aWNlIHdvdWxkIGJlIHRv
IHJld3JpdGUgdGhlIGZ1bmN0aW9uIHRvIHJldHVybiB0aGUgbWF4aW11bSBlbmQgdmlydHVhbCBh
ZGRyZXNzIChpbnN0ZWFkIG9mIGEgbGFzdCBzZWdtZW50KSBhbmQgbW92ZSBpdCB0byBlYWxfcGNp
LmMgKGFuZCBpbmNsdWRlIGRlY2xhcmF0aW9uIGZvciBpdCBpbiBpbmNsdWRlL2VhbF9wY2lfaW5p
dC5oKS4NCg0KTXkgYXBvbG9naWVzIGZvciBub3QgdGhpbmtpbmcgYWJvdXQgYW55IG9mIHRoaXMg
ZHVyaW5nIHRoZSBmaXJzdCA2IGl0ZXJhdGlvbnMgb2YgdGhlIHBhdGNoIDooDQoNClRoYW5rcywN
CkFuYXRvbHkNCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IGx4dSBbbWFpbHRv
OmxpYW5nLnh1QGNpbmZvdGVjaC5jbl0NClNlbnQ6IEZyaWRheSwgTm92ZW1iZXIgNywgMjAxNCA4
OjAxIEFNDQpUbzogZGV2QGRwZGsub3JnPG1haWx0bzpkZXZAZHBkay5vcmc+DQpDYzogQnVyYWtv
diwgQW5hdG9seTsgdGhvbWFzLm1vbmphbG9uQDZ3aW5kLmNvbTxtYWlsdG86dGhvbWFzLm1vbmph
bG9uQDZ3aW5kLmNvbT47IERlIExhcmEgR3VhcmNoLCBQYWJsbw0KU3ViamVjdDogW1BBVENIIHY2
XSBlYWw6IG1hcCB1aW8gcmVzb3VyY2VzIGFmdGVyIGh1Z2VwYWdlcyB3aGVuIHRoZSBiYXNlX3Zp
cnRhZGRyIGlzIGNvbmZpZ3VyZWQuDQoNCkEgbXVsdGlwbGUgcHJvY2VzcyBEUERLIGFwcGxpY2F0
aW9uIG11c3QgbW1hcCBodWdlcGFnZXMgYW5kIHBjaSByZXNvdXJjZSBpbnRvIHNhbWUgdmlydHVh
bCBhZGRyZXNzZXMuIEJ5IGRlZmF1bHQgdGhlIHZpcnR1YWwgYWRkcmVzc2VzIGNob3NlbiBieSB0
aGUgcHJpbWFyeSBwcm9jZXNzIGF1dG9tYXRpY2FsbHkgd2hlbiBjYWxsaW5nIHRoZSBtbWFwLiBC
dXQgc29tZXRpbWUgdGhlIHZpcnR1YWwgYWRkcmVzc2VzIGNob3NlbiBieSB0aGUgcHJpbWFyeSBw
cm9jZXNzIGlzbid0IHVzYWJsZSBhdCBzZWNvbmRhcnkgcHJvY2Vzcy4gU3VjaCBhcyB0aGUgc2Vj
b25kYXJ5IHByb2Nlc3MgbGlua2VkIHdpdGggbW9yZSBsaWJyYXJpZXMgdGhhbiBwcmltYXJ5IHBy
b2Nlc3MuIFRoZSBsaWJyYXJ5IGhhcyBiZWVuIG1hcHBlZCBpbnRvIHRoaXMgdmlydHVhbCBhZGRy
ZXNzLiBUaGUgY29tbWFuZCBsaW5lIHBhcmFtZXRlciAnYmFzZS12aXJ0YWRkcicgaGFzIGJlZW4g
YWRkZWQgZm9yIHRoaXMgc2l0dWF0aW9uLiBJZiBpdCdzIGNvbmZpZ3VyZWQsIHRoZSBodWdlcGFn
ZXMgd2lsbCBiZSBtYXBwZWQgaW50byB0aGlzIGJhc2UgYWRkcmVzcy4gQnV0IHRoZSB2aXJ0dWFs
IGFkZHJlc3Mgb2YgdWlvIHJlc291cmNlIG1hcHBlZCBzdGlsbCBkb2VzIG5vdCByZWZlciB0byB0
aGUgcGFyYW1ldGVyLiBJbiB0aGF0IGNhc2UgaXQgd291bGQgc3RpbGwgZmFpbC4NCg0KVGhpcyBw
YXRjaCB0cnkgdG8gbWFwIHVpbyByZXNvdXJjZXMgYWZ0ZXIgaHVnZXBhZ2VzIHdoZW4gdGhlIGJh
c2VfdmlydGFkZHIgaXMgY29uZmlndXJlZC4gU28gdGhlIGVycm9yIG9mICJFQUw6IHBjaV9tYXBf
cmVzb3VyY2UoKTogY2Fubm90IG1tYXAiIGNhbiBiZSByZXNvbHZlZCBieSBzZXQgYmFzZS12aXJ0
YWRkciBpbnRvIGZyZWUgdmlydHVhbCBhZGRyZXNzIHNwYWNlLg0KDQpTaWduZWQtb2ZmLWJ5OiBs
eHUgPGxpYW5nLnh1QGNpbmZvdGVjaC5jbjxtYWlsdG86bGlhbmcueHVAY2luZm90ZWNoLmNuPj4N
Ci0tLQ0KbGliL2xpYnJ0ZV9lYWwvbGludXhhcHAvZWFsL2VhbF9wY2lfdWlvLmMgfCAyOSArKysr
KysrKysrKysrKysrKysrKysrKysrKysrLQ0KMSBmaWxlIGNoYW5nZWQsIDI4IGluc2VydGlvbnMo
KyksIDEgZGVsZXRpb24oLSkNCg0KZGlmZiAtLWdpdCBhL2xpYi9saWJydGVfZWFsL2xpbnV4YXBw
L2VhbC9lYWxfcGNpX3Vpby5jIGIvbGliL2xpYnJ0ZV9lYWwvbGludXhhcHAvZWFsL2VhbF9wY2lf
dWlvLmMNCmluZGV4IDdlNjIyNjYuLmEyYzlhYjYgMTAwNjQ0DQotLS0gYS9saWIvbGlicnRlX2Vh
bC9saW51eGFwcC9lYWwvZWFsX3BjaV91aW8uYw0KKysrIGIvbGliL2xpYnJ0ZV9lYWwvbGludXhh
cHAvZWFsL2VhbF9wY2lfdWlvLmMNCkBAIC0yNzMsNiArMjczLDI0IEBAIHBjaV9nZXRfdWlvX2Rl
dihzdHJ1Y3QgcnRlX3BjaV9kZXZpY2UgKmRldiwgY2hhciAqZHN0YnVmLA0KcmV0dXJuIHVpb19u
dW07DQp9DQoNCitzdGF0aWMgaW5saW5lIGNvbnN0IHN0cnVjdCBydGVfbWVtc2VnICoNCitnZXRf
cGh5c21lbV9sYXN0KHZvaWQpDQorew0KKyBjb25zdCBzdHJ1Y3QgcnRlX21lbXNlZyAqIHNlZyA9
IHJ0ZV9lYWxfZ2V0X3BoeXNtZW1fbGF5b3V0KCk7DQorIGNvbnN0IHN0cnVjdCBydGVfbWVtc2Vn
ICogbGFzdCA9IHNlZzsNCisgdW5zaWduZWQgaSA9IDA7DQorDQorIGZvciAoaT0wOyBpPFJURV9N
QVhfTUVNU0VHOyBpKyssIHNlZysrKSB7DQorIGlmIChzZWctPmFkZHIgPT0gTlVMTCkNCisgYnJl
YWs7DQorDQorIGlmKHNlZy0+YWRkciA+IGxhc3QtPmFkZHIpDQorIGxhc3QgPSBzZWc7DQorDQor
IH0NCisgcmV0dXJuIGxhc3Q7DQorfQ0KKw0KLyogbWFwIHRoZSBQQ0kgcmVzb3VyY2Ugb2YgYSBQ
Q0kgZGV2aWNlIGluIHZpcnR1YWwgbWVtb3J5ICovIGludCBwY2lfdWlvX21hcF9yZXNvdXJjZShz
dHJ1Y3QgcnRlX3BjaV9kZXZpY2UgKmRldikgQEAgLTI5MCw2ICszMDgsMTMgQEAgcGNpX3Vpb19t
YXBfcmVzb3VyY2Uoc3RydWN0IHJ0ZV9wY2lfZGV2aWNlICpkZXYpDQpzdHJ1Y3QgbWFwcGVkX3Bj
aV9yZXNvdXJjZSAqdWlvX3JlczsNCnN0cnVjdCBwY2lfbWFwICptYXBzOw0KDQorIC8qIG1hcCB1
aW8gcmVzb3VyY2UgaW50byB1c2VyIHJlcXVpcmVkIHZpcnR1YWwgYWRkcmVzcyAqLw0KKyBzdGF0
aWMgdm9pZCAqIHJlcXVlc3RlZF9hZGRyOw0KKyBpZiAoaW50ZXJuYWxfY29uZmlnLmJhc2Vfdmly
dGFkZHIgJiYgTlVMTCA9PSByZXF1ZXN0ZWRfYWRkcikgew0KKyBjb25zdCBzdHJ1Y3QgcnRlX21l
bXNlZyAqIGxhc3QgPSBnZXRfcGh5c21lbV9sYXN0KCk7DQorIHJlcXVlc3RlZF9hZGRyID0gUlRF
X1BUUl9BREQobGFzdC0+YWRkciwgbGFzdC0+bGVuKTsNCisgfQ0KKw0KZGV2LT5pbnRyX2hhbmRs
ZS5mZCA9IC0xOw0KZGV2LT5pbnRyX2hhbmRsZS50eXBlID0gUlRFX0lOVFJfSEFORExFX1VOS05P
V047DQoNCkBAIC0zNzEsMTAgKzM5NiwxMiBAQCBwY2lfdWlvX21hcF9yZXNvdXJjZShzdHJ1Y3Qg
cnRlX3BjaV9kZXZpY2UgKmRldikNCmlmIChtYXBzW2pdLmFkZHIgIT0gTlVMTCkNCmZhaWwgPSAx
Ow0KZWxzZSB7DQotIG1hcGFkZHIgPSBwY2lfbWFwX3Jlc291cmNlKE5VTEwsIGZkLCAob2ZmX3Qp
b2Zmc2V0LA0KKyBtYXBhZGRyID0gcGNpX21hcF9yZXNvdXJjZShyZXF1ZXN0ZWRfYWRkciwgZmQs
IChvZmZfdClvZmZzZXQsDQooc2l6ZV90KW1hcHNbal0uc2l6ZSk7DQppZiAobWFwYWRkciA9PSBO
VUxMKQ0KZmFpbCA9IDE7DQorIGVsc2UgaWYgKE5VTEwgIT0gcmVxdWVzdGVkX2FkZHIpDQorIHJl
cXVlc3RlZF9hZGRyID0gUlRFX1BUUl9BREQobWFwYWRkciwgbWFwc1tqXS5zaXplKTsNCn0NCg0K
aWYgKGZhaWwpIHsNCi0tDQoxLjkuMQ0K

Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

Because vfio and uio can be used at the same time, we need save the value of requested_addr at pci_vfio_map_resource() and pci_uio_map_resource().Your guys like add a global variable "void * pci_requested_addr" or a "void** requested_addr" parameter at pci_vfio_map_resource() and pci_uio_map_resource() ?------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov@intel.com>Time:2014 Nov 7 (Fri) 18 : 03To:徐亮 <liang.xu@cinfotech.cn>, dev@dpdk.org <dev@dpdk.org>Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.



font-family: Wingdings;panose-1: 5 0 0 0 0 0 0 0 0 0;font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0in;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
*.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
size: 8.5in 11.0in;margin: 1.0in 1.0in 1.0in 1.0in;div.WordSection1 {page: WordSection1;}




Ah, yes, sorry about that, haven’t quite woke up yet
L You’re right. So it’s removing the dependency on --base-virtaddr, moving the function to eal_pci.c
  and adding similar VFIO code that’s left then.
 
Thanks,
Anatoly
 
From: XU Liang [mailto:liang.xu@cinfotech.cn]


Sent: Friday, November 7, 2014 9:57 AM

To: Burakov, Anatoly; dev@dpdk.org

Cc: thomas.monjalon@6wind.com; De Lara Guarch, Pablo

Subject: Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
 


How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.


 


If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.


 


------------------------------------------------------------------



From:Burakov, Anatoly <anatoly.burakov@intel.com>


Time:2014 Nov 7 (Fri) 17 : 47


To:徐亮
 <liang.xu@cinfotech.cn>, 
dev@dpdk.org <dev@dpdk.org>


Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>


Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.


 

The commit message looks fine to me, but VFIO code needs to be adjusted the same way.



Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the
 last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).



My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).



My apologies for not thinking about any of this during the first 6 iterations of the patch :(



Thanks,

Anatoly



-----Original Message-----

From: lxu [mailto:liang.xu@cinfotech.cn]


Sent: Friday, November 7, 2014 8:01 AM

To: dev@dpdk.org

Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo

Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.



A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary
 process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If
 it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.



This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.



Signed-off-by: lxu <liang.xu@cinfotech.cn>

---

lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-

1 file changed, 28 insertions(+), 1 deletion(-)



diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

index 7e62266..a2c9ab6 100644

--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,

return uio_num;

}



+static inline const struct rte_memseg *

+get_physmem_last(void)

+{

+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();

+ const struct rte_memseg * last = seg;

+ unsigned i = 0;

+

+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {

+ if (seg->addr == NULL)

+ break;

+

+ if(seg->addr > last->addr)

+ last = seg;

+

+ }

+ return last;

+}

+

/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)

struct mapped_pci_resource *uio_res;

struct pci_map *maps;



+ /* map uio resource into user required virtual address */

+ static void * requested_addr;

+ if (internal_config.base_virtaddr && NULL == requested_addr) {

+ const struct rte_memseg * last = get_physmem_last();

+ requested_addr = RTE_PTR_ADD(last->addr, last->len);

+ }

+

dev->intr_handle.fd = -1;

dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;



@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)

if (maps[j].addr != NULL)

fail = 1;

else {

- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,

+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,

(size_t)maps[j].size);

if (mapaddr == NULL)

fail = 1;

+ else if (NULL != requested_addr)

+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);

}



if (fail) {

--

1.9.1



From bruce.richardson@intel.com  Fri Nov  7 15:34:52 2014
Return-Path: <bruce.richardson@intel.com>
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20])
 by dpdk.org (Postfix) with ESMTP id D90B92A9
 for <dev@dpdk.org>; Fri,  7 Nov 2014 15:34:51 +0100 (CET)
Received: from orsmga001.jf.intel.com ([10.7.209.18])
 by orsmga101.jf.intel.com with ESMTP; 07 Nov 2014 06:44:14 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,332,1413270000"; d="scan'208";a="604153305"
Received: from bricha3-mobl3.ger.corp.intel.com ([10.243.20.32])
 by orsmga001.jf.intel.com with SMTP; 07 Nov 2014 06:44:11 -0800
Received: by  (sSMTP sendmail emulation); Fri, 07 Nov 2014 14:44:10 +0025
Date: Fri, 7 Nov 2014 14:44:10 +0000
From: Bruce Richardson <bruce.richardson@intel.com>
To: jigsaw <jigsaw@gmail.com>
Message-ID: <20141107144410.GC12092@bricha3-MOBL3>
References: <20141106092228.GA3056@bricha3-MOBL3> <9190772.1rnKUO3oNV@xps13>
 <545b6b74.a96db40a.26af.ffffe7fb@mx.google.com>
 <20141106135951.GB7252@bricha3-MOBL3>
 <CAHVfvh4U4PZKZSue_kKDQKATC2snb_\x10OD08LGmUtieBc_LzA@mail.gmail.com>
 <CAHVfvh5SzJ-kpQQ9h=1wmMihiitcJXeR9mcNa1in8x6Gb6tSqQ@mail.gmail.com>
 <20141107094521.GB4628@bricha3-MOBL3>
 <CAHVfvh6y4f7+bMhzmwOu5c0Y4wzwNaxj4sQPtq8cabGbdHrzXg@mail.gmail.com>
 <20141107135303.GB12092@bricha3-MOBL3>
 <CAHVfvh7ggGB_q1Rs1c3-9PRwDr_GKA+etaMXRSeKCfUKoUx8hQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAHVfvh7ggGB_q1Rs1c3-9PRwDr_GKA+etaMXRSeKCfUKoUx8hQ@mail.gmail.com>
Organization: Intel Shannon Ltd.
User-Agent: Mutt/1.5.23 (2014-03-12)
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev]
 =?utf-8?b?562U5aSNOiAgW1BBVENIXSBBZGQgdXNlciBkZWZpbmVk? =?utf-8?q?_tag_calculation_callback_tolibrte_distributor.?X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Nov 2014 14:34:52 -0000

On Fri, Nov 07, 2014 at 04:31:18PM +0200, jigsaw wrote:
> Hi Bruce,
>
> Pls have a quick look at the diff to see if this is exactly what you mean
> about the bitmask.
> I just wrote it without even compiling, just to express the idea. So it may
> leave some places unpatched.
> If this is agreed, I will make a decent test to verify it before sending
> the patch for RFC.
>
> diff --git a/lib/librte_distributor/rte_distributor.c
> b/lib/librte_distributor/rte_di
> index 585ff88..d606bcf 100644
> --- a/lib/librte_distributor/rte_distributor.c
> +++ b/lib/librte_distributor/rte_distributor.c
> @@ -92,6 +92,8 @@ struct rte_distributor {
>         unsigned num_workers;                 /**< Number of workers
> polling */
>
>         uint32_t in_flight_tags[RTE_MAX_LCORE];
> +       uint32_t in_flight_bitmask;
> +
>         struct rte_distributor_backlog backlog[RTE_MAX_LCORE];
>
>         union rte_distributor_buffer bufs[RTE_MAX_LCORE];
> @@ -188,6 +190,7 @@ static inline void
>  handle_worker_shutdown(struct rte_distributor *d, unsigned wkr)
>  {
>         d->in_flight_tags[wkr] = 0;
> +       d->in_flight_mask &= ~(1 << wkr);
>         d->bufs[wkr].bufptr64 = 0;
>         if (unlikely(d->backlog[wkr].count != 0)) {
>                 /* On return of a packet, we need to move the
> @@ -241,6 +244,7 @@ process_returns(struct rte_distributor *d)
>                         else {
>                                 d->bufs[wkr].bufptr64 = RTE_DISTRIB_GET_BUF;
>                                 d->in_flight_tags[wkr] = 0;
> +                               d->in_flight_mask &= ~(1 << wkr);
>                         }
>                         oldbuf = data >> RTE_DISTRIB_FLAG_BITS;
>                 } else if (data & RTE_DISTRIB_RETURN_BUF) {
> @@ -282,12 +286,13 @@ rte_distributor_process(struct rte_distributor *d,
>                         next_mb = mbufs[next_idx++];
>                         next_value = (((int64_t)(uintptr_t)next_mb)
>                                         << RTE_DISTRIB_FLAG_BITS);
> -                       new_tag = (next_mb->hash.rss | 1);
> +                       new_tag = next_mb->hash.rss;
>
>                         uint32_t match = 0;
>                         unsigned i;
>                         for (i = 0; i < d->num_workers; i++)
> -                               match |= (!(d->in_flight_tags[i] ^ new_tag)
> +                               match |= (((!(d->in_flight_tags[i] ^
> new_tag)) &
> +                                               (d->in_flight_bitmask >> i))

I would not do the bitmask comparison here, as that's extra instruction in the
loop. Instead, because its a bitmask, build up the match variable as it was
before, and then just do a single and operation afterwards, outside the loop
body.

/Bruce

>                                         << i);
>
>                         if (match) {
> @@ -309,6 +314,7 @@ rte_distributor_process(struct rte_distributor *d,
>                         else {
>                                 d->bufs[wkr].bufptr64 = next_value;
>                                 d->in_flight_tags[wkr] = new_tag;
> +                               d->in_flight_bitmask |= 1 << wkr;
>                                 next_mb = NULL;
>                         }
>                         oldbuf = data >> RTE_DISTRIB_FLAG_BITS;
>
>
>

[dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages

[Anatoly Burakov]

Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 19 +++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  9 ++++++++-
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 13 +++++++++++--
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 ++++++
 4 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
 		if (bar_addr == NULL) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4

Re: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages

[Burakov, Anatoly]

Nak, there are issues with the patch. There is another patch already, but I'll submit it whenever Liang verifies it works with his setup.

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov
Sent: Monday, November 10, 2014 11:35 AM
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages

Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 19 +++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  9 ++++++++-
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 13 +++++++++++--
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 ++++++
 4 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
 		if (bar_addr == NULL) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4

Re: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages

[XU Liang]

I had finished some tests. The patch works fine. My tests are included :* single process  + uio + vfio * single process  + uio + vfio + base-virtaddr * multiple processes + uio + vfio * multiple processes + uio + vfio + base-virtaddr My unlucky multiple process application still got error without base-virtaddr when initial hugepages. See the attchments: primary.txt and secondary.txt.With base-virtaddr the patch worked, both hugepages and pci resources were mapped into base-virtaddr, My application is happy. See the attchments: base-virtaddr_primary.txt and  base-virtaddr_secondary.txt. ------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov@intel.com>Time:2014 Nov 10 (Mon) 21 : 34To:Burakov, Anatoly <anatoly.burakov@intel.com>, dev@dpdk.org <dev@dpdk.org>Subject:Re: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages
Nak, there are issues with the patch. There is another patch already, but I'll submit it whenever Liang verifies it works with his setup.

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov
Sent: Monday, November 10, 2014 11:35 AM
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages

Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 19 +++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  9 ++++++++-
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 13 +++++++++++--
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 ++++++
 4 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
 		if (bar_addr == NULL) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4
From jijiang.liu@intel.com  Tue Nov 11 06:21:09 2014
Return-Path: <jijiang.liu@intel.com>
Received: from mga11.intel.com (mga11.intel.com [192.55.52.93])
 by dpdk.org (Postfix) with ESMTP id 5836C7F0D
 for <dev@dpdk.org>; Tue, 11 Nov 2014 06:21:09 +0100 (CET)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
 by fmsmga102.fm.intel.com with ESMTP; 10 Nov 2014 21:30:56 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,358,1413270000"; d="scan'208";a="620365999"
Received: from kmsmsx153.gar.corp.intel.com ([172.21.73.88])
 by fmsmga001.fm.intel.com with ESMTP; 10 Nov 2014 21:30:55 -0800
Received: from shsmsx104.ccr.corp.intel.com (10.239.110.15) by
 KMSMSX153.gar.corp.intel.com (172.21.73.88) with Microsoft SMTP Server (TLS)
 id 14.3.195.1; Tue, 11 Nov 2014 13:29:19 +0800
Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.130]) by
 SHSMSX104.ccr.corp.intel.com ([169.254.5.62]) with mapi id 14.03.0195.001;
 Tue, 11 Nov 2014 13:29:19 +0800
From: "Liu, Jijiang" <jijiang.liu@intel.com>
To: Olivier MATZ <olivier.matz@6wind.com>
Thread-Topic: [dpdk-dev] [PATCH v8 10/10] app/testpmd:test VxLAN Tx checksum
 offload
Thread-Index: AQHP8Yux0VUY397yik2xjVM3DjesSpxPpsiAgAGaY5CAABv8AIAIEl8ggAAq4wCAAVxhEA=Date: Tue, 11 Nov 2014 05:29:18 +0000
Message-ID: <1ED644BD7E0A5F4091CF203DAFB8E4CC01D8F7A7@SHSMSX101.ccr.corp.intel.com>
References: <1414376006-31402-1-git-send-email-jijiang.liu@intel.com>
 <1414376006-31402-11-git-send-email-jijiang.liu@intel.com>
 <54588BF7.309@6wind.com>
 <1ED644BD7E0A5F4091CF203DAFB8E4CC01D8510E@SHSMSX101.ccr.corp.intel.com>
 <5459FBB2.1040408@6wind.com>
 <1ED644BD7E0A5F4091CF203DAFB8E4CC01D8F399@SHSMSX101.ccr.corp.intel.com>
 <5460E512.1050609@6wind.com>
In-Reply-To: <5460E512.1050609@6wind.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.239.127.40]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v8 10/10] app/testpmd:test VxLAN Tx checksum
 offload
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 11 Nov 2014 05:21:09 -0000



> -----Original Message-----
> From: Olivier MATZ [mailto:olivier.matz@6wind.com]
> Sent: Tuesday, November 11, 2014 12:17 AM
> To: Liu, Jijiang
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v8 10/10] app/testpmd:test VxLAN Tx checksum
> offload
> 
> Hi Jijiang,
> 
> On 11/10/2014 07:03 AM, Liu, Jijiang wrote:
> >> Another thing is surprising me.
> >>
> >> - if PKT_TX_VXLAN_CKSUM is not set (legacy use case), then the
> >>    driver use l2_len and l3_len to offload inner IP/UDP/TCP checksums.
> > If the flag is not set, and imply that it is not VXLAN packet,  and do
> > TX checksum offload as regular packet.
> >
> >> - if PKT_TX_VXLAN_CKSUM is set, then the driver has to use
> >>    inner_l{23}_len instead of l{23}_len for the same operation.
> > Your understanding is not fully correct.
> > The l{23}_len is still used for TX checksum offload, please refer to
> i40e_txd_enable_checksum()  implementation.
> 
> This fields are part of public mbuf API. You cannot say to refer to i40e PMD code
> to understand how to use it.
> 
> >> Adding PKT_TX_VXLAN_CKSUM changes the semantic of l2_len and l3_len.
> >> To fix this, I suggest to remove the new fields inner_l{23}_len then
> >> add outer_l{23}_len instead. Therefore, the semantic of l2_len and
> >> l3_len would not change, and a driver would always use the same field for a
> specific offload.
> > Oh...
> 
> Does it mean you agree?

I don't agree to change inner_l{23}_len the name.
The reason is that using the "inner" word means  incoming  packet is tunneling packet or encapsulation packet.
if we add  "outer"{2,3}_len  , which will cause confusion when processing non-tunneling packet.


> >> For my TSO development, I will follow the current semantic.
> > For TSO, you still can use l{2,3} _len .
> > When I develop tunneling TSO, I will use inner_l3_len/inner_l4_len.
> 
> I've just submitted a first version, please feel free to comment it.
> 
> 
> Regards,
> Olivier

[dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

[Anatoly Burakov]

Multi-process DPDK application must mmap hugepages and PCI resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL try and map PCI BARs right after the hugepages
(instead of location chosen by mmap) in virtual memory, so that PCI BARs
have less chance of ending up in random places in virtual memory.

Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
 4 files changed, 55 insertions(+), 13 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..79fbbb8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
@@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
 	/* Map the PCI memory resource of device */
 	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
 			MAP_SHARED, fd, offset);
-	if (mapaddr == MAP_FAILED ||
-			(requested_addr != NULL && mapaddr != requested_addr)) {
+	if (mapaddr == MAP_FAILED) {
 		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
 			__func__, fd, requested_addr,
 			(unsigned long)size, (unsigned long)offset,
 			strerror(errno), mapaddr);
-		goto fail;
+	} else {
+		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
 	}
 
-	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
-
 	return mapaddr;
-
-fail:
-	return NULL;
 }
 
 /* parse the "resource" sysfs file */
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e53f06b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -35,6 +35,7 @@
 #include <fcntl.h>
 #include <dirent.h>
 #include <sys/stat.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -48,6 +49,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				/* try mapping somewhere close to the end of hugepages */
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
-				if (mapaddr == NULL)
+				if (mapaddr == MAP_FAILED)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..c1246e8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -37,6 +37,7 @@
 #include <sys/eventfd.h>
 #include <sys/socket.h>
 #include <sys/ioctl.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			/* try mapping somewhere close to the end of hugepages */
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
-		if (bar_addr == NULL) {
+		if (bar_addr == MAP_FAILED ||
+				(internal_config.process_type == RTE_PROC_SECONDARY &&
+						bar_addr != maps[i].addr)) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
 					strerror(errno));
 			close(vfio_dev_fd);
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4

Re: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

[Burakov, Anatoly]

Hi Thomas and all

Are there any objections to this patch? If there are no objections to it, could someone perhaps ack it?

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov
Sent: Tuesday, November 11, 2014 10:09 AM
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

Multi-process DPDK application must mmap hugepages and PCI resources into the same virtual address space. By default the virtual addresses are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary process - for example, secondary process is linked with more libraries than primary process, and the library occupies the same address space that the primary process has requested for PCI mappings.

This patch makes EAL try and map PCI BARs right after the hugepages (instead of location chosen by mmap) in virtual memory, so that PCI BARs have less chance of ending up in random places in virtual memory.

Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
 4 files changed, 55 insertions(+), 13 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..79fbbb8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len); }
+
+
 /* map a particular resource from a file */  void *  pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size) @@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
 	/* Map the PCI memory resource of device */
 	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
 			MAP_SHARED, fd, offset);
-	if (mapaddr == MAP_FAILED ||
-			(requested_addr != NULL && mapaddr != requested_addr)) {
+	if (mapaddr == MAP_FAILED) {
 		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
 			__func__, fd, requested_addr,
 			(unsigned long)size, (unsigned long)offset,
 			strerror(errno), mapaddr);
-		goto fail;
+	} else {
+		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
 	}
 
-	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
-
 	return mapaddr;
-
-fail:
-	return NULL;
 }
 
 /* parse the "resource" sysfs file */
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e53f06b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -35,6 +35,7 @@
 #include <fcntl.h>
 #include <dirent.h>
 #include <sys/stat.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -48,6 +49,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				/* try mapping somewhere close to the end of hugepages */
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
-				if (mapaddr == NULL)
+				if (mapaddr == MAP_FAILED)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..c1246e8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -37,6 +37,7 @@
 #include <sys/eventfd.h>
 #include <sys/socket.h>
 #include <sys/ioctl.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			/* try mapping somewhere close to the end of hugepages */
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
-		if (bar_addr == NULL) {
+		if (bar_addr == MAP_FAILED ||
+				(internal_config.process_type == RTE_PROC_SECONDARY &&
+						bar_addr != maps[i].addr)) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
 					strerror(errno));
 			close(vfio_dev_fd);
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {  TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);  extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in 
+virtual memory  */ extern void *pci_map_addr; void 
+*pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
--
1.8.1.4

Re: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

[Bruce Richardson]

On Thu, Nov 13, 2014 at 11:34:22AM +0000, Burakov, Anatoly wrote:
> Hi Thomas and all
> 
> Are there any objections to this patch? If there are no objections to it, could someone perhaps ack it?
> 

No objections, just a quick request for clarification below.

/Bruce

> Thanks,
> Anatoly
> 
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov
> Sent: Tuesday, November 11, 2014 10:09 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages
> 
> Multi-process DPDK application must mmap hugepages and PCI resources into the same virtual address space. By default the virtual addresses are chosen by the primary process automatically when calling the mmap.
> But sometimes the chosen virtual addresses aren't usable in secondary process - for example, secondary process is linked with more libraries than primary process, and the library occupies the same address space that the primary process has requested for PCI mappings.
> 
> This patch makes EAL try and map PCI BARs right after the hugepages (instead of location chosen by mmap) in virtual memory, so that PCI BARs have less chance of ending up in random places in virtual memory.
> 
> Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
>  lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
>  4 files changed, 55 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
> index 5fe3961..79fbbb8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
> @@ -97,6 +97,25 @@ error:
>  	return -1;
>  }
>  
> +void *
> +pci_find_max_end_va(void)
> +{
> +	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg *last = seg;
> +	unsigned i = 0;
> +
> +	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if (seg->addr > last->addr)
> +			last = seg;
> +
> +	}
> +	return RTE_PTR_ADD(last->addr, last->len); }
> +
> +
>  /* map a particular resource from a file */  void *  pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size) @@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
>  	/* Map the PCI memory resource of device */
>  	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
>  			MAP_SHARED, fd, offset);
> -	if (mapaddr == MAP_FAILED ||
> -			(requested_addr != NULL && mapaddr != requested_addr)) {

Why has this check been removed from here. I assume it is replaced by a new
check in secondary processes that I see added below, but perhaps you could explain
the reason for the change?

> +	if (mapaddr == MAP_FAILED) {
>  		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
>  			__func__, fd, requested_addr,
>  			(unsigned long)size, (unsigned long)offset,
>  			strerror(errno), mapaddr);
> -		goto fail;
> +	} else {
> +		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
>  	}
>  
> -	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
> -
>  	return mapaddr;
> -
> -fail:
> -	return NULL;
>  }
>  
>  /* parse the "resource" sysfs file */
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..e53f06b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -35,6 +35,7 @@
>  #include <fcntl.h>
>  #include <dirent.h>
>  #include <sys/stat.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -48,6 +49,8 @@
>  
>  static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
>  
> +void *pci_map_addr = NULL;
> +
>  
>  #define OFF_MAX              ((uint64_t)(off_t)-1)
>  static int
> @@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  			if (maps[j].addr != NULL)
>  				fail = 1;
>  			else {
> -				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
> +				/* try mapping somewhere close to the end of hugepages */
> +				if (pci_map_addr == NULL)
> +					pci_map_addr = pci_find_max_end_va();
> +
> +				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
>  						(size_t)maps[j].size);
> -				if (mapaddr == NULL)
> +				if (mapaddr == MAP_FAILED)
>  					fail = 1;
> +
> +				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
>  			}
>  
>  			if (fail) {
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> index c776ddc..c1246e8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> @@ -37,6 +37,7 @@
>  #include <sys/eventfd.h>
>  #include <sys/socket.h>
>  #include <sys/ioctl.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
>  		if (i == msix_bar)
>  			continue;
>  
> -		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> -				reg.size);
> +		if (internal_config.process_type == RTE_PROC_PRIMARY) {
> +			/* try mapping somewhere close to the end of hugepages */
> +			if (pci_map_addr == NULL)
> +				pci_map_addr = pci_find_max_end_va();
> +
> +			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
> +		} else {
> +			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +		}
>  
> -		if (bar_addr == NULL) {
> +		if (bar_addr == MAP_FAILED ||
> +				(internal_config.process_type == RTE_PROC_SECONDARY &&
> +						bar_addr != maps[i].addr)) {
>  			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
>  					strerror(errno));
>  			close(vfio_dev_fd);
> diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> index d758bee..1070eb8 100644
> --- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> +++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> @@ -59,6 +59,12 @@ struct mapped_pci_resource {  TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);  extern struct mapped_pci_res_list *pci_res_list;
>  
> +/*
> + * Helper function to map PCI resources right after hugepages in 
> +virtual memory  */ extern void *pci_map_addr; void 
> +*pci_find_max_end_va(void);
> +
>  void *pci_map_resource(void *requested_addr, int fd, off_t offset,
>  		size_t size);
>  
> --
> 1.8.1.4
> 

Re: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

[Burakov, Anatoly]

> Why has this check been removed from here. I assume it is replaced by a
> new check in secondary processes that I see added below, but perhaps you
> could explain the reason for the change?

Sure. The reason behind that change is that we can't expect that we will get a mapping at exact same address (for whatever reasons, i.e. something else is mapped there, alignment, etc.), and in primary process, it's not an error. In other words, removing this check makes it a "best-effort" type mechanism, rather than mandates PCI resources to be mapped exactly after hugepages, exactly one after another. "Wrong" mapping will still result in failure in secondary processes, and we still are risking mapping something somewhere the secondary process can't map, but that probability is decreased because we're now asking EAL to map PCI resources closer to where we most likely have some free virtual space (as evidenced by tests being done by the original submitter of the patch).

Hope that makes sense.

Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

[Bruce Richardson]

On Tue, Nov 11, 2014 at 10:09:25AM +0000, Anatoly Burakov wrote:
> Multi-process DPDK application must mmap hugepages and PCI resources
> into the same virtual address space. By default the virtual addresses
> are chosen by the primary process automatically when calling the mmap.
> But sometimes the chosen virtual addresses aren't usable in secondary
> process - for example, secondary process is linked with more libraries
> than primary process, and the library occupies the same address space
> that the primary process has requested for PCI mappings.
> 
> This patch makes EAL try and map PCI BARs right after the hugepages
> (instead of location chosen by mmap) in virtual memory, so that PCI BARs
> have less chance of ending up in random places in virtual memory.
> 
> Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

> ---
>  lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
>  lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
>  4 files changed, 55 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
> index 5fe3961..79fbbb8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
> @@ -97,6 +97,25 @@ error:
>  	return -1;
>  }
>  
> +void *
> +pci_find_max_end_va(void)
> +{
> +	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg *last = seg;
> +	unsigned i = 0;
> +
> +	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if (seg->addr > last->addr)
> +			last = seg;
> +
> +	}
> +	return RTE_PTR_ADD(last->addr, last->len);
> +}
> +
> +
>  /* map a particular resource from a file */
>  void *
>  pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
> @@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
>  	/* Map the PCI memory resource of device */
>  	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
>  			MAP_SHARED, fd, offset);
> -	if (mapaddr == MAP_FAILED ||
> -			(requested_addr != NULL && mapaddr != requested_addr)) {
> +	if (mapaddr == MAP_FAILED) {
>  		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
>  			__func__, fd, requested_addr,
>  			(unsigned long)size, (unsigned long)offset,
>  			strerror(errno), mapaddr);
> -		goto fail;
> +	} else {
> +		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
>  	}
>  
> -	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
> -
>  	return mapaddr;
> -
> -fail:
> -	return NULL;
>  }
>  
>  /* parse the "resource" sysfs file */
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..e53f06b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -35,6 +35,7 @@
>  #include <fcntl.h>
>  #include <dirent.h>
>  #include <sys/stat.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -48,6 +49,8 @@
>  
>  static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
>  
> +void *pci_map_addr = NULL;
> +
>  
>  #define OFF_MAX              ((uint64_t)(off_t)-1)
>  static int
> @@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  			if (maps[j].addr != NULL)
>  				fail = 1;
>  			else {
> -				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
> +				/* try mapping somewhere close to the end of hugepages */
> +				if (pci_map_addr == NULL)
> +					pci_map_addr = pci_find_max_end_va();
> +
> +				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
>  						(size_t)maps[j].size);
> -				if (mapaddr == NULL)
> +				if (mapaddr == MAP_FAILED)
>  					fail = 1;
> +
> +				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
>  			}
>  
>  			if (fail) {
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> index c776ddc..c1246e8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> @@ -37,6 +37,7 @@
>  #include <sys/eventfd.h>
>  #include <sys/socket.h>
>  #include <sys/ioctl.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
>  		if (i == msix_bar)
>  			continue;
>  
> -		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> -				reg.size);
> +		if (internal_config.process_type == RTE_PROC_PRIMARY) {
> +			/* try mapping somewhere close to the end of hugepages */
> +			if (pci_map_addr == NULL)
> +				pci_map_addr = pci_find_max_end_va();
> +
> +			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
> +		} else {
> +			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +		}
>  
> -		if (bar_addr == NULL) {
> +		if (bar_addr == MAP_FAILED ||
> +				(internal_config.process_type == RTE_PROC_SECONDARY &&
> +						bar_addr != maps[i].addr)) {
>  			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
>  					strerror(errno));
>  			close(vfio_dev_fd);
> diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> index d758bee..1070eb8 100644
> --- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> +++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> @@ -59,6 +59,12 @@ struct mapped_pci_resource {
>  TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
>  extern struct mapped_pci_res_list *pci_res_list;
>  
> +/*
> + * Helper function to map PCI resources right after hugepages in virtual memory
> + */
> +extern void *pci_map_addr;
> +void *pci_find_max_end_va(void);
> +
>  void *pci_map_resource(void *requested_addr, int fd, off_t offset,
>  		size_t size);
>  
> -- 
> 1.8.1.4
> 

Re: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

[Thomas Monjalon]

> > Multi-process DPDK application must mmap hugepages and PCI resources
> > into the same virtual address space. By default the virtual addresses
> > are chosen by the primary process automatically when calling the mmap.
> > But sometimes the chosen virtual addresses aren't usable in secondary
> > process - for example, secondary process is linked with more libraries
> > than primary process, and the library occupies the same address space
> > that the primary process has requested for PCI mappings.
> > 
> > This patch makes EAL try and map PCI BARs right after the hugepages
> > (instead of location chosen by mmap) in virtual memory, so that PCI BARs
> > have less chance of ending up in random places in virtual memory.
> > 
> > Signed-off-by: Liang Xu <liang.xu@cinfotech.cn>
> > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Applied

Thanks
-- 
Thomas

[dpdk-dev] [PATCH v7] eal: map uio resources after hugepages.

[lxu]

A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.

This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 25 ++++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  6 ++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  4 +++-
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  4 ++--
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)
 }
 #endif
 
+static void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
 static int
 pci_map_device(struct rte_pci_device *dev)
 {
 	int ret, mapped = 0;
+	static void * requested_addr;
+	if(NULL == requested_addr)
+		requested_addr = pci_find_max_end_va();
 
 	/* try mapping the NIC resources using VFIO if it exists */
 #ifdef VFIO_PRESENT
 	if (pci_vfio_is_enabled()) {
-		ret = pci_vfio_map_resource(dev);
+		ret = pci_vfio_map_resource(dev, &requested_addr);
 		if (ret == 0)
 			mapped = 1;
 		else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)
 #endif
 	/* map resources for devices that use igb_uio */
 	if (!mapped) {
-		ret = pci_uio_map_resource(dev);
+		ret = pci_uio_map_resource(dev, &requested_addr);
 		if (ret != 0)
 			return ret;
 	}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 
 /* map the PCI resource of a PCI device in virtual memory */
 int
-pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	int i, j;
 	char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else
+					*requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
  * primary and secondary processes follow almost exactly the same path
  */
 int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	struct vfio_group_status group_status = {
 			.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
+		maps[i].addr = *requested_addr;
 		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
 				reg.size);
 
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 			return -1;
 		}
 
+		*requested_addr = bar_addr;
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg.offset;
 		maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
 /* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr);
 
 #ifdef VFIO_PRESENT
 
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);
 int pci_vfio_mp_sync_setup(void);
 
 /* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void **requested_addr);
 int pci_vfio_get_group_fd(int iommu_group_fd);
 int pci_vfio_get_container_fd(void);
 
-- 
1.9.1

Re: [dpdk-dev] [PATCH v7] eal: map uio resources after hugepages.

[Burakov, Anatoly]

Um, not sure I agree with this implementation. I think a cleaner way would be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resource (or rather, put it in include/eal_pci_init.h, like extern void *requested_addr) but make actual use of it in pci_uio/vfio_map_resource only (and leave all of this out of eal_pci.c at all). That will also rid you of the necessity to pass around pointers to pointers.

(in that case I would also rename requested_addr to pci_map_addr or something, to make it less vague) 

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Friday, November 7, 2014 2:58 PM
To: dev@dpdk.org
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: [PATCH v7] eal: map uio resources after hugepages.

A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.

This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 25 ++++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  6 ++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  4 +++-
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  4 ++--
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)  }  #endif
 
+static void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len); }
+
 static int
 pci_map_device(struct rte_pci_device *dev)  {
 	int ret, mapped = 0;
+	static void * requested_addr;
+	if(NULL == requested_addr)
+		requested_addr = pci_find_max_end_va();
 
 	/* try mapping the NIC resources using VFIO if it exists */  #ifdef VFIO_PRESENT
 	if (pci_vfio_is_enabled()) {
-		ret = pci_vfio_map_resource(dev);
+		ret = pci_vfio_map_resource(dev, &requested_addr);
 		if (ret == 0)
 			mapped = 1;
 		else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)  #endif
 	/* map resources for devices that use igb_uio */
 	if (!mapped) {
-		ret = pci_uio_map_resource(dev);
+		ret = pci_uio_map_resource(dev, &requested_addr);
 		if (ret != 0)
 			return ret;
 	}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 
 /* map the PCI resource of a PCI device in virtual memory */  int -pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	int i, j;
 	char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else
+					*requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
  * primary and secondary processes follow almost exactly the same path
  */
 int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr)
 {
 	struct vfio_group_status group_status = {
 			.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
+		maps[i].addr = *requested_addr;
 		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
 				reg.size);
 
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 			return -1;
 		}
 
+		*requested_addr = bar_addr;
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg.offset;
 		maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
 /* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 
 #ifdef VFIO_PRESENT
 
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);  int pci_vfio_mp_sync_setup(void);
 
 /* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 int pci_vfio_get_group_fd(int iommu_group_fd);  int pci_vfio_get_container_fd(void);
 
--
1.9.1

Re: [dpdk-dev] [PATCH v7] eal: map uio resources after hugepages.

[Thomas Monjalon]

Hi Liang,

I see you are learning how to send a patch. It's a good thing.
I don't know if the idea of your patch will be accepted but here are more tips
about formatting:
- You should use checkpatch.pl from kernel.org and fix the most important issues
- The commit log should be wrapped to avoid long lines
- The title is important. You didn't update it for vfio.
And there should be no point at the end of the title.
- You should use your real name in Signed-off-by.
- You should annotate each new version to show the changes

See http://dpdk.org/dev#send and 
https://www.kernel.org/doc/Documentation/SubmittingPatches

Looking at other submissions could be inspiring also.

Thanks
-- 
Thomas

Re: [dpdk-dev] [PATCH v7] eal: map uio resources after hugepages.

[XU Liang]

I got it. Just some guys don't like global variable. I'm not sure what is DPDK code style.------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov@intel.com>Time:2014 Nov 7 (Fri) 23 : 15To:徐亮 <liang.xu@cinfotech.cn>, dev@dpdk.org <dev@dpdk.org>Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>Subject:RE: [PATCH v7] eal: map uio resources after hugepages.
Um, not sure I agree with this implementation. I think a cleaner way would be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resource (or rather, put it in include/eal_pci_init.h, like extern void *requested_addr) but make actual use of it in pci_uio/vfio_map_resource only (and leave all of this out of eal_pci.c at all). That will also rid you of the necessity to pass around pointers to pointers.

(in that case I would also rename requested_addr to pci_map_addr or something, to make it less vague) 

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Friday, November 7, 2014 2:58 PM
To: dev@dpdk.org
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: [PATCH v7] eal: map uio resources after hugepages.

A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.

This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 25 ++++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  6 ++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  4 +++-
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  4 ++--
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)  }  #endif
 
+static void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len); }
+
 static int
 pci_map_device(struct rte_pci_device *dev)  {
 	int ret, mapped = 0;
+	static void * requested_addr;
+	if(NULL == requested_addr)
+		requested_addr = pci_find_max_end_va();
 
 	/* try mapping the NIC resources using VFIO if it exists */  #ifdef VFIO_PRESENT
 	if (pci_vfio_is_enabled()) {
-		ret = pci_vfio_map_resource(dev);
+		ret = pci_vfio_map_resource(dev, &requested_addr);
 		if (ret == 0)
 			mapped = 1;
 		else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)  #endif
 	/* map resources for devices that use igb_uio */
 	if (!mapped) {
-		ret = pci_uio_map_resource(dev);
+		ret = pci_uio_map_resource(dev, &requested_addr);
 		if (ret != 0)
 			return ret;
 	}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 
 /* map the PCI resource of a PCI device in virtual memory */  int -pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	int i, j;
 	char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else
+					*requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
  * primary and secondary processes follow almost exactly the same path
  */
 int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr)
 {
 	struct vfio_group_status group_status = {
 			.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
+		maps[i].addr = *requested_addr;
 		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
 				reg.size);
 
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 			return -1;
 		}
 
+		*requested_addr = bar_addr;
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg.offset;
 		maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
 /* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 
 #ifdef VFIO_PRESENT
 
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);  int pci_vfio_mp_sync_setup(void);
 
 /* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 int pci_vfio_get_group_fd(int iommu_group_fd);  int pci_vfio_get_container_fd(void);
 
--
1.9.1
From bruce.richardson@intel.com  Fri Nov  7 16:21:25 2014
Return-Path: <bruce.richardson@intel.com>
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
 by dpdk.org (Postfix) with ESMTP id 0ACF67F29
 for <dev@dpdk.org>; Fri,  7 Nov 2014 16:21:24 +0100 (CET)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
 by fmsmga101.fm.intel.com with ESMTP; 07 Nov 2014 07:30:56 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,333,1413270000"; d="scan'208";a="618893192"
Received: from gcgray-mobl3.ger.corp.intel.com ([10.252.29.251])
 by fmsmga001.fm.intel.com with SMTP; 07 Nov 2014 07:30:54 -0800
Received: by  (sSMTP sendmail emulation); Fri, 07 Nov 2014 15:30:53 +0025
Date: Fri, 7 Nov 2014 15:30:53 +0000
From: Bruce Richardson <bruce.richardson@intel.com>
To: Manoj Viswanath <manoj.viswanath@gmail.com>
Message-ID: <20141107153053.GA10376@bricha3-MOBL3>
References: <CAC1b25pX3R2y_Cjp5UAdL6Bozrf01OkG6U6LGQvJ2aUTDXGvSA@mail.gmail.com>
 <20141105101246.GA9856@bricha3-MOBL3>
 <CAC1b25pY-wbEeVicBEVhYChOqjWXNRHO51wQwc8q4uo28DGO2g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAC1b25pY-wbEeVicBEVhYChOqjWXNRHO51wQwc8q4uo28DGO2g@mail.gmail.com>
Organization: Intel Shannon Ltd.
User-Agent: Mutt/1.5.23 (2014-03-12)
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Ports not detected by IGB_UIO in DPDK 1.7.1 in
 QEMU_KVM environment
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Nov 2014 15:21:26 -0000

On Fri, Nov 07, 2014 at 08:31:34PM +0530, Manoj Viswanath wrote:
> Hi Bruce,
>
> I was not doing anything specific for binding the NICs to IGB_UIO (like
> invoking "dpdk_nic_bind.py" script explicitly) when using my application
> with DPDK 1.6.0. The e1000 devices assigned via virt-manager to the VM were
> automatically getting picked up and initialized by IGB_UIO within each VM.
>
> The same is not working with DPDK 1.7.1 now.
>
> I tried exporting the "dpdk_nic_bind.py" script into my VM (running DPDK
> 1.7.1) and tried to check the status. The emulated devices were shown as
> neither bound to kernel nor to IGB_UIO as evident from below output:-
>
> <--------------------------------------------------------------------------------------------------->
> Network devices using DPDK-compatible driver
> ===========================================> <none>
>
> Network devices using kernel driver
> ==================================> 0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio
>
> Other network devices
> ====================> 0000:00:04.0 '82540EM Gigabit Ethernet Controller' unused=igb_uio
> 0000:00:05.0 '82540EM Gigabit Ethernet Controller' unused=igb_uio
> <--------------------------------------------------------------------------------------------------->
>
> When i tried to forcefully bind the NICs using the "--bind=igb_uio" option

Was there any output of the dpdk_nic_bind script? What does the output of
it with --status show afterwards?

Regards,
/Bruce