DPDK patches and discussions
 help / color / mirror / Atom feed
From: "XU Liang" <liang.xu@cinfotech.cn>
To: "Burakov, Anatoly" <anatoly.burakov@intel.com>,
	"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v7] eal: map uio resources after hugepages.
Date: Fri, 07 Nov 2014 23:19:26 +0800
Message-ID: <f5cf68fd-528b-4eaf-b9c0-2380d426928e@cinfotech.cn> (raw)
In-Reply-To: C6ECDF3AB251BE4894318F4E4512369780C07A7E@IRSMSX109.ger.corp.intel.com

I got it. Just some guys don't like global variable. I'm not sure what is DPDK code style.------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov@intel.com>Time:2014 Nov 7 (Fri) 23 : 15To:徐亮 <liang.xu@cinfotech.cn>, dev@dpdk.org <dev@dpdk.org>Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>Subject:RE: [PATCH v7] eal: map uio resources after hugepages.
Um, not sure I agree with this implementation. I think a cleaner way would be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resource (or rather, put it in include/eal_pci_init.h, like extern void *requested_addr) but make actual use of it in pci_uio/vfio_map_resource only (and leave all of this out of eal_pci.c at all). That will also rid you of the necessity to pass around pointers to pointers.

(in that case I would also rename requested_addr to pci_map_addr or something, to make it less vague) 

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn] 
Sent: Friday, November 7, 2014 2:58 PM
To: dev@dpdk.org
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: [PATCH v7] eal: map uio resources after hugepages.

A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.

This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 25 ++++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  6 ++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  4 +++-
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  4 ++--
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)  }  #endif
 
+static void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len); }
+
 static int
 pci_map_device(struct rte_pci_device *dev)  {
 	int ret, mapped = 0;
+	static void * requested_addr;
+	if(NULL == requested_addr)
+		requested_addr = pci_find_max_end_va();
 
 	/* try mapping the NIC resources using VFIO if it exists */  #ifdef VFIO_PRESENT
 	if (pci_vfio_is_enabled()) {
-		ret = pci_vfio_map_resource(dev);
+		ret = pci_vfio_map_resource(dev, &requested_addr);
 		if (ret == 0)
 			mapped = 1;
 		else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)  #endif
 	/* map resources for devices that use igb_uio */
 	if (!mapped) {
-		ret = pci_uio_map_resource(dev);
+		ret = pci_uio_map_resource(dev, &requested_addr);
 		if (ret != 0)
 			return ret;
 	}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 
 /* map the PCI resource of a PCI device in virtual memory */  int -pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	int i, j;
 	char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else
+					*requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
  * primary and secondary processes follow almost exactly the same path
  */
 int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr)
 {
 	struct vfio_group_status group_status = {
 			.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
+		maps[i].addr = *requested_addr;
 		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
 				reg.size);
 
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 			return -1;
 		}
 
+		*requested_addr = bar_addr;
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg.offset;
 		maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
 /* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 
 #ifdef VFIO_PRESENT
 
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);  int pci_vfio_mp_sync_setup(void);
 
 /* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 int pci_vfio_get_group_fd(int iommu_group_fd);  int pci_vfio_get_container_fd(void);
 
--
1.9.1
From bruce.richardson@intel.com  Fri Nov  7 16:21:25 2014
Return-Path: <bruce.richardson@intel.com>
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
 by dpdk.org (Postfix) with ESMTP id 0ACF67F29
 for <dev@dpdk.org>; Fri,  7 Nov 2014 16:21:24 +0100 (CET)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
 by fmsmga101.fm.intel.com with ESMTP; 07 Nov 2014 07:30:56 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,333,1413270000"; d="scan'208";a="618893192"
Received: from gcgray-mobl3.ger.corp.intel.com ([10.252.29.251])
 by fmsmga001.fm.intel.com with SMTP; 07 Nov 2014 07:30:54 -0800
Received: by  (sSMTP sendmail emulation); Fri, 07 Nov 2014 15:30:53 +0025
Date: Fri, 7 Nov 2014 15:30:53 +0000
From: Bruce Richardson <bruce.richardson@intel.com>
To: Manoj Viswanath <manoj.viswanath@gmail.com>
Message-ID: <20141107153053.GA10376@bricha3-MOBL3>
References: <CAC1b25pX3R2y_Cjp5UAdL6Bozrf01OkG6U6LGQvJ2aUTDXGvSA@mail.gmail.com>
 <20141105101246.GA9856@bricha3-MOBL3>
 <CAC1b25pY-wbEeVicBEVhYChOqjWXNRHO51wQwc8q4uo28DGO2g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAC1b25pY-wbEeVicBEVhYChOqjWXNRHO51wQwc8q4uo28DGO2g@mail.gmail.com>
Organization: Intel Shannon Ltd.
User-Agent: Mutt/1.5.23 (2014-03-12)
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Ports not detected by IGB_UIO in DPDK 1.7.1 in
 QEMU_KVM environment
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Nov 2014 15:21:26 -0000

On Fri, Nov 07, 2014 at 08:31:34PM +0530, Manoj Viswanath wrote:
> Hi Bruce,
>
> I was not doing anything specific for binding the NICs to IGB_UIO (like
> invoking "dpdk_nic_bind.py" script explicitly) when using my application
> with DPDK 1.6.0. The e1000 devices assigned via virt-manager to the VM were
> automatically getting picked up and initialized by IGB_UIO within each VM.
>
> The same is not working with DPDK 1.7.1 now.
>
> I tried exporting the "dpdk_nic_bind.py" script into my VM (running DPDK
> 1.7.1) and tried to check the status. The emulated devices were shown as
> neither bound to kernel nor to IGB_UIO as evident from below output:-
>
> <--------------------------------------------------------------------------------------------------->
> Network devices using DPDK-compatible driver
> ===========================================> <none>
>
> Network devices using kernel driver
> ==================================> 0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio
>
> Other network devices
> ====================> 0000:00:04.0 '82540EM Gigabit Ethernet Controller' unused=igb_uio
> 0000:00:05.0 '82540EM Gigabit Ethernet Controller' unused=igb_uio
> <--------------------------------------------------------------------------------------------------->
>
> When i tried to forcefully bind the NICs using the "--bind=igb_uio" option

Was there any output of the dpdk_nic_bind script? What does the output of
it with --status show afterwards?

Regards,
/Bruce

      parent reply	other threads:[~2014-11-07 15:10 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-11-05 13:25 [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured lxu
2014-11-05 15:10 ` Burakov, Anatoly
2014-11-05 15:49 ` [dpdk-dev] 答复: " XU Liang
2014-11-05 15:59   ` Burakov, Anatoly
2014-11-05 16:10   ` [dpdk-dev] 答复:答复: " XU Liang
2014-11-26  1:46     ` Qiu, Michael
2014-11-26  9:58       ` Burakov, Anatoly
2014-11-06 14:11 ` [dpdk-dev] [PATCH v2] " lxu
2014-11-06 14:27   ` Burakov, Anatoly
2014-11-06 14:48   ` [dpdk-dev] 答复:[PATCH " 徐亮
2014-11-06 14:47 ` [dpdk-dev] [PATCH v3] " lxu
2014-11-06 15:06   ` De Lara Guarch, Pablo
2014-11-06 15:07 ` [dpdk-dev] [PATCH v4] " lxu
2014-11-06 15:12   ` Thomas Monjalon
2014-11-06 15:11 ` lxu
2014-11-06 15:32 ` [dpdk-dev] [PATCH v5] " lxu
2014-11-06 15:41   ` Burakov, Anatoly
2014-11-06 15:58     ` Thomas Monjalon
2014-11-06 16:10       ` Burakov, Anatoly
2014-11-06 17:30         ` Bruce Richardson
2014-11-07  8:01 ` [dpdk-dev] [PATCH v6] " lxu
2014-11-07  9:42   ` Bruce Richardson
2014-11-07  9:47   ` Burakov, Anatoly
2014-11-07  9:57   ` XU Liang
2014-11-07 14:37     ` XU Liang
2014-11-10 11:34   ` [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages Anatoly Burakov
2014-11-10 13:33     ` Burakov, Anatoly
2014-11-11  3:53     ` XU Liang
2014-11-11 10:09     ` [dpdk-dev] [PATCH v8] " Anatoly Burakov
2014-11-13 11:34       ` Burakov, Anatoly
2014-11-13 12:58         ` Bruce Richardson
2014-11-13 13:44           ` Burakov, Anatoly
2014-11-13 13:46       ` Bruce Richardson
2014-11-25 17:17         ` Thomas Monjalon
2014-11-07 14:57 ` [dpdk-dev] [PATCH v7] eal: map uio " lxu
2014-11-07 15:14   ` Burakov, Anatoly
2014-11-07 15:15   ` Thomas Monjalon
2014-11-07 15:19   ` XU Liang [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f5cf68fd-528b-4eaf-b9c0-2380d426928e@cinfotech.cn \
    --to=liang.xu@cinfotech.cn \
    --cc=anatoly.burakov@intel.com \
    --cc=dev@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git