From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 1FABA7F4D for ; Fri, 7 Nov 2014 16:06:00 +0100 (CET) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP; 07 Nov 2014 07:15:08 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,333,1413270000"; d="scan'208";a="628361422" Received: from irsmsx102.ger.corp.intel.com ([163.33.3.155]) by fmsmga002.fm.intel.com with ESMTP; 07 Nov 2014 07:15:07 -0800 Received: from irsmsx110.ger.corp.intel.com (163.33.3.25) by IRSMSX102.ger.corp.intel.com (163.33.3.155) with Microsoft SMTP Server (TLS) id 14.3.195.1; Fri, 7 Nov 2014 15:14:27 +0000 Received: from irsmsx109.ger.corp.intel.com ([169.254.13.101]) by IRSMSX110.ger.corp.intel.com ([169.254.15.3]) with mapi id 14.03.0195.001; Fri, 7 Nov 2014 15:14:26 +0000 From: "Burakov, Anatoly" To: lxu , "dev@dpdk.org" Thread-Topic: [PATCH v7] eal: map uio resources after hugepages. Thread-Index: AQHP+ptlE8F/tniCYEuRFv3RQLuRUpxVQtDw Date: Fri, 7 Nov 2014 15:14:26 +0000 Message-ID: References: <1415193919-17361-1-git-send-email-liang.xu@cinfotech.cn> <1415372269-8723-1-git-send-email-liang.xu@cinfotech.cn> In-Reply-To: <1415372269-8723-1-git-send-email-liang.xu@cinfotech.cn> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v7] eal: map uio resources after hugepages. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2014 15:06:02 -0000 Um, not sure I agree with this implementation. I think a cleaner way would = be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resou= rce (or rather, put it in include/eal_pci_init.h, like extern void *request= ed_addr) but make actual use of it in pci_uio/vfio_map_resource only (and l= eave all of this out of eal_pci.c at all). That will also rid you of the ne= cessity to pass around pointers to pointers. (in that case I would also rename requested_addr to pci_map_addr or somethi= ng, to make it less vague)=20 Thanks, Anatoly -----Original Message----- From: lxu [mailto:liang.xu@cinfotech.cn]=20 Sent: Friday, November 7, 2014 2:58 PM To: dev@dpdk.org Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo Subject: [PATCH v7] eal: map uio resources after hugepages. A multiple process DPDK application must mmap hugepages and pci resources i= nto same virtual addresses. By default the virtual addresses chosen by the = primary process automatically when calling the mmap. But sometime the chose= n virtual addresses isn't usable at secondary process. Such as the secondar= y process linked with more libraries than primary process. The library has = been mapped into this virtual address. The command line parameter 'base-vir= taddr' has been added for this situation. If it's configured, the hugepages= will be mapped into this base address. But the virtual address of uio reso= urce mapped still does not refer to the parameter. In that case "EAL: pci_m= ap_resource(): cannot mmap" will be got. This patch try to map uio resources after hugepages. So the error can be re= solved by set base-virtaddr into free virtual address space. Signed-off-by: lxu --- lib/librte_eal/linuxapp/eal/eal_pci.c | 25 ++++++++++++++++++= ++-- lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 6 ++++-- lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 4 +++- lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 4 ++-- 4 files changed, 32 insertions(+), 7 deletions(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxap= p/eal/eal_pci.c index 5fe3961..aef6f5e 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev) } = #endif =20 +static void * +pci_find_max_end_va(void) +{ + const struct rte_memseg * seg =3D rte_eal_get_physmem_layout(); + const struct rte_memseg * last =3D seg; + unsigned i =3D 0; + + for (i=3D0; iaddr =3D=3D NULL) + break; + + if(seg->addr > last->addr) + last =3D seg; + + } + return RTE_PTR_ADD(last->addr, last->len); } + static int pci_map_device(struct rte_pci_device *dev) { int ret, mapped =3D 0; + static void * requested_addr; + if(NULL =3D=3D requested_addr) + requested_addr =3D pci_find_max_end_va(); =20 /* try mapping the NIC resources using VFIO if it exists */ #ifdef VFIO_= PRESENT if (pci_vfio_is_enabled()) { - ret =3D pci_vfio_map_resource(dev); + ret =3D pci_vfio_map_resource(dev, &requested_addr); if (ret =3D=3D 0) mapped =3D 1; else if (ret < 0) @@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev) #endif /* map resources for devices that use igb_uio */ if (!mapped) { - ret =3D pci_uio_map_resource(dev); + ret =3D pci_uio_map_resource(dev, &requested_addr); if (ret !=3D 0) return ret; } diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/lin= uxapp/eal/eal_pci_uio.c index 7e62266..e92124e 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c @@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbu= f, =20 /* map the PCI resource of a PCI device in virtual memory */ int -pci_uio= _map_resource(struct rte_pci_device *dev) +pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr) { int i, j; char dirname[PATH_MAX]; @@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev) if (maps[j].addr !=3D NULL) fail =3D 1; else { - mapaddr =3D pci_map_resource(NULL, fd, (off_t)offset, + mapaddr =3D pci_map_resource(*requested_addr, fd, (off_t)offset, (size_t)maps[j].size); if (mapaddr =3D=3D NULL) fail =3D 1; + else + *requested_addr =3D RTE_PTR_ADD(mapaddr, maps[j].size); } =20 if (fail) { diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/li= nuxapp/eal/eal_pci_vfio.c index c776ddc..2102adf 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c @@ -515,7 +515,7 @@ clear_current_group(void) * primary and secondary processes follow almost exactly the same path */ int -pci_vfio_map_resource(struct rte_pci_device *dev) +pci_vfio_map_resource(struct rte_pci_device *dev, void=20 +**requested_addr) { struct vfio_group_status group_status =3D { .argsz =3D sizeof(group_status) @@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev) if (i =3D=3D msix_bar) continue; =20 + maps[i].addr =3D *requested_addr; bar_addr =3D pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset, reg.size); =20 @@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev) return -1; } =20 + *requested_addr =3D bar_addr; maps[i].addr =3D bar_addr; maps[i].offset =3D reg.offset; maps[i].size =3D reg.size; diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librt= e_eal/linuxapp/eal/include/eal_pci_init.h index d758bee..e14fa36 100644 --- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h +++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h @@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_= t offset, size_t size); =20 /* map IGB_UIO resource prototype */ -int pci_uio_map_resource(struct rte_pci_device *dev); +int pci_uio_map_resource(struct rte_pci_device *dev, void=20 +**requested_addr); =20 #ifdef VFIO_PRESENT =20 @@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void); int pci_vfio_mp_sync_setu= p(void); =20 /* map VFIO resource prototype */ -int pci_vfio_map_resource(struct rte_pci_device *dev); +int pci_vfio_map_resource(struct rte_pci_device *dev, void=20 +**requested_addr); int pci_vfio_get_group_fd(int iommu_group_fd); int pci_vfio_get_container= _fd(void); =20 -- 1.9.1