From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 4509D7EC4 for ; Fri, 7 Nov 2014 10:38:18 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP; 07 Nov 2014 01:41:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,331,1413270000"; d="scan'208";a="618758104" Received: from irsmsx103.ger.corp.intel.com ([163.33.3.157]) by fmsmga001.fm.intel.com with ESMTP; 07 Nov 2014 01:47:47 -0800 Received: from irsmsx108.ger.corp.intel.com (163.33.3.3) by IRSMSX103.ger.corp.intel.com (163.33.3.157) with Microsoft SMTP Server (TLS) id 14.3.195.1; Fri, 7 Nov 2014 09:47:46 +0000 Received: from irsmsx109.ger.corp.intel.com ([169.254.13.101]) by IRSMSX108.ger.corp.intel.com ([169.254.11.82]) with mapi id 14.03.0195.001; Fri, 7 Nov 2014 09:47:45 +0000 From: "Burakov, Anatoly" To: lxu , "dev@dpdk.org" Thread-Topic: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured. Thread-Index: AQHP+mEdCoJJM6uhEkSpotX1GZcOhJxU6HMQ Date: Fri, 7 Nov 2014 09:47:45 +0000 Message-ID: References: <1415193919-17361-1-git-send-email-liang.xu@cinfotech.cn> <1415347284-15468-1-git-send-email-liang.xu@cinfotech.cn> In-Reply-To: <1415347284-15468-1-git-send-email-liang.xu@cinfotech.cn> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2014 09:38:20 -0000 The commit message looks fine to me, but VFIO code needs to be adjusted the= same way. Also, now that I think of it, you can't simply assume that whatever last me= mseg you have has the latest virtual address. When IVSHMEM is initialized, = it too reserves some space in the virtual memory, which can be higher than = the last hugepage, but not be the last hugepage (because IVSHMEM memory is = first to be reserved, before the main memory). My advice would be to rewrite the function to return the maximum end virtua= l address (instead of a last segment) and move it to eal_pci.c (and include= declaration for it in include/eal_pci_init.h). My apologies for not thinking about any of this during the first 6 iteratio= ns of the patch :( Thanks, Anatoly -----Original Message----- From: lxu [mailto:liang.xu@cinfotech.cn]=20 Sent: Friday, November 7, 2014 8:01 AM To: dev@dpdk.org Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo Subject: [PATCH v6] eal: map uio resources after hugepages when the base_vi= rtaddr is configured. A multiple process DPDK application must mmap hugepages and pci resource in= to same virtual addresses. By default the virtual addresses chosen by the p= rimary process automatically when calling the mmap. But sometime the virtua= l addresses chosen by the primary process isn't usable at secondary process= . Such as the secondary process linked with more libraries than primary pro= cess. The library has been mapped into this virtual address. The command li= ne parameter 'base-virtaddr' has been added for this situation. If it's con= figured, the hugepages will be mapped into this base address. But the virtu= al address of uio resource mapped still does not refer to the parameter. In= that case it would still fail. This patch try to map uio resources after hugepages when the base_virtaddr = is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can b= e resolved by set base-virtaddr into free virtual address space. Signed-off-by: lxu --- lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 +++++++++++++++++++++++++++= +- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/lin= uxapp/eal/eal_pci_uio.c index 7e62266..a2c9ab6 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c @@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstb= uf, return uio_num; } =20 +static inline const struct rte_memseg * +get_physmem_last(void) +{ + const struct rte_memseg * seg =3D rte_eal_get_physmem_layout(); + const struct rte_memseg * last =3D seg; + unsigned i =3D 0; + + for (i=3D0; iaddr =3D=3D NULL) + break; + + if(seg->addr > last->addr) + last =3D seg; + + } + return last; +} + /* map the PCI resource of a PCI device in virtual memory */ int pci_uio= _map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_= resource(struct rte_pci_device *dev) struct mapped_pci_resource *uio_res; struct pci_map *maps; =20 + /* map uio resource into user required virtual address */ + static void * requested_addr; + if (internal_config.base_virtaddr && NULL =3D=3D requested_addr) { + const struct rte_memseg * last =3D get_physmem_last(); + requested_addr =3D RTE_PTR_ADD(last->addr, last->len); + } + dev->intr_handle.fd =3D -1; dev->intr_handle.type =3D RTE_INTR_HANDLE_UNKNOWN; =20 @@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev) if (maps[j].addr !=3D NULL) fail =3D 1; else { - mapaddr =3D pci_map_resource(NULL, fd, (off_t)offset, + mapaddr =3D pci_map_resource(requested_addr, fd, (off_t)offset, (size_t)maps[j].size); if (mapaddr =3D=3D NULL) fail =3D 1; + else if (NULL !=3D requested_addr) + requested_addr =3D RTE_PTR_ADD(mapaddr, maps[j].size); } =20 if (fail) { -- 1.9.1