From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [143.182.124.21]) by dpdk.org (Postfix) with ESMTP id 40716AFDA for ; Tue, 17 Jun 2014 18:29:24 +0200 (CEST) Received: from azsmga001.ch.intel.com ([10.2.17.19]) by azsmga101.ch.intel.com with ESMTP; 17 Jun 2014 09:29:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.01,495,1400050800"; d="scan'208";a="446590459" Received: from irsmsx104.ger.corp.intel.com ([163.33.3.159]) by azsmga001.ch.intel.com with ESMTP; 17 Jun 2014 09:29:04 -0700 Received: from irsmsx106.ger.corp.intel.com (163.33.3.31) by IRSMSX104.ger.corp.intel.com (163.33.3.159) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 17 Jun 2014 17:29:04 +0100 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.239]) by IRSMSX106.ger.corp.intel.com ([169.254.8.14]) with mapi id 14.03.0123.003; Tue, 17 Jun 2014 17:29:03 +0100 From: "Ananyev, Konstantin" To: "Burakov, Anatoly" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH 1/9] eal: map shared config into exact same address as primary process Thread-Index: AQHPikIN+i2ulxxlGEWA3sP148xZf5t1fQ7Q Date: Tue, 17 Jun 2014 16:29:03 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772580EFB7832@IRSMSX105.ger.corp.intel.com> References: In-Reply-To: Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.182] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH 1/9] eal: map shared config into exact same address as primary process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2014 16:29:26 -0000 Hi Anatoly >=20 > Shared config is shared across primary and secondary processes. > However,when using rte_malloc, the malloc elements keep references to > the heap inside themselves. This heap reference might not be referencing > a local heap because the heap reference points to the heap of whatever > process has allocated that malloc element. Therefore, there can be > situations when malloc elements in a given heap actually reference > different addresses for the same heap - depending on which process has > allocated the element. This can lead to segmentation faults when dealing > with malloc elements allocated on the same heap by different processes. >=20 > To fix this problem, heaps will now have the same addresses across > processes. In order to achieve that, a new field in a shared mem_config > (a structure that holds the heaps, and which is shared across processes) > was added to keep the address of where this config is mapped in the > primary process. >=20 > Secondary process will now map the config in two stages - first, it'll > map it into an arbitrary address and read the address the primary > process has allocated for the shared config. Then, the config is > unmapped and re-mapped using the address previously read. >=20 > Signed-off-by: Anatoly Burakov > --- > lib/librte_eal/common/include/rte_eal_memconfig.h | 5 ++++ > lib/librte_eal/linuxapp/eal/eal.c | 31 +++++++++++++++++= ++---- > 2 files changed, 31 insertions(+), 5 deletions(-) >=20 > diff --git a/lib/librte_eal/common/include/rte_eal_memconfig.h b/lib/libr= te_eal/common/include/rte_eal_memconfig.h > index 30ce6fc..d6359e5 100644 > --- a/lib/librte_eal/common/include/rte_eal_memconfig.h > +++ b/lib/librte_eal/common/include/rte_eal_memconfig.h > @@ -89,6 +89,11 @@ struct rte_mem_config { >=20 > /* Heaps of Malloc per socket */ > struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES]; > + > + /* address of mem_config in primary process. used to map shared config = into > + * exact same address the primary process maps it. > + */ > + uint64_t mem_cfg_addr; > } __attribute__((__packed__)); >=20 >=20 > diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/= eal/eal.c > index 6994303..fedd82f 100644 > --- a/lib/librte_eal/linuxapp/eal/eal.c > +++ b/lib/librte_eal/linuxapp/eal/eal.c > @@ -239,6 +239,11 @@ rte_eal_config_create(void) > } > memcpy(rte_mem_cfg_addr, &early_mem_config, sizeof(early_mem_config)); > rte_config.mem_config =3D (struct rte_mem_config *) rte_mem_cfg_addr; > + > + /* store address of the config in the config itself so that secondary > + * processes could later map the config into this exact location */ > + rte_config.mem_config->mem_cfg_addr =3D (uintptr_t) rte_mem_cfg_addr; > + > } >=20 > /* attach to an existing shared memory config */ > @@ -246,6 +251,8 @@ static void > rte_eal_config_attach(void) > { > void *rte_mem_cfg_addr; > + struct rte_mem_config *mem_config; > + > const char *pathname =3D eal_runtime_config_path(); >=20 > if (internal_config.no_shconf) > @@ -257,13 +264,27 @@ rte_eal_config_attach(void) > rte_panic("Cannot open '%s' for rte_mem_config\n", pathname); > } >=20 > - rte_mem_cfg_addr =3D mmap(NULL, sizeof(*rte_config.mem_config), > - PROT_READ | PROT_WRITE, MAP_SHARED, mem_cfg_fd, 0); > - close(mem_cfg_fd); > - if (rte_mem_cfg_addr =3D=3D MAP_FAILED) > + /* map it as read-only first */ > + mem_config =3D (struct rte_mem_config *) mmap(NULL, sizeof(*mem_config)= , > + PROT_READ, MAP_SHARED, mem_cfg_fd, 0); > + if (mem_config =3D=3D MAP_FAILED) > rte_panic("Cannot mmap memory for rte_config\n"); >=20 > - rte_config.mem_config =3D (struct rte_mem_config *) rte_mem_cfg_addr; > + /* store address used by primary process */ > + rte_mem_cfg_addr =3D (void *) (uintptr_t) mem_config->mem_cfg_addr; > + > + /* unmap the config */ > + munmap(mem_config, sizeof(*mem_config)); > + > + /* map the config again, with the proper virtual address */ > + mem_config =3D (struct rte_mem_config *) mmap(rte_mem_cfg_addr, > + sizeof(*mem_config), PROT_READ | PROT_WRITE, MAP_SHARED, > + mem_cfg_fd, 0); > + if (mem_config =3D=3D MAP_FAILED || mem_config !=3D rte_mem_cfg_addr) > + rte_panic("Cannot mmap memory for rte_config\n"); > + close(mem_cfg_fd); > + > + rte_config.mem_config =3D mem_config; > } >=20 > /* Detect if we are a primary or a secondary process */ > -- I think we introduce a race window here. If secondary process would do first mmap() before rte_config.mem_config->me= m_cfg_addr was properly set by primary process, then it will try to do second mmap() with wrong address. I think we need to do second mmap() straight after rte_eal_mcfg_wait_comple= te(), or even just inside it. Konstantin =20