From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [143.182.124.21]) by dpdk.org (Postfix) with ESMTP id B2E8FB0BE for ; Wed, 18 Jun 2014 13:27:01 +0200 (CEST) Received: from azsmga001.ch.intel.com ([10.2.17.19]) by azsmga101.ch.intel.com with ESMTP; 18 Jun 2014 04:27:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.01,499,1400050800"; d="scan'208";a="446978014" Received: from irvmail001.ir.intel.com ([163.33.26.43]) by azsmga001.ch.intel.com with ESMTP; 18 Jun 2014 04:27:15 -0700 Received: from sivswdev01.ir.intel.com (sivswdev01.ir.intel.com [10.237.217.45]) by irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id s5IBREfR012012; Wed, 18 Jun 2014 12:27:14 +0100 Received: from sivswdev01.ir.intel.com (localhost [127.0.0.1]) by sivswdev01.ir.intel.com with ESMTP id s5IBREuI006308; Wed, 18 Jun 2014 12:27:14 +0100 Received: (from aburakov@localhost) by sivswdev01.ir.intel.com with id s5IBREAd006304; Wed, 18 Jun 2014 12:27:14 +0100 From: Anatoly Burakov To: dev@dpdk.org Date: Wed, 18 Jun 2014 12:27:06 +0100 Message-Id: X-Mailer: git-send-email 1.7.0.7 In-Reply-To: References: In-Reply-To: References: Subject: [dpdk-dev] [PATCH v3 1/9] eal: map shared config into exact same address as primary process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2014 11:27:02 -0000 Shared config is shared across primary and secondary processes. However,when using rte_malloc, the malloc elements keep references to the heap inside themselves. This heap reference might not be referencing a local heap because the heap reference points to the heap of whatever process has allocated that malloc element. Therefore, there can be situations when malloc elements in a given heap actually reference different addresses for the same heap - depending on which process has allocated the element. This can lead to segmentation faults when dealing with malloc elements allocated on the same heap by different processes. To fix this problem, heaps will now have the same addresses across processes. In order to achieve that, a new field in a shared mem_config (a structure that holds the heaps, and which is shared across processes) was added to keep the address of where this config is mapped in the primary process. Secondary process will now map the config in two stages - first, it'll map it into an arbitrary address and read the address the primary process has allocated for the shared config. Then, the config is unmapped and re-mapped using the address previously read. Signed-off-by: Anatoly Burakov --- lib/librte_eal/common/include/rte_eal_memconfig.h | 5 +++ lib/librte_eal/linuxapp/eal/eal.c | 44 ++++++++++++++++++++--- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/lib/librte_eal/common/include/rte_eal_memconfig.h b/lib/librte_eal/common/include/rte_eal_memconfig.h index 30ce6fc..d6359e5 100644 --- a/lib/librte_eal/common/include/rte_eal_memconfig.h +++ b/lib/librte_eal/common/include/rte_eal_memconfig.h @@ -89,6 +89,11 @@ struct rte_mem_config { /* Heaps of Malloc per socket */ struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES]; + + /* address of mem_config in primary process. used to map shared config into + * exact same address the primary process maps it. + */ + uint64_t mem_cfg_addr; } __attribute__((__packed__)); diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c index 6994303..fee375c 100644 --- a/lib/librte_eal/linuxapp/eal/eal.c +++ b/lib/librte_eal/linuxapp/eal/eal.c @@ -239,13 +239,19 @@ rte_eal_config_create(void) } memcpy(rte_mem_cfg_addr, &early_mem_config, sizeof(early_mem_config)); rte_config.mem_config = (struct rte_mem_config *) rte_mem_cfg_addr; + + /* store address of the config in the config itself so that secondary + * processes could later map the config into this exact location */ + rte_config.mem_config->mem_cfg_addr = (uintptr_t) rte_mem_cfg_addr; + } /* attach to an existing shared memory config */ static void rte_eal_config_attach(void) { - void *rte_mem_cfg_addr; + struct rte_mem_config *mem_config; + const char *pathname = eal_runtime_config_path(); if (internal_config.no_shconf) @@ -257,13 +263,40 @@ rte_eal_config_attach(void) rte_panic("Cannot open '%s' for rte_mem_config\n", pathname); } - rte_mem_cfg_addr = mmap(NULL, sizeof(*rte_config.mem_config), - PROT_READ | PROT_WRITE, MAP_SHARED, mem_cfg_fd, 0); + /* map it as read-only first */ + mem_config = (struct rte_mem_config *) mmap(NULL, sizeof(*mem_config), + PROT_READ, MAP_SHARED, mem_cfg_fd, 0); + if (mem_config == MAP_FAILED) + rte_panic("Cannot mmap memory for rte_config\n"); + + rte_config.mem_config = mem_config; +} + +/* reattach the shared config at exact memory location primary process has it */ +static void +rte_eal_config_reattach(void) +{ + struct rte_mem_config *mem_config; + void *rte_mem_cfg_addr; + + if (internal_config.no_shconf) + return; + + /* save the address primary process has mapped shared config to */ + rte_mem_cfg_addr = (void *) (uintptr_t) rte_config.mem_config->mem_cfg_addr; + + /* unmap original config */ + munmap(rte_config.mem_config, sizeof(struct rte_mem_config)); + + /* remap the config at proper address */ + mem_config = (struct rte_mem_config *) mmap(rte_mem_cfg_addr, + sizeof(*mem_config), PROT_READ | PROT_WRITE, MAP_SHARED, + mem_cfg_fd, 0); close(mem_cfg_fd); - if (rte_mem_cfg_addr == MAP_FAILED) + if (mem_config == MAP_FAILED || mem_config != rte_mem_cfg_addr) rte_panic("Cannot mmap memory for rte_config\n"); - rte_config.mem_config = (struct rte_mem_config *) rte_mem_cfg_addr; + rte_config.mem_config = mem_config; } /* Detect if we are a primary or a secondary process */ @@ -301,6 +334,7 @@ rte_config_init(void) case RTE_PROC_SECONDARY: rte_eal_config_attach(); rte_eal_mcfg_wait_complete(rte_config.mem_config); + rte_eal_config_reattach(); break; case RTE_PROC_AUTO: case RTE_PROC_INVALID: -- 1.8.1.4