From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 573FDAFDA for ; Tue, 17 Jun 2014 17:36:05 +0200 (CEST) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP; 17 Jun 2014 08:36:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.01,494,1400050800"; d="scan'208";a="529878625" Received: from irvmail001.ir.intel.com ([163.33.26.43]) by orsmga001.jf.intel.com with ESMTP; 17 Jun 2014 08:36:08 -0700 Received: from sivswdev01.ir.intel.com (sivswdev01.ir.intel.com [10.237.217.45]) by irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id s5HFa71C019312; Tue, 17 Jun 2014 16:36:07 +0100 Received: from sivswdev01.ir.intel.com (localhost [127.0.0.1]) by sivswdev01.ir.intel.com with ESMTP id s5HFa7PB031825; Tue, 17 Jun 2014 16:36:07 +0100 Received: (from aburakov@localhost) by sivswdev01.ir.intel.com with id s5HFa7wL031821; Tue, 17 Jun 2014 16:36:07 +0100 From: Anatoly Burakov To: dev@dpdk.org Date: Tue, 17 Jun 2014 16:35:59 +0100 Message-Id: X-Mailer: git-send-email 1.7.0.7 In-Reply-To: References: In-Reply-To: References: Subject: [dpdk-dev] [PATCH 1/9] eal: map shared config into exact same address as primary process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2014 15:36:06 -0000 Shared config is shared across primary and secondary processes. However,when using rte_malloc, the malloc elements keep references to the heap inside themselves. This heap reference might not be referencing a local heap because the heap reference points to the heap of whatever process has allocated that malloc element. Therefore, there can be situations when malloc elements in a given heap actually reference different addresses for the same heap - depending on which process has allocated the element. This can lead to segmentation faults when dealing with malloc elements allocated on the same heap by different processes. To fix this problem, heaps will now have the same addresses across processes. In order to achieve that, a new field in a shared mem_config (a structure that holds the heaps, and which is shared across processes) was added to keep the address of where this config is mapped in the primary process. Secondary process will now map the config in two stages - first, it'll map it into an arbitrary address and read the address the primary process has allocated for the shared config. Then, the config is unmapped and re-mapped using the address previously read. Signed-off-by: Anatoly Burakov --- lib/librte_eal/common/include/rte_eal_memconfig.h | 5 ++++ lib/librte_eal/linuxapp/eal/eal.c | 31 +++++++++++++++++++---- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/lib/librte_eal/common/include/rte_eal_memconfig.h b/lib/librte_eal/common/include/rte_eal_memconfig.h index 30ce6fc..d6359e5 100644 --- a/lib/librte_eal/common/include/rte_eal_memconfig.h +++ b/lib/librte_eal/common/include/rte_eal_memconfig.h @@ -89,6 +89,11 @@ struct rte_mem_config { /* Heaps of Malloc per socket */ struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES]; + + /* address of mem_config in primary process. used to map shared config into + * exact same address the primary process maps it. + */ + uint64_t mem_cfg_addr; } __attribute__((__packed__)); diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c index 6994303..fedd82f 100644 --- a/lib/librte_eal/linuxapp/eal/eal.c +++ b/lib/librte_eal/linuxapp/eal/eal.c @@ -239,6 +239,11 @@ rte_eal_config_create(void) } memcpy(rte_mem_cfg_addr, &early_mem_config, sizeof(early_mem_config)); rte_config.mem_config = (struct rte_mem_config *) rte_mem_cfg_addr; + + /* store address of the config in the config itself so that secondary + * processes could later map the config into this exact location */ + rte_config.mem_config->mem_cfg_addr = (uintptr_t) rte_mem_cfg_addr; + } /* attach to an existing shared memory config */ @@ -246,6 +251,8 @@ static void rte_eal_config_attach(void) { void *rte_mem_cfg_addr; + struct rte_mem_config *mem_config; + const char *pathname = eal_runtime_config_path(); if (internal_config.no_shconf) @@ -257,13 +264,27 @@ rte_eal_config_attach(void) rte_panic("Cannot open '%s' for rte_mem_config\n", pathname); } - rte_mem_cfg_addr = mmap(NULL, sizeof(*rte_config.mem_config), - PROT_READ | PROT_WRITE, MAP_SHARED, mem_cfg_fd, 0); - close(mem_cfg_fd); - if (rte_mem_cfg_addr == MAP_FAILED) + /* map it as read-only first */ + mem_config = (struct rte_mem_config *) mmap(NULL, sizeof(*mem_config), + PROT_READ, MAP_SHARED, mem_cfg_fd, 0); + if (mem_config == MAP_FAILED) rte_panic("Cannot mmap memory for rte_config\n"); - rte_config.mem_config = (struct rte_mem_config *) rte_mem_cfg_addr; + /* store address used by primary process */ + rte_mem_cfg_addr = (void *) (uintptr_t) mem_config->mem_cfg_addr; + + /* unmap the config */ + munmap(mem_config, sizeof(*mem_config)); + + /* map the config again, with the proper virtual address */ + mem_config = (struct rte_mem_config *) mmap(rte_mem_cfg_addr, + sizeof(*mem_config), PROT_READ | PROT_WRITE, MAP_SHARED, + mem_cfg_fd, 0); + if (mem_config == MAP_FAILED || mem_config != rte_mem_cfg_addr) + rte_panic("Cannot mmap memory for rte_config\n"); + close(mem_cfg_fd); + + rte_config.mem_config = mem_config; } /* Detect if we are a primary or a secondary process */ -- 1.8.1.4