From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 1EFB07CDC for ; Wed, 21 Jun 2017 17:50:46 +0200 (CEST) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Jun 2017 08:49:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,369,1493708400"; d="scan'208";a="983564421" Received: from smonroyx-mobl.ger.corp.intel.com (HELO [10.237.221.32]) ([10.237.221.32]) by orsmga003.jf.intel.com with ESMTP; 21 Jun 2017 08:49:28 -0700 To: dev@dpdk.org References: <25388622.XuEPg0MkIL@polaris> From: Sergio Gonzalez Monroy Cc: Thomas Monjalon , Ilya Maximets , "Richardson, Bruce" , Jerin Jacob , "hemant.agrawal@nxp.com" Message-ID: <8bbfaa04-fa99-8c2a-c0ec-a91059809b73@intel.com> Date: Wed, 21 Jun 2017 16:49:27 +0100 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <25388622.XuEPg0MkIL@polaris> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH] eal: use get_mempolicy(2) to find numa socket on Linux X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 15:50:47 -0000 On 18/06/2017 16:52, Gregory Etelson wrote: > Use get_mempolicy(2) to find numa socket on Linux > instead of parsing /proc/self/numa_maps. > When process maps around 1K hugepages > numa_maps file can miss huge records in older Linux kernels. > get_mempolicy() proved more reliable > > Requires numactl dev package > > Signed-off-by: Gregory Etelson > --- > lib/librte_eal/linuxapp/eal/eal_memory.c | 85 ++++---------------------------- > mk/exec-env/linuxapp/rte.vars.mk | 1 + > mk/internal/rte.build-pre.mk | 1 + > 3 files changed, 11 insertions(+), 76 deletions(-) > > diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c > index e17c9cb..48b71bc 100644 > --- a/lib/librte_eal/linuxapp/eal/eal_memory.c > +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c > @@ -54,6 +54,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -495,86 +496,20 @@ unmap_all_hugepages_orig(struct hugepage_file *hugepg_tbl, struct hugepage_info > return 0; > } > > -/* > - * Parse /proc/self/numa_maps to get the NUMA socket ID for each huge > - * page. > - */ > static int > find_numasocket(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi) > { > - int socket_id; > - char *end, *nodestr; > - unsigned i, hp_count = 0; > - uint64_t virt_addr; > - char buf[BUFSIZ]; > - char hugedir_str[PATH_MAX]; > - FILE *f; > - > - f = fopen("/proc/self/numa_maps", "r"); > - if (f == NULL) { > - RTE_LOG(NOTICE, EAL, "cannot open /proc/self/numa_maps," > - " consider that all memory is in socket_id 0\n"); > - return 0; > - } > - > - snprintf(hugedir_str, sizeof(hugedir_str), > - "%s/%s", hpi->hugedir, internal_config.hugefile_prefix); > - > - /* parse numa map */ > - while (fgets(buf, sizeof(buf), f) != NULL) { > - > - /* ignore non huge page */ > - if (strstr(buf, " huge ") == NULL && > - strstr(buf, hugedir_str) == NULL) > - continue; > - > - /* get zone addr */ > - virt_addr = strtoull(buf, &end, 16); > - if (virt_addr == 0 || end == buf) { > - RTE_LOG(ERR, EAL, "%s(): error in numa_maps parsing\n", __func__); > - goto error; > - } > - > - /* get node id (socket id) */ > - nodestr = strstr(buf, " N"); > - if (nodestr == NULL) { > - RTE_LOG(ERR, EAL, "%s(): error in numa_maps parsing\n", __func__); > - goto error; > - } > - nodestr += 2; > - end = strstr(nodestr, "="); > - if (end == NULL) { > - RTE_LOG(ERR, EAL, "%s(): error in numa_maps parsing\n", __func__); > - goto error; > - } > - end[0] = '\0'; > - end = NULL; > - > - socket_id = strtoul(nodestr, &end, 0); > - if ((nodestr[0] == '\0') || (end == NULL) || (*end != '\0')) { > - RTE_LOG(ERR, EAL, "%s(): error in numa_maps parsing\n", __func__); > - goto error; > - } > - > - /* if we find this page in our mappings, set socket_id */ > - for (i = 0; i < hpi->num_pages[0]; i++) { > - void *va = (void *)(unsigned long)virt_addr; > - if (hugepg_tbl[i].orig_va == va) { > - hugepg_tbl[i].socket_id = socket_id; > - hp_count++; > - } > + unsigned int i; > + for (i = 0; i < hpi->num_pages[0]; i++) { > + if (get_mempolicy(&hugepg_tbl[i].socket_id, > + NULL, 0, hugepg_tbl[i].orig_va, > + MPOL_F_NODE | MPOL_F_ADDR) < 0) { > + RTE_LOG(ERR, EAL, "Failed to find NUMA socket for %p\n", > + hugepg_tbl[i].orig_va); > + return -1; > } > } > - > - if (hp_count < hpi->num_pages[0]) > - goto error; > - > - fclose(f); > return 0; > - > -error: > - fclose(f); > - return -1; > } > > static int > @@ -1051,8 +986,6 @@ rte_eal_hugepage_init(void) > } > > if (find_numasocket(&tmp_hp[hp_offset], hpi) < 0){ > - RTE_LOG(DEBUG, EAL, "Failed to find NUMA socket for %u MB pages\n", > - (unsigned)(hpi->hugepage_sz / 0x100000)); > goto fail; > } > > diff --git a/mk/exec-env/linuxapp/rte.vars.mk b/mk/exec-env/linuxapp/rte.vars.mk > index 9a71699..0da977a 100644 > --- a/mk/exec-env/linuxapp/rte.vars.mk > +++ b/mk/exec-env/linuxapp/rte.vars.mk > @@ -59,5 +59,6 @@ LINK_USING_CC := 1 > EXECENV_LDFLAGS += -export-dynamic > # Add library to the group to resolve symbols > EXECENV_LDLIBS += -ldl > +EXECENV_LDLIBS += -lnuma > > export EXECENV_CFLAGS EXECENV_LDFLAGS EXECENV_ASFLAGS EXECENV_LDLIBS > diff --git a/mk/internal/rte.build-pre.mk b/mk/internal/rte.build-pre.mk > index 560cf82..9bc0ffd 100644 > --- a/mk/internal/rte.build-pre.mk > +++ b/mk/internal/rte.build-pre.mk > @@ -29,6 +29,7 @@ > # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE > # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > > +LDLIBS += -lnuma > _BUILD_TARGETS := _prebuild _build _postbuild > > comma := , I think following the discussion of libnuma dependency in another thread [1], you would need to implement a similar approach and keep the old method while providing this alternative if libnuma is present. As Ilya mentions in the thread, this is usually the job of tools such as autotools, cmake or meson but given that we do not have such tools in DPDK yet, we rely in a build time config option for libnuma. Given that we already have a libnuma config option for VHOST, we might be adding anew one for hugepage balancing, I think it would make sense to just have single CONFIG_RTE_LIBNUMA option instead. Thoughts? Thanks, Sergio [1] http://dpdk.org/ml/archives/dev/2017-June/067298.html