From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 7080A378E; Thu, 16 Feb 2017 14:32:02 +0100 (CET) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Feb 2017 05:32:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.35,169,1484035200"; d="scan'208";a="1128227585" Received: from bricha3-mobl3.ger.corp.intel.com ([10.237.221.61]) by fmsmga002.fm.intel.com with SMTP; 16 Feb 2017 05:31:58 -0800 Received: by (sSMTP sendmail emulation); Thu, 16 Feb 2017 13:31:57 +0000 Date: Thu, 16 Feb 2017 13:31:57 +0000 From: Bruce Richardson To: Ilya Maximets Cc: dev@dpdk.org, David Marchand , Sergio Gonzalez Monroy , Heetae Ahn , Yuanhan Liu , Jianfeng Tan , Neil Horman , Yulong Pei , stable@dpdk.org Message-ID: <20170216133157.GB100024@bricha3-MOBL3.ger.corp.intel.com> References: <1487250070-13973-1-git-send-email-i.maximets@samsung.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1487250070-13973-1-git-send-email-i.maximets@samsung.com> Organization: Intel Research and =?iso-8859-1?Q?De=ACvel?= =?iso-8859-1?Q?opment?= Ireland Ltd. User-Agent: Mutt/1.7.2 (2016-11-26) Subject: Re: [dpdk-dev] [PATCH] mem: balanced allocation of hugepages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2017 13:32:03 -0000 On Thu, Feb 16, 2017 at 04:01:10PM +0300, Ilya Maximets wrote: > Currently EAL allocates hugepages one by one not paying > attention from which NUMA node allocation was done. > > Such behaviour leads to allocation failure if number of > available hugepages for application limited by cgroups > or hugetlbfs and memory requested not only from the first > socket. > > Example: > # 90 x 1GB hugepages availavle in a system > > cgcreate -g hugetlb:/test > # Limit to 32GB of hugepages > cgset -r hugetlb.1GB.limit_in_bytes=34359738368 test > # Request 4GB from each of 2 sockets > cgexec -g hugetlb:test testpmd --socket-mem=4096,4096 ... > > EAL: SIGBUS: Cannot mmap more hugepages of size 1024 MB > EAL: 32 not 90 hugepages of size 1024 MB allocated > EAL: Not enough memory available on socket 1! > Requested: 4096MB, available: 0MB > PANIC in rte_eal_init(): > Cannot init memory > > This happens beacause all allocated pages are > on socket 0. > > Fix this issue by setting mempolicy MPOL_PREFERRED for each > hugepage to one of requested nodes in a round-robin fashion. > In this case all allocated pages will be fairly distributed > between all requested nodes. > > New config option RTE_LIBRTE_EAL_NUMA_AWARE_HUGEPAGES > introduced and disabled by default because of external > dependency from libnuma. > I think this highlights a general technical problem we need to resolve in DPDK. If we want to add support for a new feature in DPDK by leveraging functionality in an existing library, we are caught in a sort of catch-22: * If we want to leverage the existing library, we have to have the feature off-by-default, as we don't want to increase the minimum requirements for DPDK. * If we want the feature enabled by default we need to avoid the dependency, and so reimplement some or all of the functionality inside DPDK itself. That will be rejected on the basis that it duplicates existing library functionality. I suspect the solution to this is more dynamic build-time configuration to start enabling things based on installed dependencies, but I'm open to other opinions. I see a gap here, however. /Bruce