From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 1C40B234 for ; Fri, 6 Feb 2015 12:00:18 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP; 06 Feb 2015 02:55:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.09,529,1418112000"; d="scan'208";a="662499431" Received: from bricha3-mobl3.ger.corp.intel.com ([10.243.20.42]) by fmsmga001.fm.intel.com with SMTP; 06 Feb 2015 03:00:15 -0800 Received: by (sSMTP sendmail emulation); Fri, 06 Feb 2015 11:00:14 +0025 Date: Fri, 6 Feb 2015 11:00:14 +0000 From: Bruce Richardson To: Stefan Puiu Message-ID: <20150206110014.GA16144@bricha3-MOBL3> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Organization: Intel Shannon Ltd. User-Agent: Mutt/1.5.23 (2014-03-12) Cc: dev@dpdk.org Subject: Re: [dpdk-dev] upper limit on the size of allocation through rte_malloc in dpdk-1.8.0? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 11:00:19 -0000 On Wed, Feb 04, 2015 at 05:24:58PM +0200, Stefan Puiu wrote: > Hi, > > I'm trying to alter an existing program to use the Intel DPDK. I'm > using 1.8.0, compiled by me as a shared library > (CONFIG_RTE_BUILD_COMBINE_LIBS=y and CONFIG_RTE_BUILD_SHARED_LIB=y in > .config) on Ubuntu 12.04. The program needs to allocate large blocks > of memory (between 1 and 4 chunks of 4.5GB, also 1-4 chunks of 2.5 > GB). I tried changing my C++ code to use an array allocated using > rte_malloc() instead of the std::vector I was using beforehand, but it > seems the call to rte_malloc() fails. I then made a simple test > program using the DPDK that takes a size to allocate and if that > fails, tries again with sizes of 100MB less, basically the code below. > This is C++ code (well, now that I look it could've been plain C, but > I need C++) compiled with g++-4.6 with '-std=gnu++0x': > > int main(int argc, char **argv) > { > int ret = rte_eal_init(argc, argv); > if (ret < 0) > rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); > argc -= ret; > argv += ret; > > [... check argc >= 2] > size_t size = strtoul(argv[1], NULL, 10); > size_t s = size; > > for (size_t i = 0; i < 30; ++i) { > printf("Trying to allocate %'zu bytes\n", s); > buf = rte_malloc("test", s, 0); > if (!buf) > printf ("Failed!\n"); > else { > printf ("Success!\n"); > rte_free(buf); > break; > } > > s = s - (100 * 1024ULL * 1024ULL); > } > > return 0; > } > > I'm getting: > Trying to allocate 4,832,038,656 bytes > Failed! > Trying to allocate 4,727,181,056 bytes > Failed! > [...] > Trying to allocate 2,944,601,856 bytes > Success! > > It's not always the same value, but usually somewhere around 3GB > rte_malloc() succeeds. I'm running on a physical (non-VM) NUMA machine > with 2 physical CPUs, each having 64GBs of local memory. The machine > also runs Ubuntu 12.04 server. I've created 16384 hugepages of 2MB: > > echo 16384 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > > I'm running the basic app like this: > > sudo numactl --membind=0 ~/src/test/dpdk_test/alloc -c 1f -n 4 -w > 04:00.0 --socket-mem=16384,0 -- 4832038656 > > I'm trying to run only on NUMA node 0 and only allocate memory from > there - that's what the app I'm moving to the DPDK works like (using > numactl --membind=x and --cpunodebind=x). > > Is there an upper limit on the amount of memory rte_malloc() will try > to allocate? I tried both after a reboot and when the machine had been > running for a while with not much success. Am I missing something? > It's a bit weird to be only able to allocate 3GB out of the 32GB > assigned to the app... > > On a related note, what would be a good way to compile the DPDK with > debug info (and preferably -O0)? There's quite a web of .mk files used > and I haven't figured out where the optimization level / debug options > are set. > > Thanks in advance, > Stefan. Does your system support 1G pages? I would recommend using a smaller number of 1G pages vs the huge number of 2MB pages that you are currently using. There may be issues with the allocations failing due to a lack of contiguous blocks of memory due to the 2MB pages being spread across memory. Regards, /Bruce