From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f181.google.com (mail-ig0-f181.google.com [209.85.213.181]) by dpdk.org (Postfix) with ESMTP id A5543ADEE for ; Wed, 4 Feb 2015 16:24:59 +0100 (CET) Received: by mail-ig0-f181.google.com with SMTP id hn18so4676924igb.2 for ; Wed, 04 Feb 2015 07:24:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=YWRYA+KyXZtA0AzYdDU+faEzLUP4X99HbkgcrxkDmuE=; b=Yl87IRg0Wtxubm0uBmXpnHa/LbTwXliu0gbIiBtxQWyqz5QuK1y7LfrRoSaXslGC7L HD2cg424jB5Cj9LC1Kz4eBMQgwMFOjIQCcnTiN5FJIUAEv8UywiQJgaXcCGivKh4GqKI v8XwkbSk+LGurGUCvzNTHwBIs6ksVmKO+aZeY/lXfPhDan3e/Rb3HfQozdWBY57fMnHk C7Xw6yL1PPr7MyYitdWNu0/Hwne9089PhThsFlM7eLTSzJm2b9BGrrjaxjjXwcl9FW1u PrALzpuD/FpcvcWC9QCgho+p5eZEObdQSGnNx27Cp/taAabrE05zkqpEU2DY9NUOxAvX Ke/w== MIME-Version: 1.0 X-Received: by 10.43.75.69 with SMTP id yz5mr2280065icb.90.1423063499017; Wed, 04 Feb 2015 07:24:59 -0800 (PST) Received: by 10.36.60.14 with HTTP; Wed, 4 Feb 2015 07:24:58 -0800 (PST) Date: Wed, 4 Feb 2015 17:24:58 +0200 Message-ID: From: Stefan Puiu To: dev@dpdk.org Content-Type: text/plain; charset=UTF-8 Subject: [dpdk-dev] upper limit on the size of allocation through rte_malloc in dpdk-1.8.0? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2015 15:25:00 -0000 Hi, I'm trying to alter an existing program to use the Intel DPDK. I'm using 1.8.0, compiled by me as a shared library (CONFIG_RTE_BUILD_COMBINE_LIBS=y and CONFIG_RTE_BUILD_SHARED_LIB=y in .config) on Ubuntu 12.04. The program needs to allocate large blocks of memory (between 1 and 4 chunks of 4.5GB, also 1-4 chunks of 2.5 GB). I tried changing my C++ code to use an array allocated using rte_malloc() instead of the std::vector I was using beforehand, but it seems the call to rte_malloc() fails. I then made a simple test program using the DPDK that takes a size to allocate and if that fails, tries again with sizes of 100MB less, basically the code below. This is C++ code (well, now that I look it could've been plain C, but I need C++) compiled with g++-4.6 with '-std=gnu++0x': int main(int argc, char **argv) { int ret = rte_eal_init(argc, argv); if (ret < 0) rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); argc -= ret; argv += ret; [... check argc >= 2] size_t size = strtoul(argv[1], NULL, 10); size_t s = size; for (size_t i = 0; i < 30; ++i) { printf("Trying to allocate %'zu bytes\n", s); buf = rte_malloc("test", s, 0); if (!buf) printf ("Failed!\n"); else { printf ("Success!\n"); rte_free(buf); break; } s = s - (100 * 1024ULL * 1024ULL); } return 0; } I'm getting: Trying to allocate 4,832,038,656 bytes Failed! Trying to allocate 4,727,181,056 bytes Failed! [...] Trying to allocate 2,944,601,856 bytes Success! It's not always the same value, but usually somewhere around 3GB rte_malloc() succeeds. I'm running on a physical (non-VM) NUMA machine with 2 physical CPUs, each having 64GBs of local memory. The machine also runs Ubuntu 12.04 server. I've created 16384 hugepages of 2MB: echo 16384 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages I'm running the basic app like this: sudo numactl --membind=0 ~/src/test/dpdk_test/alloc -c 1f -n 4 -w 04:00.0 --socket-mem=16384,0 -- 4832038656 I'm trying to run only on NUMA node 0 and only allocate memory from there - that's what the app I'm moving to the DPDK works like (using numactl --membind=x and --cpunodebind=x). Is there an upper limit on the amount of memory rte_malloc() will try to allocate? I tried both after a reboot and when the machine had been running for a while with not much success. Am I missing something? It's a bit weird to be only able to allocate 3GB out of the 32GB assigned to the app... On a related note, what would be a good way to compile the DPDK with debug info (and preferably -O0)? There's quite a web of .mk files used and I haven't figured out where the optimization level / debug options are set. Thanks in advance, Stefan.