From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <stefan.puiu@gmail.com>
Received: from mail-ig0-f181.google.com (mail-ig0-f181.google.com
 [209.85.213.181]) by dpdk.org (Postfix) with ESMTP id A5543ADEE
 for <dev@dpdk.org>; Wed,  4 Feb 2015 16:24:59 +0100 (CET)
Received: by mail-ig0-f181.google.com with SMTP id hn18so4676924igb.2
 for <dev@dpdk.org>; Wed, 04 Feb 2015 07:24:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:date:message-id:subject:from:to:content-type;
 bh=YWRYA+KyXZtA0AzYdDU+faEzLUP4X99HbkgcrxkDmuE=;
 b=Yl87IRg0Wtxubm0uBmXpnHa/LbTwXliu0gbIiBtxQWyqz5QuK1y7LfrRoSaXslGC7L
 HD2cg424jB5Cj9LC1Kz4eBMQgwMFOjIQCcnTiN5FJIUAEv8UywiQJgaXcCGivKh4GqKI
 v8XwkbSk+LGurGUCvzNTHwBIs6ksVmKO+aZeY/lXfPhDan3e/Rb3HfQozdWBY57fMnHk
 C7Xw6yL1PPr7MyYitdWNu0/Hwne9089PhThsFlM7eLTSzJm2b9BGrrjaxjjXwcl9FW1u
 PrALzpuD/FpcvcWC9QCgho+p5eZEObdQSGnNx27Cp/taAabrE05zkqpEU2DY9NUOxAvX
 Ke/w==
MIME-Version: 1.0
X-Received: by 10.43.75.69 with SMTP id yz5mr2280065icb.90.1423063499017; Wed,
 04 Feb 2015 07:24:59 -0800 (PST)
Received: by 10.36.60.14 with HTTP; Wed, 4 Feb 2015 07:24:58 -0800 (PST)
Date: Wed, 4 Feb 2015 17:24:58 +0200
Message-ID: <CACKs7VDkpaBH4Gv1iWmiqmwjdPJnjL0nnWB2JYqWNqdW3+j-LQ@mail.gmail.com>
From: Stefan Puiu <stefan.puiu@gmail.com>
To: dev@dpdk.org
Content-Type: text/plain; charset=UTF-8
Subject: [dpdk-dev] upper limit on the size of allocation through rte_malloc
	in dpdk-1.8.0?
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2015 15:25:00 -0000

Hi,

I'm trying to alter an existing program to use the Intel DPDK. I'm
using 1.8.0, compiled by me as a shared library
(CONFIG_RTE_BUILD_COMBINE_LIBS=y and CONFIG_RTE_BUILD_SHARED_LIB=y in
.config) on Ubuntu 12.04. The program needs to allocate large blocks
of memory (between 1 and 4 chunks of 4.5GB, also 1-4 chunks of 2.5
GB). I tried changing my C++ code to use an array allocated using
rte_malloc() instead of the std::vector I was using beforehand, but it
seems the call to rte_malloc() fails. I then made a simple test
program using the DPDK that takes a size to allocate and if that
fails, tries again with sizes of 100MB less, basically the code below.
This is C++ code (well, now that I look it could've been plain C, but
I need C++) compiled with g++-4.6 with '-std=gnu++0x':

int main(int argc, char **argv)
{
    int ret = rte_eal_init(argc, argv);
    if (ret < 0)
        rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
    argc -= ret;
    argv += ret;

[... check argc >= 2]
    size_t size = strtoul(argv[1], NULL, 10);
    size_t s = size;

    for (size_t i = 0; i < 30; ++i) {
        printf("Trying to allocate %'zu bytes\n", s);
        buf = rte_malloc("test", s, 0);
        if (!buf)
            printf ("Failed!\n");
        else {
            printf ("Success!\n");
            rte_free(buf);
            break;
        }

        s = s - (100 * 1024ULL * 1024ULL);
    }

    return 0;
}

I'm getting:
Trying to allocate 4,832,038,656 bytes
Failed!
Trying to allocate 4,727,181,056 bytes
Failed!
[...]
Trying to allocate 2,944,601,856 bytes
Success!

It's not always the same value, but usually somewhere around 3GB
rte_malloc() succeeds. I'm running on a physical (non-VM) NUMA machine
with 2 physical CPUs, each having 64GBs of local memory. The machine
also runs Ubuntu 12.04 server. I've created 16384 hugepages of 2MB:

echo 16384 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

I'm running the basic app like this:

sudo  numactl --membind=0 ~/src/test/dpdk_test/alloc -c 1f -n 4 -w
04:00.0 --socket-mem=16384,0 -- 4832038656

I'm trying to run only on NUMA node 0 and only allocate memory from
there - that's what the app I'm moving to the DPDK works like (using
numactl --membind=x and --cpunodebind=x).

Is there an upper limit on the amount of memory rte_malloc() will try
to allocate? I tried both after a reboot and when the machine had been
running for a while with not much success. Am I missing something?
It's a bit weird to be only able to allocate 3GB out of the 32GB
assigned to the app...

On a related note, what would be a good way to compile the DPDK with
debug info (and preferably -O0)? There's quite a web of .mk files used
and I haven't figured out where the optimization level / debug options
are set.

Thanks in advance,
Stefan.