From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <bruce.richardson@intel.com>
Received: from mga03.intel.com (mga03.intel.com [134.134.136.65])
by dpdk.org (Postfix) with ESMTP id 1C40B234
for <dev@dpdk.org>; Fri, 6 Feb 2015 12:00:18 +0100 (CET)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
by orsmga103.jf.intel.com with ESMTP; 06 Feb 2015 02:55:31 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.09,529,1418112000"; d="scan'208";a="662499431"
Received: from bricha3-mobl3.ger.corp.intel.com ([10.243.20.42])
by fmsmga001.fm.intel.com with SMTP; 06 Feb 2015 03:00:15 -0800
Received: by (sSMTP sendmail emulation); Fri, 06 Feb 2015 11:00:14 +0025
Date: Fri, 6 Feb 2015 11:00:14 +0000
From: Bruce Richardson <bruce.richardson@intel.com>
To: Stefan Puiu <stefan.puiu@gmail.com>
Message-ID: <20150206110014.GA16144@bricha3-MOBL3>
References: <CACKs7VDkpaBH4Gv1iWmiqmwjdPJnjL0nnWB2JYqWNqdW3+j-LQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CACKs7VDkpaBH4Gv1iWmiqmwjdPJnjL0nnWB2JYqWNqdW3+j-LQ@mail.gmail.com>
Organization: Intel Shannon Ltd.
User-Agent: Mutt/1.5.23 (2014-03-12)
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] upper limit on the size of allocation through
rte_malloc in dpdk-1.8.0?
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 06 Feb 2015 11:00:19 -0000
On Wed, Feb 04, 2015 at 05:24:58PM +0200, Stefan Puiu wrote:
> Hi,
>
> I'm trying to alter an existing program to use the Intel DPDK. I'm
> using 1.8.0, compiled by me as a shared library
> (CONFIG_RTE_BUILD_COMBINE_LIBS=y and CONFIG_RTE_BUILD_SHARED_LIB=y in
> .config) on Ubuntu 12.04. The program needs to allocate large blocks
> of memory (between 1 and 4 chunks of 4.5GB, also 1-4 chunks of 2.5
> GB). I tried changing my C++ code to use an array allocated using
> rte_malloc() instead of the std::vector I was using beforehand, but it
> seems the call to rte_malloc() fails. I then made a simple test
> program using the DPDK that takes a size to allocate and if that
> fails, tries again with sizes of 100MB less, basically the code below.
> This is C++ code (well, now that I look it could've been plain C, but
> I need C++) compiled with g++-4.6 with '-std=gnu++0x':
>
> int main(int argc, char **argv)
> {
> int ret = rte_eal_init(argc, argv);
> if (ret < 0)
> rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
> argc -= ret;
> argv += ret;
>
> [... check argc >= 2]
> size_t size = strtoul(argv[1], NULL, 10);
> size_t s = size;
>
> for (size_t i = 0; i < 30; ++i) {
> printf("Trying to allocate %'zu bytes\n", s);
> buf = rte_malloc("test", s, 0);
> if (!buf)
> printf ("Failed!\n");
> else {
> printf ("Success!\n");
> rte_free(buf);
> break;
> }
>
> s = s - (100 * 1024ULL * 1024ULL);
> }
>
> return 0;
> }
>
> I'm getting:
> Trying to allocate 4,832,038,656 bytes
> Failed!
> Trying to allocate 4,727,181,056 bytes
> Failed!
> [...]
> Trying to allocate 2,944,601,856 bytes
> Success!
>
> It's not always the same value, but usually somewhere around 3GB
> rte_malloc() succeeds. I'm running on a physical (non-VM) NUMA machine
> with 2 physical CPUs, each having 64GBs of local memory. The machine
> also runs Ubuntu 12.04 server. I've created 16384 hugepages of 2MB:
>
> echo 16384 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>
> I'm running the basic app like this:
>
> sudo numactl --membind=0 ~/src/test/dpdk_test/alloc -c 1f -n 4 -w
> 04:00.0 --socket-mem=16384,0 -- 4832038656
>
> I'm trying to run only on NUMA node 0 and only allocate memory from
> there - that's what the app I'm moving to the DPDK works like (using
> numactl --membind=x and --cpunodebind=x).
>
> Is there an upper limit on the amount of memory rte_malloc() will try
> to allocate? I tried both after a reboot and when the machine had been
> running for a while with not much success. Am I missing something?
> It's a bit weird to be only able to allocate 3GB out of the 32GB
> assigned to the app...
>
> On a related note, what would be a good way to compile the DPDK with
> debug info (and preferably -O0)? There's quite a web of .mk files used
> and I haven't figured out where the optimization level / debug options
> are set.
>
> Thanks in advance,
> Stefan.
Does your system support 1G pages? I would recommend using a smaller number of
1G pages vs the huge number of 2MB pages that you are currently using. There
may be issues with the allocations failing due to a lack of contiguous blocks
of memory due to the 2MB pages being spread across memory.
Regards,
/Bruce