[dpdk-dev] buffer allocation failure in NUMA platform

[dpdk-dev] buffer allocation failure in NUMA platform

[Kypriotis Angelos]

Hi,

Here is my topology.

*************************************************************
NUMA Topology
*************************************************************
NUMA domains: 2
-------------------------------------------------------------
Domain 0:
Processors:  0 1 2 3 8 9 10 11
Relative distance to nodes:  10 21
Memory: 5815.68 MB free of total 5977.69 MB
-------------------------------------------------------------
Domain 1:
Processors:  4 5 6 7 12 13 14 15
Relative distance to nodes:  21 10
Memory: 4893.15 MB free of total 6058.52 MB

I use dpdk-1.6.0r1 and I have setup 1024 pages of 2M per NUMA node, as early
as possible ( boot time ) using the hugepages=2048 parameter.


cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
--> 1024
cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
--> 1024

When I try to pin my process to specific nodes like this :

numactl --physcpubind=12-15 -- ./my_simulator -c f000 -n 2 -b 
0000:01:00.0 --socket-mem=0,2048 -- my_simulator.conf  ,

I get 'RING:cannot reserve memory' error. Here is the relevant part :

EAL: Setting up memory...
EAL: Ask a virtual area of 0x1200000 bytes
EAL: Virtual area found at 0x7f217a600000 (size = 0x1200000)
EAL: Ask a virtual area of 0x3ec00000 bytes
EAL: Virtual area found at 0x7f213b800000 (size = 0x3ec00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f213b400000 (size = 0x200000)
EAL: Ask a virtual area of 0x3fc00000 bytes
EAL: Virtual area found at 0x7f20fb600000 (size = 0x3fc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f20fb200000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f20fae00000 (size = 0x200000)
EAL: Ask a virtual area of 0x7fc00000 bytes
EAL: Virtual area found at 0x7f207b000000 (size = 0x7fc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f207ac00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f207a800000 (size = 0x200000)
EAL: Requesting 1024 pages of size 2MB from socket 1
EAL: TSC frequency is ~1600000 KHz
EAL: Master core 12 is ready (tid=81c69840)
EAL: Core 13 is ready (tid=7fd55700)
EAL: Core 15 is ready (tid=7ed53700)
EAL: Core 14 is ready (tid=7f554700)
RING: Cannot reserve memory
Failed to allocate packet buffer pool.

If I do not use --socket-mem=0,2048 memory ( 1024 pages ) is requested 
from both sockets 0 and 1 and it works, but I want to use node 1 only.

In addition in order to request for 1024 pages in node 1 I must pass the 
--socket-mem parameter as above.  I would expect that what I should pass 
is --socket-mem=0,1024, but then the actual memory requested is 512.  Am 
I missing something here or is it a bug?

Regards,

-- 
Angel Kypriotis
MBB LC EPC GW RD Athens PET GR
Nokia
14 Km Athinon-Lamias National Road
145 64 N. Kifissia / Athens / Hellas

Re: [dpdk-dev] buffer allocation failure in NUMA platform

[Burakov, Anatoly]

Hi Kypriotis,

> cat /sys/devices/system/node/node1/hugepages/hugepages-
> 2048kB/nr_hugepages
> --> 1024
> cat /sys/devices/system/node/node0/hugepages/hugepages-
> 2048kB/nr_hugepages
> --> 1024
> 
> When I try to pin my process to specific nodes like this :
> 
> numactl --physcpubind=12-15 -- ./my_simulator -c f000 -n 2 -b
> 0000:01:00.0 --socket-mem=0,2048 -- my_simulator.conf  ,
> 
> I get 'RING:cannot reserve memory' error. Here is the relevant part :
> 
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x1200000 bytes
> EAL: Virtual area found at 0x7f217a600000 (size = 0x1200000)
> EAL: Ask a virtual area of 0x3ec00000 bytes
> EAL: Virtual area found at 0x7f213b800000 (size = 0x3ec00000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f213b400000 (size = 0x200000)
> EAL: Ask a virtual area of 0x3fc00000 bytes
> EAL: Virtual area found at 0x7f20fb600000 (size = 0x3fc00000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f20fb200000 (size = 0x200000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f20fae00000 (size = 0x200000)
> EAL: Ask a virtual area of 0x7fc00000 bytes
> EAL: Virtual area found at 0x7f207b000000 (size = 0x7fc00000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f207ac00000 (size = 0x200000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f207a800000 (size = 0x200000)
> EAL: Requesting 1024 pages of size 2MB from socket 1
> EAL: TSC frequency is ~1600000 KHz
> EAL: Master core 12 is ready (tid=81c69840)
> EAL: Core 13 is ready (tid=7fd55700)
> EAL: Core 15 is ready (tid=7ed53700)
> EAL: Core 14 is ready (tid=7f554700)
> RING: Cannot reserve memory
> Failed to allocate packet buffer pool.
> 
> If I do not use --socket-mem=0,2048 memory ( 1024 pages ) is requested
> from both sockets 0 and 1 and it works, but I want to use node 1 only.
> 
> In addition in order to request for 1024 pages in node 1 I must pass the --
> socket-mem parameter as above.  I would expect that what I should pass is --
> socket-mem=0,1024, but then the actual memory requested is 512.  Am I
> missing something here or is it a bug?

This is by design. If you want memory from specific socket, use --socket-mem. The "512" number you get is the number of pages. Each page is 2MB in size, so that amounts to 512 * 2MB == 1024MB requested, just as you specified on your command-line.

Best regards,
Anatoly Burakov
DPDK SW Engineer