[dpdk-users] Mempool allocation fails on first boot; but succeeds after system reboot

[dpdk-users] Mempool allocation fails on first boot; but succeeds after system reboot

[Sarthak Ray]

Hi,

I am using dpdk-2.1.0 for a platform appliance, where I am facing issue with mempool allocation.

On the firstboot of the newly installed appliance, my dpdk application is not coming up saying failure in mbuf allocation on socket 0. But once I reboot the system, it comes up without any issues.

I tried "rte_malloc_dump_stats" api to check the heap statistics right before allocating mbuf pools.

Heap Statistics on first boot (with --socket-mem=128,128)
Socket:0
    Heap_size:134215808,
    Free_size:127706432,
    Alloc_size:6509376,
    Greatest_free_size:8388544, // This value is very less than the "contiguous memory block" that my app is trying to allocate
    Alloc_count:29,
    Free_count:31,

Please Note: Increasing --socket-mem value from 128 to 192 has no impact on Greatest_free_size value and I don't see this fragmentation on socket 1.

Heap Statistics after reboot (with --socket-mem=128,128)
Socket:0
    Heap_size:134217600,
    Free_size:127708224,
    Alloc_size:6509376,
    Greatest_free_size:125982080,
    Alloc_count:29,
    Free_count:3,

After reboot, the largest free block size is increased drastically resulting successful mbuf pool allocation. So looks like this is heap fragmentation issue on Socket 0.

Output of "numactl -H" on my sytem
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
node 0 size: 65170 MB
node 0 free: 49476 MB
node 1 cpus: 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
node 1 size: 65536 MB
node 1 free: 50759 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

Kernel Boot Arguments for hugepage setting
hugepagesz=1g hugepages=24

Can anyone please comment on how to address this issue? Is there any way to reserve hugepages that can't be fragmented?

Thanks in advance for the valuable suggestion.

Regards,
Sarthak

Re: [dpdk-users] Mempool allocation fails on first boot; but succeeds after system reboot

[Stephen Hemminger]

On Wed, 17 Aug 2016 13:10:35 +0000
Sarthak Ray <sarthak_ray@outlook.com> wrote:

> Hi,
> 
> I am using dpdk-2.1.0 for a platform appliance, where I am facing issue with mempool allocation.
> 
> On the firstboot of the newly installed appliance, my dpdk application is not coming up saying failure in mbuf allocation on socket 0. But once I reboot the system, it comes up without any issues.

How big is the huge page memory on the system?
On initial boot, the free memory is usually contiguous after a lot of programs
run it gets fragmented.  You can configure grub command line to reserve huge pages
before userspace starts; see kernel documentation Documentation/vm/hugepages.txt

Re: [dpdk-users] Mempool allocation fails on first boot; but succeeds after system reboot

[Sarthak Ray]

Thanks Stephen for your quick response.


My dpdk application uses 2MB hugepages; but I was reserving 1GB hugepages during bootup. Corrected the same and now it is working fine.


Thanks again,

Sarthak

________________________________
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Wednesday, August 17, 2016 9:09:48 PM
To: Sarthak Ray
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Mempool allocation fails on first boot; but succeeds after system reboot

On Wed, 17 Aug 2016 13:10:35 +0000
Sarthak Ray <sarthak_ray@outlook.com> wrote:

> Hi,
>
> I am using dpdk-2.1.0 for a platform appliance, where I am facing issue with mempool allocation.
>
> On the firstboot of the newly installed appliance, my dpdk application is not coming up saying failure in mbuf allocation on socket 0. But once I reboot the system, it comes up without any issues.

How big is the huge page memory on the system?
On initial boot, the free memory is usually contiguous after a lot of programs
run it gets fragmented.  You can configure grub command line to reserve huge pages
before userspace starts; see kernel documentation Documentation/vm/hugepages.txt