Re: [dpdk-dev] rte_mempool_create fails with ENOMEM

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

Hi 

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Newman Poborsky
> Sent: Thursday, December 18, 2014 1:26 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] rte_mempool_create fails with ENOMEM
> 
> Hi,
> 
> could someone please provide any explanation why sometimes mempool creation
> fails with ENOMEM?
> 
> I run my test app several times without any problems and then I start
> getting ENOMEM error when creating mempool that are used for packets. I try
> to delete everything from /mnt/huge, I increase the number of huge pages,
> remount /mnt/huge but nothing helps.
> 
> There is more than enough memory on server. I tried to debug
> rte_mempool_create() call and it seems that after server is restarted free
> mem segments are bigger than 2MB, but after running test app for several
> times, it seems that all free mem segments have a size of 2MB, and since I
> am requesting 8MB for my packet mempool, this fails.  I'm not really sure
> that this conclusion is correct.

Yes,rte_mempool_create uses  rte_memzone_reserve() to allocate
single physically continuous chunk of memory.
If no such chunk exist, then it would fail.
Why physically continuous?
Main reason - to make things easier for us, as in that case we don't have to worry
about situation when mbuf crosses page boundary. 
So you can overcome that problem like that:
Allocate max amount of memory you would need to hold all mbufs in worst case (all pages physically disjoint)
using rte_malloc().
Figure out it's physical mappings.
Call  rte_mempool_xmem_create().
You can look at: app/test-pmd/mempool_anon.c as a reference.
It uses same approach to create mempool over 4K pages.

We probably add similar function into mempool API (create_scatter_mempool or something)
or just add a new flag (USE_SCATTER_MEM) into rte_mempool_create().
Though right now it is not there.

Another quick alternative - use 1G pages.

Konstantin

> 
> Does anybody have any idea what to check and how running my test app
> several times affects hugepages?
> 
> For me, this doesn't make any since because after test app exits, resources
> should be freed, right?
> 
> This has been driving me crazy for days now. I tried reading a bit more
> theory about hugepages, but didn't find out anything that could help me.
> Maybe it's something else and completely trivial, but I can't figure it
> out, so any help is appreciated.
> 
> Thank you!
> 
> BR,
> Newman P.