Re: [dpdk-dev] rte_mempool_create fails with ENOMEM

[Newman Poborsky <newman555p@gmail.com>]

On Sat, Dec 20, 2014 at 2:34 AM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> You can reserve hugepages on the kernel cmdline (GRUB).

Great, thanks, I'll try that!

Newman

>
> On Fri, Dec 19, 2014 at 12:13 PM, Newman Poborsky <newman555p@gmail.com>
> wrote:
>>
>> On Thu, Dec 18, 2014 at 9:03 PM, Ananyev, Konstantin <
>> konstantin.ananyev@intel.com> wrote:
>>
>> >
>> >
>> > > -----Original Message-----
>> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev,
>> > > Konstantin
>> > > Sent: Thursday, December 18, 2014 5:43 PM
>> > > To: Newman Poborsky; dev@dpdk.org
>> > > Subject: Re: [dpdk-dev] rte_mempool_create fails with ENOMEM
>> > >
>> > > Hi
>> > >
>> > > > -----Original Message-----
>> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Newman Poborsky
>> > > > Sent: Thursday, December 18, 2014 1:26 PM
>> > > > To: dev@dpdk.org
>> > > > Subject: [dpdk-dev] rte_mempool_create fails with ENOMEM
>> > > >
>> > > > Hi,
>> > > >
>> > > > could someone please provide any explanation why sometimes mempool
>> > creation
>> > > > fails with ENOMEM?
>> > > >
>> > > > I run my test app several times without any problems and then I
>> > > > start
>> > > > getting ENOMEM error when creating mempool that are used for
>> > > > packets.
>> > I try
>> > > > to delete everything from /mnt/huge, I increase the number of huge
>> > pages,
>> > > > remount /mnt/huge but nothing helps.
>> > > >
>> > > > There is more than enough memory on server. I tried to debug
>> > > > rte_mempool_create() call and it seems that after server is
>> > > > restarted
>> > free
>> > > > mem segments are bigger than 2MB, but after running test app for
>> > several
>> > > > times, it seems that all free mem segments have a size of 2MB, and
>> > since I
>> > > > am requesting 8MB for my packet mempool, this fails.  I'm not really
>> > sure
>> > > > that this conclusion is correct.
>> > >
>> > > Yes,rte_mempool_create uses  rte_memzone_reserve() to allocate
>> > > single physically continuous chunk of memory.
>> > > If no such chunk exist, then it would fail.
>> > > Why physically continuous?
>> > > Main reason - to make things easier for us, as in that case we don't
>> > have to worry
>> > > about situation when mbuf crosses page boundary.
>> > > So you can overcome that problem like that:
>> > > Allocate max amount of memory you would need to hold all mbufs in
>> > > worst
>> > case (all pages physically disjoint)
>> > > using rte_malloc().
>> >
>> > Actually my wrong: rte_malloc()s wouldn't help you here.
>> > You probably need to allocate some external (not managed by EAL) memory
>> > in
>> > that case,
>> > may be mmap() with MAP_HUGETLB, or something similar.
>> >
>> > > Figure out it's physical mappings.
>> > > Call  rte_mempool_xmem_create().
>> > > You can look at: app/test-pmd/mempool_anon.c as a reference.
>> > > It uses same approach to create mempool over 4K pages.
>> > >
>> > > We probably add similar function into mempool API
>> > (create_scatter_mempool or something)
>> > > or just add a new flag (USE_SCATTER_MEM) into rte_mempool_create().
>> > > Though right now it is not there.
>> > >
>> > > Another quick alternative - use 1G pages.
>> > >
>> > > Konstantin
>> >
>>
>>
>> Ok, thanks for the explanation. I understand that this is probably an OS
>> question more than DPDK, but is there a way to again allocate a contiguous
>> memory for n-th run of my test app?  It seems that hugepages get
>> divded/separated to individual 2MB hugepage. Shouldn't OS's memory
>> management system try to group those hupages back to one contiguous chunk
>> once my app/process is done?   Again, I know very little about Linux
>> memory
>> management and hugepages, so forgive me if this is a stupid question.
>> Is rebooting the OS the only way to deal with this problem?  Or should I
>> just try to use 1GB hugepages?
>>
>> p.s. Konstantin, sorry for the double reply, I accidentally forgot to
>> include dev list in my first reply  :)
>>
>> Newman
>>
>> >
>> > > >
>> > > > Does anybody have any idea what to check and how running my test app
>> > > > several times affects hugepages?
>> > > >
>> > > > For me, this doesn't make any since because after test app exits,
>> > resources
>> > > > should be freed, right?
>> > > >
>> > > > This has been driving me crazy for days now. I tried reading a bit
>> > > > more
>> > > > theory about hugepages, but didn't find out anything that could help
>> > me.
>> > > > Maybe it's something else and completely trivial, but I can't figure
>> > > > it
>> > > > out, so any help is appreciated.
>> > > >
>> > > > Thank you!
>> > > >
>> > > > BR,
>> > > > Newman P.
>> >
>
>