[dpdk-dev] Is contiguous hugepages memory still required in latest DPDKs?

[dpdk-dev] Is contiguous hugepages memory still required in latest DPDKs?

[Ruslan Bilovol]

Hi,

Recently after moving to 4.4 Linux kernel we found that DPDK v16.07
can't find physically contiguous chunk of hugepages memory.

I've tracked this down to kernel commits
  81c0a2bb515f ("mm: page_alloc: fair zone allocator policy")
  fff4068cba48 ("mm: page_alloc: revert NUMA aspect of fair allocation policy")
  4ffeaf3560a5 ("mm: page_alloc: reduce cost of the fair zone allocation policy")

These commits changed default page allocator behavior so it now allocates
memory proportionally from preferred and lower zones. Hugepages
are scattered proportionally among few memory zones, so possibility
to find big physically contiguous chunk of hugepages memory is much lower.

I see that there were some attempts to move from contiguous hugepages
approach, like http://dpdk.org/ml/archives/dev/2016-March/035201.html
Also some discussion here: http://dpdk.org/ml/archives/users/2016-October/001050.html

The question: is contiguous hugepages memory still required in latest DPDKs,
and if not, since which version?

Thanks,
Ruslan

Re: [dpdk-dev] Is contiguous hugepages memory still required in latest DPDKs?

[Sergio Gonzalez Monroy]

The DPDK will still try to allocate contiguous hugepages.
Since DPDK v16.07 the mempool library is able to create a mempool with 
multiple chunks of memory.
You would still have the contig memory restriction for any other DPDK 
library or app.

Are you pre-allocating hugepages on boot (kernel cmdline) or runtime?

Sergio

On 21/03/2017 14:20, Ruslan Bilovol wrote:
> Hi,
>
> Recently after moving to 4.4 Linux kernel we found that DPDK v16.07
> can't find physically contiguous chunk of hugepages memory.
>
> I've tracked this down to kernel commits
>  81c0a2bb515f ("mm: page_alloc: fair zone allocator policy")
>  fff4068cba48 ("mm: page_alloc: revert NUMA aspect of fair allocation 
> policy")
>  4ffeaf3560a5 ("mm: page_alloc: reduce cost of the fair zone 
> allocation policy")
>
> These commits changed default page allocator behavior so it now allocates
> memory proportionally from preferred and lower zones. Hugepages
> are scattered proportionally among few memory zones, so possibility
> to find big physically contiguous chunk of hugepages memory is much 
> lower.
>
> I see that there were some attempts to move from contiguous hugepages
> approach, like http://dpdk.org/ml/archives/dev/2016-March/035201.html
> Also some discussion here: 
> http://dpdk.org/ml/archives/users/2016-October/001050.html
>
> The question: is contiguous hugepages memory still required in latest 
> DPDKs,
> and if not, since which version?
>
> Thanks,
> Ruslan
>

Re: [dpdk-dev] Is contiguous hugepages memory still required in latest DPDKs?

[Ruslan Bilovol]

On 03/21/2017 04:41 PM, Sergio Gonzalez Monroy wrote:
> The DPDK will still try to allocate contiguous hugepages.
> Since DPDK v16.07 the mempool library is able to create a mempool with multiple chunks of memory.
> You would still have the contig memory restriction for any other DPDK library or app.
>
> Are you pre-allocating hugepages on boot (kernel cmdline) or runtime?
I tried both cases, and there is no much difference between them.
(I used debug prints in kernel to see physical address of preallocated hugepages

Main issue is that linux kernel memory allocation algorithm was changed
with "fair zone allocator policy" commits, and in my case need to
pre-allocate 2x or even 3x more hugepages now.

Regards,
Ruslan

>
> Sergio
>
> On 21/03/2017 14:20, Ruslan Bilovol wrote:
>> Hi,
>>
>> Recently after moving to 4.4 Linux kernel we found that DPDK v16.07
>> can't find physically contiguous chunk of hugepages memory.
>>
>> I've tracked this down to kernel commits
>>  81c0a2bb515f ("mm: page_alloc: fair zone allocator policy")
>>  fff4068cba48 ("mm: page_alloc: revert NUMA aspect of fair allocation policy")
>>  4ffeaf3560a5 ("mm: page_alloc: reduce cost of the fair zone allocation policy")
>>
>> These commits changed default page allocator behavior so it now allocates
>> memory proportionally from preferred and lower zones. Hugepages
>> are scattered proportionally among few memory zones, so possibility
>> to find big physically contiguous chunk of hugepages memory is much lower.
>>
>> I see that there were some attempts to move from contiguous hugepages
>> approach, like http://dpdk.org/ml/archives/dev/2016-March/035201.html
>> Also some discussion here: http://dpdk.org/ml/archives/users/2016-October/001050.html
>>
>> The question: is contiguous hugepages memory still required in latest DPDKs,
>> and if not, since which version?
>>
>> Thanks,
>> Ruslan
>>
>

Re: [dpdk-dev] Is contiguous hugepages memory still required in latest DPDKs?

[Ruslan Bilovol]

On 03/22/2017 02:08 PM, Ruslan Bilovol wrote:
> On 03/21/2017 04:41 PM, Sergio Gonzalez Monroy wrote:
>> The DPDK will still try to allocate contiguous hugepages.
>> Since DPDK v16.07 the mempool library is able to create a mempool with multiple chunks of memory.
>> You would still have the contig memory restriction for any other DPDK library or app.

Another question: are there any improvements in contiguous hugepages requirements in latest DPDK (v17.02)?

Thanks,
Ruslan


>>
>> Are you pre-allocating hugepages on boot (kernel cmdline) or runtime?
> I tried both cases, and there is no much difference between them.
> (I used debug prints in kernel to see physical address of preallocated hugepages
>
> Main issue is that linux kernel memory allocation algorithm was changed
> with "fair zone allocator policy" commits, and in my case need to
> pre-allocate 2x or even 3x more hugepages now.
>
> Regards,
> Ruslan
>
>>
>> Sergio
>>
>> On 21/03/2017 14:20, Ruslan Bilovol wrote:
>>> Hi,
>>>
>>> Recently after moving to 4.4 Linux kernel we found that DPDK v16.07
>>> can't find physically contiguous chunk of hugepages memory.
>>>
>>> I've tracked this down to kernel commits
>>>  81c0a2bb515f ("mm: page_alloc: fair zone allocator policy")
>>>  fff4068cba48 ("mm: page_alloc: revert NUMA aspect of fair allocation policy")
>>>  4ffeaf3560a5 ("mm: page_alloc: reduce cost of the fair zone allocation policy")
>>>
>>> These commits changed default page allocator behavior so it now allocates
>>> memory proportionally from preferred and lower zones. Hugepages
>>> are scattered proportionally among few memory zones, so possibility
>>> to find big physically contiguous chunk of hugepages memory is much lower.
>>>
>>> I see that there were some attempts to move from contiguous hugepages
>>> approach, like http://dpdk.org/ml/archives/dev/2016-March/035201.html
>>> Also some discussion here: http://dpdk.org/ml/archives/users/2016-October/001050.html
>>>
>>> The question: is contiguous hugepages memory still required in latest DPDKs,
>>> and if not, since which version?
>>>
>>> Thanks,
>>> Ruslan
>>>
>>
>

Re: [dpdk-dev] Is contiguous hugepages memory still required in latest DPDKs?

[Sergio Gonzalez Monroy]

On 23/03/2017 16:24, Ruslan Bilovol wrote:
> On 03/22/2017 02:08 PM, Ruslan Bilovol wrote:
>> On 03/21/2017 04:41 PM, Sergio Gonzalez Monroy wrote:
>>> The DPDK will still try to allocate contiguous hugepages.
>>> Since DPDK v16.07 the mempool library is able to create a mempool 
>>> with multiple chunks of memory.
>>> You would still have the contig memory restriction for any other 
>>> DPDK library or app.
>
> Another question: are there any improvements in contiguous hugepages 
> requirements in latest DPDK (v17.02)?
>

Not that I can think of.

Sergio

> Thanks,
> Ruslan
>
>
>>>
>>> Are you pre-allocating hugepages on boot (kernel cmdline) or runtime?
>> I tried both cases, and there is no much difference between them.
>> (I used debug prints in kernel to see physical address of 
>> preallocated hugepages
>>
>> Main issue is that linux kernel memory allocation algorithm was changed
>> with "fair zone allocator policy" commits, and in my case need to
>> pre-allocate 2x or even 3x more hugepages now.
>>
>> Regards,
>> Ruslan
>>
>>>
>>> Sergio
>>>
>>> On 21/03/2017 14:20, Ruslan Bilovol wrote:
>>>> Hi,
>>>>
>>>> Recently after moving to 4.4 Linux kernel we found that DPDK v16.07
>>>> can't find physically contiguous chunk of hugepages memory.
>>>>
>>>> I've tracked this down to kernel commits
>>>>  81c0a2bb515f ("mm: page_alloc: fair zone allocator policy")
>>>>  fff4068cba48 ("mm: page_alloc: revert NUMA aspect of fair 
>>>> allocation policy")
>>>>  4ffeaf3560a5 ("mm: page_alloc: reduce cost of the fair zone 
>>>> allocation policy")
>>>>
>>>> These commits changed default page allocator behavior so it now 
>>>> allocates
>>>> memory proportionally from preferred and lower zones. Hugepages
>>>> are scattered proportionally among few memory zones, so possibility
>>>> to find big physically contiguous chunk of hugepages memory is much 
>>>> lower.
>>>>
>>>> I see that there were some attempts to move from contiguous hugepages
>>>> approach, like http://dpdk.org/ml/archives/dev/2016-March/035201.html
>>>> Also some discussion here: 
>>>> http://dpdk.org/ml/archives/users/2016-October/001050.html
>>>>
>>>> The question: is contiguous hugepages memory still required in 
>>>> latest DPDKs,
>>>> and if not, since which version?
>>>>
>>>> Thanks,
>>>> Ruslan
>>>>
>>>
>>
>