strange DPDK memory allocator behavior

strange DPDK memory allocator behavior

[Thomas Monjalon]

Hello,

After merging the lcore variables allocation from Mattias,
we've seen an issue in unit tests running on 32-bit plaftorms:
	https://build.opensuse.org/package/live_build_log/home:bluca:dpdk/dpdk/Debian_Testing/i586
The initialization has this error message "most of the time":
	EAL: Cannot get a virtual area: Cannot allocate memory

David did more tests with this command:
 
# (for i in $(seq 10); do echo | LD_LIBRARY_PATH=lib:drivers app/dpdk-test --no-huge -m 2048 -d drivers; done) |& grep 'Cannot get a virtual area: Cannot allocate memory'

EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory

There are 2 interesting findings:

- the issue disappears when changing allocation size 2048
  to more (4096) or less (1024)

- the issue disappears when removing some big static arrays from unit tests
  like "sarray_lcore_state" in test_lcore_var_perf.c
  Should we avoid having such big arrays in our unit tests?

I don't understand how "-m 2048" is impacted by the static allocations.
And I wonder how it could impact memory fragmentation
so 2048 MB cannot be allocated but more or less are working fine.

It looks like an issue in the DPDK allocator.
Note that we are using --no-huge for these tests.
Any ideas?

Re: strange DPDK memory allocator behavior

[Mattias Rönnblom]

On 2024-11-08 16:03, Thomas Monjalon wrote:
> Hello,
> 
> After merging the lcore variables allocation from Mattias,
> we've seen an issue in unit tests running on 32-bit plaftorms:
> 	https://build.opensuse.org/package/live_build_log/home:bluca:dpdk/dpdk/Debian_Testing/i586
> The initialization has this error message "most of the time":
> 	EAL: Cannot get a virtual area: Cannot allocate memory
> 
> David did more tests with this command:
>   
> # (for i in $(seq 10); do echo | LD_LIBRARY_PATH=lib:drivers app/dpdk-test --no-huge -m 2048 -d drivers; done) |& grep 'Cannot get a virtual area: Cannot allocate memory'
> 
> EAL: Cannot get a virtual area: Cannot allocate memory
> EAL: Cannot get a virtual area: Cannot allocate memory
> EAL: Cannot get a virtual area: Cannot allocate memory
> 
> There are 2 interesting findings:
> 
> - the issue disappears when changing allocation size 2048
>    to more (4096) or less (1024)
> 
> - the issue disappears when removing some big static arrays from unit tests
>    like "sarray_lcore_state" in test_lcore_var_perf.c
>    Should we avoid having such big arrays in our unit tests?

Wouldn't that be just removing a test that triggers a DPDK bug, rather 
than fixing the bug? 128 kB is not big and some other app could easily 
have such objects in BSS.

> 
> I don't understand how "-m 2048" is impacted by the static allocations.

The ~128 kB of BSS will change the addresses of the heap to generally 
somewhat higher numbers, since the heap is mapped in after BSS (I 
think). But with ASLR, I'm not sure it would matter. I'm also not sure 
how ASLR works on 32-bit systems, so maybe it could matter.

Are there any attempts made to mmap() something at some particular 
hardcoded address? Far fetched.

> And I wonder how it could impact memory fragmentation
> so 2048 MB cannot be allocated but more or less are working fine.
> 

If you get an error during EAL initialization, pages to host those 
test_lcore_var_perf static arrays aren't allocated yet.

> It looks like an issue in the DPDK allocator.
> Note that we are using --no-huge for these tests.
> Any ideas?
> 
> 

Re: strange DPDK memory allocator behavior

[Mattias Rönnblom]

On 2024-11-08 17:08, Mattias Rönnblom wrote:
> On 2024-11-08 16:03, Thomas Monjalon wrote:
>> Hello,
>>
>> After merging the lcore variables allocation from Mattias,
>> we've seen an issue in unit tests running on 32-bit plaftorms:
>>     https://build.opensuse.org/package/live_build_log/home:bluca:dpdk/ 
>> dpdk/Debian_Testing/i586
>> The initialization has this error message "most of the time":
>>     EAL: Cannot get a virtual area: Cannot allocate memory
>>
>> David did more tests with this command:
>> # (for i in $(seq 10); do echo | LD_LIBRARY_PATH=lib:drivers app/dpdk- 
>> test --no-huge -m 2048 -d drivers; done) |& grep 'Cannot get a virtual 
>> area: Cannot allocate memory'
>>
>> EAL: Cannot get a virtual area: Cannot allocate memory
>> EAL: Cannot get a virtual area: Cannot allocate memory
>> EAL: Cannot get a virtual area: Cannot allocate memory
>>
>> There are 2 interesting findings:
>>
>> - the issue disappears when changing allocation size 2048
>>    to more (4096) or less (1024)
>>
>> - the issue disappears when removing some big static arrays from unit 
>> tests
>>    like "sarray_lcore_state" in test_lcore_var_perf.c
>>    Should we avoid having such big arrays in our unit tests?
> 
> Wouldn't that be just removing a test that triggers a DPDK bug, rather 
> than fixing the bug? 128 kB is not big and some other app could easily 
> have such objects in BSS.
> 

128 kB should be 16 MB. What is a factor of 128 between friends?

>>
>> I don't understand how "-m 2048" is impacted by the static allocations.
> 
> The ~128 kB of BSS will change the addresses of the heap to generally 
> somewhat higher numbers, since the heap is mapped in after BSS (I 
> think). But with ASLR, I'm not sure it would matter. I'm also not sure 
> how ASLR works on 32-bit systems, so maybe it could matter.
> 
> Are there any attempts made to mmap() something at some particular 
> hardcoded address? Far fetched.
> 
>> And I wonder how it could impact memory fragmentation
>> so 2048 MB cannot be allocated but more or less are working fine.
>>
> 
> If you get an error during EAL initialization, pages to host those 
> test_lcore_var_perf static arrays aren't allocated yet.
> 
>> It looks like an issue in the DPDK allocator.
>> Note that we are using --no-huge for these tests.
>> Any ideas?
>>
>>
>