Issue with Cannot allocate memory when using 32-bit DPDK application

Issue with Cannot allocate memory when using 32-bit DPDK application

[洪全]

[-- Attachment #1: Type: text/plain, Size: 2667 bytes --]

Dear DPDK community,


I am encountering an issue when attempting to run a 32-bit DPDK application on Linux. Specifically, I am facing a "Cannot allocate memory" error during initialization. While I can mitigate this issue by using the `--no-huge` option, it adversely affects the performance of my application.


Here is the error output I receive:


```
sudo ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 --vdev=net_tap001,iface=tap001 --no-pci
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```


When debugging with `--log-level=eal,8`, the relevant portion of the output indicates attempts to allocate memory:


```
EAL: Attempting to preallocate 2048M on socket 0
EAL: Ask a virtual area of 0xc000 bytes
EAL: Virtual area found at 0xeb077000 (size = 0xc000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```


System information:
- Hugepages configured: `echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`
- Hugepages mounted: `mount -t hugetlbfs hugetlbfs /dev/hugepages`
- NUMA node information: `numactl --hardware`
  - available: 1 nodes (0)
  - node 0 cpus: 0 1
  - node 0 size: 7896 MB
  - node 0 free: 3915 MB
  - node distances:
    - node 0: 10


DPDK version: 22.03
Distribution: Ubuntu 22.04
Kernel information: Linux hq-virtual-machine 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux


I believe the issue stems from the attempt to preallocate 2048M on socket 0, but using `-m` or `--socket-mem` options did not resolve the problem.


Could you please provide guidance on how to properly configure DPDK to avoid this memory allocation issue while maximizing performance?


Thank you for your assistance.




Best regards,

Quan Hong

[-- Attachment #2: Type: text/html, Size: 5599 bytes --]

Re: Issue with Cannot allocate memory when using 32-bit DPDK application

[Stephen Hemminger]

On Wed, 3 Jul 2024 10:18:34 +0800 (GMT+08:00)
洪全 <hongquan@iie.ac.cn> wrote:

> My CPU architecture is x86-64

If you have x86 then there is no good reason to bother with 32 bit.
The lack of virtual space is going to limit you probably to 1Gb or less of huge pages.

Re: Issue with Cannot allocate memory when using 32-bit DPDK application

[Stephen Hemminger]

On Tue, 2 Jul 2024 18:17:08 +0800 (GMT+08:00)
洪全 <hongquan@iie.ac.cn> wrote:

> Dear DPDK community,
> 
> 
> I am encountering an issue when attempting to run a 32-bit DPDK application on Linux. Specifically, I am facing a "Cannot allocate memory" error during initialization. While I can mitigate this issue by using the `--no-huge` option, it adversely affects the performance of my application.
> 
> 
> Here is the error output I receive:
> 
> 
> ```
> sudo ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 --vdev=net_tap001,iface=tap001 --no-pci
> EAL: Detected CPU lcores: 2
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: Cannot get a virtual area: Cannot allocate memory
> EAL: Cannot allocate VA space for memseg list, retrying with different page size
> EAL: Cannot allocate VA space on socket 0
> EAL: FATAL: Cannot init memory
> EAL: Cannot init memory
> app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
> Aborted
> ```
> 
> 
> When debugging with `--log-level=eal,8`, the relevant portion of the output indicates attempts to allocate memory:
> 
> 
> ```
> EAL: Attempting to preallocate 2048M on socket 0
> EAL: Ask a virtual area of 0xc000 bytes
> EAL: Virtual area found at 0xeb077000 (size = 0xc000)
> EAL: Memseg list allocated at socket 0, page size 0x800kB
> EAL: Ask a virtual area of 0x80000000 bytes
> EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
> EAL: Cannot get a virtual area: Cannot allocate memory
> EAL: Cannot allocate VA space for memseg list, retrying with different page size
> EAL: Cannot allocate VA space on socket 0
> EAL: FATAL: Cannot init memory
> EAL: Cannot init memory
> app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
> Aborted
> ```
> 
> 
> System information:
> - Hugepages configured: `echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`
> - Hugepages mounted: `mount -t hugetlbfs hugetlbfs /dev/hugepages`
> - NUMA node information: `numactl --hardware`
>   - available: 1 nodes (0)
>   - node 0 cpus: 0 1
>   - node 0 size: 7896 MB
>   - node 0 free: 3915 MB
>   - node distances:
>     - node 0: 10
> 
> 
> DPDK version: 22.03
> Distribution: Ubuntu 22.04
> Kernel information: Linux hq-virtual-machine 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> I believe the issue stems from the attempt to preallocate 2048M on socket 0, but using `-m` or `--socket-mem` options did not resolve the problem.
> 
> 
> Could you please provide guidance on how to properly configure DPDK to avoid this memory allocation issue while maximizing performance?

What CPU architecture? Traditionally on 32 bit x86 has 3GB for userspace and 1GB is the
shadow kernel. Some other architecture may do the same thing.

In userspace, you then have some memory for programs code, data and stack. The
DPDK EAL init then tries to map all of available huge pages (2G) and fails to find
enough contiguous virtual address space to fit.

Try smaller amount of huge pages.

Issue with Cannot allocate memory when using 32-bit DPDK application

[洪全]

[-- Attachment #1: Type: text/plain, Size: 2667 bytes --]

Dear DPDK community,


I am encountering an issue when attempting to run a 32-bit DPDK application on Linux. Specifically, I am facing a "Cannot allocate memory" error during initialization. While I can mitigate this issue by using the `--no-huge` option, it adversely affects the performance of my application.


Here is the error output I receive:


```
sudo ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 --vdev=net_tap001,iface=tap001 --no-pci
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```


When debugging with `--log-level=eal,8`, the relevant portion of the output indicates attempts to allocate memory:


```
EAL: Attempting to preallocate 2048M on socket 0
EAL: Ask a virtual area of 0xc000 bytes
EAL: Virtual area found at 0xeb077000 (size = 0xc000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```


System information:
- Hugepages configured: `echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`
- Hugepages mounted: `mount -t hugetlbfs hugetlbfs /dev/hugepages`
- NUMA node information: `numactl --hardware`
  - available: 1 nodes (0)
  - node 0 cpus: 0 1
  - node 0 size: 7896 MB
  - node 0 free: 3915 MB
  - node distances:
    - node 0: 10


DPDK version: 22.03
Distribution: Ubuntu 22.04
Kernel information: Linux hq-virtual-machine 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux


I believe the issue stems from the attempt to preallocate 2048M on socket 0, but using `-m` or `--socket-mem` options did not resolve the problem.


Could you please provide guidance on how to properly configure DPDK to avoid this memory allocation issue while maximizing performance?


Thank you for your assistance.




Best regards,

Quan Hong

[-- Attachment #2: Type: text/html, Size: 5601 bytes --]