* Issue with Cannot allocate memory when using 32-bit DPDK application
@ 2024-07-02 10:17 洪全
2024-07-02 15:08 ` Stephen Hemminger
0 siblings, 1 reply; 6+ messages in thread
From: 洪全 @ 2024-07-02 10:17 UTC (permalink / raw)
To: users
[-- Attachment #1: Type: text/plain, Size: 2667 bytes --]
Dear DPDK community,
I am encountering an issue when attempting to run a 32-bit DPDK application on Linux. Specifically, I am facing a "Cannot allocate memory" error during initialization. While I can mitigate this issue by using the `--no-huge` option, it adversely affects the performance of my application.
Here is the error output I receive:
```
sudo ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 --vdev=net_tap001,iface=tap001 --no-pci
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```
When debugging with `--log-level=eal,8`, the relevant portion of the output indicates attempts to allocate memory:
```
EAL: Attempting to preallocate 2048M on socket 0
EAL: Ask a virtual area of 0xc000 bytes
EAL: Virtual area found at 0xeb077000 (size = 0xc000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```
System information:
- Hugepages configured: `echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`
- Hugepages mounted: `mount -t hugetlbfs hugetlbfs /dev/hugepages`
- NUMA node information: `numactl --hardware`
- available: 1 nodes (0)
- node 0 cpus: 0 1
- node 0 size: 7896 MB
- node 0 free: 3915 MB
- node distances:
- node 0: 10
DPDK version: 22.03
Distribution: Ubuntu 22.04
Kernel information: Linux hq-virtual-machine 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
I believe the issue stems from the attempt to preallocate 2048M on socket 0, but using `-m` or `--socket-mem` options did not resolve the problem.
Could you please provide guidance on how to properly configure DPDK to avoid this memory allocation issue while maximizing performance?
Thank you for your assistance.
Best regards,
Quan Hong
[-- Attachment #2: Type: text/html, Size: 5601 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Issue with Cannot allocate memory when using 32-bit DPDK application
2024-07-02 10:17 Issue with Cannot allocate memory when using 32-bit DPDK application 洪全
@ 2024-07-02 15:08 ` Stephen Hemminger
2024-07-03 2:18 ` 洪全
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Hemminger @ 2024-07-02 15:08 UTC (permalink / raw)
To: 洪全; +Cc: users
On Tue, 2 Jul 2024 18:17:08 +0800 (GMT+08:00)
洪全 <hongquan@iie.ac.cn> wrote:
> Dear DPDK community,
>
>
> I am encountering an issue when attempting to run a 32-bit DPDK application on Linux. Specifically, I am facing a "Cannot allocate memory" error during initialization. While I can mitigate this issue by using the `--no-huge` option, it adversely affects the performance of my application.
>
>
> Here is the error output I receive:
>
>
> ```
> sudo ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 --vdev=net_tap001,iface=tap001 --no-pci
> EAL: Detected CPU lcores: 2
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: Cannot get a virtual area: Cannot allocate memory
> EAL: Cannot allocate VA space for memseg list, retrying with different page size
> EAL: Cannot allocate VA space on socket 0
> EAL: FATAL: Cannot init memory
> EAL: Cannot init memory
> app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
> Aborted
> ```
>
>
> When debugging with `--log-level=eal,8`, the relevant portion of the output indicates attempts to allocate memory:
>
>
> ```
> EAL: Attempting to preallocate 2048M on socket 0
> EAL: Ask a virtual area of 0xc000 bytes
> EAL: Virtual area found at 0xeb077000 (size = 0xc000)
> EAL: Memseg list allocated at socket 0, page size 0x800kB
> EAL: Ask a virtual area of 0x80000000 bytes
> EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
> EAL: Cannot get a virtual area: Cannot allocate memory
> EAL: Cannot allocate VA space for memseg list, retrying with different page size
> EAL: Cannot allocate VA space on socket 0
> EAL: FATAL: Cannot init memory
> EAL: Cannot init memory
> app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
> Aborted
> ```
>
>
> System information:
> - Hugepages configured: `echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`
> - Hugepages mounted: `mount -t hugetlbfs hugetlbfs /dev/hugepages`
> - NUMA node information: `numactl --hardware`
> - available: 1 nodes (0)
> - node 0 cpus: 0 1
> - node 0 size: 7896 MB
> - node 0 free: 3915 MB
> - node distances:
> - node 0: 10
>
>
> DPDK version: 22.03
> Distribution: Ubuntu 22.04
> Kernel information: Linux hq-virtual-machine 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
>
>
> I believe the issue stems from the attempt to preallocate 2048M on socket 0, but using `-m` or `--socket-mem` options did not resolve the problem.
>
>
> Could you please provide guidance on how to properly configure DPDK to avoid this memory allocation issue while maximizing performance?
What CPU architecture? Traditionally on 32 bit x86 has 3GB for userspace and 1GB is the
shadow kernel. Some other architecture may do the same thing.
In userspace, you then have some memory for programs code, data and stack. The
DPDK EAL init then tries to map all of available huge pages (2G) and fails to find
enough contiguous virtual address space to fit.
Try smaller amount of huge pages.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: Issue with Cannot allocate memory when using 32-bit DPDK application
2024-07-02 15:08 ` Stephen Hemminger
@ 2024-07-03 2:18 ` 洪全
2024-07-03 5:43 ` Stephen Hemminger
0 siblings, 1 reply; 6+ messages in thread
From: 洪全 @ 2024-07-03 2:18 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: users
> -----原始邮件-----
> 发件人: "Stephen Hemminger" <stephen@networkplumber.org>
> 发送时间: 2024-07-02 23:08:37 (星期二)
> 收件人: "洪全" <hongquan@iie.ac.cn>
> 抄送: users@dpdk.org
> 主题: Re: Issue with Cannot allocate memory when using 32-bit DPDK application
>
> On Tue, 2 Jul 2024 18:17:08 +0800 (GMT+08:00)
> 洪全 <hongquan@iie.ac.cn> wrote:
>
> > Dear DPDK community,
> >
> >
> > I am encountering an issue when attempting to run a 32-bit DPDK application on Linux. Specifically, I am facing a "Cannot allocate memory" error during initialization. While I can mitigate this issue by using the `--no-huge` option, it adversely affects the performance of my application.
> >
> >
> > Here is the error output I receive:
> >
> >
> > ```
> > sudo ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 --vdev=net_tap001,iface=tap001 --no-pci
> > EAL: Detected CPU lcores: 2
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected shared linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
> > EAL: Selected IOVA mode 'PA'
> > EAL: Cannot get a virtual area: Cannot allocate memory
> > EAL: Cannot allocate VA space for memseg list, retrying with different page size
> > EAL: Cannot allocate VA space on socket 0
> > EAL: FATAL: Cannot init memory
> > EAL: Cannot init memory
> > app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
> > Aborted
> > ```
> >
> >
> > When debugging with `--log-level=eal,8`, the relevant portion of the output indicates attempts to allocate memory:
> >
> >
> > ```
> > EAL: Attempting to preallocate 2048M on socket 0
> > EAL: Ask a virtual area of 0xc000 bytes
> > EAL: Virtual area found at 0xeb077000 (size = 0xc000)
> > EAL: Memseg list allocated at socket 0, page size 0x800kB
> > EAL: Ask a virtual area of 0x80000000 bytes
> > EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
> > EAL: Cannot get a virtual area: Cannot allocate memory
> > EAL: Cannot allocate VA space for memseg list, retrying with different page size
> > EAL: Cannot allocate VA space on socket 0
> > EAL: FATAL: Cannot init memory
> > EAL: Cannot init memory
> > app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
> > Aborted
> > ```
> >
> >
> > System information:
> > - Hugepages configured: `echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`
> > - Hugepages mounted: `mount -t hugetlbfs hugetlbfs /dev/hugepages`
> > - NUMA node information: `numactl --hardware`
> > - available: 1 nodes (0)
> > - node 0 cpus: 0 1
> > - node 0 size: 7896 MB
> > - node 0 free: 3915 MB
> > - node distances:
> > - node 0: 10
> >
> >
> > DPDK version: 22.03
> > Distribution: Ubuntu 22.04
> > Kernel information: Linux hq-virtual-machine 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> > I believe the issue stems from the attempt to preallocate 2048M on socket 0, but using `-m` or `--socket-mem` options did not resolve the problem.
> >
> >
> > Could you please provide guidance on how to properly configure DPDK to avoid this memory allocation issue while maximizing performance?
>
> What CPU architecture? Traditionally on 32 bit x86 has 3GB for userspace and 1GB is the
> shadow kernel. Some other architecture may do the same thing.
>
> In userspace, you then have some memory for programs code, data and stack. The
> DPDK EAL init then tries to map all of available huge pages (2G) and fails to find
> enough contiguous virtual address space to fit.
>
> Try smaller amount of huge pages.
My CPU architecture is x86-64.
Does trying a small number of huge pages mean allocating several 1GB huge pages?
According to the output of EAL: mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0). Do I need to allocate 3 1GB huge pages to satisfy this request? As you said, 32 bit x86 has 3GB for userspace. If I allocate all 3GB to huge pages, will there be an error?
In addition, I tried to reduce the amount of memory in the above mmap by using the --socket-mem or -m option, but it did not work.</hongquan@iie.ac.cn></hongquan@iie.ac.cn></stephen@networkplumber.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Issue with Cannot allocate memory when using 32-bit DPDK application
2024-07-03 2:18 ` 洪全
@ 2024-07-03 5:43 ` Stephen Hemminger
2024-07-03 6:04 ` 洪全
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Hemminger @ 2024-07-03 5:43 UTC (permalink / raw)
To: 洪全; +Cc: users
On Wed, 3 Jul 2024 10:18:34 +0800 (GMT+08:00)
洪全 <hongquan@iie.ac.cn> wrote:
> My CPU architecture is x86-64
If you have x86 then there is no good reason to bother with 32 bit.
The lack of virtual space is going to limit you probably to 1Gb or less of huge pages.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: Issue with Cannot allocate memory when using 32-bit DPDK application
2024-07-03 5:43 ` Stephen Hemminger
@ 2024-07-03 6:04 ` 洪全
0 siblings, 0 replies; 6+ messages in thread
From: 洪全 @ 2024-07-03 6:04 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: users
Thank you for your prompt reply.
When I allocated 3 1GB large pages and 128 2MB pages in the system. When I ran it again, the program log output was as follows:
```
EAL: Ask a virtual area of 0x6000 bytes
EAL: Virtual area found at 0xed4a4000 (size = 0x6000)
EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: Bus vdev wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus dsa wants IOVA as 'DC'
EAL: Bus auxiliary wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is not available, selecting IOVA as PA mode.
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x2b000 bytes
EAL: Virtual area found at 0xec477000 (size = 0x2b000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Attempting to preallocate 2048M on socket 0
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0xee701000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Cannot mmap((nil), 0xc0000000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Attempting to preallocate 2048M on socket 0
EAL: Ask a virtual area of 0xc000 bytes
EAL: Virtual area found at 0xec46b000 (size = 0xc000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```
Even if more large pages are allocated, the 32-bit virtual space limit causes mmap to fail.
So does this mean that 32-bit programs cannot use DPDK without disabling large pages?
Is it possible to adjust the 2048M space pre-allocated during DPDK initialization?
> -----原始邮件-----
> 发件人: "Stephen Hemminger" <stephen@networkplumber.org>
> 发送时间: 2024-07-03 13:43:31 (星期三)
> 收件人: "洪全" <hongquan@iie.ac.cn>
> 抄送: users@dpdk.org
> 主题: Re: Issue with Cannot allocate memory when using 32-bit DPDK application
>
> On Wed, 3 Jul 2024 10:18:34 +0800 (GMT+08:00)
> 洪全 <hongquan@iie.ac.cn> wrote:
>
> > My CPU architecture is x86-64
>
> If you have x86 then there is no good reason to bother with 32 bit.
> The lack of virtual space is going to limit you probably to 1Gb or less of huge pages.
</hongquan@iie.ac.cn></hongquan@iie.ac.cn></stephen@networkplumber.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Issue with Cannot allocate memory when using 32-bit DPDK application
@ 2024-07-02 10:11 洪全
0 siblings, 0 replies; 6+ messages in thread
From: 洪全 @ 2024-07-02 10:11 UTC (permalink / raw)
To: users
[-- Attachment #1: Type: text/plain, Size: 2667 bytes --]
Dear DPDK community,
I am encountering an issue when attempting to run a 32-bit DPDK application on Linux. Specifically, I am facing a "Cannot allocate memory" error during initialization. While I can mitigate this issue by using the `--no-huge` option, it adversely affects the performance of my application.
Here is the error output I receive:
```
sudo ./app -l 0-1 --proc-type=primary --file-prefix=pmd1 --vdev=net_tap001,iface=tap001 --no-pci
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```
When debugging with `--log-level=eal,8`, the relevant portion of the output indicates attempts to allocate memory:
```
EAL: Attempting to preallocate 2048M on socket 0
EAL: Ask a virtual area of 0xc000 bytes
EAL: Virtual area found at 0xeb077000 (size = 0xc000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Cannot mmap((nil), 0x80200000, 0x0, 0x22, -1, 0x0): Cannot allocate memory
EAL: Cannot get a virtual area: Cannot allocate memory
EAL: Cannot allocate VA space for memseg list, retrying with different page size
EAL: Cannot allocate VA space on socket 0
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
app: main.c:284: main: Assertion `(ret = rte_eal_init(argc, (char **) argv)) >= 0' failed.
Aborted
```
System information:
- Hugepages configured: `echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`
- Hugepages mounted: `mount -t hugetlbfs hugetlbfs /dev/hugepages`
- NUMA node information: `numactl --hardware`
- available: 1 nodes (0)
- node 0 cpus: 0 1
- node 0 size: 7896 MB
- node 0 free: 3915 MB
- node distances:
- node 0: 10
DPDK version: 22.03
Distribution: Ubuntu 22.04
Kernel information: Linux hq-virtual-machine 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
I believe the issue stems from the attempt to preallocate 2048M on socket 0, but using `-m` or `--socket-mem` options did not resolve the problem.
Could you please provide guidance on how to properly configure DPDK to avoid this memory allocation issue while maximizing performance?
Thank you for your assistance.
Best regards,
Quan Hong
[-- Attachment #2: Type: text/html, Size: 5599 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-07-03 6:04 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-07-02 10:17 Issue with Cannot allocate memory when using 32-bit DPDK application 洪全
2024-07-02 15:08 ` Stephen Hemminger
2024-07-03 2:18 ` 洪全
2024-07-03 5:43 ` Stephen Hemminger
2024-07-03 6:04 ` 洪全
-- strict thread matches above, loose matches on Subject: below --
2024-07-02 10:11 洪全
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).