[dpdk-dev] EAL reports "ot enough memory available on socket 0!"

[dpdk-dev] EAL reports "ot enough memory available on socket 0!"

[Huertas García, Víctor]

Hi all,


This is my first post in this mailing list. I am glad to become a member of this community.


I am a newbie using DPDK and I have tried to compile and execute an example app called "l3fwd-acl".


The DPDK command is DPDK_Test -c 0xf0 -n 2 --socket-mem 64

but the application tells me the following:


EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Cannot obtain physical addresses: Success. Only vfio will function.
EAL: Not enough memory available on socket 0! Requested: 64MB, available: 0MB
EAL: FATAL: Cannot init memory

EAL: Cannot init memory

EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters



I don't know what else I can do to make this app run as I have reserved hugepages as the user guide tells as well as mounted the /mnt/huge folder.


cat /proc/meminfo | grep Huge
AnonHugePages:    520192 kB
HugePages_Total:      64
HugePages_Free:       64
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB


My machine is has this topology (only one Numa node is detected):


udo usertools/cpu_layout.py
[sudo] password for root:
======================================================================
Core and Socket Information (as reported by '/sys/devices/system/cpu')
======================================================================

cores =  [0, 1, 2, 3]
sockets =  [0, 1]

       Socket 0    Socket 1
       --------    --------
Core 0 [0]         [4]
Core 1 [1]         [5]
Core 2 [2]         [6]
Core 3 [3]         [7]



I have compiled the application using Eclipse IDE (g++) under an OpenSuse 42.3 Leap. Everything seemed to compile well and vfio-pci driver could be loaded on two NICs.

So everything seems to be OK but I am totally stuck on this EAL initialization problem.


I would appreciate it if some expert could shed a light on my problem.


Thanks a lot!!



________________________________

Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, contiene información de carácter confidencial exclusivamente dirigida a su destinatario o destinatarios. Si no es vd. el destinatario indicado, queda notificado que la lectura, utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. En el caso de haber recibido este correo electrónico por error, se ruega notificar inmediatamente esta circunstancia mediante reenvío a la dirección electrónica del remitente.
Evite imprimir este mensaje si no es estrictamente necesario.

This email and any file attached to it (when applicable) contain(s) confidential information that is exclusively addressed to its recipient(s). If you are not the indicated recipient, you are informed that reading, using, disseminating and/or copying it without authorisation is forbidden in accordance with the legislation in effect. If you have received this email by mistake, please immediately notify the sender of the situation by resending it to their email address.
Avoid printing this message if it is not absolutely necessary.

Re: [dpdk-dev] EAL reports "ot enough memory available on socket 0!"

[Wiles, Keith]

> On Feb 6, 2018, at 12:38 PM, Huertas García, Víctor <vhuertas@indra.es> wrote:
> 
> Hi all,
> 
> 
> This is my first post in this mailing list. I am glad to become a member of this community.
> 
> 
> I am a newbie using DPDK and I have tried to compile and execute an example app called "l3fwd-acl".
> 
> 
> The DPDK command is DPDK_Test -c 0xf0 -n 2 --socket-mem 64

ran this command as root or used sudo correct?

> 
> but the application tells me the following:
> 
> 
> EAL: Detected 8 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> EAL: Not enough memory available on socket 0! Requested: 64MB, available: 0MB
> EAL: FATAL: Cannot init memory
> 
> EAL: Cannot init memory
> 
> EAL: Error - exiting with code: 1
>  Cause: Invalid EAL parameters
> 
> 
> 
> I don't know what else I can do to make this app run as I have reserved hugepages as the user guide tells as well as mounted the /mnt/huge folder.
> 
> 
> cat /proc/meminfo | grep Huge
> AnonHugePages:    520192 kB
> HugePages_Total:      64
> HugePages_Free:       64
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB

I assume you added an entry into /etc/sysctl.conf and rebooted, just to make sure the huge pages are allocated early in the boot process.

I use 1G huge pages and I assume you are using 2M huge pages

vm.nr_hugepages=32  # 32x2M is 64M

I also disable randomize_va_space, which is not required for those tests.

kernel.randomize_va_space=1


> 
> 
> My machine is has this topology (only one Numa node is detected):
> 
> 
> udo usertools/cpu_layout.py
> [sudo] password for root:
> ======================================================================
> Core and Socket Information (as reported by '/sys/devices/system/cpu')
> ======================================================================
> 
> cores =  [0, 1, 2, 3]
> sockets =  [0, 1]
> 
>       Socket 0    Socket 1
>       --------    --------
> Core 0 [0]         [4]
> Core 1 [1]         [5]
> Core 2 [2]         [6]
> Core 3 [3]         [7]

The CPU layout shows two sockets and normally that means 2 numa node.

I would change the command above —-socket-mem 32,32  (note: two ‘-‘ in front) this command gives 32M socket 0 and 32M to socket 1

I have seen some funny memory configs, with huge pages if you do not use /etc/sysctl.conf and reboot. Sometimes memory will be allocated to one socket.

I hope that helps.

> 
> 
> 
> I have compiled the application using Eclipse IDE (g++) under an OpenSuse 42.3 Leap. Everything seemed to compile well and vfio-pci driver could be loaded on two NICs.
> 
> So everything seems to be OK but I am totally stuck on this EAL initialization problem.
> 
> 
> I would appreciate it if some expert could shed a light on my problem.
> 
> 
> Thanks a lot!!
> 
> 
> 
> ________________________________
> 
> Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, contiene información de carácter confidencial exclusivamente dirigida a su destinatario o destinatarios. Si no es vd. el destinatario indicado, queda notificado que la lectura, utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. En el caso de haber recibido este correo electrónico por error, se ruega notificar inmediatamente esta circunstancia mediante reenvío a la dirección electrónica del remitente.
> Evite imprimir este mensaje si no es estrictamente necesario.
> 
> This email and any file attached to it (when applicable) contain(s) confidential information that is exclusively addressed to its recipient(s). If you are not the indicated recipient, you are informed that reading, using, disseminating and/or copying it without authorisation is forbidden in accordance with the legislation in effect. If you have received this email by mistake, please immediately notify the sender of the situation by resending it to their email address.
> Avoid printing this message if it is not absolutely necessary.

Regards,
Keith

Re: [dpdk-dev] EAL reports "ot enough memory available on socket 0!"

[Bruce Richardson]

On Tue, Feb 06, 2018 at 09:09:26PM +0000, Wiles, Keith wrote:
> 
> 
> > On Feb 6, 2018, at 12:38 PM, Huertas García, Víctor <vhuertas@indra.es> wrote:
> > 
> > Hi all,
> > 
> > 
> > This is my first post in this mailing list. I am glad to become a member of this community.
> > 
> > 
> > I am a newbie using DPDK and I have tried to compile and execute an example app called "l3fwd-acl".
> > 
> > 
> > The DPDK command is DPDK_Test -c 0xf0 -n 2 --socket-mem 64
> 
> ran this command as root or used sudo correct?
> 
> > 
> > but the application tells me the following:
> > 
> > 
> > EAL: Detected 8 lcore(s)
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> > EAL: Not enough memory available on socket 0! Requested: 64MB, available: 0MB
> > EAL: FATAL: Cannot init memory
> > 
> > EAL: Cannot init memory
> > 
> > EAL: Error - exiting with code: 1
> >  Cause: Invalid EAL parameters
> > 
> > 
> > 
> > I don't know what else I can do to make this app run as I have reserved hugepages as the user guide tells as well as mounted the /mnt/huge folder.
> > 
> > 
> > cat /proc/meminfo | grep Huge
> > AnonHugePages:    520192 kB
> > HugePages_Total:      64
> > HugePages_Free:       64
> > HugePages_Rsvd:        0
> > HugePages_Surp:        0
> > Hugepagesize:       2048 kB
> 
> I assume you added an entry into /etc/sysctl.conf and rebooted, just to make sure the huge pages are allocated early in the boot process.
> 
> I use 1G huge pages and I assume you are using 2M huge pages
> 
> vm.nr_hugepages=32  # 32x2M is 64M
> 
> I also disable randomize_va_space, which is not required for those tests.
> 
> kernel.randomize_va_space=1
> 
> 
> > 
> > 
> > My machine is has this topology (only one Numa node is detected):
> > 
> > 
> > udo usertools/cpu_layout.py
> > [sudo] password for root:
> > ======================================================================
> > Core and Socket Information (as reported by '/sys/devices/system/cpu')
> > ======================================================================
> > 
> > cores =  [0, 1, 2, 3]
> > sockets =  [0, 1]
> > 
> >       Socket 0    Socket 1
> >       --------    --------
> > Core 0 [0]         [4]
> > Core 1 [1]         [5]
> > Core 2 [2]         [6]
> > Core 3 [3]         [7]
> 
> The CPU layout shows two sockets and normally that means 2 numa node.
> 
> I would change the command above —-socket-mem 32,32  (note: two ‘-‘ in front) this command gives 32M socket 0 and 32M to socket 1
> 
> I have seen some funny memory configs, with huge pages if you do not use /etc/sysctl.conf and reboot. Sometimes memory will be allocated to one socket.
> 
> I hope that helps.
> 
I would also suggest increasing the amount of memory reserved. DPDK can
sometimes be quite memory hungry, so perhaps up the amount of hugepages
reserved in the kernel. Also, if you are ok with DPDK using all the
hugepages you have statically reserved, you can omit the --socket-mem
parameter.

/Bruce

Re: [dpdk-dev] EAL reports "ot enough memory available on socket 0!"

[Huertas García, Víctor]

Thanks Bruce and Keith,

I have run the application with root privileges and at least the memory error reported by EAL has been solved. Thanks a lot! Now it runs, although it complains about other issues (it doesn't detect interfaces despite the fact I have associated them with vfio-pci driver using dpdk-setup.sh script) but I think I can handle it.

Furthermore, I managed to finally enable NUMA nodes in my architecture. I tell you what I did just in case someone has found the same problem. I discovered in the BIOS a parameter in the NorthBridge Chipset Configuration called "Crystal Beach/DMA" which was disabled. I enabled it and appeared an additional hidden parameter called "Crystal Beach /DCA" which, I don't what does it mean, but I left it disabled. Then I restarted the machine and the numactl --hardware showed me two marvelous NUMA nodes (one for each socket as Keith told me). I just have to guess to which NUMA node the NIC cards are associated in order to have full hardware affinity. I have found plenty of info about how to know it.

In the next experimentation I will try to increase the number of hugepages to use but I have one question. If for example the AnnonHugePages returned by the /proc/meminfo tells me that there are 210944 kB and the size of the HugePageSize is 2048kB, does the maximum number of hugepages configurable equivalent to dividing 210944 / 2048 = 103 ? If I configured more than 103 it would generate an error?

Thanks for your quick answer.

BTW I will tell you how I managed to configure the Eclipse IDE to run in Debug mode the DPDK application with root privileges. Maybe the rest of developer community already know it but I publish it anyway as I think that telling how to use Eclipse and DPDK is very interesting for future developers (a tutorial on how to configure Eclipse and DPDK for developing purposes is something that would be very helpful and I missed it in the DPDK web page).

Step 1: tell the Linux not to ask you the root password if you run the "sudo gdb" in the Debug environment .

$ sudo visudo

Add after the line 'root ALL=(ALL) ALL' the following line
<your user> ALL=(root) NOPASSWD:/usr/bin/gdb

Step 2: Configure the "Debug Configurations" section at Eclipse.

Once you have added your application to be executed in debug mode at Eclipse and generated a particular Debug configuration for it, you have to go: Debug Configurations -> "Your App" -> “Debugger” Tag -> “Main” Tag at "Debugger Options"

At "GDB debugger" field you have put sudo gdb instead of just gdb

Apply it and that's it.

Regards,

Víctor Huertas García
Senior Engineer
Space - Communication Systems


-----Mensaje original-----
De: Bruce Richardson [mailto:bruce.richardson@intel.com]
Enviado el: miércoles, 07 de febrero de 2018 10:33
Para: Wiles, Keith
CC: Huertas García, Víctor; dev@dpdk.org
Asunto: Re: [dpdk-dev] EAL reports "ot enough memory available on socket 0!"

On Tue, Feb 06, 2018 at 09:09:26PM +0000, Wiles, Keith wrote:
>
>
> > On Feb 6, 2018, at 12:38 PM, Huertas García, Víctor <vhuertas@indra.es> wrote:
> >
> > Hi all,
> >
> >
> > This is my first post in this mailing list. I am glad to become a member of this community.
> >
> >
> > I am a newbie using DPDK and I have tried to compile and execute an example app called "l3fwd-acl".
> >
> >
> > The DPDK command is DPDK_Test -c 0xf0 -n 2 --socket-mem 64
>
> ran this command as root or used sudo correct?
>
> >
> > but the application tells me the following:
> >
> >
> > EAL: Detected 8 lcore(s)
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> > EAL: Not enough memory available on socket 0! Requested: 64MB,
> > available: 0MB
> > EAL: FATAL: Cannot init memory
> >
> > EAL: Cannot init memory
> >
> > EAL: Error - exiting with code: 1
> >  Cause: Invalid EAL parameters
> >
> >
> >
> > I don't know what else I can do to make this app run as I have reserved hugepages as the user guide tells as well as mounted the /mnt/huge folder.
> >
> >
> > cat /proc/meminfo | grep Huge
> > AnonHugePages:    520192 kB
> > HugePages_Total:      64
> > HugePages_Free:       64
> > HugePages_Rsvd:        0
> > HugePages_Surp:        0
> > Hugepagesize:       2048 kB
>
> I assume you added an entry into /etc/sysctl.conf and rebooted, just to make sure the huge pages are allocated early in the boot process.
>
> I use 1G huge pages and I assume you are using 2M huge pages
>
> vm.nr_hugepages=32  # 32x2M is 64M
>
> I also disable randomize_va_space, which is not required for those tests.
>
> kernel.randomize_va_space=1
>
>
> >
> >
> > My machine is has this topology (only one Numa node is detected):
> >
> >
> > udo usertools/cpu_layout.py
> > [sudo] password for root:
> > ====================================================================
> > == Core and Socket Information (as reported by
> > '/sys/devices/system/cpu')
> > ====================================================================
> > ==
> >
> > cores =  [0, 1, 2, 3]
> > sockets =  [0, 1]
> >
> >       Socket 0    Socket 1
> >       --------    --------
> > Core 0 [0]         [4]
> > Core 1 [1]         [5]
> > Core 2 [2]         [6]
> > Core 3 [3]         [7]
>
> The CPU layout shows two sockets and normally that means 2 numa node.
>
> I would change the command above —-socket-mem 32,32  (note: two ‘-‘ in
> front) this command gives 32M socket 0 and 32M to socket 1
>
> I have seen some funny memory configs, with huge pages if you do not use /etc/sysctl.conf and reboot. Sometimes memory will be allocated to one socket.
>
> I hope that helps.
>
I would also suggest increasing the amount of memory reserved. DPDK can sometimes be quite memory hungry, so perhaps up the amount of hugepages reserved in the kernel. Also, if you are ok with DPDK using all the hugepages you have statically reserved, you can omit the --socket-mem parameter.

/Bruce

________________________________

Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, contiene información de carácter confidencial exclusivamente dirigida a su destinatario o destinatarios. Si no es vd. el destinatario indicado, queda notificado que la lectura, utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. En el caso de haber recibido este correo electrónico por error, se ruega notificar inmediatamente esta circunstancia mediante reenvío a la dirección electrónica del remitente.
Evite imprimir este mensaje si no es estrictamente necesario.

This email and any file attached to it (when applicable) contain(s) confidential information that is exclusively addressed to its recipient(s). If you are not the indicated recipient, you are informed that reading, using, disseminating and/or copying it without authorisation is forbidden in accordance with the legislation in effect. If you have received this email by mistake, please immediately notify the sender of the situation by resending it to their email address.
Avoid printing this message if it is not absolutely necessary.