[dpdk-dev] l3fwd fails : Cause: Unable to create the l3fwd LPM table on socket 0

[dpdk-dev] l3fwd fails : Cause: Unable to create the l3fwd LPM table on socket 0

[Niraj Sharma (nirajsha)]

Hello! I am new to DPDK.

When I run "l3fwd" on a Ubuntu 12.04 Virtual Machine, the error is  "Cause: Unable to create the l3fwd LPM table on socket 0". The information about my set-up is as follows:

1) The command that I used to start the VM is:
qemu-system-x86_64 -enable-kvm -m 1024 -localtime -boot d -vga std -name DPDK \
-hda ./ubuntu_desktop_dpdk.img -cpu host \
-net nic,vlan=1,model=virtio,macaddr=00:40:47:1a:0a:44 \
-net tap,vlan=1,ifname=tap-dpdk1,script=no \
-net nic,vlan=2,model=virtio,macaddr=00:40:47:1a:0b:44 \
-net tap,vlan=2,ifname=tap-dpdk2,script=no \
-net nic,vlan=3,model=virtio,macaddr=00:40:47:1a:0c:44 \
-net tap,vlan=3,ifname=tap0,script=no

Host has 4 cores Xeon 4771, but this VM ends up having only one core. Note that I used option "-cpu host" in this qemu command.
On VM, I give control to two ports to the igb_uio. For huge pages, I gave 64x2MB memory to NUMA system using tools/setup.sh script. In another variation, I gave the same amount to non-NUMA system also.

2) The actual command used to start l3fwd and its output (1 core, 1 memory channel, enabled all ports, and ports 0 & 1 both assigned to core 0):

./l3fwd -c 0x1 -n 1 -- -p 0xf --config="(0,0,0),(1,0,0)"
....
....
....
....
EAL: Skip lcore 63 (not detected)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f67b7800000 (size = 0x200000)
EAL: Ask a virtual area of 0x7c00000 bytes
EAL: Virtual area found at 0x7f67afa00000 (size = 0x7c00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f67af600000 (size = 0x200000)
EAL: Requesting 64 pages of size 2MB from socket 0
EAL: TSC frequency is ~3491919 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: Master core 0 is ready (tid=b86e2800)
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL: PCI Port IO found start=0xc000 with size=0x20
EAL: PCI device 0000:00:04.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL: PCI Port IO found start=0xc020 with size=0x20
EAL: PCI device 0000:00:05.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   0000:00:05.0 not managed by UIO driver, skipping
Initializing port 0 ... Creating queues: nb_rxq=1 nb_txq=1...  Address:00:40:47:1A:0A:44, All
ocated mbuf pool on socket 0
LPM: Adding route 0x01010100 / 24 (0)
LPM: Adding route 0x02010100 / 24 (1)
LPM: Adding route 0x03010100 / 24 (2)
LPM: Adding route 0x04010100 / 24 (3)
LPM: Adding route 0x05010100 / 24 (4)
LPM: Adding route 0x06010100 / 24 (5)
LPM: Adding route 0x07010100 / 24 (6)
LPM: Adding route 0x08010100 / 24 (7)
LPM: LPM memory allocation failed
EAL: Error - exiting with code: 1
  Cause: Unable to create the l3fwd LPM table on socket 0

I will appreciate any help. Thanks,

-- Niraj

Re: [dpdk-dev] l3fwd fails : Cause: Unable to create the l3fwd LPM table on socket 0

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Niraj Sharma (nirajsha)
> Sent: Friday, May 30, 2014 7:59 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] l3fwd fails : Cause: Unable to create the l3fwd LPM table on
> socket 0
>

> ....
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f67b7800000 (size = 0x200000)
> EAL: Ask a virtual area of 0x7c00000 bytes
> EAL: Virtual area found at 0x7f67afa00000 (size = 0x7c00000)
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7f67af600000 (size = 0x200000)
> EAL: Requesting 64 pages of size 2MB from socket 0

Just by way of explanation, the 64 pages are getting mapped into three blocks, two with a single page each, i.e. 2MB in size, and the rest as a contiguous 124MB block.

> EAL: TSC frequency is ~3491919 KHz
> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable
> clock cycles !
> EAL: Master core 0 is ready (tid=b86e2800)
> EAL: PCI device 0000:00:03.0 on NUMA socket -1
> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> EAL: PCI Port IO found start=0xc000 with size=0x20
> EAL: PCI device 0000:00:04.0 on NUMA socket -1
> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> EAL: PCI Port IO found start=0xc020 with size=0x20
> EAL: PCI device 0000:00:05.0 on NUMA socket -1
> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> EAL:   0000:00:05.0 not managed by UIO driver, skipping
> Initializing port 0 ... Creating queues: nb_rxq=1 nb_txq=1...
> Address:00:40:47:1A:0A:44, All
> ocated mbuf pool on socket 0
> LPM: Adding route 0x01010100 / 24 (0)
> LPM: Adding route 0x02010100 / 24 (1)
> LPM: Adding route 0x03010100 / 24 (2)
> LPM: Adding route 0x04010100 / 24 (3)
> LPM: Adding route 0x05010100 / 24 (4)
> LPM: Adding route 0x06010100 / 24 (5)
> LPM: Adding route 0x07010100 / 24 (6)
> LPM: Adding route 0x08010100 / 24 (7)
> LPM: LPM memory allocation failed
> EAL: Error - exiting with code: 1
>   Cause: Unable to create the l3fwd LPM table on socket 0
> 

The error message indicates that you are running out of memory, or running out of contiguous memory on your system when running the application. From the output, it looks like the LPM table for IPv4 is getting created and 8 rules are getting added to it. However, thereafter the application is attempting to create an LPM table for IPv6 traffic and there is not enough memory available for this. 
Running a quick test on my system, the l3fwd app is requesting just over 32MB of memory for the LPM4 table and 128MB of memory for the LPM6 one. This size of memory requests would explain the observed behaviour, given you only have 128MB of hugepage memory in total - and only 124MB of that in a contiguous block, as can be seen from the statements mapping the memory.

Regards,
/Bruce

Re: [dpdk-dev] l3fwd fails : Cause: Unable to create the l3fwd LPM table on socket 0

[Niraj Sharma (nirajsha)]

Bruce,
    Thanks for the insight. I should be able to solve this problem based
on this information. Once I understand the applications, I plan to deep
dive into DPDK code.
-- Niraj

On 6/2/14 12:01 PM, "Richardson, Bruce" <bruce.richardson@intel.com> wrote:

>
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Niraj Sharma
>>(nirajsha)
>> Sent: Friday, May 30, 2014 7:59 PM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] l3fwd fails : Cause: Unable to create the l3fwd LPM
>>table on
>> socket 0
>>
>
>> ....
>> EAL: Setting up memory...
>> EAL: Ask a virtual area of 0x200000 bytes
>> EAL: Virtual area found at 0x7f67b7800000 (size = 0x200000)
>> EAL: Ask a virtual area of 0x7c00000 bytes
>> EAL: Virtual area found at 0x7f67afa00000 (size = 0x7c00000)
>> EAL: Ask a virtual area of 0x200000 bytes
>> EAL: Virtual area found at 0x7f67af600000 (size = 0x200000)
>> EAL: Requesting 64 pages of size 2MB from socket 0
>
>Just by way of explanation, the 64 pages are getting mapped into three
>blocks, two with a single page each, i.e. 2MB in size, and the rest as a
>contiguous 124MB block.
>
>> EAL: TSC frequency is ~3491919 KHz
>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>>unreliable
>> clock cycles !
>> EAL: Master core 0 is ready (tid=b86e2800)
>> EAL: PCI device 0000:00:03.0 on NUMA socket -1
>> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
>> EAL: PCI Port IO found start=0xc000 with size=0x20
>> EAL: PCI device 0000:00:04.0 on NUMA socket -1
>> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
>> EAL: PCI Port IO found start=0xc020 with size=0x20
>> EAL: PCI device 0000:00:05.0 on NUMA socket -1
>> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
>> EAL:   0000:00:05.0 not managed by UIO driver, skipping
>> Initializing port 0 ... Creating queues: nb_rxq=1 nb_txq=1...
>> Address:00:40:47:1A:0A:44, All
>> ocated mbuf pool on socket 0
>> LPM: Adding route 0x01010100 / 24 (0)
>> LPM: Adding route 0x02010100 / 24 (1)
>> LPM: Adding route 0x03010100 / 24 (2)
>> LPM: Adding route 0x04010100 / 24 (3)
>> LPM: Adding route 0x05010100 / 24 (4)
>> LPM: Adding route 0x06010100 / 24 (5)
>> LPM: Adding route 0x07010100 / 24 (6)
>> LPM: Adding route 0x08010100 / 24 (7)
>> LPM: LPM memory allocation failed
>> EAL: Error - exiting with code: 1
>>   Cause: Unable to create the l3fwd LPM table on socket 0
>> 
>
>The error message indicates that you are running out of memory, or
>running out of contiguous memory on your system when running the
>application. From the output, it looks like the LPM table for IPv4 is
>getting created and 8 rules are getting added to it. However, thereafter
>the application is attempting to create an LPM table for IPv6 traffic and
>there is not enough memory available for this.
>Running a quick test on my system, the l3fwd app is requesting just over
>32MB of memory for the LPM4 table and 128MB of memory for the LPM6 one.
>This size of memory requests would explain the observed behaviour, given
>you only have 128MB of hugepage memory in total - and only 124MB of that
>in a contiguous block, as can be seen from the statements mapping the
>memory.
>
>Regards,
>/Bruce