[dpdk-dev] Setting up hugepage memory failed when there is a java process also using hugepages

[dpdk-dev] Setting up hugepage memory failed when there is a java process also using hugepages

[Carlos Franco]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 6400 bytes --]

Hello
I am having some rare problem with hugepages. I am running DPDK 1.5.1 in CenOS 6.4. Everything is running smothly and we have done quite good progress integrating DPDK into our solution. But sometimes, there is an error with the init of the huge pages. The problem is when there is another process started that also used hugepages, in this case, a java 1.7 process (jdk 1.7.0_45 64-bits).
The machine only has one socket with 1 Xeon CPU with 4 cores and 4 GB. I have configured 256 pages of 2MB. Everything is running in 64 bits. 
grep Huge /proc/meminfo
AnonHugePages:    253952 kB
HugePages_Total:     256
HugePages_Free:      256
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
When I start the DPDK process (with –c 01 –n 4 –m 32 options), this is the trace:
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Setting up hugepage memory...
EAL: Ask a virtual area of 0x394264576 bytes
EAL: Virtual area found at 0x7fc0c9000000 (size = 0x17800000)
EAL: Ask a virtual area of 0x4194304 bytes
EAL: Virtual area found at 0x7fc0c8a00000 (size = 0x400000)
EAL: Ask a virtual area of 0x132120576 bytes
EAL: Virtual area found at 0x7fc0c0a00000 (size = 0x7e00000)
EAL: Ask a virtual area of 0x4194304 bytes
EAL: Virtual area found at 0x7fc0c0400000 (size = 0x400000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7fc0c0000000 (size = 0x200000)
EAL: Requesting 16 pages of size 2MB from socket 0
.......
Everything goes well. After the start, this is the hugepages information:
AnonHugePages:    280576 kB
HugePages_Total:     256
HugePages_Free:      220
HugePages_Rsvd:       35
HugePages_Surp:        0
Hugepagesize:       2048 kB

If I started then the java process, it starts ok and this is the hugepages information:

HugePages_Total:     256
HugePages_Free:      204
HugePages_Rsvd:       74
HugePages_Surp:        0
Hugepagesize:       2048 kB

If I stop and start the DPDK process, it fails with the following trace:

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Setting up hugepage memory...
EAL: map_all_hugepages(): mmap failed: Cannot allocate memory
EAL: Failed to mmap 2 MB hugepages
PANIC in rte_eal_init():
Cannot init memory

And after the fail, this is the hugepage information:

AnonHugePages:    251904 kB
HugePages_Total:     256
HugePages_Free:       40
HugePages_Rsvd:       40
HugePages_Surp:        0
Hugepagesize:       2048 kB

It seems that it tries to map all the hugepages even if it should be limited with –m 32. After the fail, there are 201 rtemap_X files in /mnt/huge.

When I stop the java process, the DPDK process starts without any problem. I don’t know if this happens also with any other process different from java and that uses hugepages.

Am I doing something wrong? 

Thanks a lot

Carlos




From dan@nyansa.com  Tue Mar 11 20:53:40 2014
Return-Path: <dan@nyansa.com>
Received: from mail-pb0-f41.google.com (mail-pb0-f41.google.com
 [209.85.160.41]) by dpdk.org (Postfix) with ESMTP id 7806018F
 for <dev@dpdk.org>; Tue, 11 Mar 2014 20:53:40 +0100 (CET)
Received: by mail-pb0-f41.google.com with SMTP id jt11so18614pbb.14
 for <dev@dpdk.org>; Tue, 11 Mar 2014 12:55:09 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d\x1e100.net; s 130820;
 h=x-gm-message-state:from:content-type:subject:message-id:date:to
 :mime-version;
 bh=kyA/3NBS5fTaUI6llIe/WT1D9VRgfX3XUHhBlnp+w2c=;
 bð6GX3dywPkCtXbXNnmsiPAflNc2L1632hRlGbA4di82QrOu/oYDPwErNU3GiRx+zw
 /fOCIJYPRCsYnjfrYsuMzrPshRAZbGLmtfFbj4zfojLhm6vhqIRNeHzLMmGaDZcPDuHh
 5iaqvXzOemFxgUtpA5fR+Evv62jM0uTiD1321bHIXhLDKAXIe3KPm/74bc+ih0ZbmmLw
 QPJNOiCrBJVTXo9bjiQVJOQuijozSUFjJh6FFshUePpJPVKNPLgSrf9z8DU07YckciRH
 d5rvO49I+KdVuE2dsrSETJy8uVXD+Fr4rGtlrhgqfMoNp8poIUqaA0U/SIStIoQk9Ils
 i9eA=X-Gm-Message-State: ALoCoQmSpEFQKIwdim8lFJmN54mGAKnSVjG5zMg6kRGe56Bnn2ULKRj14QnjvGZdDUccTLdNIkdJ
X-Received: by 10.68.237.133 with SMTP id vc5mr15700pbc.92.1394567709673;
 Tue, 11 Mar 2014 12:55:09 -0700 (PDT)
Received: from [192.168.128.242] ([50.242.95.29])
 by mx.google.com with ESMTPSA id lh13sm309595pab.4.2014.03.11.12.55.07
 for <dev@dpdk.org>
 (version=TLSv1 cipherìDHE-RSA-RC4-SHA bits\x128/128);
 Tue, 11 Mar 2014 12:55:08 -0700 (PDT)
From: Daniel Kan <dan@nyansa.com>
Message-Id: <3274259C-EE9A-461C-9586-D7B0820CDE2B@nyansa.com>
Date: Tue, 11 Mar 2014 12:55:05 -0700
To: dev@dpdk.org
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
X-Mailer: Apple Mail (2.1874)
Content-Type: text/plain;
	charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Subject: [dpdk-dev] RSS, performance, and stability issues vmxnet3-pmd
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 11 Mar 2014 19:53:40 -0000

I’m unable to get RSS to work properly with vmxnet3-pmd. The first issue is that the number of rxqs must be power of 2. Otherwise, rte_eth_dev_start() fails due to inability to activate vmxnet3 NIC. This is not too big of a deal, but physical NICs don’t have this requirement. 

The second issue is that RSS is just not working at all for me. The rxmode is set to ETH_MQ_RX_RSS and rss_hf = ETH_RSS_IPV4_TCP | ETH_RSS_IPV4_UDP | ETH_RSS_IPV4 | ETH_RSS_IPV6. The same configuration works for a real NIC. When I checked mb->pkt.hash, the value is all zeroed out. 

Even if I disabled RSS, I found the performance of vmxnet3-pmd to be quite poor, peaking out at 600k pps with 64 byte packet, while libpcap can do 650k pps. 

Lastly, there is a stability issue. On a number of occasions, vmxnet3-pmd stops receiving packets after some random time and several million packets. 

I’m not sure if anyone else is having as much issue as I’m, I will give vmxnet3-usermap a try. 

Finally, does either vmxnet3-usermap or vmxnet3-pmd work well for non-Intel-based underlying physical NIC? 

Thanks. 

Dan