Hello I am having some rare problem with hugepages. I am running DPDK 1.5.1 in CenOS 6.4. Everything is running smothly and we have done quite good progress integrating DPDK into our solution. But sometimes, there is an error with the init of the huge pages. The problem is when there is another process started that also used hugepages, in this case, a java 1.7 process (jdk 1.7.0_45 64-bits). The machine only has one socket with 1 Xeon CPU with 4 cores and 4 GB. I have configured 256 pages of 2MB. Everything is running in 64 bits. grep Huge /proc/meminfo AnonHugePages: 253952 kB HugePages_Total: 256 HugePages_Free: 256 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB When I start the DPDK process (with –c 01 –n 4 –m 32 options), this is the trace: EAL: Detected lcore 0 as core 0 on socket 0 EAL: Setting up hugepage memory... EAL: Ask a virtual area of 0x394264576 bytes EAL: Virtual area found at 0x7fc0c9000000 (size = 0x17800000) EAL: Ask a virtual area of 0x4194304 bytes EAL: Virtual area found at 0x7fc0c8a00000 (size = 0x400000) EAL: Ask a virtual area of 0x132120576 bytes EAL: Virtual area found at 0x7fc0c0a00000 (size = 0x7e00000) EAL: Ask a virtual area of 0x4194304 bytes EAL: Virtual area found at 0x7fc0c0400000 (size = 0x400000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7fc0c0000000 (size = 0x200000) EAL: Requesting 16 pages of size 2MB from socket 0 ....... Everything goes well. After the start, this is the hugepages information: AnonHugePages: 280576 kB HugePages_Total: 256 HugePages_Free: 220 HugePages_Rsvd: 35 HugePages_Surp: 0 Hugepagesize: 2048 kB If I started then the java process, it starts ok and this is the hugepages information: HugePages_Total: 256 HugePages_Free: 204 HugePages_Rsvd: 74 HugePages_Surp: 0 Hugepagesize: 2048 kB If I stop and start the DPDK process, it fails with the following trace: EAL: Detected lcore 0 as core 0 on socket 0 EAL: Setting up hugepage memory... EAL: map_all_hugepages(): mmap failed: Cannot allocate memory EAL: Failed to mmap 2 MB hugepages PANIC in rte_eal_init(): Cannot init memory And after the fail, this is the hugepage information: AnonHugePages: 251904 kB HugePages_Total: 256 HugePages_Free: 40 HugePages_Rsvd: 40 HugePages_Surp: 0 Hugepagesize: 2048 kB It seems that it tries to map all the hugepages even if it should be limited with –m 32. After the fail, there are 201 rtemap_X files in /mnt/huge. When I stop the java process, the DPDK process starts without any problem. I don’t know if this happens also with any other process different from java and that uses hugepages. Am I doing something wrong? Thanks a lot Carlos From dan@nyansa.com Tue Mar 11 20:53:40 2014 Return-Path: Received: from mail-pb0-f41.google.com (mail-pb0-f41.google.com [209.85.160.41]) by dpdk.org (Postfix) with ESMTP id 7806018F for ; Tue, 11 Mar 2014 20:53:40 +0100 (CET) Received: by mail-pb0-f41.google.com with SMTP id jt11so18614pbb.14 for ; Tue, 11 Mar 2014 12:55:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d100.net; s 130820; h=x-gm-message-state:from:content-type:subject:message-id:date:to :mime-version; bh=kyA/3NBS5fTaUI6llIe/WT1D9VRgfX3XUHhBlnp+w2c=; bð6GX3dywPkCtXbXNnmsiPAflNc2L1632hRlGbA4di82QrOu/oYDPwErNU3GiRx+zw /fOCIJYPRCsYnjfrYsuMzrPshRAZbGLmtfFbj4zfojLhm6vhqIRNeHzLMmGaDZcPDuHh 5iaqvXzOemFxgUtpA5fR+Evv62jM0uTiD1321bHIXhLDKAXIe3KPm/74bc+ih0ZbmmLw QPJNOiCrBJVTXo9bjiQVJOQuijozSUFjJh6FFshUePpJPVKNPLgSrf9z8DU07YckciRH d5rvO49I+KdVuE2dsrSETJy8uVXD+Fr4rGtlrhgqfMoNp8poIUqaA0U/SIStIoQk9Ils i9eA=X-Gm-Message-State: ALoCoQmSpEFQKIwdim8lFJmN54mGAKnSVjG5zMg6kRGe56Bnn2ULKRj14QnjvGZdDUccTLdNIkdJ X-Received: by 10.68.237.133 with SMTP id vc5mr15700pbc.92.1394567709673; Tue, 11 Mar 2014 12:55:09 -0700 (PDT) Received: from [192.168.128.242] ([50.242.95.29]) by mx.google.com with ESMTPSA id lh13sm309595pab.4.2014.03.11.12.55.07 for (version=TLSv1 cipherìDHE-RSA-RC4-SHA bits8/128); Tue, 11 Mar 2014 12:55:08 -0700 (PDT) From: Daniel Kan Message-Id: <3274259C-EE9A-461C-9586-D7B0820CDE2B@nyansa.com> Date: Tue, 11 Mar 2014 12:55:05 -0700 To: dev@dpdk.org Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) X-Mailer: Apple Mail (2.1874) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] RSS, performance, and stability issues vmxnet3-pmd X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 19:53:40 -0000 I’m unable to get RSS to work properly with vmxnet3-pmd. The first issue is that the number of rxqs must be power of 2. Otherwise, rte_eth_dev_start() fails due to inability to activate vmxnet3 NIC. This is not too big of a deal, but physical NICs don’t have this requirement. The second issue is that RSS is just not working at all for me. The rxmode is set to ETH_MQ_RX_RSS and rss_hf = ETH_RSS_IPV4_TCP | ETH_RSS_IPV4_UDP | ETH_RSS_IPV4 | ETH_RSS_IPV6. The same configuration works for a real NIC. When I checked mb->pkt.hash, the value is all zeroed out. Even if I disabled RSS, I found the performance of vmxnet3-pmd to be quite poor, peaking out at 600k pps with 64 byte packet, while libpcap can do 650k pps. Lastly, there is a stability issue. On a number of occasions, vmxnet3-pmd stops receiving packets after some random time and several million packets. I’m not sure if anyone else is having as much issue as I’m, I will give vmxnet3-usermap a try. Finally, does either vmxnet3-usermap or vmxnet3-pmd work well for non-Intel-based underlying physical NIC? Thanks. Dan