From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-x232.google.com (mail-we0-x232.google.com [IPv6:2a00:1450:400c:c03::232]) by dpdk.org (Postfix) with ESMTP id 4F742DE4 for ; Tue, 11 Mar 2014 10:27:53 +0100 (CET) Received: by mail-we0-f178.google.com with SMTP id u56so9543105wes.23 for ; Tue, 11 Mar 2014 02:29:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:from:to:subject:date:mime-version:content-type :importance; bh=dFMhUlmUsTWEejSJ2s6kORktHWacAOh5RMKwvyTG2Zg=; b=OImoDfIzTQlLHf882uFs22OF484EttuZQEgcqJouCqW4MyL4C41/7eJ5rWHzHaisbo UE0piQgpw+gOWDb41GPxocBaKhHq5xvsYLfJLg2qmpjCdl09DsuAo54I/rmaerXThbld XS03qHkVMvytN91wQP55dCv6Gc0FS3YW3hz2Nr07cntANSlNq+70OqpGKVoO9dRuSCUZ oUL7hqiL8Z87iWvnuj2CJzLm9DOD7ZqL2p4zLOvCsLjS+nvIk1ORLmrevRKlRMGpJOQF kzb6x/Ga7UEKDmgqCLKraBV5wdYFVUGrnC+1IfJBarlEF93jobnIPrsF/3D6yGbFXIyv z6qw== X-Received: by 10.180.19.35 with SMTP id b3mr2208136wie.20.1394530162556; Tue, 11 Mar 2014 02:29:22 -0700 (PDT) Received: from CARLOSPC ([195.57.230.177]) by mx.google.com with ESMTPSA id lg9sm15987496wjb.3.2014.03.11.02.29.21 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 11 Mar 2014 02:29:21 -0700 (PDT) Message-ID: From: "Carlos Franco" To: Date: Tue, 11 Mar 2014 10:29:20 +0100 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 16.4.3505.912 X-MimeOLE: Produced By Microsoft MimeOLE V16.4.3505.912 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] Setting up hugepage memory failed when there is a java process also using hugepages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 09:27:53 -0000 Hello I am having some rare problem with hugepages. I am running DPDK 1.5.1 in = CenOS 6.4. Everything is running smothly and we have done quite good = progress integrating DPDK into our solution. But sometimes, there is an = error with the init of the huge pages. The problem is when there is = another process started that also used hugepages, in this case, a java = 1.7 process (jdk 1.7.0_45 64-bits). The machine only has one socket with 1 Xeon CPU with 4 cores and 4 GB. I = have configured 256 pages of 2MB. Everything is running in 64 bits.=20 grep Huge /proc/meminfo AnonHugePages: 253952 kB HugePages_Total: 256 HugePages_Free: 256 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB When I start the DPDK process (with =E2=80=93c 01 =E2=80=93n 4 = =E2=80=93m 32 options), this is the trace: EAL: Detected lcore 0 as core 0 on socket 0 EAL: Setting up hugepage memory... EAL: Ask a virtual area of 0x394264576 bytes EAL: Virtual area found at 0x7fc0c9000000 (size =3D 0x17800000) EAL: Ask a virtual area of 0x4194304 bytes EAL: Virtual area found at 0x7fc0c8a00000 (size =3D 0x400000) EAL: Ask a virtual area of 0x132120576 bytes EAL: Virtual area found at 0x7fc0c0a00000 (size =3D 0x7e00000) EAL: Ask a virtual area of 0x4194304 bytes EAL: Virtual area found at 0x7fc0c0400000 (size =3D 0x400000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7fc0c0000000 (size =3D 0x200000) EAL: Requesting 16 pages of size 2MB from socket 0 ....... Everything goes well. After the start, this is the hugepages = information: AnonHugePages: 280576 kB HugePages_Total: 256 HugePages_Free: 220 HugePages_Rsvd: 35 HugePages_Surp: 0 Hugepagesize: 2048 kB If I started then the java process, it starts ok and this is the = hugepages information: HugePages_Total: 256 HugePages_Free: 204 HugePages_Rsvd: 74 HugePages_Surp: 0 Hugepagesize: 2048 kB If I stop and start the DPDK process, it fails with the following trace: EAL: Detected lcore 0 as core 0 on socket 0 EAL: Setting up hugepage memory... EAL: map_all_hugepages(): mmap failed: Cannot allocate memory EAL: Failed to mmap 2 MB hugepages PANIC in rte_eal_init(): Cannot init memory And after the fail, this is the hugepage information: AnonHugePages: 251904 kB HugePages_Total: 256 HugePages_Free: 40 HugePages_Rsvd: 40 HugePages_Surp: 0 Hugepagesize: 2048 kB It seems that it tries to map all the hugepages even if it should be = limited with =E2=80=93m 32. After the fail, there are 201 rtemap_X files = in /mnt/huge. When I stop the java process, the DPDK process starts without any = problem. I don=E2=80=99t know if this happens also with any other = process different from java and that uses hugepages. Am I doing something wrong?=20 Thanks a lot Carlos >From dan@nyansa.com Tue Mar 11 20:53:40 2014 Return-Path: Received: from mail-pb0-f41.google.com (mail-pb0-f41.google.com [209.85.160.41]) by dpdk.org (Postfix) with ESMTP id 7806018F for ; Tue, 11 Mar 2014 20:53:40 +0100 (CET) Received: by mail-pb0-f41.google.com with SMTP id jt11so18614pbb.14 for ; Tue, 11 Mar 2014 12:55:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:subject:message-id:date:to :mime-version; bh=kyA/3NBS5fTaUI6llIe/WT1D9VRgfX3XUHhBlnp+w2c=; b=F06GX3dywPkCtXbXNnmsiPAflNc2L1632hRlGbA4di82QrOu/oYDPwErNU3GiRx+zw /fOCIJYPRCsYnjfrYsuMzrPshRAZbGLmtfFbj4zfojLhm6vhqIRNeHzLMmGaDZcPDuHh 5iaqvXzOemFxgUtpA5fR+Evv62jM0uTiD1321bHIXhLDKAXIe3KPm/74bc+ih0ZbmmLw QPJNOiCrBJVTXo9bjiQVJOQuijozSUFjJh6FFshUePpJPVKNPLgSrf9z8DU07YckciRH d5rvO49I+KdVuE2dsrSETJy8uVXD+Fr4rGtlrhgqfMoNp8poIUqaA0U/SIStIoQk9Ils i9eA== X-Gm-Message-State: ALoCoQmSpEFQKIwdim8lFJmN54mGAKnSVjG5zMg6kRGe56Bnn2ULKRj14QnjvGZdDUccTLdNIkdJ X-Received: by 10.68.237.133 with SMTP id vc5mr15700pbc.92.1394567709673; Tue, 11 Mar 2014 12:55:09 -0700 (PDT) Received: from [192.168.128.242] ([50.242.95.29]) by mx.google.com with ESMTPSA id lh13sm309595pab.4.2014.03.11.12.55.07 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 11 Mar 2014 12:55:08 -0700 (PDT) From: Daniel Kan Message-Id: <3274259C-EE9A-461C-9586-D7B0820CDE2B@nyansa.com> Date: Tue, 11 Mar 2014 12:55:05 -0700 To: dev@dpdk.org Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) X-Mailer: Apple Mail (2.1874) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] RSS, performance, and stability issues vmxnet3-pmd X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 19:53:40 -0000 I=92m unable to get RSS to work properly with vmxnet3-pmd. The first = issue is that the number of rxqs must be power of 2. Otherwise, = rte_eth_dev_start() fails due to inability to activate vmxnet3 NIC. This = is not too big of a deal, but physical NICs don=92t have this = requirement.=20 The second issue is that RSS is just not working at all for me. The = rxmode is set to ETH_MQ_RX_RSS and rss_hf =3D ETH_RSS_IPV4_TCP | = ETH_RSS_IPV4_UDP | ETH_RSS_IPV4 | ETH_RSS_IPV6. The same configuration = works for a real NIC. When I checked mb->pkt.hash, the value is all = zeroed out.=20 Even if I disabled RSS, I found the performance of vmxnet3-pmd to be = quite poor, peaking out at 600k pps with 64 byte packet, while libpcap = can do 650k pps.=20 Lastly, there is a stability issue. On a number of occasions, = vmxnet3-pmd stops receiving packets after some random time and several = million packets.=20 I=92m not sure if anyone else is having as much issue as I=92m, I will = give vmxnet3-usermap a try.=20 Finally, does either vmxnet3-usermap or vmxnet3-pmd work well for = non-Intel-based underlying physical NIC?=20 Thanks.=20 Dan