From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f193.google.com (mail-qt0-f193.google.com [209.85.216.193]) by dpdk.org (Postfix) with ESMTP id A3F915A44 for ; Wed, 4 Jan 2017 07:44:32 +0100 (CET) Received: by mail-qt0-f193.google.com with SMTP id j29so1879843qtc.1 for ; Tue, 03 Jan 2017 22:44:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VOdJxJuPRfC17ZMFH0YUoJC/MEXHJfVGphZlQ5I/pdQ=; b=dXa1hGXk3HN/F8UKN7CJSAK93VGsU0FyIGQXwhU4Mv/IQvapiyE/7bBjdFV0fG7DAd CXIvRSgRimWq4SxCgNmoQ3q+zWqhFVX0pmGuejT0FVsJk32lbPtYnghj7TcQ/BArT9ad sG44qto1KUTOQ0QDDmggDhanmslnuZsK7GzCvdyP0eAb00ujxOMsSCPy28wucGxhSrPm LWMHzlc56E6w09+jDWhwj5ah2tIiaE1nPTm4ONfmfXIn1Nf+8bwBWojYYkoaoyosvLlD LuKofXdX/Kc2ywhcYlgJ3+49i0ZPm6D2h71u6hGhIeHCbtlFtx2gchEiFgEEWAj9b+Es mIxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VOdJxJuPRfC17ZMFH0YUoJC/MEXHJfVGphZlQ5I/pdQ=; b=B3LWfXl9okH2IoRiMQXDDNUhvRRL9bQSk40qrLeL5mOTUySCXS8h7Nf4e9/c+UID0a hu9QI3Sityi6ntHoMHXwb2ys8tLq5MM4VATqRkM5a24TkgDZvv24O5xxZH/7Yzen6cGV APSUKHmlOJNriV4YXFqKpjqE6lnnLsVy7IXo5wxFBXl1fR7i0IYftO5SzvhWXYbq6a6W e9dVf6/GHrAo6Yko6mG95a4xlakBKHMM5yPK/J9q8oHAUbBXRm83nm8hC4dm9wUCqJe3 Mb58abuuHBSIqdBlO1HfcTmWxhsnP4KAyBv9ocCQSYqgTHl1hmnOSO541y9UuiUibcih 7zkg== X-Gm-Message-State: AIkVDXJbyUOFfKywvHzd49xj6sZskVlqVnQWtTxXU3baTAq/vm9OXXqU/OmDTsQ3f8PMmBDTmPn9sM5oVmGv6w== X-Received: by 10.237.60.200 with SMTP id e8mr61198020qtf.248.1483512271975; Tue, 03 Jan 2017 22:44:31 -0800 (PST) MIME-Version: 1.0 References: <88A92D351643BA4CB23E30315517062662F595DF@SHSMSX103.ccr.corp.intel.com> <20161227105253.4dc7d44a@xeon-e3> In-Reply-To: From: edgar helmut Date: Wed, 04 Jan 2017 06:44:21 +0000 Message-ID: To: Stephen Hemminger Cc: "Hu, Xuekun" , "Wiles, Keith" , "users@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Dpdk poor performance on virtual machine X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jan 2017 06:44:33 -0000 finally hugepages issue was solved. it looks like there are some confusion between /dev/hugepages and /run/hugepages. solved by changing the "Where" at /lib/systemd/system/dev-hugepages.mount from "/dev/hugepage" to "/run/hugepages/kvm" ... hope this will help others= . The bad news are, it didn't improve the performance dramatically. digging into that and comparing with real passthrough of the pci it looks like there is a real problem with the macvtap performance as suggested by Hu. Using macvtap the dpdk in the vm could poll up to ~5-6 gbps and after that it is plenty with drops (missed pkts??), furthermore, the testpmd also lost packets in its pipline (e.g. rx X pkts and tx Y pkts while X > Y), but I don't know if it is because of the testpmd pipeling or the tx to the second macvtap based interface. However on pure pci passthrough i could transfer 10gbps without any problem... All of this were tested under 2, 3, 4 cores for the testpmd. so to summarize, don't use macvtap (virtio, vhost) if you want to scale to more than few gbps, use pci passthrough. these are bad news for me as I can't have 10gbps throughput while capable to count pkts in/out to and from the vm. If someone has an idea how to do that i will be more than happy to hear. edgar On Wed, Dec 28, 2016 at 10:09 AM edgar helmut wrote: > I tried this procedure as well as few others. > at all of the procedures the HugePages_Free or HugePages_Rsvd doesn't > change when creating the guest, though only AnonHugePages increases. > I am struggling with it for a while without real success. > I do suspect that this anonymous hugepages doesn't really make the job. > > any idea? > > > On Tue, Dec 27, 2016 at 8:52 PM Stephen Hemminger < > stephen@networkplumber.org> wrote: > > On Tue, 27 Dec 2016 15:59:08 +0000 > edgar helmut wrote: > > > short explanation for how to read the comparison: > > first row is packet length > > throughput is half duplex, means: > > second row is vm throughput of port 1 to 2 (port 2 to 1 has approximate= ly > > same throughput) in gbps. > > third row is host throughput of port 1 to 2 (port 2 to 1 has > approximately > > same throughput) in gbps. > > > > i.e. on 1500 bytes packet size testpmd delivers ~9.82 gbps from port 1 > to 2 > > and another ~9.82 gbps from port 2 to 1, while at the vm it only delive= rs > > ~3.9 gbps for each direction. > > > > > > On Tue, Dec 27, 2016 at 5:52 PM edgar helmut > > wrote: > > > > > Thanks. That's the document i am following. > > > For the best i can only ask that the hugepages won't be shared with > > > others, but it never reserve it from the pre allocated hugepages of t= he > > > host. > > > Did you have a chance to use hugepages for a guest > > > > > > as for the interfaces, i am using the virtio/vhost which creates the > > > macvtap: > > > > > > > > > > > > > > > > > >
> > function=3D'0x0'/> > > > > > > > > > The following is a performance comparison host vs. vm using testpmd. = as > > > you can see vm performance is poor. > > > > > > (sudo x86_64-native-linuxapp-gcc/app/testpmd -c 0x1f -n 3 -m 1024 -- > > > --coremask=3D0x1e --portmask=3D3 -i) > > > > > > > > > 64 128 256 500 800 1000 1500 > > > vm 0.23 0.42 0.75 1.3 2.3 2.7 3.9 > > > host 3.6 6.35 8.3 9.5 9.7 9.8 9.82 > > > > > > I have to improve it dramatically. > > > > > > > > > > > > On Mon, Dec 26, 2016 at 2:52 AM Hu, Xuekun > wrote: > > > > > > Searching =E2=80=9Chugepages=E2=80=9D in https://libvirt.org/formatdo= main.html > > > > > > > > > > > > If you are looking for to measure in and out packets through host, > maybe > > > you can look at vhost/virtio interface also. > > > > > > > > > > > > After your testing, if you can report the performace out with macvtap= , > > > that also helps us. J > > > > > > > > > > > > > > > > > > *From:* edgar helmut [mailto:helmut.edgar100@gmail.com] > > > *Sent:* Saturday, December 24, 2016 11:53 PM > > > > > > > > > *To:* Hu, Xuekun > > > *Cc:* Wiles, Keith ; users@dpdk.org > > > *Subject:* Re: [dpdk-users] Dpdk poor performance on virtual machine > > > > > > > > > > > > any idea how to reserve hugepages for a guest (and not > > > transparent/anonymous hugepages) ? > > > > > > i am using libvirt and any backing method I am trying results with > > > anonymous hugepage. > > > > > > disabling the transparent hugepages resulted without any hugepages. > > > > > > Thanks > > > > > > > > > > > > On Sat, Dec 24, 2016 at 10:06 AM edgar helmut < > helmut.edgar100@gmail.com> > > > wrote: > > > > > > I am looking for a mean to measure in and out packets to and from the > vm > > > (without asking the vm itself). While pure passthrough doesn't expose > an > > > interface to query for in/out pkts the macvtap exposes such an > interface. > > > > > > As for the anonymous hugepages I was looking for a more flexible meth= od > > > and I assumed there is no much difference. > > > > > > I will make the test with reserved hugepages. > > > > > > However is there any knowledge about macvtap performance issues when > > > delivering 5-6 gbps? > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > On 24 Dec 2016 9:06 AM, "Hu, Xuekun" wrote: > > > > > > Now your setup has a new thing, =E2=80=9Cmacvtap=E2=80=9D. I don=E2= =80=99t know what=E2=80=99s the > > > performance of using macvtap. I only know it has much worse perf than > the > > > =E2=80=9Creal=E2=80=9D pci pass-through. > > > > > > > > > > > > I also don=E2=80=99t know why you select such config for your setup, = anonymous > > > huge pages and macvtap. Any specific purpose? > > > > > > > > > > > > I think you should get a baseline first, then to get how much perf > dropped > > > if using anonymous hugepages or macvtap=E3=80=82 > > > > > > 1. Baseline: real hugepage + real pci pass-through > > > > > > 2. Anon hugepages vs hugepages > > > > > > 3. Real pci pass-through vs. macvtap > > > > > > > > > > > > *From:* edgar helmut [mailto:helmut.edgar100@gmail.com] > > > *Sent:* Saturday, December 24, 2016 3:23 AM > > > *To:* Hu, Xuekun > > > *Cc:* Wiles, Keith ; users@dpdk.org > > > > > > > > > *Subject:* Re: [dpdk-users] Dpdk poor performance on virtual machine > > > > > > > > > > > > Hello, > > > > > > I changed the setup but still performance are poor :( and I need your > help > > > to understand the root cause. > > > > > > the setup is (sorry for long description): > > > > > > (test equipment is pktgen using dpdk installed on a second physical > > > machine coonected with 82599 NICs) > > > > > > host: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz with single socket , > > > ubuntu 16.04, with 4 hugepages of 1G each. > > > > > > hypervizor (kvm): QEMU emulator version 2.5.0 > > > > > > guest: same cpu as host, created with 3 vcpus, using ubuntu 16.04 > > > > > > dpdk: tried 2.2, 16.04, 16.07, 16.11 - using testpmd and 512 pages of > 2M > > > each. > > > > > > guest total memory is 2G and all of it is backed by the host with > > > transparent hugepages (I can see the AnonHugePages consumed at guest > > > creation). This memory includes the 512 hugepages for the testpmd > > > application. > > > > > > I pinned and isolated the guest's vcpus (using kernel option isolcapu= ), > > > and could see clearly that the isolation functions well. > > > > > > > > > > > > 2 x 82599 NICs connected as passthrough using macvtap interfaces to t= he > > > guest, so the guest receives and forwards packets from one interface > to the > > > second and vice versa. > > > > > > at the guest I bind its interfaces using igb_uio. > > > > > > the testpmd at guest starts dropping packets at about ~800mbps betwee= n > > > both ports bi-directional using two vcpus for forwarding (one for the > > > application management and two for forwarding). > > > > > > at 1.2 gbps it drops a lot of packets. > > > > > > the same testpmd configuration on the host (between both 82599 NICs) > > > forwards about 5-6gbps on both ports bi-directional. > > > > > > I assumed that forwarding ~5-6 gbps between two ports should be > trivial, > > > so it will be great if someone can share its configuration for a test= ed > > > setup. > > > > > > Any further idea will be highly appreciated. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > On Sat, Dec 17, 2016 at 2:56 PM edgar helmut < > helmut.edgar100@gmail.com> > > > wrote: > > > > > > That's what I afraid. > > > > > > In fact i need the host to back the entire guest's memory with > hugepages. > > > > > > I will find the way to do that and make the testing again. > > > > > > > > > > > > > > > > > > On 16 Dec 2016 3:14 AM, "Hu, Xuekun" wrote: > > > > > > You said VM=E2=80=99s memory was 6G, while transparent hugepages was = only used > ~4G > > > (4360192KB). So some were mapped to 4K pages. > > > > > > > > > > > > BTW, the memory used by transparent hugepage is not the hugepage you > > > reserved in kernel boot option. > > > > > > > > > > > > *From:* edgar helmut [mailto:helmut.edgar100@gmail.com] > > > *Sent:* Friday, December 16, 2016 1:24 AM > > > *To:* Hu, Xuekun > > > *Cc:* Wiles, Keith; users@dpdk.org > > > *Subject:* Re: [dpdk-users] Dpdk poor performance on virtual machine > > > > > > > > > > > > in fact the vm was created with 6G RAM, its kernel boot args are > defined > > > with 4 hugepages of 1G each, though when starting the vm i noted that > > > anonhugepages increased. > > > > > > The relevant qemu process id is 6074, and the following sums the > amount of > > > allocated AnonHugePages: > > > sudo grep -e AnonHugePages /proc/6074/smaps | awk '{ if($2>0) print > $2} > > > '|awk '{s+=3D$1} END {print s}' > > > > > > which results with 4360192 > > > > > > so not all the memory is backed with transparent hugepages though it = is > > > more than the amount of hugepages the guest supposed to boot with. > > > > > > How can I be sure that the required 4G hugepages are really allocated= ?, > > > and not, for example, only 2G out of the 4G are allocated (and the > rest 2 > > > are mapping of the default 4K)? > > > > > > > > > > > > thanks > > > > > > > > > > > > On Thu, Dec 15, 2016 at 4:33 PM, Hu, Xuekun > wrote: > > > > > > Are you sure the anonhugepages size was equal to the total VM's memor= y > > > size? > > > Sometimes, transparent huge page mechanism doesn't grantee the app is > using > > > the real huge pages. > > > > > > > > > > > > -----Original Message----- > > > From: users [mailto:users-bounces@dpdk.org] On Behalf Of edgar helmut > > > Sent: Thursday, December 15, 2016 9:32 PM > > > To: Wiles, Keith > > > Cc: users@dpdk.org > > > Subject: Re: [dpdk-users] Dpdk poor performance on virtual machine > > > > > > I have one single socket which is Intel(R) Xeon(R) CPU E5-2640 v4 @ > > > 2.40GHz. > > > > > > I just made two more steps: > > > 1. setting iommu=3Dpt for better usage of the igb_uio > > > 2. using taskset and isolcpu so now it looks like the relevant dpdk > cores > > > use dedicated cores. > > > > > > It improved the performance though I still see significant difference > > > between the vm and the host which I can't fully explain. > > > > > > any further idea? > > > > > > Regards, > > > Edgar > > > > > > > > > On Thu, Dec 15, 2016 at 2:54 PM, Wiles, Keith > > > wrote: > > > > > > > > > > > > On Dec 15, 2016, at 1:20 AM, edgar helmut < > helmut.edgar100@gmail.com> > > > > wrote: > > > > > > > > > > Hi. > > > > > Some help is needed to understand performance issue on virtual > machine. > > > > > > > > > > Running testpmd over the host functions well (testpmd forwards 10= g > > > > between > > > > > two 82599 ports). > > > > > However same application running on a virtual machine over same > host > > > > > results with huge degradation in performance. > > > > > The testpmd then is not even able to read 100mbps from nic withou= t > > > drops, > > > > > and from a profile i made it looks like a dpdk application runs > more > > > than > > > > > 10 times slower than over host=E2=80=A6 > > > > > > > > Not sure I understand the overall setup, but did you make sure the > > > NIC/PCI > > > > bus is on the same socket as the VM. If you have multiple sockets o= n > your > > > > platform. If you have to access the NIC across the QPI it could > explain > > > > some of the performance drop. Not sure that much drop is this > problem. > > > > > > > > > > > > > > Setup is ubuntu 16.04 for host and ubuntu 14.04 for guest. > > > > > Qemu is 2.3.0 (though I tried with a newer as well). > > > > > NICs are connected to guest using pci passthrough, and guest's cp= u > is > > > set > > > > > as passthrough (same as host). > > > > > On guest start the host allocates transparent hugepages > (AnonHugePages) > > > > so > > > > > i assume the guest memory is backed with real hugepages on the > host. > > > > > I tried binding with igb_uio and with uio_pci_generic but both > results > > > > with > > > > > same performance. > > > > > > > > > > Due to the performance difference i guess i miss something. > > > > > > > > > > Please advise what may i miss here? > > > > > Is this a native penalty of qemu?? > > > > > > > > > > Thanks > > > > > Edgar > > Did you setup KVM host to run guest in huge pages? > > https://access.redhat.com/solutions/36741 > >