DPDK usage discussions
 help / color / mirror / Atom feed
From: Stephen Hemminger <stephen@networkplumber.org>
To: "Hu, Xuekun" <xuekun.hu@intel.com>
Cc: edgar helmut <helmut.edgar100@gmail.com>,
	"Wiles, Keith" <keith.wiles@intel.com>,
	"users@dpdk.org" <users@dpdk.org>
Subject: Re: [dpdk-users] Dpdk poor performance on virtual machine
Date: Thu, 15 Dec 2016 09:17:40 -0800	[thread overview]
Message-ID: <20161215091740.0d34defe@xeon-e3> (raw)
In-Reply-To: <88A92D351643BA4CB23E30315517062662F3C939@SHSMSX103.ccr.corp.intel.com>

On Thu, 15 Dec 2016 14:33:25 +0000
"Hu, Xuekun" <xuekun.hu@intel.com> wrote:

> Are you sure the anonhugepages size was equal to the total VM's memory size? 
> Sometimes, transparent huge page mechanism doesn't grantee the app is using
> the real huge pages. 
>  
> 
> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of edgar helmut
> Sent: Thursday, December 15, 2016 9:32 PM
> To: Wiles, Keith
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] Dpdk poor performance on virtual machine
> 
> I have one single socket which is Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz.
> 
> I just made two more steps:
> 1. setting iommu=pt for better usage of the igb_uio
> 2. using taskset and isolcpu so now it looks like the relevant dpdk cores
> use dedicated cores.
> 
> It improved the performance though I still see significant difference
> between the vm and the host which I can't fully explain.
> 
> any further idea?
> 
> Regards,
> Edgar
> 
> 
> On Thu, Dec 15, 2016 at 2:54 PM, Wiles, Keith <keith.wiles@intel.com> wrote:
> 
> >  
> > > On Dec 15, 2016, at 1:20 AM, edgar helmut <helmut.edgar100@gmail.com>  
> > wrote:  
> > >
> > > Hi.
> > > Some help is needed to understand performance issue on virtual machine.
> > >
> > > Running testpmd over the host functions well (testpmd forwards 10g  
> > between  
> > > two 82599 ports).
> > > However same application running on a virtual machine over same host
> > > results with huge degradation in performance.
> > > The testpmd then is not even able to read 100mbps from nic without drops,
> > > and from a profile i made it looks like a dpdk application runs more than
> > > 10 times slower than over host…  
> >
> > Not sure I understand the overall setup, but did you make sure the NIC/PCI
> > bus is on the same socket as the VM. If you have multiple sockets on your
> > platform. If you have to access the NIC across the QPI it could explain
> > some of the performance drop. Not sure that much drop is this problem.
> >  
> > >
> > > Setup is ubuntu 16.04 for host and ubuntu 14.04 for guest.
> > > Qemu is 2.3.0 (though I tried with a newer as well).
> > > NICs are connected to guest using pci passthrough, and guest's cpu is set
> > > as passthrough (same as host).
> > > On guest start the host allocates transparent hugepages (AnonHugePages)  
> > so  
> > > i assume the guest memory is backed with real hugepages on the host.
> > > I tried binding with igb_uio and with uio_pci_generic but both results  
> > with  
> > > same performance.
> > >
> > > Due to the performance difference i guess i miss something.
> > >
> > > Please advise what may i miss here?
> > > Is this a native penalty of qemu??
> > >
> > > Thanks
> > > Edgar  
> >
> > Regards,
> > Keith
> >
> >  

Also make sure you run host with 1G hugepages and run guest in hugepage
memory. If not, the IOMMU has to do 4K operations and thrashes.

  reply	other threads:[~2016-12-15 17:17 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CABc_bMBYNa7jdhqu2SNZkwtoJskCwQWGOZrB71RYzBZx5-OTfw@mail.gmail.com>
     [not found] ` <CABc_bMBMxhbkGSg82tw7CKLw7AiccEESLAJaHGA6PAaYAPCmTg@mail.gmail.com>
     [not found]   ` <CABc_bMB2vnXEpacfeiWiC5X-x8iuB7jO6fR67n5jj67Pspcf2g@mail.gmail.com>
     [not found]     ` <CABc_bMB7C2zRjEkFbtAyk7buaP=QWwxCSwz=60AVJT_rjoPZTQ@mail.gmail.com>
     [not found]       ` <CABc_bMDgU62=i5uE52d12VGd8wUzqHq9siOjYKSh8v811TYpug@mail.gmail.com>
     [not found]         ` <CABc_bMCn9LivEn_PJCh+dQy7pPypLdmE+eZFQZtzphsHEgr08g@mail.gmail.com>
     [not found]           ` <CABc_bMADQ3PaZz5ovLmGu=3_w2pTO3c6BvNPM6-BUzJ5BXjxLQ@mail.gmail.com>
     [not found]             ` <CABc_bMBJ+NZLuo___qw_cBB=z=5nzgVx6rMd15nMiSYf_q3WoQ@mail.gmail.com>
     [not found]               ` <CABc_bMC6qP3K-kqVQORc9XKfbcXX63UfN=AdZ+sksHUN+Bx5kw@mail.gmail.com>
     [not found]                 ` <CABc_bMDowcZKMrKc8omf-JqpUW=uPn-fq8sfLYk=AktbH9-aNw@mail.gmail.com>
     [not found]                   ` <CABc_bMACisgaHBZedG05ZzJ3wzmudgBeTHdRr93M3-QOGKDKNw@mail.gmail.com>
     [not found]                     ` <CABc_bMCTRDYy+9ZO6py+KupRStR=Rc4md+J0NhPcyTqaZhKxTA@mail.gmail.com>
     [not found]                       ` <CABc_bMAu8BkZXzBSZLWNs=R6AJgMAw9WrTki=cEzMzjC7Z8LAQ@mail.gmail.com>
     [not found]                         ` <CABc_bMDtzYE4NvhAT8nqi3qZbzV4WauzJLW-tcY_Wi5i88F7yQ@mail.gmail.com>
     [not found]                           ` <CABc_bMAr0EjgH4f7UT-oi353FBqYwxt-UgfP6candcP=jkuyLg@mail.gmail.com>
     [not found]                             ` <CABc_bMCyJ4o94T=VRWPjfyhXP1T3uDmeYVsu+0OrXi1AqUkLrg@mail.gmail.com>
     [not found]                               ` <CABc_bMBK=AQ31=mQX0f7HMXs6REqj_bkqQ9KPD96vbtoKKHWCw@mail.gmail.com>
     [not found]                                 ` <CABc_bMA8z6XBEtnxwPE_LDpngx_DYRathn17KqFxNx2V5DFbng@mail.gmail.com>
     [not found]                                   ` <CABc_bMBwFAL5rc5goMP9pt+2z=TOM=VNfp76AznZ3jend9aS_Q@mail.gmail.com>
     [not found]                                     ` <CABc_bMBr1c_Evd5zxB89SinNFVMUGHFTBDoLusVHdMySBCyaBA@mail.gmail.com>
     [not found]                                       ` <CABc_bMChD5hpWmj1zhvZ5tocsxDCFH0QdE34TKWXsdo_danH-Q@mail.gmail.com>
     [not found]                                         ` <CABc_bMCqu-4V4gn=JPqO491BF6Cnjj=4SaAey9qyTQcha134yw@mail.gmail.com>
2016-12-15  7:20                                           ` edgar helmut
2016-12-15 12:54                                             ` Wiles, Keith
2016-12-15 13:32                                               ` edgar helmut
2016-12-15 14:33                                                 ` Hu, Xuekun
2016-12-15 17:17                                                   ` Stephen Hemminger [this message]
2016-12-15 17:29                                                     ` edgar helmut
2016-12-15 19:14                                                       ` Stephen Hemminger
2016-12-15 19:29                                                         ` Jes Nielsen
2016-12-15 17:24                                                   ` edgar helmut
2016-12-16  1:14                                                     ` Hu, Xuekun
2016-12-17 12:56                                                       ` edgar helmut
2016-12-23 19:22                                                         ` edgar helmut
2016-12-24  7:06                                                           ` Hu, Xuekun
2016-12-24  8:06                                                             ` edgar helmut
2016-12-24 15:52                                                               ` edgar helmut
2016-12-26  0:52                                                                 ` Hu, Xuekun
2016-12-27 15:52                                                                   ` edgar helmut
2016-12-27 15:59                                                                     ` edgar helmut
2016-12-27 18:52                                                                       ` Stephen Hemminger
2016-12-28  8:09                                                                         ` edgar helmut
2017-01-04  6:44                                                                           ` edgar helmut

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20161215091740.0d34defe@xeon-e3 \
    --to=stephen@networkplumber.org \
    --cc=helmut.edgar100@gmail.com \
    --cc=keith.wiles@intel.com \
    --cc=users@dpdk.org \
    --cc=xuekun.hu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).