DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] Will huge page have negative effect on guest vm in qemu enviroment?
@ 2017-06-21  3:32 Sam
  2017-06-21  3:35 ` Sam
  0 siblings, 1 reply; 7+ messages in thread
From: Sam @ 2017-06-21  3:32 UTC (permalink / raw)
  To: dev, users, qemu-devel, qemu-discuss

Hi all,

We plan to use DPDK on HP host machine with several core and big memory. We
plan to use qemu-kvm enviroment. The host will carry 4 or more guest vm and
1 ovs.

Ovs-dpdk is much faster then normal ovs, but to use ovs-dpdk, we have to
enable huge page globally.

My question is, will huge page enabled globally have negative effect on
guest vm's memory orperate or something? If it is, how to prevent this, or
could I enable huge page on some core or enable huge page for a part of
memory?

Thank you~

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] Will huge page have negative effect on guest vm in qemu enviroment?
  2017-06-21  3:32 [dpdk-users] Will huge page have negative effect on guest vm in qemu enviroment? Sam
@ 2017-06-21  3:35 ` Sam
  2017-06-21  6:15   ` [dpdk-users] [dpdk-dev] " Pavel Shirshov
  0 siblings, 1 reply; 7+ messages in thread
From: Sam @ 2017-06-21  3:35 UTC (permalink / raw)
  To: dev, users, qemu-devel, qemu-discuss

BTW, we also think about use ovs-dpdk in docker enviroment, but test result
said it's not good idea, we don't know why.

2017-06-21 11:32 GMT+08:00 Sam <batmanustc@gmail.com>:

> Hi all,
>
> We plan to use DPDK on HP host machine with several core and big memory.
> We plan to use qemu-kvm enviroment. The host will carry 4 or more guest vm
> and 1 ovs.
>
> Ovs-dpdk is much faster then normal ovs, but to use ovs-dpdk, we have to
> enable huge page globally.
>
> My question is, will huge page enabled globally have negative effect on
> guest vm's memory orperate or something? If it is, how to prevent this, or
> could I enable huge page on some core or enable huge page for a part of
> memory?
>
> Thank you~
>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] [dpdk-dev] Will huge page have negative effect on guest vm in qemu enviroment?
  2017-06-21  3:35 ` Sam
@ 2017-06-21  6:15   ` Pavel Shirshov
  2017-06-21  7:22     ` Sam
  0 siblings, 1 reply; 7+ messages in thread
From: Pavel Shirshov @ 2017-06-21  6:15 UTC (permalink / raw)
  To: Sam; +Cc: dev, users, qemu-devel, qemu-discuss

Hi Sam,

Below I'm saying about KVM. I don't have experience with vbox and others.
1. I'd suggest don't use dpdk inside of VM if you want to see best
perfomance on the box.
2. huge pages enabled globally will not have any bad effect to guest
OS. Except you have to enable huge pages inside of VM and provide real
huge page for VM's huge pages from the host system. Otherwise dpdk
will use "hugepages" inside of VM, but this "huge pages" will not real
ones. They will be constructed from normal pages outside. Also when
you enable huge pages OS will reserve them from start and your OS will
not able use them for other things. Also you can't swap out huge
pages, KSM will not work for them and so on.
3. You can enable huge pages just for one numa node. It's impossible
to enable them just for one core. Usually you reserve some memory for
hugepages when the system starts and you can't use this memory in
normal applications unless normal application knows how to use them.

Also why it didn't work inside of the docker?


On Tue, Jun 20, 2017 at 8:35 PM, Sam <batmanustc@gmail.com> wrote:
> BTW, we also think about use ovs-dpdk in docker enviroment, but test result
> said it's not good idea, we don't know why.
>
> 2017-06-21 11:32 GMT+08:00 Sam <batmanustc@gmail.com>:
>
>> Hi all,
>>
>> We plan to use DPDK on HP host machine with several core and big memory.
>> We plan to use qemu-kvm enviroment. The host will carry 4 or more guest vm
>> and 1 ovs.
>>
>> Ovs-dpdk is much faster then normal ovs, but to use ovs-dpdk, we have to
>> enable huge page globally.
>>
>> My question is, will huge page enabled globally have negative effect on
>> guest vm's memory orperate or something? If it is, how to prevent this, or
>> could I enable huge page on some core or enable huge page for a part of
>> memory?
>>
>> Thank you~
>>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] [dpdk-dev] Will huge page have negative effect on guest vm in qemu enviroment?
  2017-06-21  6:15   ` [dpdk-users] [dpdk-dev] " Pavel Shirshov
@ 2017-06-21  7:22     ` Sam
  2017-06-21  9:03       ` Gaëtan Rivet
                         ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Sam @ 2017-06-21  7:22 UTC (permalink / raw)
  To: Pavel Shirshov; +Cc: dev, users, qemu-devel, qemu-discuss

Thank you~

1. We have a compare test on qemu-kvm enviroment with huge page and without
huge page. Qemu start process is much longer in huge page enviromwnt. And I
write an email titled with '[DPDK-memory] how qemu waste such long time
under dpdk huge page envriment?'. I could resend it later.

2. Then I have another test on qemu-kvm enviroment with huge page and
without huge page, which I didn't start ovs-dpdk and vhostuser port in qemu
start process. And I found Qemu start process is also much longer in huge
page enviroment.

So I think huge page enviroment, which grub2.cfg file is specified in
‘[DPDK-memory]
how qemu waste such long time under dpdk huge page envriment?’, will really
have negative effect on qemu start up process.

That's why we don't like to use ovs-dpdk. Althrough ovs-dpdk is faster, but
the start up process of qemu is much longer then normal ovs, and the reason
is nothing with ovs but huge page. For customers, vm start up time is
important then network speed.

BTW, ovs-dpdk start up process is also longer then normal ovs. But I know
the reason, it's dpdk EAL init process with forking big continous memory
and zero this memory. For qemu, I don't know why, as there is no log to
report this.

2017-06-21 14:15 GMT+08:00 Pavel Shirshov <pavel.shirshov@gmail.com>:

> Hi Sam,
>
> Below I'm saying about KVM. I don't have experience with vbox and others.
> 1. I'd suggest don't use dpdk inside of VM if you want to see best
> perfomance on the box.
> 2. huge pages enabled globally will not have any bad effect to guest
> OS. Except you have to enable huge pages inside of VM and provide real
> huge page for VM's huge pages from the host system. Otherwise dpdk
> will use "hugepages" inside of VM, but this "huge pages" will not real
> ones. They will be constructed from normal pages outside. Also when
> you enable huge pages OS will reserve them from start and your OS will
> not able use them for other things. Also you can't swap out huge
> pages, KSM will not work for them and so on.
> 3. You can enable huge pages just for one numa node. It's impossible
> to enable them just for one core. Usually you reserve some memory for
> hugepages when the system starts and you can't use this memory in
> normal applications unless normal application knows how to use them.
>
> Also why it didn't work inside of the docker?
>
>
> On Tue, Jun 20, 2017 at 8:35 PM, Sam <batmanustc@gmail.com> wrote:
> > BTW, we also think about use ovs-dpdk in docker enviroment, but test
> result
> > said it's not good idea, we don't know why.
> >
> > 2017-06-21 11:32 GMT+08:00 Sam <batmanustc@gmail.com>:
> >
> >> Hi all,
> >>
> >> We plan to use DPDK on HP host machine with several core and big memory.
> >> We plan to use qemu-kvm enviroment. The host will carry 4 or more guest
> vm
> >> and 1 ovs.
> >>
> >> Ovs-dpdk is much faster then normal ovs, but to use ovs-dpdk, we have to
> >> enable huge page globally.
> >>
> >> My question is, will huge page enabled globally have negative effect on
> >> guest vm's memory orperate or something? If it is, how to prevent this,
> or
> >> could I enable huge page on some core or enable huge page for a part of
> >> memory?
> >>
> >> Thank you~
> >>
>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] [dpdk-dev] Will huge page have negative effect on guest vm in qemu enviroment?
  2017-06-21  7:22     ` Sam
@ 2017-06-21  9:03       ` Gaëtan Rivet
  2017-06-21 19:16       ` [dpdk-users] [Qemu-devel] " Dr. David Alan Gilbert
  2017-06-22  9:24       ` [dpdk-users] " Alejandro Lucero
  2 siblings, 0 replies; 7+ messages in thread
From: Gaëtan Rivet @ 2017-06-21  9:03 UTC (permalink / raw)
  To: Sam; +Cc: Pavel Shirshov, dev, users

Hi Sam,

On Wed, Jun 21, 2017 at 03:22:45PM +0800, Sam wrote:
> Thank you~
> 
> 1. We have a compare test on qemu-kvm enviroment with huge page and without
> huge page. Qemu start process is much longer in huge page enviromwnt. And I
> write an email titled with '[DPDK-memory] how qemu waste such long time
> under dpdk huge page envriment?'. I could resend it later.
> 

Are you using 2M hugepages? Do you see any difference with 1G hugepages?
The smaller ones should not incur such delay.

On a side note, if the VM is properly configured the performance loss
should be negligible, and mostly visible in benchmark contexts with
little or no treatment done on the packets.

> 2. Then I have another test on qemu-kvm enviroment with huge page and
> without huge page, which I didn't start ovs-dpdk and vhostuser port in qemu
> start process. And I found Qemu start process is also much longer in huge
> page enviroment.
> 
> So I think huge page enviroment, which grub2.cfg file is specified in
> ‘[DPDK-memory]
> how qemu waste such long time under dpdk huge page envriment?’, will really
> have negative effect on qemu start up process.
> 
> That's why we don't like to use ovs-dpdk. Althrough ovs-dpdk is faster, but
> the start up process of qemu is much longer then normal ovs, and the reason
> is nothing with ovs but huge page. For customers, vm start up time is
> important then network speed.
> 
> BTW, ovs-dpdk start up process is also longer then normal ovs. But I know
> the reason, it's dpdk EAL init process with forking big continous memory
> and zero this memory. For qemu, I don't know why, as there is no log to
> report this.
> 
> 2017-06-21 14:15 GMT+08:00 Pavel Shirshov <pavel.shirshov@gmail.com>:
> 
> > Hi Sam,
> >
> > Below I'm saying about KVM. I don't have experience with vbox and others.
> > 1. I'd suggest don't use dpdk inside of VM if you want to see best
> > perfomance on the box.
> > 2. huge pages enabled globally will not have any bad effect to guest
> > OS. Except you have to enable huge pages inside of VM and provide real
> > huge page for VM's huge pages from the host system. Otherwise dpdk
> > will use "hugepages" inside of VM, but this "huge pages" will not real
> > ones. They will be constructed from normal pages outside. Also when
> > you enable huge pages OS will reserve them from start and your OS will
> > not able use them for other things. Also you can't swap out huge
> > pages, KSM will not work for them and so on.
> > 3. You can enable huge pages just for one numa node. It's impossible
> > to enable them just for one core. Usually you reserve some memory for
> > hugepages when the system starts and you can't use this memory in
> > normal applications unless normal application knows how to use them.
> >
> > Also why it didn't work inside of the docker?
> >
> >
> > On Tue, Jun 20, 2017 at 8:35 PM, Sam <batmanustc@gmail.com> wrote:
> > > BTW, we also think about use ovs-dpdk in docker enviroment, but test
> > result
> > > said it's not good idea, we don't know why.
> > >
> > > 2017-06-21 11:32 GMT+08:00 Sam <batmanustc@gmail.com>:
> > >
> > >> Hi all,
> > >>
> > >> We plan to use DPDK on HP host machine with several core and big memory.
> > >> We plan to use qemu-kvm enviroment. The host will carry 4 or more guest
> > vm
> > >> and 1 ovs.
> > >>
> > >> Ovs-dpdk is much faster then normal ovs, but to use ovs-dpdk, we have to
> > >> enable huge page globally.
> > >>
> > >> My question is, will huge page enabled globally have negative effect on
> > >> guest vm's memory orperate or something? If it is, how to prevent this,
> > or
> > >> could I enable huge page on some core or enable huge page for a part of
> > >> memory?
> > >>
> > >> Thank you~
> > >>
> >

-- 
Gaëtan Rivet
6WIND

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] [Qemu-devel] [dpdk-dev] Will huge page have negative effect on guest vm in qemu enviroment?
  2017-06-21  7:22     ` Sam
  2017-06-21  9:03       ` Gaëtan Rivet
@ 2017-06-21 19:16       ` Dr. David Alan Gilbert
  2017-06-22  9:24       ` [dpdk-users] " Alejandro Lucero
  2 siblings, 0 replies; 7+ messages in thread
From: Dr. David Alan Gilbert @ 2017-06-21 19:16 UTC (permalink / raw)
  To: Sam; +Cc: Pavel Shirshov, dev, qemu-discuss, users, qemu-devel

* Sam (batmanustc@gmail.com) wrote:
> Thank you~
> 
> 1. We have a compare test on qemu-kvm enviroment with huge page and without
> huge page. Qemu start process is much longer in huge page enviromwnt. And I
> write an email titled with '[DPDK-memory] how qemu waste such long time
> under dpdk huge page envriment?'. I could resend it later.

> 2. Then I have another test on qemu-kvm enviroment with huge page and
> without huge page, which I didn't start ovs-dpdk and vhostuser port in qemu
> start process. And I found Qemu start process is also much longer in huge
> page enviroment.
> 
> So I think huge page enviroment, which grub2.cfg file is specified in
> ‘[DPDK-memory]
> how qemu waste such long time under dpdk huge page envriment?’, will really
> have negative effect on qemu start up process.
> 
> That's why we don't like to use ovs-dpdk. Althrough ovs-dpdk is faster, but
> the start up process of qemu is much longer then normal ovs, and the reason
> is nothing with ovs but huge page. For customers, vm start up time is
> important then network speed.

How are you setting up hugepages?  What values are you putting in the
various /proc or cmdline options and how are you specifying them on
QEMU's commandline.

I think one problem is that with hugepages qemu normally allocates them
all at the start;  I think there are cases where that means moving a lot
of memory about, especially if you lock it to particular NUMA nodes.

> BTW, ovs-dpdk start up process is also longer then normal ovs. But I know
> the reason, it's dpdk EAL init process with forking big continous memory
> and zero this memory. For qemu, I don't know why, as there is no log to
> report this.

I suspect it's the mmaping and madvising of those hugepages - you should
be able to see it with an strace of a qemu startup, or perhaps a
'perf top'  on the host as it's in that pause.

I'm told that hugepages are supposed to be especially useful with IOMMU
performance for cards passed through to the guest, so it might still
be worth doing.

Dave

> 2017-06-21 14:15 GMT+08:00 Pavel Shirshov <pavel.shirshov@gmail.com>:
> 
> > Hi Sam,
> >
> > Below I'm saying about KVM. I don't have experience with vbox and others.
> > 1. I'd suggest don't use dpdk inside of VM if you want to see best
> > perfomance on the box.
> > 2. huge pages enabled globally will not have any bad effect to guest
> > OS. Except you have to enable huge pages inside of VM and provide real
> > huge page for VM's huge pages from the host system. Otherwise dpdk
> > will use "hugepages" inside of VM, but this "huge pages" will not real
> > ones. They will be constructed from normal pages outside. Also when
> > you enable huge pages OS will reserve them from start and your OS will
> > not able use them for other things. Also you can't swap out huge
> > pages, KSM will not work for them and so on.
> > 3. You can enable huge pages just for one numa node. It's impossible
> > to enable them just for one core. Usually you reserve some memory for
> > hugepages when the system starts and you can't use this memory in
> > normal applications unless normal application knows how to use them.
> >
> > Also why it didn't work inside of the docker?
> >
> >
> > On Tue, Jun 20, 2017 at 8:35 PM, Sam <batmanustc@gmail.com> wrote:
> > > BTW, we also think about use ovs-dpdk in docker enviroment, but test
> > result
> > > said it's not good idea, we don't know why.
> > >
> > > 2017-06-21 11:32 GMT+08:00 Sam <batmanustc@gmail.com>:
> > >
> > >> Hi all,
> > >>
> > >> We plan to use DPDK on HP host machine with several core and big memory.
> > >> We plan to use qemu-kvm enviroment. The host will carry 4 or more guest
> > vm
> > >> and 1 ovs.
> > >>
> > >> Ovs-dpdk is much faster then normal ovs, but to use ovs-dpdk, we have to
> > >> enable huge page globally.
> > >>
> > >> My question is, will huge page enabled globally have negative effect on
> > >> guest vm's memory orperate or something? If it is, how to prevent this,
> > or
> > >> could I enable huge page on some core or enable huge page for a part of
> > >> memory?
> > >>
> > >> Thank you~
> > >>
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] [dpdk-dev] Will huge page have negative effect on guest vm in qemu enviroment?
  2017-06-21  7:22     ` Sam
  2017-06-21  9:03       ` Gaëtan Rivet
  2017-06-21 19:16       ` [dpdk-users] [Qemu-devel] " Dr. David Alan Gilbert
@ 2017-06-22  9:24       ` Alejandro Lucero
  2 siblings, 0 replies; 7+ messages in thread
From: Alejandro Lucero @ 2017-06-22  9:24 UTC (permalink / raw)
  To: Sam; +Cc: Pavel Shirshov, dev, users, qemu-devel, qemu-discuss

On Wed, Jun 21, 2017 at 8:22 AM, Sam <batmanustc@gmail.com> wrote:

> Thank you~
>
> 1. We have a compare test on qemu-kvm enviroment with huge page and without
> huge page. Qemu start process is much longer in huge page enviromwnt. And I
> write an email titled with '[DPDK-memory] how qemu waste such long time
> under dpdk huge page envriment?'. I could resend it later.
>
>
DPDK vhost code does some work when configuring the VM virtio port. If
dequeue_zero_copy is set, the work to do is heavier, but I do not think
this could imply huge longer bootup times.


> 2. Then I have another test on qemu-kvm enviroment with huge page and
> without huge page, which I didn't start ovs-dpdk and vhostuser port in qemu
> start process. And I found Qemu start process is also much longer in huge
> page enviroment.
>

If hugepages are available, starting a qemu VM with hugepages should not
take too long. So in this case, I would say the hugepages need to be
allocated before the VM can boot. The only reason I can think of is
transparent hugepages in use, then the system needs to work for undoing
those transparent hugepages for allocating space to the VM hugepages.

Did you check how many hugepages are available just before starting the
QEMU VM?

cat /sys/kernel/mm/hugepages/hugepages-*/free_hugepages

If this is the problem, easy to solve just disabling transparent hugepages:

echo never > /sys/kernel/mm/transparent_hugepage/enabled


> So I think huge page enviroment, which grub2.cfg file is specified in
> ‘[DPDK-memory]
> how qemu waste such long time under dpdk huge page envriment?’, will really
> have negative effect on qemu start up process.
>
> That's why we don't like to use ovs-dpdk. Althrough ovs-dpdk is faster, but
> the start up process of qemu is much longer then normal ovs, and the reason
> is nothing with ovs but huge page. For customers, vm start up time is
> important then network speed.
>
> BTW, ovs-dpdk start up process is also longer then normal ovs. But I know
> the reason, it's dpdk EAL init process with forking big continous memory
> and zero this memory. For qemu, I don't know why, as there is no log to
> report this.
>

We had also problems with DPDK apps initialization with a server using
256GB. I guess this is something that maybe could be improved.


>
> 2017-06-21 14:15 GMT+08:00 Pavel Shirshov <pavel.shirshov@gmail.com>:
>
> > Hi Sam,
> >
> > Below I'm saying about KVM. I don't have experience with vbox and others.
> > 1. I'd suggest don't use dpdk inside of VM if you want to see best
> > perfomance on the box.
> > 2. huge pages enabled globally will not have any bad effect to guest
> > OS. Except you have to enable huge pages inside of VM and provide real
> > huge page for VM's huge pages from the host system. Otherwise dpdk
> > will use "hugepages" inside of VM, but this "huge pages" will not real
> > ones. They will be constructed from normal pages outside. Also when
> > you enable huge pages OS will reserve them from start and your OS will
> > not able use them for other things. Also you can't swap out huge
> > pages, KSM will not work for them and so on.
> > 3. You can enable huge pages just for one numa node. It's impossible
> > to enable them just for one core. Usually you reserve some memory for
> > hugepages when the system starts and you can't use this memory in
> > normal applications unless normal application knows how to use them.
> >
> > Also why it didn't work inside of the docker?
> >
> >
> > On Tue, Jun 20, 2017 at 8:35 PM, Sam <batmanustc@gmail.com> wrote:
> > > BTW, we also think about use ovs-dpdk in docker enviroment, but test
> > result
> > > said it's not good idea, we don't know why.
> > >
> > > 2017-06-21 11:32 GMT+08:00 Sam <batmanustc@gmail.com>:
> > >
> > >> Hi all,
> > >>
> > >> We plan to use DPDK on HP host machine with several core and big
> memory.
> > >> We plan to use qemu-kvm enviroment. The host will carry 4 or more
> guest
> > vm
> > >> and 1 ovs.
> > >>
> > >> Ovs-dpdk is much faster then normal ovs, but to use ovs-dpdk, we have
> to
> > >> enable huge page globally.
> > >>
> > >> My question is, will huge page enabled globally have negative effect
> on
> > >> guest vm's memory orperate or something? If it is, how to prevent
> this,
> > or
> > >> could I enable huge page on some core or enable huge page for a part
> of
> > >> memory?
> > >>
> > >> Thank you~
> > >>
> >
>

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2017-06-22  9:24 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-21  3:32 [dpdk-users] Will huge page have negative effect on guest vm in qemu enviroment? Sam
2017-06-21  3:35 ` Sam
2017-06-21  6:15   ` [dpdk-users] [dpdk-dev] " Pavel Shirshov
2017-06-21  7:22     ` Sam
2017-06-21  9:03       ` Gaëtan Rivet
2017-06-21 19:16       ` [dpdk-users] [Qemu-devel] " Dr. David Alan Gilbert
2017-06-22  9:24       ` [dpdk-users] " Alejandro Lucero

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).