* [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment?
@ 2017-06-16 8:11 Sam
2017-06-16 8:26 ` Sam
0 siblings, 1 reply; 5+ messages in thread
From: Sam @ 2017-06-16 8:11 UTC (permalink / raw)
To: dev, users
Hi all,
I'm running `QEMU_CMD ...` to create a vm under dpdk huge page envriment
(which set huge page 1G). And I enable all events in qemu.
For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail log is:
> 30012@1497443246.678304:object_dynamic_cast_assert
qemu:memory-region->qemu:memory-region (/home/hu
> anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn)
> 30012@1497443256.274866:object_dynamic_cast_assert
qio-channel-socket->qio-channel-socket (io/chann
> el-socket.c:389:qio_channel_socket_init)
I don't know why qemu doing 'memory_region_initfn' function in this 10
second, does anyone know this?
static void memory_region_initfn(Object *obj)
> {
> MemoryRegion *mr = MEMORY_REGION(obj);
> ObjectProperty *op;
> mr->ops = &unassigned_mem_ops;
> mr->enabled = true;
> mr->romd_mode = true;
> mr->global_locking = true;
> mr->destructor = memory_region_destructor_none;
> QTAILQ_INIT(&mr->subregions);
> QTAILQ_INIT(&mr->coalesced);
> op = object_property_add(OBJECT(mr), "container",
> "link<" TYPE_MEMORY_REGION ">",
> memory_region_get_container,
> NULL, /* memory_region_set_container */
> NULL, NULL, &error_abort);
> op->resolve = memory_region_resolve_container;
> object_property_add(OBJECT(mr), "addr", "uint64",
> memory_region_get_addr,
> NULL, /* memory_region_set_addr */
> NULL, NULL, &error_abort);
> object_property_add(OBJECT(mr), "priority", "uint32",
> memory_region_get_priority,
> NULL, /* memory_region_set_priority */
> NULL, NULL, &error_abort);
> object_property_add_bool(OBJECT(mr), "may-overlap",
> memory_region_get_may_overlap,
> NULL, /* memory_region_set_may_overlap */
> &error_abort);
> object_property_add(OBJECT(mr), "size", "uint64",
> memory_region_get_size,
> NULL, /* memory_region_set_size, */
> NULL, NULL, &error_abort);
> }
Thank you~
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment?
2017-06-16 8:11 [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? Sam
@ 2017-06-16 8:26 ` Sam
2017-06-16 8:59 ` Bruce Richardson
0 siblings, 1 reply; 5+ messages in thread
From: Sam @ 2017-06-16 8:26 UTC (permalink / raw)
To: dev, users
BTW, while running ovs-dpdk, this log is also take long time, does that
mean dpdk request large memory take long time?
EAL: Setting up physically contiguous memory...
2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>:
> Hi all,
>
> I'm running `QEMU_CMD ...` to create a vm under dpdk huge page envriment
> (which set huge page 1G). And I enable all events in qemu.
>
> For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail log
> is:
>
> > 30012@1497443246.678304:object_dynamic_cast_assert
> qemu:memory-region->qemu:memory-region (/home/hu
> > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn)
> > 30012@1497443256.274866:object_dynamic_cast_assert
> qio-channel-socket->qio-channel-socket (io/chann
> > el-socket.c:389:qio_channel_socket_init)
>
>
> I don't know why qemu doing 'memory_region_initfn' function in this 10
> second, does anyone know this?
>
> static void memory_region_initfn(Object *obj)
>> {
>> MemoryRegion *mr = MEMORY_REGION(obj);
>> ObjectProperty *op;
>> mr->ops = &unassigned_mem_ops;
>> mr->enabled = true;
>> mr->romd_mode = true;
>> mr->global_locking = true;
>> mr->destructor = memory_region_destructor_none;
>> QTAILQ_INIT(&mr->subregions);
>> QTAILQ_INIT(&mr->coalesced);
>> op = object_property_add(OBJECT(mr), "container",
>> "link<" TYPE_MEMORY_REGION ">",
>> memory_region_get_container,
>> NULL, /* memory_region_set_container */
>> NULL, NULL, &error_abort);
>> op->resolve = memory_region_resolve_container;
>> object_property_add(OBJECT(mr), "addr", "uint64",
>> memory_region_get_addr,
>> NULL, /* memory_region_set_addr */
>> NULL, NULL, &error_abort);
>> object_property_add(OBJECT(mr), "priority", "uint32",
>> memory_region_get_priority,
>> NULL, /* memory_region_set_priority */
>> NULL, NULL, &error_abort);
>> object_property_add_bool(OBJECT(mr), "may-overlap",
>> memory_region_get_may_overlap,
>> NULL, /* memory_region_set_may_overlap */
>> &error_abort);
>> object_property_add(OBJECT(mr), "size", "uint64",
>> memory_region_get_size,
>> NULL, /* memory_region_set_size, */
>> NULL, NULL, &error_abort);
>> }
>
>
> Thank you~
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment?
2017-06-16 8:26 ` Sam
@ 2017-06-16 8:59 ` Bruce Richardson
2017-06-19 3:12 ` Sam
0 siblings, 1 reply; 5+ messages in thread
From: Bruce Richardson @ 2017-06-16 8:59 UTC (permalink / raw)
To: Sam; +Cc: dev, users
On Fri, Jun 16, 2017 at 04:26:40PM +0800, Sam wrote:
> BTW, while running ovs-dpdk, this log is also take long time, does that
> mean dpdk request large memory take long time?
>
> EAL: Setting up physically contiguous memory...
>
When running with 1G pages, I found that the mmap system call takes a
considerable amount of time to execute. I think this is due to the
kernel zero-ing out the 1G pages. IIRC on one system I measured it as taking
about 0.4 seconds per 1G page.
/Bruce
>
> 2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>:
>
> > Hi all,
> >
> > I'm running `QEMU_CMD ...` to create a vm under dpdk huge page envriment
> > (which set huge page 1G). And I enable all events in qemu.
> >
> > For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail log
> > is:
> >
> > > 30012@1497443246.678304:object_dynamic_cast_assert
> > qemu:memory-region->qemu:memory-region (/home/hu
> > > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn)
> > > 30012@1497443256.274866:object_dynamic_cast_assert
> > qio-channel-socket->qio-channel-socket (io/chann
> > > el-socket.c:389:qio_channel_socket_init)
> >
> >
> > I don't know why qemu doing 'memory_region_initfn' function in this 10
> > second, does anyone know this?
> >
> > static void memory_region_initfn(Object *obj)
> >> {
> >> MemoryRegion *mr = MEMORY_REGION(obj);
> >> ObjectProperty *op;
> >> mr->ops = &unassigned_mem_ops;
> >> mr->enabled = true;
> >> mr->romd_mode = true;
> >> mr->global_locking = true;
> >> mr->destructor = memory_region_destructor_none;
> >> QTAILQ_INIT(&mr->subregions);
> >> QTAILQ_INIT(&mr->coalesced);
> >> op = object_property_add(OBJECT(mr), "container",
> >> "link<" TYPE_MEMORY_REGION ">",
> >> memory_region_get_container,
> >> NULL, /* memory_region_set_container */
> >> NULL, NULL, &error_abort);
> >> op->resolve = memory_region_resolve_container;
> >> object_property_add(OBJECT(mr), "addr", "uint64",
> >> memory_region_get_addr,
> >> NULL, /* memory_region_set_addr */
> >> NULL, NULL, &error_abort);
> >> object_property_add(OBJECT(mr), "priority", "uint32",
> >> memory_region_get_priority,
> >> NULL, /* memory_region_set_priority */
> >> NULL, NULL, &error_abort);
> >> object_property_add_bool(OBJECT(mr), "may-overlap",
> >> memory_region_get_may_overlap,
> >> NULL, /* memory_region_set_may_overlap */
> >> &error_abort);
> >> object_property_add(OBJECT(mr), "size", "uint64",
> >> memory_region_get_size,
> >> NULL, /* memory_region_set_size, */
> >> NULL, NULL, &error_abort);
> >> }
> >
> >
> > Thank you~
> >
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment?
2017-06-16 8:59 ` Bruce Richardson
@ 2017-06-19 3:12 ` Sam
2017-06-19 3:13 ` Sam
0 siblings, 1 reply; 5+ messages in thread
From: Sam @ 2017-06-19 3:12 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, users
Hi,
I print all system call by `strace -f -T -tt -e trace=all -o output.txt
$QEMU_CMD`, and I found this:
5900 11:08:11.288701 nanosleep({0, 10000000}, 0x7ff103c13a80) = 0
> <0.010171>
> 5900 11:08:11.299052 futex(0x7ff10be24340, FUTEX_WAIT_PRIVATE, 2, NULL
> <unfinished ...>
> 5899 11:08:20.138492 rt_sigaction(SIGBUS, {0x7ff10b5b1040, [],
> SA_RESTORER|SA_SIGINFO, 0x7ff1085fd100}, NULL, 8) = 0 <0.000012>
> 5899 11:08:20.138598 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO],
> NULL, 8) = 0 <0.000008>
> 5899 11:08:20.138646 mmap(NULL, 266240, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b380000 <0.000013>
> 5899 11:08:20.138793 mmap(NULL, 266240, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b33f000 <0.000009>
I don't know who call `futex`, and why waste such long time. And for `mmap`
system call, I don't found this system call waste too much time.
WHY....
2017-06-16 16:59 GMT+08:00 Bruce Richardson <bruce.richardson@intel.com>:
> On Fri, Jun 16, 2017 at 04:26:40PM +0800, Sam wrote:
> > BTW, while running ovs-dpdk, this log is also take long time, does that
> > mean dpdk request large memory take long time?
> >
> > EAL: Setting up physically contiguous memory...
> >
>
> When running with 1G pages, I found that the mmap system call takes a
> considerable amount of time to execute. I think this is due to the
> kernel zero-ing out the 1G pages. IIRC on one system I measured it as
> taking
> about 0.4 seconds per 1G page.
>
> /Bruce
>
> >
> > 2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>:
> >
> > > Hi all,
> > >
> > > I'm running `QEMU_CMD ...` to create a vm under dpdk huge page
> envriment
> > > (which set huge page 1G). And I enable all events in qemu.
> > >
> > > For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail
> log
> > > is:
> > >
> > > > 30012@1497443246.678304:object_dynamic_cast_assert
> > > qemu:memory-region->qemu:memory-region (/home/hu
> > > > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn)
> > > > 30012@1497443256.274866:object_dynamic_cast_assert
> > > qio-channel-socket->qio-channel-socket (io/chann
> > > > el-socket.c:389:qio_channel_socket_init)
> > >
> > >
> > > I don't know why qemu doing 'memory_region_initfn' function in this 10
> > > second, does anyone know this?
> > >
> > > static void memory_region_initfn(Object *obj)
> > >> {
> > >> MemoryRegion *mr = MEMORY_REGION(obj);
> > >> ObjectProperty *op;
> > >> mr->ops = &unassigned_mem_ops;
> > >> mr->enabled = true;
> > >> mr->romd_mode = true;
> > >> mr->global_locking = true;
> > >> mr->destructor = memory_region_destructor_none;
> > >> QTAILQ_INIT(&mr->subregions);
> > >> QTAILQ_INIT(&mr->coalesced);
> > >> op = object_property_add(OBJECT(mr), "container",
> > >> "link<" TYPE_MEMORY_REGION ">",
> > >> memory_region_get_container,
> > >> NULL, /* memory_region_set_container */
> > >> NULL, NULL, &error_abort);
> > >> op->resolve = memory_region_resolve_container;
> > >> object_property_add(OBJECT(mr), "addr", "uint64",
> > >> memory_region_get_addr,
> > >> NULL, /* memory_region_set_addr */
> > >> NULL, NULL, &error_abort);
> > >> object_property_add(OBJECT(mr), "priority", "uint32",
> > >> memory_region_get_priority,
> > >> NULL, /* memory_region_set_priority */
> > >> NULL, NULL, &error_abort);
> > >> object_property_add_bool(OBJECT(mr), "may-overlap",
> > >> memory_region_get_may_overlap,
> > >> NULL, /* memory_region_set_may_overlap */
> > >> &error_abort);
> > >> object_property_add(OBJECT(mr), "size", "uint64",
> > >> memory_region_get_size,
> > >> NULL, /* memory_region_set_size, */
> > >> NULL, NULL, &error_abort);
> > >> }
> > >
> > >
> > > Thank you~
> > >
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment?
2017-06-19 3:12 ` Sam
@ 2017-06-19 3:13 ` Sam
0 siblings, 0 replies; 5+ messages in thread
From: Sam @ 2017-06-19 3:13 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, users
Additional, this is DPDK-QEMU enviroment.
2017-06-19 11:12 GMT+08:00 Sam <batmanustc@gmail.com>:
> Hi,
>
> I print all system call by `strace -f -T -tt -e trace=all -o output.txt
> $QEMU_CMD`, and I found this:
>
> 5900 11:08:11.288701 nanosleep({0, 10000000}, 0x7ff103c13a80) = 0
>> <0.010171>
>> 5900 11:08:11.299052 futex(0x7ff10be24340, FUTEX_WAIT_PRIVATE, 2, NULL
>> <unfinished ...>
>> 5899 11:08:20.138492 rt_sigaction(SIGBUS, {0x7ff10b5b1040, [],
>> SA_RESTORER|SA_SIGINFO, 0x7ff1085fd100}, NULL, 8) = 0 <0.000012>
>> 5899 11:08:20.138598 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO],
>> NULL, 8) = 0 <0.000008>
>> 5899 11:08:20.138646 mmap(NULL, 266240, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b380000 <0.000013>
>> 5899 11:08:20.138793 mmap(NULL, 266240, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b33f000 <0.000009>
>
>
> I don't know who call `futex`, and why waste such long time. And for
> `mmap` system call, I don't found this system call waste too much time.
>
> WHY....
>
>
> 2017-06-16 16:59 GMT+08:00 Bruce Richardson <bruce.richardson@intel.com>:
>
>> On Fri, Jun 16, 2017 at 04:26:40PM +0800, Sam wrote:
>> > BTW, while running ovs-dpdk, this log is also take long time, does that
>> > mean dpdk request large memory take long time?
>> >
>> > EAL: Setting up physically contiguous memory...
>> >
>>
>> When running with 1G pages, I found that the mmap system call takes a
>> considerable amount of time to execute. I think this is due to the
>> kernel zero-ing out the 1G pages. IIRC on one system I measured it as
>> taking
>> about 0.4 seconds per 1G page.
>>
>> /Bruce
>>
>> >
>> > 2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>:
>> >
>> > > Hi all,
>> > >
>> > > I'm running `QEMU_CMD ...` to create a vm under dpdk huge page
>> envriment
>> > > (which set huge page 1G). And I enable all events in qemu.
>> > >
>> > > For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail
>> log
>> > > is:
>> > >
>> > > > 30012@1497443246.678304:object_dynamic_cast_assert
>> > > qemu:memory-region->qemu:memory-region (/home/hu
>> > > > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region
>> _initfn)
>> > > > 30012@1497443256.274866:object_dynamic_cast_assert
>> > > qio-channel-socket->qio-channel-socket (io/chann
>> > > > el-socket.c:389:qio_channel_socket_init)
>> > >
>> > >
>> > > I don't know why qemu doing 'memory_region_initfn' function in this 10
>> > > second, does anyone know this?
>> > >
>> > > static void memory_region_initfn(Object *obj)
>> > >> {
>> > >> MemoryRegion *mr = MEMORY_REGION(obj);
>> > >> ObjectProperty *op;
>> > >> mr->ops = &unassigned_mem_ops;
>> > >> mr->enabled = true;
>> > >> mr->romd_mode = true;
>> > >> mr->global_locking = true;
>> > >> mr->destructor = memory_region_destructor_none;
>> > >> QTAILQ_INIT(&mr->subregions);
>> > >> QTAILQ_INIT(&mr->coalesced);
>> > >> op = object_property_add(OBJECT(mr), "container",
>> > >> "link<" TYPE_MEMORY_REGION ">",
>> > >> memory_region_get_container,
>> > >> NULL, /* memory_region_set_container */
>> > >> NULL, NULL, &error_abort);
>> > >> op->resolve = memory_region_resolve_container;
>> > >> object_property_add(OBJECT(mr), "addr", "uint64",
>> > >> memory_region_get_addr,
>> > >> NULL, /* memory_region_set_addr */
>> > >> NULL, NULL, &error_abort);
>> > >> object_property_add(OBJECT(mr), "priority", "uint32",
>> > >> memory_region_get_priority,
>> > >> NULL, /* memory_region_set_priority */
>> > >> NULL, NULL, &error_abort);
>> > >> object_property_add_bool(OBJECT(mr), "may-overlap",
>> > >> memory_region_get_may_overlap,
>> > >> NULL, /* memory_region_set_may_overlap
>> */
>> > >> &error_abort);
>> > >> object_property_add(OBJECT(mr), "size", "uint64",
>> > >> memory_region_get_size,
>> > >> NULL, /* memory_region_set_size, */
>> > >> NULL, NULL, &error_abort);
>> > >> }
>> > >
>> > >
>> > > Thank you~
>> > >
>>
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-06-19 3:13 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-16 8:11 [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? Sam
2017-06-16 8:26 ` Sam
2017-06-16 8:59 ` Bruce Richardson
2017-06-19 3:12 ` Sam
2017-06-19 3:13 ` Sam
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).