* [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? @ 2017-06-16 8:11 Sam 2017-06-16 8:26 ` Sam 0 siblings, 1 reply; 5+ messages in thread From: Sam @ 2017-06-16 8:11 UTC (permalink / raw) To: dev, users Hi all, I'm running `QEMU_CMD ...` to create a vm under dpdk huge page envriment (which set huge page 1G). And I enable all events in qemu. For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail log is: > 30012@1497443246.678304:object_dynamic_cast_assert qemu:memory-region->qemu:memory-region (/home/hu > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn) > 30012@1497443256.274866:object_dynamic_cast_assert qio-channel-socket->qio-channel-socket (io/chann > el-socket.c:389:qio_channel_socket_init) I don't know why qemu doing 'memory_region_initfn' function in this 10 second, does anyone know this? static void memory_region_initfn(Object *obj) > { > MemoryRegion *mr = MEMORY_REGION(obj); > ObjectProperty *op; > mr->ops = &unassigned_mem_ops; > mr->enabled = true; > mr->romd_mode = true; > mr->global_locking = true; > mr->destructor = memory_region_destructor_none; > QTAILQ_INIT(&mr->subregions); > QTAILQ_INIT(&mr->coalesced); > op = object_property_add(OBJECT(mr), "container", > "link<" TYPE_MEMORY_REGION ">", > memory_region_get_container, > NULL, /* memory_region_set_container */ > NULL, NULL, &error_abort); > op->resolve = memory_region_resolve_container; > object_property_add(OBJECT(mr), "addr", "uint64", > memory_region_get_addr, > NULL, /* memory_region_set_addr */ > NULL, NULL, &error_abort); > object_property_add(OBJECT(mr), "priority", "uint32", > memory_region_get_priority, > NULL, /* memory_region_set_priority */ > NULL, NULL, &error_abort); > object_property_add_bool(OBJECT(mr), "may-overlap", > memory_region_get_may_overlap, > NULL, /* memory_region_set_may_overlap */ > &error_abort); > object_property_add(OBJECT(mr), "size", "uint64", > memory_region_get_size, > NULL, /* memory_region_set_size, */ > NULL, NULL, &error_abort); > } Thank you~ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? 2017-06-16 8:11 [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? Sam @ 2017-06-16 8:26 ` Sam 2017-06-16 8:59 ` Bruce Richardson 0 siblings, 1 reply; 5+ messages in thread From: Sam @ 2017-06-16 8:26 UTC (permalink / raw) To: dev, users BTW, while running ovs-dpdk, this log is also take long time, does that mean dpdk request large memory take long time? EAL: Setting up physically contiguous memory... 2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>: > Hi all, > > I'm running `QEMU_CMD ...` to create a vm under dpdk huge page envriment > (which set huge page 1G). And I enable all events in qemu. > > For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail log > is: > > > 30012@1497443246.678304:object_dynamic_cast_assert > qemu:memory-region->qemu:memory-region (/home/hu > > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn) > > 30012@1497443256.274866:object_dynamic_cast_assert > qio-channel-socket->qio-channel-socket (io/chann > > el-socket.c:389:qio_channel_socket_init) > > > I don't know why qemu doing 'memory_region_initfn' function in this 10 > second, does anyone know this? > > static void memory_region_initfn(Object *obj) >> { >> MemoryRegion *mr = MEMORY_REGION(obj); >> ObjectProperty *op; >> mr->ops = &unassigned_mem_ops; >> mr->enabled = true; >> mr->romd_mode = true; >> mr->global_locking = true; >> mr->destructor = memory_region_destructor_none; >> QTAILQ_INIT(&mr->subregions); >> QTAILQ_INIT(&mr->coalesced); >> op = object_property_add(OBJECT(mr), "container", >> "link<" TYPE_MEMORY_REGION ">", >> memory_region_get_container, >> NULL, /* memory_region_set_container */ >> NULL, NULL, &error_abort); >> op->resolve = memory_region_resolve_container; >> object_property_add(OBJECT(mr), "addr", "uint64", >> memory_region_get_addr, >> NULL, /* memory_region_set_addr */ >> NULL, NULL, &error_abort); >> object_property_add(OBJECT(mr), "priority", "uint32", >> memory_region_get_priority, >> NULL, /* memory_region_set_priority */ >> NULL, NULL, &error_abort); >> object_property_add_bool(OBJECT(mr), "may-overlap", >> memory_region_get_may_overlap, >> NULL, /* memory_region_set_may_overlap */ >> &error_abort); >> object_property_add(OBJECT(mr), "size", "uint64", >> memory_region_get_size, >> NULL, /* memory_region_set_size, */ >> NULL, NULL, &error_abort); >> } > > > Thank you~ > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? 2017-06-16 8:26 ` Sam @ 2017-06-16 8:59 ` Bruce Richardson 2017-06-19 3:12 ` Sam 0 siblings, 1 reply; 5+ messages in thread From: Bruce Richardson @ 2017-06-16 8:59 UTC (permalink / raw) To: Sam; +Cc: dev, users On Fri, Jun 16, 2017 at 04:26:40PM +0800, Sam wrote: > BTW, while running ovs-dpdk, this log is also take long time, does that > mean dpdk request large memory take long time? > > EAL: Setting up physically contiguous memory... > When running with 1G pages, I found that the mmap system call takes a considerable amount of time to execute. I think this is due to the kernel zero-ing out the 1G pages. IIRC on one system I measured it as taking about 0.4 seconds per 1G page. /Bruce > > 2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>: > > > Hi all, > > > > I'm running `QEMU_CMD ...` to create a vm under dpdk huge page envriment > > (which set huge page 1G). And I enable all events in qemu. > > > > For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail log > > is: > > > > > 30012@1497443246.678304:object_dynamic_cast_assert > > qemu:memory-region->qemu:memory-region (/home/hu > > > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn) > > > 30012@1497443256.274866:object_dynamic_cast_assert > > qio-channel-socket->qio-channel-socket (io/chann > > > el-socket.c:389:qio_channel_socket_init) > > > > > > I don't know why qemu doing 'memory_region_initfn' function in this 10 > > second, does anyone know this? > > > > static void memory_region_initfn(Object *obj) > >> { > >> MemoryRegion *mr = MEMORY_REGION(obj); > >> ObjectProperty *op; > >> mr->ops = &unassigned_mem_ops; > >> mr->enabled = true; > >> mr->romd_mode = true; > >> mr->global_locking = true; > >> mr->destructor = memory_region_destructor_none; > >> QTAILQ_INIT(&mr->subregions); > >> QTAILQ_INIT(&mr->coalesced); > >> op = object_property_add(OBJECT(mr), "container", > >> "link<" TYPE_MEMORY_REGION ">", > >> memory_region_get_container, > >> NULL, /* memory_region_set_container */ > >> NULL, NULL, &error_abort); > >> op->resolve = memory_region_resolve_container; > >> object_property_add(OBJECT(mr), "addr", "uint64", > >> memory_region_get_addr, > >> NULL, /* memory_region_set_addr */ > >> NULL, NULL, &error_abort); > >> object_property_add(OBJECT(mr), "priority", "uint32", > >> memory_region_get_priority, > >> NULL, /* memory_region_set_priority */ > >> NULL, NULL, &error_abort); > >> object_property_add_bool(OBJECT(mr), "may-overlap", > >> memory_region_get_may_overlap, > >> NULL, /* memory_region_set_may_overlap */ > >> &error_abort); > >> object_property_add(OBJECT(mr), "size", "uint64", > >> memory_region_get_size, > >> NULL, /* memory_region_set_size, */ > >> NULL, NULL, &error_abort); > >> } > > > > > > Thank you~ > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? 2017-06-16 8:59 ` Bruce Richardson @ 2017-06-19 3:12 ` Sam 2017-06-19 3:13 ` Sam 0 siblings, 1 reply; 5+ messages in thread From: Sam @ 2017-06-19 3:12 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev, users Hi, I print all system call by `strace -f -T -tt -e trace=all -o output.txt $QEMU_CMD`, and I found this: 5900 11:08:11.288701 nanosleep({0, 10000000}, 0x7ff103c13a80) = 0 > <0.010171> > 5900 11:08:11.299052 futex(0x7ff10be24340, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > 5899 11:08:20.138492 rt_sigaction(SIGBUS, {0x7ff10b5b1040, [], > SA_RESTORER|SA_SIGINFO, 0x7ff1085fd100}, NULL, 8) = 0 <0.000012> > 5899 11:08:20.138598 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], > NULL, 8) = 0 <0.000008> > 5899 11:08:20.138646 mmap(NULL, 266240, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b380000 <0.000013> > 5899 11:08:20.138793 mmap(NULL, 266240, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b33f000 <0.000009> I don't know who call `futex`, and why waste such long time. And for `mmap` system call, I don't found this system call waste too much time. WHY.... 2017-06-16 16:59 GMT+08:00 Bruce Richardson <bruce.richardson@intel.com>: > On Fri, Jun 16, 2017 at 04:26:40PM +0800, Sam wrote: > > BTW, while running ovs-dpdk, this log is also take long time, does that > > mean dpdk request large memory take long time? > > > > EAL: Setting up physically contiguous memory... > > > > When running with 1G pages, I found that the mmap system call takes a > considerable amount of time to execute. I think this is due to the > kernel zero-ing out the 1G pages. IIRC on one system I measured it as > taking > about 0.4 seconds per 1G page. > > /Bruce > > > > > 2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>: > > > > > Hi all, > > > > > > I'm running `QEMU_CMD ...` to create a vm under dpdk huge page > envriment > > > (which set huge page 1G). And I enable all events in qemu. > > > > > > For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail > log > > > is: > > > > > > > 30012@1497443246.678304:object_dynamic_cast_assert > > > qemu:memory-region->qemu:memory-region (/home/hu > > > > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region_initfn) > > > > 30012@1497443256.274866:object_dynamic_cast_assert > > > qio-channel-socket->qio-channel-socket (io/chann > > > > el-socket.c:389:qio_channel_socket_init) > > > > > > > > > I don't know why qemu doing 'memory_region_initfn' function in this 10 > > > second, does anyone know this? > > > > > > static void memory_region_initfn(Object *obj) > > >> { > > >> MemoryRegion *mr = MEMORY_REGION(obj); > > >> ObjectProperty *op; > > >> mr->ops = &unassigned_mem_ops; > > >> mr->enabled = true; > > >> mr->romd_mode = true; > > >> mr->global_locking = true; > > >> mr->destructor = memory_region_destructor_none; > > >> QTAILQ_INIT(&mr->subregions); > > >> QTAILQ_INIT(&mr->coalesced); > > >> op = object_property_add(OBJECT(mr), "container", > > >> "link<" TYPE_MEMORY_REGION ">", > > >> memory_region_get_container, > > >> NULL, /* memory_region_set_container */ > > >> NULL, NULL, &error_abort); > > >> op->resolve = memory_region_resolve_container; > > >> object_property_add(OBJECT(mr), "addr", "uint64", > > >> memory_region_get_addr, > > >> NULL, /* memory_region_set_addr */ > > >> NULL, NULL, &error_abort); > > >> object_property_add(OBJECT(mr), "priority", "uint32", > > >> memory_region_get_priority, > > >> NULL, /* memory_region_set_priority */ > > >> NULL, NULL, &error_abort); > > >> object_property_add_bool(OBJECT(mr), "may-overlap", > > >> memory_region_get_may_overlap, > > >> NULL, /* memory_region_set_may_overlap */ > > >> &error_abort); > > >> object_property_add(OBJECT(mr), "size", "uint64", > > >> memory_region_get_size, > > >> NULL, /* memory_region_set_size, */ > > >> NULL, NULL, &error_abort); > > >> } > > > > > > > > > Thank you~ > > > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? 2017-06-19 3:12 ` Sam @ 2017-06-19 3:13 ` Sam 0 siblings, 0 replies; 5+ messages in thread From: Sam @ 2017-06-19 3:13 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev, users Additional, this is DPDK-QEMU enviroment. 2017-06-19 11:12 GMT+08:00 Sam <batmanustc@gmail.com>: > Hi, > > I print all system call by `strace -f -T -tt -e trace=all -o output.txt > $QEMU_CMD`, and I found this: > > 5900 11:08:11.288701 nanosleep({0, 10000000}, 0x7ff103c13a80) = 0 >> <0.010171> >> 5900 11:08:11.299052 futex(0x7ff10be24340, FUTEX_WAIT_PRIVATE, 2, NULL >> <unfinished ...> >> 5899 11:08:20.138492 rt_sigaction(SIGBUS, {0x7ff10b5b1040, [], >> SA_RESTORER|SA_SIGINFO, 0x7ff1085fd100}, NULL, 8) = 0 <0.000012> >> 5899 11:08:20.138598 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], >> NULL, 8) = 0 <0.000008> >> 5899 11:08:20.138646 mmap(NULL, 266240, PROT_READ|PROT_WRITE, >> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b380000 <0.000013> >> 5899 11:08:20.138793 mmap(NULL, 266240, PROT_READ|PROT_WRITE, >> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff10b33f000 <0.000009> > > > I don't know who call `futex`, and why waste such long time. And for > `mmap` system call, I don't found this system call waste too much time. > > WHY.... > > > 2017-06-16 16:59 GMT+08:00 Bruce Richardson <bruce.richardson@intel.com>: > >> On Fri, Jun 16, 2017 at 04:26:40PM +0800, Sam wrote: >> > BTW, while running ovs-dpdk, this log is also take long time, does that >> > mean dpdk request large memory take long time? >> > >> > EAL: Setting up physically contiguous memory... >> > >> >> When running with 1G pages, I found that the mmap system call takes a >> considerable amount of time to execute. I think this is due to the >> kernel zero-ing out the 1G pages. IIRC on one system I measured it as >> taking >> about 0.4 seconds per 1G page. >> >> /Bruce >> >> > >> > 2017-06-16 16:11 GMT+08:00 Sam <batmanustc@gmail.com>: >> > >> > > Hi all, >> > > >> > > I'm running `QEMU_CMD ...` to create a vm under dpdk huge page >> envriment >> > > (which set huge page 1G). And I enable all events in qemu. >> > > >> > > For qemu and ovs-dpdk(ovs-2.4.9 with dpdk-2.2.0) environment, detail >> log >> > > is: >> > > >> > > > 30012@1497443246.678304:object_dynamic_cast_assert >> > > qemu:memory-region->qemu:memory-region (/home/hu >> > > > anghuai/cloud/contrib/qemu-2.6.0/memory.c:1076:memory_region >> _initfn) >> > > > 30012@1497443256.274866:object_dynamic_cast_assert >> > > qio-channel-socket->qio-channel-socket (io/chann >> > > > el-socket.c:389:qio_channel_socket_init) >> > > >> > > >> > > I don't know why qemu doing 'memory_region_initfn' function in this 10 >> > > second, does anyone know this? >> > > >> > > static void memory_region_initfn(Object *obj) >> > >> { >> > >> MemoryRegion *mr = MEMORY_REGION(obj); >> > >> ObjectProperty *op; >> > >> mr->ops = &unassigned_mem_ops; >> > >> mr->enabled = true; >> > >> mr->romd_mode = true; >> > >> mr->global_locking = true; >> > >> mr->destructor = memory_region_destructor_none; >> > >> QTAILQ_INIT(&mr->subregions); >> > >> QTAILQ_INIT(&mr->coalesced); >> > >> op = object_property_add(OBJECT(mr), "container", >> > >> "link<" TYPE_MEMORY_REGION ">", >> > >> memory_region_get_container, >> > >> NULL, /* memory_region_set_container */ >> > >> NULL, NULL, &error_abort); >> > >> op->resolve = memory_region_resolve_container; >> > >> object_property_add(OBJECT(mr), "addr", "uint64", >> > >> memory_region_get_addr, >> > >> NULL, /* memory_region_set_addr */ >> > >> NULL, NULL, &error_abort); >> > >> object_property_add(OBJECT(mr), "priority", "uint32", >> > >> memory_region_get_priority, >> > >> NULL, /* memory_region_set_priority */ >> > >> NULL, NULL, &error_abort); >> > >> object_property_add_bool(OBJECT(mr), "may-overlap", >> > >> memory_region_get_may_overlap, >> > >> NULL, /* memory_region_set_may_overlap >> */ >> > >> &error_abort); >> > >> object_property_add(OBJECT(mr), "size", "uint64", >> > >> memory_region_get_size, >> > >> NULL, /* memory_region_set_size, */ >> > >> NULL, NULL, &error_abort); >> > >> } >> > > >> > > >> > > Thank you~ >> > > >> > > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-06-19 3:13 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-06-16 8:11 [dpdk-dev] [DPDK-memory] how qemu waste such long time under dpdk huge page envriment? Sam 2017-06-16 8:26 ` Sam 2017-06-16 8:59 ` Bruce Richardson 2017-06-19 3:12 ` Sam 2017-06-19 3:13 ` Sam
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).