* [dpdk-dev] dpdk/vhost-user and VM migration @ 2015-10-13 21:16 Michael S. Tsirkin 2015-10-16 7:37 ` Yuanhan Liu 0 siblings, 1 reply; 4+ messages in thread From: Michael S. Tsirkin @ 2015-10-13 21:16 UTC (permalink / raw) To: dev Hello! I am currently looking at how using dpdk on host, accessing VM memory using the vhost-user interface, interacts with VM migration. The issue is that any changes made to VM memory need to be tracked so that updates can be sent from migration source to destination. At the moment, there's a proposal of an interface extension to vhost-user which adds ability to do this tracking through shared memory. dpdk would then be responsible for tracking these updates using atomic operations to set bits (per page written) in a memory bitmap. This only needs to happen during migration, at other times there could be a jump to skip this logging. Is this a reasonable approach? Would performance degradation during migration associated with atomics affect the performance to a level where it's no longer useful? Pls note these logs aren't latency sensitive, so can be done on a separate core, and can be batched. One alternative I'm considering is extending linux kernel so it can do this tracking automatically, by marking pages read-only, detecting a pagefault and logging the write, then making the pages writeable. This would mean higher worst-case overhead (pagefaults are expensive) but lower average one (not extra code after the first fault). Not sure how feasible this is yet, this would be harder to implement and it will only be apply to newer host kernels. Any feedback would be appreciated. -- MST ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] dpdk/vhost-user and VM migration 2015-10-13 21:16 [dpdk-dev] dpdk/vhost-user and VM migration Michael S. Tsirkin @ 2015-10-16 7:37 ` Yuanhan Liu 2015-10-28 9:52 ` Amnon Ilan 0 siblings, 1 reply; 4+ messages in thread From: Yuanhan Liu @ 2015-10-16 7:37 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: dev On Wed, Oct 14, 2015 at 12:16:29AM +0300, Michael S. Tsirkin wrote: > Hello! > I am currently looking at how using dpdk on host, accessing VM memory > using the vhost-user interface, interacts with VM migration. > > The issue is that any changes made to VM memory need to be tracked so > that updates can be sent from migration source to destination. > > At the moment, there's a proposal of an interface extension to > vhost-user which adds ability to do this tracking through shared memory. > dpdk would then be responsible for tracking these updates using atomic > operations to set bits (per page written) in a memory bitmap. > > This only needs to happen during migration, at other times there could > be a jump to skip this logging. > > Is this a reasonable approach? Hi Michael, As I stated in another email, adding dpdk/vhost-user vm migration support is my second TODO. However, I barely know anything about vm migration so far, that I can't tell now. I will re-visit this question when I finished my first item and after some more investigation. --yliu > Would performance degradation during > migration associated with atomics affect the performance to a level > where it's no longer useful? Pls note these logs aren't latency > sensitive, so can be done on a separate core, and can be batched. > > > One alternative I'm considering is extending linux kernel so it can do > this tracking automatically, by marking pages read-only, detecting a > pagefault and logging the write, then making the pages writeable. This > would mean higher worst-case overhead (pagefaults are expensive) but > lower average one (not extra code after the first fault). Not sure how > feasible this is yet, this would be harder to implement and it will only > be apply to newer host kernels. > > Any feedback would be appreciated. > > -- > MST ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] dpdk/vhost-user and VM migration 2015-10-16 7:37 ` Yuanhan Liu @ 2015-10-28 9:52 ` Amnon Ilan 2015-10-28 12:30 ` Yuanhan Liu 0 siblings, 1 reply; 4+ messages in thread From: Amnon Ilan @ 2015-10-28 9:52 UTC (permalink / raw) To: Yuanhan Liu; +Cc: dev, Marc-André Lureau, Michael S. Tsirkin ----- Original Message ----- > From: "Yuanhan Liu" <yuanhan.liu@linux.intel.com> > To: "Michael S. Tsirkin" <mst@redhat.com> > Cc: dev@dpdk.org > Sent: Friday, October 16, 2015 10:37:38 AM > Subject: Re: [dpdk-dev] dpdk/vhost-user and VM migration > > On Wed, Oct 14, 2015 at 12:16:29AM +0300, Michael S. Tsirkin wrote: > > Hello! > > I am currently looking at how using dpdk on host, accessing VM memory > > using the vhost-user interface, interacts with VM migration. > > > > The issue is that any changes made to VM memory need to be tracked so > > that updates can be sent from migration source to destination. > > > > At the moment, there's a proposal of an interface extension to > > vhost-user which adds ability to do this tracking through shared memory. > > dpdk would then be responsible for tracking these updates using atomic > > operations to set bits (per page written) in a memory bitmap. > > > > This only needs to happen during migration, at other times there could > > be a jump to skip this logging. > > > > Is this a reasonable approach? > > Hi Michael, > > As I stated in another email, adding dpdk/vhost-user vm migration > support is my second TODO. However, I barely know anything about > vm migration so far, that I can't tell now. > > I will re-visit this question when I finished my first item and > after some more investigation. Yuanhan, Live-migration for vhost-user is now available upstream. Do you need some guidance on how to implement it in DPDK? Amnon > > --yliu > > > Would performance degradation during > > migration associated with atomics affect the performance to a level > > where it's no longer useful? Pls note these logs aren't latency > > sensitive, so can be done on a separate core, and can be batched. > > > > > > One alternative I'm considering is extending linux kernel so it can do > > this tracking automatically, by marking pages read-only, detecting a > > pagefault and logging the write, then making the pages writeable. This > > would mean higher worst-case overhead (pagefaults are expensive) but > > lower average one (not extra code after the first fault). Not sure how > > feasible this is yet, this would be harder to implement and it will only > > be apply to newer host kernels. > > > > Any feedback would be appreciated. > > > > -- > > MST > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] dpdk/vhost-user and VM migration 2015-10-28 9:52 ` Amnon Ilan @ 2015-10-28 12:30 ` Yuanhan Liu 0 siblings, 0 replies; 4+ messages in thread From: Yuanhan Liu @ 2015-10-28 12:30 UTC (permalink / raw) To: Amnon Ilan; +Cc: dev, Marc-André Lureau, Michael S. Tsirkin On Wed, Oct 28, 2015 at 05:52:44AM -0400, Amnon Ilan wrote: > > > ----- Original Message ----- > > From: "Yuanhan Liu" <yuanhan.liu@linux.intel.com> > > To: "Michael S. Tsirkin" <mst@redhat.com> > > Cc: dev@dpdk.org > > Sent: Friday, October 16, 2015 10:37:38 AM > > Subject: Re: [dpdk-dev] dpdk/vhost-user and VM migration > > > > On Wed, Oct 14, 2015 at 12:16:29AM +0300, Michael S. Tsirkin wrote: > > > Hello! > > > I am currently looking at how using dpdk on host, accessing VM memory > > > using the vhost-user interface, interacts with VM migration. > > > > > > The issue is that any changes made to VM memory need to be tracked so > > > that updates can be sent from migration source to destination. > > > > > > At the moment, there's a proposal of an interface extension to > > > vhost-user which adds ability to do this tracking through shared memory. > > > dpdk would then be responsible for tracking these updates using atomic > > > operations to set bits (per page written) in a memory bitmap. > > > > > > This only needs to happen during migration, at other times there could > > > be a jump to skip this logging. > > > > > > Is this a reasonable approach? > > > > Hi Michael, > > > > As I stated in another email, adding dpdk/vhost-user vm migration > > support is my second TODO. However, I barely know anything about > > vm migration so far, that I can't tell now. > > > > I will re-visit this question when I finished my first item and > > after some more investigation. > > Yuanhan, > > Live-migration for vhost-user is now available upstream. Hi Ammon, Yes, I'm aware of that. > Do you need some guidance on how to implement it in DPDK? Appreciate a lot for offering the help! However, I haven't started it yet, and I should be able to start it in two or three weeks if everything goes well here. I may ask your help then if I am in trouble. --yliu > > > Would performance degradation during > > > migration associated with atomics affect the performance to a level > > > where it's no longer useful? Pls note these logs aren't latency > > > sensitive, so can be done on a separate core, and can be batched. > > > > > > > > > One alternative I'm considering is extending linux kernel so it can do > > > this tracking automatically, by marking pages read-only, detecting a > > > pagefault and logging the write, then making the pages writeable. This > > > would mean higher worst-case overhead (pagefaults are expensive) but > > > lower average one (not extra code after the first fault). Not sure how > > > feasible this is yet, this would be harder to implement and it will only > > > be apply to newer host kernels. > > > > > > Any feedback would be appreciated. > > > > > > -- > > > MST > > ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-10-28 12:29 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-10-13 21:16 [dpdk-dev] dpdk/vhost-user and VM migration Michael S. Tsirkin 2015-10-16 7:37 ` Yuanhan Liu 2015-10-28 9:52 ` Amnon Ilan 2015-10-28 12:30 ` Yuanhan Liu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).