[dpdk-dev] dpdk/vhost-user and VM migration

[dpdk-dev] dpdk/vhost-user and VM migration

[Michael S. Tsirkin]

Hello!
I am currently looking at how using dpdk on host, accessing VM memory
using the vhost-user interface, interacts with VM migration.

The issue is that any changes made to VM memory need to be tracked so
that updates can be sent from migration source to destination.

At the moment, there's a proposal of an interface extension to
vhost-user which adds ability to do this tracking through shared memory.
dpdk would then be responsible for tracking these updates using atomic
operations to set bits (per page written) in a memory bitmap.

This only needs to happen during migration, at other times there could
be a jump to skip this logging.

Is this a reasonable approach? Would performance degradation during
migration associated with atomics affect the performance to a level
where it's no longer useful?  Pls note these logs aren't latency
sensitive, so can be done on a separate core, and can be batched.


One alternative I'm considering is extending linux kernel so it can do
this tracking automatically, by marking pages read-only, detecting a
pagefault and logging the write, then making the pages writeable.  This
would mean higher worst-case overhead (pagefaults are expensive) but
lower average one (not extra code after the first fault).  Not sure how
feasible this is yet, this would be harder to implement and it will only
be apply to newer host kernels.

Any feedback would be appreciated.

-- 
MST

Re: [dpdk-dev] dpdk/vhost-user and VM migration

[Yuanhan Liu]

On Wed, Oct 14, 2015 at 12:16:29AM +0300, Michael S. Tsirkin wrote:
> Hello!
> I am currently looking at how using dpdk on host, accessing VM memory
> using the vhost-user interface, interacts with VM migration.
> 
> The issue is that any changes made to VM memory need to be tracked so
> that updates can be sent from migration source to destination.
> 
> At the moment, there's a proposal of an interface extension to
> vhost-user which adds ability to do this tracking through shared memory.
> dpdk would then be responsible for tracking these updates using atomic
> operations to set bits (per page written) in a memory bitmap.
> 
> This only needs to happen during migration, at other times there could
> be a jump to skip this logging.
> 
> Is this a reasonable approach?

Hi Michael,

As I stated in another email, adding dpdk/vhost-user vm migration
support is my second TODO. However, I barely know anything about
vm migration so far, that I can't tell now.

I will re-visit this question when I finished my first item and
after some more investigation.

	--yliu

> Would performance degradation during
> migration associated with atomics affect the performance to a level
> where it's no longer useful?  Pls note these logs aren't latency
> sensitive, so can be done on a separate core, and can be batched.
> 
> 
> One alternative I'm considering is extending linux kernel so it can do
> this tracking automatically, by marking pages read-only, detecting a
> pagefault and logging the write, then making the pages writeable.  This
> would mean higher worst-case overhead (pagefaults are expensive) but
> lower average one (not extra code after the first fault).  Not sure how
> feasible this is yet, this would be harder to implement and it will only
> be apply to newer host kernels.
> 
> Any feedback would be appreciated.
> 
> -- 
> MST

Re: [dpdk-dev] dpdk/vhost-user and VM migration

[Amnon Ilan]

----- Original Message -----
> From: "Yuanhan Liu" <yuanhan.liu@linux.intel.com>
> To: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: dev@dpdk.org
> Sent: Friday, October 16, 2015 10:37:38 AM
> Subject: Re: [dpdk-dev] dpdk/vhost-user and VM migration
> 
> On Wed, Oct 14, 2015 at 12:16:29AM +0300, Michael S. Tsirkin wrote:
> > Hello!
> > I am currently looking at how using dpdk on host, accessing VM memory
> > using the vhost-user interface, interacts with VM migration.
> > 
> > The issue is that any changes made to VM memory need to be tracked so
> > that updates can be sent from migration source to destination.
> > 
> > At the moment, there's a proposal of an interface extension to
> > vhost-user which adds ability to do this tracking through shared memory.
> > dpdk would then be responsible for tracking these updates using atomic
> > operations to set bits (per page written) in a memory bitmap.
> > 
> > This only needs to happen during migration, at other times there could
> > be a jump to skip this logging.
> > 
> > Is this a reasonable approach?
> 
> Hi Michael,
> 
> As I stated in another email, adding dpdk/vhost-user vm migration
> support is my second TODO. However, I barely know anything about
> vm migration so far, that I can't tell now.
> 
> I will re-visit this question when I finished my first item and
> after some more investigation.

Yuanhan, 

Live-migration for vhost-user is now available upstream.
Do you need some guidance on how to implement it in DPDK?

Amnon

> 
> 	--yliu
> 
> > Would performance degradation during
> > migration associated with atomics affect the performance to a level
> > where it's no longer useful?  Pls note these logs aren't latency
> > sensitive, so can be done on a separate core, and can be batched.
> > 
> > 
> > One alternative I'm considering is extending linux kernel so it can do
> > this tracking automatically, by marking pages read-only, detecting a
> > pagefault and logging the write, then making the pages writeable.  This
> > would mean higher worst-case overhead (pagefaults are expensive) but
> > lower average one (not extra code after the first fault).  Not sure how
> > feasible this is yet, this would be harder to implement and it will only
> > be apply to newer host kernels.
> > 
> > Any feedback would be appreciated.
> > 
> > --
> > MST
> 

Re: [dpdk-dev] dpdk/vhost-user and VM migration

[Yuanhan Liu]

On Wed, Oct 28, 2015 at 05:52:44AM -0400, Amnon Ilan wrote:
> 
> 
> ----- Original Message -----
> > From: "Yuanhan Liu" <yuanhan.liu@linux.intel.com>
> > To: "Michael S. Tsirkin" <mst@redhat.com>
> > Cc: dev@dpdk.org
> > Sent: Friday, October 16, 2015 10:37:38 AM
> > Subject: Re: [dpdk-dev] dpdk/vhost-user and VM migration
> > 
> > On Wed, Oct 14, 2015 at 12:16:29AM +0300, Michael S. Tsirkin wrote:
> > > Hello!
> > > I am currently looking at how using dpdk on host, accessing VM memory
> > > using the vhost-user interface, interacts with VM migration.
> > > 
> > > The issue is that any changes made to VM memory need to be tracked so
> > > that updates can be sent from migration source to destination.
> > > 
> > > At the moment, there's a proposal of an interface extension to
> > > vhost-user which adds ability to do this tracking through shared memory.
> > > dpdk would then be responsible for tracking these updates using atomic
> > > operations to set bits (per page written) in a memory bitmap.
> > > 
> > > This only needs to happen during migration, at other times there could
> > > be a jump to skip this logging.
> > > 
> > > Is this a reasonable approach?
> > 
> > Hi Michael,
> > 
> > As I stated in another email, adding dpdk/vhost-user vm migration
> > support is my second TODO. However, I barely know anything about
> > vm migration so far, that I can't tell now.
> > 
> > I will re-visit this question when I finished my first item and
> > after some more investigation.
> 
> Yuanhan, 
> 
> Live-migration for vhost-user is now available upstream.

Hi Ammon,

Yes, I'm aware of that.

> Do you need some guidance on how to implement it in DPDK?

Appreciate a lot for offering the help!  However, I haven't started
it yet, and I should be able to start it in two or three weeks if
everything goes well here.  I may ask your help then if I am in
trouble.

	--yliu

> > > Would performance degradation during
> > > migration associated with atomics affect the performance to a level
> > > where it's no longer useful?  Pls note these logs aren't latency
> > > sensitive, so can be done on a separate core, and can be batched.
> > > 
> > > 
> > > One alternative I'm considering is extending linux kernel so it can do
> > > this tracking automatically, by marking pages read-only, detecting a
> > > pagefault and logging the write, then making the pages writeable.  This
> > > would mean higher worst-case overhead (pagefaults are expensive) but
> > > lower average one (not extra code after the first fault).  Not sure how
> > > feasible this is yet, this would be harder to implement and it will only
> > > be apply to newer host kernels.
> > > 
> > > Any feedback would be appreciated.
> > > 
> > > --
> > > MST
> >