Re: [dpdk-dev] dpdk/vpp and cross-version migration for vhost

[Yuanhan Liu <yuanhan.liu@linux.intel.com>]

On Thu, Nov 17, 2016 at 09:47:09AM +0100, Maxime Coquelin wrote:
> 
> 
> On 11/17/2016 09:29 AM, Yuanhan Liu wrote:
> >As usaual, sorry for late response :/
> >
> >On Thu, Oct 13, 2016 at 08:50:52PM +0300, Michael S. Tsirkin wrote:
> >>Hi!
> >>So it looks like we face a problem with cross-version
> >>migration when using vhost. It's not new but became more
> >>acute with the advent of vhost user.
> >>
> >>For users to be able to migrate between different versions
> >>of the hypervisor the interface exposed to guests
> >>by hypervisor must stay unchanged.
> >>
> >>The problem is that a qemu device is connected
> >>to a backend in another process, so the interface
> >>exposed to guests depends on the capabilities of that
> >>process.
> >>
> >>Specifically, for vhost user interface based on virtio, this includes
> >>the "host features" bitmap that defines the interface, as well as more
> >>host values such as the max ring size.  Adding new features/changing
> >>values to this interface is required to make progress, but on the other
> >>hand we need ability to get the old host features to be compatible.
> >
> >It looks like to the same issue of vhost-user reconnect to me. For example,
> >
> >- start dpdk 16.07 & qemu 2.5
> >- kill dpdk
> >- start dpdk 16.11
> >
> >Though DPDK 16.11 has more features comparing to dpdk 16.07 (say, indirect),
> >above should work. Because qemu saves the negotiated features before the
> >disconnect and stores it back after the reconnection.
> >
> >    commit a463215b087c41d7ca94e51aa347cde523831873
> >    Author: Marc-André Lureau <marcandre.lureau@redhat.com>
> >    Date:   Mon Jun 6 18:45:05 2016 +0200
> >
> >        vhost-net: save & restore vhost-user acked features
> >
> >        The initial vhost-user connection sets the features to be negotiated
> >        with the driver. Renegotiation isn't possible without device reset.
> >
> >        To handle reconnection of vhost-user backend, ensure the same set of
> >        features are provided, and reuse already acked features.
> >
> >        Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> >
> >So we could do similar to vhost-user? I mean, save the acked features
> >before migration and store it back after it. This should be able to
> >keep the compatibility. If user downgrades DPDK version, it also could
> >be easily detected, and then exit with an error to user: migration
> >failed due to un-compatible vhost features.
> >
> >Just some rough thoughts. Makes tiny sense?
> 
> My understanding is that the management tool has to know whether
> versions are compatible before initiating the migration:

Makes sense. How about getting and restoring the acked features through
qemu command lines then, say, through the monitor interface?

With that, it would be something like:

- start vhost-user backend (DPDK, VPP, or whatever) & qemu in the src host

- read the acked features (through monitor interface)

- start vhost-user backend in the dst host

- start qemu in the dst host with the just queried acked features

  QEMU then is expected to use this feature set for the later vhost-user
  feature negotitation. Exit if features compatibility is broken.

Thoughts?

	--yliu

>  1. The downtime could be unpredictable if a VM has to move from hosts
>     to hosts multiple times, which is problematic, especially for NFV.
>  2. If migration is not possible, maybe the management tool would
>     prefer not to interrupt the VM on current host.
> 
> I have little experience with migration though, so I could be mistaken.
> 
> Thanks,
> Maxime
> 
> >
> >	--yliu
> >>
> >>To solve this problem within qemu, qemu has a versioning system based on
> >>a machine type concept which fundamentally is a version string, by
> >>specifying that string one can get hardware compatible with a previous
> >>qemu version. QEMU also reports the latest version and list of versions
> >>supported so libvirt records the version at VM creation and then is
> >>careful to use this machine version whenever it migrates a VM.
> >>
> >>One might wonder how is this solved with a kernel vhost backend. The
> >>answer is that it mostly isn't - instead an assumption is made, that
> >>qemu versions are deployed together with the kernel - this is generally
> >>true for downstreams.  Thus whenever qemu gains a new feature, it is
> >>already supported by the kernel as well.  However, if one attempts
> >>migration with a new qemu from a system with a new to old kernel, one
> >>would get a failure.
> >>
> >>In the world where we have multiple userspace backends, with some of
> >>these supplied by ISVs, this seems non-realistic.
> >>
> >>IMO we need to support vhost backend versioning, ideally
> >>in a way that will also work for vhost kernel backends.
> >>
> >>So I'd like to get some input from both backend and management
> >>developers on what a good solution would look like.
> >>
> >>If we want to emulate the qemu solution, this involves adding the
> >>concept of interface versions to dpdk.  For example, dpdk could supply a
> >>file (or utility printing?) with list of versions: latest and versions
> >>supported. libvirt could read that and
> >>- store latest version at vm creation
> >>- pass it around with the vm
> >>- pass it to qemu
> >>
> >>>From here, qemu could pass this over the vhost-user channel,
> >>thus making sure it's initialized with the correct
> >>compatible interface.
> >>
> >>As version here is an opaque string for libvirt and qemu,
> >>anything can be used - but I suggest either a list
> >>of values defining the interface, e.g.
> >>any_layout=on,max_ring=256
> >>or a version including the name and vendor of the backend,
> >>e.g. "org.dpdk.v4.5.6".
> >>
> >>Note that typically the list of supported versions can only be
> >>extended, not shrunk. Also, if the host/guest interface
> >>does not change, don't change the current version as
> >>this just creates work for everyone.
> >>
> >>Thoughts? Would this work well for management? dpdk? vpp?
> >>
> >>Thanks!
> >>
> >>--
> >>MST