Re: [dpdk-dev] [vpp-dev] dpdk/vpp and cross-version migration for vhost

[Thomas F Herbert <therbert@redhat.com>]

+Billy McFall


On 11/17/2016 04:49 AM, Yuanhan Liu wrote:
> On Thu, Nov 17, 2016 at 09:47:09AM +0100, Maxime Coquelin wrote:
>>
>> On 11/17/2016 09:29 AM, Yuanhan Liu wrote:
>>> As usaual, sorry for late response :/
>>>
>>> On Thu, Oct 13, 2016 at 08:50:52PM +0300, Michael S. Tsirkin wrote:
>>>> Hi!
>>>> So it looks like we face a problem with cross-version
>>>> migration when using vhost. It's not new but became more
>>>> acute with the advent of vhost user.
>>>>
>>>> For users to be able to migrate between different versions
>>>> of the hypervisor the interface exposed to guests
>>>> by hypervisor must stay unchanged.
>>>>
>>>> The problem is that a qemu device is connected
>>>> to a backend in another process, so the interface
>>>> exposed to guests depends on the capabilities of that
>>>> process.
>>>>
>>>> Specifically, for vhost user interface based on virtio, this includes
>>>> the "host features" bitmap that defines the interface, as well as more
>>>> host values such as the max ring size.  Adding new features/changing
>>>> values to this interface is required to make progress, but on the other
>>>> hand we need ability to get the old host features to be compatible.
>>> It looks like to the same issue of vhost-user reconnect to me. For example,
>>>
>>> - start dpdk 16.07 & qemu 2.5
>>> - kill dpdk
>>> - start dpdk 16.11
>>>
>>> Though DPDK 16.11 has more features comparing to dpdk 16.07 (say, indirect),
>>> above should work. Because qemu saves the negotiated features before the
>>> disconnect and stores it back after the reconnection.
>>>
>>>     commit a463215b087c41d7ca94e51aa347cde523831873
>>>     Author: Marc-André Lureau <marcandre.lureau@redhat.com>
>>>     Date:   Mon Jun 6 18:45:05 2016 +0200
>>>
>>>         vhost-net: save & restore vhost-user acked features
>>>
>>>         The initial vhost-user connection sets the features to be negotiated
>>>         with the driver. Renegotiation isn't possible without device reset.
>>>
>>>         To handle reconnection of vhost-user backend, ensure the same set of
>>>         features are provided, and reuse already acked features.
>>>
>>>         Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>>>
>>>
>>> So we could do similar to vhost-user? I mean, save the acked features
>>> before migration and store it back after it. This should be able to
>>> keep the compatibility. If user downgrades DPDK version, it also could
>>> be easily detected, and then exit with an error to user: migration
>>> failed due to un-compatible vhost features.
>>>
>>> Just some rough thoughts. Makes tiny sense?
>> My understanding is that the management tool has to know whether
>> versions are compatible before initiating the migration:
> Makes sense. How about getting and restoring the acked features through
> qemu command lines then, say, through the monitor interface?
>
> With that, it would be something like:
>
> - start vhost-user backend (DPDK, VPP, or whatever) & qemu in the src host
>
> - read the acked features (through monitor interface)
>
> - start vhost-user backend in the dst host
>
> - start qemu in the dst host with the just queried acked features
>
>    QEMU then is expected to use this feature set for the later vhost-user
>    feature negotitation. Exit if features compatibility is broken.
>
> Thoughts?
>
> 	--yliu
>
>>   1. The downtime could be unpredictable if a VM has to move from hosts
>>      to hosts multiple times, which is problematic, especially for NFV.
>>   2. If migration is not possible, maybe the management tool would
>>      prefer not to interrupt the VM on current host.
>>
>> I have little experience with migration though, so I could be mistaken.
>>
>> Thanks,
>> Maxime
>>
>>> 	--yliu
>>>> To solve this problem within qemu, qemu has a versioning system based on
>>>> a machine type concept which fundamentally is a version string, by
>>>> specifying that string one can get hardware compatible with a previous
>>>> qemu version. QEMU also reports the latest version and list of versions
>>>> supported so libvirt records the version at VM creation and then is
>>>> careful to use this machine version whenever it migrates a VM.
>>>>
>>>> One might wonder how is this solved with a kernel vhost backend. The
>>>> answer is that it mostly isn't - instead an assumption is made, that
>>>> qemu versions are deployed together with the kernel - this is generally
>>>> true for downstreams.  Thus whenever qemu gains a new feature, it is
>>>> already supported by the kernel as well.  However, if one attempts
>>>> migration with a new qemu from a system with a new to old kernel, one
>>>> would get a failure.
>>>>
>>>> In the world where we have multiple userspace backends, with some of
>>>> these supplied by ISVs, this seems non-realistic.
>>>>
>>>> IMO we need to support vhost backend versioning, ideally
>>>> in a way that will also work for vhost kernel backends.
>>>>
>>>> So I'd like to get some input from both backend and management
>>>> developers on what a good solution would look like.
>>>>
>>>> If we want to emulate the qemu solution, this involves adding the
>>>> concept of interface versions to dpdk.  For example, dpdk could supply a
>>>> file (or utility printing?) with list of versions: latest and versions
>>>> supported. libvirt could read that and
>>>> - store latest version at vm creation
>>>> - pass it around with the vm
>>>> - pass it to qemu
>>>>
>>>> >From here, qemu could pass this over the vhost-user channel,
>>>> thus making sure it's initialized with the correct
>>>> compatible interface.
>>>>
>>>> As version here is an opaque string for libvirt and qemu,
>>>> anything can be used - but I suggest either a list
>>>> of values defining the interface, e.g.
>>>> any_layout=on,max_ring=256
>>>> or a version including the name and vendor of the backend,
>>>> e.g. "org.dpdk.v4.5.6".
>>>>
>>>> Note that typically the list of supported versions can only be
>>>> extended, not shrunk. Also, if the host/guest interface
>>>> does not change, don't change the current version as
>>>> this just creates work for everyone.
>>>>
>>>> Thoughts? Would this work well for management? dpdk? vpp?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> MST
> _______________________________________________
> vpp-dev mailing list
> vpp-dev@lists.fd.io
> https://lists.fd.io/mailman/listinfo/vpp-dev

-- 
*Thomas F Herbert*
SDN Group
Office of Technology
*Red Hat*