Re: [dpdk-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin <maxime.coquelin@redhat.com>]

On 02/03/2017 11:11 AM, Daniel P. Berrange wrote:
> On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote:
>>
>>
>> On 02/03/2017 10:27 AM, Daniel P. Berrange wrote:
>>> On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
>>>> On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
>>>>> On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
>>>>>> On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
>>>>>>> On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
>>>>>>>> On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
>>>>>>>>> On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
>>>>>>>>>> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
>>>>>>>>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt
>>>>>>>>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
>>>>>>>>>>>>> cli tool. We pass through arguments from the port profile stored in the
>>>>>>>>>>>>> XML config.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   <interface type='bridge'>
>>>>>>>>>>>>>     <source bridge='ovsbr'/>
>>>>>>>>>>>>>     <virtualport type='openvswitch'>
>>>>>>>>>>>>>       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>>>>>>>>>>>>>     </virtualport>
>>>>>>>>>>>>>   </interface>
>>>>>>>>>>>>>
>>>>>>>>>>>>> eg those things in <parameters/> get passed as cli args to the 'add-port'
>>>>>>>>>>>>> command. Soo if add-port needs this new version string, then we'd need
>>>>>>>>>>>>> to add the version to the openvswitch virtualport XML.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If the version is provided to OVS in a different command, then it would
>>>>>>>>>>>>> probably be outside scope of libvirt.
>>>>>>>>>>>>
>>>>>>>>>>>> I think it would make sense to be a parameter of the add-port command.
>>>>>>>>>>>> But it would be for vhost-user related add-port command, I didn't find
>>>>>>>>>>>> where/if this is managed in libvirt XML.
>>>>>>>>>>>
>>>>>>>>>>> For vhost-user, libvirt does not have any interaction with OVS at
>>>>>>>>>>> all. If the thing that's using the vhost-user UNIX socket, in turn
>>>>>>>>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
>>>>>>>>>>> OVS it seems like that job is for Nova / os-vif to solve.
>>>>>>>>>>
>>>>>>>>>> I don't think they currently understand the issues involved in
>>>>>>>>>> cross-version migration though.  This is a complex subject and easy to
>>>>>>>>>> get wrong.  It would be significantly better to figure it out at libvirt
>>>>>>>>>> level since it already deals with cross-version migration issues.
>>>>>>>>>
>>>>>>>>> Libvirt considers vhost-user to be a blackbox though - it just exposes
>>>>>>>>> a UNIX socket, and whatever is on the other end is completely opaque.
>>>>>>>>> The fact that the other end might plumb the data stream into openvswitch
>>>>>>>>> is not something libvirt should know, as we don't want to end up having
>>>>>>>>> to add custom code to libvirt for every different vhost-user server
>>>>>>>>> impl.
>>>>>>>>>
>>>>>>>>> IOW, if the version str can be passed to QEMU, and then onto vhost-user
>>>>>>>>> backend in QEMU, then libvirt can be involved. If the version str has
>>>>>>>>> to be given to openvswitch that's not for libvirt to deall with.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Daniel
>>>>>>>>
>>>>>>>> It's not just about passing it to QEMU. The following is needed:
>>>>>>>> - need to query version when creating VM/device
>>>>>>>> - need to query supported versions when migrating VM
>>>>>>>
>>>>>>> Those are both things that nova can do, since it knows what the vhost-user
>>>>>>> device in question is connected to, and so can query the versions, and check
>>>>>>> versions before triggering migration with libvirt
>>>>>>
>>>>>> It can, but then it will need to query libvirt or source for the version
>>>>>> string since it's in the XML.
>>>>>
>>>>> No, it wouldn't be in the XML at all. Nova on the source queries what
>>>>> vhostuser version it has and what the target host has, and can prevent
>>>>> the migration if they're incompatible.
>>>>
>>>> This is not sufficient. Exactly the same as qemu machine type,
>>>> this must be preserved from time of install and moved
>>>> wherever VM goes.
>>>
>>> Nova has the ability todo that.
>>>
>>>>> I dont think libvirt has to be
>>>>> involved at all for this, as all the info can be obtained by nova/os-vif
>>>>> from the vhostuser impl it has configured.
>>>>
>>>> Given we are already confused at libvirt level, I find it
>>>> highly unlikely upper levels will do the right thing.
>>>
>>> The only confusion is about what must consume the data. I mistakenly
>>> thought it was being proposed that the qemu vhostuser backend was
>>> being given a version string. Now it seems the version info must be
>>> set against the 3rd party component that vhostuser connects to, and
>>> that is outside scope of libvirt. That is entirely managed by Nova
>>> today, so Nova is the right thing to manage this versioning info.
>>
>> This is my understanding, with as example using Nova:
>> 1. Before starting the VM, Nova queries source host's OVS and all
>>    possible destination hosts' OVS to retrieve supported versions
>> 2. Nova chooses the first common version supported by all hosts
>> 3. Nova create the dpdkvhostuser interfaces in OVS with passing the
>>    selected version as parameter
>>
>> That should be enough, right? Or maybe I miss something?
>
> It isn't realistic to check all possible destination hosts when first
> starting QEMU, not least becasue they may not yet exist. New hardware
> may be deployed *after* QEMU is first started.
>
> When starting the VM, it should just default to using whatever the
> latest version is on the host in question.
>
> There can be an optional host configuration parameter that admins can
> set to force use of an older version.
>
> This is how Nova deals with machine type - it defaults to latest, but
> you can set hw_machine_type in nova.conf to force an older version if
> you need migration compatibility between hosts with varying versions.

Ok, I see.
I thought it could be automated somehow if a list of available host
is already present.

> At time of migration, Nova need to request the version of the target
> host. If this version is not compatible with what the VM is currently
> using, then Nova should *not* start the migration, it should raise
> an error such that the scheduler retries and picks a different target
> host

Thanks for the clarification.

Maxime