From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id D95F11150 for ; Fri, 3 Feb 2017 12:36:48 +0100 (CET) Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 281E561BB5; Fri, 3 Feb 2017 11:36:49 +0000 (UTC) Received: from [10.36.116.81] (ovpn-116-81.ams2.redhat.com [10.36.116.81]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v13BagXE020175 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 3 Feb 2017 06:36:44 -0500 To: "Daniel P. Berrange" , "dev@openvswitch.org" References: <20170202150621.GQ2915@redhat.com> <20170202181827-mutt-send-email-mst@kernel.org> <20170202171028.GT2915@redhat.com> <20170202191248-mutt-send-email-mst@kernel.org> <20170202172908.GW2915@redhat.com> <20170202193041-mutt-send-email-mst@kernel.org> <20170202182155.GA30916@redhat.com> <20170202202450-mutt-send-email-mst@kernel.org> <20170203092735.GA10350@redhat.com> <34bf53f0-7595-fd90-300d-41db10a43ece@redhat.com> <20170203101126.GE10350@redhat.com> Cc: "Michael S. Tsirkin" , Michal Privoznik , Kevin Traynor , Ciara Loftus , mark.b.kavanagh@intel.com, Flavio Leitner , Yuanhan Liu , Daniele Di Proietto , "dev@dpdk.org" , "libvir-list@redhat.com" From: Maxime Coquelin Message-ID: <41847c4b-e7ec-6e42-8265-2efeb1e2c0b8@redhat.com> Date: Fri, 3 Feb 2017 12:36:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: <20170203101126.GE10350@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Fri, 03 Feb 2017 11:36:49 +0000 (UTC) Subject: Re: [dpdk-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Feb 2017 11:36:49 -0000 On 02/03/2017 11:11 AM, Daniel P. Berrange wrote: > On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote: >> >> >> On 02/03/2017 10:27 AM, Daniel P. Berrange wrote: >>> On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote: >>>> On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote: >>>>> On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote: >>>>>> On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote: >>>>>>> On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: >>>>>>>> On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote: >>>>>>>>> On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: >>>>>>>>>> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote: >>>>>>>>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt >>>>>>>>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs >>>>>>>>>>>>> cli tool. We pass through arguments from the port profile stored in the >>>>>>>>>>>>> XML config. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> eg those things in get passed as cli args to the 'add-port' >>>>>>>>>>>>> command. Soo if add-port needs this new version string, then we'd need >>>>>>>>>>>>> to add the version to the openvswitch virtualport XML. >>>>>>>>>>>>> >>>>>>>>>>>>> If the version is provided to OVS in a different command, then it would >>>>>>>>>>>>> probably be outside scope of libvirt. >>>>>>>>>>>> >>>>>>>>>>>> I think it would make sense to be a parameter of the add-port command. >>>>>>>>>>>> But it would be for vhost-user related add-port command, I didn't find >>>>>>>>>>>> where/if this is managed in libvirt XML. >>>>>>>>>>> >>>>>>>>>>> For vhost-user, libvirt does not have any interaction with OVS at >>>>>>>>>>> all. If the thing that's using the vhost-user UNIX socket, in turn >>>>>>>>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user >>>>>>>>>>> OVS it seems like that job is for Nova / os-vif to solve. >>>>>>>>>> >>>>>>>>>> I don't think they currently understand the issues involved in >>>>>>>>>> cross-version migration though. This is a complex subject and easy to >>>>>>>>>> get wrong. It would be significantly better to figure it out at libvirt >>>>>>>>>> level since it already deals with cross-version migration issues. >>>>>>>>> >>>>>>>>> Libvirt considers vhost-user to be a blackbox though - it just exposes >>>>>>>>> a UNIX socket, and whatever is on the other end is completely opaque. >>>>>>>>> The fact that the other end might plumb the data stream into openvswitch >>>>>>>>> is not something libvirt should know, as we don't want to end up having >>>>>>>>> to add custom code to libvirt for every different vhost-user server >>>>>>>>> impl. >>>>>>>>> >>>>>>>>> IOW, if the version str can be passed to QEMU, and then onto vhost-user >>>>>>>>> backend in QEMU, then libvirt can be involved. If the version str has >>>>>>>>> to be given to openvswitch that's not for libvirt to deall with. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Daniel >>>>>>>> >>>>>>>> It's not just about passing it to QEMU. The following is needed: >>>>>>>> - need to query version when creating VM/device >>>>>>>> - need to query supported versions when migrating VM >>>>>>> >>>>>>> Those are both things that nova can do, since it knows what the vhost-user >>>>>>> device in question is connected to, and so can query the versions, and check >>>>>>> versions before triggering migration with libvirt >>>>>> >>>>>> It can, but then it will need to query libvirt or source for the version >>>>>> string since it's in the XML. >>>>> >>>>> No, it wouldn't be in the XML at all. Nova on the source queries what >>>>> vhostuser version it has and what the target host has, and can prevent >>>>> the migration if they're incompatible. >>>> >>>> This is not sufficient. Exactly the same as qemu machine type, >>>> this must be preserved from time of install and moved >>>> wherever VM goes. >>> >>> Nova has the ability todo that. >>> >>>>> I dont think libvirt has to be >>>>> involved at all for this, as all the info can be obtained by nova/os-vif >>>>> from the vhostuser impl it has configured. >>>> >>>> Given we are already confused at libvirt level, I find it >>>> highly unlikely upper levels will do the right thing. >>> >>> The only confusion is about what must consume the data. I mistakenly >>> thought it was being proposed that the qemu vhostuser backend was >>> being given a version string. Now it seems the version info must be >>> set against the 3rd party component that vhostuser connects to, and >>> that is outside scope of libvirt. That is entirely managed by Nova >>> today, so Nova is the right thing to manage this versioning info. >> >> This is my understanding, with as example using Nova: >> 1. Before starting the VM, Nova queries source host's OVS and all >> possible destination hosts' OVS to retrieve supported versions >> 2. Nova chooses the first common version supported by all hosts >> 3. Nova create the dpdkvhostuser interfaces in OVS with passing the >> selected version as parameter >> >> That should be enough, right? Or maybe I miss something? > > It isn't realistic to check all possible destination hosts when first > starting QEMU, not least becasue they may not yet exist. New hardware > may be deployed *after* QEMU is first started. > > When starting the VM, it should just default to using whatever the > latest version is on the host in question. > > There can be an optional host configuration parameter that admins can > set to force use of an older version. > > This is how Nova deals with machine type - it defaults to latest, but > you can set hw_machine_type in nova.conf to force an older version if > you need migration compatibility between hosts with varying versions. Ok, I see. I thought it could be automated somehow if a list of available host is already present. > At time of migration, Nova need to request the version of the target > host. If this version is not compatible with what the VM is currently > using, then Nova should *not* start the migration, it should raise > an error such that the scheduler retries and picks a different target > host Thanks for the clarification. Maxime