From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <maxime.coquelin@redhat.com>
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28])
 by dpdk.org (Postfix) with ESMTP id 322DE1BA68
 for <dev@dpdk.org>; Tue, 18 Dec 2018 15:02:02 +0100 (CET)
Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com
 [10.5.11.12])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.redhat.com (Postfix) with ESMTPS id 46A0446263;
 Tue, 18 Dec 2018 14:02:01 +0000 (UTC)
Received: from [10.36.112.55] (ovpn-112-55.ams2.redhat.com [10.36.112.55])
 by smtp.corp.redhat.com (Postfix) with ESMTPS id 4A19767164;
 Tue, 18 Dec 2018 14:01:59 +0000 (UTC)
To: Xiao Wang <xiao.w.wang@intel.com>, tiwei.bie@intel.com
Cc: alejandro.lucero@netronome.com, dev@dpdk.org, zhihong.wang@intel.com,
 xiaolong.ye@intel.com
References: <20181213100910.13087-2-xiao.w.wang@intel.com>
 <20181214211612.167681-1-xiao.w.wang@intel.com>
From: Maxime Coquelin <maxime.coquelin@redhat.com>
Message-ID: <6c0c0bba-b395-6b3a-ac14-556b1c251e5c@redhat.com>
Date: Tue, 18 Dec 2018 15:01:57 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.3.1
MIME-Version: 1.0
In-Reply-To: <20181214211612.167681-1-xiao.w.wang@intel.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
 (mx1.redhat.com [10.5.110.29]); Tue, 18 Dec 2018 14:02:01 +0000 (UTC)
Subject: Re: [dpdk-dev] [PATCH v4 00/10] support SW assisted VDPA live
	migration
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Dec 2018 14:02:02 -0000



On 12/14/18 10:16 PM, Xiao Wang wrote:
> In the previous VDPA implementation we have enabled live migration support
> by HW accelerator doing all the stuff, including dirty page logging and
> device status report/restore. In this mode VDPA sample daemon and device
> driver just takes care of the control path and does not involve in data
> path, so there's almost 0 CPU resource usage. This mode requires device
> to have dirty page logging capability.
> 
> This patch series adds live migration support for devices without logging
> capability. VDPA driver could set up a relay thread standing between the
> guest and device when live migration happens, this relay intervenes into
> the communication between guest virtio driver and physical virtio
> accelerator, it helps device to do a vring relay and passingly log dirty
> pages. Thus some CPU resource will be consumed in this scenario, percentage
> depending on the network throughput.
> 
> Some new helpers are added into vhost lib for this VDPA SW fallback:
> - rte_vhost_host_notifier_ctrl, to enable/disable the VDPA direct-IO
>    datapath.
> - rte_vdpa_relay_vring_avail, to relay the available request from guest vring
>    to mediate vring.
> - rte_vdpa_relay_vring_used, to relay the used response from mediate vring to
>    guest vring.
> 
> Some existing helpers are also leveraged for SW fallback setup, like VFIO
> interrupt configuration, IOMMU table programming, etc.
> 
> This patch enables this SW assisted VDPA live migration in ifc driver.
> Since ifcvf also supports HW dirty page logging, we add a new devarg
> for user to select if the SW mode is used or not.
> 
> v4:
> * Add a patch to remove the unused vhost internal API: vhost_detach_vdpa_device().
> 
> v3:
> * Fix indent in relay code.
> * Fix the iova access mode issue of buffer check.
> * Rename the relay API to be more generic, and add more API note for used
>    ring handling.
> * Add kvargs lib dependency in ifc driver.
> * Add commit message for the doc update patch for checkpatch warning.
> 
> v2:
> * Reword the vdpa host notifier control API comment.
> * Make the vring relay API parameter as "void *" to accomodate the future
>    potential new ring layout, e.g. packed ring.
> * Add parameter check for the new API.
> * Add memory barrier for ring idx update.
> * Remove the used ring logging in the relay.
> * Some comment fix and code cleaning according to Tiwei's comment.
> * Add release note update.
> 
> Xiao Wang (10):
>    vhost: remove unused internal API
>    vhost: provide helper for host notifier ctrl
>    vhost: provide helpers for virtio ring relay
>    net/ifc: dump debug message for error
>    net/ifc: store only registered device instance
>    net/ifc: detect if VDPA mode is specified
>    net/ifc: add devarg for LM mode
>    net/ifc: use lib API for used ring logging
>    net/ifc: support SW assisted VDPA live migration
>    doc: update ifc NIC document
> 
>   doc/guides/nics/ifc.rst                |   8 +
>   doc/guides/rel_notes/release_19_02.rst |   6 +
>   drivers/net/ifc/Makefile               |   1 +
>   drivers/net/ifc/base/ifcvf.h           |   1 +
>   drivers/net/ifc/ifcvf_vdpa.c           | 461 ++++++++++++++++++++++++++++++---
>   lib/librte_vhost/rte_vdpa.h            |  57 ++++
>   lib/librte_vhost/rte_vhost_version.map |   3 +
>   lib/librte_vhost/vdpa.c                | 194 ++++++++++++++
>   lib/librte_vhost/vhost.c               |  13 -
>   lib/librte_vhost/vhost.h               |  41 ++-
>   lib/librte_vhost/vhost_user.c          |   7 +-
>   lib/librte_vhost/virtio_net.c          |  39 ---
>   12 files changed, 741 insertions(+), 90 deletions(-)
> 


Applied to dpdk-next-virtio

Thanks,
Maxime