Re: [PATCH V2] doc: add iavf live migration guide

["Cao, Yahui" <yahui.cao@intel.com>]

Hi Lingyu,

On 7/4/2023 10:27 AM, Lingyu Liu wrote:
> Describe iavf live migration steps.
>
> Signed-off-by: Lingyu Liu <lingyu.liu@intel.com>
> ---
>    v2: Fixed CI.
>        Added brief introduction about live migration.
>        Clarified this is iavf feature.
> ---
>   doc/guides/nics/intel_vf.rst           | 101 +++++++++++++++++++++++++
>   doc/guides/rel_notes/release_23_07.rst |   3 +
>   2 files changed, 104 insertions(+)
>
> diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst
> index d365dbc185..dd75aadf51 100644
> --- a/doc/guides/nics/intel_vf.rst
> +++ b/doc/guides/nics/intel_vf.rst
> @@ -622,6 +622,107 @@ which belongs to the destination VF on the VM.
>      Inter-VM Communication
>   
>   
> +Live Migrating a VM running DPDK
> +--------------------------------
> +
> +Live migration refers to the process of moving a running virtual machine (VM) or application
> +between different physical machines without disconnecting the client or application
> +(see https://en.wikipedia.org/wiki/Live_migration for more information).
> +


Please also add some introduction about the vfio migration and the link 
for qemu vfio migration doc: 
https://qemu.readthedocs.io/en/latest/devel/vfio-migration.html


> +This part describes how to migrate a Virtual Machine which has a iavf device.
> +
> +The following describes a target environment:
> +
> +*   Host Operating System: Ubuntu 20.04.5
> +
> +*   Guest Operating System: Ubuntu 20.04.5
> +
> +*   Linux Kernel Version: 5.15.0-72-generic
> +
> +*   Target Applications: dpdk-testpmd
> +
> +*   Ice Kernel Driver Version: 1.11.17.1 `<https://www.intel.com/content/www/us/en/download/19630>`_
> +
> +*   Qemu Version: 7.2
> +
> +The setup procedure is as follows:
> +
> +#.  Before booting the Host OS, open **BIOS setup** and enable **Intel® VT features**.
> +


Can you add some description about kernel boot parameters?


> +#.  In the Host OS
> +
> +    Install the ice driver and migration driver:
> +
> +    .. code-block:: console
> +
> +        insmod ice.ko
> +        insmod ice-vfio-pci.ko
> +
> +    Create 2 VFs and bind them to vfio pci driver:
> +
> +    .. code-block:: console
> +
> +        echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs
> +        echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id
> +        dpdk-devbind.py -b ice-vfio-pci 0000:ca:11.0
> +        dpdk-devbind.py -b ice-vfio-pci 0000:ca:11.1


ice-vfio-pci binding is not supported on dpdk-devbind.py yet. Please use 
sysfs to bind device and driver


> +
> +    .. note::
> +
> +        The command above creates two vfs for device 0000:ca:00.1:
> +
> +    .. code-block:: console
> +
> +        0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
> +        0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
> +
> +#.  Now, start the migration source Virtual Machine by running the following command:
> +
> +    .. code-block:: console
> +
> +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio
> +
> +    .. note::
> +        The vfio-pci,host=0000:ca:11.0 value indicates that you want to attach a vfio PCI device
> +        to a Virtual Machine and the respective (Bus:Device.Function) numbers should be passed,
> +        x-enable-migration=true indicates that this VF supports migration. Dirty page tracking
> +        is not supported, so set x-pre-copy-dirty-page-tracking=off.
> +
> +#.  In VM, install iavf driver and vfio-pci driver
> +
> +    .. code-block:: console
> +
> +        insmod iavf.ko
> +        modprobe vfio enable_unsafe_noiommu_mode=1
> +        moodprobe vfio-pci
> +
> +#.  Bind net device to vfio-pci driver and launch dpdk-testpmd
> +
> +    .. code-block:: console
> +


We need to ensure dpdk-testpmd should be running independently of ssh 
console and suggest to put it in a background process like tmux/screen 
so that migration will not causing ssh console exit and dpdk-testpmd 
killed. Please add related description.


> +        dpdk-testpmd -l 0-1 -- -i
> +        testpmd> set txpkts 64
> +        testpmd> start tx_first
> +
> +#. Start the migration destination Virtual Machine
> +
> +    .. code-block:: console
> +
> +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5556-:22 -monitor stdio -incoming tcp:127.0.0.1:4444
> +
> + #. Start migration by issuing the command in qemu console
> +
> +    .. code-block:: console
> +
> +        migrate -d tcp:127.0.0.1:4444
> +
> +#. Log in the destination VM, and dpdk-testpmd is not interrupt
> +
> +.. note::
> +
> +        Host kernel version above 5.17 is currently not supported due to kAPI change.
> +
> +
>   Windows Support
>   ---------------
>   
> diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
> index 4459144140..a187cb6146 100644
> --- a/doc/guides/rel_notes/release_23_07.rst
> +++ b/doc/guides/rel_notes/release_23_07.rst
> @@ -200,6 +200,9 @@ New Features
>   
>     Enhanced the GRO library to support TCP packets over IPv6 network.
>   
> +* **Updated Intel iavf driver.**
> +
> +  Added the document for iavf driver live migration.


Suggest to highlight iavf driver live migration based on KVM vfio migration.


>   
>   Removed Items
>   -------------