From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CB3D842DE3; Thu, 6 Jul 2023 09:24:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A03AA40A79; Thu, 6 Jul 2023 09:24:58 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id DD1C340144 for ; Thu, 6 Jul 2023 09:24:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688628297; x=1720164297; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=T3g+gGujxEIIZ3R4t791WMwpbn3goqwOO3XcblAh2fc=; b=CNg37ujqR1meg5jRM7SInxkvVVCFZKTe0GU35aulNJ8nXwo0EWtdoYKx Z3h2KEeK9gpHmS7JyonqZzk0Yjc+0zi47NhlmnXCliKNiP5ZnhxyeUvpY c/k692lShlq0KvhqhuHBa38AGNSbNYB2MW5GCidQ70IfPaeCmq+gBM1wY 1B3Z0iaDWa2p1q4Hz/mddKL9YZqFu8nPxegPhqo2zZ9RyoA9d/KMHhBVw 6RNdutW0iN9FpaDXChHyZDDIGs1/e3fVRJpOR3Tn7rI9SBjedFO6gT80r xZU/5lgdFEmTZg+h1lGPWP19PuK22uziaqHM3ZmJXvxzW6z1q/1CXnq3X g==; X-IronPort-AV: E=McAfee;i="6600,9927,10762"; a="343858093" X-IronPort-AV: E=Sophos;i="6.01,185,1684825200"; d="scan'208";a="343858093" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jul 2023 00:24:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10762"; a="843571738" X-IronPort-AV: E=Sophos;i="6.01,185,1684825200"; d="scan'208";a="843571738" Received: from dpdk-liulingy-2.sh.intel.com ([10.67.119.34]) by orsmga004.jf.intel.com with ESMTP; 06 Jul 2023 00:24:53 -0700 From: Lingyu Liu To: dev@dpdk.org, yahui.cao@intel.com, qi.z.zhang@intel.com, beilei.xing@intel.com Cc: Lingyu Liu Subject: [PATCH v3] doc: add iavf live migration guide Date: Thu, 6 Jul 2023 07:24:48 +0000 Message-Id: <20230706072448.30063-1-lingyu.liu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230704022718.227538-1-lingyu.liu@intel.com> References: <20230704022718.227538-1-lingyu.liu@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add iavf live migration steps based on KVM VFIO migration. Signed-off-by: Lingyu Liu --- v2: Fixed CI. Added brief introduction about live migration. Clarified this is iavf feature. v3: Added intro and link about vfio live migration. Added description about kernel boot parameters. Changed to use sysfs to bind device to driver. Noted for running dpdk-testpmd. Highlighted KVM vfio migration. --- doc/guides/nics/intel_vf.rst | 113 +++++++++++++++++++++++++ doc/guides/rel_notes/release_23_07.rst | 3 + 2 files changed, 116 insertions(+) diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst index d365dbc185..8c24485bdd 100644 --- a/doc/guides/nics/intel_vf.rst +++ b/doc/guides/nics/intel_vf.rst @@ -622,6 +622,119 @@ which belongs to the destination VF on the VM. Inter-VM Communication +Live Migrating a VM running DPDK +-------------------------------- + +Live migration refers to the process of moving a running virtual machine (VM) or application +between different physical machines without disconnecting the client or application +(see https://en.wikipedia.org/wiki/Live_migration for more information). + +VFIO device migration refers to migrating a VM which have VFIO device pass-through +(see https://qemu.readthedocs.io/en/latest/devel/vfio-migration.html for more information). + +This part describes stpes to migrate a VM which has a iavf device pass through. + +The following describes a target environment: + +* Host Operating System: Ubuntu 20.04.5 + +* Guest Operating System: Ubuntu 20.04.5 + +* Linux Kernel Version: 5.15.0-72-generic + +* Target Applications: dpdk-testpmd + +* Ice Kernel Driver Version: 1.11.17.1 ``_ + +* Qemu Version: 7.2 + +The setup procedure is as follows: + +#. Before booting the Host OS, open **BIOS setup** and enable **Intel® VT features**. + +#. While booting the Host OS kernel, pass the intel_iommu=on kernel command line argument using GRUB. + +#. In the Host OS + + Install the ice driver and migration driver: + + .. code-block:: console + + insmod ice.ko + insmod ice-vfio-pci.ko + + Create 2 VFs and bind them to vfio pci driver: + + .. code-block:: console + + echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs + echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id + echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind + echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind + echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind + echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind + + .. note:: + + The command above creates two vfs for device 0000:ca:00.1: + + .. code-block:: console + + 0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf + 0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf + +#. Now, start the migration source Virtual Machine by running the following command: + + .. code-block:: console + + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio + + .. note:: + The vfio-pci,host=0000:ca:11.0 value indicates that you want to attach a vfio PCI device + to a Virtual Machine and the respective (Bus:Device.Function) numbers should be passed, + x-enable-migration=true indicates that this VF supports migration. Dirty page tracking + is not supported, so set x-pre-copy-dirty-page-tracking=off. + +#. In VM, install iavf driver and vfio-pci driver + + .. code-block:: console + + insmod iavf.ko + modprobe vfio enable_unsafe_noiommu_mode=1 + moodprobe vfio-pci + +#. Bind net device to vfio-pci driver and launch dpdk-testpmd + + .. code-block:: console + + dpdk-testpmd -l 0-1 -- -i + testpmd> set txpkts 64 + testpmd> start tx_first + + .. note:: + Please ensure dpdk-testpmd to run independently of ssh console. + Suggest to put it in a background process like tmux/screen so that + migration will not causing ssh console exit and dpdk-testpmd killed. + +#. Start the migration destination Virtual Machine + + .. code-block:: console + + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5556-:22 -monitor stdio -incoming tcp:127.0.0.1:4444 + + #. Start migration by issuing the command in qemu console + + .. code-block:: console + + migrate -d tcp:127.0.0.1:4444 + +#. Log in the destination VM, and dpdk-testpmd is not interrupt + +.. note:: + + Host kernel version above 5.17 is currently not supported due to kAPI change. + + Windows Support --------------- diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst index 4459144140..794a86eba3 100644 --- a/doc/guides/rel_notes/release_23_07.rst +++ b/doc/guides/rel_notes/release_23_07.rst @@ -200,6 +200,9 @@ New Features Enhanced the GRO library to support TCP packets over IPv6 network. +* **Updated Intel iavf driver.** + + Added the document for iavf driver live migration based on KVM vfio migration. Removed Items ------------- -- 2.25.1